You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Current state: Accepted



Keywords: Kafka

Released: with Milvus 2.1 



 Milvus supports Kafka as a message stream,  we can use the configuration option to decide to use Pulsar or Kafka on cluster mode.


The log broker is a pub-sub system within Milvus, It is responsible for streaming data persistence,  event notification, recovery etc.  Now Milvus cluster mode uses Pulsar as a log broker, and standalone mode uses RocksDB. 

Apache Kafka is a distributed event store and stream-processing platform, and it is a popular solution for data streaming needs.  Many community users expect Milvus to support Kafka because they have already used it in the production environment.

Design Details

  • add kafka and zookepper dev docker
  • optimization mq_factory configuration initialization
  • remove reader
  • implement msg_stream with kafka 


Kafka Client SDK

  • Sarama 
  • confluent-kafka-go

Interface Implementation


  • standalone
    • docker
  • Cluster
    • Helm Chart
    • Operator

Test Plan

  • pass the unittest
  • performance testing

  • No labels