Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Current state: Accepted



Keywords: Kafka

Released: with Milvus 2.1 



The log broker is a pub-sub system within Milvus, It is responsible for streaming data persistence,  event notification, recovery etc.  Now Milvus cluster mode uses Pulsar as a log broker, and standalone mode uses RocksDB. 

Apache Kafka is a distributed event store and stream-processing platform, and it is a popular solution for data streaming needs.  Many community users expect Milvus to support Kafka because they have already used it in the production environment.

Image Source


 Milvus supports Kafka as a message stream,  we can use the configuration option to decide to use Pulsar or Kafka on cluster mode.

  address: localhost # Address of pulsar
  port: 6650 # Port of pulsar
  maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.

  address: localhost
  port: 9092

Design Details

  • add kafka and zookepper dev docker
  • optimization mq_factory configuration initialization
  • remove reader
  • implement msg_stream with kafka 


Kafka Client SDK

  • Sarama 
  • confluent-kafka-go

Interface Implementation


  • standalone
    • docker
  • Cluster
    • Helm Chart
    • Operator

Test Plan

  • pass the unittest
  • performance testing

  • No labels