Current state: Accepted
Released: with Milvus 2.1
The log broker is a pub-sub system within Milvus, It is responsible for streaming data persistence, event notification, recovery etc. Now Milvus cluster mode uses Pulsar as a log broker, and standalone mode uses RocksDB.
Apache Kafka is a distributed event store and stream-processing platform, and it is a popular solution for data streaming needs. Many community users expect Milvus to support Kafka because they have already used it in the production environment.
Milvus supports Kafka as a message stream, we can use the configuration option to decide to use Pulsar or Kafka on cluster mode.
- add kafka and zookepper dev docker
- optimization mq_factory configuration initialization
- remove reader
- implement msg_stream with kafka
Kafka Client SDK
- Helm Chart
- pass the unittest
- performance testing