Current state: Accepted
ISSUE: https://github.com/milvus-io/milvus/issues/5218
PRs:
Keywords: Kafka
Released: with Milvus 2.1
Authors:
Motivation
The log broker is a pub-sub system within Milvus, It is responsible for streaming data persistence, event notification, recovery etc. Now Milvus cluster mode uses Pulsar as a log broker, and standalone mode uses RocksDB.
Apache Kafka is a distributed event store and stream-processing platform, and it is a popular solution for data streaming needs. Many community users expect Milvus to support Kafka because they have already used it in the production environment.
Summary
Milvus supports Kafka as a message stream, we can use the configuration option to decide to use Pulsar or Kafka on cluster mode.
Design Details
- add kafka and zookepper dev docker
- optimization mq_factory configuration initialization
- remove reader
- implement msg_stream with kafka
Configuration
Kafka Client SDK
- Sarama
- confluent-kafka-go
Interface Implementation
Deployments
- standalone
- docker
- Cluster
- Helm Chart
- Operator
Test Plan
- pass the unittest
- performance testing