...
In consideration of the data size, especially when inserting or and search result returning search results, Milvus takes considerable efforts a lot of CPU cycles to do serialization and deserialization.
In this enhancement proposal, we suggest to adopt Apache Arrow as Milvus in-memory data format. Since in the field of big data, Apache Arrow has been a
factor standard to solve this kind of problems.
...
for in-memory analytics.
...
It specifies a standardized language-independent columnar memory format
...
.
...
Zero-copy shared memory and RPC-based data movement
...
Reading and writing file formats (like CSV, Apache ORC, and Apache Parquet)
...
Motivation(required)
From a data perspective, Milvus includes 2 data flows mainly:
...