...
See following "Search Flow" chart, `FloatSearchBruteForce` and `BinarySearchBruteForce` are copied from knowhere::IDMAP/BinaryIDMAP's interface Query() and modified a little. This will introduce more code maintenance effort. And when realize new feature on IDMAP/BinaryIDMAP in Knowhere, such as range search, we have to also copy these codes implementation to Milvus.
One good solution is to If we enhance IDMAP/BinaryIDMAP, not to add real vector data in, but only hold an external vector data pointer in the index, we can use knowhere::IDMAP/BinaryIDMAP's interface Query() directly without any costs. User need guarantee that the memory is contiguous and safe.
...
- no CPU time, memory and disk consumption when creating index
- resource saving, only query node is involved in
- no code duplication for growing segment search
- unified search result for sealed segment and growing segment
Proposal 1 (only take IDMAP as an example)
- Faiss adds new field "codes_ex" and new interface "add_ex" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
- Knowhere adds a new interface `AddExWithoutIds()` for IDMAP.
- In Milvus, re-write API "FloatSearchBruteForce()" and "BinarySearchBruteForce()", let they use enhanced IDMAP to search instead of calling Faiss interfaces.
Advantages: Little code change
Cons: Need add new interfaces in both Faiss and Knowhere
Proposal 2 (only take IDMAP as an example)
Public Interfaces(optional)
...