Page History

Current state: ["Under DiscussionDeclined"]

ISSUE: #17754

PRs: #240 #243 #18277

Keywords: IDMAP, BinaryIDMAP, brute force search, chunk

...

1. Faiss adds new field "codes_ex" and new interface "add_ex()" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
2. Knowhere adds a new interface `AddExWithoutIds()` for IDMAP.
3. In Milvus, re-write API "FloatSearchBruteForce()", let it use enhanced IDMAP to search.

...

1. Faiss adds new field "codes_ex" and new interface "add_ex()" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
2. Knowhere adds a new index type IDMAP_EX, which use API "add_ex()" to insert vector data.
3. In Milvus, re-write API "FloatSearchBruteForce()", let it use new IDMAP_EX to search.

...

Cons: This change dis-obey knowhere's design concept. By now all operations in knowhere is for an index, but this API is for all metric types, not for an index.

Since both Proposal 1 and Proposal 2 need add new interface in Faiss, and Milvus team think this will break Faiss's encapsulation, we adopt Proposal 3.

Public Interfaces(optional)

Add new interface only in Knowhere

class BruteForce {
 public:
    static DatasetPtr
    Search(const DatasetPtr base_dataset,
           const DatasetPtr query_dataset,
           const Config& config,
           const faiss::BitsetView bitset);

    static DatasetPtr
    RangeSearch(const DatasetPtr base_dataset,
                const DatasetPtr query_dataset,
                const Config& config,
                const faiss::BitsetView bitset);
};

BruteForce::Search() is wrapper API to call Faiss brute force search for all metric types.

BruteForce::RangeSearch() is wrapper API to call Faiss brute force range search for all metric types. 

Design Details(required)

For growing segment in segcore, all brute force search related code can be removed, call knowhere's BruteForceSearch() and BruteForceRangeSearch() instead.

Compatibility, Deprecation, and Migration Plan(optional)

• This MEP will be transparent for users, and will not introduce any compatibility issue.

Test Plan(required)

...

New testcases are added in Knowhere to test "BruteForceSearch()" and "BruteForceRangeSearch()".

• in knowhere/unittest/test_bruteforce.cpp

No extra testcases need be added in Milvus because current growing segment search testcases can cover this change.

...