Page History

...

1. Faiss adds new field "codes_ex" and new interface "add_ex()" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
2. Knowhere adds a new interface `AddExWithoutIds()` for IDMAP.
3. In Milvus, re-write API "FloatSearchBruteForce()", let it use enhanced IDMAP to search.

...

1. Faiss adds new field "codes_ex" and new interface "add_ex()" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
2. Knowhere adds a new index type IDMAP_EX, which use API "add_ex()" to insert vector data.
3. In Milvus, re-write API "FloatSearchBruteForce()", let it use new IDMAP_EX to search.

...

Cons: This change dis-obey knowhere's design concept. By now all operations in knowhere is for an index, but this API is for all metric types, not for an index.

Since both Proposal 1 and Proposal 2 need add new interface in Faiss, and Milvus team think this will break Faiss's encapsulation, we adopt Proposal 3.

Public Interfaces(optional)

Add new interface only in Knowhere

/** knowhere wrapper API to call faiss brute force search for all metric types
 *
 * @param metric_type
 * @param xb            training vecors, size nb * dim
 * @param xq            query vecors, size nq * dim
 * @param dim
 * @param nb            rows of training vectors
 * @param nq            rows of query vectors
 * @param topk
 * @param labels        output, memory allocated and freed by caller
 * @param distances     output, memory allocated and freed by coller
 * @param bitset
 */
void
BruteForceSearch(
    const knowhere::MetricType& metric_type,
    const void* xb,
    const void* xq,
    const int64_t dim,
    const int64_t nb,
    const int64_t nq,
    const int64_t topk,
    int64_t* labels,
    float* distances,
    const faiss::BitsetView bitset);

/** knowhere wrapper API to call faiss brute force range search for all metric types
 *
 * @param metric_type
 * @param xb            training vecors, size nb * dim
 * @param xq            query vecors, size nq * dim
 * @param dim
 * @param nb            rows of training vectors
 * @param nq            rows of query vectors
 * @param radius        range search radius
 * @param labels        output, memory allocated inside and freed by caller
 * @param distances     output, memory allocated inside and freed by coller
 * @param lims          output, memory allocated inside and freed by coller
 * @param bitset
 */
void
BruteForceRangeSearch(
    const knowhere::MetricType& metric_type,
    const void* xb,
    const void* xq,
    const int64_t dim,
    const int64_t nb,
    const int64_t nq,
    const float radius,
    int64_t*& labels,
    float*& distances,
    size_t*& lims,
    const faiss::BitsetView bitset);

BruteForceSearch() is wrapper API to call Faiss brute force search for all metric types. The output parameters "labels" and "distances" are pointers to the memory allocated by caller. Caller will also free these memory after use.

BruteForceRangeSearch() is wrapper API to call Faiss brute force range search for all metric types. The output parameters "labels", "distances" and "lims" are reference of pointers to the memory allocated in Faiss. This is because the length of range search result is unknown, caller cannot allocate memory in advance. Caller need free these memory after use.

Design Details(required)

For growing segment in segcore, all brute force search related code can be removed, call knowhere's BruteForceSearch() and BruteForceRangeSearch() instead.

Compatibility, Deprecation, and Migration Plan(optional)

...