...
Public Interfaces(optional)
Faiss needs add new field "xb_ex" and new interface "add_ex" for structure IndexBinaryFlat; also add new field "codes_ex" and new interface "add_ex" for structure IndexFlat.
In IndexBinaryFlat, "xb" and "xb_ex" are mutual exclusive, user cannot set them at the same time; it's same in IndexFlat, "codes" and "codes_ex" are also mutual exclusive, user cannot set both of them.
| Code Block | ||
|---|---|---|
| ||
//============================================================================
struct IndexBinaryFlat : IndexBinary {
/// database vectors, size ntotal * d / 8
std::vector<uint8_t> xb;
/// external database vectors, size ntotal * d / 8
uint8_t* xb_ex = nullptr; // <==== new added
... ...
}
void IndexBinaryFlat::add_ex(idx_t n, const uint8_t* x) {
xb_ex = (uint8_t*)x;
ntotal = n;
}
//============================================================================
struct IndexFlatCodes : Index {
... ...
/// encoded dataset, size ntotal * code_size
std::vector<uint8_t> codes;
// external encoded dataset , size ntotal * code_size
uint8_t* codes_ex = nullptr; // <==== new added
... ...
}
void IndexFlatCodes::add_ex(idx_t n, const float* x) {
FAISS_THROW_IF_NOT(is_trained);
FAISS_THROW_IF_NOT(codes.empty());
codes_ex = (uint8_t*)x;
ntotal = n;
} |
Knowhere needs add a new interface `AddExWithoutIds()` for both IDMAP and BinaryIDMAP.
| Code Block | ||
|---|---|---|
| ||
// set external data pointer instead really add data
void
AddExWithoutIds(const DatasetPtr&, const Config&); |
When Knowhere detect that "codes_ex" is used in current IDMAP index or "xb_ex" is used in current BinaryIDMAP index, serialization is banned.
For Milvus, API "FloatSearchBruteForce()" and "BinarySearchBruteForce()" will be re-written, they can use the enhanced IDMAP/BinaryIDMAP to search instead of calling Faiss interfaces.
Design Details(required)
In Milvus, when growing segment need create an enhanced IDMAP index, it can do in this way:
...
| language | cpp |
|---|
...
)
...
Compatibility, Deprecation, and Migration Plan(optional)
...