You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Current state: ["Under Discussion"]

ISSUE: #17754

PRs: 

Keywords: IDMAP, brute force search, chunk

Released: Milvus-v2.2.0

Summary(required)

In this MEP, we put forward an IDMAP Enhancement proposal that let knowhere index type IDMAP accept external vector data instead of adding real vector data in.

This Enhanced IDMAP can be used for growing segment searching to improve code reuse and reduce code maintenance effort.

Motivation(required)

Generally no one will create IDMAP index type for sealed segment, because it does not bring any search performance improvement but consumes identical size of memory and disk.

The only reasonable use scenario for IDMAP is for growing segment. But creating vector index is a resource consuming operation, because it involves all Milvus nodes in -- an index file is created by index node, saved by data node and loaded by query node, meanwhile proxy / rootcoord / indexcoord / datacoord / querycoord are also involved to coordinate all these operations. 

So currently in Milvus, it uses chunk index (this functionality is disabled for some particular reason) and brute force search (most part of the API realization is copied from knowhere IDMAP) for growing segment searching.

  • Advantages


  • Cons

Public Interfaces(optional)

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed.

Design Details(required)

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgment based on the scope of the change.

Compatibility, Deprecation, and Migration Plan(optional)

  • What impact (if any) will there be on existing users?
  • If we are changing behaviors how will we phase out the older behavior?
  • If we need special migration tools, describe them here.
  • When will we remove the existing behavior?

Test Plan(required)

Describe in a few sentences how the MEP will be tested. We are mostly interested in system tests (since unit tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives(optional)

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other ways.

References(optional)

Briefly list all references

  • No labels