Current state: Under Discussion
ISSUE: https://github.com/milvus-io/milvus/issues/9481
PRs:
Keywords: load balance, handoff, query node, query coord, sealed segment
Released:
This article focuses on the importance of handoff and load balance segment features, as well as proposing a scenario for handoff and load balance segments without interrupting queries.
The query node of milvus currently has the following issues:
So we want to do the following:
Handoff and load balance are transparent to users.
Query node logically manages the sealed segments and the growing segments in two parts, historical and streaming. As shown in the following figure, G3 represents the growing segment 3, and S8 represents the sealed segment 8.
Based on the diagram above, it can be concluded that the handoff and load balance process consists of two main steps:
However, there are several scenarios to consider in the implementation:
Based on the above considerations, we propose the following designs:
Prerequisite: The cache meta of querycoord records which sealed segments and growing segments are loaded on each query node, and which dmchannels are watched by the query node. These metas are stored synchronously in etcd.
1.First of all, querycoord automatically generates handoff tasks and load balance tasks and writes them into etcd, and then clears them from etcd until the task is completed, ensuring that querycoord can accurately restore the task after restarting.
2. Next, query coord uses its own allocation strategy to determine which querynode the sealed segment of balance or handoff should be allocated to. The allocation strategy considers whether the total size of segments on each query node is balanced, and whether the segments of each collection are evenly distributed across all querynode
3.Querycoord directly loads the allocated sealed segments to the corresponding query node, and then the same segment will exist on different querynodes. We de-duplicate the query result based on the primary key during local reduce and global reduce to avoid affecting the query result.
4.After query coord successfully loads sealed segments on querynode, update the sealed segments list of each querynode in cache meta. For example, after balancing sealed segment 8 from node 1 node 2, the meta of node1 changes from {S6, S7,S8} to { S6, S7}, node2's meta is changed from {S5} to {S5, S8}. If the query node suddenly goes down, directly follow the meta records of the coord to recover the query node. While querycoord updates the meta, it broadcasts a change info of sealed segments to querychannel. The proto of sealed segments change info is as follows:
message SealedSegmentsChangeInfo { common.MsgBase base = 1; int64 online_nodeID = 2; repeated SegmentInfo online_segments = 3; int64 offline_nodeID = 4; repeated SegmentInfo offline_segments = 5; } |
5.query coord 更新meta中的querychannelInfo,