Current state: Under Discussion

ISSUE: https://github.com/milvus-io/milvus/issues/6578

PRs: 

Keywords: etcd, datacoord, datanode

Released:

Summary

DataCoord register channels on etcd and DataNode watch etcd to do watch/release operations.

Motivation

There are several problems when DataCoord sends the WatchDmChannel to the DataNode through grpc:

  1. If datacoord cannot connect to datanode, it needs to try again. Retrial failure requires reallocation, which may result in duplicate watches.

  2. If datacoord has load balance, it needs to send unwatch and watch request, which may also lead to failure and retrying.

Public Interfaces

Remove WatchDmChannel of DataNode.

Design Details

Etcd key：channel / [nodeID] / [channelName]，value: ChannelInfo

ChanelInfo contains State,StartTime, VchannelInfo：

  1. State is a enum whose values are Unwatched, Watched. This means whether datanode watch it successfully.
  2. StartTime is the watch event start time.
  3. VchannelInfo contains all info needed to restore the channel.

If there is a new channel registration, datacoord updates channel / [nodeid] / [channelname]

Datanode monitors the ADD and DELETE events of channel / [nodeid]

DataCoord：

  1. When the datacoord is started, the channels of offline datanodes are assigned to current online nodes.
  2. When DataNode comes online, DataCoord may move some channels to the node and change the channels of different nodes through etcd transactions operation.
  3. When DataNode goes offline, DataCoord reassigns the channels to other nodes, changing them through the etcd transaction.
  4. Specially, if the last DataNode goes offline and there is no living DataNode at this time, record the channel in channel/remaining/[channelName].
  5. Start a background goroutine to check states of channels. If a channel's state has't changed to Watched for a long time, maybe we should reallocated it to another node atomically.

DataNode: 

  1. When DataNode starts, the channel of this Node on etcd must be empty, because the nodeID is incremented.
  2. When DataNode receives an Add event, execute WatchChannel, and transactionlly change state of channel on etcd to Watched.
  3. When DataNode receives Delete event, execute ReleaseChannel.

Test Plan


  • No labels

9 Comments

  1. Xiaofan

    Questions1 : how many states are there on etcd of the channel? can we have a state machine? 

    1. Xiaofan

      better to have a diagram illustrate state transformation

      1. bingyi.sun@zilliz.com

        For datanode, if a path on etcd like channel/[nodeID=1]/[channelName=vchan1] exists, then it means node whose id is 1 should watch the channel vchan1. And the value of this key is the channel information,eg: seek position. If the node crashes or some balance happens, we will use a transaction operation to delete the key and add a key like channel/[nodeID=2]/[channelName=vchan1] for node2. So for channel, there is no state.

  2. Xiaofan

    Questions 2: what happens if etcd add,delete multiple times? does etcd ganruatee watch are delivered in order?

    1. bingyi.sun@zilliz.com

      etcd does guarantee the order of operations 

      From etcd docs:
      Watches make three guarantees about events:

      • Ordered - events are ordered by revision; an event will never appear on a watch if it precedes an event in time that has already been posted.
      • Reliable - a sequence of events will never drop any subsequence of events; if there are events ordered in time as a < b < c, then if the watch receives events a and c, it is guaranteed to receive b.
      • Atomic - a list of events is guaranteed to encompass complete revisions; updates in the same revision over multiple keys will not be split over several lists of events.

  3. Xiaofan

    Q3: Is that possible that dataNode receives delete while it is still loading the channel? Do we interrupt it , or wait until load done?

    1. bingyi.sun@zilliz.com

      We should stop loading immediately. If we wait util load done, it will may spend too much time. FYI, we should load channel concurrently. 

  4. xige-16

    If datanode watch Dmchannel failed, how to notify dataCoord?

    1. bingyi.sun@zilliz.com

      Great question. I forget to handle this situation. We can add 'Unwatched' 'Watched' state to channel and change it after watching channel successfully. And datacoord starts a background goroutine to check the state of channel. If a channel has't changed for a long time, we can reallocate it to another node.