Current state: Under Discussion
Keywords: etcd, datacoord, datanode
DataCoord register channels on etcd and DataNode watch etcd to do watch/release operations.
There are several problems when DataCoord sends the WatchDmChannel to the DataNode through grpc:
- If datacoord cannot connect to datanode, it needs to try again. Retrial failure requires reallocation, which may result in duplicate watches.
If datacoord has load balance, it needs to send unwatch and watch request, which may also lead to failure and retrying.
Remove WatchDmChannel of DataNode.
Etcd key：channel / [nodeID] / [channelName]，value: ChannelInfo
ChanelInfo contains State,StartTime, VchannelInfo：
- State is a enum whose values are Unwatched, Watched. This means whether datanode watch it successfully.
- StartTime is the watch event start time.
- VchannelInfo contains all info needed to restore the channel.
If there is a new channel registration, datacoord updates channel / [nodeid] / [channelname]
Datanode monitors the ADD and DELETE events of channel / [nodeid]
- When the datacoord is started, the channels of offline datanodes are assigned to current online nodes.
- When DataNode comes online, DataCoord may move some channels to the node and change the channels of different nodes through etcd transactions operation.
- When DataNode goes offline, DataCoord reassigns the channels to other nodes, changing them through the etcd transaction.
- Specially, if the last DataNode goes offline and there is no living DataNode at this time, record the channel in channel/remaining/[channelName].
- Start a background goroutine to check states of channels. If a channel's state has't changed to Watched for a long time, maybe we should reallocated it to another node atomically.
- When DataNode starts, the channel of this Node on etcd must be empty, because the nodeID is incremented.
- When DataNode receives an Add event, execute WatchChannel, and transactionlly change state of channel on etcd to Watched.
- When DataNode receives Delete event, execute ReleaseChannel.
Questions1 : how many states are there on etcd of the channel? can we have a state machine?
better to have a diagram illustrate state transformation
For datanode, if a path on etcd like channel/[nodeID=1]/[channelName=vchan1] exists, then it means node whose id is 1 should watch the channel vchan1. And the value of this key is the channel information,eg: seek position. If the node crashes or some balance happens, we will use a transaction operation to delete the key and add a key like channel/[nodeID=2]/[channelName=vchan1] for node2. So for channel, there is no state.
Questions 2: what happens if etcd add,delete multiple times? does etcd ganruatee watch are delivered in order?
etcd does guarantee the order of operations
From etcd docs:
Watches make three guarantees about events:
Q3: Is that possible that dataNode receives delete while it is still loading the channel? Do we interrupt it , or wait until load done?
We should stop loading immediately. If we wait util load done, it will may spend too much time. FYI, we should load channel concurrently.
If datanode watch Dmchannel failed, how to notify dataCoord?
Great question. I forget to handle this situation. We can add 'Unwatched' 'Watched' state to channel and change it after watching channel successfully. And datacoord starts a background goroutine to check the state of channel. If a channel has't changed for a long time, we can reallocate it to another node.