Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
Discovery Impl 1.1.0
-
None
-
None
Description
Note: This is a fork of SLING-3432 based on a comment. So here is that comment again:
Note that the above does not solve the problem where the underlying repository is eventually consistent and the heartbeat configured is too low to catch all possible delays (that such an eventually consistent repository might produce under load). Consider the following:
- a cluster consisting of 3 nodes: A, B and C, A is the leader
- writes from B and C are fast - and can be read by all 3 nodes fast
- writes from A though are slow (ie A behaves asymmetric: slow writes but fast reads)
- at some point writes from A are slower than the configured heartbeat timeout: at this point B and C find out about this and vote on a new clusterView consisting only of B and C and (let's say) B becomes leader.
- meanwhile at A however: A is still happy: it sees the heartbeats of B and C in time and would not start a new voting.
- at some later point (with a certain read delay) A sees that B and C have declared a new /establishedViews - at this point it would (according to the new rule above) immediately send a TOPOLOGY_CHANGING and things would be 'ok' again.
- but until it does send this event - between 4. and 5. - we have two leaders: A and B! -> thus could see issues reported here in
SLING-3432still during that small timeframe (which is basically the amount of time it takes for the new established view declared by B and C to be read by A). - at a later time, when eg the delays in the repository have come down, A would rejoin the cluster - but would have to not become leader again, as the leader is B and must stay stable.
- but until it does send this event - between 4. and 5. - we have two leaders: A and B! -> thus could see issues reported here in
This IMHO highlights the problem that using an eventually consistent repository (that has no max guaranteed delay) is not pseudo-network-partition/duplicate-leader free under load.
Note that what is described here is not fixed by SLING-4627.
Attachments
Issue Links
- is related to
-
SLING-4627 TOPOLOGY_CHANGED in an eventually consistent repository
- Closed
- is required by
-
SLING-3432 pseudo network partition causes job deserialization issue in a cluster (when reading while job is being reassigned)
- Closed
- is superceded by
-
SLING-4603 discovery.oak: oak-based discovery implementation
- Closed
- relates to
-
SLING-5195 HeartbeatHandler should check for topology changes independently of long-during session.saves
- Closed