Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

Reference: https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html

It would be nice to have a similar functionality for Nimbus/Pacemaker clusters too.
As that would eliminate the need for restarting servers in the Nimbus/Pacemaker clusters whenever a node exits or joins these clusters.

----------------------------------------------------------
Reply from Bobby Evans on the dev group:
----------------------------------------------------------

There is nothing for that right now on pacemaker.
You can do it with nimbus so long as at least one of the original nodes is still up.

But in either case it would not be too difficult to make it all fully functional.
The two critical pieces would be in giving the workers and daemons a way to reload these specific configs dynamically.
Then it would be documenting the order of operations to be sure nothing goes wrong.

Adding Pacemaker Node(s)

1. bring up the new node(s).
2. update nimbus configs to start reading from the new nodes.
3. update all of the worker nodes to let workers start writing to the new node.

Removing Pacemaker Node(s)

1. Shut down pacemaker nodes/update configs on workers (order should not matter so long as there are enough pacemaker nodes up to handle the load)
2. update the nimbus configs to not try and read from the old nodes

Adding new Nimbus Node(s)

1. Bring up the new nimbus with the new config.
2. update all of the other nodes (including any machines that clients come from) with new config (order does not matter)

Removing Nimbus Node(s)

1. Shut down the old nodes and update the configs on all the boxes in any order you want.
2. This should just work so long as you have at least one nimbus node still up.