Manual master failover

Example cluster topology setup

Two masters sharing three workers, one of the masters is read-only, and the split strategy is set to NONE on both of them.

left to right direction
actor clientapi1
actor clientapi2


node master1
node master2

note right of master2: readonly

node worker1
node worker2
node worker3

master1 <-> master2
master1 ==> worker1
master1 ==> worker2
master1 ==> worker3
master2 ==> worker1
master2 ==> worker2
master2 ==> worker3

clientapi1 ==> master1
clientapi1 ==> master2
clientapi2 ==> master1
clientapi2 ==> master2

Enabling writes on the read-only master

In case of primary master failure, the second master will continue to serve reads. A manual intervention is needed to enable the writes on the second master:

  1. Stop the primary master, so that it does not have access to the workers. Otherwise, it might lead to replications being initiated, while the second master is in control.

  2. Unpeer the primary master from the second one (optional).

    If you change the read-only flag on second master to ``false`` (Step 3) while connected to the first master, the second master will wait for the first to recover. If you unpeer the master and then change the flag, the second master, being the only master available, will start processing requests.

  3. On the second (failover) master, set the read-only flag to false.

  4. On the primary master, change the read-only flag in cluster.properties to true.

  5. Start the primary master. It becomes the new read-only master.

  6. Peer back the two masters (addSyncPeer operation).

As a result, you will get the following cluster topology.

left to right direction
actor clientapi1
actor clientapi2


node master1
node master2

note right of master1: readonly

node worker1
node worker2
node worker3

master1 <-> master2
master1 ==> worker1
master1 ==> worker2
master1 ==> worker3
master2 ==> worker1
master2 ==> worker2
master2 ==> worker3

clientapi1 ==> master1
clientapi1 ==> master2
clientapi2 ==> master1
clientapi2 ==> master2

Network failure between the masters

The client APIs see both masters

In case of a network failure between the masters, but with the client APIs seeing both masters, everything should be fine because the client APIs will hit the same master for writes.

left to right direction
actor clientapi1
actor clientapi2


node master1
node master2

master1 <.> master2


node worker1
node worker2
node worker3

master1 ==> worker1
master1 ==> worker2
master1 ==> worker3
master2 ==> worker1
master2 ==> worker2
master2 ==> worker3

clientapi1 ==> master1
clientapi1 ==> master2
clientapi2 ==> master1
clientapi2 ==> master2

note right of master2: readonly

Client APIs can either see the primary master, or the second one

In case of a network failure between the masters, the client APIs can either see the primary master, or the second one. In this case, reads will be available for both clientapi1 and clientapi2, but writes can only be executed through clientapi1. clientapi2 will get an exception because master2 is in read-only mode. (Currently, there is no workaround for this limitation).

left to right direction
actor clientapi1
actor clientapi2


node master1
node master2

master1 <.> master2

note right of master2: readonly

node worker1
node worker2
node worker3

master1 ==> worker1
master1 ==> worker2
master1 ==> worker3
master2 ==> worker1
master2 ==> worker2
master2 ==> worker3

clientapi1 ==> master1
clientapi2 ==> master2

Primary master dies during a master split event (Unhandled case)

During a master split event, the primary master continues to execute writes but they are not synced with the second one. So in case of primary master failure, the log for these transactions will be lost. If you execute the manual failover procedure, the second master will be marked as writable, and all workers will be marked as out of sync.