Cluster states and election procedures

This section expands on the topics introduced in Overview of clusters.

See also

• Using a cluster provides an overview of using the GraphDB client API to manage your cluster.

• Creating and managing a cluster describes how to deploy a cluster in GraphDB using the Workbench or via the REST API.

• Multi-region cluster deployments details how to configure multiple clusters in different zones to achieve high availability in multi-region deployments.

The GraphDB cluster is built on the Raft consensus algorithm for managing a replicated log on a distributed state machine. Raft allows a collection of machines to work as a coherent group and survive failure of some of its members.

A cluster is made up of two basic node types: leaders and followers. Usually, it comprises an odd number of nodes in order to tolerate failures.

At any given time, each of the nodes is in one of four states:

• Leader: Usually, there is one leader that handles all client requests. If a client contacts a follower, the follower redirects it to the leader.

• Follower: A cluster is made up of one leader and all other servers are followers. They are passive, meaning that they issue no requests on their own but simply respond to requests from leaders and candidates.

• Candidate: This state is used when electing a new leader.

• Restricted: In this state, the node cannot respond to requests from other nodes and cannot participate in election. A node goes into this state when there is a license issue such as an invalid or an expired license.

Leader election

Raft uses a heartbeat mechanism to trigger leader election. When nodes start up, they begin as followers. A node remains in the follower state as long as it is receiving valid remote procedure calls (RPCs) from a leader or candidate. Leaders send periodic heartbeats (AppendEntries RPCs that carry no log entries) to all followers in order to maintain their authority. If a follower receives no communication over a period of time known as the election timeout, then it assumes there is no viable leader and begins an election to choose a new leader. A candidate wins an election if it receives votes from a majority of the servers in the full cluster for that term. Each node can vote for at most one candidate in a given term, on a first-come-first-served basis.

If the cluster gets into a situation where only one node is left, that node will switch to read-only mode. The state shown in its cluster status will switch to candidate, as it cannot achieve a quorum with other cluster nodes when new data is written.

The leader election process follows these steps:

1. After the initial configuration request has been sent, a random node will be set as the leader.

2. If the current leader node stops working for some reason, a new election is held to promote the most voted follower nodes to candidate status, and one of those candidates will become the new leader.

3. The leader node sends a constant heartbeat (a form of node status check to see if the node is present and able to perform its tasks).

4. If only one node is left active for some reason, its status will change to candidate. It will switch to read-only mode to prevent further tinkering with it until more nodes appear in the cluster group.

Image source: Stanford Digital Repository

Terms

The Raft algorithm divides time into terms of arbitrary length. Terms are numbered with consecutive integers. Each term begins with an election in which one or more candidates attempt to become leader. If a candidate wins the election, then it serves as leader for the rest of the term.

In some situations an election results in a split vote. In this case the term will end with no leader; a new term (with a new election) will commence. Raft ensures that there is at most one leader in a given term.

Different servers may observe the transitions between terms at different times. Raft terms act as a logical clock in Raft, and they allow servers to detect obsolete information such as stale leaders. Each server stores a current term number, which increments with term passings. Current terms are exchanged whenever servers communicate; if one server’s current term is smaller than the other’s, then it updates its current term to the larger value. If a candidate or leader discovers that its term is out of date, it immediately reverts to the follower state. If a server receives a request with a stale term number, it rejects the request.

Cluster states

All possible node, sync and connection states are displayed on the bottom left of the Cluster view inside Workbench. You can toggle the Cluster states legend by clicking the question mark icon.

Cluster node states

Legend	Name	Description
	Leader	The server node that manages the handling of all client requests, including delegation of jobs to the followers.
	Follower	A passive server that doesn’t issue requests on its own but simply responds to requests from leaders and candidates — typically, read requests.
	Out of sync	A node that is not in sync and should be recovered.

Cluster sync states

Legend	Name	Description
	Candidate	A node that is trying to become a new leader.
	No cluster	A node that is not part of the cluster.
	Out of sync	A node that is not in sync and should be recovered.
	No connection	A node that has no connection to the server.
	Read only	A node that is read only and cannot be written to. A read only node can only be a follower and cannot become a leader.
	Restricted	A node that doesn’t have the required license to be able to become a part of the cluster.
	Searching for node	An out of sync node that is searching for another node that can provide a snapshot to recover from.
	Waiting for snapshot	An out of sync node that is waiting to receive a snapshot from another node.
	Building a snapshot	A node that is preparing a snapshot, either as part of a recovery procedure or as a scheduled action.
	Sending a snapshot	A node that is currently sending data to recover from.
	Receiving a snapshot	A node that is currently receiving data to recover from.
	Applying a snapshot	A cluster that is applying a received snapshot as part of the recovery procedure.

Cluster link states

Legend	Name	Description
	In sync	Displayed between a Leader and a Follower node.
	Syncing	Displayed between the Leader and a node that is currently receiving entries from the Leader.
	Out of sync	Displayed between the Leader node and a node that’s out of sync.
	Receiving a snapshot	Shows the direction of a snapshot being sent from one node to another.

Transaction fingerprints

Nodes use fingerprints — checksums used to determine whether two repositories are in the same condition and contain the same data. Every transaction performed on a repository returns a fingerprint, which is then compared against the fingerprint of the same repository on the leader node.

When fingerprints do not match, GraphDB automatically resolves the issue by replicating the offending nodes.

Log replication

The GraphDB cluster nodes communicate using remote procedure calls (RPCs). The basic consensus algorithm requires only two types of RPCs:

• RequestVote: RPCs that are initiated by candidates during elections

• AppendEntries: RPCs that are initiated by leaders to replicate log entries and to provide a form of heartbeat

Servers retry RPCs if they do not receive a response in a timely manner. They issue RPCs in parallel for best performance.

The log replication resembles a two-phase commit where:

1. The user sends a commit transaction request.

2. The transaction is replicated in the local transaction log.

3. The transaction is replicated to other followers in parallel.

4. The leader waits until enough members (total/2 + 1) have replicated the entry.

5. The leader starts applying the entry to GraphDB successfully.

6. The leader sends a heartbeat until there is a successful commit in GraphDB.

7. The leader sends a second RPC informing followers to apply the log entry to GraphDB.

8. The leader informs the client that the transaction is successful.

Note

If followers crash or run slowly, or if network packets are lost, the leader retries AppendEntries RPCs indefinitely in parallel — even after it has responded to the client — until all followers eventually store all log entries.

Only updates relating to repositories, data manipulation, and access are replicated between logs. This includes addition and deletion of repositories, user permission changes, SQL views, smart updates, and standard repository updates.