Overview of clusters

Note

This feature requires a GraphDB Enterprise license.

A key to GraphDB’s scalability is its ability to run clusters. A cluster is a collection of interconnected GraphDB instances, each assigned to a unique URL address and running on a separate machine, that work together as a coherent group. With this arrangement, a GraphDB cluster can survive the failures of some of its members.

You can create a cluster through the GraphDB Workbench or by using the REST API. The Workbench also provides a Cluster view, which visualizes the current state of the cluster and its nodes.

The cluster is an essential tool for building a reliable large-scale system and achieving high availability. A cluster deployment has high uptime, recovers smoothly from failures, greatly reduces the chance of data loss, and overall, handles and adapts to unexpected situations and scenarios.

In order for a cluster to process INSERT and DELETE operations, more than half of its nodes — in other words, a majority quorum of nodes — should be alive (that is, up and running). If 50% or more of the nodes are dead, the cluster will not be able to process INSERT or DELETE operations. You can learn more about this in the Quorum-based replication section of the cluster documentation.

See also

The Raft consensus algorithm

The GraphDB cluster is based on a coordination mechanism known as the Raft consensus algorithm. Raft allows a collection of machines to work as a coherent group and survive failure of some of its members.

The algorithm implements consensus by first electing a leader and then assigning it full responsibility for managing the replicated log that makes the node redundancy possible. The leader accepts log entries from clients, replicates them on other servers, and tells servers when it is safe to apply the actions described by each log entry to their state machines, which are used to keep track of the node states. It can tolerate \(n \geq 2m + 1\) failures. Because of this, Raft plays a key role in achieving an enterprise-level high availability GraphDB cluster deployment.

GraphDB modifies and extends the original Raft protocol to include several improvements that are necessary for a higher level of cluster consistency. However, the leader election and automatic failover mechanism provided by Raft are implemented without changes:

  • An update is considered committed when a majority of GraphDB instances actually commit it locally rather than simply storing it in their transaction logs.

  • Followers also verify with the Leader whether a received entry is processed by the rest of cluster.

  • An additional OUT_OF_SYNC state has been introduced for faulty nodes that require recovery from other nodes with correct state inside the cluster.

  • Node snapshot replication, leveraging Raft’s log index and replication mechanism, has been introduced to recover out-of-sync nodes.

Cluster roles

The total amount of leader and follower nodes is usually an odd number in order to tolerate failures. At any given time, each of the nodes is in one of four states:

  • Leader: Usually, there is one leader that handles all client requests. For example, if a client contacts a follower, the follower redirects it to the leader.

  • Follower: A cluster is made up of one leader and all other servers are followers. They are passive, meaning that they issue no requests on their own but simply respond to requests from leaders and candidates.

  • Candidate: This state is used when electing a new leader.

  • Restricted: In this state, the node cannot respond to requests from other nodes and cannot participate in election. A node goes into this state when there is a license issue — for example, an invalid or expired license.

Note

To learn more about the different states a node can take, see the Cluster states section of the documentation.

Internal and external proxy

Internal proxy

In normal working conditions, the cluster nodes have two states: leader and follower. The follower nodes can accept read requests, but cannot write any data. To make it easier for the user to communicate with the cluster, an integrated proxy will redirect all requests (with some exceptions) to the leader node. This ensures that regardless of which cluster node is reached, it can accept all user requests.

However, if a GraphDB cluster node is unavailable, another node on another URL must take its place. This means that all cluster node addresses must be tracked and the reached node must be healthy and online.

External proxy

For better usability, you can deploy the proxy separately on its own URL. This way, you do not need to know where all cluster nodes are. Instead, there is a single URL that will always point to the leader node.

The externally deployed proxy will behave like a regular GraphDB instance, including opening and using the Workbench. It will always know which one is currently the leader and will always serve all requests to that leader.

High availability and fault-tolerance

A system can be characterized as having high availability if it meets criteria such as having high uptime, smooth recovery, zero data loss, and the ability to handle and adapt to unexpected situations and scenarios. The Raft consensus algorithm helps to ensure these criteria and achieve enterprise-grade high availability deployment.

What is consensus?

One of the main problems of distributed systems is achieving consensus between its servers — in other words, getting the multiple individual machines that are part of the deployment to agree on values. Once those machines agree on a value, that decision is final. With most typical consensus algorithms, a deployment can continue operating as long as a majority of its servers are available. For example, a cluster of five servers will continue working as long as at least three of its servers are operational. However, if three or more of the servers fail, the cluster will stop making INSERT and DELETE operations, although it will never return an incorrect value.

Consensus algorithms aim to be fault-tolerant, where faults can be classified in two categories:

  • Crash failure: The component abruptly stops functioning and does not resume. The other components can detect the crash and adjust their local decisions in time.

  • Byzantine failure: The component behaves arbitrarily with no absolute conditions. It can send contradictory messages to other components or simply remain silent. It may look normal from outside.

Consensus is usually associated with replicated state machines, a general approach to building fault-tolerant systems. In a replicated state machine, each instance has a log. A consensus algorithm is used to agree on the commands in the servers’ logs. The consensus algorithm ensures that each state machine processes the same series of commands and thus produces the same series of results, arriving at the same series of states.

Note

In addition to the Raft consensus algorithm, to prevent a Byzantine failure GraphDB also stops all write procedures while adding a node to the cluster that is currently being processed. You can learn more about adding nodes to the cluster in the Manage cluster membership section of the documentation.

High availability deployment

A typical scenario would be a deployment in a cloud infrastructure with the ability to deploy GraphDB instances in different regions or zones so that if a region or zone fails, the GraphDB cluster will continue functioning without any issues for the end user.

To achieve high availability, we recommend deployment of GraphDB instances in different zones or regions while considering the need for a majority quorum in order to be able to accept INSERT and DELETE requests. This means that the deployment should always have over 50% of the instances running.

Another recommendation is to distribute the GraphDB instances so that you do not have exactly half of the GraphDB instances in one zone and the other half in another zone, as this way it would be easy to lose the majority quorum. In such cases, it is better to use three zones.

For complex multi-zone deployments with interregional latency, it is best to use the Multi-region clusters mechanism.

Cluster group with three nodes

In a cluster with three nodes, you need at least two in order to write data successfully. In this case, the best deployment strategy is to have three GraphDB instances distributed in three zones in the same region. This way, if one zone fails, the other two instances will still form a quorum and the cluster will accept all requests.

Note

Having the instances in different regions may introduce latency.

Cluster group with five nodes

In a cluster with five nodes, you need three nodes for a quorum. If you have three available regions or zones, you can deploy two instances in zone 1, two in zone 2, and one in zone 3.

If any of the zones fail, you would still have at least three more GraphDB instances that will form a quorum.

Quorum-based replication

The GraphDB cluster relies on quorum-based replication, meaning that the cluster should have over 50% alive nodes in order to be able to execute INSERT or DELETE operations. This ensures that there will always be a majority of GraphDB nodes that always have up-to-date data.

If any nodes are unavailable when an INSERT or DELETE operation is executed, but more than 50% of the nodes are alive, the request will be accepted, distributed among the reachable alive nodes, and saved if everything is OK. Once the unavailable nodes come back online, the transactions will be distributed to them as well.

If fewer than 50% of the nodes are available, any INSERT or DELETE operations will be rejected.

Query load balancer

For maximum efficiency, the GraphDB cluster distributes the incoming read queries to all nodes, prioritizing the nodes that have fewer running queries. This ensures the optimal hardware resource utilization of all nodes.

Local consistency

GraphDB supports two types of local consistency: None and Last Committed.

  • None is the default setting and is used when no local consistency is needed. In this mode, queries will be sent to any readable node with no guarantee of strong consistency. This is suitable for cases where eventual consistency is sufficient or when enforcing strong consistency is too costly.

  • Last Committed is used when strong consistency is required, ensuring that the results reflect the state of the system after all transactions have been committed. This could lead to lower scalability, however, as the set of nodes to which a query could be load-balanced is smaller. In this mode, queries will be sent to a readable node that has advanced to the last transaction.

The choice between None and Last Committed depends on the specific requirements and constraints of the application and use case. In general, if query results should always reflect the up-to-date state of the database, Last Committed should be used. Otherwise, None is sufficient.