Cluster basics

GraphDB EE is a high-performance, clustered semantic repository scaling in production environments where simultaneous loading, querying and inferencing billions of RDF statements. It supports automatic failover, synchronisation and load balancing to maximise cluster utilisation.

Every GraphDB cluster has at least a single master node managing one or more worker nodes each hosting the full database copy. The master node provides an automatic failover and load balancing between worker nodes. Multiple masters enables continuous cluster performance, even in the event of a master node failure. The cluster supports dynamic reconfiguration when operational, which enables changing the cluster topology at runtime.

Structure

The GraphDB EE cluster is made up of two basic node types: masters and workers.

../_images/GraphDB_Cluster0.png

Master node

The master node manages and distributes atomic requests (query evaluations and update transactions) to a set of workers. It does not store any RDF data itself, therefore requires limited resources.

Hint

The master repository fully implements the RDF4J API. From the client perspective, it is a standard database node, which ensures the transparent upgrade from a single database instance to a cluster.

The master node is responsible for:

  • coordinating all read and write operations;
  • ensuring that all worker nodes are synchronised;
  • propagating updates (insert and delete tasks) across all workers and checking updates for inconsistencies;
  • load balancing read requests (query execution tasks) between all available worker nodes;
  • providing a uniform entry point for client software, where the client interacts with the cluster as though it is just a normal RDF4J repository;
  • providing a JMX interface for monitoring and administrating the cluster;
  • automatic cluster re-configuration in the event of failure of one or more worker nodes;
  • user-directed dynamic configuration of the cluster to add/remove worker nodes.

Tip

A cluster can contain more than one master node. Thus, every master monitors the health of all workers and can distribute query execution tasks between them, effectively allowing a cluster to have multiple entry points for queries and lack a single point of failure.

Worker nodes

Worker nodes are like non-clustered database serving the entire repository. They remain accessible for direct read operation and act like a standard repository. However, a worker node part of the cluster will reject any direct write transaction not sent from the master.

Worker nodes require extensive resources as they are responsible for:

  • Storing all the information;
  • Executing all read/write operations.

WRITE requests (loading data)

Every WRITE request is tested on a worker node to verify its validity. If it is successfully processed, the master will apply it to all other available workers. The master keeps the sequence of all write operations into a transaction log to guarantee that the workers will remain in a consistent state.

../_images/update_success.png

After the processing of a specific WRITE, if the target worker does not result in the same data state as the first test worker, it will start an automatic replication from a repository copy in a correct state.

../_images/update_failure.png

READ requests (query evaluations)

Every READ request is dispatched to an available worker node. The master selects a worker based on their system load and the configured consistency model. When the cluster is strict consistency mode or the request includes the local consistency flag, the master will dispatch the query to a worker on the latest time stamp.

All read requests goes to a priority queue rearranged after every READ operation. If an unexpected error occurs like server time-out or a lost connection, the request is resent to another worker node. Requests that are consistently failing result in an error message returned to the client.

../_images/GraphDB_read.png

Worker node synchronisation and replication

When a master detects a new worker node, it will choose how to synchronise it automatically. If the transaction log contains the full transaction history, the master may apply all missing updates to the worker.

Note

Masters keep only limited transactions controlled by the LogMaxSize & LogMaxDepth parameters in the JMX console.

  • LogMaxSize (JMX) / tx-log-max-size (cluster.properties): the maximum number of stored transactions in the log (default 100`000);
  • LogMaxDepth (JMX) / tx-log-max-depth (cluster.properties): the maximum hours to store transactions (default 7 days);
../_images/GraphDB_synchronise.png

Otherwise, if the master no longer keeps the missing transaction, the only possible option for the master is to replicate the worker from another instance or a cluster backup.

../_images/GraphDB_replication1.png

In th event of an out-of-sync worker, when it reports an unknown state, the master will also instructs the failed worker to replicate directly from a good worker.

../_images/GraphDB_out-of-sync.png

Tip

Having a cluster with a master and at least three workers ensures that the cluster will remain fully functional in the event of any node failure.

During replication, the master remains fully functional.

../_images/GraphDB_replication2.png

Master peering

To operate multiple master nodes, they need to maintain knowledge over the current cluster state. The peering allows the master nodes to synchronize their transaction logs according to the master modes:

  • Read/write mode: the default mode where the master can process new write transaction and will broadcast them to all its peers before applying the write operation to all linked worker nodes;
  • Read-only mode: the master can receive write transaction only from other masters and will not apply the remote transactions to any workers nodes. All the other masters will wait for this master to acknowledge the transaction before sending it to their workers;
  • Muted mode: the master can receive transactions only from other masters and will apply the remote transaction to all locally managed workers. All other masters will not wait for this master to acknowledge the transaction before sending it to their workers;
../_images/GraphDB_peers.png

Tip

Having a cluster with two peered masters ensures high-availability without any single point of failure.

Cluster backup

The master selects a worker with a full transaction log, switches it to read-only and instructs it to create a backup. After finishing the backup, the read-only worker goes in and read-write mode and synchronises with the other workers.

../_images/GraphDB_backup.png

Tip

During the backup operation, the cluster remains in read-write mode and can process updates if there are other workers to process the requests.