Creating and managing a cluster¶

What’s in this document?

Prerequisites
Create cluster
Manage cluster membership
- Managing cluster membership through Workbench
- Managing cluster membership using the REST API
Manage cluster configuration properties
- Using the Workbench
- Using the REST API
Monitor cluster status
- Cluster group
- Cluster node
Delete cluster
- Force parameter
Configure external cluster proxy
- Start the external proxy
- External proxy for cluster running over SSL
Cluster security
- Encryption
- Access control
Truncate cluster transaction log
Disaster recovery
- Recovering individual nodes
- Rebuilding the cluster from a single node
  - Kubernetes

These instructions walk through the steps for creating and monitoring a cluster group.

Prerequisites ¶

At least three licensed GraphDB installations are available. The Raft algorithm recommends an odd number of nodes, so a cluster of five nodes is a good choice.
All instances have the same security settings, in particular the same shared Token secret. For each instance, inside the graphdb.properties file set the graphdb.auth.token.secret property to the desired Token secret value graphdb.auth.token.secret = <my-shared-secret-key> where <my-shared-secret-key> can be any string.
All nodes must have networking configured correctly. The hostname provided by the operating system must be resolvable to the correct IP address on each node in the cluster.
Some of the GraphDB configuration properties need to be shared between all nodes. See below.

Tip

In case your networking is not configured correctly or you are unsure if it is, you can set the hostname for each node by configuring graphdb.hostname = <your.example.hostname> in each graphdb.properties file on each instance. If you do not have resolvable hostnames, you can supply an IP address instead.

The examples below assume that there are five nodes reachable at the hostnames graphdb1.example.com, graphdb2.example.com, graphdb3.example.com, graphdb4.example.com, and graphdb5.example.com.

Ports ¶

The following table outlines the network connectivity requirements for the GraphDB cluster nodes and optionally, the cluster proxies.

The table lists the default ports used by GraphDB, but can be changed using configuration properties.

Source	Destination	Port/Protocol	Direction	Purpose
Nodes	Nodes	7200/TCP	Bidirectional	Request proxy/Configuration synchronization
Nodes	Nodes	7300/TCP	Bidirectional	Raft communication
Cluster proxy	Nodes	7200/TCP	Incoming	Request proxy
Cluster proxy	Nodes	7300	Bidirectional	Node state observer

Configuration properties prerequisites ¶

Some of the GraphDB properties should be identical on all nodes in the cluster — notably, all security-related properties have to be identical on all nodes. The list below lists all configuration properties and configuration property groups that must be identical between the nodes.

graphdb.auth.*

graphdb.ontop.*

graphdb.s3.*

graphdb.audit.*

graphdb.cluster.*

graphdb.raft.*

graphdb.engine.*

graphdb.health.*

graphdb.llm.*

graphdb.openai.*

graphdb.ttyg.*

graphdb.gpt-sparql.*

graphdb.request.id.alternatives

graphdb.append.request.id.headers

graphdb.wait.report.minutes

graphdb.jsonld.whitelist

graphdb.heapdump.enable

graphdb.inference.buffer

graphdb.inference.concurrency

graphdb.workbench.cors.enable

graphdb.workbench.cors.origin

graphdb.workbench.cors.expose-headers

graphdb.workbench.maxConnections

graphdb.workbench.maxUploadSize

graphdb.proxy.followerRetries

graphdb.persistent.parallel.inferencers

graphdb.connector.ssl.enabled

graphdb.connector.ssl.certificateKeystorePassword

graphdb.connector.ssl.certificateKeyAlias

graphdb.connector.ssl.certificateKeyPassword

graphdb.connector.ssl.truststorePassword

graphdb.data.encryption.type

graphdb.data.encryption.file

graphdb.data.encryption.keystore.password

graphdb.data.encryption.keystore.alias

Achieving a high availability deployment ¶

A typical scenario would be a deployment in a cloud infrastructure with an odd number of GraphDB instances in different regions or zones. No single zone should contain more than 50% of the nodes in the cluster, so that the GraphDB cluster can continue to function even if one of the zones fails. We recommend deploying clusters with either three or five nodes as the optimal option. To learn more about high availability deployment, check the examples of high availability cluster deployments.

Create cluster ¶

A cluster can be created interactively from the Workbench or programmatically via the REST API.

Using the Workbench ¶

Open any of the GraphDB instances that you want to be part of the cluster, for example http://graphdb1.example.com:7200, and go to Setup ‣ Cluster.

Click the icon to create a cluster group.
In the dialog that opens, add all the nodes you want to be part of the cluster. To add a cluster node, click Add node and enter the address of the node you want to add, then click on the check icon to confirm it:

Note

The nodes will become a part of the cluster only once you confirm the cluster creation by clicking on Save changes.

Clicking on Advanced settings opens an additional panel with settings that affect the entire cluster group. You can also leave the default parameters when creating the cluster, and if needed change them later once the cluster has been created.
Click Save changes to create the cluster. One of the nodes will become a candidate and initiate an election (you may see it briefly flash in green), and eventually a leader will be elected and shown in dark red.

In this example, graphdb1.example.com became the leader but it could have been any of the other four nodes. The fact that graphdb1.example.com was used to create the cluster does not affect the leader election process in any way.

Tip

You can open a legend listing all cluster states by clicking on the Question mark on the bottom left of the screen. To learn more about the various cluster states, check the cluster states documentation, which explains the legend in further detail.
You can also add, remove or replace nodes from the cluster group, as well as delete it.

Using the REST API ¶

You can also create a cluster using the respective REST API — see Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller for the interactive REST API documentation.

The examples below use the curl command line utility.

To create the cluster group, simply POST the desired cluster configuration to the /rest/cluster/config endpoint of any of the nodes (in this case http://graphdb1.example.com:7200):

curl -X POST -H 'Content-type: application/json' \
    http://graphdb1.example.com:7200/rest/cluster/config \
    -d '{
        "nodes": [
            "graphdb1.example.com:7300",
            "graphdb2.example.com:7300",
            "graphdb3.example.com:7300"
        ]
    }'

Each node uses the default HTTP port of 7200 and the default RPC port of 7300.

Tip

The default RPC port is the HTTP port + 100. Thus, when the HTTP port is 7200, the RPC port will be 7300. You can set a custom RPC port in the graphdb.properties file of a given node’s GraphDB instance using graphdb.rpc.port = NNNN, where NNNN is the chosen port.

Just like in the Workbench, you do not need to specify the advanced settings if you want to use the defaults. If needed, you can specify them like this:

curl -X POST -H 'Content-type: application/json' \
    http://graphdb1.example.com:7200/rest/cluster/config \
    -d '{
        "nodes": [
            "graphdb1.example.com:7300",
            "graphdb2.example.com:7300",
            "graphdb3.example.com:7300"
        ],
        "electionMinTimeout": 8000,
        "electionRangeTimeout": 6000,
        "heartbeatInterval": 2000,
        "messageSizeKB": 64,
        "transactionLogMaximumSizeGB": 50,
        "verificationTimeout": 300
    }'

The HTTP return code for this operation may be 201, 400, or 412:

201: Cluster successfully created
If the cluster group has been successfully created, you will get a 201 Success response code, and the returned response body will indicate that the cluster is created on all nodes:
{
  "graphdb1.example.com:7300": "CREATED",
  "graphdb2.example.com:7300": "CREATED",
  "graphdb3.example.com:7300": "CREATED"
}
400: Invalid cluster configuration

If the JSON description of the cluster group configuration is invalid, the returned response code will be 400 Bad request.
412: Unreachable nodes or cluster already existing on a node
If during the creation of the cluster some nodes are unreachable or a cluster group already exists on a given node, the response code will be 412 Precondition failed. The error will be shown in the JSON response body:
Unreachable node:
{
  "graphdb1.example.com:7301": "NOT_CREATED",
  "graphdb2.example.com:7302": "NO_CONNECTION",
  "graphdb3.example.com:7303": "NOT_CREATED"
}
Cluster already existing on a node:
{
  "graphdb1.example.com:7301": "NOT_CREATED",
  "graphdb2.example.com:7302": "ALREADY_EXISTS",
  "graphdb3.example.com:7303": "NOT_CREATED"
}

Creation parameters ¶

The cluster group configuration has several properties that have typical default values:

Parameter	Default value	Description
`electionMinTimeout`	8000	The minimum wait time in milliseconds for a heartbeat from a leader.
`electionRangeTimeout`	6000	The variable portion of each waiting period in milliseconds for a heartbeat.
`heartbeatInterval`	2000	The interval in milliseconds between each heartbeat that is sent to follower nodes by the leader.
`verificationTimeout`	300	The amount of time in milliseconds that a follower node would wait before attempting to verify the last committed entry when the first verification is unsuccessful.
`messageSizeKB`	64	The size of the data blocks, in kilobytes, transferred during data replication streaming through the RPC protocol.
`transactionLogMaximumSizeGB`	50	Maximum size of the transaction log in GBs. The transaction log will be automatically truncated if it becomes bigger than this value. The minimum transaction log size is 1 GB. Setting it to a negative value will disable automatic transaction log truncation.
`batchUpdateInterval`	5000	The interval in milliseconds between requesting updates from the primary cluster. Used only when the cluster is in secondary mode.

Manage cluster membership ¶

You can add, remove, or replace cluster nodes at runtime without having to stop the entire cluster group. To achieve this, when making a change to the cluster membership, GraphDB checks whether there is total consensus between the nodes in the new configuration.

Tip

A total consensus means that all nodes in the configuration have successfully processed the last entry

If you try to join a node that used to be a part of the cluster but has since disconnected from it, only the necessary metadata will be sent to that node, provided the transaction log has not changed since disconnecting. This way bulky snapshot replications are avoided when joining previously in-sync nodes.

If the majority of nodes fail to respond to heartbeats, you can remove the non-responsive nodes all at once. In that case, a majority consensus on the new configuration would be enough for this operation to be executed successfully.

Note

To maintain a majority consensus, GraphDB will prevent you from removing 50% or more of the nodes from a current configuration at once.

You can also directly replace a node in the cluster with another instance, even if the address of the node you attempt to add is not part of the cluster.

The replace procedure allows you to replace more than one node, or to add a number of new nodes different than the number of old nodes you are removing — as long as you are not removing the majority of nodes and there is majority consensus between the nodes in the new group.

Warning

When adding nodes, GraphDB stops all current write operations in order to prevent a Byzantine fault issue. The writes are resumed once the add node procedure has concluded.

Managing cluster membership through Workbench ¶

Clicking on the Edit cluster button opens up a pop-up interface that allows you to easily add, remove and replace nodes in the current configuration. You can simultaneously perform any combination of those three actions, as long as you preserve the total consensus between the nodes in the new configuration.

The Edit cluster interface also displays additional information, such as the status of the individual nodes as well as the current node.

_images/creating-a-cluster-manage-cluster-membership-workbench.png

Note

Any changes made to the cluster configuration will not be preserved until you click on Save changes to create the cluster.

Add nodes through the Workbench ¶

From the cluster view, open the Edit cluster interface.
Click on Add node.
Add the URL address of the node you want to add, and confirm by clicking on the check icon.
Save the changes to the cluster.

Remove nodes through the Workbench ¶

From the cluster view, open the Edit cluster interface.
Click on the Remove node icon next to the node you want to remove.
Save the changes to the cluster.

Warning

To preserve the stability of the cluster, you cannot remove half or more of the active nodes at the same time. If you want to remove more than 50% of the nodes, you will have to split the process in two or more steps.

Remove nodes through the Workbench ¶

From the cluster view, open the Edit cluster interface.
Click on the Replace node icon next to the node you want to replace.
Add the URL address of the node you want to add in its place, and confirm by clicking on the check icon.
Save the changes to the cluster.

Warning

To preserve the stability of the cluster, you cannot replace half or more of the active nodes at the same time. If you want to replace more than 50% of the nodes, you will have to split the process in two or more steps.

Managing cluster membership using the REST API ¶

You can also use the respective REST API — see Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller for the interactive REST API documentation.

Add nodes using the REST API ¶

From Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller, use a curl command like the following to send a POST request to the /rest/cluster/config/node endpoint:

curl -X POST -H 'Content-Type: application/json' \
  'http://graphdb1.example.com:7200/rest/cluster/config/node' \
  -d '{
   "nodes": [
     "graphdb3.example.com:7300"
   ]
 }'

If one of the nodes from the group or from the newly added ones has no connection to any of the nodes, an error message will be returned. This is because a total consensus between the nodes in the new group is needed to accept the configuration, which means that all of them should see each other.

If the added node is part of a different cluster, an error message will be returned.

You can make cluster membership changes only if the cluster has a leader node. If you try to add a node to the cluster group while there is no leader, an error message will be returned.

You can add nodes that have previously been part of the same cluster, even if they are not empty. If you add such a node, the cluster will automatically synchronize it. If the added node has not previously been part of the cluster, it should be either empty or in the same state as the cluster, which means that it should have the same repositories and namespaces as the nodes in the cluster. If one of these conditions is not met, you will not be able to add the node.

Note

You can’t add a node that was successfully replaced or removed from the cluster back to the cluster if that node is behind. It can be added only if it is completely empty, or if no write actions have been performed on the cluster since the last time that node was a part of the cluster.

If a node was not connected to the cluster when it was removed, but “truncate transaction log” was performed on the cluster, the node cannot be added back to the cluster.

Remove nodes using the REST API ¶

From Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller, send a DELETE request to the /rest/cluster/config/node endpoint:

curl -X DELETE -H 'Content-Type: application/json' \
'http://graphdb1.example.com:7200/rest/cluster/config/node' \
  -d '{
   "nodes": [
     "graphdb3.example.com:7300"
   ]
 }'

If one of the nodes remaining in the new cluster configuration is down or not visible to the others, the operation will not be successful. This is because a total consensus between all nodes in the new configuration is needed, so all of them should see each other.

You can make cluster membership changes only if the cluster has a leader node. If you try to remove a node from the cluster group while there is no leader, an error message will be returned.

You can also remove the leader node — in that case, the cluster will elect a new leader at random.

If a node is down, it can still be removed as it will not be part of the new configuration. If started again, the node will “think” that it is still part of the cluster and will be stuck in candidate state. The rest of the nodes will not accept any communication coming from it. In such a case, you can only manually delete the cluster (using Show cluster configuration ‣ Delete cluster) from this node.

_images/delete-cluster-on-detached-node.png

Replace nodes using the REST API ¶

From Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller, send a POST request to the /rest/cluster/config/node endpoint:

curl -X PATCH -H 'Content-Type: application/json' \
  'http://graphdb1.example.com:7200/rest/cluster/config/node' \
  -d '{
   "addNodes": [
     graphdb4.example.com:7300
     graphdb5.example.com:7300
   ],
   "removeNodes": [
     graphdb3.example.com:7300
   ]
 }'

A total consensus between the nodes in the resulting group is needed to accept the configuration, which means that all of the nodes in the cluster (including the remaining and the newly added nodes) should see each other. If one of them has no connection to any of the nodes, replace operation will fail, an error message will be returned, and no changes to the current cluster group will be made.

If the added node is part of a different cluster, an error message will be returned.

You can make cluster membership changes only if the cluster has a leader node. If you try to replace nodes in the cluster group while there is no leader, an error message will be returned.

You can also replace the leader node — in that case, the cluster will elect a new leader at random.

Manage cluster configuration properties ¶

You can view and manage the cluster configuration properties both from the Workbench and the REST API.

Using the Workbench ¶

To view the properties, go to Setup ‣ Cluster and click the cog icon on the top right.

It will open a panel showing the cluster group configuration properties and a list of its nodes.

To modify the configuration properties, click Edit configuration.

Warning

Editing of these properties is only possible on the leader node.

Using the REST API ¶

To view the cluster configuration properties, go to Help ‣ REST API ‣ GraphDB Workbench API ‣ cluster-group-controller and perform a GET request to the /rest/cluster/config endpoint on any of the nodes:

curl http://graphdb1.example.com:7200/rest/cluster/config

To check the cluster configuration, send a GET request to the GET /rest/cluster/config endpoint. The HTTP return code for this operation may be 200 or 404:

200: Returns cluster configuration

If the cluster configuration has passed successfully, the response code will be 200 Success.

404: Cluster not found

If no cluster group has been found, the returned response code will be 404. One such case could be when you attempt to create a cluster group with just one GraphDB node.

To update the configuration properties, perform a PATCH request containing the parameters of the new configuration to the /rest/cluster/config endpoint:

curl -X PATCH -H 'Content-Type: application/json' \
  'http://graphdb1.example.com:7200/rest/cluster/config' \
  -d '{
   "electionMinTimeout": 8000,
   "electionRangeTimeout": 6000,
   "heartbeatInterval": 2000,
   "messageSizeKB": 64,
   "transactionLogMaximumSizeGB": 50,
   "verificationTimeout": 300
 }'

If one of the cluster nodes is down or was not able to accept the new configuration, the operation will not be successful. This is a total consensus between the nodes is required. In case one or more of them cannot append the new configuration, it will be rejected by all of the nodes.

Monitor cluster status ¶

To check the current status of the cluster, including the current leader, open the Workbench and go to Setup ‣ Cluster.

The solid green lines indicate that the leader is IN_SYNC with all followers.

Clicking on a node displays some basic information about it, such its state (leader or follower) and RPC address. Clicking on its URL will open the node in a new browser tab.

You can also use the REST API to get more detailed information or to automate monitoring.

Cluster group ¶

To check the status of the entire cluster group, send a GET request to the /rest/cluster/group/status endpoint of any of the nodes, for example:

curl http://graphdb1.example.com:7200/rest/cluster/group/status

If there are no issues with the cluster group, the returned response code will be 200 with a result similar to the following:

[
   {
      "address" : "graphdb1.example.com:7300",
      "endpoint" : "http://graphdb1.example.com:7200",
      "lastLogIndex" : 0,
      "lastLogTerm" : 0,
      "nodeState" : "LEADER",
      "syncStatus" : {
         "graphdb2.example.com:7300" : "IN_SYNC",
         "graphdb3.example.com:7300" : "IN_SYNC"
      },
      "term" : 2,
      "recoveryStatus" : {}
   },
   {
      "address" : "graphdb2.example.com:7300",
      "endpoint" : "http://graphdb2.example.com:7200",
      "lastLogIndex" : 0,
      "lastLogTerm" : 0,
      "nodeState" : "FOLLOWER",
      "syncStatus" : {},
      "term" : 2,
      "recoveryStatus" : {}
   },
   {
      "address" : "graphdb3.example.com:7300",
      "endpoint" : "http://graphdb3.example.com:7200",
      "lastLogIndex" : 0,
      "lastLogTerm" : 0,
      "nodeState" : "FOLLOWER",
      "syncStatus" : {},
      "term" : 2,
      "recoveryStatus" : {}
   }
]

Response code 404 will be returned if no cluster has been found.

Note

Any node, regardless of whether it is a leader or a follower, will return the status for all nodes in the cluster group.

A recoveryStatus value has the following structure:

"recoveryStatus":{
            state:RecoveryStateEnum,
            message:String,
            affectedNodes:[<list of affected node addresses>]
    }

The RecoveryStateEnum value can be one of the following:

SEARCHING_FOR_NODE
WAITING_FOR_SNAPSHOT
RECEIVING_SNAPSHOT
APPLYING_SNAPSHOT
BUILDING_SNAPSHOT
SENDING_SNAPSHOT

Below are two example values for recoveryStatus:

"recoveryStatus": {
  "state": "SENDING_SNAPSHOT",
  "message": "Sending a snapshot to node http://graphdb2.example.com:7200",
  "affectedNodes": [
    "http://graphdb2.example.com:7200 "
  ]
}

"recoveryStatus": {
  "state": "WAITING_FOR_SNAPSHOT",
  "message": "Waiting for snapshot from node http://graphdb1.example.com:7200",
  "affectedNodes": [
    "http://graphdb1.example.com:7200"
  ]
}

Cluster node ¶

To check the status of a single cluster node, send a GET request to the /rest/cluster/node/status endpoint of the node, for example:

curl http://graphdb1.example.com:7200/rest/cluster/node/status

If there are no issues with the node, the returned response code will be 200 with the following information (for a leader):

{
   "address" : "graphdb1.example.com:7300",
   "endpoint" : "http://graphdb1.example.com:7200",
   "lastLogIndex" : 0,
   "lastLogTerm" : 0,
   "nodeState" : "LEADER",
   "syncStatus" : {
      "graphdb2.example.com:7300" : "IN_SYNC",
      "graphdb3.example.com:7300" : "IN_SYNC"
   },
   "term" : 2,
   "recoveryStatus" : {}
}

For a follower node (in this case graphdb2.example.com):

{
   "address" : "graphdb2.example.com:7300",
   "endpoint" : "http://graphdb2.example.com:7200",
   "lastLogIndex" : 0,
   "lastLogTerm" : 0,
   "nodeState" : "FOLLOWER",
   "syncStatus" : {},
   "term" : 2,
   "recoveryStatus" : {}
}

Delete cluster ¶

To delete a cluster, open the Workbench and go to Setup ‣ Cluster. Click Delete cluster and confirm the operation.

Warning

This operation deletes the cluster group on all nodes, and can be executed from any node regardless of whether it is a leader or a follower. Proceed with caution.

You can also use the REST API to automate the delete operation. Send a DELETE request to the /rest/cluster/config endpoint of any of the nodes, for example:

curl -X DELETE http://graphdb1.example.com:7200/rest/cluster/config

By default, the cluster group cannot be deleted if one or more nodes are unreachable. Reachable here means that the nodes are not in status NO_CONNECTION, therefore there is an RPC connection to them. The HTTP return code for this operation may be 200 or 412:

200: Cluster deleted
If the deletion is successful, the response code will be 200 and the returned response body:
{
  "graphdb1.example.com:7300": "DELETED",
  "graphdb2.example.com:7300": "DELETED",
  "graphdb3.example.com:7300": "DELETED"
}
412: Unreachable nodes
If one or more nodes in the group are not reachable, the delete operation will fail with response code 412 and return:
{
  "graphdb1.example.com:7300": "NOT_DELETED",
  "graphdb2.example.com:7300": "NO_CONNECTION",
  "graphdb3.example.com:7300": "NOT_DELETED"
}

Force parameter ¶

The optional force parameter (false by default) enables you to bypass this restriction and delete the cluster group on the nodes that are reachable:

When set to false, the cluster configuration will not be deleted on any node if at least one of the nodes is unreachable.
When set to true, the cluster configuration will be deleted only on the reachable nodes, and not be deleted on the unreachable ones.

In such a case, the returned response will be 200:
{
  "graphdb1.example.com:7300": "DELETED",
  "graphdb2.example.com:7300": "io.grpc.StatusRuntimeException: UNAVAILABLE: io exception",
  "graphdb3.example.com:7300": "DELETED"
}

In the Workbench, it can be enabled at deletion:

Configure external cluster proxy ¶

The external cluster proxy can be deployed separately on its own URL. This way, you do not need to know where all cluster nodes are. Instead, there is a single URL that will always point to the leader node.

The externally deployed proxy will behave like a regular GraphDB instance, including opening and using the Workbench. It will always know which node is the current leader and always serve all requests to that leader.

Note

The external proxy does not require a GraphDB SE/EE license.

Start the external proxy ¶

To start the external proxy:

Execute the cluster-proxy script in the bin directory of the GraphDB distribution.
Provide the cluster secret.
Provide a GraphDB server HTTP or RPC address to at least one of the nodes in the cluster. You can provide either the HTTP or the RPC address of the node — they are interchangeable. For example:
./bin/cluster-proxy -g http://graphdb1.example.com:7200,http://graphdb2.example.com:7200
A console message will inform you that GraphDB has been started in proxy mode.

Cluster proxy options

The cluster-proxy script supports the following options:

Option	Description
-d, --daemon	Daemonize (run in background)
-r, --follower-retries <num>	Number of times to retry a request to a different node in the cluster
-g, --graphdb-hosts <address>	List of GraphDB nodes’ HTTP or RPC addresses that are part of the same cluster
-h, --help	Print command line options
p, --pid-file <file>	Write PID to <file>
-Dprop	Set Java system property
-Xprop	Set non-standard Java system property

By default, the proxy will start on port 7200. To change it, use, for example, -Dgraphdb.connector.port=7201.

As mentioned above, the default RPC port of the proxy is the HTTP proxy port + 100, which will be 7300 if you have not used a custom HTTP port. You can change the RPC port by setting, for example, -Dgraphdb.rpc.port=7301 or -Dgraphdb.rpc.address=graphdb-proxy.example.com:7301, e.g.:

./bin/cluster-proxy -Dgraphdb.connector.port=7201 -Dgraphdb.rpc.port=7301 -g http://graphdb1.example.com:7200

Note

Remember to set the -Dgraphdb.auth.token.secret=<cluster-secret> with the same secret with which you have set up the cluster. If the secrets do not match, some of the proxy functions may appear as if they are working correctly, but will still be misconfigured and you may experience unexpected behavior at any time.

The external proxy works with two types of cluster node lists: static and dynamic.

The static list is provided to the proxy through the -g/--graphdb-hosts option of the script. This is a comma-separated list of HTTP or RPC addresses of cluster nodes. At least one address to an active node should be provided. Once the proxy is started, it tries to connect to each of the nodes provided in this list. If it succeeds with one of them, it then builds the dynamic cluster node list.
The dynamic cluster node list is built by requesting the cluster’s current status from one of the nodes in the static list. The proxy then subscribes to any changes in the cluster status such as leader changes, nodes being added or removed, or nodes out of reach. The external proxy always sends all requests to the current cluster leader. If there is no leader at the moment, or the leader is unreachable, requests will go to a random node.

Note

The dynamic cluster node list is reset every time the external proxy is restarted. After each restart, the proxy knows only about the nodes listed in the static node list provided by the -g/--graphdb-hosts option of the script.

External proxy for cluster running over SSL ¶

To set up the external proxy to connect to a cluster over SSL, the same options used to set up GraphDB with security can be provided to the cluster-proxy script. The most common ones are:

-Dgraphdb.connector.ssl.enabled = true
-Dgraphdb.connector.keystoreFile=<path-to-keystore-file>
-Dgraphdb.connector.keystorePass=<keystore-pass>
-Dgraphdb.connector.keyAlias=<alias-to-key-in-keystore>

For more information on the cluster security options, please see below.

Cluster security ¶

Encryption ¶

Because there is a lot of traffic between the cluster nodes, it is important that it is encrypted. To do this, make sure that:

SSL/TLS is enabled on all cluster nodes.

The nodes’ certificates are trusted by the other nodes in the cluster.

The method of enabling SSL/TLS is already described in Configuring GraphDB instance with SSL. There are no differences when setting up a node to be used in a cluster.

See how to set up certificate trust between the nodes in the Certificate trust section of the Encryption page.

Access control ¶

Authorization and authentication methods in the cluster do not differ from those for a regular GraphDB instance. The rule of thumb is that all nodes in the cluster group must have the same security configuration.

For example, if SSL is enabled on one of the nodes, you must enable it on the other nodes as well. If you have configured OpenID on one of the nodes, it needs to be configured on the rest of them as well.

Truncate cluster transaction log ¶

The truncate log operation is used to free up storage space on all cluster nodes by clearing the current transaction log and removing cached recovery snapshots. It can be triggered with a POST request to the /rest/cluster/truncate-log endpoint.

Note

The operation requires a healthy cluster — that is, one where a leader node is present and all follower nodes are IN_SYNC. This is because the truncate log operation is propagated to each node in the cluster and truncates the log subsequently on each node through the Raft quorum mechanism.

You can truncate the cluster log with the following curl request:

curl -X POST '<base_url>/rest/cluster/truncate-log'

Disaster recovery ¶

Sometimes the machine on which a cluster node runs might become unavailable or experience hardware failures. In these cases, manual intervention may be necessary to bring the cluster back to a healthy state.

This section describes how to recover unavailable cluster nodes, or in case of unrecoverable situations, how to rebuild the entire cluster from a single node.

Recovering individual nodes ¶

The GraphDB cluster configuration relies heavily on the RPC addresses used for the nodes when the cluster was first created. By default, this is the machine’s hostname; however, it can be overridden with the RPC address.

The default setup in most cloud and virtualized environments is for the machines to get a new hostname, IP address, or both.

To recover individual nodes, determine if quorum is affected by using the cluster and node health check endpoints. If there are not enough nodes to form a quorum, consult the Rebuilding the cluster from a single node process.

In case the RPC address has changed:

Stop the new GraphDB node from running.
Remove the failed node from the cluster.
Delete the entire data directory of the new node.
Start the new GraphDB node and join it to the cluster.

In case the RPC address has not changed, simply start the GraphDB process. The leader should start sending updates to the node; just wait for the node to move into the IN_SYNC state.

Rebuilding the cluster from a single node ¶

Warning

Make sure you have a backup of the current data on all nodes before continuing with this procedure. Most cloud environments provide the ability to create volume snapshots. If you have enough space on the machine you can move the data instead of deleting it.

Sometimes multiple nodes can fail simultaneously, or nodes might be available but unable to agree on the state of the cluster. When this happens, the cluster might be in a deadlocked state and unable to process write or even read requests.

If this happens, you can rebuild the cluster from a single node serving as the source of truth for the new cluster.

Before starting the recovery process, make sure that the cluster is not currently in the process of recovering:

Examine the log files — if you see recent messages like “recording snapshot” or “recovering from snapshot” it might mean that the cluster is currently recovering from a snapshot
Monitor the size of the data directory on the GraphDB nodes — if it is rising, that might also indicate that a snapshot replication is in progress
Use the cluster and node status endpoints to check if any of the nodes are currently syncing

Also, as mentioned above, make sure you have backed up all current data on all nodes.

Stop the GraphDB processes on all nodes
Determine which node has processed the most transactions by checking the log index maintained by GraphDB, which can be retrieved for each node by following these steps:
Note

The node determined in this step will be referred to as node-x later on in this procedure.
1. Get access to the file system where GraphDB stores its data
2. Determine the size (in bytes) of the log.index file. The location of this file is relative to the configured GraphDB data directory and has the following path: raft/transaction-log/log.index
3. Calculate the log index using the following formula: \((size\_in\_bytes / 33) - 1\)
4. Select the node with the highest log index; check the contents and size of the data/repositories directory to make sure that it contains all the repositories that you have
Delete the entire data directory of GraphDB on all nodes except on node-x
On node-x, delete only the raft/ subfolder of the data directory
Start the GraphDB process again on all nodes
Create the cluster again, but make sure to issue the request to the node-x

Kubernetes ¶

Since the GraphDB pods cannot be stopped individually to perform the modifications in the data directory, you can do one of the following:

Change the command of the pods to something that will block indefinitely such as sleep infinity; also remember to disable the liveness and readiness probes — otherwise Kubernetes will kill and restart the pods
Scale the StatefulSet down to zero and mount the volumes somewhere else