Multi-region cluster deployments¶
What’s in this document?
Single-zone and region cluster deployments are great at providing the necessary redundancy for addressing failures of a limited number of instances. However, when dealing with deployments that span multiple regions or zones, interregional latency can add a significant overhead on client queries. The failure of a region or data center (for example, due to a power outage) can also force the cluster to lose its write and read availability if the majority — or all — of the nodes are deployed in the problematic region.
To address these issues and achieve high availability in multi-region deployments, GraphDB has introduced a new cluster type to the Raft protocol called a secondary cluster.
Architecture¶
Secondary cluster is a read-only cluster type that replicates the state of a primary cluster and achieves eventual consistency, even in configurations with significant latency. Similar to the observer nodes, secondary clusters cannot process new updates — they can only keep up to date with the primary cluster by pulling batched updates over a given time interval. The duration of the interval is controlled individually for each secondary cluster by the batchUpdateInterval cluster configuration parameter.
Tip
A primary cluster is any GraphDB cluster that is not set to a read-only secondary cluster mode, including any cluster in a single-region deployment. However, to distinguish between single- and multiple-region deployments, the term primary cluster is almost exclusively used only within the context of multi-region deployments.
Primary and secondary clusters work in harmony to reduce both recovery time and recovery point objectives (RTO and RPO) for multi-region and data center cluster deployments. For any given multi-region deployment, only one primary cluster can exist, and it is responsible for processing all updates. The primary cluster must be healthy (in other words, have a majority quorum) to be able to process updates. The primary cluster will always have the most up-to-date information while secondary clusters will always lag behind (unless the dataset is static) due to the asynchronous nature of the pulling update mechanism.
Note
The example diagram above shows an example configuration of a single primary and a single secondary cluster. However, a multi-region deployment can include any number of secondary clusters — but only a single primary cluster, no matter how secondary ones it has. They can also include a different number of nodes.
Both the primary and secondary clusters operate independently of each other, each with its own separate topology — including a separate leader — as well as all other fault tolerance features available in a single region cluster. Each individual cluster must elect its own leader from the local cluster topology; this leader is responsible for propagating updates and verifying quorum consistency.
Warning
The secondary cluster cannot be created to increase the availability of a primary cluster that is already down. The mechanism exists to provide high availability for writes only in an already established cluster topology.
While the secondary cluster replicates the state of the primary cluster, the primary and the secondary cluster maintain separate log indexes. For example, if a secondary cluster’s own internal last log index is 8, that does not respond to the log index 8 of the primary cluster. It also does not mean that the last log index received pulled from the primary cluster is log index 8.
Secondary cluster identifier tags¶
One of the main components of the secondary cluster mechanism is the use of identifier tags — labels used to identify individual secondary clusters by geographic zone, data center, or other unique attribute. For example, you can use a us-east tag to identify a secondary cluster located on the East coast of the Unites States.
Tags are added to a list that is part of the primary cluster’s configuration and declared by the secondary cluster when switching to a secondary mode. The primary cluster does not maintain a list of secondary clusters — only a list of tags associated with each of those clusters.
Identifier tags can take any string value as long as it is not empty. A unique identifier tag must be used for each individual secondary cluster (in other words, you cannot use the same tag to identify two different servers). Secondary clusters cannot be identified by more than one tag.
Warning
A tag must be added to the list maintained by the primary cluster before you enable the cluster’s secondary mode. Attempting to enable the secondary mode before adding that cluster’s identifier tag to the primary cluster’s tags list will result in an error.
Adding identifier tags using Workbench¶
From the interface, click the gear icon and go to the .
Once added, the tag will appear in the list of identifier tags. The Index column shows the last index received by the secondary cluster identified by the tag.
Tip
Until a cluster has declared itself as a secondary cluster using the identifier tag you just added, that value will remain zero. You can find more information below in the section on monitoring multi-region cluster deployments.
Remove tags from the list by clicking the trash bin icon.
Adding identifier tags using the REST API¶
You can also add identifier tags using the REST API — see for the interactive REST API documentation. The examples below use the curl command line utility.
To add an identifier tag to the list of tags maintained by the primary cluster, POST the that tag to the /rest/cluster/config/tag endpoint of any of the primary cluster’s nodes. The example below adds the identifier tag us-west to the primary cluster through the node http://graphdb1.example.com:7200.
curl -X POST -H 'Content-type: application/json' \
http://graphdb1.example.com:7200/rest/cluster/config/tag \
-d '{
"tag": "us-west"
}'
You can also check which tags have been added to the list by performing a GET request to /rest/cluster/group/status.
curl -X GET --header http://graphdb1.example.com:7200/rest/cluster/group/status
You can delete identifier tags from the list by using the DELETE method with the /rest/cluster/config/tag endpoint.
curl -X DELETE -H 'Content-type: application/json' \
http://graphdb1.example.com:7200/rest/cluster/config/tag \
-d '{
"tag": "us-west"
}'
Tip
You can remove identifier tags regardless of whether a secondary cluster using that tag is already connected to the primary cluster. This will not result in any failures on either cluster, but that secondary cluster will no longer replicate the state of the primary cluster until the identifier tag is added to the list once again.
Note
Only one identifier tag can be added to or deleted from the list at a time.
Enable the secondary cluster mode¶
You can switch any healthy primary cluster to secondary mode by providing the RPC address of the primary cluster node that it will replicate and then declaring the tag used to identify it. Once you have switched a cluster to secondary mode, all of its current repositories and data will be removed and replaced with the data and state of the primary cluster.
Tip
You can enable the secondary cluster mode from any of the nodes of a cluster, not just the leader node. You can also use any of the healthy nodes of the primary cluster to connect to that cluster.
Prerequisites¶
Both clusters must share the same cluster secret.
The tags necessary for identifying the secondary cluster must already be added to the primary cluster’s list of tags.
Enable the secondary cluster mode using Workbench¶
From the interface, click the gear icon and go to the . Then provide the RPC address of a node from the primary cluster, and a tag to identify the secondary cluster.
Warning
Enabling the secondary mode will delete all data on the secondary cluster and replicate the state of the primary cluster.
Once in secondary mode, you can disable secondary mode and then you will no longer be in that mode. Doing this will stop the pulling of updates from the primary cluster.
Enable the secondary cluster mode using the REST API¶
You can enable the secondary mode using the REST API. See for the interactive REST API documentation. The examples below use curl.
To enable the secondary mode of a cluster, perform a POST with that tag to the /rest/cluster/config/secondary-mode endpoint of any of the nodes of that cluster. You must provide the RPC address of a primary cluster node and declare the identifier tag of the secondary cluster. The example below enables the secondary mode on a cluster that contains the node http://graphdb-eu-west-1.example.com:7200, connects to the primary cluster through the node with RPC address graphdb1.example.com:7300, and identifies the secondary cluster with the identifier tag eu-west.
curl -X POST -H 'Content-type: application/json' \
http://graphdb-eu-west-1.example.com:7200/rest/cluster/config/secondary-mode \
-d '{
"primaryNode": "graphdb1.example.com:7300",
"tag": "eu-west"
}'
Warning
Enabling the secondary mode will delete all data on the secondary cluster and replicate the state of the primary cluster.
Monitoring multi-region cluster deployments¶
Once a multi-region cluster has been configured, you can monitor the deployment using Workbench or the REST API.
You can monitor a multi-region cluster deployment from both the primary and the secondary cluster, but there are differences in what information about the multi-region deployment can be monitored between the two.
Monitoring from the primary cluster |
Monitoring from the secondary cluster |
|
|---|---|---|
Last log index pulled in a batched update by the secondary cluster |
✅ |
✅ (but only by that specific secondary cluster) |
Current latest log index of the primary cluster |
✅ |
❌ |
RPC address of the primary cluster’s leader |
✅ |
✅ |
RPC address of the secondary cluster’s leader |
❌ |
✅ |
Full list of identifier tags |
✅ |
❌ |
Note
The primary and secondary cluster maintain different log indexes. When monitoring a multi-region deployment, it is important not to confuse the two log indexes and terms.
Tip
Keep in mind that the primary cluster doesn’t maintain a list of secondary clusters, but only a list of identifier tags. There is no way to check whether any actual secondary clusters are using those tags at the moment.
Monitoring a multi-region cluster deployment using Workbench¶
From the Workbench, navigate to .
When viewed from the primary cluster, the tab will display the list of identifier tags and the last log index that each of these tags has requested.
When viewed from the secondary cluster, the tab will display the RPC address of the node of the primary cluster to which the secondary cluster is connected. It will also display the identifier tag associated with the secondary cluster, as well as the last log index received by the secondary cluster from the perspective of the primary cluster.
Monitoring a multi-region cluster deployment using the REST API¶
You can use the REST API to monitor clusters. See for the interactive REST API documentation. The examples below use curl.
Monitoring a multi-region cluster deployment from the primary cluster¶
A GET request to the rest/cluster/group/status endpoint retrieves information about all of the cluster’s nodes as well as information about the list of identifier tags used by the multi-region deployment.
curl -X GET --header http://graphdb1.example.com:7200/rest/cluster/group/status
Information about the secondary cluster is returned inside topologyStatus. Since the primary cluster does not maintain a list of secondary clusters, monitoring the multi-region deployment from the primary cluster provides only a list of associated identifier tags. For each of those tags, the response body provides the last log index update.
In the example below, we see that the primary cluster’s last log index is 10. Under topologyStatus there is one associated identifier tag eu-west, with 8 being the last log index that was sent to it.
{
"address": "graphdb1.example.com:7300",
"nodeState": "LEADER",
"term": 2,
"syncStatus": {
"graphdb2.example.com:7300": "IN_SYNC",
"graphdb3.example.com:7300": "IN_SYNC"
},
"lastLogTerm": 2,
"lastLogIndex": 10,
"endpoint": "http://graphdb1.example.com:7200",
"recoveryStatus": {},
"topologyStatus": {
"state": "PRIMARY_NODE",
"primaryTags": {
"eu-west": 8
},
{
"address": "graphdb2.example.com:7300",
"nodeState": "FOLLOWER",
"term": 2,
"syncStatus": {},
"lastLogTerm": 2,
"lastLogIndex": 10,
"endpoint": "http://graphdb2.example.com:7200",
"recoveryStatus": {},
"topologyStatus": {
"state": "PRIMARY_NODE"
}
},
{
"address": "graphdb3.example.com:7300",
"nodeState": "FOLLOWER",
"term": 2,
"syncStatus": {},
"lastLogTerm": 2,
"lastLogIndex": 10,
"endpoint": "http://graphdb3.example.com:7200",
"recoveryStatus": {},
"topologyStatus": {
"state": "PRIMARY_NODE"
}
}
Monitoring a multi-region cluster deployment from the secondary cluster¶
A GET request to the rest/cluster/group/status endpoint retrieves information about all of the cluster’s nodes as well as and information about the primary cluster.
curl -X GET --header http://graphdb-eu-west-1.example.com:7200/rest/cluster/group/status
The response body includes the topology of the cluster, the address of the primary cluster’s leader, and the last log index received from the primary cluster.
Note
Monitoring from a secondary cluster cannot return any information about any other secondary clusters connected to the primary cluster or the full list of identifier tags added to the primary cluster.