Managing a Cluster

GraphDB cluster management is handled via Java Management Extensions Technology (JMX). Creating a cluster consists of several steps, which, depending on the topology, are setting or updating a set of JMX properties, and are executing JMX operations, e.g., adding a worker node to the master. GraphDB provides a JMX-HTTP bridge for easier management of the JMX configuration using a Jolokia request. Of course, you can use any other software for working with JMX, e.g., JConsole, Java VisualVM, as well as third party clients. e.g., Jmxterm.

Note

To use the JMX interface, you need to configure the JMX endpoint.

Managing workers

Create a worker

To create workers, follow the procedure described in Create a worker.

Clone a worker

To clone a worker:

  1. Click a worker node to view its information.

  2. Click the Clone to another location button.

  3. In the dialog that opens, enter the ID and the title, and choose the target location.

Add a worker to the cluster

Using the Workbench

  1. Go to Setup ‣ Cluster management.

    The cluster manager visualizes the nodes (masters and workers) and the connections between them. The workers are represented by a W bucket, and the masters - by an M bucket. The connections between the nodes are shown in different colors and patterns depending on the status of each worker. All possible statuses are listed in the legend on the bottom left.

    Each master has a different color, and every worker connected to it takes its color. If a worker is connected to more than one master, it is constantly switching between their respective colors. When a worker has not been connected to any master(s) yet, its color is grey.

    ../_images/cluster_management.png

  2. Drag and drop nodes to connect them. Note that you can connect workers to masters, and a master to another master, but not a worker to another worker. Masters always connect bi-directionally.

  3. Click on a worker node to check its status or to clone it to another location as described above.

  4. Click on a master node to check its status, or to control its read-only state and normal/mute mode. These configurations apply for both local and remote masters.

    ../_images/master_status.png

  5. When connecting to workers and masters from a remote location, this is indicated by the location label below each node, as is the case here with r_master1 and r_worker1:

    ../_images/remote_nodes.png

If you disconnect these remote nodes, in this case they will remain visible due to the fact that they were both connected to the local master, displayed the following way:

../_images/remote_nodes_disconnected.png

Tip

The cluster manager continually updates the visualization so you can use it as a tool for monitoring the status of the workers. In case you experience errors or issues, check if your attached remote locations are accessible, and if the JMX settings for these locations are correct.

Using Jolokia

Link a worker to a master by executing the following request in a browser:

http://masterhost:<port>/jolokia/exec/ReplicationCluster:name=ClusterInfo!/masterID/addClusterNode/http:!/!/workerhost:<port>!/repositories!/workerID/0/true

Using CURL

curl -H 'content-type: application/json' \
  -d "{\"type\":\"exec\",\"mbean\":\"ReplicationCluster:name=ClusterInfo\/masterID\",\"operation\":\"addClusterNode\",\"arguments\":[\"http://workerhost:<port>/repositories/workerID\",0,true]}" \
  http://masterhost:<port>/jolokia/

Using JMX

After the instantiation of a master node, worker nodes can be added using a JMX client application, e.g., JConsole. From the MBeans tab, select the bean associated with the master node to be modified. Each bean will be named: ReplicationCluster/ClusterInfo/<repository_id>.

  1. Configure the JMX client application.

  2. Add worker nodes using the addClusterNode operation.

  3. Check the attribute NodeStatus to ensure that the worker node has been added correctly. This is an array and needs to be expanded by double-clicking it. The newly added worker node is shown here with its full URL. Its status should be ON.

Notifications and statistics can be monitored through the JMX client.

Note

All masters are always in read-write mode. There is no additional step to put the master in RW mode, as in previous GraphDB versions.

Remove a worker from the cluster

Using the Workbench

To remove a worker from a cluster, click the links connecting the worker to the other nodes.

Using Jolokia

Remove a worker from a cluster by executing the following request in a browser:

http://masterhost:<port>/jolokia/exec/ReplicationCluster:name=ClusterInfo!/masterID/removeClusterNode/http:!/!/workerhost:<port>!/repositories!/workerID

Using CURL

curl -H 'content-type: application/json' \
  -d "{\"type\":\"exec\",\"mbean\":\"ReplicationCluster:name=ClusterInfo\/masterID\",\"operation\":\"removeClusterNode\",\"arguments\":[\"http://workerhost:<port>/repositories/workerID\"]}" \
  http://masterhost:<port>/jolokia/

Using JMX

Worker nodes are removed using the removeClusterNode operation, which requires only the repository URL as a parameter.

Note

If a master node is assigned two worker nodes of unknown but different status, then it will not be able to decide which of the two is the correct one. In this situation, both workers should be removed, the master’s transaction log cleared, and the workers added back with the correct worker first.

Reconfigure a worker

To reconfigure a worker, follow the procedure described in Configuring a repository.

Managing masters

Create a master

To create a master, follow described procedure.

Initialize a master

To initialize a master, follow the described procedure.

Reconfigure a master

To reconfigure a master, follow the described procedure.

Add a new master to the cluster

Using the Workbench

To add a master as a sync peer to a second master, follow the same procedure as for adding a worker.

Warning

Master-to-master connections are always created in both directions.

Using Jolokia

Add a master as a sync peer to a second master by executing the following request:

http://master2host:<port>/jolokia/exec/ReplicationCluster:name=ClusterInfo!/master2ID/addSyncPeer/master1ID/http:!/!/master1host:<port>!/repositories!/master1ID

Using CURL

curl -H 'content-type: application/json' \
  -d "{\"type\":\"exec\",\"mbean\":\"ReplicationCluster:name=ClusterInfo\/master1ID\",\"operation\":\"addSyncPeer\",\"arguments\":[\"master1ID\",\"http://master1host:<port>/repositories/master1ID\"]}" \
  http://master2host:<port>/jolokia/

Using JMX

Another option is to use the JMX Console. After the instantiation of a master node, other masters can be added via the MBeans tab. In order to do this, select the bean associated with the master node to be modified. Each bean will be named: ReplicationCluster/ClusterInfo/<repository_id>.

Master nodes can be added using the addSyncPeer operation with the following parameters:

Using cluster.properties

If you want to set up several master nodes for the cluster, properly configure the config file cluster.properties of each master before bringing it online:

  • location <master-repository>/cluster.properties

  • format

    version=2.0
node-id=master3

#own workers
node.1.url=http://node1:7200/repositories/worker
node.2.url=http://node2:7200/repositories/worker

#peers
peer.1.id=master1
peer.1.url=http://server1:7200/repositories/master
peer.2.id=master2
peer.2.url=http://server2:7200/repositories/master


#replication
min-time-to-consider-full-replication-s=600
net-speed-bits-per-sec=104857600

#misc
remote-master=false
slow-op-threshold-ms=0
system-transaction-password=
timeout=-1
update-timeout=-1

Remove a master from the cluster

Using the Workbench

To remove a master from a cluster, click the links connecting it to the other nodes.

Using Jolokia

Remove a master as a sync peer to a second master by executing the request:

http://master2host:<port>/jolokia/exec/ReplicationCluster:name=ClusterInfo!/master2ID/removeSyncPeer/master1ID

Using CURL

curl -H 'content-type: application/json' \
  -d "{\"type\":\"exec\",\"mbean\":\"ReplicationCluster:name=ClusterInfo\/master1ID\",\"operation\":\"removeSyncPeer\",\"arguments\":[\"master1ID\"]}" \
  http://master2host:<port>/jolokia/

Using JMX

Master nodes are removed using the removeSyncPeer operation, which requires only the ID of the master.

Warning

Keep in mind that master-to-master connections are never unidirectional, so you have to remove the sync peer in both directions.

Checking the cluster status

  1. Start all cluster nodes.

  2. Go to http://localhost:7200 - local if you have a Workbench on the same server where the master node is (http://localhost:7200/repositories/master), or a remote location if the Workbench is on another server (http://masterhost:7200/repositories/master).

  3. Copy the master repository ID.

  4. Start a JMX client, e.g., JConsole, from the Java Software Development Kit.

  5. After connecting, select the MBeans tab and open the ReplicationCluster/ClusterInfo/<repository_id> bean (if it is not visible, copy and paste in a browser http://masterhost:<port>/repositories/<master-repository-id>/size, or execute it using curl).

  6. Check the ClusterStatusSummary attribute to ensure that the cluster is available. Its status should be ON.

Secured connection

The communication between the cluster nodes can be performed over HTTPS. The configuration is straightforward:

  1. Enable HTTPS on the hosts.

  2. Make sure the Locations configured in Setup ‣ Repositories are configured to use HTTPS.

In case the nodes’ certificates are self-signed, you need to add them to the other nodes’ JVM SSL TrustStores. Here, you should keep in mind that:

  1. Both masters and workers need to trust the other nodes’ certificates. For example, if master1 and worker1 are in a cluster, you should have worker1’s certificate in master1’s TrustStore, and vice versa.

  2. The master should have its own certificate in its TrustStore, otherwise you would not be able to use the Cluster Management from the Workbench.