Migrating GraphDB Configurations¶

What’s in this document?

Compatibility between the versions of GraphDB, Connectors, and third-party connectors
Migrating without a cluster
Migrating with a cluster
Migrating connectors
Migrating plugins in a cluster
Migrating Helm charts

To migrate from one GraphDB version to another, follow the instructions in the last column of the table below, and then the steps described further down in this page.

Compatibility between the versions of GraphDB, Connectors, and third-party connectors ¶

GraphDB	RDF4J	Connectors	Elasticsearch	OpenSearch	Lucene	Solr	Kafka
10.4.x	4.3.6	16.2.0	8.9.2	2.8.0	8.11.2	9.1.1	3.5.1
10.4.x	Starting from 10.4, GraphDB uses LZ4 compression for backup and restore: Compressed backups created with GraphDB 10.4.0 and newer are not backward compatible and cannot be restored with older versions of GraphDB Restore procedures are backward compatible – in GraphDB 10.4.0 and newer you are able to restore backups created with older versions of GraphDB
10.3.x	4.3.3	16.1.0	8.8.1	2.8.0	8.11.2	9.1.1	3.3.1
10.3.x	If you used the Elasticsearch connector to connect to an OpenSearch server, follow the steps outlined in the OpenSearch documentation page. Elasticsearch connector was upgraded to use the new Java API client and new major version 8.8, which may or may not work with Elasticsearch server 7.x. For more information, see the Elasticsearch Java API Client documentation. Our tests indicate that the Elasticsearch Java API client 8.8 can connect to Elasticsearch 7.14 and newer. Solr connector was upgraded to new major version 9.1 of SolrJ (the Solr Java client). Solr previously claimed that the client is backward compatible, as outlined in the SolrJ/Solr cross-version compatibility documentation. However, the documentation for Solr 9.1 does not mention anything about version compatibility. Our tests indicate that the SolrJ 9.1 can connect to Solr server 8.x and newer.
10.2.x	4.2.2	16.0.5	7.17.7	Unsupported	8.11.2	8.11.2	3.3.1
10.2.x	By default all memory indexes are now on-heap, please adjust the memory settings according to the new sizing guidelines Introduced the `graphdb.external-url.enforce.transactions` property, which determines whether it is necessary to rewrite the Location header when no proxy is configured. Setting it to `true` will use the `graphdb.external-url` when building the transaction URLs.
10.1.x	4.2.0	16.0.2	7.16.3	Unsupported	8.11.1	8.11.1	3.3.1
10.1.x	No special attention required.
10.0.x	4.0.2	16.0.0	7.16.3	Unsupported	8.11.1	8.11.1	2.8.0
10.0.x	Introduced the new high-availability cluster where any node can be a leader or a follower (akin to the master and worker nodes in the old cluster). See the detailed migration procedure with a cluster below. Introduced new single repository type that replaces the existing Free, SE, and EE repositories: existing repositories will be automatically converted to the new type. If you have existing repository configuration templates outside a GraphDB installation, you need to convert them to the new type before using them with GraphDB 10. Redesigned filtering mechanism in the connectors: you need to rewrite the filters and recreate the connectors. See Migrating connectors below. The GraphDB REST API has been refactored, with some of the changes including: moving the Import and SPARQL template controllers to a new base URL, the use of kebab-case for compound words in URLs, the removal of the `Header X-GraphDB-Password` from the Security management controller, and more. See Using the GraphDB REST API for more information. Refactored remote locations – they cannot be activated any more but all repositories in remote locations are accessible via the Workbench. OntoRefine has been removed from GraphDB and is now developed as a separate product - see note above. If you are upgrading from GraphDB 8.x or older, please upgrade to GraphDB 9.11 before you upgrade to GraphDB 10.0.
See compatibility for GraphDB versions 9.x and older.

Migrating without a cluster ¶

To migrate your GraphDB configuration and data, follow the steps below.

Warning

Keep in mind that after that, you cannot automatically revert to GraphDB 9.x.

Stop the GraphDB 9.x instance.
Back up your repositories and configuration – this will ensure you can revert safely if something goes wrong during the upgrade.
1. To back up all repositories, copy the data directory. See also Backing up and Restoring a Repository for additional ways to backup repository data.
2. To back up all configuration, copy the work and conf directories.
Your existing GraphDB 9.x home directory (containing the conf, data, and work directories) can be used directly as the GraphDB 10.0 home directory.

Tip

You can also copy the conf, data, and work directories from the GraphDB 9.x home directory to a new directory to use as the GraphDB 10.0 home directory. In this case, your GraphDB 9.x home directory is also the backup so you may skip the backup steps.

The various directories are described in detail here.
Start GraphDB 10.0.
If you use any GraphDB connectors, please follow the guidelines in Migrating connectors.

The cluster in GraphDB 10 is based on an entirely new approach and is not directly comparable or compatible with the cluster in GraphDB 9.x. See the High Availability Cluster Basics for more details on how the new GraphDB cluster operates.

The described procedures refer to the three recommended cluster topologies in the 9.x cluster: single master with three or more workers; two masters sharing workers, one of the masters is read-only; and multiple masters with dedicated workers. See more about 9.x cluster topologies.

Understand ¶

You will need an existing GraphDB 9.x cluster in good condition before you start the migration. Data and configuration will be copied from two of the nodes:

A worker node that is in sync with the master. This node will provide:
- The data for each repository that is part of the GraphDB 9.x cluster.
- Any repositories that are not part of the cluster, e.g., an Ontop repository created on the same instance as the worker repository. Typically, these are used via internal SPARQL federation in the cluster.
A master node that will provide:
- The user database containing users, credentials, and user settings.
- Any repositories that are not part of the cluster, e.g., an Ontop repository created on the same instance as the master repository. Typically, these are used by connecting to the repository via HTTP – directly or via standard SPARQL federation.
- The graphdb.properties file that contains all GraphDB configuration properties.

The instructions below assume your GraphDB 9.x setup has a single home directory that contains the conf, data, and work directories. If your setup uses explicitly configured separate directories for any of these, you need to adjust the instructions accordingly. The various directories are described in detail here.

Important

The cluster in GraphDB 10 is configured at the instance level, while the cluster in GraphDB 9.x is defined per repository. This means that every repository you migrate following the steps below will automatically become part of the cluster.

Once a cluster is created, it is not possible to have a repository that is not part of the cluster in GraphDB 10.

Prepare ¶

In order to minimize downtime during the migration, you may want to keep the GraphDB 9.x cluster running in read-only mode while performing the migration.

To make a master read-only, go to Setup ‣ Cluster, click on the master node and enable the read-only setting:

Alternatively, you can reconfigure your application such that it does not do any writes during the migration.

Procedure ¶

To migrate a cluster configuration from GraphDB version 9.x to the 10.0 cluster, please follow the steps outlined below.

Warning

The instructions are written in such a way that your existing GraphDB 9.x setup is preserved so you can abort the migration at any point and revert to your previous setup. Note that once you decide to go live with the migrated GraphDB 10 setup, there is no automatic way to revert that configuration to GraphDB 9.x.

First, choose a temporary GraphDB 10 home directory that will be used to copy files and directories and bootstrap all the nodes.

Tip

All instructions below mean this directory when “temporary GraphDB 10 home directory” is mentioned.
Select one of the worker nodes that is in sync with the master.
Stop the GraphDB 9.x instance where the worker node is located – the rest of the GraphDB 9.x cluster will remain operational.
Locate the data directory within the GraphDB 9.x home directory of the worker node and copy it to the temporary GraphDB 10 home directory.
- The data/repositories directory contains all repositories and their data.
- If any repository is a master repository, delete it from the copy.
Select one of the master nodes.
Stop the GraphDB 9.x instance where the master node is located – you may want to point your application to another master or a worker repository so that read operations will continue to work during the migration.
Locate the data directory within the GraphDB 9.x home directory of the master node and copy it to the temporary GraphDB home directory.
- The data/repositories directory contains all repositories and their data.
- If any repository is a master repository, do not copy it.
- If you have only master repositories on the master node you can skip this step.
Locate the work directory within the GraphDB 9.x home directory of the master node and copy it to the temporary GraphDB home directory.
- On GraphDB 9.x, the work directory contains the user database.
  
  Note
  
  After copying the work directory from the master to the new nodes, the old locations of the GraphDB 9.x cluster workers will be visible in the Workbench of the new nodes. We recommend deleting the old locations.
Locate the conf directory within the GraphDB 9.x home directory of the master node and copy it to the temporary GraphDB home directory.
- The conf directory contains the graphdb.properties file.
Choose the number of nodes for the new cluster. Due to the nature of the Raft consensus algorithm on which the GraphDB 10 cluster is based, an odd number of nodes is recommended, e.g., three, five, or seven.

As a rule of thumb, use as many nodes as the number of workers you had but add or remove a node to make the number odd. For example:
- If you had three workers, use three nodes.
- If you had six workers, use five or seven nodes.
Copy the temporary GraphDB 10 home directory to each node to serve as the GraphDB 10 on that node.
Edit the graphdb.properties file on each node to reflect any settings specific to that node, e.g., graphdb.external-url or SSL certificate properties but keep general properties, especially graphdb.auth.token.secret and any security-related properties identical on all nodes.
- If necessary, consult the graphdb.properties file on that node from your GraphDB 9.x setup.
- If the nodes are hosted on the same machine, edit the graphdb.connector.port property so that it is different for each node.
- See also the notes on configuring networking properties related to the GraphDB 10 cluster.
Start GraphDB 10 on each node.
- Make sure each node is up and has a valid EE license. If no license is applied, you will be able to create the cluster with all nodes in state Follower - no leader will be elected. However, if you attempt to run a query on any of them, their state will change to Restricted.
On any of the instances that you just created, go to Setup ‣ Cluster in the Workbench and create the cluster group.

See more information about the new Workbench user interface for creating, configuring, and accessing a cluster.
- You can also create it via the Workbench REST API.
If you use any GraphDB connectors, please follow the guidelines in Migrating connectors.

Reverting the procedure ¶

You can revert to your old setup by restarting the worker and master nodes that you stopped while performing the migration.

If you set your master to read-only, do not forget to set it back to write mode using the same Workbench interface that you used to make it read-only.

Example migration ¶

Given the following GraphDB 9.x cluster setup consisting of two masters and three workers for each master, or a total of eight GraphDB instances:

graphdb1.example.com

Master repository master1, the primary master repository
Worker repository mydata, which is not part of any cluster

graphdb2.example.com

Master repository master2, the secondary master repository

graphdb3.example.com

Worker repository worker1 connected to master1
Ontop repository sql1

graphdb4.example.com

Worker repository worker2 connected to master1
Ontop repository sql1

graphdb5.example.com

Worker repository worker3 connected to master1
Ontop repository sql1

graphdb6.example.com

Worker repository worker4 connected to master2
Ontop repository sql1

graphdb7.example.com

Worker repository worker5 connected to master2
Ontop repository sql1

graphdb8.example.com

Worker repository worker6 connected to master2
Ontop repository sql1

You choose the worker worker1 and the master master1 to perform the migration.

After completing the steps that copy files from the worker and the master, you should have a directory structure in the temporary GraphDB 10 home that looks like this:

Directory	Description
`data/repositories/worker1/`	The worker repository copied from the worker node
`data/repositories/sql1/`	The Ontop repository copied from the worker node
`data/repositories/mydata/`	The non-clustered worker repository copied from the master node
`conf/graphdb.properties`	The GraphDB configuration file copied from the master node
`work/workbench/settings.js`	The GraphDB 9.x Workbench settings and user database copied from the master node

There may be other files in the data, conf, and work directories, e.g., conf/logback.xml, that are safe to have in the copy in order to preserve as much of the same configuration as possible.

Note, however, that you should NOT have the following directories:

Directory	Description
`data/repositories/master1/`	The master repository from the master node should NOT be copied

Since you have six workers in the GraphDB 9.x cluster, it makes sense to choose five (the number of workers minus one to make the number odd) nodes for the GraphDB 10.0 cluster.

If you proceed with the migration, your cluster will contain three repositories that are part of the same cluster:

Repository ID	Description
`worker1`	Migrated GraphDB repository – note it uses the repository ID from the worker node you used to copy the files from
`sql1`	Migrated Ontop repository
`mydata`	Migrated GraphDB repository that previously was not part of any cluster

Configuring external cluster proxy ¶

See how to configure the external GraphDB 10.0 cluster proxy here.

Migrating connectors ¶

GraphDB 10.0 introduces major changes to the filtering mechanism of the connectors. Existing connector instances will not be usable and attempting to use them for queries or updates will throw an error.

If your connector definitions do not include an entity filter, you can simply repair them.

If your connector definitions do include an entity filter, you need to rewrite the filter using the new filter options.

See the migration steps from GraphDB 9.x for Lucene, Solr, Elasticsearch, and Kafka.

Migrating plugins in a cluster ¶

When upgrading to a newer GraphDB version, it might contain plugins that are not present in the older version. In this case, and when using a cluster, the Plugin Manager disables the newly detected plugins, so you need to enable them by executing the following SPARQL query:

insert data {
    [] <http://www.ontotext.com/owlim/system#startplugin> "plugin-name"
}

Then create your plugin following the steps described in the corresponding documentation, and also make sure to not delete the database in the plugin you are using.

You can also stop a plugin before the migration in case you deem it necessary:

insert data {
    [] <http://www.ontotext.com/owlim/system#stopplugin> "plugin_name"
}

Migrating Helm charts ¶

From version 9.8 onwards, GraphDB Enterprise Edition can be deployed with open-source Helm charts. See how to migrate them to GraphDB 10.0 here.