Benchmarks

Our engineering team invests constant efforts in measuring the database data loading and query answering performance. The section covers common database scenarios tested with popular public benchmarks and their interpretation in the context of common RDF use cases.

LDBC Semantic Publishing Benchmark 2.0

LDBC is an industry association aimed to create TPC-like benchmarks for RDF and graph databases. The association is founded by a consortium of database vendors like Ontotext, OpenLink, Neo Technologies, Oracle, IBM, and SAP, among others. The Semantic Publishing Benchmark (SPB) simulates the database load commonly faced by media or publishing organizations. The synthetic generated dataset is based on BBC’s Dynamic Semantic Publishing use case. It contains a graph of linked entities like creative works, persons, documents, products, provenance, and content management system information. All benchmark operations follow a standard authoring process - add new metadata, update the reference knowledge, and search queries hitting various checkpoints as join performance, data access locality, expression calculation, parallelism, concurrency, and correlated subqueries.

Data loading

This section illustrates how quickly GraphDB can do an initial data load. The SPB-256 dataset represents the size of a mid-sized production database managing documents and metadata. The data loading test run measures how the GraphDB edition and the selection of i3 instances affect the processing of 237K explicit statements, including the materialization of the inferred triples generated by the reasoner.

Table 1: Loading time of the LDBC SPB-256 dataset with the default RDFS-Plus-optimized ruleset in minutes

Editions

Ruleset

Explicit statements

Total statements

AWS instance

Cores

Loading time (minutes)

9.2 Free

RDFS-Plus-optimized

237,802,643

385,168,491

i3.large

1*

481

9.2 SE/EE

RDFS-Plus-optimized

237,802,643

385,168,491

i3.large

2

391

9.2 SE/EE

RDFS-Plus-optimized

237,802,643

385,168,491

i3.xlarge

4

302

9.2 SE/EE

RDFS-Plus-optimized

237,802,643

385,168,491

i3.2xlarge

8

254

9.2 SE/EE

RDFS-Plus-optimized

237,802,643

385,168,491

i3.4xlarge

16

246

* GraphDB Free uses a single CPU core only.

Loading the dataset with RDF-Plus-optimized ruleset generates an additional nearly 150M implicit statements or expansion of 1:1.6 from the imported explicit triples. GraphDB Free produces the slowest performance due to a limitation of a single write thread. The Standard and Enterprise editions scale with the increase of the available CPU cores until the I/O performance throughput becomes a major limiting factor.

Table 2: Loading time of the LDBC SPB-256 dataset with the default OWL2-RL ruleset in minutes

Editions

Ruleset

Explicit statements

Total statements

AWS instance

Cores

Loading time (minutes)

9.2 SE/EE

OWL2-RL

237,802,643

752,341,659

i3.large

2

1254

9.2 SE/EE

OWL2-RL

237,802,643

752,341,659

i3.xlarge

4

817

9.2 SE/EE

OWL2-RL

237,802,643

752,341,659

i3.2xlarge

8

652

9.2 SE/EE

OWL2-RL

237,802,643

752,341,659

i3.4xlarge

16

613

The same dataset tested with OWL2-RL ruleset produces nearly 515M implicit statements, or an expansion of 1:3.2 from the imported explicit triples. The data loading performance scales much better with the increase of additional CPU cores due to much higher computational complexity. Once again, the I/O performance throughput becomes a major limiting factor, but the conclusion is that datasets with a higher reasoning complexity benefit more from the additional CPU cores.

Production load

The test demonstrates the execution speed of small-sized transactions and read queries against the SPB-256 dataset preloaded with RDFS-Plus-optimized ruleset. The query mix includes transactions generating updates and information searches with simple or complex aggregate queries. The different runs compare the database performance according to the number of concurrent read and write clients.

Table 3: The number of executed query mixes per second (higher is better) vs. the number of concurrent clients.

Server instance

Price

Disk

Concurrent read agents

Read query mixes per second

Concurrent write agents

Write per second

c4.4xlarge

$0.796

EBS (5K IOPS)

0

4

8.03

i3.4xlarge

$1.248

local NVMe SSD

0

4

27.08

c5d.4xlarge

$0.768

local NVMe SSD

0

4

33.02

c4.4xlarge

$0.796

EBS (5K IOPS)

16

23.06

0

i3.4xlarge

$1.248

local NVMe SSD

16

56.54

0

c5d.4xlarge

$0.768

local NVMe SSD

16

87.78

0

c4.4xlarge

$0.796

EBS (5K IOPS)

8

20.37

4

5.19

i3.4xlarge

$1.248

local NVMe SSD

8

34.25

4

18.75

c5d.4xlarge

$0.768

local NVMe SSD

8

50.28

4

22.12

c4.4xlarge

$0.796

EBS (5K IOPS)

12

21.80

4

1.97

i3.4xlarge

$1.248

local NVMe SSD

12

43.60

4

12.38

c5d.4xlarge

$0.768

local NVMe SSD

12

66.53

4

17.69

c4.4xlarge

$0.796

EBS (5K IOPS)

16

21.37

4

1.33

i3.4xlarge

$1.248

local NVMe SSD

16

52.36

4

3.52

c5d.4xlarge

$0.768

local NVMe SSD

16

79.94

4

8.81

Notes: All runs use the same configuration limited to 20GB heap size on instances with 16 vCPU. The AWS price is based on the US East coast for an on-demand type of instance (Q1 2020), and does not include the EBS volume charges that are substantial only for IOP partitions.

The instances with local NVMe SSD devices substantially outperform any EBS drives due to the lower disk latency and higher bandwidth. In the case of standard and cheapest EBS gp2 volumes, the performance is even slower after the AWS IOPs throttling starts to limit the disk operations. The c5d.4xlarge instances achieve consistently fastest results with the main limitation of small local disks. Next in the list are i3.4xlarge instances offering substantially bigger local disks. Our recommendation is to avoid using the slow EBS volumes, except for cases where you plan to limit the database performance load. Cluster setup with multiple worker nodes with local storage will always outperform significantly any instance with EBS volumes.

Berlin SPARQL Benchmark (BSBM)

BSBM is a popular benchmark combining read queries with frequent updates. It covers a less demanding use case without reasoning, generally defined as eCommerce, describing relations between products and producers, products and offers, offers and vendors, products and reviews.

The benchmark features two runs, where the “explore” run generates requests like “find products for a given set of generic features”, “retrieve basic information about a product for display purpose”, “get recent review”, etc. The “explore and update” run mixes all read queries with information updates.

Table 4: BSBM 100M query mixes per hour on AWS instance - c5d.4xlarge, local NVMe SSD with GraphDB 9.2 EE and ruleset RDFS-Plus-optimized

Threads

explore (query mixes per hour)

explore & update (query mixes per hour)

1

8,766

7,726

2

14,618

13,095

4

25,353

21,559

8

38,778

29,484

12

46,962

30,681

16

52,787

30,503