Benchmarks

Our engineering team invests constant efforts in measuring the database data loading and query answering performance. The section covers common database scenarios tested with popular public benchmarks and their interpretation in the context of common RDF use cases.

LDBC Semantic Publishing Benchmark 2.0

LDBC is an industry association aimed to create TPC-like benchmarks for RDF and graph databases. The association is founded by a consortium of database vendors like Ontotext, OpenLink, Neo Technologies, Oracle, IBM, and SAP, among others. The SPB (Semantic Publishing Benchmark) simulates the database load commonly faced by media or publishing organizations. The synthetic generated dataset is based on BBC’s Dynamic Semantic Publishing use case. It contains a graph of linked entities like creative works, persons, documents, products, provenance, and content management system information. All benchmark operations follow a standard authoring process – add new metadata, update the reference knowledge, and search queries hitting various choke points as join performance, data access locality, expression calculation, parallelism, concurrency, and correlated subqueries.

Data loading

This section illustrates how quickly GraphDB can do an initial data load. The SPB-256 dataset represents the size of a mid-sized production database managing documents and metadata. The data loading test run measures how the GraphDB edition and the selection of i3 instances affect the processing of 237K explicit statements, including the materialization of the inferred triples generated by the reasoner.

Table 1: Loading time of the LDBC SPB-256 dataset with the default RDFS-Plus-optimized ruleset in minutes

Editions

Ruleset

Explicit statements

Total statements

AWS instance

Cores

Loading time (minutes)

10.4 Free

RDFS-Plus-optimized

256,057,106

401,961,832

r6id.xlarge

1*

247

10.4 SE/EE

RDFS-Plus-optimized

256,057,106

401,961,832

r6id.xlarge

2

232

10.4 SE/EE

RDFS-Plus-optimized

256,057,106

401,961,832

r6id.xlarge

4

230

10.4 SE/EE

RDFS-Plus-optimized

256,057,106

401,961,832

r6id.2xlarge

8

207

10.4 SE/EE

RDFS-Plus-optimized

256,057,106

401,961,832

r6id.4xlarge

16

206

* GraphDB Free uses a single CPU core only.

Loading the dataset with RDF-Plus-optimized ruleset generates an additional nearly 150M implicit statements or expansion of 1:1.6 from the imported explicit triples. GraphDB Free produces the slowest performance due to a limitation of a single write thread. The Standard and Enterprise editions scale with the increase of the available CPU cores until the I/O performance throughput becomes a major limiting factor.

Table 2: Loading time of the LDBC SPB-256 dataset with the default OWL2-RL ruleset in minutes

Editions

Ruleset

Explicit statements

Total statements

AWS instance

Cores

Loading time (minutes)

10.4 SE/EE

OWL2-RL

256,057,106

775,059,522

r6id.xlarge

2

600

10.4 SE/EE

OWL2-RL

256,057,106

775,059,522

r6id.xlarge

4

580

10.4 SE/EE

OWL2-RL

256,057,106

775,059,522

r6id.2xlarge

8

470

10.4 SE/EE

OWL2-RL

256,057,106

775,059,522

r6id.4xlarge

16

486

The same dataset tested with OWL2-RL ruleset produces nearly 515M implicit statements, or an expansion of 1:3.2 from the imported explicit triples. The data loading performance scales much better with the increase of additional CPU cores due to much higher computational complexity. Once again, the I/O performance throughput becomes a major limiting factor, but the conclusion is that datasets with a higher reasoning complexity benefit more from the additional CPU cores.

Production load

The test demonstrates the execution speed of small-sized transactions and read queries against the SPB-256 dataset preloaded with RDFS-Plus-optimized ruleset. The query mix includes transactions generating updates and information searches with simple or complex aggregate queries. The different runs compare the database performance according to the number of concurrent read and write clients.

Table 3: The number of executed query mixes per second (higher is better) vs. the number of concurrent clients.

Server instance

Price

Disk

Concurrent read agents

Read query mixes per second

Concurrent write agents

Write per second

r6i.4xlarge

$1.008

EBS (5K IOPS)

0

4

22.31

i3.4xlarge

$1.248

local NVMe SSD

0

4

21.61

r6id.4xlarge

$1.209

local NVMe SSD

0

4

33.63

r6i.4xlarge

$1.008

EBS (5K IOPS)

16

179.75

0

i3.4xlarge

$1.248

local NVMe SSD

16

77.58

0

r6id.4xlarge

$1.209

local NVMe SSD

16

173.40

0

r6i.4xlarge

$1.008

EBS (5K IOPS)

8

115.95

4

17.38

i3.4xlarge

$1.248

local NVMe SSD

8

51.97

4

14.47

r6id.4xlarge

$1.209

local NVMe SSD

8

111.80

4

23.80

r6i.4xlarge

$1.008

EBS (5K IOPS)

12

147.77

4

15.02

i3.4xlarge

$1.248

local NVMe SSD

12

63.70

4

12.10

r6id.4xlarge

$1.209

local NVMe SSD

12

145.20

4

20.14

r6i.4xlarge

$1.008

EBS (5K IOPS)

16

157.07

4

11.24

i3.4xlarge

$1.248

local NVMe SSD

16

70.60

4

8.53

r6id.4xlarge

$1.209

local NVMe SSD

16

158.37

4

15.26

Notes: All runs use the same configuration limited to 20GB heap size on instances with 16 vCPU. The AWS price is based on the US East coast for an on-demand type of instance (Q1 2020), and does not include the EBS volume charges that are substantial only for IOP partitions.

The instances with local NVMe SSD devices substantially outperform any EBS drives due to the lower disk latency and higher bandwidth. In the case of standard and cheapest EBS gp2 volumes, the performance is even slower after the AWS IOPs throttling starts to limit the disk operations. The c5d.4xlarge instances achieve consistently fastest results with the main limitation of small local disks. Next in the list are i3.4xlarge instances offering substantially bigger local disks. Our recommendation is to avoid using the slow EBS volumes, except for cases where you plan to limit the database performance load.

Berlin SPARQL Benchmark (BSBM)

BSBM is a popular benchmark combining read queries with frequent updates. It covers a less demanding use case without reasoning, generally defined as eCommerce, describing relations between products and producers, products and offers, offers and vendors, products and reviews.

The benchmark features two runs, where the “explore” run generates requests like “find products for a given set of generic features”, “retrieve basic information about a product for display purpose”, “get recent review”, etc. The “explore and update” run mixes all read queries with information updates.

Table 4: BSBM 100M query mixes per hour on AWS instance – r6id.4xlarge, local NVMe SSD with GraphDB 10.4 EE, ruleset RDFS-Plus-optimized and excluded Query 5

Threads

explore (query mixes per hour)

explore & update (query mixes per hour)

1

62,170

9,884

2

119,241

13,286

4

217,509

13,320

8

13,008

12

13,389

16

13,307