Benchmarks

Our engineering team invests constant efforts in measuring the database data loading and query answering performance. The section covers common database scenarios tested with popular public benchmarks and their interpretation in the context of common RDF use cases.

LDBC Semantic Publishing Benchmark 2.0

LDBC is an industry association aimed to create TPC-like benchmarks for RDF and graph databases. The association is founded by a consortium of database vendors like Ontotext, OpenLink, Neo Technologies, Oracle, IBM, and SAP, among others. The Semantic Publishing Benchmark (SPB) simulates the database load commonly faced by media or publishing organizations. The synthetic generated dataset is based on BBC’s Dynamic Semantic Publishing use case. It contains a graph of linked entities like creative works, persons, documents, products, provenance, and content management system information. All benchmark operations follow a standard authoring process - add new metadata, update the reference knowledge, and search queries hitting various checkpoints as join performance, data access locality, expression calculation, parallelism, concurrency, and correlated subqueries.

Data loading

This section illustrates how quickly GraphDB can do an initial data load. The SPB-256 dataset represents the size of a mid-sized production database managing documents and metadata. The data loading test run measures how the GraphDB edition and the selection of i3 instances affect the processing of 237K explicit statements, including the materialization of the inferred triples generated by the reasoner.

Table 1: Loading time of the LDBC SPB-256 dataset with the default RDFS-Plus-optimized ruleset in minutes

Editions Ruleset Explicit statements Total statements AWS instance Cores Loading time (minutes)
9.0 Free RDFS-Plus-optimized 237,802,643 385,168,491 i3.large 2* 571
9.0 SE/EE RDFS-Plus-optimized 237,802,643 385,168,491 i3.large 2 374
9.0 SE/EE RDFS-Plus-optimized 237,802,643 385,168,491 i3.xlarge 4 300
9.0 SE/EE RDFS-Plus-optimized 237,802,643 385,168,491 i3.2xlarge 8 253
9.0 SE/EE RDFS-Plus-optimized 237,802,643 385,168,491 i3.4xlarge 16 234

* GraphDB Free uses a single CPU core only.

Loading the dataset with RDF-Plus-optimized ruleset generates an additional nearly 150M implicit statements or expansion of 1:1.6 from the imported explicit triples. GraphDB Free produces the slowest performance due to a limitation of a single write thread. The Standard and Enterprise editions scale with the increase of the available CPU cores until the I/O performance throughput becomes a major limiting factor.

Table 2: Loading time of the LDBC SPB-256 dataset with the default OWL2-RL ruleset in minutes

Editions Ruleset Explicit statements Total statements AWS instance Cores Loading time (minutes)
9.0 SE/EE OWL2-RL 237,802,643 752,341,659 i3.large 2 1425
9.0 SE/EE OWL2-RL 237,802,643 752,341,659 i3.xlarge 4 958
9.0 SE/EE OWL2-RL 237,802,643 752,341,659 i3.2xlarge 8 699
9.0 SE/EE OWL2-RL 237,802,643 752,341,659 i3.4xlarge 16 644

The same dataset tested with OWL2-RL ruleset produces nearly 515M implicit statements, or an expansion of 1:3.2 from the imported explicit triples. The data loading performance scales much better with the increase of additional CPU cores due to much higher computational complexity. Once again, the I/O performance throughput becomes a major limiting factor, but the conclusion is that datasets with a higher reasoning complexity benefit more from the additional CPU cores.

Production load

The test demonstrates the execution speed of small-sized transactions and read queries against the SPB-256 dataset preloaded with RDFS-Plus-optimized ruleset. The query mix includes transactions generating updates and information searches with simple or complex aggregate queries. The different runs compare the database performance according to the number of concurrent read and write clients.

Table 3: The number of executed query mixes per second (higher is better) vs. the number of concurrent clients.

Server instance Price Disk Concurrent read agents Read query mixes per second Concurrent write agents Write per second
c4.4xlarge $0.796 EBS (5K IOPS) 0
4 8.95
i3.4xlarge $1.248 local NVMe SSD 0
4 28.66
c5d.4xlarge $0.768 local NVMe SSD 0
4 33.55
c4.4xlarge $0.796 EBS (5K IOPS) 16 17.33 0
i3.4xlarge $1.248 local NVMe SSD 16 60.71 0
c5d.4xlarge $0.768 local NVMe SSD 16 94.81 0
c4.4xlarge $0.796 EBS (5K IOPS) 8 15.63 4 2.32
i3.4xlarge $1.248 local NVMe SSD 8 35.68 4 19.04
c5d.4xlarge $0.768 local NVMe SSD 8 51.28 4 23.01
c4.4xlarge $0.796 EBS (5K IOPS) 12 21.66 4 1.53
i3.4xlarge $1.248 local NVMe SSD 12 45.48 4 16.46
c5d.4xlarge $0.768 local NVMe SSD 12 63.07 4 17.40
c4.4xlarge $0.796 EBS (5K IOPS) 16 17.85 4 1.23
i3.4xlarge $1.248 local NVMe SSD 16 56.25 4 4.29
c5d.4xlarge $0.768 local NVMe SSD 16 77.68 4 10.52

Notes: All runs use the same configuration limited to 20GB heap size on instances with 16 vCPU. The AWS price is based on the US East coast for an on-demand type of instance (Q3 2019), and does not include the EBS volume charges that are substantial only for IOP partitions.

The instances with local NVMe SSD devices substantially outperform any EBS drives due to the lower disk latency and higher bandwidth. In the case of standard and cheapest EBS gp2 volumes, the performance is even slower after the AWS IOPs throttling starts to limit the disk operations. The c5d.4xlarge instances achieve consistently fastest results with the main limitation of small local disks. Next in the list are i3.4xlarge instances offering substantially bigger local disks. Our recommendation is to avoid using the slow EBS volumes, except for cases where you plan to limit the database performance load. Cluster setup with multiple worker nodes with local storage will always outperform significantly any instance with EBS volumes.

Berlin SPARQL Benchmark (BSBM)

BSBM is a popular benchmark combining read queries with frequent updates. It covers a less demanding use case without reasoning, generally defined as eCommerce, describing relations between products and producers, products and offers, offers and vendors, products and reviews.

The benchmark features two runs, where the “explore” run generates requests like “find products for a given set of generic features”, “retrieve basic information about a product for display purpose”, “get recent review”, etc. The “explore and update” run mixes all read queries with information updates.

Table 4: BSBM 100M query mixes per hour on AWS instance - r5.4xlarge, 500 gigabytes gp2 EBS, 1500 IOPSa with GraphDB 9.0 EE and ruleset RDFS-Plus-optimized

Threads explore (query mixes per hour) explore & update (query mixes per hour)
1 9,010 7,447
2 15,866 13,196
4 27,434 22,238
8 42,571 32,720
12 49,754 36,297
16 54,346 36,810