MongoDB integration¶

What’s in this document?

Usage
Setup and maintenance

The MongoDB integration feature is a GraphDB plugin that lets you use SPARQL to query MongoDB databases with heterogeneous joins of that data to RDF data. This section describes how to configure GraphDB and MongoDB to work together.

MongoDB is a document-based database with a huge community of developers and users. It is part of the MEAN technology stack and guarantees scalability and performance well beyond the throughput supported in GraphDB.

MongoDB is a NoSQL JSON document store and does not natively support joins, SPARQL, or RDF-enabled linked data. The integration between GraphDB and MongoDB is done by a plugin that sends a request to MongoDB and then transforms the result to an RDF model.

Usage ¶

The basic steps for using MongoDB with GraphDB are:

Installing MongoDB
Preparing and loading JSON-LD documents in MongoDB
Configuring GraphDB with MongoDB connection settings by creating an index

In order to be converted to RDF models, the documents in MongoDB should be in the JSON-LD format.

JSON-LD documents have a hierarchical structure, which allows more complex querying of embedded and nested documents.

Each document can be in a separate context. That way, the relation between statements in GraphDB and documents in MongoDB is preserved when extracting parts of the documents and importing them into GraphDB in order to make inferred statements. The import of parts is an option for future development.

Below we see a sample document in MongoDB from the LDBC SPB benchmark:

{
        "_id": { "$oid": "5c0fb7f329298f15dc37bb81"},
        "@graph":
        [{
                "@id": "http://www.bbc.co.uk/things/1#id",
                "@type": "cwork:NewsItem",
                "bbc:primaryContentOf":
                [{
                        "@id": "bbcd:3#id",
                        "bbc:webDocumentType": {
                                "@id": "bbc:HighWeb"
                        }
                },
                {
                        "@id": "bbcd:4#id",
                        "bbc:webDocumentType": {
                                "@id": "bbc:Mobile"
                        }
                }],
                "cwork:about":
                [{
                        "@id": "dbpedia:AccessAir"
                },
                {
                        "@id": "dbpedia:Battle_of_Bristoe_Station"
                },
                {
                        "@id": "dbpedia:Nicolas_Bricaire_de_la_Dixmerie"
                },
                {
                        "@id": "dbpedia:Bernard_Roberts"
                },
                {
                        "@id": "dbpedia:Bartolomé_de_Medina"
                },
                {
                        "@id": "dbpedia:Don_Bonker"
                },
                {
                        "@id": "dbpedia:Cornel_Nistorescu"
                },
                {
                        "@id": "dbpedia:Clete_Roberts"
                },
                {
                        "@id": "dbpedia:Mark_Palansky"
                },
                {
                        "@id": "dbpedia:Paul_Green_(taekwondo)"
                },
                {
                        "@id": "dbpedia:Mostafa_Abdel_Satar"
                },
                {
                        "@id": "dbpedia:Tommy_O'Connell_(hurler)"
                },
                {
                        "@id": "dbpedia:Ahmed_Ali_Salaad"
                }],
                "cwork:altText": "thumbnail atlText for CW http://www.bbc.co.uk/context/1#id",
                "cwork:audience": {
                        "@id": "cwork:NationalAudience"
                },
                "cwork:category": {
                        "@id": "http://www.bbc.co.uk/category/Company"
                },
                "cwork:dateCreated": {
                        "@type": "xsd:dateTime",
                        "@value": "2011-02-15T07:13:29.495+02:00"
                },
                "cwork:dateModified": {
                        "@type": "xsd:dateTime",
                        "@value": "2012-02-14T12:43:13.165+02:00"
                },
                "cwork:description": " constipate meant breaking felt glitzier democrat's huskily breeding solicit gargling.",
                "cwork:liveCoverage": {
                        "@type": "xsd:boolean",
                        "@value": "false"
                },
                "cwork:mentions": {
                        "@id": "geonames:2862704/"
                },
                "cwork:primaryFormat":
                [{
                        "@id": "cwork:TextualFormat"
                },
                {
                        "@id": "cwork:InteractiveFormat"
                }],
                "cwork:shortTitle": " closest subsystem merit rebuking disengagement cerebrums caravans conduction disbelieved might.",
                "cwork:thumbnail": {
                        "@id": "bbct:1361611547"
                },
                "cwork:title": "Beckhoff greatly agitators constructed racquets industry restrain spews pitifully undertone stultification."
        }],
        "@id": "bbcc:1#id",
        "@context": {
                "bbcevent": "http://www.bbc.co.uk/ontologies/event/",
                "geo-pos": "http://www.w3.org/2003/01/geo/wgs84_pos#",
                "bbc": "http://www.bbc.co.uk/ontologies/bbc/",
                "time": "http://www.w3.org/2006/time#",
                "event": "http://purl.org/NET/c4dm/event.owl#",
                "music-ont": "http://purl.org/ontology/mo/",
                "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
                "foaf": "http://xmlns.com/foaf/0.1/",
                "provenance": "http://www.bbc.co.uk/ontologies/provenance/",
                "owl": "http://www.w3.org/2002/07/owl#",
                "cms": "http://www.bbc.co.uk/ontologies/cms/",
                "news": "http://www.bbc.co.uk/ontologies/news/",
                "cnews": "http://www.bbc.co.uk/ontologies/news/cnews/",
                "cconcepts": "http://www.bbc.co.uk/ontologies/coreconcepts/",
                "dbp-prop": "http://dbpedia.org/property/",
                "geonames": "http://sws.geonames.org/",
                "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
                "domain": "http://www.bbc.co.uk/ontologies/domain/",
                "dbpedia": "http://dbpedia.org/resource/",
                "geo-ont": "http://www.geonames.org/ontology#",
                "bbc-pont": "http://purl.org/ontology/po/",
                "tagging": "http://www.bbc.co.uk/ontologies/tagging/",
                "sport": "http://www.bbc.co.uk/ontologies/sport/",
                "skosCore": "http://www.w3.org/2004/02/skos/core#",
                "dbp-ont": "http://dbpedia.org/ontology/",
                "xsd": "http://www.w3.org/2001/XMLSchema#",
                "core": "http://www.bbc.co.uk/ontologies/coreconcepts/",
                "curric": "http://www.bbc.co.uk/ontologies/curriculum/",
                "skos": "http://www.w3.org/2004/02/skos/core#",
                "cwork": "http://www.bbc.co.uk/ontologies/creativework/",
                "fb": "http://rdf.freebase.com/ns/",
                "ot": "http://www.ontotext.com/",
                "ldbcspb": "http://www.ldbcouncil.org/spb#",
                "bbcd": "http://www.bbc.co.uk/document/",
                "bbcc": "http://www.bbc.co.uk/context/",
                "bbct": "http://www.bbc.co.uk/thumbnail/"
        }
}

Note the following about the sample MongoDB document above:

_id key is a MongoDB internal key.
@graph node represents the RDF context in the JSON-LD document.
@type xsd:dateTime date has a @date key with an ISODate(...) value. This is not related to the JSON-LD standard and is ignored when the document is parsed as an RDF model. The dates are extended for faster search and sorting. The ISODate in MongoDB is its internal way to store dates and is optimized for searching. This step will make querying and sorting by this date field easier but is optional.

Note

The keys in MongoDB cannot contain "." nor start with "$". Although the JSON-LD standard allows it, MongoDB does not. Therefore, either use namespaces (see the sample above) or encoding the . and $, respectively. Only the JSON keys are subject to decoding.

Setup and maintenance ¶

Installing MongoDB ¶

The instructions below assume that you have the MongoDB server installed and running on a computer you can access. Setting up and maintaining a MongoDB database is a separate task and must be accomplished outside of GraphDB. See the MongoDB documentation for details.

The GraphDB integration plugin uses MongoDB Java driver version 3.8. More information about the compatibility between MongoDB Java driver and MongoDB version is available in the MongoDB compatibility tables.

Creating an index ¶

To configure GraphDB with MongoDB connection settings, you need to set:

The server where MongoDB is running

The port on which MongoDB is listening

The name of the database you are using

The name of the MongoDB collection you are using

The credentials (optional unless you are using authentication) — the username and password that will allow you to connect to the database

This sample update request demonstrates how to create a MongoDB index:

PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
INSERT DATA {
    mongodb-index:spb1000 mongodb:service "mongodb://localhost:27017" ;
        mongodb:database "ldbc" ;
        mongodb:collection "creativeWorks" .
}

Supported predicates:

:service — MongoDB connection string
:database — MongoDB database
:collection — MongoDB collection
:user — (optional) MongoDB user for the connection
:password — (optional) the user’s password
:authDb — (optional) the database where the user is authenticated

Upgrading an index ¶

When upgrading to a newer GraphDB version, it may contain plugins that are not present in the older version. If so, the PluginManager disables the newly detected plugin, so you need to enable it by executing the following SPARQL query:

insert data { [] <http://www.ontotext.com/owlim/system#startplugin> "mongodb" }

Then create the plugin by executing the SPARQL update request above under Creating an index. Also, make sure that you don’t delete the database in the plugin you are using.

Deleting an index ¶

Delete a MongoDB index using the mongodb:drop predicate in an update request like the following:

PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
INSERT DATA {
    mongodb-index:spb1000 mongodb:drop [] .
}

Loading sample data ¶

MongoDB’s mongoimport tool lets you import data into collections from the command line. The following command will import the cwork1000.json file, which has 1,000 CreativeWork documents, in MongoDB database “ldbc” and the collection “creativeWorks”.

mongoimport --db ldbc --collection creativeWorks --file cwork1000.json

Querying MongoDB ¶

This sample query returns the dateModified for documents with the specified audience:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>

SELECT ?creativeWork ?modified WHERE {
        ?search a mongodb-index:spb1000 ;
                mongodb:find '{"@graph.cwork:audience.@id" : "cwork:NationalAudience"}' ;
                mongodb:entity ?entity .
        GRAPH mongodb-index:spb1000 {
                ?creativeWork cwork:dateModified ?modified .
        }
}

In a query, use the exact values from the documents. For example, if the full URIs are used instead of "cwork:NationalAudience" or "@graph.cwork:audience.@id", there would not be any matching results.

The :find argument is a valid JSON document.

Note

The results are returned in a named graph to indicate when the plugin should bind the variables. This is an API plugin limitation. The variables to be bound by the plugin are in a named graph. This allows GraphDB to determine whether to bind the specific variable using MongoDB or not.

Supported predicates:

mongodb:find: Accepts a single JSON object provided as an RDF literal and sets a query string. The value is used to call db.collection.find().
mongodb:project: Accepts a single JSON object provided as an RDF literal. The value is used to select the projection for the results returned by mongodb:find. Find more information at MongoDB: Project Fields to Return from Query.
mongodb:aggregate: Accepts an array of JSON objects. Calls db.collection.aggregate(). This is the most flexible way to make a MongoDB query, because the find() method is just a single phase of the aggregation pipeline. The mongodb:aggregate predicate takes precedence over mongodb:find and mongodb:project. This means that if both mongodb:aggregate and mongodb:find are used, mongodb:find will be ignored.
mongodb:graph: Accepts an IRI. Specifies the IRI of the named graph in which the bound variables should be. Its default value is the name of the index itself.
mongodb:entity : Returns the IRI of the MongoDB document. If the JSON-LD has context, the value of @graph.@id is used. In case of multiple values, the first one is chosen and a warning is logged. If the JSON-LD has no context, the value of the @id node is used. Even if the value from this predicate is not used, the query must include it in order to inform the plugin that the graph part of the current iteration is completed.
mongodb:hint: Specifies the index to be used when executing the query by calling the MongoDB method cursor.hint().
mongodb:collation Accepts JSON. Specifies language-specific rules for string comparison such as rules for lettercase and accent marks. It is applied to a mongodb:find or an mongodb:aggregate query.

Batch processing ¶

While streaming is the default and preferred process for handling of the result documents, it can lead to unexpected behavior in certain edge cases, such as queries returning no or partial results despite matching data in MongoDB. This can happen when the mongodb:find {...} matches multiple documents, but each of the them contains only partial data. Without batch processing, when the second part of the query is evaluated, it filters out the documents that does not match the provided statements, thus resulting in unexpected behavior.

Using batch processing prevents this issue. When enabled, batch processing will collect the documents matching the input query up to the defined maximum and load them in-memory, before returning them as named graph collection.

Batch processing can be enabled via the additional MongoDB plugin predicate mongodb:batchSize, which controls the maximum amount of documents that will be loaded:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>

SELECT ?creativeWork ?modified WHERE {
        ?search a mongodb-index:spb1000 ;
                mongodb:find '{"@graph.cwork:audience.@id" : "cwork:NationalAudience"}' ;
                mongodb:batchSize 1000 ;
                mongodb:entity ?entity .
        GRAPH mongodb-index:spb1000 {
                ?creativeWork cwork:dateModified ?modified .
        }
}

In addition to the query predicate, there is also a system property called graphdb.mongodb.maxBatchSize, which controls the maximum value that can be set to the query predicate. The default value for the configuration is 1000, with a maximum possible value of 10000. These limits aim to minimize the risk of memory explosion and OOM errors in GraphDB.

Multiple index calls in the same query ¶

The same query can support multiple MongoDB calls. There are two approaches:

Each index call to be in a separate SUBSELECT (Example 1);
Each index call to use different named graph. If querying different indexes, this comes out of the box. If not, use the :graph predicate. (Example 2).

Example 1:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
SELECT ?creativeWork ?modified WHERE {
    {
        SELECT ?creativeWork ?modified {
            ?search a mongodb-index:spb1000 ;
                mongodb:find '{"@graph.@id" : "http://www.bbc.co.uk/things/1#id"}' ;
                mongodb:entity ?creativeWork .
            GRAPH mongodb-index:spb1000 {
                ?creativeWork cwork:dateModified ?modified ;
            }
        }
    }
    UNION
    {
        SELECT ?creativeWork ?modified WHERE {
            ?search a mongodb-index:spb1000 ;
                mongodb:find '{"@graph.@id" : "http://www.bbc.co.uk/things/2#id"}' ;
                mongodb:entity ?entity .
            GRAPH mongodb-index:spb1000 {
                ?creativeWork cwork:dateModified ?modified ;
            }
        }
    }
}

Example 2:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
SELECT ?creativeWork ?modified WHERE {
    {
        ?search a mongodb-index:spb1000 ;
                mongodb:graph mongodb:search1 ;
                mongodb:find '{"@graph.@id" : "http://www.bbc.co.uk/things/1#id"}' ;
                mongodb:entity ?creativeWork .
        GRAPH mongodb:search1 {
                ?creativeWork cwork:dateModified ?modified ;
        }
    }
    UNION
    {
        ?search a mongodb-index:spb1000 ;
                mongodb:graph mongodb:search2 ;
                mongodb:find '{"@graph.@id" : "http://www.bbc.co.uk/things/2#id"}' ;
                mongodb:entity ?entity .
        GRAPH mongodb:search2 {
                ?creativeWork cwork:dateModified ?modified ;
        }
    }
}

Both examples return the same result.

_images/mongodb-multiple-queries-ex1.png

Using aggregation functions ¶

MongoDB supports a number of aggregation functions such as min, max, and size. These functions are called using the :aggregate predicate. The data of the retrieved results must be converted to an RDF model. The example below shows how to retrieve the RDF context of a MongoDB document.

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

SELECT ?s ?o {
    ?search a mongodb-index:spb1000 ;
        mongodb:aggregate '''[{"$match": {"@graph.@id": "http://www.bbc.co.uk/things/1#id"}},
                {'$addFields': {'@graph.cwork:graph.@id' :  '$@id'}}]''' ;
        mongodb:entity ?entity .
    GRAPH mongodb-index:spb1000 {
        ?s cwork:graph ?o .
    }
}

The $addFields phrase adds a new nested document in the JSON-LD stored in MongoDB. The newly added document is then parsed to the following RDF statement:

<http://www.bbc.co.uk/things/1#id> cwork:graph <http://www.bbc.co.uk/context/1#id>

We retrieve the context of the document using the cwork:graph predicate.

This approach is very flexible but is prone to error. Let’s look at an example:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

SELECT ?g1 ?g2 {
    ?search a mongodb-index:spb1000 ;
        mongodb:aggregate '''[{"$match": {"@graph.@id": "http://www.bbc.co.uk/things/1#id"}},
                {'$addFields': {'@graph.inst:graph.@id' :  '$@id'}}]''' ;
        mongodb:entity ?entity .
    GRAPH mongodb-index:spb1000 {
        OPTIONAL {
            ?s mongodb-index:graph ?g1 .
        }
        ?s mongodb-index:graph ?g2 .
    }
}

It looks very similar to the first one except that instead of @graph.cwork:graph.@id we are writing the value to @graph.inst:graph.@id, so ?g1 will not get bound. This happens because in the JSON-LD stored in MongoDB we are aware of the cwork context but not of the inst context, so ?g2 will get bound instead.

Custom fields ¶

The following demonstrates the use of custom fields:

PREFIX cwork: <http://www.bbc.co.uk/ontologies/creativework/>
PREFIX mongodb-index: <http://www.ontotext.com/connectors/mongodb/instance#>
PREFIX mongodb: <http://www.ontotext.com/connectors/mongodb#>

SELECT ?size ?halfSize {
    ?search a mongodb-index:spb1000 ;
        mongodb:aggregate '''[{"$match": {"@graph.@type": "cwork:NewsItem"}},
                {"$count": "size"},
                {"$project": {"custom.size": "$size", "custom.halfSize": {"$divide": ["$size", 2]}}}]''' ;
        mongodb:entity ?entity .
    GRAPH mongodb-index:spb1000 {
        ?s mongodb-index:size ?size ;
        mongodb-index:halfSize ?halfSize .
    }
}

_images/mongodb-custom-field-example.png

The values are projected as child elements of a custom node. After JSON-LD is taken from MongoDB, a pre-processing step follows in order to retrieve all child elements of custom and create statements with predicates in the <http://www.ontotext.com/connectors/mongodb/instance#> namespace.

Note

The returned values are always string literals.

Authentication ¶

You can take advantage of all types of authentication by setting the credentials in the connection string. However, because it is not a good practice to store the passwords in plain text, the :user, :password, and :authDb predicates are available. If you use one of those predicates, you must set the other two as well. These predicates set credentials for SCRAM and LDAP authentication and the password is stored encrypted with a symmetrical algorithm on the disk. For x.509 and Kerberos authentication the connection string should be used, because no passwords are being stored.