Overview and features¶

The GraphDB Connectors provide extremely fast normal and faceted (aggregation) searches, typically implemented by an external component or a service such as Lucene but have the additional benefit of staying automatically up-to-date with the GraphDB repository data.

The Connectors provide synchronisation at the entity level, where an entity is defined as having a unique identifier (a URI) and a set of properties and property values. In terms of RDF, this corresponds to a set of triples that have the same subject. In addition to simple properties (defined by a single triple), the Connectors support property chains. A property chain is defined as a sequence of triples where each triple’s object is the subject of the following triple.

The main features of the GraphDB Connectors are:

• maintaining an index that is always in sync with the data stored in GraphDB;
• multiple independent instances per repository;
• the entities for synchronisation are defined by:
  • a list of fields (on the Lucene side) and property chains (on the GraphDB side) whose values will be synchronised;
  • a list of rdf:type‘s of the entities for synchronisation;
  • a list of languages for synchronisation (the default is all languages);
  • additional filtering by property and value.
• full-text search using native Lucene queries;
• snippet extraction: highlighting of search terms in the search result;
• faceted search;
• sorting by any preconfigured field;
• paging of results using offset and limit;
• custom mapping of RDF types to Lucene types;
• specifying which Lucene analyzer to use (the default is Lucene’s StandardAnalyzer);
• stripping HTML/XML tags in literals (the default is not to strip markup);
• boosting an entity by the numeric value of one or more predicates;
• custom scoring expressions at query time to evaluate score based on Lucene score and entity boost.

Each feature is described in detail below.

Usage¶

All interactions with the Lucene GraphDB Connector shall be done through SPARQL queries.

There are three types of SPARQL queries:

• INSERT for creating and deleting connector instances;
• SELECT for listing connector instances and querying their configuration parameters;
• INSERT/SELECT for storing and querying data as part of the normal GraphDB data workflow.

In general, this corresponds to INSERT adds or modifies data and SELECT queries existing data.

Each connector implementation defines its own URI prefix to distinguish it from other connectors. For the Lucene GraphDB Connector, this is http://www.ontotext.com/connectors/lucene#. Each command or predicate executed by the connector uses this prefix, e.g., http://www.ontotext.com/connectors/lucene#createConnector to create a connector instance for Lucene.

Individual instances of a connector are distinguished by unique names that are also URIs. They have their own prefix to avoid clashing with any of the command predicates. For Lucene, the instance prefix is http://www.ontotext.com/connectors/lucene/instance#.

Sample data

All examples use the following sample data, which describes five fictitious wines: Yoyowine, Franvino, Noirette, Blanquito and Rozova as well as the grape varieties required to make these wines. The minimum required ruleset level in GraphDB is RDFS.

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix : <http://www.ontotext.com/example/wine#> .

:RedWine rdfs:subClassOf :Wine .
:WhiteWine rdfs:subClassOf :Wine .
:RoseWine rdfs:subClassOf :Wine .

:Merlo
    rdf:type :Grape ;
    rdfs:label "Merlo" .

:CabernetSauvignon
    rdf:type :Grape ;
    rdfs:label "Cabernet Sauvignon" .

:CabernetFranc
    rdf:type :Grape ;
    rdfs:label "Cabernet Franc" .

:PinotNoir
    rdf:type :Grape ;
    rdfs:label "Pinot Noir" .

:Chardonnay
    rdf:type :Grape ;
    rdfs:label "Chardonnay" .

:Yoyowine
    rdf:type :RedWine ;
    :madeFromGrape :CabernetSauvignon ;
    :hasSugar "dry" ;
    :hasYear "2013"^^xsd:integer .

:Franvino
    rdf:type :RedWine ;
    :madeFromGrape :Merlo ;
    :madeFromGrape :CabernetFranc ;
    :hasSugar "dry" ;
    :hasYear "2012"^^xsd:integer .

:Noirette
    rdf:type :RedWine ;
    :madeFromGrape :PinotNoir ;
    :hasSugar "medium" ;
    :hasYear "2012"^^xsd:integer .

:Blanquito
    rdf:type :WhiteWine ;
    :madeFromGrape :Chardonnay ;
    :hasSugar "dry" ;
    :hasYear "2012"^^xsd:integer .

:Rozova
    rdf:type :RoseWine ;
    :madeFromGrape :PinotNoir ;
    :hasSugar "medium" ;
    :hasYear "2013"^^xsd:integer .

Setup and maintenance¶

Third-party component versions

This version of the Lucene GraphDB Connector uses Lucene version 5.5.0.

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

INSERT DATA {
    inst:my_index :createConnector '''
{
  "types": [
    "http://www.ontotext.com/example/wine#Wine"
  ],
  "fields": [
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.ontotext.com/example/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ]
    },
    {
      "fieldName": "sugar",
      "propertyChain": [
        "http://www.ontotext.com/example/wine#hasSugar"
      ],
      "multivalued": false
    },
    {
      "fieldName": "year",
      "propertyChain": [
        "http://www.ontotext.com/example/wine#hasYear"
      ]
    }
  ]
}
''' .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

INSERT DATA {
  inst:my_index :dropConnector "" .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>

SELECT ?cntUri ?cntStr {
  ?cntUri :listConnectors ?cntStr .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>

SELECT ?cntUri ?cntStatus {
  ?cntUri :connectorStatus ?cntStatus .
}

Working with data¶

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?entity {
  ?search a inst:my_index ;
      :query "grape:cabernet" ;
      :entities ?entity .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>
PREFIX wine: <http://www.ontotext.com/example/wine#>

SELECT ?entity ?grape ?year {
  ?search a inst:my_index ;
      :query "grape:cabernet" ;
      :entities ?entity .
  ?entity wine:madeFromGrape ?grape .
  ?entity wine:hasYear ?year
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?entity ?score {
  ?search a inst:my_index ;
      :query "grape:cabernet" ;
      :entities ?entity .
  ?entity :score ?score
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?facetName ?facetValue ?facetCount WHERE {
  # note empty query is allowed and will just match all documents, hence no :query
  ?r a inst:my_index ;
    :facetFields "year,sugar" ;
    :facets _:f .
  _:f :facetName ?facetName .
  _:f :facetValue ?facetValue .
  _:f :facetCount ?facetCount .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?entity {
  ?search a inst:my_index ;
      :query "year:2013" ;
      :orderBy "-sugar" ;
      :entities ?entity .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?entity {
  ?search a inst:my_index ;
      :query "sugar:dry" ;
      :offset "1" ;
      :limit "1" ;
      :entities ?entity .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?entity ?snippetField ?snippetText {
  ?search a inst:my_index ;
      :query "grape:cabernet" ;
      :entities ?entity .
  ?entity :snippets _:s .
  _:s :snippetField ?snippetField ;
     :snippetText ?snippetText .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?totalHits {
    ?r a inst:my_index ;
       :query "year:2012" ;
       :totalHits ?totalHits .
}

List of creation parameters¶

The creation parameters define how a connector instance is created by the :createConnector predicate. Some are required and some are optional. All parameters are provided together in a JSON object, where the parameter names are the object keys. Parameter values may be simple JSON values such as a string or a boolean, or they can be lists or objects.

All of the creation parameters can also be set conveniently from the Create Connector user interface in the GraphDB Workbench without any knowledge of JSON.

analyzer (string), optional, specifies Lucene analyser

The Lucene Connector supports custom Analyser implementations. They may be specified via the analyzer parameter whose value must be a fully qualified name of a class that extends org.apache.lucene.analysis.Analyzer. The class requires either a default constructor or a constructor with exactly one parameter of type org.apache.lucene.util.Version. For example, these two classes are valid implementations:

package com.ontotext.example;

import org.apache.lucene.analysis.Analyzer;

public class FancyAnalyzer extends Analyzer {
    public FancyAnalyzer() {
        ...
    }
    ...
}

package com.ontotext.example;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.util.Version;

public class SmartAnalyzer extends Analyzer {
    public SmartAnalyzer(Version luceneVersion) {
        ...
    }
    ...
}

FancyAnalyzer and SmartAnalyzer can then be used by specifying their fully qualified names, for example:

...
    "analyzer": "com.ontotext.example.SmartAnalyzer",
...

types (list of URI), required, specifies the types of entities to sync

The RDF types of entities to sync are specified as a list of URIs. At least one type URI is required.

languages (list of string), optional, valid languages for literals

RDF data is often multilingual but you can map only some of the languages represented in the literal values. This can be done by specifying a list of language ranges to be matched to the language tags of literals according to RFC 4647, Section 3.3.1. Basic Filtering. In addition, an empty range can be used to include literals that have no language tag. The list of language ranges maps all existing literals that have matching language tags.

fields (list of field object), required, defines the mapping from RDF to Lucene

The fields define exactly what parts of each entity will be synchronised as well as the specific details on the connector side. The field is the smallest synchronisation unit and it maps a property chain from GraphDB to a field in Lucene. The fields are specified as a list of field objects. At least one field object is required. Each field object has further keys that specify details.

• fieldName (string), required, the name of the field in Lucene

  The name of the field defines the mapping on the connector side. It is specified by the key fieldName with a string value. The field name is used at query time to refer to the field. There are few restrictions on the allowed characters in a field name but to avoid unnecessary escaping (which depends on how Lucene parses its queries), we recommend to keep the field names simple.

• propertyChain (list of URI), required, defines the property chain to reach the value

  The property chain (propertyChain) defines the mapping on the GraphDB side. A property chain is defined as a sequence of triples where the entity URI is the subject of the first triple, its object is the subject of the next triple and so on. In this model, a property chain with a single element corresponds to a direct property defined by a single triple. Property chains are specified as a list of URIs where at least one URI must be provided.

  See Copy fields for defining multiple fields with the same property chain.

  See Multiple property chains per field for defining a field whose values are populated from more than one property chain.

• defaultValue (string), optional, specifies a default value for the field

  The default value (defaultValue) provides means for specifying a default value for the field when the property chain has no matching values in GraphDB. The default value can be a plain literal, a literal with a datatype (xsd: prefix supported), a literal with language, or a URI. It has no default value.

• indexed (boolean), optional, default true

  If indexed, a field is available for Lucene queries. true by default.

  This option corresponds to Lucene’s field option "indexed".

• stored (boolean), optional, default true

  Fields can be stored in Lucene and this is controlled by the Boolean option "stored". Stored fields are required for retrieving snippets. true by default.

  This options corresponds to Lucene’s property "stored".

• analyzed (boolean), optional, default true

  When literal fields are indexed in Lucene, they will be analysed according to the analyser settings. Should you require that a given field is not analysed, you may use "analyzed". This option has no effect for URIs (they are never analysed). true by default.

  This option corresponds to Lucene’s property “tokenized”.

• multivalued (boolean), optional, default true

  RDF properties and synchronised fields may have more than one value. If "multivalued" is set to true, all values will be synchronised to Lucene. If set to false, only a single value will be synchronised. true by default.

• facet (boolean), optional, default true

  Lucene needs to index data in a special way, if it will be used for faceted search. This is controlled by the Boolean option “facet”. True by default. Fields that are not synchronised for faceting are also not available for faceted search.

• datatype (string), optional, the manual datatype override

  By default, the Lucene GraphDB Connector uses datatype of literal values to determine how they must be mapped to Lucene types. For more information on the supported datatypes, see Datatype mapping.

  The datatype mapping can be overridden through the parameter "datatype", which can be specified per field. The value of "datatype" can be any of the xsd: types supported by the automatic mapping.

...
  "fields": [
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.ontotext.com/example/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ],
      "analyzed": true,
    },
    {
      "fieldName": "grapeFacet",
      "propertyChain": [
        "@grape"
      ],
      "analyzed": false,
    }
  ]
...

...
  "fields": [
    {
      "fieldName": "name/1",
      "propertyChain": [
        "http://www.ontotext.com/example#canonicalName"
      ],
      "fieldName": "name/2",
      "propertyChain": [
        "http://www.ontotext.com/example#historicalName"
      ]
      ...
    },
...

Datatype mapping¶

The Lucene GraphDB Connector maps different types of RDF values to different types of Lucene values according to the basic type of the RDF value (URI or literal) and the datatype of literals. The autodetection uses the following mapping:

The datatype mapping can be affected by the synchronisation options too, e.g., a non-analysed field that has xsd:long values is indexed with a StringField.

Advanced filtering and fine tuning¶

entityFilter (string)

The entityFilter parameter is used to fine-tune the set of entities and/or individual values for the configured fields, based on the field value. Entities and field values are synchronised to Lucene if, and only if, they pass the filter. The entity filter is similar to a FILTER() inside a SPARQL query but not exactly the same. Each configured field can be referred to, in the entity filter, by prefixing it with a ?, much like referring to a variable in SPARQL. Several operators are supported:

In addition to the operators, there are some constructions that can be used to write filters based not on the values but on values related to them:

Accessing the previous element in the chain

The construction parent(?var) is used for going to a previous level in a property chain. It can be applied recursively as many times as needed, e.g., parent(parent(parent(?var))) goes back in the chain three times. The effective value of parent(?var) can be used with the in or not in operator like this: parent(?company) in (<urn:a>, <urn:b>), or in the bound operator like this: parent(bound(?var)).

Accessing an element beyond the chain

The construction ?var -> uri (alternatively, ?var o uri or just ?var uri) is used for accessing additional values that are accessible through the property uri. In essence, this construction corresponds to the triple pattern value uri ?effectiveValue, where ?value is a value bound by the field var. The effective value of ?var -> uri can be used with the in or not in operator like this: ?company -> rdf:type in (<urn:c>, <urn:d>). It can be combined with parent() like this: parent(?company) -> rdf:type in (<urn:c>, <urn:d>). The same construction can be applied to the bound operator like this: bound(?company -> <urn:hasBranch>), or even combined with parent() like this: bound(parent(?company) -> <urn:hasGroup>).

The URI parameter can be a full URI within < > or the special string rdf:type (alternatively, just type), which will be expanded to http://www.w3.org/1999/02/22-rdf-syntax-ns#type.

Filtering by RDF graph

The construction graph(?var) is used for accessing the RDF graph of a field’s value. The typical use case is to sync only explicit values: graph(?a) not in (<http://www.ontotext.com/implicit>). The construction can be combined with parent() like this: graph(parent(?a)) in (<urn:a>).

Entity filters and default values

Entity filters can be combined with default values in order to get more flexible behaviour.

A typical use-case for an entity filter is having soft deletes, i.e., instead of deleting an entity, it is marked as deleted by the presence of a specific value for a given property.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix : <http://www.ontotext.com/example#> .

# the entity bellow will be synchronised because it has a matching value for city: ?city in ("London")
:alpha
    rdf:type :gadget ;
    :name "John Synced" ;
    :city "London" .

# the entity below will not be synchronised because it lacks the property completely: bound(?city)
:beta
    rdf:type :gadget ;
    :name "Peter Syncfree" .

# the entity below will not be synchronised because it has a different city value:
# ?city in ("London") will remove the value "Liverpool" so bound(?city) will be false
:gamma
    rdf:type :gadget ;
    :name "Mary Syncless" ;
    :city "Liverpool" .

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

INSERT DATA {
  inst:my_index :createConnector '''
    {
      "types": ["http://www.ontotext.com/example#gadget"],
      "fields": [
         {
           "fieldName": "name",
           "propertyChain": ["http://www.ontotext.com/example#name"]
         },
         {
           "fieldName": "city",
           "propertyChain": ["http://www.ontotext.com/example#city"]
         }
      ],
      "entityFilter":"bound(?city) && ?city in (\\"London\\")"
    }
  ''' .
}

...
         {
           "fieldName": "city",
           "propertyChain": ["http://www.ontotext.com/example#city"],
           "defaultValue": "London"
         }
...
}

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix : <http://www.ontotext.com/example2#> .

:Berlin
    rdf:type :Location ;
    rdfs:label "Berlin" .

:Mozart
    rdf:type :Person ;
    rdfs:label "Wolfgang Amadeus Mozart" .

:Einstein
    rdf:type :Person ;
    rdfs:label "Albert Einstein" .

:Cannes-FF
    rdf:type :Event ;
    rdfs:label "Cannes Film Festival" .

:Article1
    rdf:type :Article ;
    rdfs:comment "An article about a film about Einstein's life while he was a professor in Berlin." ;
    :taggedWith :Berlin ;
    :taggedWith :Einstein ;
    :taggedWith :Cannes-FF .

:Article2
    rdf:type :Article ;
    rdfs:comment "An article about Berlin." ;
    :taggedWith :Berlin .

:Article3
    rdf:type :Article ;
    rdfs:comment "An article about Mozart's life." ;
    :taggedWith :Mozart .

:Article4
    rdf:type :Article ;
    rdfs:comment "An article about classical music in Berlin." ;
    :taggedWith :Berlin ;
    :taggedWith :Mozart .

:Article5
    rdf:type :Article ;
    rdfs:comment "A boring article that has no tags." .

:Article6
    rdf:type :Article ;
    rdfs:comment "An article about the Cannes Film Festival in 2013." ;
    :taggedWith :Cannes-FF .

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

INSERT DATA {
  inst:my_index :createConnector '''
    {
      "types": ["http://www.ontotext.com/example2#Article"],
      "fields": [
         {
            "fieldName": "comment",
            "propertyChain": ["http://www.w3.org/2000/01/rdf-schema#comment"]
         },
         {
           "fieldName": "taggedWithPerson",
           "propertyChain": ["http://www.ontotext.com/example2#taggedWith"]
         },
         {
           "fieldName": "taggedWithLocation",
           "propertyChain": ["http://www.ontotext.com/example2#taggedWith"]
         }
      ],
      "entityFilter": "?taggedWithPerson type in (<http://www.ontotext.com/example2#Person>)
                        && ?taggedWithLocation type in (<http://www.ontotext.com/example2#Location>)"
    }
  ''' .
}

PREFIX : <http://www.ontotext.com/connectors/lucene#>
PREFIX inst: <http://www.ontotext.com/connectors/lucene/instance#>

SELECT ?facetName ?facetValue ?facetCount {
  ?search a inst:my_index ;
      :facetFields "taggedWithLocation,taggedWithPerson" ;
      :facets _:f .
  _:f :facetName ?facetName ;
      :facetValue ?facetValue ;
      :facetCount ?facetCount .
}

Overview of connector predicates¶

The following diagram shows a summary of all predicates that can administer (create, drop, check status) connector instances or issue queries and retrieve results. It can be used as a quick reference of what a particular predicate needs to be attached to. For example, to retrieve entities, you need to use :entities on a search instance and to retrieve snippets, you need to use :snippets on an entity. Variables that are bound as a result of a query are shown in green, blank helper nodes are shown in blue, literals in red, and URIs in orange. The predicates are represented by labelled arrows.

Caveats¶

Upgrading from previous versions¶