Elasticsearch GraphDB Connector¶

What’s in this document?

Usage
Setup and maintenance
Working with data
List of creation parameters
- Updating parameters at runtime
- Special field definitions
Datatype mapping
- Date and time conversion
Entity filtering
Overview of connector predicates
Caveats
- Order of control
Upgrading from previous versions
- Migrating from GraphDB 9.x

Note

This feature requires a GraphDB Enterprise license.

Note

GraphDB supports full-text search options as well.

The GraphDB Connectors provide extremely fast normal and faceted (aggregation) searches, typically implemented by an external component or a service such as Elasticsearch but with the additional benefit of staying automatically up to date with the GraphDB repository data.

The Connectors provide synchronization at the entity level, where an entity is defined as having a unique identifier (an IRI) and a set of properties and property values. In terms of RDF, this corresponds to a set of triples that have the same subject. In addition to simple properties (defined by a single triple), the Connectors support property chains. A property chain is a sequence of triples where each triple’s object is the subject of the following triple.

Elasticsearch also allows for vector search, which uses semantic similarity to ensure better recall than full-text search. The embeddings used for this vector search can either be created by Elasticsearch, or by a client for an external embedding model configured in the connector. One of the options for embedding models is the default local model deployed with the Graphwise Transformer tool. You can learn more about configuring such a model under Embedding model configuration for vector search.

The main features of the GraphDB Connectors are:

maintenance of an index that is always in sync with the data stored in GraphDB
multiple independent instances per repository
entity synchronization defined by:
- a list of fields (on the Elasticsearch side) and property chains (on the GraphDB side) whose values will be synchronized
- a list of rdf:type values of the entities for synchronization
- a list of languages for synchronization (the default is all languages)
- additional filtering by property and value
full-text search using native Elasticsearch queries
snippet extraction: highlighting of search terms in the search result
faceted search
sorting by any preconfigured field
paging of results using OFFSET and LIMIT
custom mapping of RDF types to Elasticsearch types

Each feature is described in detail below.

Usage ¶

All interactions with the Elasticsearch GraphDB Connector are done through SPARQL queries.

There are three types of SPARQL queries:

INSERT for creating, updating, and deleting connector instances
SELECT for listing connector instances and querying their configuration parameters
INSERT and SELECT for storing and querying data as part of the normal GraphDB data workflow

In general, this corresponds to INSERT update requests that add or modify data and to SELECT queries of existing data.

Each connector implementation uses its own namespace IRI to distinguish it from other connectors. For the Elasticsearch GraphDB Connector, this is http://www.ontotext.com/connectors/elasticsearch#. Each command or predicate executed by the connector uses terms from this namespace (for example, http://www.ontotext.com/connectors/elasticsearch#createConnector) to create a connector instance for Elasticsearch.

Individual instances of a connector are distinguished by unique names that are also IRIs. They have their own namespace to avoid clashing with any of the command predicates. For Elasticsearch, the instance IRI is http://www.ontotext.com/connectors/elasticsearch/instance#.

Warning

Deleting the repository does not remove the indexes in Elasticsearch. In order to create a connector instance with the same name as one of the instances in the deleted repository, you may need to delete the corresponding Elasticsearch index first.

Warning

Changing the Elasticsearch URL will not reindex the data automatically. If you need the data to be reindexed, you can repair the connector instance.

Sample data

All examples use the following sample data that describes five fictitious wines: Yoyowine, Franvino, Noirette, Blanquito, and Rozova, as well as the grape varieties required to make these wines. The minimum required ruleset level in GraphDB is RDFS.

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix wine: <http://www.example.com/demo/wine#> .

wine:RedWine rdfs:subClassOf wine:Wine .
wine:WhiteWine rdfs:subClassOf wine:Wine .
wine:RoseWine rdfs:subClassOf wine:Wine .

wine:Merlot
    rdf:type wine:Grape ;
    rdfs:label "Merlot" .

wine:CabernetSauvignon
    rdf:type wine:Grape ;
    rdfs:label "Cabernet Sauvignon" .

wine:CabernetFranc
    rdf:type wine:Grape ;
    rdfs:label "Cabernet Franc" .

wine:PinotNoir
    rdf:type wine:Grape ;
    rdfs:label "Pinot Noir" .

wine:Chardonnay
    rdf:type wine:Grape ;
    rdfs:label "Chardonnay" .

wine:Yoyowine
    rdf:type wine:RedWine ;
    wine:madeFromGrape wine:CabernetSauvignon ;
    wine:hasSugar "dry" ;
    wine:hasYear "2013"^^xsd:integer .

wine:Franvino
    rdf:type wine:RedWine ;
    wine:madeFromGrape wine:Merlot ;
    wine:madeFromGrape wine:CabernetFranc ;
    wine:hasSugar "dry" ;
    wine:hasYear "2012"^^xsd:integer .

wine:Noirette
    rdf:type wine:RedWine ;
    wine:madeFromGrape wine:PinotNoir ;
    wine:hasSugar "medium" ;
    wine:hasYear "2012"^^xsd:integer .

wine:Blanquito
    rdf:type wine:WhiteWine ;
    wine:madeFromGrape wine:Chardonnay ;
    wine:hasSugar "dry" ;
    wine:hasYear "2012"^^xsd:integer .

wine:Rozova
    rdf:type wine:RoseWine ;
    wine:madeFromGrape wine:PinotNoir ;
    wine:hasSugar "medium" ;
    wine:hasYear "2013"^^xsd:integer .

Setup and maintenance ¶

Prerequisites ¶

Third-party component versions ¶

The Elasticsearch Connector in GraphDB 11.5 supports Elasticsearch version 9.1, including any of the patch releases for that version.

Prerequisites for vector search ¶

Elasticsearch Enterprise is required if you want to Elasticsearch to create the embeddings when using vector search — check Elasticsearch’s available subscription options for more details.

When using another embedding model to generate the embeddings, you need to configure the embedding model before creating the connector.

Creating a connector instance ¶

Create a connector instance by sending a SPARQL query with the following configuration data:

the name of the connector instance, such as my_index
an Elasticsearch instance to synchronize to
classes to synchronize
properties to synchronize

The configuration data must be provided as a JSON string representation and passed together with the create command.

You can create connectors via Workbench or by using a SPARQL update query (create command).

If you create the connector using Workbench, no matter which way you use, you will be presented with a pop-up screen showing you the connector creation progress.

Using Workbench ¶

Go to Setup ‣ Connectors.
Click New Connector in the tab of the respective Connector type you want to create.
Fill out the configuration form.
If you are using the Wine example dataset, you can provide the following configuration parameters:
- Name: my_index
- Fields:
  Field name: grape
  
  Property chain:
  
  http://www.example.com/demo/wine#madeFromGrape
  
  http://www.w3.org/2000/01/rdf-schema#label
  
  Nested object fields: Select Indexed, Stored, Analyzed and Multivalued
- Types: http://www.example.com/demo/wine#Wine
- ES node: http://localhost:9200 (or the address of your Elasticsearch instance)
- Select ES cluster sniff, ES manage index and ES manage mappings
- ES bulk update batch size: 5000
- ES index create settings: 5242880
If you want to use vector search with your connector, you need to configure the fields for similarity search.
1. If you want the embeddings to be created by GraphDB, provide the embeddings model you want to use.
  
  Tip
  
  If you want Elasticsearch to create the embeddings, you do not need to change the embeddings model parameter.
2. Specify the datatype of the field to use for similarity search. The datatype depends on whether the embeddings are created by Elasticsearch or by an embedding model via the connector, as described in the documentation for the datatype field below.
Execute the CREATE statement from the form by clicking OK. Alternatively, you can view its SPARQL query by clicking View SPARQL Query, and then copy it to execute it manually or integrate it in automation scripts.

See also

List of creation parameters provides thorough explanation of all creation parameters.

Using the create command ¶

The create command is triggered by a SPARQL INSERT with the elastic:createConnector predicate. In the example, it creates a connector instance called my_index, which synchronizes the wines from the sample data above.

To be able to use newlines and quotes without the need for escaping, the update request uses SPARQL’s multi-line string delimiter consisting of 3 apostrophes: '''...'''. You can also use 3 quotes instead: """...""".

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
    elastic-index:my_index elastic:createConnector '''
{
  "elasticsearchNode": "localhost:9200",
  "types": [
    "http://www.example.com/demo/wine#Wine"
  ],
  "fields": [
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.example.com/demo/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ]
    },
    {
      "fieldName": "sugar",
      "propertyChain": [
        "http://www.example.com/demo/wine#hasSugar"
      ],
      "analyzed": false
    },
    {
      "fieldName": "year",
      "propertyChain": [
        "http://www.example.com/demo/wine#hasYear"
      ],
      "analyzed": false
    }
  ]
}
''' .
}

The above command creates a new Elasticsearch connector instance that connects to the Elasticsearch instance accessible at port 9200 on the localhost as specified by the elasticsearchNode key.

The "types" key defines the RDF type of the entities to synchronize. In the example, it is only entities of the type http://www.example.com/demo/wine#Wine and its subtypes if RDFS or higher-level reasoning is enabled. The "fields" key defines the mapping from RDF to Elasticsearch. The basic building block is the property chain: a sequence of RDF properties where the object of each property is the subject of the following property. In the example, three bits of information are mapped — the grape the wines are made of, sugar content, and year. Each chain is assigned a short and convenient field name: “grape”, “sugar”, and “year”. The field names are used later in the queries.

The field grape is an example of a property chain composed of more than one property. First, we take the wine’s madeFromGrape property, the object of which is an instance of the type Grape, and then we take the rdfs:label of this instance. The fields sugar and year are both composed of a single property that links the value directly to the wine.

The fields sugar and year contain discrete values such as medium, dry, 2012, and 2013, so it is best to specify the option analyzed: false as well. See analyzed in Defining fields for more information.

Mapping and index management ¶

By default, GraphDB manages (creates, deletes, or updates if needed) the Elasticsearch index and the Elasticsearch mapping. This makes it easier to use Elasticsearch as everything is done automatically. This behavior can be changed by the following options:

manageIndex: if true, GraphDB manages the index. true by default.
manageMapping: if true, GraphDB manages the mapping. true by default.

Note

If either of the options is set to false, you have to create, update or remove the index/mapping and, in case Elasticsearch is misconfigured, the connector instance will not function correctly.

Using a non-managed schema ¶

The present version provides no support for changing some advanced options, such as stop words, on a per-field basis. The recommended way to do this for now is to manage the mapping yourself and tell the connector to just sync the object values in the appropriate fields. Here is an example:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
    elastic-index:my_index elastic:createConnector '''
{
  "elasticsearchNode": "localhost:9200",
  "types": [
    "http://www.example.com/demo/wine#Wine"
  ],
  "fields": [
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.example.com/demo/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ]
    },
    {
      "fieldName": "sugar",
      "propertyChain": [
        "http://www.example.com/demo/wine#hasSugar"
      ],
      "analyzed": false
    },
    {
      "fieldName": "year",
      "propertyChain": [
        "http://www.example.com/demo/wine#hasYear"
      ],
      "analyzed": false
    }
  ],
  "manageMapping": false
}
''' .
}

This creates the same connector instance as above but it expects fields with the specified field names to be already present in the index mapping, as well as some internal GraphDB fields. For the example, you must have the following fields:

field name	Elasticsearch config
`_graphdb_id`	“type”:”long”, “index”:true, “store”:tru
`grape`	“type”:”text”, “index”:true, “store”:true
`sugar`	“type”:”keyword”, “index”:true, “store”:true
`year`	“type”:”keyword”, “index”:true, “store”:true

_graphdb_id is used internally by GraphDB and is always required.

Working with secured Elasticsearch ¶

GraphDB allows the access of a secured Elasticsearch instance by passing the arbitrary elasticsearchBasicAuthUser and elasticsearchBasicAuthPassword parameters.

Instead of supplying the username and password as part of the connector instance configuration, you can also implement a custom authenticator class and set it via the authenticationConfiguratorClass option. See these connector authenticator examples for more information and example projects that implement such a custom class.

See the List of creation parameters for more information.

Dropping a connector instance ¶

Dropping a connector instance removes all references to its external store from GraphDB as well as the Elasticsearch index associated with it.

The drop command is triggered by a SPARQL INSERT with the elastic:dropConnector predicate where the name of the connector instance must be in the subject position. For example, the following removes the connector my_index:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:dropConnector [] .
}

You can also force drop a connector if a normal delete operation does not work. The force delete will remove the connector even if part of the operation fails. Go to Setup ‣ Connectors where you will see the already existing connectors that you have created. Click the delete icon , check Force delete in the dialog box, and click that dialog box’s Delete button.

Retrieving the create options for a connector instance ¶

You can view the options string that was used to create a particular connector instance with the following query:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?createString {
  elastic-index:my_index elastic:listOptionValues ?createString .
}

Listing available connector instances ¶

In the Connectors management view ¶

Existing instances of a given Connector type are shown above the New connector button for that connector type. Click the name of an instance to view its configuration and SPARQL query or click the Repair or Delete icons to perform these operations. Click the Copy icon to copy the connector definition query to your clipboard.

_images/view-existing-connectors-elastic.png

With a SPARQL query ¶

Listing connector instances returns all previously created instances. It is a SELECT query with the listConnectors predicate:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>

SELECT ?cntUri ?cntStr {
  ?cntUri elastic:listConnectors ?cntStr .
}

In this example, ?cntUri is bound to the IRI of the connector instance that was used during creation such as http://www.ontotext.com/connectors/elasticsearch/instance#my_index, while ?cntStr is bound to a string representing the local name such as “my_index” in this example.

Instance status check ¶

You can query the internal state of each connector instance using a SELECT query with the connectorStatus predicate:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>

SELECT ?cntUri ?cntStatus {
  ?cntUri elastic:connectorStatus ?cntStatus .
}

In this example, ?cntUri is bound to the IRI of the connector instance, while ?cntStatus is bound to a string representation of the status of the connector represented by this IRI. The status is key-value based.

Working with data ¶

Adding, updating and deleting data ¶

From the user point of view, all synchronization happens transparently without using any additional predicates or naming a specific store explicitly. You simply execute standard SPARQL INSERT and DELETE update requests. This is achieved by intercepting all changes in the plugin and determining which Elasticsearch documents need to be updated.

Simple queries ¶

Once a connector instance has been created, you can use SPARQL to query data from it. For each matching Elasticsearch document, the connector instance returns the document subject. In its simplest form, you query that data using a SELECT query that includes an Elasticsearch query as the object of the elastic:query predicate:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity {
  ?search a elastic-index:my_index ;
      elastic:query "grape:cabernet" ;
      elastic:entities ?entity .
}

The result of this query binds ?entity to the two wines made from grapes that have “cabernet” in their name: :Yoyowine and :Franvino.

Note

You must use the field names you chose when you created the connector instance. They can be identical to the property IRIs but you must escape any special characters according to what Elasticsearch expects.

The creation of such a query follows these basic steps:

Get a query instance of the requested connector instance by using the RDF notation "X a Y" (= X rdf:type Y), where X is a variable and Y is a connector instance IRI. X is bound to a query instance of the connector instance.
Assign a query to the query instance by using the system predicate elastic:query.
Request the matching entities through the elastic:entities predicate.

You can also provide per-query search options by using one or more option predicates. The option predicates are described in detail below.

Raw queries ¶

To access an Elasticsearch query parameter that is not exposed through a special predicate, use a raw query. Instead of providing a full-text query in the :query part, specify raw Elasticsearch parameters. For example, to boost some parts of your full-text query as described in the Elastic documentation, execute the following query:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity {
  ?search a elastic-index:my_index ;
      elastic:query '''
      {
        "query" : {
          "bool" : {
            "should" : [ {
              "query_string" : {
                "query" : "<full-text-query-not-boosted>"
              }
            }, {
              "query_string" : {
                "query" : "<full-text-query-boosted>",
                "boost" : 4.0
              }
            } ]
          }
        }
      }
      ''' ;
      elastic:entities ?entity .
}

Combining Elasticsearch results with GraphDB data ¶

You can use the bound ?entity variable in other SPARQL triple patterns to build complex queries that join to or fetch additional data from GraphDB. For example, to see the actual grapes in the matching wines as well as the year they were made:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>
PREFIX wine: <http://www.example.com/demo/wine#>

SELECT ?entity ?grape ?year {
  ?search a elastic-index:my_index ;
      elastic:query "grape:cabernet" ;
      elastic:entities ?entity .
  ?entity wine:madeFromGrape ?grape .
  ?entity wine:hasYear ?year
}

The result looks like this:

_images/connectors-combining-results-with-gdb-data.png

Note

:Franvino is returned twice because it is made from two different grapes, both of which are returned.

Entity match score ¶

It is possible to access the match score returned by Elasticsearch with the score predicate. As each entity has its own score, the predicate should come at the entity level. For example:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity ?score {
  ?search a elastic-index:my_index ;
      elastic:query "grape:cabernet" ;
      elastic:entities ?entity .
  ?entity elastic:score ?score
}

The result looks like this but the actual score might be different as it depends on the specific Elasticsearch version:

_images/connectors-entity-match-score.png

Vector search queries ¶

You can conduct vector searches in Elasticsearch by using the similarity predicate in SPARQL or with raw Knn (K nearest neighbors) queries. However, querying data through raw Knn queries differs based on whether the embeddings are generated by Elasticsearch or through another embedding model via the connector.

Vector search using the similarity predicate ¶

You can use SPARQL to query data using the similarity predicate. The queries are the same regardless of what creates the embeddings.

PREFIX conn:<http://www.ontotext.com/connectors/elasticsearch#>
PREFIX inst:<http://www.ontotext.com/connectors/elasticsearch/instance#>


SELECT ?entity WHERE {
   ?search a inst:my-index ;
          conn:similarity "myvectorfield:mysearchterm1 mysearchterm2" ;
          conn:entities ?entity .
}

Vector search using raw Knn queries ¶

You can also use raw Knn queries to access data not exposed through a special predicate.

Raw query with embeddings created by an embedding model:

PREFIX conn:<http://www.ontotext.com/connectors/elasticsearch#>
PREFIX inst:<http://www.ontotext.com/connectors/elasticsearch/instance#>


SELECT ?entity WHERE {
   ?search a inst:my-index ;
          conn:query '''{
                       "knn": {
                         "field": "myvectorfield",
                         "query_vector": "mysearchterm1 mysearchterm2",
                         "k": 100,
                         "num_candidates": 150
                       }
                   }''' ;
          conn:entities ?entity .
}

You can also combine Knn vector search queries and full text search in a raw query:

PREFIX conn:<http://www.ontotext.com/connectors/elasticsearch#>
PREFIX inst:<http://www.ontotext.com/connectors/elasticsearch/instance#>


SELECT ?entity WHERE {
   ?search a inst:my-index ;
conn:query '''{
                     "query": {
                       "bool": {
                         "must": [
                           {
                             "match": {
                               "mytextfield": {
                                 "query": "mysearchterm",
                                 "boost": 1
                               }
                             }
                           },
                           {
                             "knn": {
                               "field": "myvectorfield",
                               "query_vector": "mysearchterm1 mysearchterm2",
                               "k": 10,
                               "boost": 2
                             }
                           }
                         ]
                       }
                     }
                 }''' ;
          conn:entities ?entity .
}

Raw query with embeddings created by Elasticsearch:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>


SELECT ?entity ?name ?abstract {
  ?search a elastic-index:my-index;
      elastic:query '''{
                            "query": {
                              "semantic": {
                                "field": "mysemantictextfield",
                                "query": "mysearchterm"
                              }
                            }
                          }''' ;
      elastic:entities ?entity .
    ?entity <http://schema.org/name> ?name.
    ?entity <http://schema.org/abstract> ?abstract .
}

Vector search in nested objects ¶

If you have nested objects with "vector" or "native:semantic_text" fields, you can also execute vector queries that will search in the nested objects. To do so, you need to define the query as a nested query and provide the full path to the object in the query.

Advanced facet and aggregation queries ¶

While basic faceting allows for simple counting of documents based on the discrete values of a particular field, there are more complex faceted or aggregation searches in Elasticsearch. The Elasticsearch GraphDB Connector provides a mapping from Elasticsearch results to RDF results but no mechanism for specifying the queries other than executing Raw queries.

RDF mapping of the results ¶

The results are accessed through the predicate aggregations (much like the basic facets are accessed through facets). The predicate binds multiple blank nodes that each contains a single aggregation bucket. The individual bucket items can be accessed through these predicates:

Predicate	Meaning	Elasticsearch counterpart
`:name`	Bucket name	`getName()`
`:key`	Key or value associated with the bucket	`getValue()` or `getKey()`
`:count`	Count of documents in the bucket	`getDocCount()`, `getValue()`
`:from`	Start of range	`getFrom()`, `getFromAsDate()`
`:to`	End of range (RangeFacet)	`getTo()`, `getToAsDate()`
`:min`	Minimum value	`getMin()`, `getValue()`
`:max`	Maximum value	`getMax()`, `getValue()`
`:sum`	Sum value	`getSum()`, `getValue()`
`:avg`	Average value	`getAvg()`, `getValue()`
`:sum_of_squares`	Sum of squares value	`getSumOfSquares()`
`:variance`	Variance value	`getVariance()`
`:std_deviation`	Standard deviation value	`getStdDeviation()`
`:parent`	Sub-aggregations: points to the parent (upper level) blank node
`:level`	Sub-aggregations: level number where 1 is the uppermost level and the following levels are 2, 3 and so on
`:levelName`	Sub-aggregations: level name	`getKey()` or `getValue()`

Sorting ¶

You can sort the entities returned by a connector query according to one or more fields. To do this, assign a comma-delimited list of sort fields to the orderBy predicate. Each field can be prefixed with a minus to indicate sorting in descending order. For example:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>
PREFIX wine: <http://www.example.com/demo/wine#>
SELECT ?entity ?sugar{
  ?search a elastic-index:my_index ;
      elastic:query "year:2013" ;
      elastic:orderBy "-sugar" ;
      elastic:entities ?entity.
  ?entity wine:hasSugar ?sugar
}

The result of this sample query contains wines produced in 2013 sorted according to their sugar content in descending order:

By default, entities are sorted according to their matching score in descending order.

Note

If you join the entity from the connector query to other triples stored in GraphDB, GraphDB might scramble the order. Avoid this by using SPARQL’s ORDER BY instruction.

Tip

Sorting by an analyzed textual field works but might produce unexpected results. Analyzed textual fields are composed of tokens and sorting uses the least (in the lexicographical sense) token. For example, “North America” will be sorted before “Europe” because the token “america” is lexicographically smaller than the token “europe”. If you need to sort by a textual field and still do full-text search on it, it is best to create a copy of the field with the setting "analyzed": false. For more information, see Copy fields.

Limit and offset ¶

Limit and offset are supported on the Elasticsearch side of the query. You can set these values using the predicates limit and offset. Consider this example in which an offset of 1 and a limit of 1 are specified:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity {
  ?search a elastic-index:my_index ;
      elastic:query "sugar:dry" ;
      elastic:offset "1" ;
      elastic:limit "1" ;
      elastic:entities ?entity .
}

offset is counted from 0. The result contains a single wine, Franvino. If you execute the query without the limit and offset, Franvino will be second in the list:

Note

The specific order in which GraphDB returns the results depends on how Elasticsearch returns the matches, unless sorting is specified.

Snippet extraction ¶

Snippet extraction lets you retrieve highlighted snippets of text that match a query. You can access the snippets by using the the dedicated predicate elastic:snippets. This binds a blank node that in turn provides the actual snippets via the predicates elastic:snippetField and elastic:snippetText. The predicate snippets must be attached to the entity, as each entity has a different set of snippets. For example, in a search for Cabernet:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity ?snippetField ?snippetText {
  ?search a elastic-index:my_index ;
      elastic:query "grape:cabernet" ;
      elastic:entities ?entity .
  ?entity elastic:snippets ?snippet .
  ?snippet elastic:snippetField ?snippetField ;
     elastic:snippetText ?snippetText .
}

the query returns the two wines made from Cabernet Sauvignon or Cabernet Franc grapes as well as the respective matching fields and snippets:

_images/connectors-snippet-extraction.png

Note

The actual snippets might be different depending on the specific Elasticsearch implementation.

You can tweak how the snippets are collected and composed by using the following option predicates:

elastic:snippetSize: sets the maximum size of the extracted text fragment, 250 by default
elastic:snippetSpanOpen: the text to insert before the highlighted text, <em> by default
elastic:snippetSpanClose: the text to insert after the highlighted text, </em> by default

The option predicates are set on the query instance, much like the elastic:query predicate.

Snippets from nested documents ¶

Note

This section uses the Nested objects dataset described later in this page.

Snippets extracted from nested documents (when a nested query is used) will be available through the same mechanism as snippets from non-nested fields. In addition, nested snippet results provide the nested search path via the snippetInnerField predicate. For example, in a nested search on the field “grandChildren” (specified by “path”) and a match query for “tylor” on the nested field “grandChildren.name”:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity ?snippetInnerField ?snippetField ?snippetText {
  ?search a elastic-index:my_index ;
      elastic:query '''
        {
           "query":{
              "nested":{
                 "path":"grandChildren",
                 "query":{
                    "bool":{
                       "must":[
                          {
                             "match":{
                                "grandChildren.name":"tylor"
                             }
                          }
                       ]
                    }
                 }
              }
           }
        }
      ''' ;
      elastic:entities ?entity .
  ?entity elastic:snippets ?snippet .
  ?snippet elastic:snippetInnerField ?snippetInnerField ;
           elastic:snippetField ?snippetField ;
           elastic:snippetText ?snippetText .
}

the query returns all people who have a grandchild whose name matches “tylor”, as well as the highlighted snippets:

`?entity`	`?snippetInnerField`	`?snippetField`	`?snippetText`
`urn:Eva`	grandChildren	grandChildren.name	`John-<em>Tylor</em>`
`urn:John`	grandChildren	grandChildren.name	`John-<em>Tylor</em>`
`urn:John`	grandChildren	grandChildren.name	`<em>Tylor</em>`
`urn:Mary`	grandChildren	grandChildren.name	`<em>Tylor</em>`

Note that the matching field whose matching values are highlighted is provided via the snippetField predicate, just like extracting snippets with non-nested searches, while the predicate snippetInnerField provides the field on which the nested search was executed.

Total hits ¶

You can get the total number of matching Elasticsearch documents (hits) by using the elastic:totalHits predicate. For the my_index connector instance used above, a query for all wines made in 2012 might look like this:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?totalHits {
    ?r a elastic-index:my_index ;
       elastic:query "year:2012" ;
       elastic:totalHits ?totalHits .
}

As there are three wines made in 2012, the value 3 (of type xsd:long) binds to ?totalHits.

As you see above, you can omit the return of any of the matching entities. This can be useful if there are many hits and you want to calculate pagination parameters.

List of creation parameters ¶

The creation parameters define how the elastic:createConnector predicate creates a connector instance. Some are required and some are optional. All parameters are provided together in a JSON object, where the parameter names are the object keys. Parameter values may be simple JSON values such as a string or a boolean, or they can be lists or objects.

All of the creation parameters can also be set conveniently from the Create Connector user interface without any knowledge of JSON.

readonly (boolean), optional, read-only mode

A read-only connector will index all existing data in the repository at creation time, but, unlike non-read-only connectors, it will:

Not react to updates. Changes will not be synced to the connector.

Not keep any extra structures (such as the internal internal data structures for tracking updates to chains).

The only way to index changes in data after the connector has been created is to repair (or drop and recreate) the connector.

importGraph (boolean), optional, specifies that the RDF data from which to create the connector is in a special virtual graph

Used to make an Elasticsearch index from temporary RDF data inserted in the same transaction. It requires read-only mode and creates a connector whose data will come from statements inserted into a special virtual graph instead of data contained in the repository. The virtual graph is elastic:graph, where the prefix elastic: represents the IRI http://www.ontotext.com/connectors/elasticsearch# as before. The data must be inserted into this graph before the connector create statement is executed.

Both the insertion into the special graph and the create statement must be in the same transaction. In Workbench, you can do this by pasting them one after another in the SPARQL editor and putting a semicolon at the end of the first INSERT. This feature requires read-only mode.

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
INSERT {
    GRAPH elastic:graph {
        ...
    }
} WHERE {
        ...
};
PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>
INSERT DATA {
    elastic-index:my_index elastic:createConnector '''
{
  "readonly": true,
  "importGraph": true,
  "fields": [],
  "languages": [],
  "types": [],
}
''' .
}

importFile (string), optional, an RDF file with data from which to create the connector: Creates a connector whose data will come from an RDF file on the file system instead of data contained in the repository. The value must be the full path to the RDF file. This feature requires readonly mode.

detectFields (boolean), optional, detects fields

This mode introduces automatic field detection when creating a connector. You can omit the specification of fields in JSON. Instead, you will get automatic fields, each corresponding to a single predicate with a field name the same as the predicate, so you must use escaping when issuing Elasticsearch queries.

In this mode, specifying types is also optional. If types are not provided, then all types will be indexed. This mode requires importGraph or importFile.

Once the connector is created, you can inspect the detected fields in the Connector management section of Workbench.

elasticsearchNode (string), required, the Elasticsearch instance to sync to As Elasticsearch is a third-party service, you have to specify the node where it is running. The node value takes the form http://hostname.domain:port or https://hostname.domain:port. No default value. Can be updated at runtime with no need to rebuild the index.

Note

Elasticsearch exposes two protocols — the native transport* protocol over port 9300 and the RESTful API over port 9200. The Elasticsearch GraphDB Connector uses the RESTful API over port 9200.

indexCreateSettings (json), optional, the settings for creating the Elasticsearch index: This option is passed directly to Elasticsearch when creating the index.

elasticsearchBasicAuthUser (string), optional, the settings for supplying the authentication user: No default value. Can be updated at runtime with no need to rebuild the index.

elasticsearchBasicAuthPassword (string), optional, the settings for supplying the authentication password: A password is a string with a single value that is not logged or printed. No default value. Can be updated at runtime with no need to rebuild the index.

elasticsearchClusterSniff (boolean), controls whether to build the server address list by sniffing on the Elasticsearch cluster: Corresponds to the Elasticsearch client.transport.sniff option. true by default Can be updated at runtime with no need to rebuild the index.

bulkUpdateBatchSize (integer), controls the maximum number of documents sent per bulk request: Default value is 5,000. Can be updated at runtime with no need to rebuild the index.

authenticationConfiguratorClass provides custom authentication behavior: Optional.

types (list of IRIs), required, specifies the types of entities to sync

The RDF types of entities to sync are specified as a list of IRIs. At least one type IRI is required.

Use the pseudo-IRI $any to sync entities that have at least one RDF type.

Use the pseudo-IRI $untyped to sync entities regardless of whether they have any RDF type. See also the examples in General full-text search with the connectors.

languages (list of strings), optional, valid languages for literals: RDF data is often multilingual, but only some of the languages represented in the literal values can be mapped. You can do this by specifying a list of language ranges to be matched to the language tags of literals according to RFC 4647, Section 3.3.1. Basic Filtering. You can also use an empty range to include literals that have no language tag. The list of language ranges maps all existing literals that have matching language tags.

fields (list of field objects), required, defines the mapping from RDF to Elasticsearch

The fields specify exactly which parts of each entity will be synchronized as well as the specific details on the connector side. The field is the smallest synchronization unit, and it maps a property chain from GraphDB to a field in Elasticsearch. The fields are specified as a list of field objects. At least one field object is required. Each field object has further keys that specify details.

fieldName (string), required, the name of the field in Elasticsearch
The name of the field defines the mapping on the connector side. It is specified by the key fieldName with a string value. The field name is used at query time to refer to the field. There are few restrictions on the allowed characters in a field name, but to avoid unnecessary escaping (which depends on how Elasticsearch parses its queries), we recommend keeping the field names simple.

fieldNameTransform (one of none, predicate or predicate.localName), optional, none by default
Defines an optional transformation of the field name. Although fieldName is always required, it is ignored if fieldNameTransform is predicate or predicate.localName.
- none: The field name is supplied via the fieldName option.
- predicate: The field name is equal to the full IRI of the last predicate of the chain. For example, if the last predicate was http://www.w3.org/2000/01/rdf-schema#label, then the field name will also be http://www.w3.org/2000/01/rdf-schema#label.
- predicate.localName: The field name is derived from the local name of the IRI of the last predicate of the chain. For example, if the last predicate was http://www.w3.org/2000/01/rdf-schema#comment, then the field name will be comment.
See Indexing all literals in distinct fields for an example.
propertyChain (list of IRI), required, defines the property chain to reach the value
The property chain defines the mapping on the GraphDB side. A property chain is a sequence of triples where the entity IRI is the subject of the first triple, its object is the subject of the next triple, etc. In this model, a property chain with a single element corresponds to a direct property defined by a single triple. Property chains are specified as a list of IRIs where at least one IRI must be provided.

The IRI of the document will be synchronized to the special field _id in Elasticsearch. You may use it to query Elasticsearch directly and to retrieve the matching entity IRI.

See Copy fields for defining multiple fields with the same property chain.

See Multiple property chains per field for defining a field whose values are populated from more than one property chain.

See Indexing language tags for defining a field whose values are populated with the language tags of literals.

See Indexing the IRI of an entity for defining a field whose values are populated with the IRI of the indexed entity.

See Wildcard literal indexing for defining a field whose values are populated with literals regardless of their predicate.
valueFilter (string), optional, specifies the value filter for the field
See also Entity filtering.
documentFilter (string), optional, specifies the nested document filter for the field
Only for fields that define nested documents. See also Entity filtering.
defaultValue (string), optional, specifies a default value for the field
Provides a means to specify a default value for the field when the property chain has no matching values in GraphDB. The default value can be a plain literal, a literal with a datatype (xsd: prefix supported), a literal with language, or a IRI. It has no default value.
indexed (boolean), optional, default true
If indexed, a field is available for Elasticsearch queries. true by default.

If true, this option corresponds to "index" = true. If false, it corresponds to "index" = false.
stored (boolean), optional, default true
Whether fields are stored in Elasticsearch. true by default.

This option corresponds to the property "store" in the Elasticsearch mapping.
analyzed (boolean), optional, default true
When literal fields are indexed in Elasticsearch, they will be analyzed according to the analyzer settings. Should you require that a given field is not analyzed, you may use "analyzed". This option has no effect for IRIs, which are never analyzed. true by default.

If true, this option will use automatic or manual (datatype option) type for the Elasticsearch mapping. If false, it corresponds to "type" = "keyword" (i.e., the default type will be changed to keyword).
multivalued (boolean), optional, default true
RDF properties and synchronized fields may have more than one value. If multivalued is set to true, all values will be synchronized to Elasticsearch. If set to false, only a single value will be synchronized. true by default.
ignoreInvalidValues (boolean), optional, default false
Per-field option that controls what happens when a value cannot be converted to the requested (or previously detected) type. False by default.

Example use: when an invalid date literal like "2021-02-29"^^xsd:date (2021 is not a leap year) needs to be indexed as a date, or when an IRI needs to be indexed as a number.

Note

Some conversions are always valid: any literal to an FTS field, any non-literal (IRI, blank node, embedded triple) to a non-analyzed field. When true, such values will be skipped with a note in the logs. When false, such values will break the transaction.
array (boolean), optional, default false
Normally, Elasticsearch creates an array only if more than value is present for a given field. If array is set to true, Elasticsearch will always create an array even for single values. If set to false, Elasticsearch will create arrays for multiple values only. False by default.
fielddata (boolean), optional, default false
Allows fielddata to be built in memory for text fields. This can consume a lot of heap space, especially when loading high cardinality text fields. False by default.

datatype (string), optional, the manual datatype override
By default, the Elasticsearch GraphDB Connector uses datatype of literal values to determine how they should be mapped to Elasticsearch types. For more information on the supported datatypes, see Datatype mapping.

You can override the datatype mapping with the "datatype" parameter, which can be specified per field. The value of datatype can be any of the xsd: types supported by the automatic mapping or a native Elasticsearch type prefixed by native:. For example, both xsd:long and native:long map to the long type in Elasticsearch.

When using vector similarity search, you need to provide the datatype of the field that will be used for similarity search:

If the embeddings are created by an externally provided model (such as a locally deployed model or an OpenAI model) via GraphDB, specify "vector" as the value of the "datatype" parameter. This datatype creates a dense_vector field.

If the embeddings are created by Elasticsearch, specify "native:semantic_text".

Note

If you want to use an inference pipeline from Elasticsearch, specify "nativeSettings": {"inference_id": "your_inference_id"}.

You can also create a single vector from multiple fields by specifying "vector" or "native:semantic_text" (depending on where the embeddings are generated) as the datatype for each of the fields you want to use.

nativeSettings (json), optional, custom field settings
The setting for the Elasticsearch mapping parameters of the respective field such as the format of the datatype. Native field settings require an explicit native datatype.

nativeSettings are not allowed for the following parameters so as to avoid conflicts with the existing way to specify them: type, index, store, analyzer, fielddata.
objectFields (objects array), optional, nested object mapping
When native:object, native:nested, or native:geo_point is used as a datatype value, this provides a mapping for the nested object’s fields. If datatype is not provided, then native:object will be assumed.

For the difference between object and nested, refer to the Elastic nested field type. The geo_point type must have exactly two fields named lat and long (required by Elastic, see geo-point field type).

Nested objects support further nested objects with a limit of five levels of nesting. See Nested objects for an example.

startFromParent (integer), optional, default 0
Start processing the property chain from the N-th parent instead of the root of the current nested object. 0 is the root of the current nested object, 1 is the parent of the nested object, 2 is the parent of the parent and so on.
analyzer (string), optional, per field analyzer
The Elasticsearch analyzer that is used for indexing the field can be specified with the parameter analyzer. It will be passed directly to Elasticsearch’s property analyzer when creating the mapping (see Custom Analyzers in the Elasticsearch documentation). For example:
{
...
  "fields": [
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.example.com/demo/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ],
      "analyzer": "my_analyzer"
    },
...
}

valueFilter (string), optional, specifies the top-level value filter for the document: See also Entity filtering.
documentFilter (string), optional, specifies the top-level document filter for the document: See also Entity filtering.
embeddingModel (string), optional, specifies the embedding model used to create the embeddings when using vector search: See also Embedding model configuration for vector search

Updating parameters at runtime ¶

As mentioned above, you can update the following connection parameters at runtime with no need to rebuild the index:

elasticsearchNode
elasticsearchClusterSniff
elasticsearchBasicAuthUser
elasticsearchBasicAuthPassword
bulkUpdateBatchSize

The following SPARQL update request demonstrates this by updating the user and password values:

PREFIX conn:<http://www.ontotext.com/connectors/elasticsearch#>
PREFIX inst:<http://www.ontotext.com/connectors/elasticsearch/instance#>
INSERT DATA {
inst:proper_index conn:updateConnector '''
    {
        "elasticsearchBasicAuthUser": "foo",
        "elasticsearchBasicAuthPassword": "bar"
    }
''' .
}

Special field definitions ¶

Nested objects ¶

Nested objects are Elasticsearch documents that are used as values in the main document or in other nested objects. Up to five levels of nesting is possible. They are defined with the objectFields option.

Let’s look at an example that uses the following data, which consists of children and grandchildren relations:

<urn:John>
    a <urn:Person> ;
    <urn:name> "John" ;
    <urn:gender> <urn:Male> ;
    <urn:age> 60 ;
    <urn:hasSpouse> <urn:Mary> ;
    <urn:hasChild> <urn:Billy> ;
    <urn:hasChild> <urn:Annie> .

<urn:Mary>
    a <urn:Person> ;
    <urn:name> "Mary" ;
    <urn:gender> <urn:Female> ;
    <urn:age> 58 ;
    <urn:hasSpouse> <urn:John> ;
    <urn:hasChild> <urn:Billy> .

<urn:Eva>
    a <urn:Person> ;
    <urn:name> "Eva" ;
    <urn:gender> <urn:Female> ;
    <urn:age> 45 ;
    <urn:hasChild> <urn:Annie> .

<urn:Billy>
    a <urn:Person> ;
    <urn:name> "Billy" ;
    <urn:gender> <urn:Male> ;
    <urn:age> 35 ;
    <urn:hasChild> <urn:Tylor> ;
    <urn:hasChild> <urn:Melody> .

<urn:Annie>
    a <urn:Person> ;
    <urn:name> "Annie" ;
    <urn:gender> <urn:Female> ;
    <urn:age> 28 ;
    <urn:hasChild> <urn:Sammy> .

<urn:Tylor>
    a <urn:Person> ;
    <urn:name> "Tylor" ;
    <urn:gender> <urn:Male> ;
    <urn:age> 5 .

<urn:Melody>
    a <urn:Person> ;
    <urn:name> "Melody" ;
    <urn:gender> <urn:Female> ;
    <urn:age> 2 .

<urn:Sammy>
    a <urn:Person> ;
    <urn:name> "Sammy" ;
    <urn:gender> <urn:Male> ;
    <urn:age> 10 .

<urn:Male> <urn:label> "male" .

<urn:Female> <urn:label> "female" .

We can create a nested objects index that consists of children and grandchildren with their corresponding fields defining their gender and age. We use the native:nested type because we want to query the nested objects independently of each other:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["urn:Person"],
      "fields": [
        {
          "fieldName": "name",
          "propertyChain": [
            "urn:name"
          ]
        },
        {
          "fieldName": "age",
          "propertyChain": [
            "urn:age"
          ],
          "datatype": "xsd:long"
        },
        {
          "fieldName": "hasSpouse",
          "propertyChain": [
            "urn:hasSpouse"
          ]
        },
        {
          "fieldName": "gender",
          "propertyChain": [
            "urn:gender",
            "urn:label"
          ]
        },
        {
          "fieldName": "children",
          "propertyChain": [
            "urn:hasChild"
          ],
          "datatype": "native:nested",
          "objectFields": [
            {
              "fieldName": "id",
              "propertyChain": [
                "$self"
              ]
            },
            {
              "fieldName": "name",
              "propertyChain": [
                "urn:name"
              ]
            },
            {
              "fieldName": "age",
              "propertyChain": [
                "urn:age"
              ],
              "datatype": "xsd:long"
            },
            {
              "fieldName": "gender",
              "propertyChain": [
                "urn:gender",
                "urn:label"
              ]
            },
            {
              "fieldName": "children",
              "propertyChain": [
                "urn:hasChild"
              ],
              "objectFields": [
                {
                  "fieldName": "id",
                  "propertyChain": [
                    "$self"
                  ]
                },
                {
                  "fieldName": "name",
                  "propertyChain": [
                    "urn:name"
                  ]
                },
                {
                  "fieldName": "age",
                  "propertyChain": [
                    "urn:age"
                  ],
                  "datatype": "xsd:long"
                }
              ]
            }
          ]
        },
        {
          "fieldName": "grandChildren",
          "valueFilter": "$this -> type in (<urn:Person>)",
          "propertyChain": [
            "urn:hasChild",
            "urn:hasChild"
          ],
          "datatype": "native:nested",
          "objectFields": [
            {
              "fieldName": "id",
              "propertyChain": [
                "$self"
              ]
            },
            {
              "fieldName": "name",
              "propertyChain": [
                "urn:name"
              ]
            },
            {
              "fieldName": "age",
              "propertyChain": [
                "urn:age"
              ],
              "datatype": "xsd:long"
            },
            {
              "fieldName": "gender",
              "propertyChain": [
                "urn:gender",
                "urn:label"
              ]
            }
          ]
        },
      ]
    }
 '''
}

To find male grandchildren of age 5 and older, we will use the following query:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?entity {
    ?search a elastic-index:my_index ;
        elastic:query '''
            {
               "query" : {
                  "nested" : {
                     "path" : "grandChildren",
                     "query" : {
                        "bool" : {
                           "must" : [
                              {
                                 "match" : {
                                    "grandChildren.gender" : "male"
                                 }
                              },
                              {
                                 "range" : {
                                    "grandChildren.age" : {
                                       "gt" : 5
                                    }
                                 }
                              }
                           ]
                        }
                     }
                  }
               }
            }
        ''' ;
        elastic:entities ?entity .
}
ORDER BY ?entity

The result looks like this:

?entity

urn:Eva

urn:John

Nested objects with vector datatypes¶

You can also execute vector search queries with nested objects by adding fields to the nested object with "vector" or "native:semantic_text as the value of the "datatype" parameter.

Copy fields ¶

It is often convenient to synchronize one and the same data multiple times with different settings to accommodate for different use cases such as faceting or sorting vs full-text search. The Elasticsearch GraphDB Connector has explicit support for fields that copy their value from another field. To do this, specify a single element in the property chain of the form @otherFieldName, where otherFieldName is another non-copy field. Take the following example:

...
  "fields": [
    {
      "fieldName": "grape",
      "facet": false,
      "propertyChain": [
        "http://www.example.com/demo/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ],
      "analyzed": true
    },
    {
      "fieldName": "grapeFacet",
      "propertyChain": [
        "@grape"
      ],
      "analyzed": false
    }
  ]
...

The snippet creates an analyzed field “grape” and a non-analyzed field “grapeFacet”; both fields are populated with the same values and “grapeFacet” is defined as a copy field that refers to the field “facet”.

Note

The connector handles copy fields in a more optimal way than specifying a field with exactly the same property chain as another field.

Multiple property chains per field ¶

Sometimes, you have to work with data models that define the same concept (in terms of what you want to index in Elasticsearch) with more than one property chain — for example, the concept of “name” could be defined as a single canonical name, multiple historical names and some unofficial names. If you want to index these together as a single field in Elasticsearch, you can define this as a multiple property chains field.

Fields with multiple property chains are defined as a set of separate virtual fields that will be merged into a single physical field when indexed. Virtual fields are distinguished by the suffix $xyz, where xyz is any convenient alphanumeric sequence. For example, we can define the fields name$1 and name$2 like this:

...
  "fields": [
    {
      "fieldName": "name$1",
      "propertyChain": [
        "http://www.example.com/demo#canonicalName"
      ],
      "fieldName": "name$2",
      "propertyChain": [
        "http://www.example.com/demo#historicalName"
      ]
      ...
    },
...

The values of the fields name$1 and name$2 will be merged and synchronized to the name field in Elasticsearch.

Note

You cannot mix suffixed and unsuffixed fields with the same name. For example, if you defined myField$new and myField$old, you cannot have a field called just myField.

Filters and fields with multiple property chains¶

You can use filters with fields defined with multiple property chains. Both the physical field values and the individual virtual field values are available:

Physical fields are specified without the suffix (for example, ?myField)
Virtual fields are specified with the suffix (for example, ?myField$2 or ?myField$alt)

Note

Physical fields cannot be combined with parent() as their values come from different property chains. If you really need to filter the same parent level, you can rewrite parent(?myField) in (<urn:x>, <urn:y>) as parent(?myField$1) in (<urn:x>, <urn:y>) || parent(?myField$2) in (<urn:x>, <urn:y>) || parent(?myField$3) ... and surround it with parentheses if it is a part of a bigger expression.

Indexing language tags ¶

The language tag of an RDF literal can be indexed by specifying a property chain whose last element is the pseudo-IRI lang(). The property preceding lang() must lead to a literal value. For example:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["http://www.example.com/demo#gadget"],
      "fields": [
         {
           "fieldName": "name",
           "propertyChain": [
             "http://www.example.com/demo#name"
           ]
         },
         {
           "fieldName": "nameLanguage",
           "propertyChain": [
             "http://www.example.com/demo#name",
             "lang()"
           ]
         }
      ],
    }
  ''' .
}

The above connector will index the language tag of each literal value of the property http://www.example.com/demo#name into the field nameLanguage.

Indexing named graphs ¶

You can index a given value’s named graph by ending a property chain with the special pseudo-URI graph(). Indexing the named graph of the value instead of the value itself allows searching by named graph.

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["http://www.example.com/demo#gadget"],
      "fields": [
         {
           "fieldName": "name",
           "propertyChain": [
             "http://www.example.com/demo#name"
           ]
         },
         {
           "fieldName": "nameGraph",
           "propertyChain": [
             "http://www.example.com/demo#name",
             "graph()"
           ]
         }
      ],
    }
  ''' .
}

The above connector will index the named graph of each value of the property http://www.example.com/demo#name into the field nameGraph.

Indexing local names ¶

You can index the local name of a given IRI value by ending a property chain with the special pseudo-URI localName(). Indexing the local name instead of the full IRI is convenient when the local name is a human-readable meaningful string.

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["http://www.example.com/demo#gadget"],
      "fields": [
         {
           "fieldName": "name",
           "propertyChain": [
             "http://www.example.com/demo#name"
           ]
         },
         {
           "fieldName": "feature",
           "propertyChain": [
             "http://www.example.com/demo#feature",
             "localName()"
           ]
         }
      ],
    }
  ''' .
}

The above connector will index the local name of each IRI value of the property http://www.example.com/demo#feature into the field feature.

Wildcard literal indexing ¶

In this mode, the last element of a property chain is a wildcard that will match any predicate that leads to a literal value. Use the special pseudo-IRI $literal as the last element of the property chain to activate it.

Note

Currently, it really means any literal, including literals with data types.

For example:

{
    "fields" : [ {
        "propertyChain" : [ "$literal" ],
        "fieldName" : "name"
    }, {
        "propertyChain" : [ "http://example.com/description", "$literal" ],
        "fieldName" : "description"
    }
    ...
}

See Indexing all literals for a detailed example.

Indexing the IRI of an entity ¶

Sometimes you may need the IRI of each entity (for example, http://www.example.com/demo/wine#Franvino from our small example dataset) indexed as a regular field. Do this by specifying a property chain with a single property referring to the pseudo-IRI $self. For example:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
    elastic-index:my_index elastic:createConnector '''
{
  "elasticsearchNode": "localhost:9200",
  "types": [
    "http://www.example.com/demo/wine#Wine"
  ],
  "fields": [
    {
      "fieldName": "entityId",
      "propertyChain": [
        "$self"
      ],
    },
    {
      "fieldName": "grape",
      "propertyChain": [
        "http://www.example.com/demo/wine#madeFromGrape",
        "http://www.w3.org/2000/01/rdf-schema#label"
      ]
    },
  ]
}
''' .
}

The above connector will index the IRI of each wine into the field entityId.

Note

Note that GraphDB will also use the IRI of each entity as the ID of each document in Elasticsearch, which is represented by the field id.

Datatype mapping ¶

The Elasticsearch GraphDB Connector maps different types of RDF values to different types of Elasticsearch values according to the basic type of the RDF value (IRI or literal) and the datatype of literals. The auto-detection uses the following mapping:

RDF value	RDF datatype	Elasticsearch type
IRI	n/a	`keyword`
literal	any type not explicitly mentioned below	`text`
literal	`xsd:boolean`	`boolean`
literal	`xsd:double`	`double`
literal	`xsd:float`	`float`
literal	`xsd:long`	`long`
literal	`xsd:int`	`integer`
literal	`xsd:dateTime`	`date` with `format: strict_date_time`
literal	`xsd:date`	`date` with `format: strict_date`
literal	`xsd:time`	`date` with `format: strict_time_no_millis\|\|strict_time`
literal	`xsd:gYear`	`date` with `format: strict_year`
literal	`xsd:gYearMonth`	`date` with `format: strict_year_month`

For any given field, the automatic mapping uses the first value it sees. This works fine for clean datasets but might lead to problems if your dataset has non-normalized data — for example, the first value has no datatype but other values do. Because of this, it’s best to set the datatype to a fixed value such as xsd:date.

Note

There are two additional datatypes used for vector search but not by the datatype mapping mechanism — see the datatype entry in the Creation parameters section for more details on which values to use when creating an Elasticsearch connector instance with vector search capabilities.

Warning

The commonly used xsd:integer and xsd:decimal datatypes are not indexed as numbers because they represent infinite precision numbers. You can override that by using the datatype option to cast them to xsd:long, xsd:double, or xsd:float as appropriate.

Date and time conversion ¶

RDF and Elasticsearch use slightly different models for representing dates and times, even though the values might look very similar.

Years in RDF values use the XSD format and are era years, where positive values denote the common era and negative values denote years before the common era. There is no year zero.

Years in Elasticsearch use the ISO 8601 format and are proleptic years: positive values denote years from the common era with any previous eras just going down by one mathematically so there is year zero.

In short:

year 2020 CE = year 2020 in XSD = year 2020 in ISO.
…
year 1 CE = year 1 in XSD = year 1 in ISO.
year 1 BCE = year -1 in XSD = year 0 in ISO.
year 2 BCE = year -2 in XSD = year -1 in ISO.
…

All years coming from RDF literals will be converted to ISO before indexing in Elasticsearch.

Both XSD and ISO date and time values support timezones. In addition to that, XSD defines the lack of a timezone as undetermined. Since we do not want to have any undetermined state in the indexing system, GraphDB defines the undetermined time zone as UTC. For example, "2020-02-14T12:00:00"^^xsd:dateTime is equivalent to "2020-02-14T12:00:00Z"^^xsd:dateTime (Z is the UTC timezone, also known as +00:00).

Also note that XSD dates and partial dates, e.g., xsd:gYear values, may have a timezone, which leads to additional complications. For example, "2020+02:00"^^xsd:gYear (the year 2020 in the +02:00 timezone) will be normalized to 2019-12-31T22:00:00Z (the previous year!) if GraphDB followed strict timezone adherence. We have chosen to ignore the timezone on any values that do not have an associated time value. For example, all of the following values will be treated as if they specified UTC as their timezone:

"2020-02-15+02:00"^^xsd:date
"2020-02+02:00"^^xsd:gYearMonth
"2020+02:00"^^xsd:gYear

Entity filtering ¶

The Elasticsearch connector supports four kinds of entity filters used to fine-tune the set of entities and individual values for the configured fields, based on the field value. Entities and field values are synchronized to Elasticsearch if, and only if, they pass the filter. The filters are similar to a SPARQL query’s FILTER() but not exactly the same. You can reference each configured field in them by prefixing its name with a ?, much like referencing a SPARQL variable.

Types of filters ¶

Top-level value filter

The top-level value filter is specified with the valueFilter parameter. It is evaluated prior to anything else when only the document ID is known, and it may not refer to any field names but only to the special field $this that contains the current document ID. Failing to pass this filter removes the entire document early in the indexing process. It can be used to introduce more restrictions similar to the built-in filtering by type via the types property.

Top-level document filter

The top-level document filter is specified with the documentFilter parameter. This filter is evaluated last when all of the document has been collected and it decides whether to include the document in the index. You can use it to enforce global document restrictions such as that certain fields are required or a document needs to be indexed only if a certain field value meets specific conditions.

Per-field value filter

The per-field value filter is specified with the valueFilter parameter inside the field definition of the field whose values are to be filtered. The filter is evaluated while collecting the data for the field when each field value becomes available.

The variable that contains the field value is $this. You can use other field names to filter the current field’s value based on the value of another field. For example, $this > ?age will compare the current field value to the value of the field age (see also Two-variable filtering). Failing to pass the filter will remove the current field value.

On nested documents, the per-field value filter can be used to remove the entire nested document early in the indexing process, e.g., by checking the type of the nested document via next hop with rdf:type.

Nested document filter

The nested document filter is specified with the documentFilter parameter inside the field definition of the field that defines the root of a nested document. The filter is evaluated after the entire nested document has been collected. Failing to pass this filter removes the entire nested document.

Inside a nested document filter, the field names are within the context of the nested document and not within the context of the top-level document. For example, if you have a children field that defines a nested document and you use a filter like ?age < "10"^^xsd:int, it will refer to the field children.age. You can use the prefix $outer. one or more times to refer to field values from the outer document from the viewpoint of the nested document. For example, $outer.age > "25"^^xsd:int will refer to the age field that is a sibling of the children field.1

Other than the above differences, the nested document filter is equivalent to the top-level document filter from the viewpoint of the nested document.

Filter operators ¶

The filter operators test whether a given field’s value satisfies a certain condition.

Field comparisons are done on original RDF values before they are converted to Elasticsearch values using datatype mapping.

Operator	Meaning
`?var in (value1, value2, ...)`	Tests if the `var` field’s value is one of the specified values. Unlike the similar SPARQL operator, values are compared strictly — like the SPARQL `sameTerm()` function, values that do not match are treated as if they were not present in the repository. Example: `?status in ("active", "new")`
`?var not in (value1, value2, ...)`	The negated version of the `in` operator. Example: `?status not in ("archived")`
`bound(?var)`	Tests if the field `var` has a valid value. This can be used to make the field compulsory. Example: `bound(?name)`
`isExplicit(?var)`	Tests if the `var` field’s value came from an explicit statement. This will use the last element of the property chain. If you need to assert the explicit status of a previous property chain use `parent(?var)` as many times as needed. Example: `isExplicit(?name)`
`?var = value` (equal to) `?var != value` (not equal to) `?var > value` (greater than) `?var >= value` (greater than or equal to) `?var < value` (less than) `?var <= value` (less than or equal to)	RDF value comparison operators that compare RDF values similarly to the equivalent SPARQL operators. The `var` field’s value will be compared to the specified RDF value. When comparing RDF values that are literals, their datatypes must be compatible such as `xsd:integer` and `xsd:long` but not `xsd:string` and `xsd:date`. Values that do not match are treated as if they were not present in the repository. Examples: Given that `height` has a value of `"150"^^xsd:int` and `dateOfBirth` has a value of `"1989-12-31"^^xsd:date`, then: `?height = "150"^^xsd:int` is `true` `?height = "150"^^xsd:long` is `true` `?height = "150"` is `false` `?height != "151"^^xsd:int` is `true` `?height != "150"` is `true` `?height > "150"^^xsd:int` is `false` `?height >= "150"^^xsd:int` is `true` `?dateOfBirth < "1990-01-01"^^xsd:date` is `true`
`regex(?var, "pattern")` or `regex(?var, "pattern", "i")`	Tests if the `var` field’s value matches the given regular expression pattern. The “i” flag tells the function to ignore case when matching. Values that do not match are treated as if they were not present in the repository. Example: `regex(?name, "^mrs?", "i")`
`expr1 \|\| expr2` or `expr1 or expr2`	Logical disjunction of expressions `expr1` and `expr2`. Examples: `bound(?name) \|\| bound(?company)` `bound(?name) or bound(?company)`
`expr1 && expr2` or `expr1 and expr2`	Logical conjunction of expressions `expr1` and `expr2`. Examples: `bound(?status) && ?status in ("active", "new")` `bound(?status) and ?status in ("active", "new")`
`!expr`	Logical negation of expression `expr`. Example: `!bound(?company)`
`( expr )`	Grouping of expressions Example: `(bound(?name) or bound(?company)) && bound(?address)`

Filter modifiers ¶

Along with the filter operators, additional constructions let you write filters based not on the values of a field but on values related to them.

Accessing the previous element in the chain: The construction parent(?var) goes to a previous level in a property chain. You can apply it recursively as many times as as needed; for example, parent(parent(parent(?var))) goes back in the chain three times. You can use the effective value of parent(?var) with the in or not in operators (for example, parent(?company) in (<urn:a>, <urn:b>), or in the bound operator (for example, parent(bound(?var))).

Accessing an element beyond the chain

The construction ?var -> uri (alternatively, ?var o uri or just ?var uri) accesses additional values through the uri property. This construction roughly corresponds to the triple pattern value uri ?effectiveValue, where ?value is a value bound by the var field. You can use the effective value of ?var -> uri with the in or not in operator like this: ?company -> rdf:type in (<urn:c>, <urn:d>). You can combine it with parent() like this: parent(?company) -> rdf:type in (<urn:c>, <urn:d>). You can apply the same construction to the bound operator like this: bound(?company -> <urn:hasBranch>), or you can combine it with parent() like this: bound(parent(?company) -> <urn:hasGroup>).

The IRI parameter can be a full IRI within < > or the special string rdf:type (or just type), which will be expanded to http://www.w3.org/1999/02/22-rdf-syntax-ns#type.

Filtering by RDF graph

The construction graph(?var) gives you access to the RDF graph of a field’s value. A typical use case is to sync only explicit values: graph(?a) not in (<http://www.ontotext.com/implicit>) but using isExplicit(?a) is the recommended way.

You can combine this construction with the parent() function like this: graph(parent(?a)) in (<urn:a>).

Filtering by language tags: The construction lang(?var) gives you access to the language tag of field’s value. (Only RDF literals can have a language tag.) The typical use case is to sync only values written in a given language: lang(?a) in ("de", "it", "no"). You can combine this construction with the parent() function and an element beyond the chain like this: lang(parent(?a) -> <http://www.w3.org/2000/01/rdf-schema#label>) in ("en", "bg"). You can filter literal values without language tags by using an empty tag: "".

Current context variable $this: You can use the special field variable $this (and not ?this, ?$this, or $?this) to refer to the current context. In the top-level value filter and the top-level document filter, it refers to the document. In the per-field value filter, it refers to the currently filtered field value. In the nested document filter, it refers to the nested document.

ALL() quantifier

In the context of document-level filtering, a match is true if at least one of potentially many field values match. For example, ?location = <urn:Europe> would return true if the document contains { "location": ["<urn:Asia>", "<urn:Europe>"] }.

You can also use the ALL() quantifier when you need all values to match. For example, ALL(?location) = <urn:Europe> would not match with the above document because <urn:Asia> does not match.

Entity filters and default values

You can combine entity filters with default values for more flexible behavior.

If a field has no values in the RDF database, the defaultValue is used. But if a field has some values, defaultValue is NOT used, even if all values are filtered out. See an example in Basic entity filter.

A typical use case for an entity filter is having soft deletes — instead of deleting an entity, it is marked as deleted by the presence of a specific value for a given property.

Two-variable filtering ¶

Besides comparing a field value to one or more constants, or running an existential check on the field value, some use cases also require the ability to compare the field value with the value of another field in order to produce the desired result. GraphDB solves this by supporting two-variable filtering in the per-field value filter, the top-level document filter, and the nested document filter.

Note

This type of filtering is not possible in the top-level value filter because the only variable that is available there is $this.

In the top-level document filter and the nested document filter, there are no restrictions because all values are available at the time of evaluation.

In the per-field value filter, two-variable filtering will reorder the defined fields such that values for other fields are already available when the current field’s filter is evaluated. For example, let’s say we defined a filter $this > ?salary for the field price. This will force the connector to process the field salary first, apply its per-field value filter if any, and only then start collecting and filtering the values for the field price.

Cyclic dependencies will be detected and reported as an invalid filter. For example, if in addition to the above we define a per-field value filter ?price > "1000"^^xsd:int for the field salary, a cyclic dependency will be detected as both price and salary will require the other field to be indexed first.

Basic entity filter example ¶

Let’s look at an example that uses the following RDF data.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix example: <http://www.example.com/demo#> .

# the entity below will be synchronised because it has a matching value for city: ?city in ("London")
example:alpha
    rdf:type example:gadget ;
    example:name "John Synced" ;
    example:city "London" .

# the entity below will not be synchronised because it lacks the property completely: bound(?city)
example:beta
    rdf:type example:gadget ;
    example:name "Peter Syncfree" .

# the entity below will not be synchronized because it has a different city value:
# ?city in ("London") will remove the value "Liverpool" so bound(?city) will be false
example:gamma
    rdf:type example:gadget ;
    example:name "Mary Syncless" ;
    example:city "Liverpool" .

If you create a connector instance such as:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["http://www.example.com/demo#gadget"],
      "fields": [
         {
           "fieldName": "name",
           "propertyChain": ["http://www.example.com/demo#name"]
         },
         {
           "fieldName": "city",
           "propertyChain": ["http://www.example.com/demo#city"],
           "valueFilter": "$this = \\"London\\""
         }
      ],
      "documentFilter": "bound(?city)"
    }
  ''' .
}

The :beta entity is not synchronized because it has no value for city.

To handle such cases, you can modify the connector configuration to specify a default value for city:

...
         {
           "fieldName": "city",
           "propertyChain": ["http://www.example.com/demo#city"],
           "defaultValue": "London"
         }
...
}

The default value is used for the entity :beta as it has no value for city in the repository. As the value is “London”, the entity is synchronized.

Advanced entity filter example ¶

Sometimes, data represented in RDF does not map directly to a non-RDF format. For example, if you have news articles and they can be tagged with different concepts such as locations, persons, and events, one possible way to model this is a single property :taggedWith. Consider the following RDF data:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix example2: <http://www.example.com/demo2#> .

example2:Berlin
    rdf:type example2:Location ;
    rdfs:label "Berlin" .

example2:Mozart
    rdf:type example2:Person ;
    rdfs:label "Wolfgang Amadeus Mozart" .

example2:Einstein
    rdf:type example2:Person ;
    rdfs:label "Albert Einstein" .

example2:Cannes-FF
    rdf:type example2:Event ;
    rdfs:label "Cannes Film Festival" .

example2:Article1
    rdf:type example2:Article ;
    rdfs:comment "An article about a film about Einstein's life while he was a professor in Berlin." ;
    example2:taggedWith example2:Berlin ;
    example2:taggedWith example2:Einstein ;
    example2:taggedWith example2:Cannes-FF .

example2:Article2
    rdf:type example2:Article ;
    rdfs:comment "An article about Berlin." ;
    example2:taggedWith example2:Berlin .

example2:Article3
    rdf:type example2:Article ;
    rdfs:comment "An article about Mozart's life." ;
    example2:taggedWith example2:Mozart .

example2:Article4
    rdf:type example2:Article ;
    rdfs:comment "An article about classical music in Berlin." ;
    example2:taggedWith example2:Berlin ;
    example2:taggedWith example2:Mozart .

example2:Article5
    rdf:type example2:Article ;
    rdfs:comment "A boring article that has no tags." .

example2:Article6
    rdf:type example2:Article ;
    rdfs:comment "An article about the Cannes Film Festival in 2013." ;
    example2:taggedWith example2:Cannes-FF .

Assume you want to map this data to Elasticsearch, so that the property example2:taggedWith x is mapped to separate fields taggedWithPerson and taggedWithLocation, according to the type of x (where we are not interested in Events). You can map taggedWith twice to different fields and then use an entity filter to get the desired values:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

INSERT DATA {
  elastic-index:my_index elastic:createConnector '''
    {
      "elasticsearchNode": "localhost:9200",
      "types": ["http://www.example.com/demo2#Article"],
      "fields": [
         {
            "fieldName": "comment",
            "propertyChain": ["http://www.w3.org/2000/01/rdf-schema#comment"]
         },
         {
           "fieldName": "taggedWithPerson",
           "propertyChain": ["http://www.example.com/demo2#taggedWith"],
           "valueFilter": "$this -> type = <http://www.example.com/demo2#Person>"
         },
         {
           "fieldName": "taggedWithLocation",
           "propertyChain": ["http://www.example.com/demo2#taggedWith"],
           "valueFilter": "$this -> type = <http://www.example.com/demo2#Location>"
         }
      ]
    }
  ''' .
}

Note

type is the short way to write <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>.

The six articles in the RDF data above will be mapped as such:

Article IRI	Value in taggedWithPerson	Value in taggedWithLocation	Explanation
`:Article1`	`:Einstein`	`:Berlin`	`:taggedWith` has the values `:Einstein`, `:Berlin` and `:Cannes-FF`. The filter leaves only the correct values in the respective fields. The value `:Cannes-FF` is ignored because it does not match the filter.
`:Article2`		`:Berlin`	`:taggedWith` has the value `:Berlin`. After the filter is applied, only `taggedWithLocation` is populated.
`:Article3`	`:Mozart`		`:taggedWith` has the value `:Mozart`. After the filter is applied, only `taggedWithPerson` is populated.
`:Article4`	`:Mozart`	`:Berlin`	`:taggedWith` has the values `:Berlin` and `:Mozart`. The filter leaves only the correct values in the respective fields.
`:Article5`			`:taggedWith` has no values. The filter is not relevant.
`:Article6`			`:taggedWith` has the value `:Cannes-FF`. The filter removes it because it does not match.

You can check this by issuing a faceted search for taggedWithLocation and taggedWithPerson:

PREFIX elastic: <http://www.ontotext.com/connectors/elasticsearch#>
PREFIX elastic-index: <http://www.ontotext.com/connectors/elasticsearch/instance#>

SELECT ?facetName ?facetValue ?facetCount {
  ?search a elastic-index:my_index ;
      elastic:facetFields "taggedWithLocation,taggedWithPerson" ;
      elastic:facets [
        elastic:facetName ?facetName ;
        elastic:facetValue ?facetValue ;
        elastic:facetCount ?facetCount
      ]
}

If the filter was applied, you should get only :Berlin for taggedWithLocation and only :Einstein and :Mozart for taggedWithPerson:

facetName	facetValue	facetCount
taggedWithLocation	`http://www.example.com/demo2#Berlin`	3
taggedWithPerson	`http://www.example.com/demo2#Mozart`	2
taggedWithPerson	`http://www.example.com/demo2#Einstein`	1

Migrating filters from GraphDB 9.x ¶

If you used entity filters in the connectors in GraphDB 9.x (or older) with the entityFilter option, you need to rewrite them using one of the current filter types.

In general, most older connector filters can be easily rewritten using the per-field value filter and top-level document filter.

Rule of thumb:

If you want to remove individual values, i.e., if the operand is not BOUND() –> rewrite with per-field value filter.
If you want to remove entire documents, i.e., if the operand is BOUND() –> rewrite with top-level document filter.

So if we take the example:

?location = <urn:Europe> AND BOUND(?location) AND ?type IN (<urn:Foo>, <urn:Bar>)

It needs to be rewritten like this:

Per-field rule on field location: $this = <urn:Europe>
Per-field rule on field type: $this IN (<urn:Foo>, <urn:Bar>)
Top-level document filter: BOUND(?location)

Overview of connector predicates ¶

The following diagram shows a summary of all predicates that can administer (create, drop, check the status of) connector instances or issue queries and retrieve results. It can be used as a quick reference of what a particular predicate needs to be attached to. For example, to retrieve entities, you need to use :entities on a search instance, and to retrieve snippets, you need to use :snippets on an entity.

Variables that are bound as a result of a query are shown in green, blank helper nodes are shown in blue, literals in red, and IRIs in orange. The predicates are represented by labeled arrows.

$scale 0.85 left to right direction skinparam activity { BackgroundColor<<BNode>> #D1E0FF BackgroundColor<<Var>> #D1FFD1 BackgroundColor<<IRI>> #FFCC80 BackgroundColor #FFE3E3 } partition "Instance level" { "instance IRI" <<IRI>> -->[:createConnector] "JSON params" "instance IRI" -->[:dropConnector] "blank node\n or dummy value" "instance IRI" -->[:repairConnector] "blank node\n or dummy value" "instance IRI" -->[:connectorStatus] "?status" <<Var>> "_:search" <<BNode>> -->[rdf:type] "instance IRI" } partition "Search level: query and options" { "_:search" -->[:query] "query value" "_:search" -->[:limit] "limit value" "_:search" -->[:offset] "offset value" "_:search" -->[:orderBy] "order by expression" "_:search" -->[:facetFields] "field name list" "_:search" -->[:snippetSize] "snippet size value" "_:search" -->[:snippetSpanOpen] "string" "_:search" -->[:snippetSpanClose] "string" } partition "Search level: results" "_:search" -->[:entities] "?entity" <<Var>> "_:search" -->[:totalHits] "?totalHits" <<Var>> "_:search" -->[:facets] "_:facet" <<BNode>> "_:search" -->[:aggregations] "_:aggregation" <<BNode>> } partition "Entity level" { "?entity" -->[:score] "?score" <<Var>> "?entity" -->[:snippets] "_:snippet" <<BNode>> } partition "Snippet level" { "_:snippet" -->[:snippetField] "?snippetField" <<Var>> "_:snippet" -->[:snippetInnerField] "?snippetInnerField" <<Var>> "_:snippet" -->[:snippetText] "?snippetText" <<Var>> } partition "Facet level" { "_:facet" -->[:facetName] "?facetName" <<Var>> "_:facet" -->[:facetValue] "?facetValue" <<Var>> "_:facet" -->[:facetCount] "?facetCount" <<Var>> } partition "Aggregation level" { "_:aggregation" -->[:name] "?aggrName" <<Var>> "_:aggregation" -->[:key] "?aggrKey" <<Var>> "_:aggregation" -->[:count] "?aggrCount" <<Var>> "_:aggregation" -->[:from] "?aggrFrom" <<Var>> "_:aggregation" -->[:to] "?aggrTo" <<Var>> "_:aggregation" -->[:min] "?aggrMin" <<Var>> "_:aggregation" -->[:max] "?aggrMax" <<Var>> "_:aggregation" -->[:sum] "?aggrSum" <<Var>> "_:aggregation" -->[:avg] "?aggrAvg" <<Var>> "_:aggregation" -->[:sum_of_squares] "?aggrSumSq" <<Var>> "_:aggregation" -->[:std_deviation] "?aggrStdDev" <<Var>> "_:aggregation" -->[:variance] "?aggrVar" <<Var>> "_:aggregation" -->[:parent] "?aggrParent" <<Var>> "_:aggregation" -->[:level] "?aggrLevel" <<Var>> "_:aggregation" -->[:levelName] "?aggrLevelName" <<Var>> }$

Caveats ¶

Order of control ¶

Even though SPARQL per se is not sensitive to the order of triple patterns, the Elasticsearch GraphDB Connector expects to receive certain predicates before others so that queries can be executed properly. In particular, predicates that specify the query or query options must come before any predicates that fetch results.

The diagram in Overview of connector predicates provides a quick overview of the predicates.

Upgrading from previous versions ¶

Migrating from GraphDB 9.x ¶

GraphDB 10.0 introduces major changes to the filtering mechanism of the connectors. Existing connector instances will not be usable and attempting to use them for queries or updates will throw an error.

If your GraphDB 9.x (or older) connector definitions do not include an entity filter, you can simply repair them.

If your GraphDB 9.x (or older) connector definitions do include an entity filter with the entityFilter option, you need to rewrite the filter with one of the current filter types:

Save your existing connector definition.
Drop the connector instance.
In general, most older connector filters can be easily rewritten using the per-field value filter and top-level document filter. Rewrite the filters as follows.
Rule of thumb:
- If you want to remove individual values, i.e., if the operand is not BOUND() –> rewrite with per-field value filter.
- If you want to remove entire documents, i.e., if the operand is BOUND() –> rewrite with top-level document filter.
So if we take the example:
?location = <urn:Europe> AND BOUND(?location) AND ?type IN (<urn:Foo>, <urn:Bar>)
It needs to be rewritten like this:
- Per-field rule on field location: $this = <urn:Europe>
- Per-field rule on field type: $this IN (<urn:Foo>, <urn:Bar>)
- Top-level document filter: BOUND(?location)
Recreate the connector instance using the new definition.