General full-text search with the Connectors¶
What’s in this document?
The GraphDB Connectors offer an excellent solution for indexing data with a well-known schema — for example, index documents that have type A, where each document has a field F1 that can be reached by following the property chain composed of property IRIs P1 and P2.
The features described below add a more general full-text search (FTS) ability to the connectors. You can use them individually or combined as desired to meet the specific needs of the use case.
Note
See more about GraphDB’s FTS capabilities in the Full-text search section of this documentation.
The connector features described below are useful when defining a connector for general full-text search.
Indexing with wildcard literals¶
This feature lets you index literals without specifying the IRI of the predicate that leads to the literal. Use $literal as the last element of the property chain.
See more about wildcard literals in the Lucene connector, the Solr connector, the Elasticsearch connector, and the OpenSearch connector sections.
Deriving field names from a predicate URI¶
This feature lets you use dynamic field names derived from the IRI of the last predicate in the property chain.
See more about field name transformations in the Lucene connector, the Solr connector, the Elasticsearch connector, and the OpenSearch connector sections.
Any type or untyped indexing¶
Specify $any or $untyped as the sole type to index all entities that have at least one RDF type, or all entities regardless of whether they have any RDF type.
See more about types in the Lucene connector, the Solr connector, the Elasticsearch connector, and the OpenSearch connector sections.
Examples¶
All examples use the Star Wars RDF dataset. Download starwars-data.ttl and import it into a fresh repository before trying any of these examples.
The example connector definitions use the Lucene connector but can be easily adapted to Solr, Elasticsearch and OpenSearch by changing lucene in the prefix definitions to solr, elasticsearch or opensearch, and adding any additional parameters required by the respective connector such as elasticsearchNode.
Indexing all literals¶
To index all literals in the repository regardless of where they are attached in the graph, you can combine wildcard literal indexing and untyped indexing. Create a connector with an update request like the following:
PREFIX con: <http://www.ontotext.com/connectors/lucene#>
PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
INSERT DATA {
con-inst:starwars_fts con:createConnector '''
{
"fields": [
{
"fieldName": "fts",
"propertyChain": [
"$literal"
],
"facet": false
}
],
"languages": [
""
],
"types": [
"$untyped"
]
}
''' .
}
The connector defines a single field, fts, that will index all literals regardless of their predicate: $literal as the last element of the property chain. The connector has no type expectations on the entities that lead to those literals and will index any entity regardless of whether it has an RDF type: $untyped in the types parameter.
Since the Star Wars dataset contains literals in many different languages, we restrict the index definition further by specifying "" (the empty language, which means any literal without a language tag) using the languages option.
We can now search in this connector as usual, for example for the FTS query “luke skywalker”:
# Full-text search for "skywalker"
PREFIX con: <http://www.ontotext.com/connectors/lucene#>
PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?entity ?label {
[] a con-inst:starwars_fts ;
con:query "skywalker" ;
con:entities ?entity .
?entity rdfs:label ?label
FILTER(lang(?label) = "")
}
We get many different results belonging to different types; below are the first ten results:
?entity |
?label |
|---|---|
|
Shmi Skywalker |
|
Luke Skywalker |
|
Padmé Amidala |
|
Tatooine |
|
Obi-Wan Kenobi |
|
Anakin Skywalker |
|
C-3PO |
|
Darth Vader |
|
R2-D2 |
|
Wedge Antilles |
Indexing all literals in distinct fields¶
The above example indexes all literals into a single field, which is convenient for very rough full-text search. You can fine-tune it by using field names derived from the predicate. In this example, we added "fieldNameTransform": "predicate.localName" so we will get a field for every predicate whose object literal is indexed, and the field name will be derived from the local name of the predicate:
PREFIX con: <http://www.ontotext.com/connectors/lucene#>
PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
INSERT DATA {
con-inst:starwars_fts2 con:createConnector '''
{
"fields": [
{
"fieldName": "fts",
"fieldNameTransform": "predicate.localName",
"propertyChain": [
"$literal"
],
"facet": false
}
],
"languages": [
""
],
"types": [
"$untyped"
]
}
''' .
}
We can use this connector to do general full-text searches, but also more precise ones, such as a query only of entity labels (the field label is the result of taking the local name of <http://www.w3.org/2000/01/rdf-schema#label> at indexing time):
# Full-text search for "skywalker" in the field "label"
PREFIX con: <http://www.ontotext.com/connectors/lucene#>
PREFIX con-inst: <http://www.ontotext.com/connectors/lucene/instance#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?entity ?label {
[] a con-inst:starwars_fts2 ;
con:query "label:skywalker" ;
con:entities ?entity .
?entity rdfs:label ?label
FILTER(lang(?label) = "")
}
We get only three results back — the people that have “Skywalker” in their name:
?entity |
?label |
|---|---|
|
Shmi Skywalker |
|
Luke Skywalker |
|
Anakin Skywalker |