Full-text search¶

What’s in this document?

Full-text search using the GraphDB connectors
Simple full-text search index

Full-text search (FTS) indexing enables very fast queries over textual data. Typically, FTS is used to retrieve data that represents text written in a human language such as English, Spanish, or French.

GraphDB supports various mechanisms for performing full-text search depending on the use case and the needs of a given project.

Full-text search using the GraphDB connectors ¶

GraphDB connectors can index, search, and retrieve entire documents composed of a set of RDF statements. To do this, those connectors need a predefined data model that describes how every indexed document is constructed from a template of RDF statements. Queries can search in one or more document fields defined by the data model, and results return document IDs to show which documents were found.

See more about the full-text search with the GraphDB connectors, as well as the Lucene, Solr, Elasticsearch, and OpenSearch connectors.

Simple full-text search index ¶

GraphDB 10.1 introduced a simple FTS index that covers some basic FTS use cases. This index contains literals and IRIs:

There is no data model, so it is easy to set up.
Queries search in literals and IRIs.
Results return the matching literals and IRIs.

How the search works ¶

In general, searching is performed via SPARQL using a pattern like this:

?value onto:fts (query index limit)

There are three search arguments:

The query: string or language-tagged string, required
The index to search: string, optional
The limit of the search: integer, optional

The matching values will be returned as bindings of the provided variable in the triple pattern’s subject position (?value in the model above).

When no index is supplied as a parameter, GraphDB FTS will use the following:

If the query is a plain string without a language tag, then the index will be the configured index for string literals (via the Enable full-text search (FTS) index repository configuration parameter).
If the query is a language-tagged string, then the language tag will be used to determine the index name.

Note

When an index is supplied as a parameter, the language tag of the query string will be ignored.

When only the query is provided (the only required argument), it’s best to provide it directly without constructing an RDF list. So, the pattern can be simplified to:

?value onto:fts query

Some examples:

("query" "en" 10): Search for “query” in the “en” index and limit results to 10.
("query" 15): Search for “query” in the index configured via fts-string-literals-index and limit results to 15.
("query"@de 20): Search for “query” in the “de” index and limit results to 20.
("query"@de-CH 20): Search for “query” in the “de” index and limit results to 20. Note that only the language part of the tag de-CH determines the index.
("query" "fr"): Search for “query” in the “fr” index and do not apply a limit.
"query"@fr: Search for “query” in the “fr” index and do not apply a limit — when a sole argument is provided, it does not need to be inside an RDF list.

Query syntax ¶

The queries are parsed using Lucene’s StandardQueryParser class.

A query consists of clauses, field specifications, grouping and boolean operators, and interval functions.

Note

Keep in mind these details in particular:

Field specifications: There are no other field names but the default field name so there is no valid case where the user must specify a field name.
Escaping in SPARQL: All query syntax examples specify the expected Lucene query string. If you provide these strings as SPARQL literals, you may need to escape " and \ as required by SPARQL.

Warning

FTS performs no text analysis on some of the specialized query types. Lexical analysis is only run on complete terms (a term or phrase query). Query types containing incomplete terms such as prefix, wildcard, regex, or fuzzy queries skip the analysis stage and get directly added to the query tree. The only transformation applied to partial query terms is lowercasing.

This may lead to surprising results if you expect stemming or lemmatization. For example, searching for “resti*” and expecting to find “resting” will not work when using the English analyzer since the word “resting” was analyzed and indexed as “rest”.

Basic clauses ¶

A query must contain one or more clauses. A clause can be a literal term, a phrase, a wildcard expression, or any supported expression.

The following are some examples of simple one-clause queries:

Query	Description
`test`	Selects documents containing the word “test” (term clause).
`"test equipment"`	Phrase search; selects documents containing the phrase “test equipment” (phrase clause).
`"test failure"~4`	Proximity search; selects documents containing the words “test” and “failure” within 4 words (positions) from each other. The provided “proximity” is technically translated into the “edit distance” (the maximum number of atomic word-moving operations required to transform the document’s phrase into the query phrase).
`tes*`	Prefix wildcard matching; selects documents containing words starting with “tes”, such as: “test”, “testing” or “testable”.
`/(p\|n).st/`	Documents containing word roots matching the provided regular expression, such as “post” or “nest”.
`nest~2`	Fuzzy term matching; documents containing words within 2-edits distance (2 additions, removals, or replacements of a letter) from “nest”, such as “test”, “net”, or “rests”.

Boolean operators and grouping ¶

You can combine clauses using boolean AND, OR, and NOT operators to form more complex expressions, for example:

Query	Description
`test AND results`	Selects documents containing both the word “test” and the word “results”.
`test OR suite OR results`	Selects documents with at least one of “test”, “suite”, or “results”.
`test AND NOT complete`	Selects documents containing “test” and not containing “complete”.
`test AND (pass* OR fail*)`	Grouping; use parentheses to specify the precedence of terms in a boolean clause. Query will match documents containing “test” and a word starting with “pass” or “fail”.
`(pass fail skip)`	Shorthand notation; documents containing at least one of “pass”, “fail”, or “skip”.

Note

The boolean operators must be written in all caps, otherwise they are parsed as regular terms.

Range operators ¶

To search for ranges of textual or numeric values, use square or curly brackets, for example:

Query	Description
`[Jones TO Smith]`	Inclusive range; selects documents that contain any value between “Jones” and “Smith”, including boundaries.
`{Jones TO Smith}`	Exclusive range; selects documents that contain any value between “Jones” and “Smith”, excluding boundaries.
`{Jones TO *]`	One-sided range; selects documents that contain any value larger than (that is, sorted after) “Jones”.

Note

These will work intuitively only with the “iri” index, e.g., "[http://www.w3.org/2000/01/rdf-schema#comment TO http://www.w3.org/2000/01/rdf-schema#range]" will retrieve all IRIs that are alphabetically ordered between http://www.w3.org/2000/01/rdf-schema#comment and http://www.w3.org/2000/01/rdf-schema#range inclusive. If used with any of the other indexes, they will return matches but it will not be intuitive what they match.

Term boosting ¶

Terms, quoted terms, term range expressions, and grouped clauses can have a floating-point weight boost applied to them to increase their score relative to other clauses. For example:

Query	Description
`jones^2 OR smith^0.5`	Prioritize documents with “jones” term over matches on the “smith” term.
`(a OR b NOT c)^2.5 OR d`	Apply the boost to a sub-query.

Special character escaping ¶

Most search terms can be put in double quotes, making special character escaping not necessary. If the search term contains the quote character (or cannot be quoted for some reason), any character can be quoted with a backslash. For example:

Query	Description
`\:\(quoted\+term\)\:`	A single search term `(quoted+term):` with escape sequences. An alternative quoted form would be simpler: `":(quoted+term):"`.

Minimum-should-match constraint for boolean disjunction groups ¶

A minimum-should-match operator can be applied to a disjunction boolean query (a query with only “OR”-subclauses) to force the query to match documents with at least the provided number of these subclauses. For example:

Query	Description
`(blue crab fish)@2`	Matches all documents with at least two terms from the set [blue, crab, fish] in any order.
`((yellow OR blue) crab fish)@2`	Sub-clauses of a boolean query can themselves be complex queries; here the min-should-match selects documents that match at least two of the provided three sub-clauses.

Interval function clauses ¶

Interval functions are a powerful tool for expressing search needs in terms of one or more contiguous fragments of text and their relationship to one another. All interval clauses start with the fn: prefix. For example:

Query	Description
`fn:ordered(quick brown fox)`	Matches all documents with at least one ordered sequence of “quick”, “brown”, and “fox” terms.
`fn:maxwidth(5 fn:atLeast(2 quick brown fox))`	Matches all documents where at least two of the three terms “quick”, “brown”, and “fox” occur within five positions of each other.

Common use cases ¶

Before performing a full-text search you must enable the FTS index. This can be done at repository creation by setting the Enable full-text search (FTS) index to true, as well as at a later stage if you want to edit the repository configuration.

Single language ¶

Let’s say that your data is in a single supported language and you want to perform full-text search in order to find literals that match. Literals may or may not have a language tag; for example:

“This is a literal in English without a language tag”
“This is another literal in English with a language tag for the language only”@en
“This is yet another literal tagged for English in Canada”@en-CA

To configure the sample search:

Create a repository.
In its configuration menu, enable the "en" index by setting FTS indexes to build to “en”.
Literals with no language tag must also go into the "en" index, so we set FTS index for xsd:string literals to “en”.

Important

After each change applied to any FTS parameters you must restart the repository.

To try this configuration out, insert some tagged sample data by executing the following update request in the Workbench SPARQL editor:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <urn:d1> rdfs:label "This is a literal in English without a language tag",
            "This is another literal in English with a language tag for the language only"@en,
            "This is yet another literal tagged for English in Canada"@en-CA,
            "Let's pretend this literal isn't in English by tagging it as German"@de
}

So if we run the following example query against it:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # Note that this exploits the fact that we haven't enabled the default index,
    # so the index for indexing string literals (en) is the default query index
    ?value onto:fts "english literal"
}

Or this one:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The language tag of the query literal supplies the index to query
    ?value onto:fts "english literal"@en
}

Or this one:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The query string and the index to query are supplied as two separate values
    # inside an RDF list
    ?value onto:fts ("english literal" "en")
}

They will all return the three literals that are not tagged as German:

Multiple languages ¶

The strings in the following bulleted list are in English or German and we want to perform full-text search in order to find literals that match. Literals with no language tag are in one of the desired languages: English.

“This is a literal in English without a language tag”
“This is another literal in English with a language tag for the language only”@en
“This is yet another literal tagged for English in Canada”@en-CA
“Das ist ein schönes deutsches Literal”@de
“Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz”@de-CH

To configure a search through this set of literals:

Create a repository.
In its configuration menu, enable the "en" and "de" indexes by setting FTS indexes to build to “en, de”. You can extend this with additional languages by adding them to the list.
Literals with no language tag must also go into the "en" index, so set FTS index for xsd:string literals to “en”.

The following update request inserts the data shown above into the repository:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <urn:d2> rdfs:label "This is a literal in English without a language tag",
            "This is another literal in English with a language tag for the language only"@en,
            "This is yet another literal tagged for English in Canada"@en-CA,
            "Das ist ein schönes deutsches Literal"@de,
            "Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz"@de-CH
}

Searching in English is exactly the same as in the first use case above. To search the additional German index, we specify it like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The language tag of the query literal supplies the index to query
    ?value onto:fts "deutsch literal"@de
}

Or this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The query string and the index to query are supplied as two separate values
    # inside an RDF list
    ?value onto:fts ("deutsch literal" "de")
}

Both of these queries will return the two German literals.

Note

Keep in mind that if you have other data in the repository, it may affect the results.

Ignore untagged literals ¶

In this case, our data is in one or more supported languages (English and German) and we want to perform full-text search in order to find literals that match. Literals with no language tag should not be treated as any of those languages and need not be searched. Data may look like any of these:

“This is another literal in English with a language tag for the language only”@en
“This is yet another literal tagged for English in Canada”@en-CA
“Das ist ein schönes deutsches Literal”@de
“Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz”@de-CH
“This is a literal in English without a language tag” (this must not be indexed)

To configure the sample search:

Create a repository.
In its configuration menu, enable the "en" and "de" indexes by setting FTS indexes to build to “en, de”. You can extend this with additional languages by adding them to the list.
Literals with no language tag need to not be indexed, so we set FTS index for xsd:string literals to “none”.

The following update request inserts the data shown above into the repository:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <urn:d3> rdfs:label "This is another literal in English with a language tag for the language only"@en,
            "This is yet another literal tagged for English in Canada"@en-CA,
            "Das ist ein schönes deutsches Literal"@de,
            "Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz"@de-CH,
            "This is a literal in English without a language tag"
}

Searching in any of the languages requires the query to to specify the index. There is no default search index because the FTS index for xsd:string literals is set to “none”:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The language tag of the query literal supplies the index to query
    ?value onto:fts "english literal"@en
}

Or this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The query string and the index to query are supplied as two separate values
    # inside an RDF list
    ?value onto:fts ("english literal" "en")
}

Both queries will return the two literals that are tagged for English but not the untagged one.

Untagged literals not treated as any language but still searchable ¶

Here, our data is in one or more supported languages (English and German) and we want to perform full-text search in order to find literals that match.

Literals with no language tag should not be treated as any of those languages but should provide language-agnostic full-text search. These literals may be data like UUIDs or anything else that has a textual representation that we may want to search. Data may look like any of these:

“This is another literal in English with a language tag for the language only”@en
“This is yet another literal tagged for English in Canada”@en-CA
“Das ist ein schönes deutsches Literal”@de
“Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz”@de-CH
“96ac1c60-7997-45a3-8dfe-b57b24c1cb62” (this will be indexed separately)

To configure the sample search:

Create a repository.
In its configuration menu, enable the "default" index, as well as the indexes "en" and "de", by setting FTS indexes to build to “default, en, de”. You can extend this with additional languages by adding them to the list.
Literals with no language tag must be indexed in a language-agnostic manner, so we set FTS index for xsd:string literals to “default” (which is also the default value for that repository configuration property).

Important

The values of FTS indexes to build must contain the values for FTS index for xsd:string literals and FTS index for full-text indexing of IRIs, unless those are set to “none”.

The following update request inserts the data shown above into the repository:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <urn:d4> rdfs:label "This is another literal in English with a language tag for the language only"@en,
            "This is yet another literal tagged for English in Canada"@en-CA,
            "Das ist ein schönes deutsches Literal"@de,
            "Dies hier ist ebenso ein hübsches deutsches Literal, aber aus der Schweiz"@de-CH,
            "96ac1c60-7997-45a3-8dfe-b57b24c1cb62"
}

Note

The "default" index provides language-agnostic search.

Searching in any of the languages is like in the third example related to ignoring untagged literals: the query must specify the index to search.

Searching in the untagged literals can be done like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "b57*"
}

Or like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The language tag of the query literal supplies the index to query
    ?value onto:fts "b57*"@default
}

Or like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # The query string and the index to query are supplied as two separate values
    # inside an RDF list
    ?value onto:fts ("b57*" "default")
}

All of these queries will return the single untagged literal where "b57*" was matched to one of the hyphenated components.

Treat IRIs as keywords and search them ¶

Sometimes, regardless of our need to search literals, we also want to search within IRIs and treat them as keywords so that the entire IRI is considered to be a single searchable token. These can be any IRIs, such as:

<http://www.w3.org/2000/01/rdf-schema#domain>
<http://example.com/data/john>
<http://example.com/data/mary>
<http://example.com/data/william>

To configure the sample search:

Create a repository.
In its configuration menu, enable a special index called "iri" by adding it to the FTS indexes to build property. For example, if we also want English literals to be indexed, we will set FTS indexes to build to “en, iri”.
Set FTS index for xsd:string literals to “en” so that literals with no language tag will go to the “en” index.

The following update request inserts data that uses the IRIs above:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <http://example.com/data/john> rdfs:label "John" .
    <http://example.com/data/mary> rdfs:label "Mary" .
    <http://example.com/data/william> rdfs:label "William" .
}

To search the IRIs, query the "iri" index like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # Finds all IRIs that start with "http://example.com/"
    ?value onto:fts "http://example.com/*"@iri
}

Or like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # Finds all IRIs that start with "http://example.com/"
    ?value onto:fts ("http://example.com/*" "iri")
}

Both of these will return the http://example.com/xxx IRIs from the sample data.

When the entire search string is a single keyword, which is the case for the "iri" index, you can also use range searches to find IRIs that fall between two IRIs when sorted:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # Finds all IRIs that sort between http://example.com/data/kelly and http://example.com/data/william,
    # including the boundaries
    ?value onto:fts "[http://example.com/data/kelly TO http://example.com/data/william]"@iri
}

Or like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    # Finds all IRIs that sort between http://example.com/data/kelly and http://example.com/data/william,
    # including the boundaries
    ?value onto:fts "[http://example.com/data/kelly TO http://example.com/data/william]"@iri
}

Both of these queries should return http://example.com/data/mary and http://example.com/data/william.

Indexing ¶

In this scenario, regardless of our need to search literals, we also need to search within IRIs, treating them as regular text in which the IRI is split into multiple searchable tokens. These are typically IRIs that are readable and are composed of words:

<http://example.com/data/john>
<http://example.com/data/mary>
<http://example.com/data/william>

To configure the sample search:

Create a repository.
In its configuration menu, enable the index for the language we want by adding it to FTS indexes to build — for English, we set FTS indexes to build to “en”.
The value of FTS index for xsd:string literals must also be set to “en”.
We also need IRIs to be indexed for full-text search in the language we enabled, so we set FTS index for full-text indexing of IRIs to “en”.

The following update request inserts the data shown above into the repository:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

INSERT DATA {
    <http://example.com/data/john> rdfs:label "John" .
    <http://example.com/data/mary> rdfs:label "Mary" .
    <http://example.com/data/william> rdfs:label "William" .
}

IRIs are then searchable in the "en" index just like literals:

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "john"@en
}

Or like this:

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts ("john" "en")
}

Both of these queries will return the IRI http://example.com/john, as well as the literal "John".

Star Wars dataset examples ¶

These examples use the Star Wars dataset from starwars-data.ttl.

Create a repository with the following settings:

Enable full-text search (FTS) index: true
FTS indexes to build: en, de, fr, es, it
FTS index for xsd:string literals: en
FTS index for full-text indexing of IRIs: none (but “en” would also make sense for this dataset)

_images/fts-starwars-examples-config.png

Let’s look at some example queries.

All literals where “luke” and “vader” are near each other ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts '"luke vader"~5'
}

This query returns a single literal.

Note

The query above searches in the "en" index since the default index is disabled and we requested xsd:string literals to go to the "en" index.
The query uses single quotes for the query literal to avoid escaping the double quotes that are part of the full-text search query.

All literals containing “skywalker” but not “luke”¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "skywalker -luke"
}

This query returns several results, some of which are Luke’s grandmother Shmi Skywalker and Luke’s father Anakin Skywalker before he became Darth Vader.

All literals corresponding to a simple FTS query ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "striking jedis"
}

This query returns many results, some of which are “The Empire Strikes Back” and “Return of the Jedi”. This illustrates how full-text search tuned to a specific language (in this case English) is able to match “striking” to “strikes” and “jedis” to “jedi”.

Note that the query written like that does not need all tokens to be in the matched result. In other words, the query is equivalent to “striking OR jedis”.

All literals corresponding to a simple FTS query in German ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "das beste"@de
}

This query returns matches such as “Ahmed Best”, “Oscar für den besten Film” and “Oscar für die beste Regie”, illustrating the ability of FTS to match different word forms in German.

All literals corresponding to a simple FTS query in French ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "oscar acteur"@fr
}

This query returns matches like “Oscar de la meilleure actrice” and “Oscar du meilleur acteur”, illustrating the ability of FTS to match different word forms in French.

All literals corresponding to a simple FTS query in Italian ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "migliori"@it
}

This query returns matches such as “Oscar al miglior film”, “Oscar ai migliori costumi” and “Oscar alla migliore scenografia”, illustrating the ability of FTS to match different word forms in Italian.

All literals corresponding to a simple FTS query in Spanish ¶

PREFIX onto: <http://www.ontotext.com/>
select * {
    ?value onto:fts "peliculas"@es
}

This query returns matches like “Película del 2005” and “personaje de ficción el las películas de Star Wars”, illustrating the ability of FTS to match different word forms in Spanish but also the ability to ignore diacritics when searching.