Semantic similarity searches

What can the similarity plugin do for me?

The similarity plugin enables exploration and searching based on semantic similarity of RDF resources.

Statistical semantics queries can be valuable for:

• Taking text that is encoded as a literal in the database and finding the closest texts based on a vector space model. (The use of the vector space model is explained more below in How does the similarity plugin work?)

• Clustering stories from a news feed based on the events that they cover.

What does the similarity plugin do?

Humans determine the similarity between texts based on the similarity of the content words and their abstract meaning. Documents containing similar words are semantically related, and words frequently co-occurring are also considered close.

The similarity plugin supports document and term searches. A document is a literal or an aggregation of multiple literals, and a term is a word from a document.

There are four types of similarity searches:

• Term to term — supply a term to the plugin and it returns the closest semantically related terms

• Term to document — supply a term to the plugin and it returns the most representative documents for that term

• Document to term — supply a document to the plugin and it returns the most representative terms for that document

• Document to document — supply a document to the plugin and it returns a list of the closest related texts

How does the similarity plugin work?

The similarity plugin integrates the semantic vectors library and the underlying Random Indexing algorithm. The algorithm uses a tokenizer to translate documents to sequences of words (terms) and a vector space model to represent their abstract meaning. A distinctive feature of the algorithm is the dimensionality reduction approach based on Random Projection, where the initial vector state is generated randomly. With the indexing of each document, the term vectors are adjusted based on the contextual words. This approach makes the algorithm highly scalable for very large text corpora of documents, and research papers have proven that its efficiency is comparable to more sound dimensionality reduction algorithms like singular value decomposition.

After loading some news article data and creating an index, we’re going to look at several kinds of searches that the similarity plugin can do.

Download sample data

The following examples use factforge.net news data and metadata from January to April 2018. These are represented as content, creationDate, and mentionsEntity triples.

1. Go to the SPARQL editor at http://factforge.net/sparql and run the following query to pull out the news story data. As you can see from the query’s CONSTRUCT clause, for each news story the resulting dataset will have one or more mentioned entities, the news story content, and the article’s creation date:

   PREFIX pubo: <http://ontology.ontotext.com/publishing#>
   PREFIX pub: <http://ontology.ontotext.com/taxonomy/>
   PREFIX dbr: <http://dbpedia.org/resource/>
   PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
   PREFIX ff-map: <http://factforge.net/ff2016-mapping/>
   
   CONSTRUCT {
           ?document ff-map:mentionsEntity ?entity .
           ?document pubo:content ?content .
           ?document pubo:creationDate ?date .
   } WHERE {
           ?document a pubo:Document .
           ?document ff-map:mentionsEntity ?entity .
           ?document pubo:content ?content .
           ?document pubo:creationDate ?date .
       FILTER (?p NOT IN (pubo:containsMention, pubo:hasFeature, pubo:hasImage))
       FILTER ( (?date > "2018-01-01"^^xsd:dateTime) && (?date < "2018-04-30"^^xsd:dateTime))
   }
   

2. Use the Download As button in the lower right of the FactForge SPARQL editor to pull down a Turtle version of the query result. It may take a few minutes to export the data to the query-result.ttl file.

3. Open your GraphDB instance and create a new repository called “news”.

4. Move the downloaded file to the <HOME>/graphdb-import folder so that it is visible in Import ‣ Server files (see how to import server files).

5. Import the query-result.ttl file into the “news” repository.

6. Go to Setup and enable the Autocomplete index for the “news” repository. It is used for autocompletion of URLs in the SPARQL editor and the View resource page.

   

Text-based similarity searches

Create text similarity index

Create an index with the following steps:

1. Go to Explore ‣ Similarity ‣ Create similarity index ‣ Create text similarity index and change the Data query to:

   PREFIX pubo: <http://ontology.ontotext.com/publishing#>
   SELECT ?documentID ?documentText
   {
       ?documentID pubo:content ?documentText .
       FILTER(isLiteral(?documentText))
   }
   

   This will index the content, where the ID of a document in the input data is the news piece’s IRI and the text is the news piece’s content. The FILTER condition shows that we only want triple objects that are literal values.

2. Name the index allNews and click the Create button in the lower right. An image of spinning arrows to the right of the new index’s entry on the Existing Indexes list will show that the index is being created and not ready yet.

3. Once the index has been created, it will appear as a new choice on the the Existing Indexes list. This is where you select an index to use when searching.

For each listed index, you can also do the following with the icons to the right of its name:

• review or copy the SPARQL query that this index was created with

• modify the search query without having to build an index

• create a new index from an existing one

• rebuild the index

• delete the index

As described above, you can have the index search for a term and then return similar terms or documents, and you can also search for documents and have it return similar terms or documents. Select your new allNews index from the Existing Indexes list to try the examples below. This will show Search type and Result type drop-downs that each let you pick “Term” or “Document”.

Use the index to search for similar terms

This example searches on the term “novichok” and requests similar terms. Returned results are sorted by their similarity score, which is shown in the second column of the results:

Note

For all searches done with the Similarity Index search form, the View SPARQL Query link above the search results shows you the SPARQL query generated by the form to create the displayed results. You can use this query and others based on it to incorporate these searches into your applications without relying on this form to do similarity searching.

Search documents for which selected term is specific

This example also searches for the term “novichok” but requests documents that are similar to that term. The result shows URLs of relevant documents and their similarity scores.

Search specific terms in selected document

We saw that documents are identified by their URLs, and you could type out an entire URL to specify one as a Search Type value, but because you enabled autocomplete earlier you usually don’t need to type out a full URL. With a Search type of Document, enter the words “used in spy” in the allNews index’s search field, and autocomplete will enter the rest of the URL https://www.theguardian.com/uk-news/2018/apr/12/novichok-used-in-spy-poisoning-chemical-weapons-watchdog-confirms-salisbury for you. Click the Show button to find the most important words in the document and you’ll see results similar to the following:

Search for closest documents

You can also search for documents that are closest to the specified search target document. Using the same input as the previous search, a search for the closest documents shows the results below. For result URLs that include title terms, we can see that many do have content related to the input document.

Note

Returned terms are sometimes stems of words such as “nerv” and “salisburi” in the example. This means that various words from the same word family may have been found.

Search similar news within a specified time range

We’re going to execute a SPARQL query that searches for news stories similar to a particular one and then we’re going to revise the query to only return stories published within two days of the input one.

1. For the initial search:

   1. In the similarity index list, click on the allNews index to search in it.

   2. Select Search type: Document and Result type: Document.

   3. In the search field, type http://www.uawire.org/merkel-and-putin-discuss-syria-and-nord-stream-2 and click Show.

   4. Click the View SPARQL Query option above the returned results. It will contain the following query:

      PREFIX : <http://www.ontotext.com/graphdb/similarity/>
      PREFIX inst: <http://www.ontotext.com/graphdb/similarity/instance/>
      PREFIX pubo: <http://ontology.ontotext.com/publishing#>
      
      SELECT ?documentID ?score {
          ?search a inst:allNews ;
              :searchDocumentID <http://www.uawire.org/merkel-and-putin-discuss-syria-and-nord-stream-2>;
              :searchParameters "";
              :documentResult ?result .
          ?result :value ?documentID ;
                  :score ?score.
      }
      

   5. Copy the query.

   6. Paste it in Workbench’s SPARQL editor.

2. Now, we can extend this search query to get only the news similar to http://www.uawire.org/merkel-and-putin-discuss-syria-and-nord-stream-2 that have been created within days of the time of creation of this one, making it more likely to be the same news. Replace the query in the SPARQL editor with this augmented version of it and execute it:

   PREFIX : <http://www.ontotext.com/graphdb/similarity/>
   PREFIX inst: <http://www.ontotext.com/graphdb/similarity/instance/>
   PREFIX pubo: <http://ontology.ontotext.com/publishing#>
   PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
   SELECT ?documentID ?score ?matchDate ?searchDate {
       BIND (<http://www.uawire.org/merkel-and-putin-discuss-syria-and-nord-stream-2> as ?searchDocumentID )
       ?search a inst:allNews ;
           :searchDocumentID ?searchDocumentID;
           :searchParameters "";
           :documentResult ?result .
       ?result :value ?documentID ;
               :score ?score.
       ?documentID pubo:creationDate ?matchDate  .
           ?searchDocumentID pubo:creationDate ?searchDate .
       FILTER (?matchDate > ?searchDate - "P2D"^^xsd:duration && ?matchDate < ?searchDate + "P2D"^^xsd:duration)
   }
   

   The last three lines of the query do the following:

   • Find the pubo:creationDate value of any documents found with the original search and store these values in the ?matchDate variable.

   • Store the pubo:creationDate value of the input document in ?searchDate.

   • Filter search results down to just those from up to two days before or two days after ?searchDate.

   We see the results below:

Term to term search

The Term to term search gets the relevant terms by period.

For this example we’ll create four indexes — one each for the news in January, February, March, and April.

Go to Create similarity index and create a new index with the following query for January:

PREFIX : <http://www.ontotext.com/graphdb/similarity/>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX similarity: <http://www.ontotext.com/graphdb/similarity/>
PREFIX similarity-index: <http://www.ontotext.com/graphdb/similarity/instance/>

SELECT ?documentID ?documentText {
    ?documentID pubo:content ?documentText .
    ?documentID pubo:creationDate ?date .
    FILTER ( (?date > "2018-01-01"^^xsd:dateTime) && (?date < "2018-01-30"^^xsd:dateTime))
    FILTER(isLiteral(?documentText))
}

Do the same for February, March, and April by changing the date range. After creating the four indexes, go to the index for each month and select Term for both Search type and Result type. Type “korea” in the search field. See how the results change over time.

Customizing index creation

Create index parameters

When you create a new index, you can select from a list of creation parameters under more options ‣ Semantic Vectors create index parameters to further configure the similarity index.

• -vectortype: Real, Complex, and Binary Semantic Vectors.

• -dimension: Dimension of semantic vector space, with a default value of 200. Recommended values are in the hundreds for real and complex, and in the thousands for binary, since binary dimensions are single bits. Smaller dimensions make both indexing and queries faster, but if the dimension is too low it will compromise the orthogonality of the element vectors and lead to poorer results. An intuition for the optimal values is given by the Johnson-Lindenstrauss lemma.

• -seedlength: Number of nonzero entries in a sparse random vector, with a default value 10 except for when vectortype is binary, in which case a default of dimension/2 is enforced. For real and complex vectors the default value is 10, but it’s a good idea to use a higher value when the vector dimension is higher than 200. The simplest thing to do is to preserve this ratio — that is, to divide the dimension by 20. (In the original implementation of random indexing, the ratio of non-zero elements was 1/3.)

• -trainingcycles: Number of training cycles used for Reflective Random Indexing.

• -termweight: Term weighting used when constructing document vectors. Values can be none, idf, logentropy, or sqrt. It’s best to use term weighting when building indexes by adding -termweight idf as a default when creating an index. It uses inverse document frequency when building the vectors. See LuceneUtils for more details.

• -minfrequency: Minimum number of times that a term must occur in order to be indexed. The default value is 0, but it would be a bad idea to use it, because that could add a lot of big numbers, weird terms, and misspelled words to the list of word vectors. The best approach would be to set it as a fraction of the total word count in the corpus — for example 40 per million as a frequency threshold. Another approach is to start with an intuitive value, a single digit number like 3 or 4, and fine tune from there.

• -maxfrequency: Maximum number of times that a term can occur before getting removed from indexes. The default value is Integer.MAX_VALUE. Again, a better approach is to calculate it as a percentage of the total word count. Otherwise, you can use the default value and add most common English words to the stop list.

• -maxnonalphabetchars: Maximum number of non-alphabet characters a term can contain in order to be indexed. The default value is Integer.MAX_VALUE. Recommended values depend on the dataset and the type of terms it contains, but setting it to 0 works pretty well for most basic cases, as it takes care of punctuation (if data has not been preprocessed), malformed terms, and weird codes and abbreviations.

• -filternumbers: true/false, whether to index numbers.

• -mintermlength: Minimum number of characters in a term.

• -indexfileformat: Format used for serializing and deserializing vectors from disk. The default value is lucene. Another option is text, which may be used when debugging to see the actual vectors, although this can be too slow with real data.

Disabled parameters

• -luceneindexpath: Currently, you are not allowed to build your own Lucene index and create vectors from it because the creation of the index and vectors is all done in one step.

• -stoplistfile: Replaced by the <http://www.ontotext.com/graphdb/similarity/stopList> predicate. Stop words are passed as a string literal as opposed to a file.

• -elementalmethod

• -docindexing

Stop words and Lucene analyzer

In the Stop words field, you can add a custom list of stop words to be passed to the Semantic Vector plugin. If left empty, the default Lucene stop words list will be used.

The Analyzer class field lets you can set a Lucene analyzer to use during Semantic Vector indexing and query time tokenization. The default is org.apache.lucene.analysis.en.EnglishAnalyzer, but it can be any from the supported list.

The Lucene connector also supports custom Analyzer implementations. This lets you create your own analyzer and add it to a classpath. The value of the Analyzer Class parameter must be a fully-qualified class name that extends the Java class org.apache.lucene.analysis.Analyzer.

Search parameters

On the search form, expanding the Search options panel lets you configure more search parameters.

• -searchtype: Most search types involve processing combinations of vectors in different ways, building a query expression, scoring candidates against these query expressions, or both. The default is a sum that builds a query by adding together weighted vectors for each of the query terms, and search using cosine similarity.

• -matchcase: If true, matching of query terms is case-sensitive; otherwise, case-insensitive. Default: false.

• -numsearchresults: Number of search results.

• -searchresultsminscore: Search results with similarity scores below this threshold will not be returned. Default: -1.

See more about Semantic Vectors Search Options.

Delete an index using a SPARQL query

To delete an index, use the following SPARQL query, substituting your index name for my_index:

PREFIX similarity-index: <http://www.ontotext.com/graphdb/similarity/instance/>
PREFIX similarity: <http://www.ontotext.com/graphdb/similarity/>

INSERT DATA {
    similarity-index:my_index similarity:deleteIndex "" .
}

Rebuild an index using a SPARQL query

To rebuild an index, use the following SPARQL query, substituting your index name for my_index:

PREFIX similarity-index: <http://www.ontotext.com/graphdb/similarity/instance/>
PREFIX similarity: <http://www.ontotext.com/graphdb/similarity/>

INSERT DATA {
    similarity-index:my_index similarity:rebuildIndex "" .
}

Search in the index during rebuild with no downtime

GraphDB lets you use the similarity index with no downtime while the database is being modified. While rebuilding the index, its last successfully built version is preserved until the new index is ready. This way, when you search in it during rebuild, the retrieved results will be from this last version. The following message will notify you of this:

The outdated image is then replaced.

Locality-sensitive hashing

Note

As locality-sensitive hashing does not guarantee the retrieval of the most similar results, this hashing is not the most suitable option if precision is essential. Hashing with the same configuration over the same data does not guarantee the same search results.

Locality-sensitive hashing can reduce search times. Without a hashing algorithm, a search consists of the following steps:

1. A search vector is generated.

2. All vectors in the store are compared to this search vector, and the most similar ones are returned as matches.

While this approach is complete and accurate, it is also time-consuming. Hashing can speed the process by reducing the number of candidates for most similar vectors. This is where Locality-sensitive hashing can help.

The Locality-sensitive hashing algorithm has two parameters that can be passed either during index creation, or as search options:

• -lsh_hashes_num: The number of n random vectors used for hashing. Default: 0

• -lsh_max_bits_diff: The m number of bits by which two hashes can differ and still be considered similar. Default: 0

The hashing workflow is as follows:

1. An n number of random orthogonal vectors are generated.

2. Each vector in store is compared to each of those vectors (checking whether their scalar product is positive or not).

3. Given this data, a hash is generated for each of the vectors in store.

A search workflow follows these steps:

1. A search vector is generated.

2. A hash is generated for this search vector by comparing it to the n number of random vectors used during the initial hashing.

3. All hashes similar to that of the searched vector are found. (A hash is considered similar when it has up to m bits difference from the original one.)

4. All vectors with such hash values are collected and compared to the generated vector in order to get the closest ones, based on the assumption that the vectors with similar hashes will be close to each other.

Note

If both parameters have the same value, then all possible hashes are considered similar and therefore no filtering is done. For optimization purposes in this scenario, the entire hashing logic has been bypassed.

If one of the parameters is specified during the index creation, then its value will be used as the default one for searching.

Depending on its configuration, the hash can perform in different ways.

A higher -lsh_hashes_num value leads to more hash buckets with fewer elements in them. Conversely, a lower number of hashes would mean fewer but bigger buckets. The n number of hashes leads to 2^n potential buckets.

A higher -lsh_max_bits_diff value leads to more buckets being checked and vice versa. More precisely, an m number of -lsh_max_bits_diff with an n number of hashes leads to m-combinations of n + (m - 1) -combination of n + ... + 0 -combinations of n checked buckets.

By modifying these parameters, you can control the number of checked vectors. A lower number of checked vectors leads to higher performance, but also increases the chance of missing a similar vector.

Different settings perform well for different vector store sizes. A reasonable initial configuration is (3, 1). If you want to slightly increase the precision, you can change it to (3, 2), but this will substantially increase the number of checked vectors and reduce performance.

To make finer calibration, you would need a higher number of hashes — for instance, (6, 2) is also a possible configuration.

To increase performance, you could change the configuration to (6, 1) or (8, 2), but this will reduce precision.

If increasing the precision at the cost of performance is an acceptable option for you, you could use the configuration of (6, 3).

Note

If the -lsh_max_bits_diff value is too close to the -lsh_hashes_num value, the performance can suffer compared to the default one because of the computational overhead.

Boosting a term’s weight

You can boost a given term’s weight in the text-based similarity index for term-based searches (Term to term or Term to document). Boosting a term’s weight can be done by using the caret symbol ^ followed by a boosting factor — a positive decimal number term^factor.

For example, UK Brexit^3 EU will perform a search in which the term “Brexit” will have 3 times more weight than “UK” and “EU”, and you can expect the results to be mainly related to “Brexit”.

The default boosting factor is 1. Setting a boosting factor of 0 will completely ignore the given term. Escaping the caret symbol ^ is done with a double backslash \\^.

Note

The boosting will not work in document-based searches (Document to term or Document to document), meaning that the caret following by a number will not be treated as a weight boosting symbol.

Predication-based Semantic Indexing

Predication-based Semantic Indexing, or PSI, is an application of distributional semantic techniques for reasoning and inference. PSI starts with a collection of known facts or observations and combines them into a single semantic vector model in which both concepts and relationships are represented. With this technique, the usual ways for constructing query vectors and searching for results in semantic vectors can be used to suggest similar concepts based on the knowledge graph.

Load example data

The predication-based semantic search examples are based on Person data from the DBpedia dataset. The sample dataset contains over 730,000 triples for over 101,000 persons born between 1960 and 1970.

1. Download the provided persons-1960-1970 dataset.

2. Unzip it and import the .ttl file into a repository.

3. Enable the Autocomplete index for the repository from Setup ‣ Autocomplete.

For ease of use, you may add the following namespaces for the example dataset (done from Setup ‣ Namespaces):

• dbo: http://dbpedia.org/ontology/

• dbr: http://dbpedia.org/resource/

• foaf: http://xmlns.com/foaf/0.1/

Create a predication-based index

1. From Explore ‣ Similarity ‣ Create similarity index, select Create predication index.

   

2. Fill in the index name shown above and under more options add -termweight idf in the Semantic Vectors create index parameters field so that the index uses term weighting. As another parameter option, setting -dimension higher than the default value of 200 in that same field can improve search results.

3. Configure the Data query. This SPARQL SELECT query determines the data that will be indexed. The SELECT query must bind the following variables:

   • ?subject

   • ?predicate

   • ?object

   The Data query is executed during index creation to obtain the actual data for the index. When data in your repository changes, you must rebuild the index to have it reflect the changes. It is a subquery of a more complicated query that you can see with the View Index Query button.

   For the given example, leave the default Data query. This will create an index with all triples in the repository:

   SELECT ?subject ?predicate ?object
     WHERE {
         ?subject ?predicate ?object .
     }
   

4. Set the Search query. This SELECT query determines the data that will be fetched on search. Add more bindings by modifying this query to see more data in the results table.

   For this example, set the Search query to:

   PREFIX similarity: <http://www.ontotext.com/graphdb/similarity/>
   PREFIX similarity-index: <http://www.ontotext.com/graphdb/similarity/instance/>
   PREFIX psi: <http://www.ontotext.com/graphdb/similarity/psi/>
   PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
   PREFIX dbo: <http://dbpedia.org/ontology/>
   PREFIX foaf: <http://xmlns.com/foaf/0.1/>
   
   SELECT ?entity ?name ?description ?birthDate ?birthPlace ?gender ?score {
       ?search a ?index ;
           ?searchType ?query;
           psi:searchPredicate ?psiPredicate;
           similarity:searchParameters ?parameters;
       ?resultType ?result .
       ?result similarity:value ?entity ;
           similarity:score ?score .
       ?entity foaf:name ?name .
       OPTIONAL { ?entity <http://purl.org/dc/terms/description> ?description . }
       OPTIONAL { ?entity dbo:birthPlace ?birthPlace . }
       OPTIONAL { ?entity dbo:birthDate ?birthDate . }
       OPTIONAL { ?entity foaf:gender ?gender . }
   }
   

5. Click Create to start index creation.

Once the index has been built, you have the same options as for the text similarity index: View SPARQL query, Edit query, Create index from existing one, Rebuild, and Delete index. If you want to edit an index query, you can do it for both the Search and the Analogical queries, which are described below.

Search predication-based index

In the list of Existing indexes, select the people_60s index that you will search in.

In our example, we will look for individuals similar to Hristo Stoichkov — the most famous Bulgarian football player.

In the results, you can see Bulgarian football players born in the same town, other Bulgarian athletes born in the same place, as well as other people with the same birth date.

Analogical searches

Along with searching explicit relations and similarities, PSI can also be used for analogical search.

Suppose you have a dataset with currencies and countries, and want to know the following: “If I use dollars in the USA, what do I use in Mexico?” By using the predicate index, you do not need to know the predicate (“has currency”).

1. Import the Nations.ttl sample dataset into a repository.

2. Build an Autocomplete index for the repository.

3. Build a predication index following the steps above. (There is no need to set the Search query because you’ll be using an Analogical query instead.)

4. Once the index is built, you can use the Analogical search option of your index. In logical terms, your query will translate to “If USA implies dollars, what does Mexico imply?”

   

As you can see, the first result is peso, the Mexican currency. The rest of the results are not relevant in this situation since they are part of a very small dataset.

Why is this important?

PSI supplements traditional tools for artificial inference by giving “nearby” results. In cases where there is a single clear winner, this is essentially the behavior of giving “one right answer”. But in cases where there are several possible plausible answers, having robust approximate answers can provide great benefit.

Hybrid indexing

A Predication index creates a random vector for each entity in the database and uses these vectors to generate the similarity vectors that will be used for similarity searches. This approach does not take into consideration the similarity between the literals themselves. Let’s examine the following example, using the FactForge data from above:

<express:donald-tusk-eu-poland-leave-european-union-polexit> <pubo:formattedDate> 1/11/2018
<telegraph:donald-tusk-warnspoland-could-hold-brexit-style-eu-referendum> <pubo:formattedDate> 1/11/2018
<express:POLAND-s-bid-for-World-War-2-reparations-is-bolstered-by-a-poll-which-found-that-a-majorit> <pubo:formattedDate> 1/6/2018

Naturally we would expect the first news article to be more similar to the second one than to the third one, not only based on their topics — Poland’s relationship with the EU — but also because of their dates. However, the normal Predication index would not take into account the similarity of the dates, and all news would have fairly close scores. In order to handle this type of scenario, we can first create a Text similarity index. It will find that the dates of the three articles are similar, and will then use this information when building the Predication index.

The following describes how to create one.

Edit the FactForge data

Dates, as presented in FactForge, are not literals that the similarity plugin can handle easily. This is why you need to format them to something easier to parse. Do this with the following query on the Workbench SPARQL view to insert pubo:formattedDate triples with a valueof the form mm/dd/yyyy:

PREFIX pub: <http://ontology.ontotext.com/taxonomy/>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
INSERT {
    ?x pubo:formattedDate ?displayDate
}
WHERE {
    ?x pubo:creationDate ?date.
    BIND (CONCAT(STR(MONTH(?date)),
                "/",
                STR(DAY(?date)),
                "/",
            STR(YEAR(?date))) as ?displayDate)
}

Go to Setup and enable the Autocomplete index for allNews.

Create a Literal index

The Literal index is a subtype of the Text index. To build it, create a normal Text index by ticking the Literal index checkbox from the More options menu. These types of indexes can only be used as input indexes for predication indexes and will be indicated in the Similarity page. They can not be used for similarity searching. The index will include all literals returned by the ?documentText variable from the Data query.

Make sure to filter out the mentions, so the data in the Literal index only contains the news. When creating the index, use the following Data query:

SELECT ?documentID ?documentText {
        ?documentID ?p ?documentText .
        filter(isLiteral(?documentText))
        filter (?p != <http://factforge.net/ff2016-mapping/mentionsEntity>)
}

Click Create to create the index. It may take a few minutes, so make sure to wait until it finishes before proceeding.

Use the Literal index

Create a new predication index. From its More options menu, select Input Literal Index -> the index created in the previous step.

Since you do not want to look at mentions, and in this sense the default data format is useless, you need to filter them out from the data used in the predication index. Add the following Data query:

SELECT ?subject ?predicate ?object
WHERE {
    ?subject ?predicate ?object .
        filter (?predicate != <http://factforge.net/ff2016-mapping/mentionsEntity>)
        filter (?predicate != <http://ontology.ontotext.com/publishing#creationDate>)
}

For the purposes of the test, we want to also display the new formatted date when retrieving data. Go to the search query tab and add the following query:

PREFIX similarity: <http://www.ontotext.com/graphdb/similarity/>
PREFIX similarity-index: <http://www.ontotext.com/graphdb/similarity/instance/>
PREFIX psi: <http://www.ontotext.com/graphdb/similarity/psi/>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>

SELECT ?entity ?score ?content ?date {
    ?search a ?index ;
        ?searchType ?query;
        psi:searchPredicate ?psiPredicate;
        similarity:searchParameters ?parameters;
        ?resultType ?result .
    ?result similarity:value ?entity ;
            similarity:score ?score .
    ?entity pubo:content ?content .
    ?entity pubo:formattedDate ?date .
}

With those two queries in place, the data returned from the index should be more useful. Create your hybrid predication index and wait for the process to be completed. Then, open it and run a query for “donald tusk”, selecting the Express article about “Polexit” from the Autocomplete suggest box. You will see that the first results are related to the Polexit and dated the same.

Indexing behavior

When building the Literal index, it’s a good idea to index all literals that will be indexed in the Predication index, or at least all literals of the same type. Continuing with the example above, let’s say that the Literal index you have created only returns these three news pieces. Add the following triple about a hypothetical Guardian article and create a Predication index to index all news:

<guardian:poland-grain-exports> <pubo:formattedDate> 12/08/2017

Based on the triples, one would expect that the first article will be equally similar to the third and the new one — their contents and dates have little in common. However, depending on the binding method used when creating the Predication index, you can get a higher score for the third article compared to the new one only because the third article has been indexed by the Literal index. There are two ways to easily avoid this: either index all literals, or at least index all the dates.

Manual creation

If you are not using the Similarity page, you could pass the following options when creating the indexes:

• -literal_index true: passed to a Text index creates a Literal index

• -input_index <literaIndex> (replacing <literalIndex> with the name of an existing Literal index): passed to a Predication index creates a hybrid index based on a Literal index

Using training cycles

When building Text and Predication indexes, you can use training cycles to increase the accuracy of the index. The number of training cycles can be set by passing the option -trainingcycles <numOfCycles>. The default number of training cycles is 0.

Text and Predication indexes have quite different implementations of the training cycles.

Text indexes just repeat the same algorithm multiple times, which leads to algorithm convergence.

Predication indexes initially start the training with a random vector for each entity in the database. On each cycle, the initially random elemental vectors are replaced with the product of the previous cycle, and the algorithm is run again. In addition to the entity vectors, the predicate vectors get trained as well. This leads to higher computational time for a cycle compared to the initial run with trainingcycles = 0.

Note

Each training cycle costs time and computing cycles, and a higher number of cycles will greatly increase the building time.