Text mining plugin¶
What’s in this document?
The GraphDB text mining plugin lets you consume the output of text mining APIs as SPARQL binding variables. Depending on the annotations returned by the concrete API, the plugin makes it possible to:
Generate semantic annotations by linking fragments from texts to knowledge graph entities (entity linking)
Transform and filter the text annotations to a concrete RDF data model using SPARQL
Enrich the knowledge graph with additional information suggested by the information extraction or invalidate their input
Evaluate and control the quality of the text annotations by comparing different versions
Implement complex text mining use cases when combined with the Kafka GraphDB connector
The plugin supports the protocols of these services:
Ontotext’s Tag API
You can also use any text mining service that outputs JSON responses as long as you provide a JSLT transformation to remodel the output from the service output to an output that the plugin understands. See the examples below for querying the Google Cloud Natural Language API and the Refinitiv API using the generic client.
Usage examples¶
A typical use case would be analyzing a piece of text such as a news story to recognize people, organizations, and locations fragments. Ideally, we will link them to entity IRIs that are already known in the knowledge graph such as Wikidata or PermID IRIs, which provide further possibilities for graph enrichment.
Let’s say we have the following text that mentions Dyson as the company “Dyson Ltd.”, the person “James Dyson”, and also only as “Dyson”.
“Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore. The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company.”
The remaining sections of this page demonstrate how different services can annotate this text.
Note
Keep in mind that query results shown below may vary depending on the service.
spaCy server¶
The spaCy server is a containerized HTTP API that provides industrial-strength natural language processing whose named entity recognition (NER) component is used by the plugin.
The NER pipeline is the only spaCy component currently supported by the text mining plugin.
Creating a spaCy client¶
Run the spaCy server through its Docker image with the following commands:
docker pull neelkamath/spacy-server:2-en_core_web_sm-sense2vec docker run --rm -p 8000:8000 neelkamath/spacy-server:2-en_core_web_sm-sense2vecIn the Workbench SPARQL editor, execute the following query, where
http://localhost:8000is the location of the spaCy server that was set up using the above Docker image:PREFIX txtm: <http://www.ontotext.com/textmining#> PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#> INSERT DATA { txtm-inst:localSpacy txtm:connect txtm:Spacy; txtm:service "http://localhost:8000" . }
Note
The sense2vec similarity feature is enabled by default. If your Docker image does not support it or you want to disable it when creating the client, set it to false in the SPARQL query:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:localSpacy txtm:connect txtm:Spacy;
txtm:service "http://localhost:8000";
txtm:sense2vec "false" .
}
Finding spaCy entities through GraphDB¶
The simplest query will return all annotations with their types and offsets. Since spaCy also provides sentence grouping, we can also get the text where each annotation was found.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?annotationText ?sentence ?annotationType ?annotationStart ?annotationEnd
WHERE {
?searchDocument a txtm-inst:localSpacy;
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company''' .
graph txtm-inst:localSpacy {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationKey ?annotationKey;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
optional {
?annotation txtm:hasSentence/txtm:sentenceText ?sentence.
}
}
}
We see that spaCy succeeds in assigning the correct types to each “Dyson” found in the text:
Each service attaches its own metadata, which can be obtained through the feature predicate, to the annotations. In spaCy’s case, we can reach the sense2vec similarity data using the following query:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?annotationText ?sentence ?annotationType ?annotationStart ?annotationEnd ?feature ?value ?featureItem ?featureValue
WHERE {
?searchDocument a txtm-inst:localSpacy;
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company''' .
graph txtm-inst:localSpacy {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
optional {
?annotation txtm:hasSentence/txtm:sentenceText ?sentence.
}
optional {
?annotation txtm:features ?item .
?item ?feature ?value .
optional {
?value ?featureItem ?featureValue .
}
}
}
}
spaCy’s sense2vec similarity feature provides us with the additional knowledge that Dyson is somehow related to “vacuums” and “Miele”.
GATE Cloud¶
The GATE Cloud text analytics service provides several pipelines to choose from. Its ANNIE named entity recognizer used by the plugin identifies basic entity types such as Person, Location, Organization, Money amounts, Time and Date expressions.
Creating a GATE client¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:gateService txtm:connect txtm:Gate;
txtm:service "https://cloud-api.gate.ac.uk/process-document/annie-named-entity-recognizer?annotations=:Address&annotations=:Date&annotations=:Location&annotations=:Organization&annotations=:Person&annotations=:Money&annotations=:Percent&annotations=:Sentence" .
}
You can provide the annotation types you are interested in using the query parameters.
Finding GATE entities through GraphDB¶
The following query uses the txtm:text predicate to pass the text that is to be analyzed to the GATE service.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value
WHERE {
?searchDocument a txtm-inst:gateService;
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company''' .
graph txtm-inst:gateService {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
optional { ?annotation txtm:features ?item . ?item ?feature ?value }
}
}
In GATE, sentences are returned as annotations, so they will appear as annotations in the response.
Tag¶
Ontotext’s Tag API lets you semantically enrich content with annotations about both known and novel concepts.
Using machine learning algorithms and data from DBpedia and Wikidata, it recognizes mentions of entities such as Person, Organization, and Location, various relationships between them, and general topics and key phrases. Visit the NOW demonstrator to explore such entities found in news.
Creating a Tag client¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:tagService txtm:connect txtm:Ces;
txtm:service "https://tagger.ontotext.com/extract" ;
txtm:header "X-API-Key: qQ-H4f>.G}9Tqbfi&=%i" .
}
Note
The API key shown above is for demo purposes and will return a limited number of results in the course of a day. Contact your Ontotext representative about obtaining your own API key if you are interested in using the Tag service as part of your own applications.
Finding Tag entities through GraphDB¶
The following query uses the txtm:text predicate to pass the text to be analyzed to the Tag service:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
SELECT ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value
WHERE {
?searchDocument a txtm-inst:tagService;
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company.''' .
graph txtm-inst:tagService {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
{
?annotation txtm:features ?item .
?item ?feature ?value
}
}
}
For some annotations, it finds an exact match to one or more IRIs in the knowledge graph and provides annotation data in the annotation variables.
Tag also succeeds in assigning the proper type “Person” for “Dyson”.
Here are some details about the features that Tag provides for each annotation:
txtm:inst: The ID of the concept from the knowledge graph which was assigned to this annotation, or an ID of a generated concept in case it is not trusted (seetxtm:isTrustedbelow).For example, http://ontology.ontotext.com/resource/9cafep — you can find a short description and news that mention this entity in the NOW web application at http://now.ontotext.com/#/concept&uri=http://ontology.ontotext.com/resource/9cafep, using the IRI value as uri parameter.
txtm:class: The class of the concept from the knowledge graph which was assigned to this annotation.txtm:isTrusted: Has valuetruewhen the entity is mapped to an existing entity in the database.txtm:isGenerated: Has valuetruewhen the annotation has been generated by the pipeline itself — that is, from NER taggers for which there is no suitable concept in the knowledge graph. Note that generation of this does not mean that the annotation is not trusted.txtm:relevanceScore: A floating point number that represents the level of relevancy of the annotation to the target document.txtm:confidence: A floating point number that represents the confidence score for the annotation to be produced.
Extracting Tag entities using the Web Annotation Data Model¶
The Tag service provides a way to serve entities and their features as RDF. The model is based on the Web Annotation Data Model. The following headers should be passed when creating the Tag client:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:tagInstJSONLD txtm:connect txtm:Ces;
txtm:service "https://tagger.ontotext.com/extract";
txtm:header "Accept: application/vnd.ontotext.ces+json+ld";
txtm:header "Content-type: application/vnd.ontotext.ces+json+ld" ;
txtm:header "X-API-Key: qQ-H4f>.G}9Tqbfi&=%i" .
}
The common model applied for all services is no longer used because you get the Tag response in RDF as it is formed by the service.
The following request type (Content-type) and response type (Accept) combinations are supported:
Content-type: text/plain—Accept: application/vnd.ontotext.ces+json(the default if nothing is specified)Content-type: application/vnd.ontotext.ces+json+ld—Accept: application/vnd.ontotext.ces+jsonContent-type: application/vnd.ontotext.ces+json+ld—Accept: application/vnd.ontotext.ces+json+ld
Not supported:
Content-type: text/plain—Accept: application/vnd.ontotext.ces+json+ldContent-type: application/vnd.ontotext.ces+json
Note
This means that JSON-LD as a response type requires that the request is JSON-LD and nothing else. The default text/plain will not work, so when creating the plugin, you need to pass the Content-type explicitly.
When the request type is JSON-LD, the response type can be JSON or JSON-LD.
When using JSON-LD, the following document features are required. They must be passed using the txtm:features predicate on ?annotatedDocument and in the following order:
txtm:features (?id ?title ?type ?author ?source ?category ?date).
Here is a sample query:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX resource: <http://ontology.ontotext.com/resource/>
PREFIX content: <http://data.ontotext.com/content/>
PREFIX onto: <http://www.ontotext.com/>
CONSTRUCT { ?subject ?predicate ?object }
WHERE {
?searchDocument a txtm-inst:tagInstJSONLD;
txtm:features (resource:guid-for-the-annotated-document "Dyson Ltd. hires 450 people globally" "Article" "The author" <https://the_doc_source_uri> content:My_Category "2019-03-01T00:11:15Z");
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore. The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company. ''' ;
graph txtm-inst:tagInstJSONLD {
?subject ?predicate ?object
}
}
You can also use the txtm:rawInput predicate to provide your own raw JSON-LD document. Revising the query above to do this, it will look like the following, and will return the same results:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX resource: <http://ontology.ontotext.com/resource/>
PREFIX content: <http://data.ontotext.com/content/>
PREFIX onto: <http://www.ontotext.com/>
CONSTRUCT { ?subject ?predicate ?object }
WHERE {
?searchDocument a txtm-inst:tagInstJSONLD;
txtm:rawInput '''{
"@id": "resource:some-new-guid-for-the-annotated-document-resource",
"@graph": [
{
"@id": "resource:some-new-guid-for-the-annotated-document-resource",
"@type": "AnnotatedDocument",
"document": {
"@id": "http://ontology.ontotext.com/resource/guid-for-the-annotated-document",
"@type": "Article",
"author": "The author",
"documentSource": "https://the_doc_source_uri",
"category": "http://data.ontotext.com/content/My_Category",
"publishDate": "2019-03-01T00:11:15Z",
"title": "Dyson Ltd. hires 450 people globally",
"docContent": "Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore. The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company. "
}
}
],
"@context": [
"http://www.w3.org/ns/anno.jsonld",
{
"ann": "http://data.ontotext.com/annotation/",
"ontoa": "http://ontology.ontotext.com/annotation#",
"ontocontent": "http://ontology.ontotext.com/content#",
"onto": "http://ontology.ontotext.com/taxonomy/",
"content": "http://data.ontotext.com/content/",
"resource": "http://ontology.ontotext.com/resource/",
"xsd": "http://www.w3.org/2001/XMLSchema#",
"nif": "http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#",
"Article": "ontocontent:Article",
"AnnotatedDocument": "ontocontent:AnnotatedDocument",
"document": "ontocontent:document",
"annotations": "ontocontent:annotations",
"author": {
"@id": "ontocontent:author",
"@type": "xsd:string"
},
"documentSource": {
"@id": "ontocontent:source",
"@type": "@id"
},
"category": {
"@id": "ontocontent:category",
"@type": "@id"
},
"publishDate": {
"@id": "ontocontent:publishDate",
"@type": "xsd:dateTime"
},
"title": {
"@id": "ontocontent:title",
"@type": "xsd:string"
},
"docContent": "ontocontent:content",
"tagType": {
"@id": "ontoa:tagType",
"@type": "@id"
},
"relevanceScore": {
"@id": "ontoa:relevanceScore",
"@type": "xsd:double"
},
"confidence": {
"@id": "nif:confidence",
"@type": "xsd:double"
},
"type": {
"@id": "ontoa:type",
"@type": "xsd:string"
},
"class": {
"@id": "ontoa:class",
"@type": "@id"
},
"status": {
"@id": "ontoa:status",
"@type": "xsd:string"
},
"isTrusted": {
"@id": "ontoa:isTrusted",
"@type": "xsd:boolean"
},
"isGenerated": {
"@id": "ontoa:isGenerated",
"@type": "xsd:boolean"
},
"annotationSetName": {
"@id": "ontoa:annotationSetName",
"@type": "xsd:string"
},
"annotationType": {
"@id": "ontoa:type",
"@type": "xsd:string"
}
}
]
} ''' ;
graph txtm-inst:tagInstJSONLD {
?subject ?predicate ?object
}
}
The supported returned response formats are JSON and JSON-LD.
Extracting annotations from another NER service¶
To register a service in the text mining plugin, the service must provide a REST interface with a POST endpoint. The response Content-Type value must be application/json. Pass the POST request headers using the predicate http://www.ontotext.com/textmining#header. Pass the request body with the predicate http://www.ontotext.com/textmining#text.
The following curl request
curl -X POST --header "HEADER1: VALUE1" --header "HEADER2: VALUE2" -d 'body' 'https://endoint.com?queryParam1=param1'
corresponds to the following configuration,
PREFIX : <http://www.ontotext.com/textmining#>
PREFIX inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
inst:myService :connect :Provider;
:service "https://endoint.com?queryParam1=param1";
:header "HEADER1: VALUE1";
:header "HEADER2: VALUE2";
:transformation '''
...
'''.
}
and to the following query for consuming the annotations:
PREFIX : <http://www.ontotext.com/textmining#>
PREFIX inst: <http://www.ontotext.com/textmining/instance#>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
SELECT ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value
WHERE {
?searchDocument a inst:myService;
:text '''body''' .
graph inst:myService {
?annotatedDocument :annotations ?annotation .
?annotation :annotationText ?annotationText ;
:annotationType ?annotationType ;
:annotationStart ?annotationStart ;
:annotationEnd ?annotationEnd ;
{
?annotation :features ?item .
?item ?feature ?value
}
}
}
To extract annotations using another named entity recognition provider, create a client for these services using a JSLT transformation. The transformation will convert the JSON returned by the target service to a JSON model that the text mining plugin understands. The target JSON should follow this model:
{
"content":"",
"sentences":[ ],
"features":{ },
"annotations":[
{
"text":"Google",
"type":"Company",
"startOffset":78,
"endOffset":84,
"confidence":0.0,
"features":{ }
}
]
}
where the only required part is the following:
{
"annotations":[
{
"text":"Google",
"type":"Company",
"startOffset":78,
"endOffset":84,
}
]
}
Google Cloud Natural Language API¶
The Google Cloud Natural Language API associates information such as salience and mentions with annotations, where an annotation represents a phrase in the text that is a known entity such as a person, an organization, or a location. This also requires a token to access the API.
Creating a Google Cloud Natural Language API client¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:myGoogleService txtm:connect txtm:Provider;
txtm:service "https://language.googleapis.com/v1/documents:annotateText";
txtm:header "Authorization: Bearer <your API token>";
txtm:transformation '''
{"annotations" : flatten([for (.entities)
let type = .type
let metadata = .metadata
let salience = .salience
let mentions = [for (.mentions) {
"type" : $type,
"text" : .text.content,
"startOffset" : .text.beginOffset,
"endOffset" : .text.beginOffset + size(.text.content),
"features" : {
"salience" : $salience,
"metadata" : $metadata
}
}]
$mentions
])}
'''.
}
Extracting entities from Google Cloud Natural Language API¶
Once created, you can list annotations using a model similar to the other services. Note that you need to provide the input in the way the service expects it. No transformation is applied to the request content.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value ?featureItem ?featureValue
WHERE {
?searchDocument a txtm-inst:myGoogleService;
txtm:text '''
{
"document":{
"type":"PLAIN_TEXT",
"content":"Net income was $9.4 million compared to the prior year of $2.7 million. Google is a big company.
Revenue exceeded twelve billion dollars, with a loss of $1b"
}, "features": {'extractEntities': 'true', 'extractSyntax': 'true'},
'encodingType':'UTF8',
}
''' .
graph txtm-inst:myGoogleService {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
optional {
?annotation txtm:features ?item .
?item ?feature ?value .
optional { ?value ?featureItem ?featureValue . }
}
}
}
The results will look like this:
Refinitiv API¶
Refinitiv’s PermIDs are open, permanent, and universal identifiers where underlying attributes capture the context of the identity they each represent.
Creating a Refinitiv API client¶
The following provides a model for creating a Refinitiv API client.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:refinitiv txtm:connect txtm:Provider;
txtm:service "https://api-eit.refinitiv.com/permid/calais";
txtm:header "X-AG-Access-Token: <your_access_token>";
txtm:header "Content-Type: text/raw";
txtm:header "x-calais-selectiveTags: company,person,industry,socialtags,topic";
txtm:header "outputformat: application/json";
txtm:transformation '''
{
"content" : string(.doc.info.document),
"rawSource" : string(.),
"language" : .doc.meta.language,
"features" : {for (.) .key : {for (.value) .key : .value }
if (.value._typeGroup and .value._typeGroup != "entities" and .value._typeGroup != "relations"
and .value._typeGroup != "language" and .value._typeGroup != "versions") },
"annotations" : flatten([for (.)
if (.value._typeGroup == "entities")
let type = .value._type
let text = .value.name
let features = {for (.value) .key : .value
if (.key != "_type" and .key != "name" and .key != "instances" and .key != "offset")}
let instances = [for (.value.instances){
"type" : $type,
"text" : $text,
"startOffset": .offset,
"endOffset" : .offset + size($text),
"features" : $features
}]
$instances
else if (.value._typeGroup == "relations")
let type = .value._type
let features = {for (.value) .key : .value
if (.key != "_type" and .key != "instances")}
let instances = [for (.value.instances){
"type" : $type,
"text" : .exact,
"startOffset": .offset,
"endOffset" : .offset + size(.exact),
"features" : $features
}]
$instances
else
[]
])
}
'''.
}
Extracting Refinitiv PermID entities¶
The following query uses the txtm:text predicate to pass the text to be analyzed to the Refinitiv service:
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?searchDocument ?annotation ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value ?featureItem ?featureValue
WHERE {
?searchDocument a txtm-inst:refinitiv;
txtm:text '''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company.''' .
graph txtm-inst:refinitiv {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText ;
txtm:annotationType ?annotationType ;
txtm:annotationStart ?annotationStart ;
txtm:annotationEnd ?annotationEnd ;
optional {
?annotation txtm:features ?item . ?item ?feature ?value .
optional { ?value ?featureItem ?featureValue . }
}
}
}
The tricky part of the integration of an arbitrary NER provider is to write the JSLT transformation, but once you get used to the language, you can enrich your text document with any entity provider of your choice, and extend your knowledge graph solely with the power of SPARQL and GraphDB.
Escaping special characters¶
The following example demonstrates how quotation marks within the quoted input string can be escaped with double-backslash sequences:
PREFIX : <http://www.ontotext.com/textmining#>
PREFIX inst: <http://www.ontotext.com/textmining/instance#>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
SELECT ?annotationText ?annotationType ?annotationStart ?annotationEnd ?feature ?value
WHERE {
?searchDocument a inst:razor;
:text '''
{"text":"Prosecutors want NFL's Peterson arrested on alleged bond violation | Reuters
Prosecutors want NFL's Peterson arrested on alleged bond violation
By Eric Kelsey
(Reuters) --- Suspended Minnesota Vikings star Adrian Peterson faced new legal trouble on Thursday after Texas prosecutors in his child abuse case asked a court to order his arrest on a possible drug-related bond violation.
Peterson, 29, who has been accused of injuring his 4-year-old son while disciplining him with the thin end of a tree branch, allegedly told a drug-testing administrator on Wednesday he had smoked marijuana before submitting to a urinalysis test, court papers said.
\\"During this process the defendant admitted ... that he smoked a little weed,\\" according to the motion filed by Montgomery County District Attorney Brett Ligon.
A court date has not been set on the possible bond violation. Peterson's next scheduled court date is Nov. 4.
It is unclear when a judge would rule on the motion as prosecutors' request to have the current judge recused must be heard first.
Peterson's attorney, Rusty Hardin, declined to comment until a judge is settled on in the case.
The Vikings said in a statement they were aware of the allegation and \\"will await the results of that hearing before having further comment.\\"
The National Football League did not respond to a request for comment.
Peterson was arrested and posted $15,000 bond on Sept. 12 on a charge of injury to a child. He was later suspended indefinitely with pay by the Vikings until the matter is resolved.
He has admitted using a switch, the thin end of a tree branch, to discipline his son, but said he was not trying to injure him.
Peterson could be sentenced to up to two years in prison and fined $10,000 if convicted.
The charge against Peterson came as the NFL faced public criticism for its handling of a spate of domestic violence cases among its players. A number of corporate sponsors rebuked America's most popular professional sports league, which has overhauled how it deals with player behavior and punishment.
(Reporting by Eric Kelsey in Los Angeles; Editing by Peter Cooney )
"}
''' .
graph inst:razor {
?annotatedDocument :annotations ?annotation .
?annotation :annotationText ?annotationText ;
:annotationType ?annotationType ;
:annotationStart ?annotationStart ;
:annotationEnd ?annotationEnd ;
{
?annotation :features ?item .
?item ?feature ?value
}
}
}
Since the text enclosed within the ''' marks represents a literal string, SPARQL will store it as is and keep new lines and paragraphs. The only special characters that need to be escaped with a double backslash are the quotation marks: \\".
Comparing annotations between services¶
The text mining plugin generates meaningful IRIs for the ?annotatedDocument and ?annotation variables. It provides the additional txtm:annotationKey
predicate that binds an IRI for the annotation, based on the text and offsets, to the ?annotationKey variable. This means that regardless of the service that generated the annotation, the same pieces of text will have the same ?annotationKey IRIs. You can use this to compare annotations over the same piece of text provided by different services.
The following query compares annotation types obtained from spaCy and Tag for annotations that have the same key and text — that is, they refer to the same piece of text.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT ?spacyDocument ?tagDocument ?spacyAnnotation ?tagAnnotation ?spacyType ?tagType ?annotationKey ?annotationText
WHERE {
BIND ('''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company''' as ?text)
?searchDocument1 a txtm-inst:localSpacy;
txtm:text ?text.
graph txtm-inst:localSpacy {
?spacyDocument txtm:annotations ?spacyAnnotation .
?spacyAnnotation txtm:annotationText ?annotationText ;
txtm:annotationKey ?annotationKey;
txtm:annotationType ?spacyType .
}
?searchDocument2 a txtm-inst:tagService;
txtm:text ?text .
graph txtm-inst:tagService {
?tagDocument txtm:annotations ?tagAnnotation .
?tagAnnotation txtm:annotationText ?annotationText ;
txtm:annotationKey ?annotationKey;
txtm:annotationType ?tagType .
}
}
The query returns the following:
The IRIs generated by the text mining plugin have the following meaning:
?annotatedDocument(?tagDocumentor?spacyDocumentin the above query):<http://www.ontotext.com/textmining/document/<md5-content>>wheremd5-contentis the MD5 code of the document content.For example:
<http://www.ontotext.com/textmining/document/ffa3feed18dacea1c195492cc1c06847>.Note that document IRIs will be the same for the same pieces of text, regardless of the service.
?annotation:<http://www.ontotext.com/textmining/document/<md5-content>/annotation/<start>/<end>/<service-name>/<index>><start>/<end>: The start/end offsets of the annotation in the text.<service-name>: The name of the service that provided the annotation.<index>: A unique number of the annotation within the document, meaning that if there are different annotation for the same pieces of text, they will have different IRIs.
For example:
<http://www.ontotext.com/textmining/document/ffa3feed18dacea1c195492cc1c06847/annotation/102/111/localSpacy/4><http://www.ontotext.com/textmining/document/ffa3feed18dacea1c195492cc1c06847/annotation/102/111>?annotationKey:<http://www.ontotext.com/textmining/document/<md5-content>/annotation/<start>/<end>>: The annotation key IRI marks only a piece of text in the document and can be used to find annotation over the same piece of text, but provided by different services.For example:
<http://www.ontotext.com/textmining/document/ffa3feed18dacea1c195492cc1c06847/annotation/102/111>
Enriching documents with mentions of known entities¶
Using the Tag txtm:exactMatch feature and our own mentions predicate, we can generate the following triples and enrich our dataset with entities from DBpedia.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
PREFIX my-kg: <http://my.knowledge.graph.com/textmining#>
CONSTRUCT {
?tagDocument my-kg:mentions ?value
}
WHERE {
BIND ('''Dyson Ltd. plans to hire 450 people globally, with more than half the recruits in its headquarters in Singapore.
The company best known for its vacuum cleaners and hand dryers will add 250 engineers in the city-state. This comes short before the founder James Dyson announced he is moving back to the UK after moving residency to Singapore. Dyson, a prominent Brexit supporter who is worth US$29 billion, faced criticism from British lawmakers for relocating his company''' as ?text)
?searchDocument a txtm-inst:tagService;
txtm:text ?text .
graph txtm-inst:tagService {
?tagDocument txtm:annotations ?tagAnnotation .
?tagAnnotation txtm:features ?item .
?item txtm:exactMatch ?value
}
}
The query returns the following:
Of course, the power of RDF allows you to construct any graph you want based on the response from the named entity recognition service.
Error handling¶
To see some error handling in action, let’s first create three documents to annotate in one batch. Use the following update request to insert the three sample documents in your repository:
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX my-kg: <http://my.knowledge.graph.com/textmining#>
INSERT DATA
{
GRAPH <http://my.knowledge.graph.com> {
my-kg:doc1 my-kg:content "SOFIA, March 14 (Reuters) --- Bulgaria expects Azeri state energy company SOCAR to start investing in the Balkan country's retail gas distribution network this year, Prime Minister said on Thursday".
my-kg:doc2 my-kg:content "Bulgaria is looking to secure gas supplies for its planned gas hub at the Black Sea port of Varna and Borissov said he had discussed the possibility of additional Azeri gas shipments for the plan.".
my-kg:doc3 my-kg:content "In the Sunny Beach resort, this one-bedroom apartment is 150m from the sea. It is in the Yassen complex, which has a communal pool and gardens. On the third floor, the 66sq m (718sq ft) apartment has a livingroom, with kitchen, that opens to a balcony overlooking the pool. There are also a bedroom and bathroom. The property is being sold with furniture. The service charge is €8 a square metre, making it about €528. Burgas Airport is about 12km away. Varna is 40km away.".
}
}
You can send all of them for annotation with a single query. By default, if the service fails for one document, the whole query will fail. As a result, you will miss the results for the documents that were successfully annotated. To prevent this from happening, you can use the txtm:serviceErrors predicate that defines a maximum number of errors allowed before the query fails, where -1 means that an infinite number of errors is allowed. As a result of the following query, you will either get an error for the document, or its annotations.
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
PREFIX pubo: <http://ontology.ontotext.com/publishing#>
PREFIX my-kg: <http://my.knowledge.graph.com/textmining#>
SELECT ?content ?annotationText ?errorFeature
WHERE {
?myDocument my-kg:content ?content.
OPTIONAL {
?searchDocument a txtm-inst:localSpacy;
txtm:text ?content;
txtm:serviceErrors -1 .
GRAPH txtm-inst:localSpacy {
OPTIONAL {
?annotatedDocument txtm:annotations ?annotation .
?annotation txtm:annotationText ?annotationText .
}
OPTIONAL {
?annotatedDocument txtm:features ?docFeature .
?docFeature ?errorFeature ?errorFeature.
}
}
}
}
The following results will be returned if the spaCy service successfully annotates the first document but is then stopped. We can simulate this by stopping the spaCy Docker during the query execution using Ctrl+C in the terminal where Docker is running. The error message is returned as a document feature.
Managing text mining instances¶
Use the queries below to explore the instances of text mining clients you have in the repository with their configurations, as well as to remove them.
Listing all clients¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT * where {
?instance a txtm:Service .
}
Getting a client’s configuration¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
SELECT * WHERE {
txtm-inst:localSpacy ?p ?o .
}
Dropping an instance¶
PREFIX txtm: <http://www.ontotext.com/textmining#>
PREFIX txtm-inst: <http://www.ontotext.com/textmining/instance#>
INSERT DATA {
txtm-inst:localSpacy txtm:dropService "".
}
Monitoring annotation progress¶
If you are annotating multiple documents in one transaction, you may want to get feedback on the progress. This is done by setting the log level of the text mining plugin to DEBUG in the conf/logback.xml file of the GraphDB distribution:
<logger name="com.ontotext.graphdb.plugins.textmining" level="DEBUG"/>
You will see a message for each document sent for annotation in the GraphDB main log file in the logs directory.
[DEBUG] 2021-05-19 08:39:40,893 [repositories/ff-news | c.o.g.p.t.c.ClientBase] Annotating document content starting with: "Australia's Cardinal Pell sentenced to six years jail for sexually...
MELBOURNE (Reuters) - Former ..." with length: 911
[DEBUG] 2021-05-19 08:39:41,851 [repositories/ff-news | c.o.g.p.t.c.ClientBase] Annotating document content starting with: "Google engineer calls for accessibility in design
Laura D'Aquila, a software engineer at Google and..." with length: 2455
[DEBUG] 2021-05-19 08:39:45,610 [repositories/ff-news | c.o.g.p.t.c.ClientBase] Annotating document content starting with: "NBA Fines Russell Westbrook $25,000, Utah Jazz Permanently Bans Fan Following Verbal Altercation
Op..." with length: 4932