Summary
-------
Cube for named entities source and recognition (NER).
This cube provides:
- the notion of NerSource (i.e. Named Entities Source), e.g. dbpedia or dbpedia-en (for
Dbpedia in english).
- the notion of NerEntry, which is a token/word/entry that could be recognized.
Basically it requires a "label" and a "cwuri", but an "unormalize_label"
could be given for quicker match, a "weight" for disambiguation or
a "lang" for sorting. It should be related to a NerSource.
- the notion of NerProcess, which is an entity type that stores the parameters
for a Named Entities Recognition: a "name", an "host" (appid or url of a sparql endpoint),
a request (RQL or SPARQL, with the "token" key for substitution), a type ('rql' or 'sparql')
for now, and a lang (for sorting).
Basically a lexic could be defined (NerSource), that contains entries (NerEntry).
Thus processes (NerProcess) could be defined in other applications to retrieve these entries
in some content.
Installation
------------
Creation of the instance:
* Create an instance using: cubicweb-ctl create ner <name-of-instance>
* Create the instance's database using: cubicweb-ctl db-create <name-of-instance>
Creating entities
-----------------
For creating a NerSource (in a cw shell):
session.create_entity('NerSource', name=<name of the source>)
E.g.:
session.create_entity('NerSource', name=u"dbpedia-38-en")
For creating a simple NerEntry (in a cw shell):
session.create_entity('NerEntry', label=<label of the entry>, cwuri=<uri of the entry>)
E.g.:
session.create_entity('NerEntry', label=u"Barack Obama", cwuri=u"http://dbpedia.org/page/Barack_Obama"
ner_source=123)
or :
session.create_entity('NerEntry', label=u"Barack Obama", cwuri=u"http://dbpedia.org/page/Barack_Obama",
ner_source=123, unormalize_label=u"barack obama", lang=u'fr', weight=1)
For creating a NerProcess, probably in another application (in a cw shell):
session.create_entity('NerProcess', name=<name of the process>, host=<name/url of the host>,
type=<rql or sparql>, request=<rql or sparql query with %(token)s>)
E.g.:
session.create_entity('NerProcess', name=u'dbpedia38-en', host=u'ner',
type=u'rql', lang=u'en',
request=u'Any U WHERE X label %(token)s, X cwuri U, '
'X ner_source NS, NS name "dbpedia38-en"')
or :
session.create_entity('NerProcess', name=u'dbpedia-sparql', host=u'http://dbpedia.org/sparql',
type=u'sparql', lang=u'en',
request=u'''SELECT ?uri
WHERE{
?uri rdfs:label "%(w)s"@en .
?uri rdf:type ?type
FILTER(?type IN (dbpedia-owl:Agent, dbpedia-owl:Event,
dbpedia-owl:MeanOfTransportation,
dbpedia-owl:Place,
dbpedia-owl:TopicalConcept))}''')
Commands
--------
A command "NerImportDbpedia" exists to import the labels from a dbpedia dump:
* Download the 'labels_en.nt' from Dbpedia (e.g. http://wiki.dbpedia.org/Downloads38),
in the Dataset "Titles". WARNING ! You should download the NT file.
* Decompress the file
* Use the command:
cubicweb-ctl ner-import-dbpedia <instance name> labels_en.nt --name=<name of the source>
where <name of the source> could be "dbpedia38-en" for example.
Adapters
--------
The "INamedEntitiesContentAbstract" adapter could be use to imply that an etype
has a content where Named Entities Recognition could be applied.
