Enhancing Named Entity Recognition in Twitter Messages Using Entity Linking

In this paper, we describe our approach for Named Entity Recognition in Twitter , a shared task for ACL 2015 Workshop on Noisy User-generated Text (Baldwin et al., 2015). Because of the noisy, short, and colloquial nature of Twitter, the performance of Named Entity Recognition (NER) degrades signiﬁcantly. To address this problem, we propose a novel method to enhance the performance of the Twitter NER task by using Entity Linking which is a method for detecting entity mentions in text and resolving them to corresponding entries in knowledge bases such as Wikipedia. Our method is based on supervised machine-learning and uses the high-quality knowledge obtained from several open knowledge bases. In comparison with the other systems proposed for this shared task, our method achieved the best performance.


Introduction
Named Entity Recognition (NER) refers to the task of identifying mentions of entities (e.g., persons, locations, organizations) within text. Because of the noisy, short, and colloquial nature of Twitter messages (or tweets), the performance of standard NER software significantly suffers. For example, Derczynski et al. (Derczynski et al., 2015) recently demonstrated that the performance of various state-of-the-art NER software (e.g., Stanford NER and ANNIE) is typically lower than 50% F1 1 for tweets.
Entity Linking (EL) refers to the task of detecting textual entity mentions and linking them to corresponding entries within knowledge bases (e.g., Wikipedia, DBpedia (Auer et al., 2007), 1 The harmonic mean of precision and recall.
Freebase (Bollacker et al., 2008)). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant attention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015;Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because existing EL methods usually address the mention detection task by using external NER software whose performance is unreliable when processing tweets. Consequently, several approaches (Guo et al., 2013;Yamada et al., 2015) have been proposed with enhanced abilities that address the task in an end-to-end manner without completely depending on NER software.
The main objective of this study is to investigate the possibility of enhancing the performance of Twitter NER by using an end-to-end EL. Although EL is typically performed after NER in most of the existing methods, our approach performs EL before NER and uses the EL results to enhance the NER performance. Resolving the entity mentions to the KB entries enables us to use the high-quality knowledge in KB for enhancing the NER performance. This knowledge includes things such as the popularity of the entity, the classes of the entity, and the likelihood that the entity appears in the given context.
We begin by briefly introducing our end-to-end EL method that specifically focuses on tweets. Our EL method is based on supervised machinelearning and addresses the task in an end-to-end manner. It considers every possible n-gram as a candidate entity mention and detects the mention with a corresponding link to a KB entry if the mention exists in the KB. Furthermore, it can handle mentions that appear as irregular forms (e.g., misspellings, abbreviations, acronyms) using several approximate string matching algorithms.
The NER task is split into two separate subtasks: segmentation and classification. During segmentation, the entity mentions are detected from tweets. Then, the entity mentions are classified into the predefined entity types. Both tasks involve supervised machine-learning with various features.
For the segmentation task, we use data obtained from the KB of the corresponding entity mention detected by the EL and the output of a NER software as the main machine-learning features. Furthermore, we include several common features used in traditional NER methods.
For the classification task, the following three types of features are used as primary features: 1) the KB types of the entity detected by the EL, 2) the entity types detected by the NER software, and 3) the vector representation of the entity mention derived from word embeddings. The entity's KB types are extracted from the corresponding entries in DBpedia and Freebase. Furthermore, the vector representation of the entity mention is derived using GloVe word embeddings (Pennington et al., 2014).
To train and evaluate our system, we used the dataset given by the Named Entity Recognition in Twitter shared task. Our proposed method significantly outperformed the second ranked system by a wide margin; 10.3% F1 at the segmentation task, and 5.0% F1 at the end-to-end (both the segmentation and the classification) task.

Preprocessing
The system first assigns part-of-speech tags to the resulting tokens using ARK Twitter Part-of-Speech Tagger (Gimpel et al., 2011). It also tokenizes Twitter hashtags using our enhanced implementation of the hashtag tokenization.

Entity Linking
We formalize our EL task as follows: Given a tweet, our goal is to recognize a set of entity mentions (e.g., Obama, President Obama, Barack Obama) that appear in a tweet, and then resolve the mentions into entities (e.g., Barack Obama) in Wikipedia if they exist.
Our EL system addresses the task using the following two steps; mention candidate generation and mention detection and disambiguation.

Mention Candidate Generation
Our system first generates a set of candidate entity mentions with the set of corresponding referent entities. The system takes all the n-grams of n ≤ 10 and looks up each n-gram in a dictionary, treats an n-gram as a candidate mention if it exists in the dictionary, and finally, generates an output of pairs of mentions and their associated possible referent entities.
Mention-Entity Dictionary: The system uses a mention-entity dictionary that maps a mention surface (e.g., apple) to the possible referent entities (e.g., Apple Inc., Apple (food)). The possible mention surfaces of an entity are extracted from the corresponding Wikipedia page title, the page titles of the Wikipedia pages that redirect to the page of the entity, and anchor texts in Wikipedia articles that point to the page of the entity. We constructed this dictionary using the January 2015 dump of Wikipedia.
Approximate Candidate Generation: One major problem of the mention candidate generation task is that many entity mentions in tweets cannot be detected because they appear as irregular forms (e.g., misspellings, abbreviations). In order to address this problem, we introduce the following three approximate string-matching methods to improve the ability of this task: • Fuzzy match searches the mention candidates that have text surfaces within a certain distance of the surface of the n-gram measured by edit distance.
• Approximate token search obtains mention candidates whose text surfaces have a significant ratio of words in common with the surface of the n-gram.
• Acronym search retrieves mention candidates with possible acronyms 2 that include the surface of the n-gram.
When using the above methods, we observed that the number of mention candidates becomes very large. To deal with this, we use a simple filtering method based on soft tf-idf (Cohen et al., 2003); we simply use only the mention candidates that have a similarity greater than a threshold measured by the soft tf-idf. We use 0.9 as the threshold because this achieves the best performance in our experiments of EL.

Mention Detection and Disambiguation
Given a pair of a mention and its possible referent entity, it needs to be determined if the possible referent entity is indeed the correct one for its associated mention.
In this system, we use a supervised machinelearning algorithm to assign a relevance score to each of the pairs and select the entity mention with the highest score. We use random forest as the machine-learning algorithm.
Here, we use machine-learning features that are mostly identical to the method proposed previously (Yamada et al., 2015). Basically, we use various features that are commonly observed in EL studies and enhance the performance further by introducing two new features: 1) the entity popularity knowledge extracted from Wikipedia page views 3 , and 2) the contextual similarity between the entity and the tweet measured by word embeddings.

Named Entity Recognition
We address the NER task by performing two subtasks: segmentation and classification.

Segmentation of Named Entities
In this step, entity mentions are detected from tweets. We formalize this task as follows. Given an n-gram in a tweet, the goal of this task is assigning a binary label that represents whether the n-gram should be detected as an entity mention. Note that in order to enable the straightforward integration of EL and this task, we formalize this task as simply classifying n-grams instead of the commonly-used IOB labeling approach (Ramshaw and Marcus, 1995).
The basic strategy that we adopt here is to combine the output of NER software and the KB knowledge of the corresponding entity mention detected by the EL using supervised machinelearning. We again use random forest as the machine-learning algorithm.
We use Stanford NER 4 as the NER software that achieves relatively better performance in the Twitter NER task in a recent study (Derczynski et al., 2015). Here, we adopt two models of Stanford NER to enhance the performance: 1) the standard three-class model which is included in the software and 2) a model that does not use capitalization as a feature, in order to deal with the unreliability of capitalization in tweets.
The results of the NER and the KB knowledge of the corresponding entity mention detected by the EL are used as the primary machine-learning features. We also include features that are traditionally used in NER such as part-of-speech tags and the capitalization features. Furthermore, the ratio of the capitalized words in the tweet is also used as an indicator of the reliability of the capitalization.
The machine-learning features for this step include: • EL relevance score * : The relevance score of the entity mention assigned by the previous EL step.
• Link probability * : The probability of the entity mention appearing as an anchor text in Wikipedia.
• Capitalization probability * : The probability of the entity mention being capitalized in Wikipedia.
• The number of inbound links * : The number of inbound links of the corresponding entity in Wikipedia.
• The average page view * : The average page view of the corresponding entity in Wikipedia.
• NER span match: Binary values that represent whether the n-gram is detected by NER models.
• Part-of-speech tags: Part-of-speech tags of the previous, first, last, and next words of the n-gram.
• Context capitalization: Binary values that represent whether the previous, first, last, and next words of the n-gram are capitalized.
• Character length: The number of characters read in the surface of the n-gram.
• Token length: The number of tokens read in the n-gram.
Note that some features (marked with * ) are based on an entity mention detected by EL, thus these features can be missing if there is no corresponding entity mention detected by the EL.
We also resolve overlaps of mentions by iteratively selecting the longest entity mention from the beginning of a tweet.

Classification of Named Entities
In this step, detected entity mentions are classified into the predefined types (i.e., person, geo-loc, facility, product, company, movie, sportsteam, musicartist, tvshow, and other) using supervised machine-learning. Here, linear support vector machine is used as the machine-learning model.
One main machine-learning feature of this step is the corresponding entity types retrieved from KBs. We obtain KB entity types from the corresponding entries in DBpedia 5 and Freebase 6 .
One problem in this step is that there are several entity mentions that cannot be detected by EL because of various reasons (e.g., a non-existent entity in the KB, an error performing EL). In addition, some minor entities might not have entity types in the KBs. In order to deal with this problem, we first include the entity types predicted by Stanford NER as features. However, because the target entity types of our task do not directly correspond to the ones given in Stanford NER (i.e., location, person, and organization), the effectiveness of these features is obviously limited. Therefore, we introduce another type of feature based on word embeddings. For this, we use GloVe word embeddings 7 to calculate an average vector of vectors of words in n-gram text.
We also include the relevance score assigned by the previous EL step that indicates the reliability of the KB entity types to the model. The number of words and the number of characters in the ngram text are also included as features to enhance the expressiveness of our model even further.
The machine-learning features for this step include: • KB entity types: The entity types in KBs. The KBs used include DBpedia and Freebase.
• NER detected type: The detected entity types of the NER model. As mentioned in Section • N-gram vector: The vector representation of the n-gram derived using the method explained above and includes each dimension of the vector as a separate feature.
• EL relevance score: The relevance score assigned by the previous EL step.
• Character length: The number of characters read in the n-gram text.
• Token length: The number of tokens read in the n-gram.

Experimental Setup
To train our proposed EL method, we used the #Microposts 2015 EL dataset  that contains 3,998 tweets and 3,993 annotations of entities. 8 The performance of our EL method using this particular dataset is reported in (Yamada et al., 2015). For this shared task, we trained and evaluated our proposed Twitter NER using the dataset provided by the workshop. 9 Table 1 shows the results of the segmentation task of the five top-ranking systems. Our proposed method significantly outperforms the second ranked method by 10.3% F1.

Results
The end-to-end results (both segmentation and classification tasks) of the five top-ranking systems are shown in Table 2. Here, our method significantly outperforms the second ranked method by 5.0% F1.   Table 3: Performance of our system at both segmentation and classification tasks broken down by entity types