Implicit Entity Recognition in Clinical Documents

With the increasing automation of health care information processing, it has become crucial to extract meaningful information from textual notes in electronic medical records. One of the key challenges is to extract and normalize entity mentions. State-of-the-art approaches have focused on the recognition of entities that are explicitly mentioned in a sentence. However, clinical documents often contain phrases that indicate the entities but do not contain their names. We term those implicit entity mentions and introduce the problem of implicit entity recognition (IER) in clinical documents. We propose a solution to IER that leverages entity deﬁnitions from a knowledge base to create entity models, projects sentences to the entity models and identiﬁes implicit entity mentions by evaluating semantic similarity between sentences and entity models. The evaluation with 857 sentences selected for 8 different entities shows that our al-gorithm outperforms the most closely related unsupervised solution. The similarity value calculated by our algorithm proved to be an effective feature in a supervised learning setting, helping it to improve over the baselines, and achieving F1 scores of .81 and .73 for different classes of implicit mentions. Our gold standard annotations are made available to encourage further research


Abstract
With the increasing automation of health care information processing, it has become crucial to extract meaningful information from textual notes in electronic medical records. One of the key challenges is to extract and normalize entity mentions. State-of-the-art approaches have focused on the recognition of entities that are explicitly mentioned in a sentence. However, clinical documents often contain phrases that indicate the entities but do not contain their names. We term those implicit entity mentions and introduce the problem of implicit entity recognition (IER) in clinical documents. We propose a solution to IER that leverages entity definitions from a knowledge base to create entity models, projects sentences to the entity models and identifies implicit entity mentions by evaluating semantic similarity between sentences and entity models. The evaluation with 857 sentences selected for 8 different entities shows that our algorithm outperforms the most closely related unsupervised solution. The similarity value calculated by our algorithm proved to be an effective feature in a supervised learning setting, helping it to improve over the baselines, and achieving F1 scores of .81 and .73 for different classes of implicit mentions. Our gold standard annotations are made available to encourage further research in the area of IER.

Introduction
Consider the following sentence, extracted from a clinical document: "Patient has shortness of breath with reaccumulation of fluid in extremities." It states that the patient has 'shortness of breath' and 'edema'. The former is explicitly mentioned, while the latter is implied by the semantics of the phrase 'reaccumulation of fluid in extremities'. We term such occurrences implicit entity mentions.
While implicit entity mentions are common in many domains, resolving them is particularly valuable in the clinical domain. Clinical documents are rich in information content that plays a central role in understanding patients' health status and improving the quality of the delivered services. It is a common practice to employ computer assisted coding (CAC) solutions to assist expert "coders" in determining the unique identifier (e.g., ICD9 or ICD10) for each medical condition or combination of conditions. These identifiers are important to unambiguously represent the medical conditions, to prepare the post-discharge plan, and to perform secondary data analysis tasks. A human coder reading the sentence 'Patient has shortness of breath with reaccumulation of fluid in extremities' would generate the corresponding codes for entities 'shortness of breath' and 'edema'. However, the solutions developed to perform entity recognition in clinical documents (Aronson, 2006) (Friedman et al., 1994) (Savova et al., 2010 (Friedman et al., 2004) (Fu and Ananiadou, 2014) (Pradhan et al., 2015) do not recognize the presence of entity 'edema' in this sentence.
Implicit entity mentions are a common occurrence in clinical documents as they are often typed during a patient visit in a way that is natural in spoken language and meant for consumption by the professionals with similar backgrounds. An analysis with 300 documents in our corpus showed that 35% of the 'edema' mentions and 40% of the 'shortness of breath' mentions are implicit.
Recognizing implicit mentions is particularly 228 challenging since, besides the fact that they lack the entity name, they can be embedded with negations. For example, the semantics of the sentence 'The patients' respiration become unlabored' implies that the patient does not have 'shortness of breath'. Identification of the negated mentions of entities in clinical documents is crucial as they provide valuable insights into the patients' health status.
We propose an unsupervised solution to the IER problem that leverages knowledge embedded in entity definitions obtained for each entity from the Unified Medical Language System (UMLS) (Bodenreider, 2004). UMLS provides a standard vocabulary for the clinical domain. Our solution: a) Creates an entity model from these definitions, b) Identifies the sentences in input text that may contain implicit entity mentions, c) Projects these sentences onto our entity model, and d) Classifies the sentences to distinguish between those containing implicit entity mentions or negated implicit mentions, by calculating the semantic similarity between the entity model and the projected sentences.
The contributions of this work are as follows: 1. We introduce the problem of implicit entity recognition (IER) in clinical documents.
2. We propose an unsupervised solution to IER that outperforms the most relevant unsupervised baseline and improves the results of a supervised baseline.
3. We create a gold standard corpus annotated for IER in the clinical domain and make it available to encourage research in this area.

Related Work
To the best of our knowledge, this is the first work to address the problem of Implicit Entity Recognition (IER) in clinical documents. However, there is a large body of research that is relevant to the problem, including Named Entity Recognition (NER), Entity Linking (EL), Coreference Resolution, Paraphrase Recognition, and Textual Entailment Recognition. Much like IER, both NER and EL have the objective of binding a natural language expression to a semantic identifier. However, related work in NER and EL expect the proper name (explicit mention) of entities and assume the presence of noun phrases (Collins andSinger, 1999) (Bunescu andPasca, 2006). The solutions developed for NER leverage regularities on morphological and syntactical features that are unlikely to hold in the case of IER. The most successful NER approaches use word-level features (such as capitalization, prefixes/suffixes, and punctuation), list lookup features (such as gazetteers, lexicons, or dictionaries), as well as corpus-level features (such as multiple occurrences, syntax, and frequency) (Nadeau and Sekine, 2007) that are not exhibited by the phrases with implicit entity mentions.
Many approaches couple NER with a follow up EL step (Hachey et al., 2013) in order to assign an unique entity identifiers to mentions. Therefore, the inadequacy of NER techniques will limit the capability of recognizing implicit entity mentions by a solution developed for EL. Moreover, state-of-theart EL approaches include a 'candidate mapping' step that uses entity names to narrow down the space of possible entity identifiers, which is also a limiting factor in the IER case. Finally, neither NER nor EL deal with the negated mentions of entities.
Coreference resolution (CR) focuses on grouping multiple mentions of the same entity with different surface forms. The solutions to CR focus on mapping explicit mentions of entity names to other pronouns and noun phrases referring to the same entity (Ng, 2010) (Durrett and Klein, 2013). In IER implicit mentions occur without co-referring corresponding entity. Hence, they must be resolved without dependencies on co-referents.
In contrast to NER, EL, and CR problems and their solutions, IER addresses instances where neither explicit mention of an entity nor noun phrases or any of the above mentioned features are guaranteed to appear in the text but still have a reference to a known entity. Hence, IER solutions require treatment for implied meaning of the phrases beyond its syntactic features.
Since our solution to IER establishes a relationship between entity definitions and the input text, the tasks of paraphrase recognition (Barzilay and Elhadad, 2003) (Dolan et al., 2004) and textual entailment recognition (Giampiccolo et al., 2007) are related to our solution. However, these tasks are fundamentally different in two aspects: 1) Both paraphrase recognition and textual entailment recogni-229 tion are defined at the sentence level, whereas text phrases considered for IER can exist as a sentence fragment or span across multiple sentences, and 2) The objective of IER is to find whether a given text phrase has a mention of an entity-as opposed to determining whether two sentences are similar or entail one another. However, our solution benefits from the lessons learned from both tasks.
The question answering solutions cope with the questions that describe the characteristics of a concept and expect that concept as the answer. This particular type of questions resembles implicit entity mentions. However, they assume that the questions are referring to some concept and the problem is to uncover which one, whereas the implicit entity mention problem requires us to first check whether a particular sentence/phrase has a mention of an entity at all. Furthermore, question answering systems benefit from the presence of pronouns, nouns, and noun phrases in the questions and the candidate answers to derive helpful syntactic and semantic features (Lally et al., 2012) (Wang, 2006), while phrases with implicit entity mentions may not contain such features.
The existing work on clinical document annotation focused on explicit entity mentions with contiguous phrases (Aronson, 2006) (Savova et al., 2010) (Friedman et al., 2004) (Fu and Ananiadou, 2014). Going one step beyond, the SemEval 2014 task 7 recognized the need for identifying discontiguous mentions of explicit entities (Pradhan et al., 2014). However, the recognition of implicit entities has yet to address by this community.

Implicit Entity Recognition (IER) in Clinical Documents
We define the Implicit Entity Recognition (IER) task in clinical documents as: given input text that does not have explicit mentions of target entities, find which target entities are implied (including implied negations) in the input text. Negation detection is traditionally separated from the entity recognition task because negation indicating terms can be recognized separately from the phrases that contain explicit mention of an entity. In contrast, implicit mention can involve an antonym that fuses the entity indication with negated sense  Figure 1: Components of the Proposed Solution (e.g., 'patient denies shortness of breath' vs 'patient is breathing comfortably'). Hence, negation detection is considered as a sub-task of IER.
Typical entity recognition task considers the detection of the boundaries of the phrases with entities (i.e., segmentation) as a sub-task. We consider boundary detection of the implicit entity mentions as an optional step due to two reasons: 1) It is considered an optional step in biomedical entity recognition task (Tsai et al., 2006), and 2) The phrases with implicit entity mentions can be noncontiguous and span multiple sentences. Further, in some cases, even domain experts disagree on the precise phrase boundaries.
We define the IER as a classification task. Given an input text, classify it to one of the three categories: TP e if the text has a mention of entity e, or Tneg e if the text has a negated mention of entity e, or TN e if the entity e is not mentioned at all. As mentioned, the phrases with implicit entity mentions can span to multiple sentences. However, this work will focus only on implicit mentions exist within a sentence. Our unsupervised solution to this classification task: 1) Creates an entity model from the entity definitions, 2) Selects candidate sentences that may contain implicit entity mentions, 3) Projects the candidate sentences into entity model space, and 4) Calculates the semantic similarity between projected sentences and the entity model. Figure 1 shows the components of our solution which are discussed below in detail.
In order to facilitate these sub-tasks, our algorithm introduces the concept of an entity representative term for each entity and propose an automatic way to select these terms from entity definitions. 230

Entity Representative Term Selection
Entity representative term (ERT) selection finds a term with high representative power to an entity and plays an important role in defining it.
The representative power of a term t for entity e is defined based on two properties: its dominance among the definitions of entity e, and its ability to discriminate the mentions of entity e from other entities. This is formalized in eq. (1). Consider the entity 'appendicitis' as an example. It is defined as 'acute inflammation of appendix'. Intuitively, both terms inflammation and appendix are candidates to explain the entity appendicitis. However, the term appendix has more potential to discriminate the implicit mentions of appendicitis than the term inflammation, because the term inflammation is used to describe many entities. Also, none of the definitions define appendicitis without using the term appendix; therefore, appendix is the dominant term, and consequently it has the most representative power for the entity 'appendicitis'.
We used a score inspired by the TF-IDF measure to capture this intuition. The IDF (inverse document frequency) value measures the specificity of a term in the definitions. The TF (term frequency) captures the dominance of a term. Hence the representative power of a term t for entity e (r t ) is defined as, Q e is the set of definitions of entity e, E is the set of all entities. f req(t, Q e ) is the frequency of term t in set Q e , |E| is the size of the set E (3962 in our corpus), and the denominator |E t | calculates the number of entities defined using term t. We expand the ERT found for an entity with this technique by adding its synonyms obtained from WordNet.
We can define entity representative terms based on the definition of representative power.
Definition 3.1 (Entity Representative Term). Let L e = {t 1 , t 2 , ..., t n } be the set of terms in a definitions of an entity e. Let R Le = {r t 1 , r t 2 , ..., r tn } be the representative power calculated for each term t i in L e for e. We select term t m as the entity representative term of the entity e if its representative power is maximum, i.e., r tm ≥ r t i for all i where 1 ≤ i ≤ n.

Entity Model Creation
Our algorithm creates an entity indicator from a definition of the entity. An entity indicator consists of terms that describe the entity. Consider the definition 'A disorder characterized by an uncomfortable sensation of difficulty breathing' for 'shortness of breath', for which the selected ERT is 'breathing'. The terms uncomfortable, sensation, difficulty, and breathing collectively describe the entity. Adding other terms in this definition to the entity indicator negatively affects the similarity calculation with the candidate sentences since they are less likely to appear in a candidate sentence. We exploited the neighborhood of the ERT in the definition to create the entity indicator and automatically selected the nouns, verbs, adjectives, and adverbs in the definition within a given window size to the left and to the right of the ERT. We used a window size of four in our experiments.
An entity can have multiple definitions each explaining it using diverse vocabulary. On average, an entity in our corpus had 3 definitions. We create an entity indicator from each definition of the entity, hence an entity has multiple indicators. We call the collection of indicators of an entity as its entity model. In other words, an entity model consists of multiple entity indicators that capture diverse and orthogonal ways an entity can be expressed in the text.

Candidate Sentence Selection
The sentences with ERT in an input text are identified as candidate sentences containing implicit mention of the corresponding entity. A sentence may contain multiple ERTs and consequently become a candidate sentence for multiple entities. This step reduces the complexity of the classification task as now a sentence has only a few target entities.

Candidate Sentence Pruning
In order to evaluate the similarity between any given candidate sentence and the entity model, we perform a projection of candidate sentences onto the same semantic space. We perform this by pruning the terms in candidate sentences that does not participate in forming the segment with implicit entity mentions. Candidate sentences are pruned by fol-231 lowing the same steps followed to create the entity indicators from the entity definitions.

Semantic Similarity Calculation
As the last step, our solution calculates the similarity between the entity model and the pruned candidate sentence. The sentences with implicit entity mentions often use adjectives and adverbs to describe the entity and they may indicate the absence of the entities using antonyms or explicit negations. These two characteristics pose challenges to the applicability of existing text similarity algorithms such as MCS (Mihalcea et al., 2006) and matrixJcn (Fernando and Stevenson, 2008) which are proven to perform well among the unsupervised algorithms in paraphrase identification task (ACLWiki, 2014).
The existing text similarity algorithms largely benefit from the WordNet similarity measures. Most of these measures use the semantics of the hierarchical arrangement of the terms in WordNet. Unfortunately, adjectives and adverbs are not arranged in a hierarchy, and terms with different part of speech (POS) tags cannot be mapped to the same hierarchy. Hence, they are limited in calculating the similarity between terms of these categories. This limitation negatively affects the performance of IER as the entity models and pruned sentences often contain terms from these categories. Consider the following examples: 1. Her breathing is still uncomfortable adjective .
2. She is breathing comfortably adverb in room air.
3. His tip of the appendix was inflamed verb .
The first two examples use an adjective and an adverb to mention the entity 'shortness of breath' implicitly. The third example uses a verb to mention the entity 'appendicitis' implicitly instead of the noun inflammation that is used by its definition.
We have developed a text similarity measure overcoming these challenges and weigh the contributions of the words in the entity model to the similarity value based on their representative power.
Handling adjectives, adverbs and words with different POS tags: To get the best out of all Word-Net similarity measures, we exploited the relationships between different forms of the terms in Word-Net to find the noun form of the terms in the entity models and pruned sentences before calculating the similarity. We found the adjective for an adverb using relationship 'pertainym' and noun for an adjective or a verb using the relationship 'derivationally related form' in WordNet.
Handling negations: Negations are of two types: 1) Negations mentioned with explicit terms such as no, not, and deny, and 2) Negations indicated with antonyms (e.g., 2 nd example in above list). We used the NegEx algorithm (Chapman et al., 2001) to address the first type of negations. To address the second type of negations, we exploited the antonym relationships in the WordNet.
The similarity between an entity model and the pruned candidate sentence is calculated by computing the similarities of their terms. The term similarity is computed by forming an ensemble using the standard WordNet similarity measures namely, WUP (Wu and Palmer, 1994), LCH (Leacock and Chodorow, 1998), Resnik (Resnik, 1995), LIN (Lin, 1998), JCN (Jiang and Conrath, 1997), as well as a predict vector-based measure Word2vec (Mikolov et al., 2013) and a morphology-based similarity metric Levenshtein 1 as: where t 1 and t 2 are input terms and M is the set of above mentioned similarity measures. This ensemble-based similarity measure exploits orthogonal ways of comparing terms: semantic, statistical, and syntactic. An ensemble-based approach is preferable over picking one of them exclusively since they are complementary in nature, that is, each outperforms the other two in certain scenarios.
The similarity values calculated by WordNet similarity measures in sim m (t 1 , t 2 ) are normalized to range between 0 and 1.
The similarity of a pruned candidate sentence to the entity model is calculated by calculating its similarity to each entity indicator in the entity model, and picking the maximum value as the final similarity value for the candidate sentence. The similarity between entity indicator e and pruned sentence s, sim(e, s), is calculated by summing the similarities calculated for each term t e in the entity indicator weighted by its representative power as defined in eq. (1). If t e is an antonym for any term in s (t s ), it contributes negatively to the overall similarity value, else it contributes in linear portion of the maximum similarity value between t e and some t s (eqs. (4) and (5)). The overall similarity value is normalized based on the total representative power of all the terms t e s (eq. (1)) and ranges between -1 and +1.
Note that this formulation weighs the contribution of each term according to its importance in defining the entity. The higher similarity with a term that has higher representative power leads to higher overall similarity value, while the lower similarity with such terms leads to a lower total similarity value. The special treatment for antonyms takes care of the negated mentions of an entity.
sim(e, s) = te∈e f (t e , s) * r te te∈e r te (3) α(t e , s) = ts∈s 0 if t e is an antonym of t s 1 otherwise (5) Finally, the sentences are classified based on a configurable threshold values selected between -1 and +1.

Evaluation
We reannotated a sample of the corpus created for SemEval-2014task 7 (Pradhan et al., 2014 to include implicit mention annotations and measured the performance of our proposed method in classifying entities annotated with TP and Tneg mentions 2 .

Gold Standard Dataset
The SemEval-2014 task 7 corpus consists of 24,147 de-identified clinical notes. We used this corpus to create a gold standard for IER with the help of three domain experts. The gold standard consists of 857 2 We do not explicitly report performance on TN because our focus is to find sentences that contain entity mentions rather than those devoid of mentions.
We have annotated the corpus for explicit mentions of the entities using cTAKES (Savova et al., 2010) and ranked the entities based on their frequency.
The domain experts on our team then selected a subset of these entities that they judged to be frequently mentioned implicitly in clinical documents. For example, the frequent entity 'shortness of breath' was selected but not 'chest pain' since the former is mentioned implicitly often but not the latter. We used four frequently implicitly mentioned entities as the primary focus of our evaluation. We refer to these as primary entities from here on (the first four entities in Table 1). To test the generalizability of our method, as well as to evaluate its robustness when lacking training data, we selected another four entities (the last four entities in Table 1). We then selected a random sample of candidate sentences for each of these entities based on their ERTs and winnowed it down further by manually selecting a subset that exhibits syntactic diversity. Ultimately, our corpus consisted of 120-200 sentences for each primary entity and additional 80 sentences selected from the other four entities.
Each candidate sentence was annotated as TP e (contains a mention of entity e), Tneg e (contains a negated mention of entity e), or TN e (does not contain a mention of entity e). Each sentence was annotated by two domain experts, and we used the third one to break the ties. The Cohens' kappa value for the annotation agreement was 0.58. While the annotators have good agreement on annotating sentences in category TP, they agreed less on the categories Tneg and TN. The latter categories are indeed difficult to distinguish. For example, annotators often argue whether 'patient breathing at a rate of 15-20' 233 means the negation of entity 'shortness of breath' (because that is a normal breathing pattern) or just lacks a mention of the entity. The final annotation label for a sentence is decided based on majority voting. Table 1 shows the statistics of the annotated candidate sentences. The prepared data set is available at http://knoesis.org/ researchers/sujan/data-sets.html

Implicit Entity Recognition Performance
Since IER is a novel task, there are no baseline algorithms that can be directly applied such that it would yield a fair comparison with our algorithm. However, we deem some of the related algorithms to have good potential applicability for this task. Therefore, we included two strong algorithms from the closest related work as baseline solutions to the problem.
The first baseline is the well-known text similarity algorithm MCS (Mihalcea et al., 2006). MCS is one of the best performing unsupervised algorithms in paraphrase recognition task (ACLWiki, 2014). It uses an ensemble of statistical and semantic similarity measures, which is a preferable feature for the IER as opposed to one measure used by the ma-trixJcn (Fernando and Stevenson, 2008). Both MCS and our algorithm classify the candidate sentences based on threshold values selected experimentally.
To include also a supervised baseline, we trained an SVM (Cortes and Vapnik, 1995) one of the state-of-the-art learning algorithms, shown to perform remarkably well in a number of classification tasks. We trained separate SVMs for each primary entity, considering unigrams, bigrams, and trigrams as the features. It has been shown that SVM trained on ngrams performed well on text classification tasks (Pang et al., 2002) (Zhang and Lee, 2003). The SVMs trained with bigrams consistently produced the best results for the 4-fold cross validation. Therefore, our testing phase used the SVMs trained with the bigrams.
Preparation of training and testing datasets: We created training and testing datasets by splitting the dataset annotated for each primary entity as 70% (training) and 30% (testing). The training datasets were used to train the SVM models for each primary entity and to select the threshold values for both MCS and our algorithm.
The classification performance of each algorithm is studied in the TP and Tneg categories using precision, recall, and F-measure. The precision (PP) and recall (PR) for category TP at threshold t are defined as: Similarly, NP and NR for Tneg are defined as: where S T P and S T neg denote the sentences annotated with TP and Tneg respectively by domain experts and sim is the calculated similarity value for the pruned sentence.
Selecting threshold value: The threshold values for both MCS and our algorithm are selected based on their classification performance in the training dataset. The MCS algorithm produced the best F1 score for the TP category with a threshold value of 0.5, and for the Tneg category with a threshold value of 0.9, while our algorithm produced the best F1 for the TP category with 0.4 and for the Tneg category with 0.3. We examined threshold values that produce best F1 scores by the two algorithms by starting with 10% of the training data and gradually increasing the size of the training data. The threshold values with best F1 scores were stabilized after adding 30% of the training data. Hence, we could select the threshold values with just 50% of the training data.

Classification Performance
The first experiment evaluates the classification performance of our algorithm, MCS, and SVM.  Our algorithm outperforms the other unsupervised solution MCS, but the SVM was able to leverage supervision to outperform our algorithm in the TP category in terms of F-measure (P F 1 on Table 2). For example, the sentence 'he was placed on mechanical ventilation shortly after presentation' is annotated as TP in the gold standard for the entity 'shortness of breath' since 'mechanical ventilation' indicates the presence of 'shortness of breath'. This annotation requires domain knowledge that was not present in the entity definitions that we used to build entity models. However, with enough examples, the SVM was able to learn the importance of the bigram 'mechanical ventilation' and classify it as TP.
For the Tneg category, however, our algorithm outperforms the SVM (N F 1 on Table 2). This is due to the explicit treatment for the negated mentions by our algorithm to capture different variations of the negated mentions.
The MCS algorithm underperformed in both categories. We observed that this was mostly due to its limitations described in Section 3.5. The overall classification accuracy-the accuracy of classifying both TP and Tneg instances-of our algorithm, MCS, and SVM are 0.7, 0.4, and 0.7 respectively.  The second experiment evaluates the impact of including the similarity scores calculated by MCS and our algorithm for each candidate sentence as a feature to the best performing SVM model. Table 3 shows that the inclusion of MCS scores as a feature did not help to improve the SVM results. In fact, it negatively affected the results for the Tneg category. Since the MCS showed low precision for the Tneg category in the previous experiment (Table 2), it is potentially introducing too much noise that the SVM is not able to linearly separate. However, the similarity value calculated by our algorithm improves the SVM classifiers. It increased the precision and recall values for both the TP and Tneg categories. This shows that the similarity value calculated by our algorithm can be used as an effective feature for a learning algorithm that is designed to solve the IER problem. The overall classification accuracy of SVM, SVM+MCS, and SVM+Our con-figurations are 0.7, 0.7, and 0.74 respectively.
We were interested in exploring how much labeled data would be needed for supervised solution to outperform our unsupervised score alone. We analyzed the behavior of the three configurations of the SVM with our unsupervised approach with different training set sizes. Figure 2 shows the F1 values obtained by gradually increasing the size of the training dataset 3 , while testing on the same test set. The F1 value of our approach remains constant after 50% training data since it has already decided the threshold values. Figure 2 shows that the SVM trained with bigrams needs 76% of the training dataset to achieve the F1 value achieved by our unsupervised approach in the TP category, and it does not achieve the F1 achieved by our algorithm in Tneg category (note the crossing points of the line marked with 'X' and line marked with circles). Figure 2 also shows that the similarity score calculated by our algorithm complements the SVM at each data point. After adding our similarity score to the SVM as a feature, it achieved the F1 achieved by our unsupervised algorithm with 50% of the training data in the TP category and with 90% of the training data in the Tneg category (note the crossing points of the line marked with 'X' and line marked with '+'). Also, SVM+Our configuration achieved the best F1 value for SVM with just 70% of the training data in the TP category and with just 50% of the training data in the Tneg category. This shows that our similarity score can be used as an effective feature to reduce manual labeling effort and to improve the supervised learning algorithms to solve the IER problem.
Finally, to evaluate the generalization ability of our algorithm and to demonstrate its usage in situations with a lack of training data, we applied it to a set of 80 sentences selected for four new entities (the last four entities in Table 1). Our algorithm produced the following results for these entities when we classify their sentences with the threshold values selected using the training dataset created for the primary entities. and Tneg (right) categories with varying sizes of training datasets. These graphs show that the SVM trained with bigrams needs 76% of the training data to achieve the F1 score of our unsupervised method in the TP category while it does not achieve the F1 score of our algorithm in the Tneg category. This also shows that the similarity value calculated by our algorithm complements the SVM trained with bigrams at each data point and helps it to beat or perform on par with our algorithm. The paired T-test values calculated for SVM and SVM+Our configurations show that this is not a random behavior (t-T-test value, df-degree of freedom, p-probability value).
Although negation detection with NegEx is not a contribution of our work, our algorithm enables its application to IER. This is not possible for MCS. NegEx requires two inputs: 1) The sentence, and 2) The term being considered for possible negation. MCS does not detect the key term in the sentence, hence it is not possible to apply NegEx with MCS. However, our algorithm starts with identifying the ERT which is considered for possible negation.

Limitations
The candidate sentence selection based on the ERT can be seen as a limitation of our approach since it does not select sentences with implicit entity mentions that do not use the selected ERT. However, we do not expect this limitation to have a major impact. We asked our domain experts to come up with sentences that contain implicit mentions of the entity 'shortness of breath' without using its ERT 'breathing' or its synonyms ('respiration' and 'ventilation'). According to them, the sentences 'the patient had low oxygen saturation', 'the patient was gasping for air', and 'patient was air hunger' are such sentences (the emphasis indicates the phrases that imply 'shortness of breath'). However, we found only 113 occurrences of these phrases as opposed to 8990 occurrences of its ERTs in our corpus.

Conclusion and Future Work
We defined the problem of implicit entity recognition in clinical documents and proposed an unsupervised solution that recognizes the implicit mentions of entities using a model built from their definitions in a knowledge base. We showed that our algorithm outperforms the most relevant unsupervised method and it can be used as an effective feature for a supervised learning solution based on an SVM. The ability to capture the diverse ways in which an entity can be implicitly mentioned by exploiting their definitions with special treatment for two types of negations are the main strengths of our method.
In the future, we will explore the ability to detect the boundary of the phrases with implicit mentions, capture the sentences with implicit mentions without selected ERT, and investigate more intensive exploitation of domain knowledge for IER.