WNUT 2020 Shared Task-1: Conditional Random Field(CRF) based Named Entity Recognition(NER) for Wet Lab Protocols

The paper describes how classifier model built using Conditional Random Field detects named entities in wet lab protocols.


Introduction
Wet laboratories are laboratories for conducting biology and chemistry experiments. These require handling of various types of chemicals and potential "wet" hazards. These experiments are guided by a sequence of instructions collectively referred as wet lab protocols.
The instructions are mostly composed of imperative statements which are meant to describe an action. Figure 1 shows a representative wet lab protocol. Figure 2 shows BRAT annotations (entities and relations) on two sentences from the representative protocol. For each protocol, annotators had identified and marked every span of text corresponding to action or one of the 17 types of entities. To remove template DNA, add 30 l nuclease-free water to each 20 l reaction, followed by 2 l of DNase I (RNase-free), mix and incubate for 15 minutes at 37C. Proceed with purification of synthesized RNA or analysis of transcription products by gel electrophoresis.

Named Entity Recognition Methodology
A Conditional Random Fields (CRF) classifier was trained to recognize named entities. The CRF NER model was implemented using sklearn-crfsuite 2 which is a Python wrapper over C++ based CRFsuite 3 . It utilized L-BFGS [3], a limited memory quasi-Newton algorithm for large scale numerical optimization. The classifier was trained with both L1 and L2 regularization.

Features
Three types of features have been extracted using Python library spaCy [4].

Experiments
The experiments were based on the datasets provided by the organizers of W-NUT 2020 shared task on Entity and Relation Recognition over Wet Lab Protocols [5]. The dataset ( Table 2) was annotated in both StandOff and CoNLL formats. Entities and relations of 615 protocols were annotated in brat with 3 annotators with 0.75 inter-annotator agreement, measured by span-level Cohen's Kappa.

Results
For the experiments, the classifier was trained on the training data and evaluated on development and test data. The reported averages are defined as follows: • Macro average: averaging the unweighted mean per label.
• Micro average: averaging the total true positives, false negatives and false positives.
• Weighted average: averaging the supportweighted mean per label Table 3 and Table 4 show results at token and entity level respectively.

Error Analysis
Entity type wise performance metrics on development dataset are available in Table 6. This is based on strict evaluation mode of matching as defined in SemEval'13 [6]. As per strict evaluation, a predicted entity is correct only if it matches with goldstandard in both exact boundary and type. Used seqeval 4 for the evaluation. Table 7 shows the poorly performing entity classes along with their frequent confusers.
Errors are of primarily two types: 4 https://github.com/chakki-works/ seqeval   • Predicted entity text span matches truth but entity class is incorrect.
-Partial match: Example shown in Table  8. -Complete mis-match: Example shown in Table 9. Table 8 and Table 9 show examples of misclassification for the highlighted text portion of the corresponding sentences. water. Whereas the system predicted a single entity(Reagent) over the entire text span.    The entity's text span is covered by the subtree having Solution as its root and of as its head.

Conclusion
This paper has proposed a CRF-based named entity extraction system to extract Action and 17 Entities of wet lab protocols. Future plan: • Analyse the errors in more detail and extract richer features.
• Extract global structured information features of the dependency trees [7] as shown in Figure 4. Currently as the system only uses local dependency features, it predicts Fe Stock Solution as Reagent entity and misses 10g/L.