YEDDA: A Lightweight Collaborative Text Span Annotation Tool

In this paper, we introduce Yedda, a lightweight but efficient and comprehensive open-source tool for text span annotation. Yedda provides a systematic solution for text span annotation, ranging from collaborative user annotation to administrator evaluation and analysis. It overcomes the low efficiency of traditional text annotation tools by annotating entities through both command line and shortcut keys, which are configurable with custom labels. Yedda also gives intelligent recommendations by learning the up-to-date annotated text. An administrator client is developed to evaluate annotation quality of multiple annotators and generate detailed comparison report for each annotator pair. Experiments show that the proposed system can reduce the annotation time by half compared with existing annotation tools. And the annotation time can be further compressed by 16.47% through intelligent recommendation.


Introduction
Natural Language Processing (NLP) systems rely on largescale training data for robust accuracies. However, annotation can be time-consuming and expensive. Despite detailed annotation standards and rules, inter-annotator disagreement is inevitable because of human mistakes, language phenomenons which are not covered by the annotation rules and the ambiguity of language itself. Existing annotation tools (Morton and LaCivita, 2003;Cunningham et al., 2002) mainly focus on user annotation process but rarely consider the post-annotation quality analysis, which is necessary due to the inter-annotator disagreement. In addition to the annotation quality, efficiency is also critical in large-scale annotation task, while it is less optimized in many annotation tools (Stenetorp et al., 2012;Ogren, 2006). To address the challenges above, we propose YEDDA, a lightweight and efficient annotation tool for text span annotation. A snapshot is shown in Figure 2. Here a text span boundary is selected and assigned with a label, which can be useful for Named Entity Recognition (NER) (Tjong Kim Sang and De Meulder, 2003), word segmentation (Sproat and Emerson, 2003), chunking (Tjong Kim Sang and Buchholz, 2000) .etc. To keep annotation efficient and accurate, YEDDA provides a systematic solution across the whole annotation process, which includes the shortcut annotation, batch annotation with command line, intelligent recommendation, format exporting and administrator evaluation/analysis. Figure 1 shows the general framework of YEDDA. It offers annotators with a clear and efficient Graphical User Interface (GUI) to annotate raw text. For the administrator, it provides two useful toolkits to evaluate multi-annotated text and generate detailed comparison report for each annotator pair. The main advantages of our tool are listed: • Efficient: it supports both shortcut and command line model to accelerate the annotation process.
• Intelligent: it offers user with real-time system suggestion to reduce duplicated annotation.

Admin Toolkits
Annotator Interface  • Comprehensive: it integrates useful toolkits to give statistical index of multi-user collaborative annotation results and generate detailed content comparison for annotator pairs. This paper is organized as follows: Section 2 gives an overview of previous text annotation tools and the comparison with ours. Section 3 describes the architecture of YEDDA and its detail functions. Finally, Section 4 concludes this paper and give the future plans.

Related Work
There exist many text span annotation tools which focus on different aspects of the annotation process. Stanford manual annotation tool 1 is a lightweight tool but with limited function. Knowtator (Ogren, 2006) is a general-task annotation tool which links to a biomedical onto ontology to help identify named entities and relations, it supports quality control during the annotation process by integrating inter-annotator evaluation and analysis. Besides, GATE 2 1 http://nlp.stanford.edu/software/ stanford-manual-annotation-tool-2004-05-16. tar.gz 2 GATE is a general NLP tool which includes annotation function.
(Bontcheva et al., 2013) includes a web-based with collaborative annotation framework which allows users to work collaboratively by annotating online with shared text storage. Unlike the tools mentioned above, WordFreak (Morton and LaCivita, 2003) adds a system recommendation function and brings the thought of active learning to rank the unannotated sentences based on the recommend confidence.
BRAT (Stenetorp et al., 2012) is one of the most popular text annotation tools in recent years, it supports both online and local model and provides powerful annotation functions and rich visualization function. However, its online annotation is slow and local annotation requires complex system configuration. Table 1 concludes the differences of YEDDA with above annotation tools, where "Self Consistency" represents whether the tool works independently or it relies on preinstalled packages. YEDDA provides a lighter but more systematic choice with more flexibility, efficiency and less dependence on system environment for text span annotation. Besides, YEDDA offers administrator useful toolkits for evaluating the annotation quality and analyze the disagreements within annotators.

YEDDA
YEDDA provides a simple and clear interface for annotators and useful analysis toolkits for the administrator. It is developed based on standard Python GUI library Tkinter 3 , and hence does not need complex environment configuration and is compatible with all Operating System (OS) platforms with Python installation. It offers two user-friendly interfaces for annotators and administrator, respectively, which are introduced in detail in Section 3.1. and Section 3.2., respectively.

Annotator Client
The client is designed to accelerate the annotation process as much as possible. It supports shortcut annotation to reduce the user operation time. Command line annotation is designed to annotate multi-span in batch. Besides, the client provides system recommendations to lessen the workload of duplicated span annotation. Figure 2 shows the interface of annotator client on an English entity annotation file. The interface consists of 5 parts. The working area in the up-left which shows the texts with different colors (blue: annotated entities, green: recommended entities and orange: selected text span). The entry at the bottom is the command line which accepts annotation command. There are several control buttons in the middle of the interface, which are used to set annotation model. The status area is below the control buttons, it shows the cursor position and the status of recommending model. The right side shows the shortcut map, which can be configured 3.1.1. Shortcut Key Annotation YEDDA provides the function of annotating text span by selecting using mouse and press shortcut key to map the selection into a specific label. It is a common annotation process in many annotation tools (Stenetorp et al., 2012;Bontcheva et al., 2013). It binds each label with one custom shortcut key, this is shown in the "Shortcuts map Labels" part of Figure 2. The annotator needs only two steps to annotate one text span, i.e. "select and press". The annotated file updates simultaneously with each key pressing process.

Command Line Annotation
YEDDA also support the command line annotation function (see the command entry in the bottom of Figure 2) which can execute multi-span annotation at once. The system will parse the command automatically and convert the command into multi-span annotation instructions and execute in batch. It is quite efficient for character-based languages, such as Chinese and Japanese. The command follows a simple rule which is "n1 + key1 + n2 + key2 + n3 + key3 + ... , where 'n1, n2, n3' are the length of the entities and 'key1, key2, key3' is the corresponding shortcut key. For example, command "2a3d2b" represents annotating following 2 characters as label 'a' (mapped into a specific label name), the following 3 characters as label 'd' and 2 characters further as label 'b'.

System Recommendation
It has been proven that using pre-annotated text and manual correction increases the annotation efficiency in many annotation tasks (Meurs et al., 2011). YEDDA offers annotators with system recommendation and correction function. Algorithm 1 shows the forward maximum matching algorithm used in our recommending system on an entity annotation example. It parses the annotated text part a and adds the annotated entities into a dictionary d. For the unannotated sentence sent, recommending system scans the sentence from the left to right with an span with initial length ml, the entity will be added into an entity list s if the entity can be found in d, otherwise the last character of the span will be stripped and rematch. The final predicted s is returned and colored in the annotation interface, as the green part shown in Figure 2. Annotators can use shortcut to confirm, correct or veto the suggestions. The recommending system keeps online updating during the whole annotation process, which learns the up-to-date annotation information. The recommending system is designed as "pluggable" which ensures that the recommending algorithm can be easily extended into other sequence labeling models, such as Conditional Random Field (CRF) 5 (Lafferty et al., 2001). The recommendation can be controled through two buttons "RMOn" and "RMOff", which enables and disables the recommending function, respectively.

Annotation Modification
It is inevitable that the annotator or the recommending system gives incorrect annotations or suggestions. Based on our annotation experience, we found that the time cost of annotation correction can not be neglected. Therefore, YEDDA provides several efficient modification actions to revise the annotation: • Action Withdraw: annotators can cancel their previous action and let system return to the last status by press the shortcut key Ctrl+z.
• Span Label Modification: if the selected span was in right boundary but got an incorrect label, annotator only needs to put the cursor inside the span (or select the span) and press the shortcut key of the right label to correct label.
• Label Deletion: when the plain text was incorrectly annotated (recommended) with a label by annotator (system), similar to the label modification, the annotator can put the cursor inside the span and press shortcut key q to remove the annotated (recommended) label.

Export Annotated Text
As the annotated file is saved in .ann format, YEDDA provides the "Export" function which exports the annotated text as standard format (ended with .anns). Each line includes one word/character and its label, sentences are separated by an empty line. The exported label can be chosen in either BIO or BIOES format (Ratinov and Roth, 2009).

Administrator Toolkits
For the administrator, it is important and necessary to evaluate the quality of annotated files and analyze the detailed disagreements of different annotators. Shown as Figure 3, YEDDA provides a simple interface with several toolkits for administrator monitoring the annotation process.

Multi-Annotator Analysis
To evaluate and monitor the annotation quality of different annotators, our Multi-Annotator Analysis (MAA) toolkit imports all the annotated files and gives the analysis results in a matrix. As shown in Figure 4, the matrix gives the Fvalues in full level (consider both boundary and label accuracy) and boundary level (ignore the label correctness, only care about the boundary accuracy) of all annotator pairs.

Pairwise Annotators Comparison
If an administrator wants to look into the detailed disagreement of annotators, it is quite convenient by using the Pairwise Annotators Comparison (PAC). PAC loads two annotated files and generates a specific comparison report file 6 for the two annotators. As shown in Figure 5, the report is mainly in two parts: • Overall Statistics: it shows the specific precision, recall and F-measure value 7 of two files in all labels. It also gives the three accuracy indexes on overall full level and boundary level in the end.
• Content Comparison: this function gives the detailed comparison of two annotated files in whole content. It highlights the annotated parts of two annotators and assigns different color for the agreed and disagreed span. 6 The report is generated in .tex format and can be complied into .pdf file. 7 Notice that we assume "File1" as a gold standard, this only affects the order of precision and recall, while the F-value keeps same if we choose the other file as gold standard.

SUTDNLP Group
Singapore University of Technology and Design But the group was actually farther away over the weekend, moving through the Sunda Strait into the Indian Ocean . The US military's Pacific Command said on Tuesday that it had cancelled a port visit to Perth , but had completed previously scheduled training with Australia off its northwest coast after departing Singapore on 8 April.
The strike group was now "proceeding to the Western Pacific as ordered". It is not clear whether the failure to arrive was a deliberate deception, perhaps designed to frighten North Korea's leader Kim Jong-un , a change of plan

Conclusion and Future Work
We have presented a lightweight but systematic annotation tool, YEDDA, for annotating the entities in text and analyzing the annotation results efficiently. The source codes of this tool are released at https://github. com/jiesutd/SUTDAnnotator. In order to reduce the workload of annotators, we are going to integrate active learning strategy in our system recommendation part in the future. A supervised sequence labeling model (such as CRF) is trained based on the annotated text, then unannotated sentences with less confidence (predicted by this model) are reordered in the front to ensure annotators only annotate the most confusing sentences.