Coreference resolution is the task of clustering together multiple mentions of the same entity appearing in a textual document (e.g. Joe Biden, the U.S. President and he). This CodaLab-powered shared task deals with multilingual coreference resolution and is associated with the CRAC 2022 Workshop (the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference) held at COLING 2022.
IMPORTANT DATES
- May 17, 2022 - development evaluation possible via CodaLab
- June 1, 2022 - start of the evaluation phase (test evaluation via CodaLab)
- June 8, 2022 (AoE) - end of evaluation phase
- July 11, 2022 - submission of system description papers
- October 16-17, 2022 – the CRAC workshop at COLING
BACKGROUND
Recently, inspired by the Universal Dependencies initiative (UD), the coreference community has started discussions on establishing a universal annotation scheme and using it to harmonize existing corpora. The discussions at the CRAC 2020 workshop led to proposing the Universal Anaphora initiative. One of the lines of effort related to Universal Anaphora resulted in CorefUD, which is a multilingual collection of coreference data resources harmonized under a common scheme. The public edition of CorefUD 1.0 contains 13 datasets for 10 languages, namely Catalan, Czech (2×), English (2×), French, German (2×), Hungarian, Lithuanian, Polish, Russian, and Spanish. The CRAC 2022 shared task deals with coreference resolution in all these languages.
The file format used in CorefUD 1.0 represents coreference using the bracketing notation inspired by the CoNLL-2011 and CoNLL-2012 shared tasks, and inserts it into the MISC column of the CoNLL-U, the file format used in UD. The content of the other columns is fully compatible with morphological and syntactic annotations of the UD framework in CorefUD (with, for instance, automatically parsed trees added to resources that miss manual syntactic annotations). Thus, the shared task participant can easily employ UD-style morphosyntactic features for coreference prediction for all resources in a unified way, if they want to. CorefUD tokenization is UD-compliant, too.
TASK DESCRIPTION
The main rules of the CRAC 2022 shared task are the following:
- Shared task participants are supposed to both (a) identify mentions in texts and (b) predict which mentions belong to the same coreference cluster (i.e., refer to the same entity or event).
- Training and development data will be published first; evaluation data (without gold annotations) will be publicly available only after the beginning of the evaluation phase and must not be used for improving the models.
- Participants are expected to deliver their submissions exclusively via CodaLab; a submission must have the form of a zip file containing test set files with predicted coreference, ideally for all 13 CorefUD datasets; however, participants who are unable to predict coreference for all CorefUD datasets are encouraged to submit at least a subset of test set files.
- Technically, ‘files with predicted coreference’ means that coreference attributes using the CorefUD notation are filled into the MISC column of test set CoNLL-U files.
- In this shared task, only identity coreference is supposed to be predicted (even if some of the CorefUD datasets contain annotation of bridging).
- There is a single official evaluation criterion that will be used for the main ranking of all submissions within the evaluation phase. In other words, there are no subtasks delimited within this shared task. We define the criterion as the arithmetic mean (macro-average) of the CoNLL score (an average of the F1 values of MUC, B-cubed and CEAFe scores) across the 13 datasets.
- Even if there are no subtasks declared, additional evaluation criteria might be evaluated for all submissions and presented by the organizers (for instance, secondary rankings of submissions according to scores reached for individual languages).
- A deep-learning-based baseline system will be available to participants, and it is up to their decision whether they start developing their system from scratch, or by incremental improvements of this baseline.
- After the evaluation period, participants will be invited to submit their system description papers to the CRAC 2022 workshop.
REGISTRATION
If you are interested in participating in this shared task, please fill the registration form as soon as possible.
Technically, this registration will not be connected with participants' CodaLab accounts in any way. In other words, it will be possible to upload your CodaLab submissions without being registered here. However, we strongly recommend that at least one person from each participating team fills this registration form so that we can keep you informed about all updates regarding the shared task.
In addition, you can send any questions about the shared task to the organizers via corefud [at] googlegroups.com.
ORGANIZERS
- Charles University (Prague, Czechia): Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Daniel Zeman
- Institute of Computer Science, Polish Academy of Sciences (Warsaw, Poland): Maciej Ogrodniczuk
- Georgetown University (Washington D.C., USA): Yilun Zhu
- University of West Bohemia (Pilsen, Czechia): Miloslav Konopík, Ondřej Pražák, Jakub Sido