shared task

Shared task on Multilingual Grammatical Error Detection

Abbreviated Title: 
MultiGED-2023
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Location: 
Event Dates: 
Monday, 23 January 2023 to Monday, 22 May 2023
Submission Deadline: 
Friday, 3 March 2023
Contact: 
Elena Volodina
MultiGED-2003 team
Contact Email: 
elena.volodina [at] svenska.gu.se
multiged-2023 [at] googlegroups.com
City: 
Torshavn
State: 
Country: 
Faroe Islands
Description/Details: 

Call for participation - shared task on Multilingual Grammatical Error Detection (MultiGED-2023) on Czech, English, German, Italian and Swedish

Official website for the shared task: https://github.com/spraakbanken/multiged-2023

The Computational SLA working group invites you to participate in the first shared task on Multilingual Grammatical Error Detection, MultiGED-2023, which includes five languages: Czech, English, German, Italian and Swedish.

The aim of this shared task is to detect tokens in need of correction across five different languages, labeling them as either correct ("c") or incorrect ("i"), i.e. performing binary classification at the token level. You can work on one of the provided languages or any combination of languages.
More details about the task: https://github.com/spraakbanken/multiged-2023

The shared task is part of the NLP4CALL workshop, which will take place on 22 May 2023, co-located with the NoDaLiDa conference to be held in the Faroe Islands. Accepted papers with systems descriptions will be published in the workshop proceedings and double-published through the ACL anthology.

Timeline:

23 January, 2023 - first call for participation. Training and validation data released, CodaLab opens for team registrations.
14 February, 2023 - second call/reminder
27 February, 2023 - test data released
03 March, 2023 - system submission deadline (system output)
10 March, 2023 - results announced
03 April, 2023 - paper submission deadline with system descriptions. We encourage you to share models, code, fact sheets, extra data, etc. with the community through github or other repositories on paper publication.
21 April, 2023 - paper reviews sent to the authors
01 May, 2023 - camera-ready deadline
22 May, 2023 - presentations of the systems at NLP4CALL workshop

To register for/express interest in the shared task, please fill in this form: https://forms.gle/DgwTNmTCQhsmrbxq6
To ask questions and to get important information and updates about the shared task, please join the MultiGED-2023 Google Group: https://groups.google.com/g/multiged-2023
Official system evaluation will be carried out on CodaLab: https://codalab.lisn.upsaclay.fr/competitions/9784

Organizers:

Elena Volodina, University of Gothenburg, Sweden
Chris Bryant, University of Cambridge, UK
Andrew Caines, University of Cambridge, UK
Orphee De Clercq, Ghent University, Belgium
Jennifer-Carmen Frey, EURAC Research, Italy
Elizaveta Ershova, JetBrains, Cyprus
Alexandr Rosen, Charles University, Czech Republic
Olga Vinogradova, Independent researcher, Israel

SemEval-2023 Task 2 - MultiCoNER II - 2nd Call for Participation

Abbreviated Title: 
MultiCoNER II
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
Event Dates: 
Sunday, 1 January 2023 to Tuesday, 31 January 2023
Submission Deadline: 
Tuesday, 31 January 2023
Contact: 
Sudipta Kar
Contact Email: 
sudipkar [at] amazon.com
City: 
State: 
Country: 
Description/Details: 

We cordially invite you to participate in the SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II).

The task is being held as part of SemEval-2023, and all participating team will be able to publish their system description paper in the proceedings published by ACL.

This task focuses on the fine-grained detection of complex entities, such as e-commerce products (movies, books, music and product titles), in low context settings (short and uncased text).

The task provides data in 12 language. Here are some examples in different languages where entities are enclosed inside brackets with their type:

  • English: [wes anderson | Artist]'s film [the grand budapest hotel | VisualWork] opened the festival .
  • Spanish: fue superado por el [aon center | Facility] de [los ángeles | HumanSettlement] .
  • Ukranian: назва альбому походить з роману « [кінець дитинства | WrittenWork] » англійського письменника [артура кларка | Artist] .
  • Swedish: [tom hamilton | Artist] amerikansk musiker basist i [aerosmith | MusicalGRP] .
  • Portuguese: também é utilizado para se fazer [licor | Drink] e [vinhos | Drink].
  • Hindi: १७९६ में उन्हें [शाही स्वीडिश विज्ञान अकादमी | Facility] का सदस्य चुना गया।
  • French: l [amiral de coligny | Politician] réussit à s y glisser .
  • German: in [frühgeborenes | Disease] führt dies zu [irds | Symptom] .
  • Bangla [লিটল মিক্স | MusicalGrp] এ যোগদানের আগে তিনি [পিৎজা হাট | ORG] এ ওয়েট্রেস হিসাবে কাজ করেছিলেন।
  • Italian: è conservato nel [rijksmuseum | Facility] di [amsterdam | HumanSettlement] .
  • Chinese: 它的纤维穿过 [锁骨 | AnatomicalStructure] 并沿颈部侧面倾斜向上和内侧.
  • Farsi: است] ناگویا |HumanSettlement] مرکزاین استان شهر

Additionally, a multilingual NER track is also offered for multilingual systems that can process all languages. The task focuses on detecting semantically ambiguous and complex entities in short and low-context settings. Participants are welcome to build NER systems for any number of languages. And we encourage to aim for a bigger challenge of building NER systems for multiple languages.

We have released training data for 12 languages along with a baseline system to start with. Participants can submit their system for one language but are encouraged to aim for a bigger challenge and build multi-lingual NER systems.

Important Links

Shared task schedule

  • Evaluation start: January 10, 2023
  • Evaluation end: by January 31, 2023 (latest date; task organizers may choose an earlier date)
  • System description paper submissions due: February 1, 2023
  • Notification to authors: March 1, 2023

Task organizers

  • Shervin Malmasi (Amazon)
  • Besnik Fetahu (Amazon)
  • Sudipta Kar (Amazon)

Please reach out to the organizers at multiconer-semeval-organizers [at] googlegroups.com, or join the Slack workspace to connect with the other participants and organizers.

Call for Submissions - MultiCoNER II (SemEval-2023 Task 2)

Abbreviated Title: 
MultiCoNER II
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Event Dates: 
Sunday, 1 January 2023 to Tuesday, 31 January 2023
Submission Deadline: 
Tuesday, 31 January 2023
Contact: 
Sudipta Kar
Contact Email: 
sudipkar [at] amazon.com
Description/Details: 

We cordially invite you to participate in the SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER II).

The task is being held as part of SemEval-2023, and all participating team will be able to publish their system description paper in the proceedings published by ACL.

This task focuses on the fine-grained detection of complex entities, such as e-commerce products (movies, books, music and product titles), in low context settings (short and uncased text).

The task provides data in 12 language. Here are some examples in different languages where entities are enclosed inside brackets with their type:

  • English: [wes anderson | Artist]'s film [the grand budapest hotel | VisualWork] opened the festival .
  • Spanish: fue superado por el [aon center | Facility] de [los ángeles | HumanSettlement] .
  • Ukranian: назва альбому походить з роману « [кінець дитинства | WrittenWork] » англійського письменника [артура кларка | Artist] .
  • Swedish: [tom hamilton | Artist] amerikansk musiker basist i [aerosmith | MusicalGRP] .
  • Portuguese: também é utilizado para se fazer [licor | Drink] e [vinhos | Drink].
  • Hindi: १७९६ में उन्हें [शाही स्वीडिश विज्ञान अकादमी | Facility] का सदस्य चुना गया।
  • French: l [amiral de coligny | Politician] réussit à s y glisser .
  • German: in [frühgeborenes | Disease] führt dies zu [irds | Symptom] .
  • Bangla [লিটল মিক্স | MusicalGrp] এ যোগদানের আগে তিনি [পিৎজা হাট | ORG] এ ওয়েট্রেস হিসাবে কাজ করেছিলেন।
  • Italian: è conservato nel [rijksmuseum | Facility] di [amsterdam | HumanSettlement] .
  • Chinese: 它的纤维穿过 [锁骨 | AnatomicalStructure] 并沿颈部侧面倾斜向上和内侧.
  • Farsi: است] ناگویا |HumanSettlement] مرکزاین استان شهر

Additionally, a multilingual NER track is also offered for multilingual systems that can process all languages. The task focuses on detecting semantically ambiguous and complex entities in short and low-context settings. Participants are welcome to build NER systems for any number of languages. And we encourage to aim for a bigger challenge of building NER systems for multiple languages.

We have released training data for 12 languages along with a baseline system to start with. Participants can submit their system for one language but are encouraged to aim for a bigger challenge and build multi-lingual NER systems.

Important Links

Shared task schedule

  • Evaluation start: January 10, 2023
  • Evaluation end: by January 31, 2023 (latest date; task organizers may choose an earlier date)
  • System description paper submissions due: February 1, 2023
  • Notification to authors: March 1, 2023

Task organizers

  • Shervin Malmasi (Amazon)
  • Besnik Fetahu (Amazon)
  • Sudipta Kar (Amazon)

Please reach out to the organizers at multiconer-semeval-organizers [at] googlegroups.com, or join the Slack workspace to connect with the other participants and organizers.

WMT 2022 shared task on code-mixed machine translation (MixMT)

ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Submission Deadline: 
Sunday, 16 October 2022
Contact: 
Vivek Srivastava
Mayank Singh
Contact Email: 
srivastava.vivek2 [at] tcs.com
singh.mayank [at] iitgn.ac.in
Description/Details: 

Dear All,

We are organizing the WMT 2022 shared task on code-mixed machine translation (MixMT).

In this shared task, we are running two subtasks involving a code-mixed language i.e. Hinglish (code-mixing of Hindi and English). A brief description of both the subtasks is given below:

Monolingual to code-mixed machine translation (Subtask-1): In this subtask, Hindi and English are the two source languages and the target language is Hinglish. The source Hindi and English sentences are translations of each other. The Hindi language sentences are written in the Devanagari script whereas the target Hinglish language text is written in the Roman script.

Code-mixed to monolingual machine translation (Subtask-2): In this subtask, Hinglish is the source language and the target language is English. Both the English and Hinglish text are written in the Roman script.

More details about the competition are available on the competition website: https://codalab.lisn.upsaclay.fr/competitions/2861

The timeline is:
Apr 01, 2022: The training + validation phase starts.
Jul 15, 2022: The training + validation phase ends.
Jul 16, 2022: The test phase starts.
Jul 30, 2022: The test phase ends.
Sept 7, 2022: Paper submission deadline.
Oct 9, 2022: Notification/feedback.
Oct 16, 2022: Camera-ready submission.

We look forward for a great participation.

VarDial Evaluation Campaign 2022

Abbreviated Title: 
VarDial 2022
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
COLING 2022
Event Dates: 
Wednesday, 12 October 2022 to Monday, 17 October 2022
Submission Deadline: 
Wednesday, 6 July 2022
Contact: 
Yves Scherrer
Contact Email: 
yves.scherrer [at] helsinki.fi
City: 
Gyeongju
State: 
Country: 
Republic of Korea
Description/Details: 

Within the scope of the ninth VarDial workshop, co-located with COLING 2022, we are organizing an evaluation campaign on similar languages, varieties and dialects with three shared tasks. To participate and to receive the training data please fill the registration form available on the workshop website:
https://sites.google.com/view/vardial-2022/shared-tasks

The tasks we are organizing this year are the following (please check the website for more information):

1. French Cross-Domain Dialect Identification (FDI)

In the 2022 French Dialect Identification (FDI) shared task, participants have to train a model on news samples collected from a set of publication sources and evaluate it on news samples collected from a different set of publication sources. Not only the sources are different, but also the topics. Therefore, participants have to build a model for a cross-domain 4-way classification by dialect task, in which a classification model is required to discriminate between the French (FH), Swiss (CH), Belgian (BE) and Canadian (CA) dialects across different news samples. The corpus is divided into training, validation and test, such that the publication sources and topics are distinct across splits. The training set contains 358,787 samples. The development set is composed of 18,002 samples. Another set of 36,733 samples are kept for the final evaluation.

2. Identification of Languages and Dialects of Italy (ITDI)

We provide participants with Wikipedia dumps (“pages-articles-multistream.xml.bz2”, from 01.03.2022) of 11 languages and dialects of Italy for training (Piedmontese, Venetian, Sicilian, Neapolitan, Emilian-Romagnol, Tarantino, Sardinian, Ligurian, Friulian, Ladin, Lombard). The Standard Italian raw Wikipedia dump may also be used as training data, but there will not be any instances of Standard Italian in the development and test sets. Please use the provided script to download (and extract, if you wish) the dumps to make sure you work with the correct kind and date of the dump.
The task is classification, i.e. the model is required to discriminate between different language varieties. As the training data is provided in the form of raw Wikipedia dumps, careful pre-processing of the data is part of the task. The task is closed, therefore, participants are not allowed to use external data to train their models. Exceptions are off-the-shelf pre-trained language models from the HuggingFace model hub or similar, the use of which has to be clearly stated. The test set will contain newly collected text samples of a subset of the given language varieties for training. The systems will be evaluated on sentence level.

3. Dialectal Extractive Question Answering (DialQA)

The Dialectal Extractive Question Answering Shared Task invites participants to build QA systems that are robust to dialectal variation. The task builds on existing QA benchmarks (TyDi-QA and SD-QA): specifically, it uses portions of the SD-QA dataset, which recorded dialectal variations of TyDi-QA questions. The participants may either (a) use the baseline automatic speech recognition outputs for each dialect with the aim of making a robust text-based QA system, or (b) they may use the provided audio recordings of the questions with the aim of making a dialect-robust ASR system which can be then evaluated with a baseline QA system, or (c) both of the above. The shared task provides development and test data for 5 varieties of English (Nigeria, USA, South India, Australia, Philippines), 4 varieties of Arabic (Algeria, Egypt, Jordan, Tunisia), and 2 varieties of Kiswahili (Kenya, Tanzania), as well as code for training baseline systems with modified TyDi-QA data.

The test sets will be released on June 30 and submissions are due on July 6. The system description papers will be due on July 29.

Of course, VarDial also accepts research papers focusing on computational methods and language resources for closely related languages, language varieties, and dialects. The full call for papers can be found here:
https://sites.google.com/view/vardial-2022/call-for-papers

CRAC 2022 Shared Task on Multilingual Coreference Resolution

Abbreviated Title: 
CorefUD 2022
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Submission Deadline: 
Wednesday, 8 June 2022
Contact: 
Maciej Ogrodniczuk
Contact Email: 
corefud [at] googlegroups.com
Description/Details: 

Coreference resolution is the task of clustering together multiple mentions of the same entity appearing in a textual document (e.g. Joe Biden, the U.S. President and he). This CodaLab-powered shared task deals with multilingual coreference resolution and is associated with the CRAC 2022 Workshop (the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference) held at COLING 2022.

IMPORTANT DATES

  • May 17, 2022 - development evaluation possible via CodaLab
  • June 1, 2022 - start of the evaluation phase (test evaluation via CodaLab)
  • June 8, 2022 (AoE) - end of evaluation phase
  • July 11, 2022 - submission of system description papers
  • October 16-17, 2022 – the CRAC workshop at COLING

BACKGROUND

Recently, inspired by the Universal Dependencies initiative (UD), the coreference community has started discussions on establishing a universal annotation scheme and using it to harmonize existing corpora. The discussions at the CRAC 2020 workshop led to proposing the Universal Anaphora initiative. One of the lines of effort related to Universal Anaphora resulted in CorefUD, which is a multilingual collection of coreference data resources harmonized under a common scheme. The public edition of CorefUD 1.0 contains 13 datasets for 10 languages, namely Catalan, Czech (2×), English (2×), French, German (2×), Hungarian, Lithuanian, Polish, Russian, and Spanish. The CRAC 2022 shared task deals with coreference resolution in all these languages.

The file format used in CorefUD 1.0 represents coreference using the bracketing notation inspired by the CoNLL-2011 and CoNLL-2012 shared tasks, and inserts it into the MISC column of the CoNLL-U, the file format used in UD. The content of the other columns is fully compatible with morphological and syntactic annotations of the UD framework in CorefUD (with, for instance, automatically parsed trees added to resources that miss manual syntactic annotations). Thus, the shared task participant can easily employ UD-style morphosyntactic features for coreference prediction for all resources in a unified way, if they want to. CorefUD tokenization is UD-compliant, too.

TASK DESCRIPTION

The main rules of the CRAC 2022 shared task are the following:

  • Shared task participants are supposed to both (a) identify mentions in texts and (b) predict which mentions belong to the same coreference cluster (i.e., refer to the same entity or event).
  • Training and development data will be published first; evaluation data (without gold annotations) will be publicly available only after the beginning of the evaluation phase and must not be used for improving the models.
  • Participants are expected to deliver their submissions exclusively via CodaLab; a submission must have the form of a zip file containing test set files with predicted coreference, ideally for all 13 CorefUD datasets; however, participants who are unable to predict coreference for all CorefUD datasets are encouraged to submit at least a subset of test set files.
  • Technically, ‘files with predicted coreference’ means that coreference attributes using the CorefUD notation are filled into the MISC column of test set CoNLL-U files.
  • In this shared task, only identity coreference is supposed to be predicted (even if some of the CorefUD datasets contain annotation of bridging).
  • There is a single official evaluation criterion that will be used for the main ranking of all submissions within the evaluation phase. In other words, there are no subtasks delimited within this shared task. We define the criterion as the arithmetic mean (macro-average) of the CoNLL score (an average of the F1 values of MUC, B-cubed and CEAFe scores) across the 13 datasets.
  • Even if there are no subtasks declared, additional evaluation criteria might be evaluated for all submissions and presented by the organizers (for instance, secondary rankings of submissions according to scores reached for individual languages).
  • A deep-learning-based baseline system will be available to participants, and it is up to their decision whether they start developing their system from scratch, or by incremental improvements of this baseline.
  • After the evaluation period, participants will be invited to submit their system description papers to the CRAC 2022 workshop.

REGISTRATION

If you are interested in participating in this shared task, please fill the registration form as soon as possible.

Technically, this registration will not be connected with participants' CodaLab accounts in any way. In other words, it will be possible to upload your CodaLab submissions without being registered here. However, we strongly recommend that at least one person from each participating team fills this registration form so that we can keep you informed about all updates regarding the shared task.

In addition, you can send any questions about the shared task to the organizers via corefud [at] googlegroups.com.

ORGANIZERS

  • Charles University (Prague, Czechia): Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Daniel Zeman
  • Institute of Computer Science, Polish Academy of Sciences (Warsaw, Poland): Maciej Ogrodniczuk
  • Georgetown University (Washington D.C., USA): Yilun Zhu
  • University of West Bohemia (Pilsen, Czechia): Miloslav Konopík, Ondřej Pražák, Jakub Sido

Third Call for Participation:: SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition

ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Submission Deadline: 
Friday, 28 January 2022
Contact: 
Sudipta Kar
Contact Email: 
sudipkar [at] amazon.com
Description/Details: 

We cordially invite you to participate in the SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER).

The task is being held as part of SemEval-2022, and all participating team will be able to publish their system description paper in the proceedings published by ACL.

This task focuses on the detection of complex entities, such as e-commerce products (movies, books, music and product titles), in low context settings (short and uncased text).

The task covers 3 domains (sentences, search queries, and questions) and provides data in 11 languages: English, Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and Bangla. Here are some examples in English, Chinese, Bangla, Hindi, Russian, Korean, and Farsi, where entities are enclosed inside brackets with their type:

  • the original [ferrari daytona | PRODUCT] replica driven by [don johnson | PERSON] in [miami vice | CreativeWork]
  • 它 的 座 位 在 [圣 布 里 厄 | LOCATION] .
  • স্টেশনটির মালিক [টাউনস্কেয়ার মিডিয়া | CORPORATION] ।
  • यह [कनेल विभाग | LOCATION] की राजधानी है।
  • в основе фильма — стихотворение [г. сапгира | PERSON] .
  • [블루레이 디스크 | PRODUCT] : 광 기록 방식 저장매체의 하나
  • [نینتندو- | CORPORATION] / [باندای نامکو انترتینمنت | CORPORATION] – [برادران سوپر ماریو نهایی | CreativeWork].

Additionally, a multilingual NER track is also offered for multilingual systems that can process all languages. A code-mixed track allows participants to build systems that process inputs with tokens coming from two languages. For example, the following are some code-mixed examples from Turkish, Spanish, Dutch, German, and English.

  • it was produced at the [soyuzmultfilm | GROUP] studio in [moskova | LOCATION] .
  • [arturo vidal | PERSON] ( born 1987 ) , professional footballer playing for [fútbol club barcelona | GROUP]
  • daarmee promoveerde hij toen naar de [premier league | CORPORATION] .
  • piracy has been a part of the [sultanat von sulu | LOCATION] culture .

The task focuses on detecting semantically ambiguous and complex entities in short and low-context settings. Participants are welcome to build NER systems for any number of languages. And we encourage to aim for a bigger challenge of building NER systems for multiple languages. The task also aims at testing the domain adaption capability of the systems by testing on additional test sets on questions and short search queries.

We have released training data for 11 languages along with a baseline system to start with. Participants can submit their system for one language but are encouraged to aim for a bigger challenge and build multi-lingual NER systems.

Important Links

Shared task schedule

  • Training data ready: September 3, 2021
  • Evaluation data ready: December 3, 2021
  • Evaluation start: January 24, 2022
  • Evaluation end: by January 28, 2022
  • System description paper submissions due: February 23, 2022
  • Notification to authors: March 31, 2022

Task organizers

  • Shervin Malmasi (Amazon)
  • Besnik Fetahu (Amazon)
  • Anjie Fang (Amazon)
  • Sudipta Kar (Amazon)
  • Oleg Rokhlenko (Amazon)

Please reach out to the organizers at multiconer-semeval-organizers [at] googlegroups.com, or join the Slack workspace to connect with the other participants and organizers.

Second Call for Participation: SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition

ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
File attachments: 
AttachmentSize
Image icon banner_multiconer.png255.35 KB
Submission Deadline: 
Monday, 10 January 2022
Contact: 
Sudipta Kar
Contact Email: 
skar3 [at] uh.edu
Description/Details: 

We cordially invite you to participate in the SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER).

The task is being held as part of SemEval-2022, and all participating team will be able to publish their system description paper in the proceedings published by ACL.

This task focuses on the detection of complex entities, such as e-commerce products (movies, books, music and product titles), in low context settings (short and uncased text).

The task covers 3 domains (sentences, search queries, and questions) and provides data in 11 languages: English, Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and Bangla. Here are some examples in English, Chinese, Bangla, Hindi, Russian, Korean, and Farsi, where entities are enclosed inside brackets with their type:

  • the original [ferrari daytona | PRODUCT] replica driven by [don johnson | PERSON] in [miami vice | CreativeWork]
  • 它 的 座 位 在 [圣 布 里 厄 | LOCATION] .
  • স্টেশনটির মালিক [টাউনস্কেয়ার মিডিয়া | CORPORATION] ।
  • यह [कनेल विभाग | LOCATION] की राजधानी है।
  • в основе фильма — стихотворение [г. сапгира | PERSON] .
  • [블루레이 디스크 | PRODUCT] : 광 기록 방식 저장매체의 하나
  • [نینتندو- | CORPORATION] / [باندای نامکو انترتینمنت | CORPORATION] – [برادران سوپر ماریو نهایی | CreativeWork].

Additionally, a multilingual NER track is also offered for multilingual systems that can process all languages. A code-mixed track allows participants to build systems that process inputs with tokens coming from two languages. For example, the following are some code-mixed examples from Turkish, Spanish, Dutch, German, and English.

  • it was produced at the [soyuzmultfilm | GROUP] studio in [moskova | LOCATION] .
  • [arturo vidal | PERSON] ( born 1987 ) , professional footballer playing for [fútbol club barcelona | GROUP]
  • daarmee promoveerde hij toen naar de [premier league | CORPORATION] .
  • piracy has been a part of the [sultanat von sulu | LOCATION] culture .

The task focuses on detecting semantically ambiguous and complex entities in short and low-context settings. Participants are welcome to build NER systems for any number of languages. And we encourage to aim for a bigger challenge of building NER systems for multiple languages. The task also aims at testing the domain adaption capability of the systems by testing on additional test sets on questions and short search queries.

We have released training data for 11 languages along with a baseline system to start with. Participants can submit their system for one language but are encouraged to aim for a bigger challenge and build multi-lingual NER systems.

Important Links

Shared task schedule

  • Training data ready: September 3, 2021
  • Evaluation data ready: December 3, 2021
  • Evaluation start: January 10, 2022
  • Evaluation end: by January 31, 2022 (latest date; task organizers may choose an earlier date)
  • System description paper submissions due: February 23, 2022
  • Notification to authors: March 31, 2022

Task organizers

  • Shervin Malmasi (Amazon)
  • Besnik Fetahu (Amazon)
  • Anjie Fang (Amazon)
  • Sudipta Kar (Amazon)
  • Oleg Rokhlenko (Amazon)

Please reach out to the organizers at multiconer-semeval-organizers [at] googlegroups.com, or join the Slack workspace to connect with the other participants and organizers.

[BioCreative VII] Text mining drug-protein/gene interactions (DrugProt) shared task

Abbreviated Title: 
DrugProt Shared Task
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Contact: 
Martin Krallinger
Contact Email: 
krallinger.martin [at] gmail.com
Description/Details: 

DrugProt Shared Task (BioCreative VII track 1- 2021)
Text mining drug-protein/gene interactions (DrugProt) shared task

https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vii/track-1/

Previous BioCreative efforts have provided highly relevant resources for advancing biomedical text mining research, including datasets and system evaluations in both shared and interactive modes (e.g. BioBERT benchmark datasets, BioC format, ChemProt corpus, CHEMDNER corpus, etc.).

We are organizing the DrugProt track specifically focusing on the large-scale automatic extraction of relations between drugs or chemical compounds and genes/proteins of interest for drug discovery and biomedicine.

It is getting increasingly challenging to efficiently exploit drug-related information described in the growing amount of scientific literature. There are a range of different types of drug-gene/protein interactions, and their systematic extraction and characterization is essential to analyze, predict and explore key biomedical properties underlying high impact biomedical applications.

We foresee that the DrugProt track will promote the development NLP techniques to extract critical health information, generating results useful for:

- Drug discovery, drug repurposing & drug design

- Metabolic engineering, modeling drug response, pharmacogenetics

- Drug-induced adverse reactions, off target interactions

- Molecular medicine, systems biology and bioinformatics

- Biomedical knowledge graph mining

Therefore the DrugProt organizers will release a large training corpus of manually annotated entity mentions for drugs/chemicals as well as genes/proteins together with their interactions (13 different types of interactions).

DrugProt teams participating will be provided with the following training corpus:

- PubMed abstracts (3500)

- Manually annotated chemical compound mentions (> 46200)

- Manually annotated gene/protein mentions (> 43200)

- Manually annotated drug/chemical-protein/gene interactions (> 17200)

Key information:

DrugProt web: https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vii/track-1/

Corpus: https://zenodo.org/record/4955411#.YNnWl27tZGb

Registration: https://docs.google.com/forms/d/e/1FAIpQLScdMnKFMncL8qDkcRx6aV6lYRm8Pbuf...

Evaluation will be done using micro-averaged f-measure by comparing the automatically extracted relations against previously manually labelled Gold Standard relations.

Important dates

- Training set release (UPDATED!) - June 15th 2021: DrugProt training set

- Test set release- July 5th 2021: Test set abstracts and entity annotations

- Test set prediction submission due: August 30th 2021

- Test set evaluation returned to participants: September 3st 2021

- Workshop proceedings paper due: September 10th 2021

- Paper acceptance and review returned: September 16th 2021

BioCreative VII workshop proceedings and Journal Special Issue

Participating teams will be invited to contribute to the: Proceedings of the Seventh BioCreative Challenge Evaluation Workshop. Proceedings papers are free of charge.

A selected number of top performing teams will also be invited to contribute with a longer system description paper to a special issue on BioCreative VII to be published in the journal Database.

Task organizers:

Martin Krallinger, Barcelona Supercomputing Center, Spain

Antonio Miranda, Barcelona Supercomputing Center, Spain

Farrokh Mehryary, University of Turku, Finland

Jouni Luoma, University of Turku, Finland

Sampo Pyysalo, University of Turku, Finland

Alfonso Valencia, Barcelona Supercomputing Center, Spain

References

[1] Krallinger, Martin, et al. "Overview of the protein-protein interaction annotation extraction task of BioCreative II." Genome biology 9.2 (2008): 1-19.
[2] Krallinger, Martin, et al. "CHEMDNER: The drugs and chemical names extraction challenge." Journal of cheminformatics 7.1 (2015): 1-11.
[3] Krallinger, Martin, et al. "Overview of the BioCreative VI chemical-protein interaction Track." Proceedings of the sixth BioCreative challenge evaluation workshop. Vol. 1. 2017.

MEDDOPROF: Medical Documents Profession Recognition

Abbreviated Title: 
MEDDOPROF
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Submission Deadline: 
Wednesday, 9 June 2021
Contact: 
Salvador Lima-López
Martin Krallinger
Contact Email: 
salvador.limalopez [at] gmail.com
krallinger.martin [at] gmail.com
Description/Details: 

*** 2nd CALL FOR PARTICIPATION ***
MEDDOPROF Shared Task (IberLEF - SEPLN 2021)
Medical Documents Profession Recognition shared task
https://temu.bsc.es/meddoprof/
MEDDOPROF Cup Awards by BSC-Plan TL [3,000€]

Professions and employment statuses can have a radical impact in physical and mental health, habits, lifestyle choices, … An entire medical specialty (occupational medicine) is centered around them, dealing with workplace accidents, exposition to toxic substances and pathogens, mental health impact, … The current pandemic has also underscored this influence, as many people with specific occupations have been specially affected (for instance, health professionals and other essential workers). The detection of these terms should help researchers to better characterize health risks of specific occupations.

One of the main contributions of the task is the inclusion of employment status in a broad sense. Unemployed and retired people, family caregivers, people who are homeless, people who depend on government subsidies, etc. have been considered, which strengthens the social side of this project.

Outside medicine, we foresee that the systems resulting from MEDDOPROF may be used in fields such as social care, human resources, legal NLP and even gender studies. Other NLP tasks such as anonymization might also benefit from having more exhaustive annotated data in this regard.

Following the success of previously organized shared tasks (i.e. Cantemist, PharmaCoNER and Meddocan), the Barcelona Supercomputing Center is now launching the MEDDOPROF Shared Task as part of the IberLEF 2021 evaluation initiative (co-located with SEPLN 2021), with the following sub-tracks:

- MEDDOPROF-NER: automatic detection of mentions of occupations (profession, employment status and activities).
- MEDDOPROF-CLASS: finding mentions of occupations and classifying them, whether they refer to the patients themselves, their family members or to healthcare professionals.
- MEDDOPROF-NORM: mapping detected occupation mentions to their corresponding concept identifiers from standard multilingual occupation terminologies (ESCO and SNOMED-CT).

Participation is free of charge, regardless of whether you submit a system description paper or not, and there will be prizes for the top three winners of each subtask.

** Key information: **
- MEDDOPROF web: https://temu.bsc.es/meddoprof/
- Data: https://doi.org/10.5281/zenodo.4694768
- Registration: https://temu.bsc.es/meddoprof/registration

** Schedule **
- Test set release (start of evaluation period): June 1st, 2021
- End of evaluation period (system submissions): June 9th, 2021
- Working papers submission: June 21st, 2021
- Notification of acceptance (peer-reviews): June 27th, 2021
- Camera-ready system descriptions: July 4th, 2021
- IberLEF @ SEPLN 2021: September 2021

** Publications and IBERLEF/SEPLN2021 workshop **
Teams participating in MEDDOPROF will be invited to contribute a systems description paper for the IberLEF (SEPLN 2021) Working Notes proceedings, and a short presentation of their approach at the IberLEF 2021 workshop.

** Main Organizers **
- Martin Krallinger, Barcelona Supercomputing Center, Spain
- Eulàlia Farré, Barcelona Supercomputing Center, Spain
- Salvador Lima, Barcelona Supercomputing Center, Spain
- Vicent Briva-Iglesias, D-REAL, Dublin City University, Ireland
- Antonio Miranda-Escalada, Barcelona Supercomputing Center, Spain

** Scientific Committee **
- Sophia Ananadiou, Department of Computer Science, University of Manchester, UK
- Alec Chapman, Data Scientist, University of Utah
- Dina Demner-Fushman, Tenure Track Investigator, Biomedical Informatics Branch, Lister Hill National Center for Biomedical Communications
- Hercules Dalianis, Professor in Computer and Systems Science, Stockholm University, Sweden
- Hongfang Liu, Professor of Biomedical Informatics, Mayo Clinic
- Josep Maria Haro Abad, Institut de Recerca Sant Joan de Déu
- Bradley Malin, Accenture Professor of Biomedical Informatics, Biostatistics, and Computer Science, Vanderbilt
- Goran Nenadic, Department of Computer Science, University of Manchester, UK
- Aurélie Névéol, LIMSI-CNRS, Université Paris-Sud, France
- Øystein Nytrø, Department of Computer and Information Science, Norges Teknisk-Naturvitenskapelige Universitet (NTNU)
- Carlos Luis Parra Calderón, Head of Technological Innovation at Virgen del Rocío University Hospital, Institute of Biomedicine of Seville, Spain
- Kirk E. Roberts, School of Biomedical Informatics, University of Texas Health Science Center
- Francisco Javier Sanz Valero, Escuela Nacional de Medicina del Trabajo, Instituto de Salud Carlos III, Spain
- Stefan Schulz, Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Austria
- Ashish Tendulkar, Machine Learning Specialist at Google
- Michelle Turner, Assistant Research Professor at Barcelona Institute for Global Health, Secretary-Treasurer International Society for Environmental Epidemiology (ISEE)
- Ozlem Uzuner, George Mason University
- Alfonso Valencia Herrera, Barcelona Supercomputing Center (BSC-CNS), Spain

Pages

Subscribe to RSS - shared task