shared task

KSAA-CAD: Contemporary Arabic Dictionary Shared Task! For Reverse Dictionary and Word Sense Disambiguation at ArabicNLP 2024!

Abbreviated Title: 
KSAA-CAD: Contemporary Arabic Dictionary Shared Task!
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
Co-located with ACL 2024 Bangkok, Thailand
File attachments: 
AttachmentSize
PDF icon KSAA-CAD.pdf194.26 KB
Event Dates: 
Friday, 16 August 2024
Submission Deadline: 
Friday, 3 May 2024
Contact: 
Waad Alshammari
Rawan Almatham
Contact Email: 
waad.wtss [at] gmail.com
r.almatham [at] gmail.com
City: 
Bangkok
State: 
Thailand
Country: 
Thailand
Description/Details: 

Greeting
We would like to extend an invitation to participate in the KSAA-CAD: Contemporary Arabic Dictionary Shared Task! For Reverse Dictionary and Word Sense Disambiguation at ArabicNLP 2024!
Please find all the necessary information below.
https://arai.ksaa.gov.sa/sharedTask2024/

Registration deadline: 29th of April 2024.

Registration:
Participants need to register via this link: https://docs.google.com/forms/d/e/1FAIpQLSetmIT7XQ3megI6dGI9xm17wc-oY86z... !

Tasks:
Task 1: Reverse Dictionary (RD)

RDs, identified by their sequence-to-vector format, characterized as sequence-to-vector, introduce a differentiated strategy in contrast to traditional dictionary lookup methods. The RD task concentrates on the conversion of human-readable glosses into word embedding vectors.
This process entails reconstructing the word embedding vector corresponding to the defined word, a methodology aligning with the approaches of (Mickus et al., 2022; Zanzotto et al., 2010; Hill et al., 2016).
The dataset includes, lemma, lemma vector representations, and their respective gloss.
The developed model is expected to generate novel lemma vector representations for the unseen human-readable definitions in the test set. Such a strategy allows users to search for words based on anticipated definitions or meanings.

Task 2: Word Sense Disambiguation (WSD)

WSD focuses on identifying the specific sense of a word in a given context. This technique involves determining a word's intended meaning by calculating the overlap between its contextual use and the provided gloss or definition.
The dataset consists of word, context, and context ID, and corresponding gloss ID. The developed model is expected to retrieve the suitable gloss ID for the word in the context from the WSD dictionary.

Datasets:
Datasets can be downloaded from the CODALAB platform:
• CODALAB-Task1 RD: https://codalab.lisn.upsaclay.fr/competitions/18510
• CODALAB-Task2 WSD: https://codalab.lisn.upsaclay.fr/competitions/18468

Important dates:
• Release of training, dev data, and evaluation scripts: 15th of March 2024.
• Release of test data: 15th of April 2024.
• Registration deadline: 29th of April 2024.
• End of the evaluation cycle (test set submission closes): 3th of May 2024.
• Results released: 29th of April 2024
• System description paper submissions due: 10st of May 2024.

Contact:
• Email: waad.wtss [at] gmail.com
• Email: r.almatham [at] gmail.com

Awards:
We are pleased to announce the awards for the Arabic Reverse Dictionary Shared Task at ArabicNLP 2024. The top-ranked teams in each task will receive cash prizes as follows:

Task 1: Reverse Dictionary (RD)
1st Ranked: $350
2nd Ranked: $250
3rd Ranked: $150

Task 2: Word Sense Disambiguation (WSD)
1st Ranked: $350
2nd Ranked: $250
3rd Ranked: $150

Best regards!!

DialAM-2024 Shared Task - Call for Participation

Abbreviated Title: 
DialAM-2024 CfP
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
Contact: 
Contact Email: 
rruizdolz001 [at] dundee.ac.uk
City: 
State: 
Country: 
Description/Details: 

Hi all,

We would like to invite the community to participate in DialAM-2024: The First Shared Task on Dialogical Argument Mining, part of the 11th Workshop in Argument Mining (https://argmining-org.github.io/2024/index.html#shared_task)

*** Registration ***

https://dialam.arg.tech

*** Slack Channel ***

https://join.slack.com/t/dialam-2024/shared_invite/zt-2d9z2wvlj-9AVjILUM...

*** Important Dates ***

Now - Shared Task starts and training data release
17th April - Test data release start of evaluation
1st May - Evaluation end
7th May - Results announcement
20th May - Paper submission due
End of June [TBD] - Camera-ready version due
15th of August - ArgMining 2024 workshop (ACL)

*** Overview ***

With the DialAM-2024 task, we propose the first shared task in dialogue argument mining where argumentation and dialogue information is modelled together in a domain-independent framework. The Inference Anchoring Theory (IAT) framework, makes possible to obtain homogeneous annotations of dialogue argumentation including relevant information and structural data from speech and argumentation, regardless of the domain, and allowing a more complete analysis of argumentation in dialogues together with a consistent cross-domain evaluation of the resulting argument mining systems. The DialAM-2024 consists of two sub-tasks: the identification of propositional (argumentative) relations, and the identification of illocutionary (speech act) relations. For both tasks all the information belonging to argumentation and dialogue will be available for the development of the submitted systems. We invite the community to participate in the DialAM-2024 task and explore how the use of additional information from the dialogue can be integrated into the argument retrieval process, in an attempt to take a step forward from sequence modelling approaches, where much of the relevant information to argumentation remains implicit behind the natural language.

*** Tasks ***

Task A: Identification of Propositional Relations

In the first task, the goal is to detect argumentative relations existing between the propositions identified and segmented in the argumentative dialogue. Such relations are: Inference, Conflict, and Rephrase.

Task B: Identification of Illocutionary Relations

In the second task, the goal is to detect illocutonary relations existing between locutions uttered in the dialogue and and the argumentative propositions associated with them such as: Asserting, Agreeing, Arguing, or Disagreeing among others.

*** Data ***

We will use the QT30 corpus [1], the largest available corpus in dialogical argumentation in English. QT30 is a collection of 30 episodes of Question Time aired between June 2020 and November 2021, with a total of more than 29 hours of transcribed broadcast material and comprises of 19,842 locutions by more than 400 participants: one moderator, 125 panel members (7 of them appearing more than once), and 300+ audience members. The QT30 dataset contains 10,818 propositional relations divided into Default Inferences, Default Conflicts, and Default Rephrases, and 32,303 illocutionary relations divided into Asserting, Agreeing, Arguing, Disagreeing, Restating, Questioning, and Default Illocuting.

[1] Hautli-Janisz, A., Kikteva, Z., Siskou, W., Gorska, K., Becker, R., & Reed, C. (2022, June). Qt30: A corpus of argument and conflict in broadcast debate. In Proceedings of the 13th Language Resources and Evaluation Conference (pp. 3291-3300). European Language Resources Association (ELRA).

*** Organisers ***

Ramon Ruiz-Dolz (rruizdolz001 [at] dundee.ac.uk), ARG-tech, University of Dundee
John Lawrence, ARG-tech, University of Dundee
Ella Schad, ARG-tech, University of Dundee
Chris Reed, ARG-tech, University of Dundee

CfP: Shared Task on Text-Graph Representations for KGQA @ TextGraphs-17

Abbreviated Title: 
KGQA Shared Task @ TextGraphs-17
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
ACL 2024, Thailand
Event Dates: 
Thursday, 15 August 2024
Submission Deadline: 
Wednesday, 24 April 2024
Contact: 
TextGraphs-17 Google Group
Contact Email: 
textgraphs17 [at] googlegroups.com
City: 
Bangkok
Country: 
Thailand
Description/Details: 

CfP: Shared Task on Text-Graph Representations for KGQA @ TextGraphs-17

Venue: ACL 2024, TextGraph workshop
Website: https://sites.google.com/view/textgraphs2024/home/shared-task

TextGraph-17 workshop, co-located with ACL-2024, features a shared task. TextGraph fosters the investigation of synergies between methods for text and graph processing. This edition focuses on the fusion of LLMs with KGs. In line with this goal, we propose a shared task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).

The shared task is to select a KG entity (out of several candidates) that corresponds to an answer given a textual question. The specificity of the task is that for each question-answer (Q-A) pair, not only a textual Q-A pair is provided but also a graph of shortest paths in the KG from entities in the query to the LLM-generated candidate entity (including links of the intermediate nodes). This way, participants may easily experiment with various strategies of text-graph modality fusion for the given task in a controllable manner.

Participants can submit reports about their participation in the TextGraphs workshop. An example of a previous TextGraphs shared task can be found here: https://aclanthology.org/volumes/2022.textgraphs-1/.

Task Description

Participants are given:
text: question with a list of Wikidata entities mentioned,
text: 5-10 answer candidates in the form of Wikidata entities,
graph: a Wikidata sub-graph composed of the shortest paths between entities in question and entities in answer is provided.

One of the candidates is correct, and the others are incorrect. The goal is to find the correct answer, i.e., perform a binary classification. Participants are provided a train and development dataset in the form of Q-A-Subgraph triples. Besides, a submission to Codalab's public and private leaderboard of the test set Q-A solutions will be available.

Important Dates
Training dataset released: 10th March 2024
Test set released: 25 March 2024
End of evaluation: 25 April 2024
Submission deadline for technical reports: 17 May 2024

Contact
Please write all questions about the shared task to textgraphs17 [at] googlegroups.com. Also, you are invited to join our Telegram group, where you can connect to organizers and get updates: https://t.me/+kRTCZYTrpJ5jZGVi

Organizers:
Irina Nikishina, Universität Hamburg
Aida Usmanova, Leuphana University Lüneburg
Angelie Kraft, Universität Hamburg
Cedric Möller, Universität Hamburg
Debayan Banerjee, Universität Hamburg
Junbo Huang, Universität Hamburg
Longquan Jiang, Universität Hamburg
Rana Abdullah, Universität Hamburg
Xi Yan, Universität Hamburg
Andrey Sakhovskiy, KFU
Elena Tutubalina, KFU
Mikhail Salnikov, AIRI
Alexander Panchenko, AIRI
Ricardo Usbeck, Universität Hamburg

GEM 2024 multilingual data-to-text and summarization shared task

Abbreviated Title: 
GEM 2024 shared task
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
INLG
Submission Deadline: 
Friday, 5 April 2024
Contact: 
Simon Mille
João Sedoc
Contact Email: 
simon.mille [at] adaptcentre.ie
joao.sedoc [at] gmail.com
City: 
Tokyo
Country: 
Japan
Description/Details: 

We are finally launching the 2024 GEM multilingual shared task! It comprises 2 main tasks, Data-to-text generation and Summarization, each of which has 3 subtasks. You can submit automatically generated outputs for one or more subtask(s), in Arabic, Chinese, English, German, Hindi, Korean, Russian, Spanish, and/or Swahili. The deadline for pre-registering your system submissions is March 8th 23.59 AoE (see link below). Note that it will be possible for participants to pre-register after March 8th, but that doing so does not guarantee a participation in the human evaluation.

We are delighted to announce that the GEM shared task is endorsed by SIGGEN, and will be part of the Generation Challenges (GenChal) at INLG’24. Participants will have the possibility to (i) publish a system description in the GenChal proceedings (available on the ACL anthology), and (ii) present their results during the GenChal session at the INLG conference in Tokyo in September 2024. See links below.

Important dates
March 8: Deadline for pre-registering systems (pre-registration will remain open until the submission date; late pre-registrations do not guarantee participation to the human evaluation).
April 5: Deadline for output submission (all subtasks).
April 6: Human evaluation starts.
TBD: System descriptions due.
Late September: GenChal@INLG’24

GEM-ST website: https://gem-benchmark.com/shared_task.
SIGGEN page: https://siggen-acl.github.io/index.html.
GenChal’23 Proceedings: https://aclanthology.org/volumes/2023.inlg-genchal/.
Pre-registration: https://nyustern.az1.qualtrics.com/jfe/form/SV_8qRqfdN3qBy3Bqe.

ArgMining2024: Call for Shared Task Proposals

Abbreviated Title: 
ArgMining2024
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Proposals
Location: 
To be held in conjunction with ACL 2024
Event Dates: 
Thursday, 15 August 2024
Submission Deadline: 
Friday, 5 January 2024
Contact: 
Contact Email: 
City: 
Bangkok
State: 
Country: 
Thailand
Description/Details: 

CALL FOR SHARED TASK PROPOSALS

We cordially invite submissions of shared tasks as part of ArgMining 2024, the “11th Workshop on Argument Mining”. The workshop will be co-located with ACL 2024 (to be held in Bangkok, Thailand).

Argument mining (also known as “argumentation mining”) is a gradually maturing research area within computational linguistics. It involves the automatic identification of argumentative structures in free text, as well as argument quality assessment, argument persuasiveness, and the synthesis of argumentative texts.

To advance research on specific aspects of argument mining, previous editions of the ArgMining workshop series have promoted shared tasks, including key point analysis for quantitative summarization of arguments (see https://2021.argmining.org/), the validity and novelty of arguments (see https://argmining-org.github.io/2022/), multimodal argument mining and pragmatic tagging of peer reviews (see https://argmining-org.github.io/2023).

Following the success of previous workshops, ArgMining 2024 plans to share one or more unsolved problems to be investigated by the community.

Proposals for shared tasks should include:

* a title and a brief description of the task
* a description of the datasets that will be used in the task and their readiness, and a proposed plan for data collection and annotation
* previous work on the datasets, including publications (if any)
* a few lines regarding evaluation of the submitted systems
* a brief introduction of the task organizers

Shared task organizers will have the opportunity to publish a task overview paper in the workshop proceedings.

Please submit your shared task proposal via email to argmining.org [at] gmail.com. The submission deadline is January 5, 2024, and task organizers will be notified of proposal acceptance on January 19, 2024.

While exact dates are not yet available, we assume the following tentative schedule:

* End of February - Training data release
* Mid of April - Test data release, evaluation start
* End of April - Evaluation end
* Mid of May - Results announcement
* End of May - Paper submission due
* End of June - Camera-ready version due
* Mid of August - ArgMining 2024 workshop (ACL)

The timeline will be finalized with the shared task organizers.

ORGANIZERS:
Yamen Ajjour, Leibniz University Hannover
Roy Bar-Haim, IBM Research
Roxanne El Baff, German Aerospace Center (DLR) and Bauhaus-Universität
Zhexiong Liu, University of Pittsburgh
Gabriella Skitalinskaya, Leibniz University Hannover

GEM flash shared task summarisation/D2T: expression of interest

Abbreviated Title: 
GEM Shared Task
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Other
Location: 
EMNLP'23
Contact: 
João Sedoc
Simon Mille
Contact Email: 
joao.sedoc [at] gmail.com
simon.mille [at] adaptcentre.ie
City: 
Singapore
State: 
Country: 
Description/Details: 

We are collecting interests for the GEM v3 "flash" shared task, which will focus on text summarisation and data-to-text generation in a variety of languages. The submission deadline is expected to be around the end of September 2023, and we will provide a human evaluation of each participant’s main submission in the following weeks. Please fill out the very brief survey to indicate which task(s) and language(s) you would be interested in. More details on the datasets and modalities will be communicated soon. Thank you!

Survey link: https://forms.office.com/Pages/ResponsePage.aspx?id=EhAze6GHFkqLD6RgWx89...

Second WMT shared task on sign language translation

Abbreviated Title: 
WMT-SLT2023
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Location: 
Singapore
Event Dates: 
Wednesday, 6 September 2023 to Thursday, 7 September 2023
Submission Deadline: 
Tuesday, 15 August 2023
Contact: 
WMT Tasks
WMT Sign Language Task
Contact Email: 
wmt-tasks [at] googlegroups.com
wmt-sign-language-task [at] googlegroups.com
City: 
State: 
Country: 
Description/Details: 

WMT-SLT 4: Motivation 2023 https://youtu.be/QjqBiRH0e_0
WMT-SLT 5: Summary 2023 https://youtu.be/ngUDIZDvzvY

We are excited to announce the second edition of the Shared Task on Sign Language Translation (WMT-SLT 23) at WMT 2023, co-located with EMNLP 2023.

*NEW DEVELOPMENTS SINCE THE FIRST CALL*

(1) Release of all our provided training data
(2) Release of baseline systems

*OVERVIEW*

This shared task is concerned with automatic translation between signed and spoken languages. The task requires processing visual information (such as video and poses) beyond the well-known paradigm of text-to-text machine translation.

This year the shared task features two tracks: translation from a signed language to a spoken language (signed-to-spoken track) and vice versa (spoken-to-signed track). We cover the following translation directions:

signed-to-spoken:
* Swiss German Sign Language (DSGS) to German (DE)

spoken-to-signed:
* German (DE) to Swiss German Sign Language (DSGS)
* French (FR) to French Sign Language (LSF)
* Italian (IT) to Italian Sign Language (LIS)

We provide training data and baseline systems for all translation directions above.

Submissions to the signed-to-spoken track will be ranked on a leaderboard. Submissions to the spoken-to-signed track are not ranked on the leaderboard while the task is running, but we still encourage participants to submit such systems.

We aim to provide human evaluation for all submitted systems, regardless of the translation direction.

We will ask participants to submit a system description paper to WMT 2023.

*IMPORTANT DATES*

* 22 June 2023: Release of training and development data
* 22 June 2023: Release of baseline systems
* 01 August 2023: Release of test data
* 01 August 2023: System submission opens
* 15 August 2023: System submission deadline
* 05 September 2023: Paper submission deadline
* 06 October 2023: Paper acceptance notification
* 18 October 2023: Camera-ready version due
* 6 - 7 December 2023: Overview of results at WMT
All deadlines are AoE (Anywhere on Earth). Dates are in line with general deadlines of WMT.

*ORGANIZERS*

* Mathias Müller, University of Zurich
* Sarah Ebling, University of Zurich and ZHAW Winterthur
* Eleftherios Avramidis, DFKI Berlin
* Cristina España-Bonet, DFKI Saarbrücken
* Dimitar Shterionov, Tilburg University
* Amit Moryossef, Bar-Ilan University / University of Zurich
* Zifan Jiang, University of Zurich
* Oscar Koller, Microsoft
* Necati Cihan Camgoz, Meta
* Roman Grundkiewicz, Microsoft
* Davy Van Landuyt, European Union of the Deaf
* Annelies Braffort, CNRS and Paris-Saclay University
* Richard Bowden, University of Surrey
* Anne Goehring, University of Zurich
* Annette Rios, University of Zurich
* Malihe Alikhani, University of Pittsburgh
* Mert Inan, University of Pittsburgh

*MORE INFORMATION*
For more information, please visit our website: www.wmt-slt.com
If you have any comments and/or questions, do not hesitate to contact us: wmt-tasks [at] googlegroups.com

The Generation Challenges@INLG 2023

Abbreviated Title: 
GenChal@INLG 2023
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Proposals
Location: 
INLG'23
Event Dates: 
Monday, 11 September 2023 to Friday, 15 September 2023
Submission Deadline: 
Monday, 5 June 2023
Contact: 
Simon Mille
Contact Email: 
simon.mille [at] adaptcentre.ie
City: 
Prague
Country: 
Czech Republic
Description/Details: 

=======================================
GenChal @ 16th International Conference on Natural Language Generation
Prague, Czech Republic, September 11-15 2023
INLG Twitter: @inlgmeeting
=======================================
Call for new Shared Task proposals
Submission deadline: June 5th 2023

We invite submissions of papers describing ideas for future shared tasks in the general area of language generation (Generation Challenges 2023). Proposed tasks can be in the area of core NLG, or in other research areas in which language is generated. Examples include, but are not limited to: data-to-text NLG, text-to-text generation (including MT and summarisation), combining core NLG and MT, combining core NLG and text summarisation, NLG quality estimation, NLG evaluation metrics, and/or generating language from heterogeneous data, including image and video.
The Generation Challenges (GenChal) are an umbrella event designed to bring together a variety of shared-task efforts that involve the generation of natural language. This year, Generation Challenges will be held during a special session at the 16th International Conference on Natural Language Generation (INLG 2023, https://inlg2023.github.io/), scheduled on September 11-15 2023. The session will follow the format of previous GenChal results sessions, with presentations of results by the organisers of the generation challenges that are currently running, a poster session for task participants to present their submissions, as well as presentations of proposals for new shared tasks in the Task Proposals Track, and discussion sessions. You can see the previous GenChal tasks on the dedicated repository (https://sites.google.com/view/genchalrepository/home).

Submissions should describe possible future tasks in detail, including information regarding organisers, task description, motivating theoretical interest and/or application context, size and state of completion of data to be used, schedule and evaluation plans. Accepted shared tasks will be run in the 2024 iteration of INLG.

Important dates (the dates on the website will be updated soon)
Submission deadline: June 5th 2023
Notification: July 11th 2023
Camera-ready submission: July 25th 2023
INLG conference: September 11th-15th 2023

All deadlines are 11.59 pm UTC -12h ("anywhere on Earth").

Submissions and format
Submissions in the Shared Task Proposals track should be no more than 4 (four) pages long excluding citations, and should follow the ACLPUB formatting guidelines (https://acl-org.github.io/ACLPUB/formatting.html) and template files (https://2021.aclweb.org/downloads/acl-ijcnlp2021-templates.zip). Proposals should be uploaded to the SoftConf GenChal submission page (https://softconf.com/n/inlg2023/user/scmd.cgi?scmd=submitPaperCustom&pag...), using the Submission type New shared task proposal.

Submissions will be peer-reviewed by the program committee. As reviewing will not be blind, there is no need to anonymise papers.
This is not intended to be a selective process, since the aim is to discuss new potential shared tasks with INLG delegates. However, the organisers reserve the right to reject proposals which do not fall within the scope of the GenChal initiative, or which do not follow guidelines. Accepted submissions will be published in separate GenChal 2023 proceedings on the ACL Anthology, as was done in 2022 (https://aclanthology.org/volumes/2022.inlg-genchal/).

Looking forward to seeing you at INLG!

BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Location: 
ACL 2023
Event Dates: 
Thursday, 13 July 2023
Submission Deadline: 
Friday, 5 May 2023
Contact: 
Anaïs Tack
Contact Email: 
anais.tack [at] kuleuven.be
City: 
Toronto
Country: 
Canada
Description/Details: 

Conversational agents offer promising opportunities for education. They can fulfill various roles (e.g., intelligent tutors and service-oriented assistants) and pursue different objectives (e.g., improving student skills and increasing instructional efficiency) (Wollny et al. 2021). Among all of these different vocations of an educational chatbot, the most prevalent one is the AI teacher helping a student with skill improvement and providing more opportunities to practice. Some recent meta-analyses have even reported a significant effect of chatbots on skill improvement, for example in language learning (Bibauw et al. 2022). What is more, current advances in AI and natural language processing have led to the development of conversational agents that are founded on more powerful generative language models.

Despite these promising opportunities, the use of powerful generative models as a foundation for downstream tasks also presents several crucial challenges. In the educational domain in particular, it is important to ascertain whether that foundation is solid or flimsy. Bommasani et al. (2021: pp. 67-72) stressed that, if we want to put these models into practice as AI teachers, it is imperative to determine whether they can (a) speak to students like a teacher, (b) understand students, and (c) help students improve their understanding. Therefore, Tack and Piech (2022) formulated the AI teacher test challenge: How can we test whether state-of-the-art generative models are good AI teachers, capable of replying to a student in an educational dialogue?

Following the AI teacher test challenge, we organize a first shared task on the generation of teacher language in educational dialogues. The goal of the task is to use NLP and AI methods to generate teacher responses in real-world samples of teacher-student interactions. These samples are taken from the Teacher Student Chatroom Corpus (Caines et al. 2020; Caines et al. 2022). Each training sample is composed of a dialogue context (i.e., several teacher-student utterances) as well as the teacher’s response. For each test sample, participants are asked to submit their best generated teacher response.

The purpose of the task is to benchmark the ability of generative models to act as AI teachers, replying to a student in a teacher-student dialogue. Submissions will be ranked according to several automated dialogue evaluation metrics, with the top submissions selected for further human evaluation. During this manual evaluation, human raters will compare a pair of teacher responses in terms of three abilities: can speak like a teacher, can understand a student, can help a student (Tack & Piech 2022). As such, we adopt an evaluation method that is akin to ACUTE-Eval for evaluating dialogue systems (Li et al. 2019).

PARTICIPATION

The shared task is hosted on CodaLab (Pavao et al. 2022). Anyone participating in the shared task will be asked to:

  1. Register on the CodaLab platform.
  2. Fill in the registration form with their CodaLab ID. Participants must comply with the terms and conditions of the task and the TSCC data outlined in the form.
  3. Register for the CodaLab competition using the CodaLab ID. We will only accept people who submitted the registration form. Note that you can participate as a member of one team only.

IMPORTANT DATES

  • Mar 24, 2023: Training data release
  • May 1, 2023: Test data release
  • May 5, 2023: Final submissions due
  • May 8, 2023: Results announced
  • May 12, 2023: Human evaluation results announced
  • May 22, 2023: System papers due
  • May 26, 2023: Paper reviews returned
  • May 30, 2023: Camera-ready papers due
  • June 12, 2023: Pre-recorded video due
  • July 13, 2023: BEA Workshop at ACL

ORGANIZERS

  • Anaïs Tack, KU Leuven
  • Ekaterina Kochmar, MBZUAI
  • Zheng Yuan, King’s College London
  • Serge Bibauw, Universidad Central del Ecuador
  • Chris Piech, Stanford University

Webpage: https://sig-edu.org/sharedtask/2023

SIGMORPHON 2023 Shared Task on Interlinear Glossing

ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Contact: 
Miikka Silfverberg
Contact Email: 
msilfver [at] mail.ubc.ca
Description/Details: 

We are organizing a shared task on automated interlinear glossing (IGT) in the 2023 workshop of the ACL Special Interest Group on Computational Morphology and Phonology (SIGMORPHON).

Interlinear glossed text is a major annotated datatype produced in the course of linguistic fieldwork. For many low-resource languages, this is the only form of annotated data that is available for NLP work. Creation of glossed text is, however, a laborious endeavor and this shared task investigates methods to (fully or partially) automate the process.

In this task, participants build systems which generate morpheme-level grammatical descriptions of input sentences following the Leipzig glossing conventions. The input to the glossing system consists of (1) a sentence in the target language and (2) a translation of the target sentence into a language of wider communication, often English. More details are available on our task repo: https://github.com/sigmorphon/2023glossingST.

*Code*
In the task repo, we provide the code, results, and downloadable trained models for our baseline system. We also provide the evaluation script and other helpful scripts for loading and processing the task data, to facilitate easy building of novel systems.

*Training Conditions*
In this task, there are two tracks that participants may participate in. In the closed track, systems are trained solely on input sentences and glosses. In the open track, systems may additionally make use of morphological segmentations during training time. In the open track, participants may additionally use any data and resources (including dictionaries and pretrained language models). The only exception is additional interlinear glossed data which is not allowed. For the open track, we also provide some extra information like POS tags for a subset of the languages.

*Evaluation*
The main evaluation metric for the competition is token accuracy. Systems are evaluated w.r.t. generation of fully glossed tokens (chiens -> dog-PL). We will also separately evaluate glossing accuracy on bound morphemes like PL and free morphemes, i.e. stems, like dog.

*Organizers*
Michael Ginn (University of Colorado)
Mans Hulden (University of Colorado)
Sarah Moeller (University of Florida)
Garrett Nicolai (University of British Columbia)
Alexis Palmer (University of Colorado)
Miikka Silfverberg (University of British Columbia)
Anna Stacey (University of British Columbia)

*Important Dates and Deadlines*
April 1: Release of surprise language training and development data
April 24: Release of test data for all languages
April 27: Test predictions should be submitted to organizers
May 15: System description paper submission deadline
May 25: Notification of paper acceptance
May 30: Camera ready deadline for system description papers

*Read More*
https://github.com/sigmorphon/2023glossingST

Pages

Subscribe to RSS - shared task