Shared Task on Cross-lingual Open-Retrieval QA
Location: Co-located with NAACL 2022, Friday, 15 July 2022
Event Notification Type: Shared Task
State:
Washington
Country:
United States
Contact Email:
mia.nlp.workshop [at] gmail.com
akari [at] cs.washington.edu
slongpre [at] mit.edu
Website: https://mia-workshop.github.io/shared_task.html
Submission Deadline:
April 25, 2022
The Shared Task on Cross-lingual Open-Retrieval QA is collocated with MIA 2022 at NAACL 2022. This task aims at developing models for cross-lingual open question answering in 14 topologically diverse languages, and we are planning to release new evaluation data in additional languages. All of the baseline models, training data, intermediate prediction results are available at our Github Repository.
Github: https://github.com/mia-workshop/MIA-Shared-Task-2022
Submission site: https://eval.ai/web/challenges/challenge-page/1638/overview
If you are participating in our shared task, please register your team through this form:
https://forms.gle/ioWDn4UCKyftTVCk6
***Shared Task Overview***
Cross-lingual Open Question Answering is a challenging multilingual NLP task, where given questions are written in a user’s preferred language, a system needs to find evidence in large-scale document collections written in many different languages, and return an answer in the user’s preferred language, as indicated by their question. For instance, a system needs to answer in Arabic to answer an Arabic question, but it can use evidence passages written in any language included in a large-document corpus.
The evaluation is based on the macro-averaged scores across different target languages.
***Target Languages***
Our shared task will evaluate systems in 14 languages, 7 of which will not be covered in our training data. The training and evaluation data is originally from Natural Questions, XOR-TyDi QA, and MKQA (See details below in the Dataset section).
The full list of the languages:
- Languages with training data: Arabic, Bengali, English Finnish, Japanese, Korean, Russian, Telugu
- Languages without training data: Spanish, Khmer, Malay, Swedish, Turkish, Chinese (simplified)
***Evaluations***
Participants will run their systems on the evaluation files (without answer data) and then submit their predictions to our competition site hosted at eval.ai. Systems will first be evaluated using automatic metrics: Exact match and token-level F1 (Lee et al., 2019; Asai et al., 2021; Longpre et al., 2021). For non-spacing languages (i.e., Japanese, Khmer and Chinese) we use token-level tokenizers, Mecab, khmernltk and jieba to tokenize both predictions and ground-truth answers.
Due to the difference of the datasets' nature, we will calculate macro-average scores on XOR-TyDi and MKQA datasets, and then take the average of the XOR-TyDi QA average {F1, EM} and MKQA average {F1, EM}.
Although EM is often used as a primarily evaluate metric for English open-retrieval QA, the risk of surface-level mismatching (Min et al., 2021) can be more pervasive in cross-lingual open-retrieval QA. Therefore, we will use F1 as our primary metric and rank systems using their macro averaged F1 scores.
***Prizes***
- The Best {unconstrained, constrained} system: These prizes will be given to the {constrained, unconstrained} systems (see the details in the Dataset Section) obtaining the highest macro-average F1 scores.
- Special award: We plan to give additional award(s) to systems that employ a creative approach to the problem, or undertake interesting experiments to better understand the problem. This award is designed to encourage interesting contributions even from teams without access to the largest models or computational resources. Examples include attempts to improve generalization ability/language equality, reducing model sizes, or understanding the weaknesses in existing systems.
***Important Dates***
February 2022: Shared task baseline & training/development data release
Early March 2022: Shared task participant registration (soft) deadline
Late March 2022: Shared task test set to release to the participants
April 25, 2022: Shared task prediction submission deadline
April 29, 2022: Late paper submission deadline
July 10, 2022: NAACL Workshop date
***Registration***
If you are participating in our shared task, please register your team through this form:
https://forms.gle/ioWDn4UCKyftTVCk6
It will help us to plan better for human evaluation, etc., as well as occasionally send announcements when there is a major update on baseline or dataset.
For more details, please visit the shared task website: https://mia-workshop.github.io/shared_task.html
On Thu, Mar 10, 2022 at 8:47 AM Priscilla Rasmussen wrote:
Your announcement has been posted and "published", Akari.
Regards,
Priscilla
On Thu, Mar 10, 2022 at 12:07 AM Akari Asai wrote:
Dear Priscilla,
Hello, my name is Akari Asai, a Ph.D. student at the University of Washington and a lead organizer of the Workshop on Multilingual Information Access (NAACL 2022). Could you post our call for workshop papers at the ACL member portal so that we could reach out to the wide NLP community?
We also host a shared task and would like to post a call for shared task participation as well. I will send a separate email later this week.
Thank you so much for your help. If you have any questions or concerns, please let me know.
Best regards,
Akari
====
The 1st Workshop on Multilingual Information
Location: Co-located with NAACL 2022, Friday, 15 July 2022
Event Notification Type: Call for Papers
Abbreviated Title: MIA 2022
State:
Washington
Country:
United States
Contact Email:
mia.nlp.workshop [at] gmail.com
akari [at] cs.washington.edu
Website: https://mia-workshop.github.io/
Submission Deadline:
Friday, 8 April 2022
***Overview and Main focus***
State-of-the-art NLP technologies such as question answering and information retrieval systems have enabled many people to access information efficiently. However, these advances have been made in an English-first way, leaving other languages behind. Large-scale multilingual pre-trained models have achieved significant performance improvements on many multilingual NLP tasks (Hu et al., 2020; Conneau et al., 2020; Xue et al., 2020) where input text is provided. Yet, on knowledge-intensive tasks that require retrieving knowledge and generating output (Petroni et al., 2021), we observe limited progress (Asai et al., 2021). Moreover, in many languages, existing knowledge sources are critically limited (Roy et al., 2020), and thus finding knowledge in another language with abundant knowledge sources is often required. Despite much exciting recent progress (Karpukhin et al., 2020) in English knowledge-intensive tasks, transferring such progress to a wider set of languages is non-trivial as in many languages.
Our workshop focuses on building efficient, performant information access systems in a larger set of typologically diverse languages. We seek submissions on various aspects of this challenging task—(i) knowledge source, evaluation benchmark curation for low-resource languages, (ii) learning scenarios, and model architectures for building multilingual information access systems.
***Research Paper Track***
We encourage paper submissions that focus on various aspects of cross-lingual knowledge-intensive NLP tasks, including but not limited to:
- Multilingual machine reading comprehension (MRC)
- Multilingual open-retrieval question answering (QA)
- Cross-lingual information retrieval
- Multilingual information extraction
- Cross-lingual fact-checking
- Cross-lingual summarization
- System descriptions from our shared tasks (See below)
We also encourage submissions on related topics such as:
- Multilingual argument mining
- Cross-lingual semantic parsing
- Quantifying information content available in public
***Shared Task Track***
We also host a shared task on cross-lingual open-retrieval QA, which covers 15 typologically diverse languages with and without training data. See details on this page.
https://mia-workshop.github.io/shared_task.html
We accept system descriptions from our shared task as a workshop submission (Deadline: April 29).
***Submission Guidelines***
Submissions should be at least 4 and at most 8 pages, not including citations; final versions of papers will be given one additional page (up to 9 pages). All submissions will be reviewed according to the same standards, regardless of length (i.e., there are no separate short and long paper tracks). Submissions may optionally include an appendix with no length restriction. The main paper must remain fully self-contained, as reviewers will not be asked to review appendices.
Please format your papers using the standard style files for ARR submission format:
- LaTeX & Word version: https://github.com/acl-org/acl-style-files
- Overleaf template: https://www.overleaf.com/project/5f64f1fb97c4c50001b60549
We accept submissions of both previously unpublished work and work recently published elsewhere.
1. Unpublished work
Previously-unpublished work must be anonymized, as it will go through a double-blind review process, and will be included in the workshop proceedings if accepted. This year, we are using a hybrid submission system: papers may be submitted either via OpenReview.
2. Published work (non-archival track)
Recently published work does not need to be anonymized and will not go through the normal review process. Instead, authors are asked to submit their published work with the reviews, and we will conduct a meta-review to see if the published work aligns with our workshop focus. The submission should clearly indicate the original venue and will be accepted if the organizers think the work will benefit from exposure to the audience of this workshop. Work published elsewhere will not be included in the workshop proceedings. Please submit your published work here.
*Dual submissions
We allow submissions that are also under review in other venues, but please note that many conferences do not allow it, so make sure that you do not violate their policy as well. Please follow the double-submission policy from ACL. Accepted cross-submissions will be presented as posters, with an indication of the original venue if it's already accepted elsewhere.
*Anonymity period
We do not enforce an anonymity period. We do not restrict posting on preprint servers such as arXiv at any point in time.
***Important dates***
Apr 8, 2022: Research Paper Track Paper Due Date
Apr 29, 2022: Shared Task System Description Paper Due Date
May 6, 2022: Notification of Acceptance for Research Papers
May 14, 2022: Notification of Acceptance for System Descriptions
May 20, 2022: Camera-ready papers due
July 15, 2022: MIA Workshop
For any queries please contact:
Akari Asai (akari [at] cs.washington.edu) or mia.nlp.workshop [at] gmail.com.
References
Hu et al. "XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation." In ICML. 2020.
- Conneau et al. "Unsupervised cross-lingual representation learning at scale." In ACL. 2020.
- Xue et al. "mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer."" In NAACL. 2021.
- Petroni et al. "KILT: a Benchmark for Knowledge Intensive Language Tasks."" In NAACL. 2021.
- Asai et al. "XOR QA: Cross-lingual Open-Retrieval Question Answering."" In NAACL, 2021.
- Roy et al.. "Information asymmetry in Wikipedia across different languages: A statistical analysis." Journal of the Association for Information Science and Technology. 2021.
- Karpukhin et al. "Dense Passage Retrieval for Open-Domain Question Answering". In EMNLP. 2020.