The GEM Workshop at EMNLP 2022
The Second Version of Generation, Evaluation And Metrics (GEM) Workshop 2022 workshop will be held as part of EMNLP, December 7-11, 2022. It is endorsed by the ACL Special Interest Group on Natural Language Generation (SIGGEN).
========
Overview
========
Natural language generation (NLG) is one of the most active research fields in NLP. Yet, much of the work is focused on English, and too little attention is given to evaluation processes. As many of the recent developments in few-shot and in-context learning have led to the treatment of many tasks as text generation problems, the need for better NLG evaluation processes is becoming more urgent. To that end, the GEM workshop aims to encourage the development of (semi-) automatic model audits and improved human evaluation strategies, and to popularize model evaluations in languages beyond English.
We welcome submissions related, but not limited to, the following topics:
- Automatic evaluation of NLG systems (example, example)
- Creating NLG corpora and challenge sets (example, example)
- Critiques of benchmarking efforts and responsibly measuring progress in NLG (example
, example)
- Effective and/or efficient NLG methods that can be applied to a wide range of languages and tasks (example, example)
- Standardizing human evaluation and making it more robust (example, example)
We additionally invite submissions that conduct in-depth analyses of outputs of existing systems, for example through error analyses, by applying new metrics, or by testing the system on new test sets. While we encourage the use of the infrastructure the organizing team is developing as part of the GEM benchmark, its use is not required.
If you are interested in seeing last year's workshop website from ACL 2021, please check here.
========
How to submit?
========
Submissions can take either of the following forms:
- Archival Papers Papers describing original and unpublished work can be submitted in a between 4 and 8 page format.
- Non-Archival Abstracts To discuss work already presented or under review at a peer-reviewed venue, we allow the submission of 2-page abstracts.
All submissions are allowed unlimited space for references and appendices and should conform to EMNLP 2022 style guidelines. Archival paper submissions must be anonymized while abstract submissions may include author information.
You can either commit a paper reviewed through ARR at here or submit directly through SoftConf here. Note that there are separate deadlines for the two options (see below)
We additionally invite presentations by authors of papers in the Findings of the EMNLP (details to be announced at a later date).
========
Shared Task
========
We are organizing a shared task focused on multilingual summarization, including human and automatic evaluation. Participants of the shared task are invited to submit a system description of 4-8 pages.
More information to come soon!
========
Important Dates
========
Paper Submission Dates
- 7 September 2022: Workshop paper submission deadline if using Softconf
- 2 October 2022: Latest ARR commitment deadline
- 9 October 2022: Workshop paper notification deadline
- 16 October 2022: Workshop paper camera ready deadline
Workshop Dates
- 7-8 December 2022: Workshops
========
Organization
========
- Antoine Bosselut (EPFL)
- Khyathi Chandu (Carnegie Mellon University)
- Kaustubh Dhole (Emory University)
- Varun Gangal (Carnegie Mellon University)
- Sebastian Gehrmann (Google Research)
- Yacine Jernite (Hugging Face)
- Jekaterina Novikova (NoOverfitting Lab)
- Laura Perez-Beltrachini (University of Edinburgh)
Steering Committee
- Wei Xu (Georgia Tech)
- Esin Durmus (Stanford University)
- Samira Shaikh (UNC Charlotte)