CFP 
NLP Power! The First Workshop on Efficient Benchmarking in NLP


Second Call for Papers 


NLP Power! The First Workshop on Efficient Benchmarking in NLP co-located with ACL 2022, online


Workshop: May 26, 2022
ACL Conference: May 26–28, 2022
Website: https://nlp-power.github.io/ 
Paper submission: https://openreview.net/group?id=aclweb.org/ACL/2022/Workshop/NLP-Power


*** Paper submission deadline: February 28, 2022 ***



BACKGROUND

Benchmarks have played a crucial role in accelerating progress in the field of NLP, covering a wide range of research directions: natural language understanding (GLUE, SuperGLUE), natural language generation (GEM), cross-lingual knowledge transfer (XGLUE, XTREME), probing and interpretation (LINSPECTOR, SentEval), hate speech and bias (HateCheck, StereoSet, HONEST) and robustness to adversarial attacks (RobustnessGym,  AdvGLUE).  Despite the fact that the concept of benchmarking has become a standard practice for evaluating upcoming models against one another and human solvers, there are still a number of unresolved issues and methodological concerns.




TOPICS


The main objectives of this workshop are to (1) create a space for critical reflection on current benchmarks and evaluation tools, (2) encourage the development of improved or new benchmarks and evaluation tools that resolve current challenges, (3) develop better approaches to model ranking, (4) rethink benchmarking strategies that best account for computational costs, energy and ethical considerations, out-of-domain language capabilities and meeting the end-user preferences. We welcome submissions on ongoing and finished research and hope to provide an opportunity for participants to present their work and exchange ideas. Particular topics of interest include, but are not limited to:
* Computational efficiency and energy considerations in NLP benchmarks;
* New practices in measuring linguistic competence in mono- and multilingual benchmarks;
* Critical analysis of existing benchmark evaluation and construction designs;
* Guidelines for reproducibility and reliability of the benchmark results;
* Construction of zero-shot and few-shot mono- and multilingual benchmarks;
* Novel approaches to benchmark evaluation considering task complexity, model architecture, number of parameters and result aggregation;
* Applications of utility theory, voting theory and microeconomics to benchmark evaluation;
* User and application-specific model evaluation;
* New metrics and tasks for computationally lean comparison between models and measuring interpretability;
* Benchmarks and other evaluation methods to analyse ethical or social aspects of NLP tools;
* Analysis of human and model evaluation strategies in natural language understanding, text generation, knowledge transfer;
* Human evaluation protocols, specifically in the multilingual setting;
* Tracing biases and ethical issues in benchmark datasets and models.


We welcome submissions that identify with one of the aforementioned areas or present fall into these general:
1. Computational race & carbon footprints
2. Linguistic competence of the models
3. Reproducibility crisis 
4. Responsible innovation and ethics in NLP
5. Model and human evaluation design 
6. Application to real-world scenarios 
7. Data collection & leakage 


IMPORTANT DATES


* Jan. 28, 2022: Anonymity period begins
* Feb. 28, 2022: Workshop Paper Due Date 
* March 26, 2022: Notification of Acceptance
* April 10, 2022: Camera-ready papers due
* May 26-28, 2022: Workshop Dates (to be assigned)
Note: All deadlines are 11:59 PM UTC-12:00 (“anywhere on Earth”).


EVALUATION AND DECISION CRITERIA


Submissions will be reviewed in a double-blind manner and assessed based on
their novelty, technical quality, potential impact, and clarity. 


PAPER SUBMISSION

Submission is electronic, using the OpenReview conference management system.
Submission link: https://openreview.net/group?id=aclweb.org/ACL/2022/Workshop/NLP-Power

We accept three types of papers
1. Standard workshop papers: anonymized submissions describing substantially original research not previously published in other venues.
2. Extended abstracts: anonymized submissions describing preliminary but interesting ideas or results not previously published in other venues.
3. Cross-submissions: non-anonymized papers on relevant topics that have previously been accepted for publication in another venue.

Workshop papers cannot exceed 6 pages in length (excluding ethical considerations and references). The papers can have an optional appendix as described in ARR CFP guidelines. For example, preprocessing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper can be put into appendices. The reviewers are not required to consider the appendix during the review process.

The workshop will run its own review process, and papers can be submitted directly to the workshop by February 28, 2022. 

Both papers and abstracts must follow the ACL two-column format. Official style sheets: Overleaf template (https://www.overleaf.com/read/crtcwgxzjskr), Latex/Word template download (https://github.com/acl-org/ACLPUB/tree/master/templates).
Please do not modify these style files, nor should you use templates designed for other conferences. Submissions that do not conform to the required styles, including paper size, margin width, and font size restrictions, will be rejected without review.


DUAL SUBMISSIONS AND PREPRINTS

Dual submissions with the main conference are allowed, but authors must declare dual submission by entering the paper’s main conference submission id. The reviews for the submission for the main conference will be automatically forwarded to the workshop and taken into consideration when your paper is evaluated. Authors of dual-submission papers accepted to the main conference should retract them from the workshop by April 15.


Papers posted to preprint servers such as arxiv can be submitted without any restrictions on when they were posted.


ETHICS POLICY


Authors are required to honour the ethical code set out in the ACL Code of
Ethics, and comply with the ethics guidelines for ARR submissions.


CONTACT INFORMATION

Email: nlp_power@googlegroups.com



Read more:
https://nlp-power.github.io/
https://openreview.net/group?id=aclweb.org/ACL/2022/Workshop/NLP-Power

[1] https://aclrollingreview.org/cfp#multiple-submission-policy
[2] mailto:nlp_power@googlegroups.com