2024Q1 Reports: Ethics Committee Co-chairs
N.B.: This document is authored expressly for the purpose of informing the ACL Executive Board as required for the Quarter 1 2024 documentation. It is meant to be read by the ACL Executive Board and is also suitable for the ACL membership and the general public.
After 2023 Q3, the Ethics Committee responded to a few issues that required our attention. These issues sidelined scheduled work on internal documentation progress which we hope to restart. As a reminder, our committee is structure with four subcommittees:
- #Ethical Reviewing Guidelines
- #Ethics Tutorial, Education and Outreach
- Committee Scope
- Committee Procedure
The Committee Procedure and Committee Scope subcommittees have not made much progress suitable for reporting since the previous half year, so we have left them out of this update. We first discuss two issues that consumed most of the deliberative bandwidth of our committee.
Issue 1. Pre-print survey
The ACL Executive instituted an ACL Survey on Anonymity Period Policy Committee to poll the ACL constituency about their feelings on the (now past) pre-print policy. This was done without consulting our committee. We identified issues that may affect the validity and representativeness of the survey and possible action repercussions of any downstream action (that was ultimately taken by the executive).
As we felt that the issue was pertinent to ethics in its wider scope, we deliberated on our action and ultimately wrote to the committee directly so as to not to disrupt ongoing work and deliberation unknown to us. Our recommendations were acknowledge, but some of our members (we were not unanimous on our own internal deliberations) were unsatisfied with the outcomes of the survey.
We include our recommendations to the committee here for the public record:
Dear ACL Survey on Anonymity Period Policy Committee:
Thank you for doing the community a great service in helping to get aclearer picture of our membership’s varied perspectives on the topic of preprints and, relatedly, of anonymity in peer review. This is a sensitive subject with heated arguments on both sides, and care in communication is important. We are writing to you to offer any assistance we may render to you, as we feel that there are ethical issues related to the current preprint environment.
In particular, (double) blind review and the representativeness of the survey outcomes are two issues that we would like to call attention to.
Blind review has been shown to lessen bias in publication, as names of well-known institutions and authors have been scientifically shown to bias the quality judgement of work [3,4,2,5]. We believe that the institution of blind review needs to be preserved to ensure that work is only judged on scientific merit and not by any other ancillary factor, such as prestige or author demographics [1]. These are factual observations that are worthwhile to consider in your recommendations, as (changes in) pre-print policies affect anonymity. ArXiv does not allow nor mandate anonymous submission.
Our Recommendation: Any preprint platform that our community uses should protect double blind review and require anonymous submission.
We appreciate your multipronged attempt to solicit participation in the survey through multiple emails and social media advertisements. Despite this effort, there may be a self-selection bias in survey participation.
Our Recommendation: Ensure that all voices are heard. We urge the committee to attend to underrepresented minorities and groups that will be disproportionately affected by changes to the pre-print policy. Such groups may not be on social media and may be better served by other means. One concrete means of realising this is to convene focus groups that include such populations, as they may discover new issues. A survey then can validate and measure the prevalence of these issues. We encourage adding representation from ACL chapters and groups that are not currently well represented, such as researchers from the global south and EquiCL, the ACL DEI interest group.
Current social media platforms may not be the most appropriate vehicle for nuanced rationale discussion on such topics. Twitter/X’s agenda may encourage divisiveness and exclusion among our membership, which does a disservice to our community.
Our Recommendation: The survey committee needs to keep in mind that social media platforms represent only a fraction of our community. Dissemination of survey results should not rely on social media as a means of discussion.
We would urge you to consider these factors in helping to shape the narrative with respect to the interpretation of your survey’s outcome. Please do reach out to us if you feel that we can be helpful.
The ACL Ethics Committee
References: [1] Kaatz A, Gutierrez B, Carnes M. Threats to objectivity in peer review: the case of gender. Trends in pharmacological sciences. 2014;35(8):371-373.
[2] Tomkins A, Zhang M, Heavlin W. Single vs. Double Blind Reviewing at WSDM, https://arxiv.org/pdf/1702.00502.pdf
[3] Okike K, Hug KT, Kocher MS, Leopold SS. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA. 2016 Sep 27;316(12):1315-6.
[4] Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci U S A. 2017 Nov 28;114(48):12708-12713. doi: > 10.1073/pnas.1707323114. Epub 2017 Nov 14. PMID: 29138317; PMCID: PMC5715744.
[5] [FR] http://www.ethique-et-tal.org/2017/03/17/relecture-par-les-pairs-un-etat-de-lart/
Issue 2. Appropriate use of the ethics consideration section
As certain camera-ready versions of EMNLP 2023 papers contained non-peer reviewed statements that particular members of the public and our committee thought were misused, our committee also deliberated on what action to take. As of this timing, the committee has not taken an official action, as we understand that there are again various members of the executive working on these delicate issues. We are still formulating a possible action.
Ethics Survey
The report is deposited as part of the archival resources for the EACL tutorial (mentioned below):
Yulia Tsvetkov and Karën Fort and Min-Yen Kan all co-chair this subcommittee.
Ethical Reviewing Guidelines
Conferences and other events have largely taken ethics directive from the ARR Ethics guidelines for which our committee has offered our feedback on, through shared committee members (Dirk Hovy and Pascale Fung).
As part of the outreach effort at ACL 2023, Dirk Hovy will report on our tripartite (authoring, reviewing, event organising) guidelines. We will highlight some specific challenges of ensuring that ethics reviewing is considered a first-class duty, alongside and complementary to scientific reviewing.
This subcommittee is chaired by Yulia Tsvetkov, assisted by Karën Fort.
Ethics Tutorial, Education and Outreach
We ran an ethics tutorial Understanding Ethics in NLP Authoring and Reviewing at EACL 2023 (May, Dubrovnik).
Luciana Benotti, Karën Fort, Min-Yen Kan and Yulia Tsvetkov were the presenters and authors for the tutorial.
As part of the charges for the tutorial, we created educational resources in terms of an introductory 60-minute presentation merging two of the authors' (Karen and Yulia's) lecture notes. We also created a number of problematic synthetic abstracts (based on real studies, but without the stigma that would be result from attributing a problematic abstract to an author) for participants to analyse. We also described materials such as the reviewing guidelines, ethics reading list, and an analyses of Ethical Considerations and Limitations sections in NLP/CL papers (contributed by Luciana Benotti) as part of the tutorial's open-source contents.
At EACL 2023, we had about 20 participants in total join the tutorial, with 1 remote participant.
We have archived and documented all of the tutorial's materials at its associated Github repository link. The ACL Ethics Committee Report on its 2021 Survey (mentioned above) is also deposited there as well:
https://github.com/acl-org/ethics-tutorial
With ACL 2023 culminating in the successful delivery of the survey, and EACL 2023 the tutorial, we will likely work towards a special journal issue call for papers themed on ethics, again as part of outreach to build a community of scholars interested in these topics. The Computational Linguistics journal has agreed that this would be within their purview and of interest to its audience.