Review Data Collection at *ACL: Difference between revisions
Created page with "''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' ''The documents has been prepared by Iryna Gurevych and Ilia..." |
No edit summary |
||
| Line 1: | Line 1: | ||
''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ||
''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).'' | ''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).'' | ||
=== Summary === | === Summary === | ||
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | ||
=== | === Documents === | ||
# The slide deck briefly summarizes the two proposals; | # The [[:File:1._Review_collection_at_ACL_(slides).pdf| slide deck]] briefly summarizes the two proposals; | ||
# The full document elaborates on the proposals in further detail. In addition, we provide | # The [[:File: 2._Review_collection_at_ACL.pdf| full document]] elaborates on the proposals in further detail. In addition, we provide | ||
# | # the example License agreements for [[:File: 3._ACL_License_Blind_Submission_v1.pdf| peer reviewing data]] | ||
# and blind submission data | # and [[:File: 3._ACL_License_Blind_Submission_v1.pdf| blind submission data]]. | ||
=== Contact === | === Contact === | ||
Revision as of 09:47, 30 March 2021
This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).
Summary
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.
Documents
- The slide deck briefly summarizes the two proposals;
- The full document elaborates on the proposals in further detail. In addition, we provide
- the example License agreements for peer reviewing data
- and blind submission data.
Contact
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:
Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Ilia Kuznetsov, kuznetsov@ukp.informatik.tu-darmstadt.de