ACL Conference Reviewer Awards Policy

From Admin Wiki
Jump to navigation Jump to search

This document proposes a standard protocol for reviewer awards at ACL conferences. The goal of introducing the policy is to have a consistent approach that suits the scale of our conferences today. It is designed to provide positive incentives for great service by volunteer reviewers and area chairs.

Award Types

There are typically three volunteer roles in the *ACL reviewing process: reviewers, area chairs, senior area chairs. There are corresponding awards for each of these roles:

  • Great reviewers: nominated by the area chairs;
  • Great area chairs: nominated by the senior area chairs;
  • Great senior area chairs: nominated by the program chairs.

Award Criteria

Great reviewers. All reviewers are expected to follow the review guidelines and perform all their tasks on time. Above that, the area chairs may nominate the 'great' reviewers for a number of criteria, including but not limited to the following:

  • rigorous reviewers, e.g. when the reviewer examines the provided code, data, or references, and reports hard-to-spot issues, or invests significant effort into learning a new technique to perform a better review;
  • emergency reviewers, done on short notice but with very high quality;
  • champion reviewers, who helped to rescue a paper from an unfair negative review;
  • engaged reviewers, who engaged in an unusually thoughtful discussion with either the authors or other reviewers, especially when that improved the assessment of the paper;
  • open-minded reviewers, who significantly changed their mind and assessment in the light of evidence brought by the authors or other reviewers;
  • patient reviewers, who were able to provide unusually kind and helpful feedback even to low-quality submissions.

Great Area Chairs. All area chairs are expected to follow their guidelines, provide clear recommendations, and perform all their tasks on time. Above that, the senior area chairs may nominate the 'great' area chairs for a number of criteria, including but not limited to the following:

  • in the face of imperfect reviews, writing high-quality meta-reviews that would be helpful to the authors and make up for some shortcomings in the reviews;
  • above-average level of detail in the meta-review, ideally based on reading the papers in question;
  • above-average proactive engagement with the authors of low-quality reviews, helping to improve the reviews and train the reviewers for the future review cycles;
  • managing an unusually high load of unresponsive reviewers and finding emergency substitutes on time.

Great Senior Area Chairs. The program chairs may choose to nominate the senior area chairs who were the most on-time with their duties, dealt with an unusual number of unresponsive ACs, spotted serious publication ethics issues, helped to catch early any ambiguities in the instructions or potential issues in the overall process, suggested useful improvements, etc.

Selection Process

Nominations

The reviewers can be nominated by the area chairs by marking the great reviews in the peer review system, after the reviews have been finalized. The instructions for meta-review should include the instructions for entering reviewer nominations. The interface for collecting nominations should include a field for a short justification for the nomination.

The area chairs can be nominated by the senior area chairs at the stage of entering the final recommendations. Likewise, the interface for entering the nominations should include a short justification or selection from pre-defined categories. The instructions for final recommendations should include the instructions for entering area chair nominations.

The program chairs can nominate the senior area chairs of one track directly.

The award justification text should be short (1-3 sentences) and not include any information that could directly identify the papers or the reviewers.

Selection process

Ideally, all reviewers/chairs who perform great service should be at least recognized as great reviewers/chairs, and hence there is no cap on the number of such nominations.

Per agreement with the ACL exec, 1.5-2.5% of total number of chairs/reviewers can be selected for receiving a monetary award. If there are more nominations for reviewers or area chairs than 2.5% of their total number in a given review cycle, a subset of nominees for receiving monetary awards will be randomly selected. The more nominations a given reviewer/chair received in a given cycle, the higher their chances for receiving the award.

Announcement, Certificates, and Awards

The reviewer and area chair awards will be announced on the ARR website for each cycle. For cycles associated with conferences, the conference website will also copy the list of nominees to the conference website or link to the ARR list. The ARR editors responsible for the given review cycle will communicate the award(s) to the reviewers/area chairs, ideally before the closest conference (in case they would like to use the prize to attend that same conference).

All awardees will receive an electronic certificate with the name of the ARR cycle, name of the award, and their name, signed by the ARR EiCs and/or conference Program Chair(s). These certificates will be generated and distributed by ARR in each cycle. Pending the development of reviewer history tools, all nominated reviewers/chairs should also have the record of their awards.

For the cycles directly associated with a conference, the program chairs will work with ACL director of events to coordinate the availability of 'great reviewer/chair' stickers for conference badges and posters, and the logistics of their distribution, based on a list of awardees provided by ARR. Similarly to the CVPR initiative, the papers of 'great reviewers/chairs' (if any) can display a special 'great reviewer/chair' sticker in the poster sessions and the programs. A comparable design should be developed for virtual participation.

The awardees selected for the monetary awards will receive a free attendee registration to a virtual *ACL event, or a corresponding discount to the on-site attendance. These awards can be used within a year of the award, at an event where the awardee is not a paper presenter. The rationale for this choice is that many institutions do not sponsor the attendance of the events for their students and employees, if they do not have a paper to present at that event. At the same time, when they do have a paper to present, awarding a registration fee waiver actually benefits the institution rather than the awardee. With these flexible-date attended fee waivers we hope that the awardees would be able to attend more events than they would otherwise. It will also be possible to exchange the virtual registration for a discount to the on-site attendance of an *ACL event (also to be used within a year of the award).

Logistics of the flexi-date fee waiver awards

The ARR editors-in-chief will be responsible for maintaining and updating a running record of awardees, with the following information: Award Type, First Name, Last Name, Affiliation, Country, Email, Backup Email, Date of Award, Date of Award Expiration. The possible values for "Award Type" are: Great Reviewer, Great Area Chair, Great Senior Area Chair.

When there is an upcoming conference, the ACL director of events will communicate to the ARR EiCs and program chairs the registration code that the awardees can use for that event. The chairs will communicate that code and registration instructions to the awardees who have an unused fee waiver, awarded within the year prior to the event. After the event, the list of used awards will be updated based on the list of registrations. With each subsequent review cycle, the list will be updated with new awards.

Notes on the development of this proposal

1.5-2.5% target awards were chosen to match the ACL awards policy. https://www.aclweb.org/adminwiki/index.php/ACL_Conference_Awards_Policy

For conferences colocated with a non-ACL event there may need to be some adjustments depending on the policies of the other event. This process was not developed for workshops or journals.

Retrieved from "https://www.aclweb.org/adminwiki/index.php?title=ACL_Conference_Reviewer_Awards_Policy&oldid=76344"