Difference between revisions of "ACL Conference Reviewer Awards Policy"

From Admin Wiki
Jump to navigation Jump to search
Line 44: Line 44:
  
  
The award justification text should be short (1-3 sentences)
+
The award justification text should be short (1-3 sentences) and not include any information that could directly identify the papers or the reviewers.
  
 
=== Nominations ===
 
=== Nominations ===

Revision as of 04:30, 2 May 2025

This document proposes a standard protocol for reviewer awards at ACL conferences. The goal of introducing the policy is to have a consistent approach that suits the scale of our conferences today. It is designed to provide positive incentives for great service by volunteer reviewers and area chairs.

Award Types

There are typically three volunteer roles in the *ACL reviewing process: reviewers, area chairs, senior area chairs. There are corresponding awards for each of these roles:

  • Great reviewers: nominated by the area chairs;
  • Great area chairs: nominated by the senior area chairs;
  • Great senior area chairs: nominated by the program chairs.

Award Criteria

Great reviewers. All reviewers are expected to follow the review guidelines and perform all their tasks on time. Above that, the area chairs may nominate the 'great' reviewers for a number of criteria, including but not limited to the following:

  • rigorous reviewers, e.g. when the reviewer examines the provided code, data, or references, and reports hard-to-spot issues, or invests significant effort into learning a new technique to perform a better review;
  • emergency reviewers, done on short notice but with very high quality;
  • champion reviewers, who helped to rescue a paper from an unfair negative review;
  • engaged reviewers, who engaged in an unusually thoughtful discussion with either the authors or other reviewers, especially when that improved the assessment of the paper;
  • open-minded reviewers, who significantly changed their mind and assessment in the light of evidence brought by the authors or other reviewers;
  • patient reviewers, who were able to provide unusually kind and helpful feedback even to low-quality submissions.


Great Area Chairs. All area chairs are expected to follow their guidelines, provide clear recommendations, and perform all their tasks on time. Above that, the senior area chairs may nominate the 'great' area chairs for a number of criteria, including but not limited to the following:

  • in the face of imperfect reviews, writing high-quality meta-reviews that would be helpful to the authors and make up for some shortcomings in the reviews;
  • above-average level of detail in the meta-review, ideally based on reading the papers in question;
  • above-average proactive engagement with the authors of low-quality reviews, helping to improve the reviews and train the reviewers for the future review cycles;
  • managing an unusually high load of unresponsive reviewers and finding emergency substitutes on time.


Great Senior Area Chairs. The program chairs may choose to nominate the senior area chairs who were the most on-time with their duties, dealt with an unusual number of unresponsive ACs, spotted serious publication ethics issues, helped to catch early any ambiguities in the instructions or potential issues in the overall process, suggested useful improvements, etc.


Selection Process

Nominations

The reviewers can be nominated by the area chairs by marking the great reviews in the peer review system, after the reviews have been finalized. The instructions for meta-review should include the instructions for entering reviewer nominations. The interface for collecting nominations should include a field for a short justification for the nomination.


The area chairs can be nominated by the senior area chairs at the stage of entering the final recommendations. Likewise, the interface for entering the nominations should include a short justification or selection from pre-defined categories. The instructions for final recommendations should include the instructions for entering area chair nominations.


The program chairs can nominate the senior area chairs of one track directly.


The award justification text should be short (1-3 sentences) and not include any information that could directly identify the papers or the reviewers.

Nominations