DragoRadev: Created page with "I have been developing a new system for managing the reviewer pool for ACL affiliated conferences. The primary purpose of this system is to 1) facilitate reviewer signup for a..."

2014-05-09T21:28:41Z

Created page with "I have been developing a new system for managing the reviewer pool for ACL affiliated conferences. The primary purpose of this system is to 1) facilitate reviewer signup for a..."

New page

I have been developing a new system for managing the reviewer pool for
ACL affiliated conferences. The primary purpose of this system is to
1) facilitate reviewer signup for area chairs and reviewers and 2)
efficiently allocate the reviewer pool to areas based on area need and
reviewer preference. In this system, area chairs email reviewers
asking them to volunteer. A reviewer them completes an online form in
which they can indicate which areas they would be willing to review
for. After papers are submitted, we calculate how many reviewers are
needed by area, and then assign the appropriate number of reviewers
factoring in the requests of the area chairs and the reviewer
preferences. Once reviewers are assigned to area, bidding proceeds as
usual. The process can be repeated for short papers, though we usually
reuse the area assignments from long papers.

Since the initial trial of the system in 2012, we have used it for 4-5
conferences, including ACL, NAACL and EMNLP. For some conferences
where we received an unexpected surge of papers in a few areas, the
system allowed us to respond without a last minute attempt to recruit
more reviewers or increase the load on a particular area. The process
has been refined each time based on feedback from the reviewers, area
chairs and program chairs. We have reached a stable version and wrote
up detailed instructions for running the process. For both ACL 2014
and EMNLP 2014, a graduate student has been in charge of running the
process based on the provided code and instructions. We believe the
process is now at the point where any conference can use the available
software to run the process themselves.

There has been some discussion of the next steps for the process. One
idea would be to integrate the system directly into the START review
platform. This would make managing the process easier, but it would
require cooperation of the START system, and reduce flexibility for
further refinements in future years.

Going forward, each conference's program chairs should be made aware
of this system and be encouraged to use it.

(by Mark Dredze)

2014Q3 Reports: Reviewing Software - Revision history

DragoRadev: Created page with "I have been developing a new system for managing the reviewer pool for ACL affiliated conferences. The primary purpose of this system is to 1) facilitate reviewer signup for a..."