Guidelines for ACL Program Committee
Purpose of Document
Many policy questions came up during the two ACL program committee (PC) meetings I attended (1992 and 1993). In many cases, individuals who had served on the ACL PC in the past or had served on the PCs of other major conferences were able to provide suggestions that enabled decisions to be made and the meeting to progress. These individuals typically cited what was done in the past in similar situations. Unfortunately, what has been done from year to year has not always been consistent, and this led to a general sense that a set of guidelines should be collected and written down to act as a kind of institutional memory that can be passed from PC to PC each year.
To this end, an e-mail message was sent to all members of the ACL ‘93 PC and to several past PC chairs asking for views on a set of questions/issues that came up at the meeting and inviting comments on other PC-related issues. The document that follows is my summary of the responses that I received. I viewed my task to be the collection and organization of information to be passed on to the ACL Executive Committee (EC) in order to aid them in setting policy. I have tried to represent all views on the topics. Additional comments and suggestions are welcome and encouraged.
Definition of the Area
When is a paper relevant for ACL? In some cases, there were papers submitted that seemed like good work in a given discipline (e.g., "pure" linguistics or formal language theory), but it was not clear how that work related to Computational Linguistics.
There was some discussion about what to do with papers that describe tools. Those surveyed suggested that tools papers should be allowed and encouraged if:
Every year the PC chair receives several papers that arrive after the deadline and must decide what to do with these. There seems to have been an informal understanding among some in the ACL community that papers received within a "reasonable" time after the deadline would be reviewed and considered for publication in the conference as if they had arrived on time. The decision to review papers received after the deadline and the determination of what constitutes a "reasonable" time are typically left to the discretion of the program chair who could consult the entire committee if he or she chose. In 1993, Len Schubert consulted the committee about this issue and a large majority of the PC members felt strongly that papers received after the deadline indicated in the Call for Papers (CFP) should be returned to the authors unopened. Everyone agreed that a policy for late submissions must be set and made explicit in the CFP. There is a general sense that the rules should be well known and apply equally.
By far the majority still feels that the deadline should be strictly enforced. Many authors work long and hard to submit papers on time. They stay up late working round the clock and send papers via express mail. These papers could no doubt be improved if the author(s) had another few days or a week to polish them. The majority feels that in the interest of fairness, all authors should be subject to the same deadlines.
A very small minority feels that there should be a few days’ grace period, whose exact length should be covertly determined by the PC chair, based on her or his judgement and constraints at the time.
How the Deadline Should be Stated:
Several suggested that in the future the deadline be given in terms of a "postmarked by" date, rather than a "must be received by" date.
Several members of the committee suggested that We allow electronic submissions. This would:
Some pointed out that electronic submission is feasible only for authors who have the capability to get papers into postscript form and to e-mail or ftp their papers to the PC chair. Although this is true of the majority of authors in the USA, Western Europe and Japan, we cannot assume that this is true for all authors. Therefore, we must continue to allow hardcopy submissions. However, we should try to use electronic mail to the extent possible.
For those who do not have access to appropriate e-mail tools for submission, faxing is an option.
Submissions that Violate Restrictions:
Every year the PC must tackle the question of what to do with papers that clearly violate the length restriction. While a small number of PC members felt that handling of such papers should be left up to the individual reviewers that received them, the majority felt that all papers should be checked for compliance with length restrictions by an administrator and those that are in clear violation should be returned to authors without review. Many felt strongly that this decision should NOT be left up to individual reviewers as this would lead to inconsistent treatment of papers violating length restrictions.
By far the majority feel that length restrictions must be adhered to. One of the most difficult parts of writing an abstract for something like ACL is compressing ideas into 3200 words. Some PC members were appalled by the fact that the PC reviewed, and even accepted some papers that were more than double the length specified in the CFP. This was considered extremely unfair to authors who may have lessened their chances for acceptance by cutting their papers to meet the length restrictions.
Specification of Length Restrictions:
If we are going to enforce length restrictions, then we need to specify length restrictions in such a way that we can easily check whether or not a paper conforms. Word count is a very hard measure to check. Specifications that detail the font size, margin size, page size, and number of pages seem better. To help authors make sure they meet these guidelines we should provide Latex, Scribe, and MS Word style files that meet ACL specifications. (We could get these from DARPA.)
A problem that many authors have noted is that the abstact length is often longer than the final paper length, leaving authors no room to address reviewers’ comments. If we gave out style files from the beginning, we could alleviate this problem (since authors would know the final camera-ready paper length from the outset, and/or we could require that submissions be 1/2 page shorter than final papers.)
Of course, even if we move to style files, we will
need to provide specifications for authors who do not have access to Latex,
Scribe, MS Word,etc. Since we expect the number of people in this category
to be small and dwindling, this should not be a problem.
Submission to Multiple Conferences
A small number of PC members feel that ACL (and other conferences) should not allow multiple submissions. They feel that submitting to multiple conferences is an abuse of the reviewing process. It wastes reviewers’ time and causes problems with the conference schedule if authors retract their paper from ACL after it has been accepted.
However, the majority feel that allowing multiple submissions is good compromise between conference’s desires to get their programs finalized and authors’ desires to get their work published while it is still fresh. Due to the number of relevant conferences that have submission deadlines in November—February, authors must be allowed to submit to multiple conferences.
All agree that if multiple submissions are allowed, authors must declare this fact up front and agree to publish in only one place. Failure to explicitly declare multiple submissions should be grounds for rejection.
Most feel that it’s up to the PC chair to coordinate with other conferences (e.g., AAAI, COLING, EACL, Cognitive Science) to see that the same paper isn’t going to be presented at more than one of them, and it’s up to reviewers to make sure that papers haven’t been published (outside of workshop proceedings) before.
Managing the Schedule if Multiple Submissions are allowed:
A serious problem is what to do with multiply submitted papers that are accepted to ACL as well as one or more other conferences. There was concern about how this will affect the schedule. Here are several suggestions for the EC to consider:
A number of questions arose regarding the nature of prior publication of work that would make a paper ineligible for publication in the ACL proceedings. The following suggestions were almost unanimous: Workshops were the hardest to deal with.
One person suggested that guidelines for workshops should take account of:
However, if papers from a workshop are reviewed, authors are given time to revise their papers, and the papers are published in a book or proceedings, then these papers should not be eligible for publication in ACL. (For example, the AI and Planning Systems Workshop is almost indistinguishable in format from ACL, but it’s called a workshop, so we can’t simply go by the title the organizers give.)
Whatever is decided, it has been suggested that the following are good rules of thumb:
There is no agreement on this question. Here are some comments the EC may want to take into account when making their decision on this issue:
Several people felt that different reviewers were using different metrics for scoring and therefore it was difficult to compare scores given by different reviewers.
Several suggestions about how to alleviate this problem were made:
1) Papers should be available to all committee members.
Some PC members feel that it is crucial that every PC member have a chance to browse through all the abstracts and review whichever they wish to review. This year PC members saw only the papers they were officially asked to review and, since the committee was large, this number was very small (15-20). Without the context of the other abstracts, PC members felt that they couldn’t tell whether their sample of a given field was representative, or whether a given paper was better or worse than another similar paper that they weren’t officially reviewing and didn’t even know existed until the PC meeting. Some felt that it was this problem that made the PC meeting itself seem rather extraneous, since it was very hard to participate in discussions of papers one hadn’t even seen, and it was hard to have any idea whether the decisions overall were fair.
It was generally agreed that if all papers are made available to all PC members, other PC members might provide useful additional comments on papers for which they were not the primary reviewers, and this would likely strengthen the basis for decisions. However, it was noted that this could allow very ‘aggressive’ PC members to have undue influence in the process by reading all papers and voicing their opinions on them.
2) Some past PC chairs have used spread sheets to calculate statistics such as the mean, mode, and median of each PC members’ scores. This was done at the ACL ‘92 PC meeting and the ACL Applied meeting and was very useful in interpreting reviewers’ ratings.
3) Some PC members said they were accepting papers with technical problems, because the idea was interesting, while others would reject such a paper. Clearly, we want to encourage new and interesting ideas, but being incorrect seems very different than having correct but somewhat preliminary results. We need to attempt to operationalize / standardize the meaning of ratings such as "accept," "marginal" etc.
4) The discrepancy may not be so much between individual reviewers, but between subareas of the program committee. One way to overcome this might be by having more discussion between reviewers, before the plenary program committee meeting. If reviewers in each subarea identify what they think are the best papers in that area, possibly in some kind of order, then the plenary session can be devoted to constructing a program out of that, without having to compare numbers on individual papers issued by individual reviewers.
5) One person suggested that reviewers just be forewarned that for a 5-point scale from definite accept to definite reject, experience shows that very few papers with ratings less than second-highest will ultimately be included. That should discourage reviewers from employing standards they subsequently regret.
Although many felt that this was less than successful
because they were able to guess who the authors of papers were, most felt
that ACL should continue this practice. If continued, ACL should give authors
more guidelines about how to avoid recognition. In many cases, reviewers
were able to identify authors because authors lack experience about how
to reference their own work or deal with acknowledgements when submitting
a paper for blind review.
Submission by Program Committee
Most agree that members of the PC should be allowed to submit papers and that their papers should be treated in the same manner as all other submissions, with additional care to keep reviewers anonymous.
Some PC members thought this was handled very well
in ‘93, where the PC chair notified authors of accept/reject decisions
by handing out slips of paper.
Selection of Invited Speakers
Many felt that this is often handled in an ad hoc fashion because it is left to the time that remains after the decisions on papers are made. One person suggested that choosing invited speakers be done by e-mail as soon as the PC is formed in November/December. This will increase our chances of getting the speakers we want. As it is now, we end up asking a relatively famous person in mid-March to commit to speaking in late June. Several times our first choice(s) are already booked.
As to the criteria for selecting invited speakers, several PC members feel that it is preferable to invite speakers outside of the general range of ACL topics.
Although a couple of people found the ‘93 committee
consisting of 22 members to be too large, some felt that the larger size
had many advantages. It made the reviewing load more modest, allowing more
careful reviewing. It also allowed all of the reviewing (save a few instances
of delegation) to be done within the committee. This has the crucial advantages
that (a) PC members have already set aside time for the job and made a
commitment to be punctual, whereas finding reviewers ad hoc and holding
them to deadlines is much harder, and (b) all the papers get discussed
by all the reviewers at the meeting. Even with the committee of 22, some
problems arose in trying to find 3 expert reviewers for some papers.
Need For PC Meeting
Comments in favor:
The whole point of having an in-person meeting is to take advantage of as much of our expertise as possible. Yet it is virtually impossible for a person to have a reasonable opinion to contribute if he or she hasn’t seen a paper. The problems this year were largely due to the fact that most PC members saw only 15—20 of the papers. To make sure every good paper has ample chance to be discovered and every not-so-good paper has ample chance to be correctly evaluated, we should maximize papers’ exposure to members of the PC rather than concluding that we don’t need a PC meeting. We do need a PC meeting, but we nee to come to it having looked at, if not laboriously reviewed, more papers. This is what makes ACL different from other conferences, and different in a good way. It also makes the reviewing process fairer. If we make all the papers available to everyone (see above), the meeting will potentially provide new information.
This year the number of papers accepted pre-meeting was just about the right quantity. Other years might have way too many or too few papers accepted on a first pass. The meeting is useful if papers have to be reconsidered.
Many rating errors come more from personal biases that are overcome by discussion.
The sense of group ownership of the process is very important, and this would be lost in an electronic approach.
The face-to-face meeting is importantunless you have a program committee consisting solely of people who really know each other very well already. It’s perhaps only in retrospect that we think we could have done it all via email since we now know everyone on the program committee. However, I would have liked to have seen more discussion of the program that was emerging, rather than just have discussion of single papers by two or three committee members at a time.
The PC meeting can be held via e-mail. The ACL ‘93 PC committee assigned 2 reviewers to each paper. Reviewers were given a deadline by which they were expected to have written up their reviews and exchanged them with their co-reviewer. If the 2 reviewers could not come to consensus, they asked for a third review. The PC chair selected a third reviewer and typically a decision could be reached. A few members felt that this was sufficient and that no meeting was necessary.
Specific suggestions for how to manage an all electronic PC meeting:
Commitment by Program Committee
Several PC members stated that based on their own
ACL experiences, and comments they have heard from others at different
ACL program committee meetings, it is imperative that all PC members be
present at the PC meeting. Having the review is not the same as having
the reviewer for the difficult cases.
Makeup of Program Committee
In recent years there has been an attempt to bring new faces onto the ACL PC. While everyone agrees this is important for the community, we must be careful to ensure continuity. Therefore several have suggested that there be an attempt to balance the number of new PC members with persons that have served on the ACL PC in the past.
There was a proposal for a panel, yet many committee members felt that panels were not solicited in the CFP so it was unfair. However, in the past there have been panels and no solicitation, and workshops and tutorials seem to be done without a proposal solicitation. AAAI and IJCAI have explicit calls for these things, and many PC members suggest that ACL do the same. A panel, workshop, or tutorial chair could still seek out people as is done now.
Conflict of Interest
Some PC members heard of three cases where PC members
were sent papers from their home institution to review. In one case the
PC member knew there was a conflict and returned the paper. In another
case the paper was kept even though the reviewer suspected it was from
their institution (and it was). The members that reported this feel that
this must be prohibited in the future. NSF has quite an explicit policy
regarding what constitutes a conflict of interest, which might prove instructive.
Author: Johanna Moore, June 1993.