Report of the Program Chairs of ACL-2007 to the Executive Committee
Antal van den Bosch (Tilburg University, The Netherlands)
Annie Zaenen (PARC, USA)
Date: 21 May 2007
1. Planning and preparations
Planning and organisation of the reviewing process for the main
session of ACL-2007 commenced, largely via email between the two
program chairs and with additional communication with the ACL exec and
general chair John Carroll, in August 2006. The following areas and
area chairs were selected in the Fall of 2006:
Lexicon, lexical databases, ontologies, language resources:
Tim Baldwin (University of Melbourne, Australia)
Summarization, generation:
Kees van Deemter (University of Aberdeen, UK)
Pragmatics, dialog systems, discourse:
Barbara Di Eugenio (University of Illinois at Chicago, USA)
Syntax, parsing, formalisms:
Josef van Genabith (Dublin City University, Ireland)
Question answering, information extraction, information retrieval:
Claire Grover (University of Edinburgh, UK)
Semantics, lexical semantics, formal semantics, logic, textual entailment:
Diana McCarthy (University of Sussex, UK)
Machine learning, algorithms for NLP
Dan Roth (University of Illinois at Urbana-Champaign, USA)
Phonology, morphology, FS technology, tagging, word segmentation
Richard Sproat (University of Illinois at Urbana-Champaign, USA)
Speech, language modeling, spoken dialog systems
Marc Swerts (Tilburg University, The Netherlands)
Machine translation
Andy Way (Dublin City University, Ireland)
On top of the standard invitation and instructions, we asked the area
chairs to urge their reviewers to pay attention to the originality of
the submissions, to avoid the observed trend of going for safe and
sound but somewhat incremental work.
As submissions came in near deadline time, it became clear for most
areas that having about 25 reviewers would be too low for most areas
to keep the burden of reviewers at six papers, so more reviewers were
sought and added around the deadline date, to a total of 332
(i.e. over 33 on average per area). Some reviewers reviewed up to 11
papers but that was very rare; the average was 5.25, and the mode 7.
Florence Reeder provided mentoring services for 17 submissions.
She was assisted in the mentoring service by Charles Callaway, John White,
Bea Oshika, Ken Samuel, Deborah Dahl, Marilyn Kupetz, and Chrys
Chrystello.
Analogous to the scheme used for COLING-ACL 2006, we adopted a bidding
phase in which reviewers could voice preferences for reviewing
particular papers. This went smoothly and was judged positively
overall.
The schedule for the whole process was (and was kept at) the following:
Wed Jan 10 reviewers recruited by area chairs
Tue Jan 23 submissions arrive
Wed Jan 24 papers assigned to primary track/area
Thu-Mon Jan 25-29 reviewer bidding phase
Tue Jan 30 initial assignment of reviewers to papers using START
Thu Feb 1 final assignment of reviewers to papers by area chairs
Fri Feb 2 reviewing stage begins
Sun Mar 11 reviews due
Mon-Mon Mar 12-19 e-mail/START discussion amongst reviewers and
area chairs on "disagreement" papers
Mon-Thu Mar 19-22 area chair discussion/decisions on accept/reject
Fri Mar 23 author notification
Fri May 4 camera ready due
Sun-Fri Jun 24-29 ACL-2007 conference
The first call for papers was released around September 5, 2006. A
second call was released around December 1, 2006. The call was also in
ACL newsletters sent around regularly by Priscilla Rasmussen, and was
publicized on the conference web page.
2. Submission and Review Process
The submission deadline was January 23, 2007 (5pm US Eastern time,
10pm GMT). Notification of acceptance was set at March 23, 2007, after
discussion with the organizers of EMNLP-CoNLL who wanted a little room
between ACL’s notification date and their submission deadline (March
26), and sufficient room for their reviewing process. Camera ready
papers were due on May 4, 2007 and the final program was sent to the
publications chair about a week later.
We decided not to have an area chair meeting. Communication with the
area chairs (mostly through email) was unproblematic throughout the
entire process. We were confident that by maintaining a steady level
of detail, clarity, and speedy feedback in our email communication,
the decision making process could be performed using the same
channels. For the final decision making process, the two co-chairs met
for two days at Tilburg University, and interacted during that time
vigorously with the area chairs by email and sometimes by phone.
The program committee's selection of 131 papers was based on 588
submissions, a new record in an increasing trend. The original total
number of submissions was 605. The difference of 17 papers is
explained by 12 papers being withdrawn (mostly due to double
submissions, where authors opted for presentation at another venue,
while ACL reviewing was still ongoing), and by an unfortunate number
of 5 papers which did not comply with the anonymization guidelines
(after the authors were warned twice about this and were given a brief
period to correct the anonymity of their submissions).
All submissions received three reviews. A significant effort was
delivered by the area chairs who managed to get all of the reviews
completed in time for the final decision making process.
At decision making time we made the following steps. We asked all area
chairs to divide their papers into “definite accepts”, “borderline”,
and “definite reject” papers. We tentatively accepted all “definite
accepts”, totaling 100 papers. We then used the START system to rank
all remaining papers according to a formula that used the overall
recommendation score as a basis, and added the originality scores to
break the ties (using the START formula "Recommendation + (0.2 *
Originality)"). We ranked all “borderline” cases, and took a temporary
threshold of 4.27. This threshold marked 130 papers, which was the
total number of submissions that were recommended with priority by the
area chairs (the “definite accepts” plus any “borderline” papers that
they particularly recommended). We dropped “borderline” papers above
that threshold that were not recommended, and added papers from below
the threshold that did, resulting in a list of 131 papers to be
accepted. We proposed this list to the area chairs, who all agreed.
In the week leading to the notification date we decided to plan for
four parallel sessions. During decision making time it became clear
that even then we would not be able to accommodate more than about 110
papers if we would use 30 minute slots. We considered a lottery that
would reduce some papers to posters, but finally we decided to reduce
the time slots to 25 minutes, which was just enough to cover the 131
accepted papers. In sum, we managed to get the acceptance rate up to
22.3% and to accept all the papers on the area chairs’ priority
lists. Yet, to accept all the papers that were deemed worthy of being
presented at the ACL we would have had to accept around 150 papers.
See the table below for the distribution of the admissions and
accepted papers per area.
submissions
area # #accept %accept
----------------------------------------------------------
Discourse, Dialogue and Pragmatics 57 16 28
Syntax, Parsing and Formalisms 63 15 24
Lexicon, ontologies and resources 63 11 17
Morphology, etc. 60 13 22
Machine Learning, algorithms for NLP 67 16 24
IE, IR, QA 70 15 21
Machine Translation 65 15 23
Speech, Language Modeling 41 9 22
Generation and Summarization 48 10 21
Semantics etc. 54 12 22
----------------------------------------------------------
Total 588 131 22.3
The geographical distribution of the first authors of all submissions
is shown below. We received submissions from 45 countries. Asia & the
Pacific region accounted for 35% of all submissions, closely followed
by Europe also with 35%; 27% came from the North American continent,
2% from the Middle East, and less than 1% from South America and
Africa.
Continent #subm #acc
Asia & Pacific=========================================
11 countries
205/584 = 35.1% of submissions; 22/131 = 16.8% of accepted
Australia 18 4
China 74 6
Bangladesh 1
Hong Kong 10
India 11
Japan 53 3
Republic of Korea 6
Singapore 10 8
Sri Lanka 1
Taiwan 20 1
Thailand 1
Europe==================================================
23 countries
204/584 = 34.9%; 41/131 = 31.3%
Austria 1
Belgium 4 1
Czech Republic 6 1
Denmark 3
Estonia 2
Finland 3 1
France 21 4
Germany 32 6
Greece 3
Hungary 2
Iceland 2
Ireland 10 5
Italy 10 3
Netherlands 11 3
Poland 1
Portugal 3
Romania 3
Russian Federation 1
Spain 20
Sweden 8 1
Switzerland 5 2
Turkey 4
United Kingdom 49 14
South America =============================================
3/584 = <1%
Argentina 1
Brazil 1
Mexico 1
North America =============================================
157/584 = 26.9%; 61/131 = 46.6%
Canada 11 4
United States 146 57
Middle East ================================================
2 countries
11/584 = 1.9%; 6/131 = 4.6%
Iran 1
Israel 10 6
Africa =====================================================
2 countries
5/584 = <1%
Egypt 3
South Africa 1
Tunisia 1
3. Conference management system
As in previous ACL conferences, we used the START system to manage
submissions, bidding, and reviews. Things generally went well and we
got good support from Rich Gerber. Most minor problems were dealt with
quickly, and suggestions for minor improvements were either
immediately implemented or noted for later implementation.
4. Issues
The most salient issues that came up during the process and during
reflective discussions were the size of the whole process, conflicts
of interest, and dealing with dual submissions (both simultaneous and
serial submissions).
As for the size of the process, some flexibility was required from the
program and area chairs when it turned out that we would need to
process near 600 submissions, hence would need to have the reviewing
capacity for close to 1,800 reviews (in the end, 1,764 reviews were
written). Near future ACL conferences should expect similar numbers of
submissions. With ten areas it would have been optimal if the area
chairs would have been instructed to recruit about 30-35 reviewers
from the start. For the future this should be the norm, unless more
areas are discerned. We went for the ten areas listed, roughly following the EACL-2006 and ACL-2005 division of areas. This division
worked well, although we observed some overlap notably between the
three areas of semantics and lexical semantics, information extraction
and question answering, and discourse and dialog.
For the program chairs the large size of the program also meant a high
frequency of relatively small issues to handle: interactions with
authors about their submission, START issues (as all ACL events were
handled though START, where co-chair Van den Bosch handled the
initialization of START pages for all fifteen workshops and co-located
events) or merely side issues such as requests for more information.
The issue of conflicts of interest arose to a mild degree due to the
fact that most area chairs were co-authors of papers; one area chair
co-authored five submissions. Inevitably these papers were handled by
other area chairs, who had to carefully select reviewers for these
submissions. Although we have not seen evidence of this process
failing, there is an increased risk of errors. We believe that area
chairs should be made aware of this issue. At the same time we have
the impression that if area chairs would be advised against submitting
papers, it would be more difficult to recruit competent area
chairs. The program co-chairs did not co-author any submission.
Dual submissions present a problem as authors are not always
forthcoming with information, and as they tend to have different ideas
about what constitutes an almost identical paper. After some
discussion we resorted to having the authors of double submission tick
an option on the submission page to declare that the paper was
submitted elsewhere or was going to be submitted elsewhere during the
reviewing period. If ticked, submittors were also asked to fill in a
text box stating to which other conference or journal the paper was
submitted. Although as many as 43 authors provided this information,
we do not think that the problem has been solved completely
satisfactorily. At decision time we discovered that two papers that
were shortlisted for acceptance had already been presented and
published elsewhere - these papers were subsequently rejected. One
more paper was already scheduled for presentation at a workshop but
was retracted from that workshop in favor of ACL.
We discussed the problem with the area chairs, who did not express
worries so much towards simultaneous double submissions (on which the
ACL guidelines should be upheld), but rather at serial double
submissions to other events before and after ACL. The general opinion
was that a better coordination with the programme chairs of
conferences such as NODALIDA, NAACL, EMNLP, IWPT and CoNLL is
necessary, so that (information on) reviewers and reviews may be
shared, of course without compromising independence and fairness to
the authors who want of course as many chances as possible, and who in
fact may improve a rejected paper before resubmitting it
elsewhere. Information on papers and reviewers was indeed shared with
Jason Eisner, program chair of EMNLP-CoNLL.
Another proposal that came up in the course of the discussion with the
area chairs was to ask experimental papers to make the code, data and
scripts available to the reviewers.
4. Best Paper Award
This year’s conference continues the tradition of recognizing one of
the submitted papers with the Best Paper award. It will be selected by
the ten area chairs, the two program chairs, and the general chair,
based on a shortlist of papers proposed by the area chairs.
5. Acknowledgments
We thank John Carroll, General Conference Chair, the Local Arrangement
Committee headed by Eva Hajicova, and the ACL executive, especially
Dragomir Radev, for their help and advice, and last year's co-chairs,
Claire Cardie and Pierre Isabelle, for sharing their experience. Our
sincere thanks go to Su Jian for putting together the proceedings. We
thank Femke Wieme and Lauraine Sinay of Tilburg University for
checking paper formatting issues.