Report of the Program Chairs of ACL-2007 to the Executive Committee Antal van den Bosch (Tilburg University, The Netherlands) Annie Zaenen (PARC, USA) Date: 21 May 2007 1. Planning and preparations Planning and organisation of the reviewing process for the main session of ACL-2007 commenced, largely via email between the two program chairs and with additional communication with the ACL exec and general chair John Carroll, in August 2006. The following areas and area chairs were selected in the Fall of 2006: Lexicon, lexical databases, ontologies, language resources: Tim Baldwin (University of Melbourne, Australia) Summarization, generation: Kees van Deemter (University of Aberdeen, UK) Pragmatics, dialog systems, discourse: Barbara Di Eugenio (University of Illinois at Chicago, USA) Syntax, parsing, formalisms: Josef van Genabith (Dublin City University, Ireland) Question answering, information extraction, information retrieval: Claire Grover (University of Edinburgh, UK) Semantics, lexical semantics, formal semantics, logic, textual entailment: Diana McCarthy (University of Sussex, UK) Machine learning, algorithms for NLP Dan Roth (University of Illinois at Urbana-Champaign, USA) Phonology, morphology, FS technology, tagging, word segmentation Richard Sproat (University of Illinois at Urbana-Champaign, USA) Speech, language modeling, spoken dialog systems Marc Swerts (Tilburg University, The Netherlands) Machine translation Andy Way (Dublin City University, Ireland) On top of the standard invitation and instructions, we asked the area chairs to urge their reviewers to pay attention to the originality of the submissions, to avoid the observed trend of going for safe and sound but somewhat incremental work. As submissions came in near deadline time, it became clear for most areas that having about 25 reviewers would be too low for most areas to keep the burden of reviewers at six papers, so more reviewers were sought and added around the deadline date, to a total of 332 (i.e. over 33 on average per area). Some reviewers reviewed up to 11 papers but that was very rare; the average was 5.25, and the mode 7. Florence Reeder provided mentoring services for 17 submissions. She was assisted in the mentoring service by Charles Callaway, John White, Bea Oshika, Ken Samuel, Deborah Dahl, Marilyn Kupetz, and Chrys Chrystello. Analogous to the scheme used for COLING-ACL 2006, we adopted a bidding phase in which reviewers could voice preferences for reviewing particular papers. This went smoothly and was judged positively overall. The schedule for the whole process was (and was kept at) the following: Wed Jan 10 reviewers recruited by area chairs Tue Jan 23 submissions arrive Wed Jan 24 papers assigned to primary track/area Thu-Mon Jan 25-29 reviewer bidding phase Tue Jan 30 initial assignment of reviewers to papers using START Thu Feb 1 final assignment of reviewers to papers by area chairs Fri Feb 2 reviewing stage begins Sun Mar 11 reviews due Mon-Mon Mar 12-19 e-mail/START discussion amongst reviewers and area chairs on "disagreement" papers Mon-Thu Mar 19-22 area chair discussion/decisions on accept/reject Fri Mar 23 author notification Fri May 4 camera ready due Sun-Fri Jun 24-29 ACL-2007 conference The first call for papers was released around September 5, 2006. A second call was released around December 1, 2006. The call was also in ACL newsletters sent around regularly by Priscilla Rasmussen, and was publicized on the conference web page. 2. Submission and Review Process The submission deadline was January 23, 2007 (5pm US Eastern time, 10pm GMT). Notification of acceptance was set at March 23, 2007, after discussion with the organizers of EMNLP-CoNLL who wanted a little room between ACL’s notification date and their submission deadline (March 26), and sufficient room for their reviewing process. Camera ready papers were due on May 4, 2007 and the final program was sent to the publications chair about a week later. We decided not to have an area chair meeting. Communication with the area chairs (mostly through email) was unproblematic throughout the entire process. We were confident that by maintaining a steady level of detail, clarity, and speedy feedback in our email communication, the decision making process could be performed using the same channels. For the final decision making process, the two co-chairs met for two days at Tilburg University, and interacted during that time vigorously with the area chairs by email and sometimes by phone. The program committee's selection of 131 papers was based on 588 submissions, a new record in an increasing trend. The original total number of submissions was 605. The difference of 17 papers is explained by 12 papers being withdrawn (mostly due to double submissions, where authors opted for presentation at another venue, while ACL reviewing was still ongoing), and by an unfortunate number of 5 papers which did not comply with the anonymization guidelines (after the authors were warned twice about this and were given a brief period to correct the anonymity of their submissions). All submissions received three reviews. A significant effort was delivered by the area chairs who managed to get all of the reviews completed in time for the final decision making process. At decision making time we made the following steps. We asked all area chairs to divide their papers into “definite accepts”, “borderline”, and “definite reject” papers. We tentatively accepted all “definite accepts”, totaling 100 papers. We then used the START system to rank all remaining papers according to a formula that used the overall recommendation score as a basis, and added the originality scores to break the ties (using the START formula "Recommendation + (0.2 * Originality)"). We ranked all “borderline” cases, and took a temporary threshold of 4.27. This threshold marked 130 papers, which was the total number of submissions that were recommended with priority by the area chairs (the “definite accepts” plus any “borderline” papers that they particularly recommended). We dropped “borderline” papers above that threshold that were not recommended, and added papers from below the threshold that did, resulting in a list of 131 papers to be accepted. We proposed this list to the area chairs, who all agreed. In the week leading to the notification date we decided to plan for four parallel sessions. During decision making time it became clear that even then we would not be able to accommodate more than about 110 papers if we would use 30 minute slots. We considered a lottery that would reduce some papers to posters, but finally we decided to reduce the time slots to 25 minutes, which was just enough to cover the 131 accepted papers. In sum, we managed to get the acceptance rate up to 22.3% and to accept all the papers on the area chairs’ priority lists. Yet, to accept all the papers that were deemed worthy of being presented at the ACL we would have had to accept around 150 papers. See the table below for the distribution of the admissions and accepted papers per area. submissions area # #accept %accept ---------------------------------------------------------- Discourse, Dialogue and Pragmatics 57 16 28 Syntax, Parsing and Formalisms 63 15 24 Lexicon, ontologies and resources 63 11 17 Morphology, etc. 60 13 22 Machine Learning, algorithms for NLP 67 16 24 IE, IR, QA 70 15 21 Machine Translation 65 15 23 Speech, Language Modeling 41 9 22 Generation and Summarization 48 10 21 Semantics etc. 54 12 22 ---------------------------------------------------------- Total 588 131 22.3 The geographical distribution of the first authors of all submissions is shown below. We received submissions from 45 countries. Asia & the Pacific region accounted for 35% of all submissions, closely followed by Europe also with 35%; 27% came from the North American continent, 2% from the Middle East, and less than 1% from South America and Africa. Continent #subm #acc Asia & Pacific========================================= 11 countries 205/584 = 35.1% of submissions; 22/131 = 16.8% of accepted Australia 18 4 China 74 6 Bangladesh 1 Hong Kong 10 India 11 Japan 53 3 Republic of Korea 6 Singapore 10 8 Sri Lanka 1 Taiwan 20 1 Thailand 1 Europe================================================== 23 countries 204/584 = 34.9%; 41/131 = 31.3% Austria 1 Belgium 4 1 Czech Republic 6 1 Denmark 3 Estonia 2 Finland 3 1 France 21 4 Germany 32 6 Greece 3 Hungary 2 Iceland 2 Ireland 10 5 Italy 10 3 Netherlands 11 3 Poland 1 Portugal 3 Romania 3 Russian Federation 1 Spain 20 Sweden 8 1 Switzerland 5 2 Turkey 4 United Kingdom 49 14 South America ============================================= 3/584 = <1% Argentina 1 Brazil 1 Mexico 1 North America ============================================= 157/584 = 26.9%; 61/131 = 46.6% Canada 11 4 United States 146 57 Middle East ================================================ 2 countries 11/584 = 1.9%; 6/131 = 4.6% Iran 1 Israel 10 6 Africa ===================================================== 2 countries 5/584 = <1% Egypt 3 South Africa 1 Tunisia 1 3. Conference management system As in previous ACL conferences, we used the START system to manage submissions, bidding, and reviews. Things generally went well and we got good support from Rich Gerber. Most minor problems were dealt with quickly, and suggestions for minor improvements were either immediately implemented or noted for later implementation. 4. Issues The most salient issues that came up during the process and during reflective discussions were the size of the whole process, conflicts of interest, and dealing with dual submissions (both simultaneous and serial submissions). As for the size of the process, some flexibility was required from the program and area chairs when it turned out that we would need to process near 600 submissions, hence would need to have the reviewing capacity for close to 1,800 reviews (in the end, 1,764 reviews were written). Near future ACL conferences should expect similar numbers of submissions. With ten areas it would have been optimal if the area chairs would have been instructed to recruit about 30-35 reviewers from the start. For the future this should be the norm, unless more areas are discerned. We went for the ten areas listed, roughly following the EACL-2006 and ACL-2005 division of areas. This division worked well, although we observed some overlap notably between the three areas of semantics and lexical semantics, information extraction and question answering, and discourse and dialog. For the program chairs the large size of the program also meant a high frequency of relatively small issues to handle: interactions with authors about their submission, START issues (as all ACL events were handled though START, where co-chair Van den Bosch handled the initialization of START pages for all fifteen workshops and co-located events) or merely side issues such as requests for more information. The issue of conflicts of interest arose to a mild degree due to the fact that most area chairs were co-authors of papers; one area chair co-authored five submissions. Inevitably these papers were handled by other area chairs, who had to carefully select reviewers for these submissions. Although we have not seen evidence of this process failing, there is an increased risk of errors. We believe that area chairs should be made aware of this issue. At the same time we have the impression that if area chairs would be advised against submitting papers, it would be more difficult to recruit competent area chairs. The program co-chairs did not co-author any submission. Dual submissions present a problem as authors are not always forthcoming with information, and as they tend to have different ideas about what constitutes an almost identical paper. After some discussion we resorted to having the authors of double submission tick an option on the submission page to declare that the paper was submitted elsewhere or was going to be submitted elsewhere during the reviewing period. If ticked, submittors were also asked to fill in a text box stating to which other conference or journal the paper was submitted. Although as many as 43 authors provided this information, we do not think that the problem has been solved completely satisfactorily. At decision time we discovered that two papers that were shortlisted for acceptance had already been presented and published elsewhere - these papers were subsequently rejected. One more paper was already scheduled for presentation at a workshop but was retracted from that workshop in favor of ACL. We discussed the problem with the area chairs, who did not express worries so much towards simultaneous double submissions (on which the ACL guidelines should be upheld), but rather at serial double submissions to other events before and after ACL. The general opinion was that a better coordination with the programme chairs of conferences such as NODALIDA, NAACL, EMNLP, IWPT and CoNLL is necessary, so that (information on) reviewers and reviews may be shared, of course without compromising independence and fairness to the authors who want of course as many chances as possible, and who in fact may improve a rejected paper before resubmitting it elsewhere. Information on papers and reviewers was indeed shared with Jason Eisner, program chair of EMNLP-CoNLL. Another proposal that came up in the course of the discussion with the area chairs was to ask experimental papers to make the code, data and scripts available to the reviewers. 4. Best Paper Award This year’s conference continues the tradition of recognizing one of the submitted papers with the Best Paper award. It will be selected by the ten area chairs, the two program chairs, and the general chair, based on a shortlist of papers proposed by the area chairs. 5. Acknowledgments We thank John Carroll, General Conference Chair, the Local Arrangement Committee headed by Eva Hajicova, and the ACL executive, especially Dragomir Radev, for their help and advice, and last year's co-chairs, Claire Cardie and Pierre Isabelle, for sharing their experience. Our sincere thanks go to Su Jian for putting together the proceedings. We thank Femke Wieme and Lauraine Sinay of Tilburg University for checking paper formatting issues.