NAACL-01 Program Chair Report Kevin Knight June 13, 2001 1 The Program For this year's NAACL, we got 110 submissions, of which 31 papers were accepted, for a main session acceptance rate of 28%. Here is the submission profile by region: 80 from North America (73%), 19 from Europe (17%), 11 from Asia (10%). Accepted papers included 22 from North America, 8 from Europe, 1 from Asia -- according to the location of the contact author. A high degree of collaboration indicates more participation from all regions. By contrast, last year's NAACL received 166 submissions and accepted 43, for an acceptance rate of 24%. ACL-97, ACL-99, and ACL-00 had acceptance rates between 24% and 26%. I invited three speakers specifically in areas just a bit outside those researched by "typical ACLers" (as has been the custom). The speakers were: Jon Kleinberg ("Structure and Content in World Wide Web Search"), Tom Mitchell ("Machine Learning and Extracting Information from the Web"), and Aravind Joshi ("Language Modeling of Biological Data"). The paper presentations and invited talks fit into a "single-session" format. 2 The Process We used a web-based "notification of submission" form that generated code numbers for submissions. This was extremely useful in tracking papers throughout the entire process. We used hard copy for actual submission and distribution of papers to reviewers. This ensured that reviewers spent the maximum amount of time reviewing, and no time navigating or printing. This was important for them, given the extremely tight holiday-infested reviewing schedule. The burden of printing and copying was thereby placed on the submitting authors. It was not necessary to maintain a secure 24/7 web site throughout November, December, and January. We used electronic mail for review submission and discussion among reviewers. This was serviceable, but required substantial amounts of work in passing the reviews up to the senior program commitee and program chair. We used electronic mail for author notification. This worked fine. Reviewing was blind. I recruited the senior program committee (SPC): Eric Brill, Ann Copestake, Marti Hearst, Aravind Joshi, Andrew Kehler, Elliott Macklovitch, Fernando Pereira, Owen Rambow, Elizabeth Shriberg, and Ralph Weischedel. They were involved in many time-consuming activities, including assembling the program committee of 111 reviewers, assigning reviewers to papers, organizing in-depth discussions among reviewers of many papers, assembling reviews, and participating in the SPC meeting held in Los Angeles. I would like to thank them for this very hard work! In contrast to previous years, SPC members were not assigned to particular sub-areas, but recruited for wide-ranging research experience. Similarly, SPC members in turn recruited reviewers (now called the "program committee" on the suggestion of the NAACL exec) based on who would be the most capable. I suggested to some SPC members that they try to make sure certain sub-areas were covered, and that was sufficient. The SPC/PC organization was not hierarchical. In assigning reviewers to particular papers, we found it very useful to be able to draw on all of the reviewers in the program committee rather than a subset dedicated to a particular area. For example, it was easy to assemble reviewers for a paper on morphology in language XYZ that would include experts in both morphology and XYZ. There was no special track for "applications" papers this year, but papers were encouraged in the call and by the inclusion of applications-oriented researchers on the SPC. Six of the thirty one accepted papers described work on a large end-to-end system (for Q/A, speech, translation), while the rest described techniques that could be applied to many such systems. SPC members were able to inspect paper titles from the "notification of submission" web page, and they could optionally indicate interest in certain titles by e-mail to me. I assigned each paper to an individual SPC member. I distributed paper hardcopies to the relevant SPC members, along with a reviewer-assignment matrix. I requested four suggested reviewers per paper. The matrix already blocked out many reviewer/paper conflicts, which I determined manually. The matrix came in several formats; some SPC members chose the paper format, making X's on it and fedexing it back to me; others chose to do things by e-mail. I manually assembled the results, assigning three final reviewers to each paper, taking care not to overload individual reviewers. Reviewers returned reviews to SPC members in charge of individual papers (some returned different reviews to different SPC members). The SPC organized discussions in cases of disagreement, also by e-mail. Reviews for papers were packaged and sent to me. I created paper booklets that included all of the reviews, and these booklets formed the basis of discussions at the face-to-face SPC meeting. Authors were notified (ahead of schedule) as to which papers were accepted and declined. 3 Things that went well The process was very smooth. All the deadlines were made, despite a tight schedule that involved both November and December holidays. There were no problems with security (e.g., reviewer or author identities leaked incorrectly). The bulk of conflicts of interest were taken care of early in the process. The SPC/PC organization worked well. 4 Things that went wrong The deadline for paper submissions was in November, earlier than typical ACL/AAAI/IJCAI deadlines. More papers might have been submitted if this deadline were later; heavier promotion would have been good, also. As it was difficult to predict the number of submissions, we erred on the side of getting too many reviewers. This made the match between papers and reviewers very good, of course, but some reviewers were placed on "stand by." 5 Acknowledgements Lori Levin, Alon Lavie, Bob Frederking, and Ralf Brown provided plenty of assistance from the Pittsburgh end. Eric Nyberg arranged the CD-ROM version of the proceedings. Special thanks to Lauri Grier of USC/ISI for her assistance in successfully navigating the program through a dozen tight deadlines. Lauri Grier, Liz Hall, Fanny Mak, and Kary Lau kept the records straight, helped identify potential reviewer conflicts, got papers to reviewers, and did many other things to give the program committee the maximum possible amount of time to review papers. Thanks to Irene Langkilde Geary for organizing hundreds of e-mail reviews into a coherent booklet for the senior program committee meeting, and for helping with the somewhat stressful final distribution of reviews. Thanks also to Yaser Al-Onaizan, who designed the electronic notification process and coordinated the formatting codes for the authors of the papers. Final thanks to the NAACL executive committee for providing useful suggestions and answering pressing questions.