|
|
| (One intermediate revision by the same user not shown) |
| Line 1: |
Line 1: |
| NAACL HLT Conference Chair Report | | This report can be found at http://www.cs.cmu.edu/~lsl/acl/NAACL-HLT-2009-report.pdf |
|
| |
|
| As General Chair, my goal for the 2009 NAACL HLT Conference was to have high
| | Posted by Lori Levin on behalf of Mari Ostendorf |
| quality technical presentations in all areas, but particularly to increase the number and
| |
| quality of papers from the speech and IR areas. In addition, I was encouraged by the
| |
| NAACL Board to explore ways to better engage researchers from industry.
| |
| | |
| The ACL-HLT 2008 conference chair raised the issue of imbalance in the speech/IR/NLP
| |
| fields. There were many more submissions in NLP than in the other two areas. She
| |
| recommended either changing the structure of committees to reflect the actual balance of
| |
| papers or having an “additional coordinated event that involves invitations to members of
| |
| the community in order to draw more participation.” For NAACL HLT 2009, we
| |
| maintained the area distribution for the area chairs, publicity and tutorials, but not for
| |
| other committees. For committees with fewer chairs, we tried to choose people whose
| |
| expertise spanned multiple disciplines, which seemed to work well. It would probably be
| |
| reasonable to take a similar strategy for the PC chairs next year, i.e. have only 3 PC
| |
| chairs with 2 from NLP and one from speech, with one of the three having a tie to IR.
| |
| (We had one person serve as lead for long papers (Mike Collins) and one as lead for short
| |
| papers (Lucy Vanderwende), and this worked very well.)
| |
| | |
| In order to try to attract more papers in the speech and IR areas, we included two special
| |
| sessions targeting the cross-cutting topics of large scale language processing (IR and
| |
| NLP) and speech indexing and retrieval (speech and IR). The area chairs for the speech
| |
| indexing session actively recruited paper submissions, which contributed to the session’s
| |
| success. The large scale language processing area had several submissions without active
| |
| recruiting other than our highlighting it in the call for papers.
| |
| | |
| In an attempt to better engage industry researchers, we organized a lunchtime panel
| |
| discussion on “Emerging Application Areas in Computational Linguistics,” which
| |
| included representatives from different application areas and different size companies.
| |
| Thanks to Bill Dolan for organizing and moderating the discussion. Box lunches were
| |
| available for purchase. The panel discussion was very well attended. One problem was
| |
| that the panel started late because of the long line for getting lunches, which involved
| |
| participants composing their lunches. Possibilities for improving this include having the
| |
| lunches pre-boxed to simplify pickup logistics and moving the lunch pickup outside of
| |
| the panel discussion room. In addition to the industry panel, another success with
| |
| industry is that the tutorial chairs actively recruited some more practically oriented
| |
| panels, which were very well received.
| |
| | |
| Two web pages were available to provide guidance:
| |
| http://aclweb.org/adminwiki/index.php?title=Conference_Handbook
| |
| http://www.cs.jhu.edu/~jason/advice/how-to-chair-a-conference
| |
| | |
| While I should have done a better job in making all chairs aware of the guidelines, our
| |
| biggest problem (coordinating schedules with other conferences) is not addressed in these
| |
| guidelines. In addition, it would be nice to have guidance on other HLT-specific issues
| |
| (e.g. demos). NAACL might want to develop an additional web page covering special
| |
| considerations for NAACL HLT.
| |
| | |
| Several changes were instituted this year, including:
| |
| * Multi-conference coordination of sponsorship
| |
| * Multi-conference coordination of workshops
| |
| * New format for short paper reviewing and explicit call for different types of short
| |
| papers
| |
| * Including the student research workshop as a parallel session within the main
| |
| conference
| |
| * Allowing students whose papers were accepted to the student session to also
| |
| (optionally) present a poster in the main conference poster session.
| |
| | |
| All changes seemed to work well with a couple of exceptions. Most importantly, there
| |
| was a lack of clarity of responsibility among the different sponsorship chairs and a lack
| |
| of connection of the regional sponsorship chairs to the NAACL HLT 2009 meeting. The
| |
| general idea of multi-conference coordination makes sense, but it is necessary to clarify
| |
| the role of the local chair and to make this person the main point of contact. Second, as
| |
| noted in the PC chairs’ report, there were not many papers in the “negative result” and
| |
| “opinion piece” categories of short papers. The program committee also felt that it would
| |
| be better to have the author indicate the type of short paper, in addition to the reviewers.
| |
| The total funds raised through sponsorship efforts was $27,350, which includes one
| |
| $4000 contribution that resulted from the multi-conference sponsorship coordination. The
| |
| Local Arrangements Chairs were very helpful in this effort in addition to the Local
| |
| Sponsorship Chair. Companies contributing included: Rosetta Stone, CNGL, Google,
| |
| AT&T, Language Weaver, JD Powers, IBM Research, CLEAR, HLT, LDC, and John
| |
| Benjamins. One difficulty that arose was a last minute decision that funds from a sponsor
| |
| would go to a student travel award, which caused some minor program glitches. We
| |
| recommend that an advance deadline be set for designating donated funds for new
| |
| awards.
| |
| | |
| | |
| Based on discussions during and after the conference, additional areas where we think
| |
| there could be improvements include:
| |
| * Parallel review of short papers: A possible reason for serial review was to allow
| |
| papers submitted as long papers and rejected to be resubmitted as short papers.
| |
| Very few short papers are accepted this way, and authors can always submit to
| |
| other conferences. Parallel review would make scheduling much easier and
| |
| would make the reviewing process less complicated for the program committee.
| |
| * Demos: We had very few demos this year, but many people like the tradition.
| |
| Ideas for addressing this include eingither assign the Demo Chairs the task of
| |
| actively recruiting demos, or giving authors of accepted papers the opportunity to
| |
| present as a demo.
| |
| * Poster session: The evening poster session combined with a reception was well
| |
| received by poster viewers, but less so by poster presenters. People who had
| |
| posters in the first session didn’t get much food, and people who had their posters
| |
| in the later session had a smaller audience. Some ideas for improving this include:
| |
| having a better enforced time period for poster presenters to eat, having some
| |
| overlap of the two time slots so the second session runs less late, grouping posters
| |
| that are on related topics in the same area, and including an introductory session
| |
| where poster presenters give a 1-minute pitch on their poster.
| |
| * Workshops: One of the workshop chairs should be affiliated with the local
| |
| (hosting) institution, since there are a lot of local arrangements issues that arise
| |
| with the workshops.
| |
| | |
| Other suggestions are included in the reports from other chairs.
| |
| While there were several areas for improvement, overall, I consider the conference to be
| |
| a success. There were roughly 700 participants, and the quality of the tutorials,
| |
| presentations and workshops was high. The local arrangements were terrific. I am
| |
| indebted to all the chairs involved in the organization and to the NAACL Board for their
| |
| support. While there remains an imbalance between NLP, speech and IR, I am
| |
| encouraged by the quality of the papers that were included. I strongly support continued
| |
| efforts to include these different areas of HLT and make it possible for researchers to
| |
| benefit from the insights of these related field.
| |
| | |
| Mari Ostendorf, University of Washington | |
| General Chair
| |
| | |
| | |
| NAACL 2009 Program Chairs Report
| |
| | |
| In 2009 the NAACL HLT program continued to include high-quality work in the areas of
| |
| computational linguistics, information retrieval, and speech technology. The program included
| |
| full papers, short papers, demonstrations, a student research workshop, pre-conference
| |
| tutorials, and post-conference workshops. The call for papers included solicitation of papers
| |
| for 2 special sessions, “Large-Scale Language Processing” and “Speech Indexing and Retrieval”.
| |
| This year, 260 full papers were submitted, of which 75 papers were accepted (giving a 29%
| |
| acceptance rate); and 178 short papers were submitted, of which 71 were accepted (giving
| |
| a 40% acceptance rate). All full papers were presented as talks; this contrasts with some
| |
| previous years, e.g., ACL-08 HLT, where some full papers were presented as posters. Of the
| |
| short papers, 35 were presented as talks, with the remainder being presented as posters. A
| |
| full breakdown of the statistics by area is presented at the end of this report.
| |
| | |
| This year, short papers of five types were solicited: “a small, focused contribution”, “work in
| |
| progress”, “a negative result”, “an opinion piece”, or “an interesting application note”; it was
| |
| a reviewer task to determine which paper type a short paper best belonged to, alternatively,
| |
| this could be a check-box at submission time. In practice, the largest agreement among
| |
| reviewers was found in the “small, focused contribution” category, the traditional type of short
| |
| paper submitted to NAACL HLT (119/178). A majority of reviewers thought that 38/178
| |
| papers were “work in progress”, and that 10/180 were “interesting application note”. There
| |
| were only a handful of papers submitted that any of the reviewers considered to be a “negative
| |
| result” or “opinion piece”. It will take more than one conference cycle to determine the field’s
| |
| interest in writing, and then accepting, such paper types.
| |
| | |
| Reviewing was organised in a two-tier system, with eighteen senior program committee
| |
| (SPC) members (“area chairs”), who in turn recruited 352 reviewers. The SPC members
| |
| managed the review process for both the full and short paper submissions: each full paper
| |
| received at least three reviews, and each short paper received at least two reviews. Similar to
| |
| recent years, we did not have a face-to-face meeting of the area chairs, instead we held a series
| |
| of tele-conferences between individual area chairs and the PC chairs. The START conference
| |
| management system was used to manage paper submissions and the review process—Rich
| |
| Gerber and the START team gave invaluable help with the system.
| |
| | |
| Two best paper awards were given at the conference. The senior program committee members
| |
| for the conference nominated an initial set of papers that were candidates for the awards; the
| |
| final decisions were then made by a committee chaired by Candace Sidner, and with Hal
| |
| Daume III, Roland Kuhn, Ryan McDonald, and Mark Steedman as its other members.
| |
| | |
| Michael Collins, Massachusetts Institute of Technology
| |
| Shri Narayanan, University of Southern California
| |
| Douglas W. Oard, University of Maryland
| |
| Lucy Vanderwende, Microsoft Research
| |
| | |
| Table 1: Statistics for full paper submissions.
| |
| | |
| | |
| Area Submissions Acceptances (Talk)
| |
| | |
| Sentiment/Information Extraction 34 8 (24%)
| |
| Discourse 10 4 (40%)
| |
| Generation/Summarization 22 2 (9%)
| |
| Machine learning 28 8 (29%)
| |
| Phonology/Morphology/Language acquisition 14 5 (36%)
| |
| Semantics 32 7 (22%)
| |
| Syntax 33 11 (33%)
| |
| Machine translation 37 13 (35%)
| |
| Dialog 9 3 (33%)
| |
| IR/Question answering 16 4 (25%)
| |
| Large-scale language processing 11 3 (27%)
| |
| Speech indexing and retrieval 1 0 (0%)
| |
| Speech/Spoken Language Processing Algorithms 9 4 (44%)
| |
| Speech/Spoken Language Processing Applications 8 3 (38%)
| |
| | |
| | |
| Area Submissions Acceptances Acceptances
| |
| (talk) (poster)
| |
| Sentiment/Information Extraction 22 3 (14%) 4 (18%)
| |
| Discourse 7 - 2 (29%)
| |
| Generation/Summarization 10 2 (20%) 3 (30%)
| |
| Machine Learning 14 2 (14%) 2 (14%)
| |
| Phonology/Morphology/Language Aquisition 4 - 1 (25%)
| |
| Semantics 17 4 (24%) -
| |
| Syntax 16 3 (19%) 4 (25%)
| |
| Machine Translation 30 8 (27%) 5 (17%)
| |
| Dialog 11 2 (18%) 3 (27%)
| |
| IR/Question answering 16 3 (19%) 3 (19%)
| |
| Large Scale Processing 6 1 (17%) 2 (33%)
| |
| Speech Indexing and Retrieval 7 5 (71%) -
| |
| Speech/Spoken Language Algorithms 9 1 (11%) 5 (56%)
| |
| Speech/Spoken Language Applications 9 1 (11%) 2 (22%)
| |
| Table 2: Statistics for short paper submissions.
| |
| 2
| |
| Report from Student Research Workshop/ Doctoral Consortium
| |
| The Student Research Workshop provided a venue for student researchers investigating topics
| |
| in the broad fields of Computational Linguistics and Language Technologies to present their
| |
| work and receive feedback from the community. The workshop was composed of three parallel
| |
| tracks in Natural Language Processing, Information Retrieval, and Speech. We received a total
| |
| of 29 submissions (4 IR, 4 Speech, 21 NLP) from 11 countries. Submissions were up from last
| |
| year, although that was not until after we extended the deadline twice, so we wonder if there
| |
| may be too many similar venues competing for submissions, such as the ACL SRW and the
| |
| EACL SRW. Considering the uneven distribution of submissions to the three tracks, the topical
| |
| organization of the workshop into tracks should also be re-thought.
| |
| Of the 29 submission, we accepted 9 as oral presentations (one of which withdrew from the
| |
| workshop) and another 9 as poster presentations. Accepted oral presentations and posters
| |
| came form 9 different countries. Both oral presentation and poster presentation sessions were
| |
| scheduled during the main conference; each paper accepted for oral presentation was also
| |
| given a slot in the poster session. We made a special effort to schedule the sessions at times
| |
| when many senior people in the field would be able to attend and offer their valuable wisdom
| |
| from years. A total of 86 students and senior researchers agreed to serve on the program
| |
| committee, which allowed us to assign 4 to 6 reviewers per paper.
| |
| During the workshop, each oral presentation was followed by a brief panel discussion by two
| |
| panelists per paper. Despite the extra effort of having to recruit panelists, we believe that the
| |
| panels added considerable extra value to the workshop. Not only did it ensure good feedback to
| |
| the presenters, it also helped the audience put the papers into perspective within the respective
| |
| research fields. Each of the three oral presentation sessions drew an audience of 30 to 50
| |
| people.
| |
| All presenters received financial support from the U.S. National Science Foundation to assist
| |
| them in their travel to Boulder for the conference. Altogether we received $21,000 from the
| |
| National Science Foundation to fund the workshop, which included support for student
| |
| participants, student co-chairs, and the cost of the student lunch. Oral presenters were offered
| |
| $400 to defray the cost of registration and hotel as well as $500 to cover travel from within North
| |
| America, or $1000 if they were traveling internationally. Poster presenters were offered $300.00
| |
| total for reimbursement. We also budgeted a small amount of money for materials, such as
| |
| poster boards for the poster presentations.
| |
| At the student lunch on the day of the SRW, we had a group discussion to get feedback from
| |
| the student community on how the SRW could be improved, and in general what could be done
| |
| to offer mentoring to the students in our community. One issue that was raised is that it is not
| |
| very clear what sets the SRW apart from the main conference or exactly what types of
| |
| submissions are desired. Students felt that some of the feedback they received from reviewers
| |
| wasn’t consistent with what the call for participation described as the target submissions. In
| |
| response to this, we may need a more structured review form. One participant in the discussion
| |
| pointed out that the form used for ACL short papers this year was a particularly good example of
| |
| how to keep reviewers thinking along the right lines for review. Another issue that was raised is
| |
| that students are not getting encouragement from their advisors to submit to the SRW. So we
| |
| may need to go back to the faculty segment of the ACL community to find out why and what we
| |
| can do about it.
| |
| Students expressed a desire for more networking opportunities at ACL conferences, especially
| |
| to help the shyer students come out of their shells. One idea was to organize topic specific
| |
| round tables where selected faculty would attend, but which would be mainly students
| |
| interested in similar topics. Other ideas included websites to help students find roommates for
| |
| conferences and distributing contact information for all people who are registered by a particular
| |
| date, along with their affiliations and research interests, to help students plan for who they want
| |
| to try to set up meetings with, etc. during the conference.
| |
| Student Research Workshop Faculty Chairs and Student Co-Chairs
| |
| Anoop Sarkar (Faculty Chair, Simon Fraser University)
| |
| Carolyn Rose (Faculty Chair, CMU)
| |
| Svetlana Stenchikova (Student Co-Chair, Stony Brook University) - Speech
| |
| Ulrich Germann (Student Co-Chair, University of Toronto) - NLP
| |
| Chirag Shah (Student Co-Chair, University of North Carolina) - Information Retrieval
| |
| Brief Reports from Other NAACL HLT Chairs
| |
| Publicitiy Chairs
| |
| Matthew Stone (Rutgers)
| |
| Gokhan Tur (SRI)
| |
| Diana Inkpen (U Ottawa)
| |
| The Publicity Chairs were chosen from each of the three HLT areas -- IR, speech and
| |
| NLP – in order to ensure good connections to the communities. The forwarded all
| |
| announcements to mailing lists in their respective fields, including the main conference
| |
| Call for Papers and Call for Short Papers, the Calls for Workshop and Tutorial Proposals,
| |
| the Call for Demos, and the Doctoral Consortium Call for Papers. The lists and websites
| |
| used included: corpora, elsenet, acl, asis-l, linguist, webir, ISCA ISCAPad, IEEE
| |
| eNewsletter, AI Magazine, and the cognitive science society website.
| |
| Future organizers should bear in mind that many organizations produce bimonthly or
| |
| quarterly newsletters for conference announcements (e.g., AAAI's AI magazine, IEEE
| |
| signal processing speech & language technical committee), which require CFPs to be
| |
| distributed at least 2-3 months in advance of submission deadlines.
| |
| Publications Chairs
| |
| Christy Doran (MITRE)
| |
| Eric Ringger (BYU)
| |
| At the recommendation of the ACL-HLT 2008 chair, we continued the tradition from
| |
| ACL 2008 of having two Publication Chairs, which seemed to work well. The Chairs
| |
| followed the recipe written for publications chairs by Joakim Nivre and Noah Smith,
| |
| located here:
| |
| http://stp.lingfil.uu.se/~nivre/how-to-pub.html
| |
| This includes the updated recipe for using ACLPUB to assemble the actual proceedings:
| |
| http://faculty.cs.byu.edu/~ringger/naacl09/howto.html
| |
| Several improvements are in the queue for both documents as well as the ACLPUB tools.
| |
| Notes for improvement and discussion:
| |
| o Better publicizing and enforcement of publications-related deadlines.
| |
| o Mailing lists for relevant subsets of organizing committee, including: PC cochairs,
| |
| local organizers, workshop chairs and sponsorship chairs (There are no
| |
| sponsor logos in the proceedings, since the sponsorship chairs did not know about
| |
| the deadline.)
| |
| o Improved documentation of pre-requisites for hand-off to OmniPress (e.g. file
| |
| formats), especially regarding book covers.
| |
| o Per Jan Hajic, there is an opportunity to integrate some of ACLPUB into START
| |
| o Numbers of needed printed volumes continue to drop. It may be a good time to
| |
| consider going digital only.
| |
| o Recommendations for shared documentation on aclweb.org?
| |
| Tutorials Chairs
| |
| Ciprian Chelba (Google)
| |
| Paul Kantor (Rutgers)
| |
| Brian Roark (OHSU)
| |
| The tutorial chairs actively recruited submissions, received 12, and accepted 8. They
| |
| erred on the side of accepting rather than rejecting because of “ties” in the reviews, and
| |
| felt that it worked out quite well. Even though 8 tutorials is more than in most years
| |
| (typically 6), we ended up with sufficient enrollment in all 8 accepted proposals. The
| |
| complete list of tutorials is given below.
| |
| 1. Data-Intensive Text Processing with MapReduce -- Jimmy Lin and Chris Dyer
| |
| 45 participants
| |
| 2. Distributed Language Models -- Thorsten Brants and Peng Xu
| |
| 32 participants
| |
| 3. Search Algorithms in NLP: Theory and Practice with Dynamic Programming --
| |
| Liang Huang
| |
| 53 participants
| |
| 4. Extracting world/linguistic knowledge from Wikipedia -- Simone Paolo Ponzetto
| |
| and Michael Strube
| |
| 34 participants
| |
| 5. OpenFst: An Open-Source, Weighted FST Library -- Martin Jansche/Cyril
| |
| Allauzen/Michael Riley
| |
| 24 participants
| |
| 6. OntoNotes: The 90% Solution -- Sameer Pradhan and Nianwen Xue
| |
| 12 participants
| |
| 7. VerbNet overview, extensions, mappings and apps -- Karin Kipper Schuler, Anna
| |
| Korhonen, Susan W. Brown
| |
| 24 participants
| |
| 8. Writing Systems, Transliteration and Decipherment -- Richard Sproat and Kevin
| |
| Knight
| |
| 20 participants
| |
| Workshops Chairs
| |
| Mark Hasegawa-Johnson (UIUC),
| |
| Nizar Habash (Columbia)
| |
| There were 41 workshop submissions jointly to ACL, EACL, and NAACL. ACL
| |
| accepted 12, EACL accepted 10, and we accepted 11. Eleven plus the student workshop
| |
| and CoNLL gave a total of 13 workshops, listed below. Number of participants listed is
| |
| the final estimate from Priscilla Rasmussen as of May 26. For more information, see
| |
| http://isle.uiuc.edu/hltnaacl2009/.
| |
| 1. Semantic Evaluations: Recent Achievements and Future Directions
| |
| Organizers: Eneko Agirre, LluÌs Marquez, Richard Wicentowski
| |
| 42 participants
| |
| 2. BioNLP 2009
| |
| Organizers: Sophia Ananiadou, K. Bretonnel Cohen, Dina Demner-Fushman,
| |
| John Pestian, Jun'ichi Tsujii, Bonnie Webber
| |
| 74 participants
| |
| 3. Third International Workshop on Cross Lingual Information Access: Addressing
| |
| the Information Need of Multilingual Societies
| |
| Organizers: Sivaji Bandyopadhyay, Pushpak Bhattacharya, Vasudeva Varma,
| |
| Sudeshna Sarkar, A Kumaran
| |
| 14 participants
| |
| 4. Workshop on Integer Linear Programming for Natural Language Processing
| |
| Organizers: James Clarke, Sebastian Riedel
| |
| 25 participants
| |
| 5. Software engineering, testing, and quality assurance for natural language
| |
| processing
| |
| Organizers: Kevin Bretonnel Cohen, Marc Light
| |
| 34 participants
| |
| 6. Computational Approaches to Linguistic Creativity
| |
| Organizers: Birte Loenneker-Rodman, Anna Feldman
| |
| 34 participants
| |
| 7. Unsupervised and minimally supervised learning of lexical semantics
| |
| Organizers: Suresh Manandhar, Ioannis Klapaftis
| |
| 34 participants
| |
| 8. Semi-supervised Learning for NLP
| |
| Organizers: Qin Wang, Kevin Duh, Dekang Lin
| |
| 75 participants
| |
| 9. Active Learning for NLP
| |
| Organizers: Eric Ringger, Robbie Haertel, Katrin Tomanek
| |
| 35 participants
| |
| 10. Innovative Use of NLP for Building Educational Applications
| |
| Organizers: Joel Tetreault, Jill Burstein, Claudia Leacock
| |
| 34 participants
| |
| 11. Third Workshop on Syntax and Structure in Statistical Translation
| |
| Organizers: Dekai Wu, David Chiang
| |
| 48 participants
| |
| 12. Thirteenth Conference on Computational Natural Language Learning (CoNLL)
| |
| Organizers: Suzanne Stevenson and Xavier Carreras
| |
| 85 participants
| |
| The multi-conference proposal system seemed to work very well for all concerned: three
| |
| workshops that would not have been offered (because they were rejected by their firstchoice
| |
| conference) were, instead, offered at NAACL. All of the workshops had full
| |
| schedules, as indicated in the online schedule.
| |
| Demo Chairs
| |
| Fred Popowich (Simon Fraser University)
| |
| Michael Johnston (AT&T)
| |
| Six demos were submitted, and five were accepted. The demo chairs did not actively
| |
| recruit demos from specific research groups, and relied on the general publicity efforts.
| |
| Future organizers might consider opening the demos up to allow people who have
| |
| accepted papers to give demos in the demo session. From attending the demo session, the
| |
| presenters seemed to be happy with how it went (and they were glad that they were in the
| |
| high traffic area).
| |