Admin Wiki - User contributions [en]

2016Q3 Reports: TACL Journal Editor

2016-07-23T00:08:06Z

KristinaToutanova:

(We are very grateful to have received an extension on filing our report until July 22 from ACL Secretary Shiqi Zhao. --the TACL editors in chief)

== Personnel updates ==

The ACL Exec and Lillian Lee have jointly agreed to give her an extra half-term lasting up to but not including September 2018, when she goes on sabbatical. Editorial Assistant Cindy Robinson does not currently plan to stay on past that time.

We have reached the Grand Term Expiration Date of July 2016, where out of our 34 action editors, the terms of 16 have ended. We thank our departing AEs Regina Barzilay, Chris Callison-Burch, and Joakim Nivre for all their help over these pivotal years for TACL, and thank our renewing AEs David Chiang, Alexander Clark, Hal Daume III, Katrin Erk, Philipp Koehn, Hwee Tou Ng, Patrick Pantel, and Chris Potts for being willing and able to stay on board. (Other AEs with expiring terms are still pondering the question.)

218 of our elite reviewing team also had expiring terms. Renewal invitations have been issued: of these, there have been 10 declines, 19 requests pending, and --- a fact we are deeply, truly grateful for --- the rest have agreed to rejoin.

We anticipate issuing invitations to new people as well soon.

''We sometimes joke that the "A" in TACL stands for "artisanal". The reviewers and action editors are, on the whole, putting a huge amount of time and craft into their "pieces", broadly construed. (For one bit of proof, see the statistics below.) Whether this "personal touch" can (or should) be scaled to a conference-size production line is an important question.''

== Interface with conferences ==

Our checks for undeclared multiple/overlapping submissions with main conferences are surprisingly time-consuming but unfortunately do turn up problematic cases. As one consequence, we have clarified our policy language to state that ''no'' material in a TACL submission can be under consideration elsewhere while under review by TACL (see website for exact policy).

Author-response periods for conferences --- so that authors see reviews before they receive an official decision --- raise the question from authors, "After getting negative reviews from a conference, if we withdraw the paper from the conference (and revise it), can we submit it to TACL?" We consider a paper that has received reviews and is not accepted by a conference to be "rejected" from TACL's point of view. Hence, submission of a heavily revised version to TACL is allowed ''if'' the TACL submission follows the rules for papers rejected from *ACL/EMNLP; but undeclared (revisions of) such submissions will be rejected from TACL.

Counts of TACL papers (to be) presented at this year's conferences: 10 at NAACL 2016, 25 at ACL 2016, 10 or so at EMNLP 2016. Conference program chairs have been making different decisions with respect to the poster-vs-oral presentation question for TACL papers. NAACL and ACL essentially gave TACL authors their choice (a very few do choose poster), a decision perhaps aided by the conferences' decisions to have shorter talk slots and hence more talk slots available.

== Interface with preprint servers/workshop proceedings ==

In the copy-editing phase, we are requesting that authors replace references to preprints such as arXiv papers and other non-refereed papers with references to peer-reviewed versions, if available.

We have agreed that shared tasks familiar to the ACL community whose papers are put in a separate proceedings instead of the main meeting proceedings will be treated as workshop papers, and thus not subject to the multiple-submission restrictions.

== Transition to new (old) domain name ==

We are transitioning from using URLs referencing tacl2013.cs.columbia.edu to transacl.org ([https://transacl.org]) . Old links (for both http and https) should still work, but we ask that people try to use the transacl.org format and update links they control, if possible. The plan is to get DOIs that are linked to transacl.org URLs.

== Some statistics ==

=== number of submissions and timing of first decision ===
Depicted below is the history of the number of distinct submission IDs that have received a decision, and the average time to first decision, grouped by round (=nearest first-of-the-month).

[[File:First-decision-to-post.png]]

The drop in the last two months is due at least in part to 13 papers submitted in that period that have not yet received a decision. Also, not counted: papers handled using START (the early days of TACL), papers, and the 106 papers archived for some technical or formatting problem. The decision time for papers submitted late in a month is counted as starting from the first of the next month.

''Observations'':

While we don't count technical/format problems, we "do" count papers that receive an editorial decision of rejection without going out for full review. One way we EiCs are trying to keep reviewing loads down and submissions interesting for our reviewers is sometimes doing a preliminary consultation with an action editor or highly relevant reviewer for submissions that seem highly unlikely to be accepted. Such consultations and subsequent discussion can take two weeks or so, since we strive to be cautious about making such decisions, but also want to factor in the value of conserving reviewer effort.

Waves in the number of submissions seem correlated with upcoming conference deadlines.

Average decision times per month appear to correspond well with an ideal but attainable schedule of:

: ideally 7 days to make a load-balanced assignment of submissions to action editors (AEs)
: ideally 7 days for AEs to get three reviewers confirmed to review
: ideally 21 days for reviewers to review
: ideally 10 days for AE to coordinate discussion among reviewers and come to a decision
which would yield an ideal target of 45 days turnaround.

For reviewers that complete their reviewers, the average completion time is in fact 23 days!

=== Decision statistics ===

First decision breakdown for our 589 submissions.
9% (a) = accepted as is
22% (b) = conditional accept: acceptance guaranteed if conditions met
31% (c) = rejected, encourage resubmission but no guarantee of acceptance
38% (d) = rejected with 1-year moratorium on TACL submission.
= 32% rejected based on full review, 6% rejected without full review (but received editorial consideration)

72 (=12%) some of these papers are revised versions of (c) papers: TACL treats these as new submissions. Of these 72 papers, the first-decision breakdown is:
44% (a)
11% (b)
14% (c)
30% (d)

=== Publishing statistics ===

26 papers have been published so far in 2016.

Another 11 are in the publication queue.

2016Q1 Reports: TACL Journal

2016-02-20T02:22:47Z

KristinaToutanova: Created page with "== Personnel == * Mark Johnson and Kristina Toutanova have joined Lillian Lee as co-editors-in-chief! * Sincere gratitude to our retiring action editor Janyce Wiebe. We are..."

== Personnel ==

* Mark Johnson and Kristina Toutanova have joined Lillian Lee as co-editors-in-chief!
* Sincere gratitude to our retiring action editor Janyce Wiebe. We are very glad to have the new action editors Phil Blunsom (University of Oxford and Google DeepMind), Chris Quirk (Microsoft Research), Hinrich Schütze (University of Munich), Holger Schwenk (Facebook AI Research), Scott Wen-tau Yih (Microsoft Research), and Luke Zettlemoyer (University of Washington) join, and to have Bo Pang (Google Research) come back as a TACL action editor after a pause.
* We have expanded the roster of reviewers thanks mainly to Cindy's great efforts.

== Policy updates==
* We have a precedent for consequences in case of multiple submission to a conference and TACL, but have not formulated a general policy.
* Mark Johnson is developing a publication ethics statement. This is an important step in the process of getting the journal indexed by several major bodies.

== Biggest pending issues==

* ''System Hosting''.

: We are considering move to a different machine hosted somewhere else to improve the reliability of the system.
: Professional short-term help by a developer would be very useful if available.

* ''Conference interface''.

: To integrate the handling of TACL papers into the START-based workflow of the conferences, EMNLP 2015 and NAACL 2016 did or will create a special track in START. TACL provides the conference with author contact info, and the authors upload their papers and metadata into the START track. We understand from EMNLP 2015 that this worked well.

: Checking for double submissions to TACL and conferences or undeclared re-submission of conference rejected papers: The TACL co-editors and conference PC chairs exchange information on submissions to be able to do this check. We have caught multiple cases of violations of the multiple submission policy.

== Selected minor procedure/system updates:==
* We have updated the style files for submissions and now require the use of these files.
* We have updated the reviewer information in the system to improve search for suitable reviewers (thanks to Cindy).

== Some statistics==
* Coming very soon!

2014Q3 Reports: Program Chairs

2014-06-05T20:42:46Z

KristinaToutanova: /* Recommendations */

===Program Chairs (Kristina Toutanova and Hua Wu)===

== Innovations ==

As compared to ACL conferences in prior years, the main innovations this year were:
# We allocated reviewers to areas based on the reviewers’ preferences over areas, the preferences of area chairs over reviewers, and the number of submissions in each area (using a tool developed by Mark Dredze and applied to ACL with the help of Jiang Guo; this tool had been applied to NAACL and EMNLP in the past but was applied at ACL for the first time this year).
# We optimized the conference schedule based on feedback from attendees on the talks they would like to see.
# All talks will be recorded at ACL 2014, as in NAACL 2013.
# We scheduled two large poster sessions on two evenings of the conference, to accommodate the large number of poster presentations. Instead of a banquet, we will have is a no-fee light social event, and the president’s talk will be given on the first morning of the conference.
# We grouped oral TACL papers in thematic sessions together with ACL paper presentations, instead of in separate TACL-only sessions.
# We solicited nominations for outstanding reviewers and acknowledged them in the proceedings (around 14% of reviewers were acknowledged).

== Submissions and Presentations ==

ACL 2014 received a total of 1123 submissions, of which 572 were long papers and 551 were short papers. 15 long papers and 19 short papers were rejected without review due to non-anonymity or formatting issues.The remaining submissions were assigned to one of 20 areas, and managed by a program committee of 33 area chairs and 779 reviewers.

146 (26.2%) of the 557 qualifying long papers and 139 (26.1%) of the 532 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 95 were selected for oral presentation, and 51 for poster presentation. Of the accepted short papers, 51 have oral and 88 have poster presentation slots. The oral versus poster decision was made based not on the quality of the work, but the estimated appeal to a wide audience.

In addition, 19 TACL papers will be presented at ACL – 13 as talks and 6 as posters. Including TACL papers, there will be 159 oral and 145 poster presentations at the main ACL conference.
The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area. The table also shows the number of qualifying long and short papers that were withdrawn prior to the completion of the review process (11 long and 34 short papers were withdrawn).

{| class="wikitable" cellpadding="2" border="1"
| Areas||Long Received||Long Accepted||Short Received||Short Accepted||Total Submissions ||Percent of Total Submission||Total Accepts||Percent of Total Accepts||Area Accept Rate
|-
| Cognitive Modeling and Psycholinguistics ||9 ||3 ||14 ||4 ||23 ||2.11% ||7 ||2.46% ||30.43%
|-
| Dialogue and Interactive Systems ||10 ||2 ||8 ||2 ||18 ||1.65% ||4 ||1.40% ||22.22%
|-
| Discourse, Coreference, and Pragmatics ||22 ||5 ||20 ||5 ||42 ||3.86% ||10 ||3.51% ||23.81%
|-
| Document Categorization, Sentiment Analysis, and Topic Models ||53 ||14 ||48 ||13 ||101 ||9.27% ||27 ||9.47% ||26.73%
|-
| Generation ||13 ||6 ||7 ||4 ||20 ||1.84% ||10 ||3.51% ||50.00%
|-
| Information Extraction and Text Mining ||54 ||13 ||49 ||14 ||103 ||9.46% ||27 ||9.47% ||26.21%
|-
| Information Retrieval ||8 ||2 ||9 ||2 ||17 ||1.56% ||4 ||1.40% ||23.53%
|-
| Language Resources and Evaluation ||31 ||8 ||28 ||10 ||59 ||5.42% ||18 ||6.32% ||30.51%
|-
| Lexical Semantics and Ontology ||26 ||7 ||23 ||6 ||49 ||4.50% ||13 ||4.56% ||26.53%
|-
| Machine Learning for Language Processing ||39 ||13 ||15 ||5 ||54 ||4.96% ||18 ||6.32% ||33.33%
|-
| Machine Translation ||76 ||18 ||72 ||19 ||148 ||13.59% ||37 ||12.98% ||25.00%
|-
| Multilinguality and Multimodal NLP ||12 ||3 ||14 ||3 ||26 ||2.39% ||6 ||2.11% ||23.08%
|-
| NLP Applications and NLP-enabled Technology ||32 ||6 ||34 ||9 ||66 ||6.06% ||15 ||5.26% ||22.73%
|-
| NLP for the Web and Social Media ||29 ||5 ||29 ||7 ||58 ||5.33% ||12 ||4.21% ||20.69%
|-
| Question Answering ||6 ||2 ||4 ||0 ||10 ||0.92% ||2 ||0.70% ||20.00%
|-
| Semantics ||53 ||16 ||37 ||11 ||90 ||8.26% ||27 ||9.47% ||30.00%
|-
| Summarization ||19 ||6 ||11 ||4 ||30 ||2.75% ||10 ||3.51% ||33.33%
|-
| Spoken Language Processing ||9 ||2 ||10 ||3 ||19 ||1.74% ||5 ||1.75% ||26.32%
|-
| Tagging, Chunking, Syntax and Parsing ||35 ||12 ||48 ||14 ||83 ||7.62% ||26 ||9.12% ||31.33%
|-
| Phonology, Morphology and Word Segmentation ||10 ||4 ||18 ||4 ||28 ||2.57% ||8 ||2.81% ||28.57%
|-
| Withdrawn ||11 ||0 ||34 ||0 ||45 ||4.13% ||0 ||0% ||0%
|-
| Total ||557 ||146 ||532 ||139 ||1089 ||100% ||285 ||100% ||26.17%
|}

The top five areas for the highest number of submissions this year were Machine Translation, Information Extraction, Document Categorization/ Sentiment analysis/Topic Models, Semantics, and Tagging/Chunking/Syntax/Parsing.

===Review process===
As mentioned above, we used Mark Dredze’s tool for reviewer assignment, which Jiang Guo helped apply to ACL this year. Thanks to the tool and the collaboration among the area chairs, the committee organized a smooth review process with sufficient number of expert reviewers.
The submissions were reviewed under different categories and using different review forms for empirical/data-driven, theoretical, applications/tools, resources/evaluation, and survey papers. For the short papers we additionally used a negative results category and were glad to see that the community is becoming more open to enabling the publication of useful negative results (two out of six short papers submitted under the negative results category were accepted; additional papers reporting negative results but submitted under other categories were also accepted).

We modified the review forms slightly. One modification was to change the final recommendation range from the 1-to-6 range used last year to a 1-to-5 range. We believe this reduces some unnecessary variance in the scores. Another modification was to reword the questions asking reviewers to check whether any submitted code was well structured, because we thought performing a thorough code review would be an unrealistic demand from the reviewers.

We increased the page limit to 9 pages for submitted long papers and 5 pages for submitted short papers and kept the same limits for the camera-ready versions. The change will likely not be implemented next year.

Formatting and anonymity were checked with the help of the area chairs and the student volunteer Jiang Guo. More than 10% of the submissions had author names in the properties of their files – we chose to manually remove this identifying information, but it would be great if that could be automated. Jiang Guo and Jason Eisner helped us apply automated ways to do this, but they did not work in all cases. Papers that had author names listed under the title were rejected without review unless the authors sent us an email realizing their area in the 48 hours following the deadline; in the latter case we removed the author list from the paper for them.

We did not reject papers for non-anonymity of their supplementary data, software, or notes, since the call for papers did not say anything about whether such materials should be anonymized.

===Best paper awards===
The area chairs nominated sixteen papers for the long best paper award. We selected a list of seven candidates which were ranked by a specially formed best paper committee, consisting of seven members --- six area chairs and one external member. Based on the ranking, we selected a long best paper and a long best student paper winner. The selection of a best short paper is in process. The best long papers will be presented in a plenary session, whereas the best short paper will be presented in one of the parallel sessions.

===Presentations===
The oral presentations are arranged in five parallel sessions. There are two large poster sessions including dinner on two evenings of the conference.

We optimized the conference schedule based on feedback from attendees on the talks they would like to see. We collected attendee responses using a scheduling survey developed with the help of David Yarowsky and Svitlana Volkova, and we optimized the conference schedule to assign popular sessions to large conference rooms, and to reduce the chance that two talks that an attendee is interested in are scheduled at the same time. The number of responses collected by the time the schedule was due was 307. The top five topics with highest interest per paper were Semantics, Lexical Semantics, Information Extraction, Question Answering, and Discourse, Dialogue, Coreference and Pragmatics.
We started with a manual grouping of papers in groups for sessions. We then optimized the assignments of groups to rooms/times, and the order of papers within a group automatically. Compared to the original manually created schedule, the optimized schedule had 644 fewer instances where an attendee was interested in seeing two papers scheduled at the same time. More thorough and exact optimization methods could be applied in the future.

==Recommendations==
Based on our workthis year, we would like to make the following recommendations for future years:
# Recommendations for the tool for allocating reviewers to areas: the tool was very useful to ensure a balanced and sufficient number of reviewers were assigned to each area; we recommend using it in future years. (i) Integrate the tool in START: it would make things easier and less confusing for area chairs and reviewers. (ii) Enable area chairs to bid on reviewers after the submissions are in since there are some very narrow areas for which there are only a handful of expert reviewers.
# Add tools for checking non-anonymity and formatting issues to START. We rejected 15 long and 19 short papers for such issues which could have been prevented if the system could automatically alert the authors of such issues. If there are large technical problems with implementing this, there could be a student volunteer on the organizing committee who could help alert authors that submit at least a few hours before the deadline.
# Enable merging of the schedules for long, short, and TACL papers in START, as well as uploading metadata and papers for TACL papers, to simplify the process of deriving the conference program for the website, the conference handbook, and the downloadable proceedings.
# Enable the persistence of reviewers and their assignment completion rate in START.
# Issue a policy on whether self-published papers (published on ArXiv or the authors’ web pages) should constitute previously published work. This year we published a policy applicable to ACL-14 only, but this changes from year to year and CL conference to conference.
# Issue a policy on whether supplementary materials should be anonymized.

2014Q3 Reports: Program Chairs

2014-06-05T20:35:04Z

KristinaToutanova: /* Presentations */

===Program Chairs (Kristina Toutanova and Hua Wu)===

== Innovations ==

As compared to ACL conferences in prior years, the main innovations this year were:
# We allocated reviewers to areas based on the reviewers’ preferences over areas, the preferences of area chairs over reviewers, and the number of submissions in each area (using a tool developed by Mark Dredze and applied to ACL with the help of Jiang Guo; this tool had been applied to NAACL and EMNLP in the past but was applied at ACL for the first time this year).
# We optimized the conference schedule based on feedback from attendees on the talks they would like to see.
# All talks will be recorded at ACL 2014, as in NAACL 2013.
# We scheduled two large poster sessions on two evenings of the conference, to accommodate the large number of poster presentations. Instead of a banquet, we will have is a no-fee light social event, and the president’s talk will be given on the first morning of the conference.
# We grouped oral TACL papers in thematic sessions together with ACL paper presentations, instead of in separate TACL-only sessions.
# We solicited nominations for outstanding reviewers and acknowledged them in the proceedings (around 14% of reviewers were acknowledged).

== Submissions and Presentations ==

ACL 2014 received a total of 1123 submissions, of which 572 were long papers and 551 were short papers. 15 long papers and 19 short papers were rejected without review due to non-anonymity or formatting issues.The remaining submissions were assigned to one of 20 areas, and managed by a program committee of 33 area chairs and 779 reviewers.

146 (26.2%) of the 557 qualifying long papers and 139 (26.1%) of the 532 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 95 were selected for oral presentation, and 51 for poster presentation. Of the accepted short papers, 51 have oral and 88 have poster presentation slots. The oral versus poster decision was made based not on the quality of the work, but the estimated appeal to a wide audience.

In addition, 19 TACL papers will be presented at ACL – 13 as talks and 6 as posters. Including TACL papers, there will be 159 oral and 145 poster presentations at the main ACL conference.
The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area. The table also shows the number of qualifying long and short papers that were withdrawn prior to the completion of the review process (11 long and 34 short papers were withdrawn).

{| class="wikitable" cellpadding="2" border="1"
| Areas||Long Received||Long Accepted||Short Received||Short Accepted||Total Submissions ||Percent of Total Submission||Total Accepts||Percent of Total Accepts||Area Accept Rate
|-
| Cognitive Modeling and Psycholinguistics ||9 ||3 ||14 ||4 ||23 ||2.11% ||7 ||2.46% ||30.43%
|-
| Dialogue and Interactive Systems ||10 ||2 ||8 ||2 ||18 ||1.65% ||4 ||1.40% ||22.22%
|-
| Discourse, Coreference, and Pragmatics ||22 ||5 ||20 ||5 ||42 ||3.86% ||10 ||3.51% ||23.81%
|-
| Document Categorization, Sentiment Analysis, and Topic Models ||53 ||14 ||48 ||13 ||101 ||9.27% ||27 ||9.47% ||26.73%
|-
| Generation ||13 ||6 ||7 ||4 ||20 ||1.84% ||10 ||3.51% ||50.00%
|-
| Information Extraction and Text Mining ||54 ||13 ||49 ||14 ||103 ||9.46% ||27 ||9.47% ||26.21%
|-
| Information Retrieval ||8 ||2 ||9 ||2 ||17 ||1.56% ||4 ||1.40% ||23.53%
|-
| Language Resources and Evaluation ||31 ||8 ||28 ||10 ||59 ||5.42% ||18 ||6.32% ||30.51%
|-
| Lexical Semantics and Ontology ||26 ||7 ||23 ||6 ||49 ||4.50% ||13 ||4.56% ||26.53%
|-
| Machine Learning for Language Processing ||39 ||13 ||15 ||5 ||54 ||4.96% ||18 ||6.32% ||33.33%
|-
| Machine Translation ||76 ||18 ||72 ||19 ||148 ||13.59% ||37 ||12.98% ||25.00%
|-
| Multilinguality and Multimodal NLP ||12 ||3 ||14 ||3 ||26 ||2.39% ||6 ||2.11% ||23.08%
|-
| NLP Applications and NLP-enabled Technology ||32 ||6 ||34 ||9 ||66 ||6.06% ||15 ||5.26% ||22.73%
|-
| NLP for the Web and Social Media ||29 ||5 ||29 ||7 ||58 ||5.33% ||12 ||4.21% ||20.69%
|-
| Question Answering ||6 ||2 ||4 ||0 ||10 ||0.92% ||2 ||0.70% ||20.00%
|-
| Semantics ||53 ||16 ||37 ||11 ||90 ||8.26% ||27 ||9.47% ||30.00%
|-
| Summarization ||19 ||6 ||11 ||4 ||30 ||2.75% ||10 ||3.51% ||33.33%
|-
| Spoken Language Processing ||9 ||2 ||10 ||3 ||19 ||1.74% ||5 ||1.75% ||26.32%
|-
| Tagging, Chunking, Syntax and Parsing ||35 ||12 ||48 ||14 ||83 ||7.62% ||26 ||9.12% ||31.33%
|-
| Phonology, Morphology and Word Segmentation ||10 ||4 ||18 ||4 ||28 ||2.57% ||8 ||2.81% ||28.57%
|-
| Withdrawn ||11 ||0 ||34 ||0 ||45 ||4.13% ||0 ||0% ||0%
|-
| Total ||557 ||146 ||532 ||139 ||1089 ||100% ||285 ||100% ||26.17%
|}

The top five areas for the highest number of submissions this year were Machine Translation, Information Extraction, Document Categorization/ Sentiment analysis/Topic Models, Semantics, and Tagging/Chunking/Syntax/Parsing.

===Review process===
As mentioned above, we used Mark Dredze’s tool for reviewer assignment, which Jiang Guo helped apply to ACL this year. Thanks to the tool and the collaboration among the area chairs, the committee organized a smooth review process with sufficient number of expert reviewers.
The submissions were reviewed under different categories and using different review forms for empirical/data-driven, theoretical, applications/tools, resources/evaluation, and survey papers. For the short papers we additionally used a negative results category and were glad to see that the community is becoming more open to enabling the publication of useful negative results (two out of six short papers submitted under the negative results category were accepted; additional papers reporting negative results but submitted under other categories were also accepted).

We modified the review forms slightly. One modification was to change the final recommendation range from the 1-to-6 range used last year to a 1-to-5 range. We believe this reduces some unnecessary variance in the scores. Another modification was to reword the questions asking reviewers to check whether any submitted code was well structured, because we thought performing a thorough code review would be an unrealistic demand from the reviewers.

We increased the page limit to 9 pages for submitted long papers and 5 pages for submitted short papers and kept the same limits for the camera-ready versions. The change will likely not be implemented next year.

Formatting and anonymity were checked with the help of the area chairs and the student volunteer Jiang Guo. More than 10% of the submissions had author names in the properties of their files – we chose to manually remove this identifying information, but it would be great if that could be automated. Jiang Guo and Jason Eisner helped us apply automated ways to do this, but they did not work in all cases. Papers that had author names listed under the title were rejected without review unless the authors sent us an email realizing their area in the 48 hours following the deadline; in the latter case we removed the author list from the paper for them.

We did not reject papers for non-anonymity of their supplementary data, software, or notes, since the call for papers did not say anything about whether such materials should be anonymized.

===Best paper awards===
The area chairs nominated sixteen papers for the long best paper award. We selected a list of seven candidates which were ranked by a specially formed best paper committee, consisting of seven members --- six area chairs and one external member. Based on the ranking, we selected a long best paper and a long best student paper winner. The selection of a best short paper is in process. The best long papers will be presented in a plenary session, whereas the best short paper will be presented in one of the parallel sessions.

===Presentations===
The oral presentations are arranged in five parallel sessions. There are two large poster sessions including dinner on two evenings of the conference.

We optimized the conference schedule based on feedback from attendees on the talks they would like to see. We collected attendee responses using a scheduling survey developed with the help of David Yarowsky and Svitlana Volkova, and we optimized the conference schedule to assign popular sessions to large conference rooms, and to reduce the chance that two talks that an attendee is interested in are scheduled at the same time. The number of responses collected by the time the schedule was due was 307. The top five topics with highest interest per paper were Semantics, Lexical Semantics, Information Extraction, Question Answering, and Discourse, Dialogue, Coreference and Pragmatics.
We started with a manual grouping of papers in groups for sessions. We then optimized the assignments of groups to rooms/times, and the order of papers within a group automatically. Compared to the original manually created schedule, the optimized schedule had 644 fewer instances where an attendee was interested in seeing two papers scheduled at the same time. More thorough and exact optimization methods could be applied in the future.

==Recommendations==
Based on our workthis year, we would like to make the following recommendations for future years:
# Tool for allocating reviewers to areas.
# It would be great if the tool is implemented in START: it would make things easier and less confusing for area chairs and reviewers.
# It would be great if the area chairs can bid on reviewers after the submissions are in since there are some very narrow areas where only a handful of reviewers are experts in.
# It would be good if the Softconf has tools for checking non-anonymity and formatting issues. We rejected 15 long and 19 short papers for such issues which could have been prevented if the system could automatically alert the authors of such issues. If there are large technical problems with implementing this, there could be a student volunteer on the organizing committee who could help alter authors that submit at least a few hours before the deadline.
# Enable merging of the schedules for short and long papers in START, as well as uploading metadata and text of for TACL papers, to simplify the process of deriving the conference program for the website, the conference handbook, and the downloadable proceedings.
# Enable the persistence of reviewer lists in START, as well as persist the completion rate of assignments.
# Issue a policy on whether self-published papers (published on Arxiv or the authors’ web pages) should constitute previously published work. This year we published a policy applicable to ACL-14 only, but this changes from year to year and CL conference to conference.
# Issue a policy on whether supplementary materials should be anonymized.

2014Q3 Reports: Program Chairs

2014-06-05T20:28:31Z

KristinaToutanova: /* Innovations */

===Program Chairs (Kristina Toutanova and Hua Wu)===

== Innovations ==

As compared to ACL conferences in prior years, the main innovations this year were:
# We allocated reviewers to areas based on the reviewers’ preferences over areas, the preferences of area chairs over reviewers, and the number of submissions in each area (using a tool developed by Mark Dredze and applied to ACL with the help of Jiang Guo; this tool had been applied to NAACL and EMNLP in the past but was applied at ACL for the first time this year).
# We optimized the conference schedule based on feedback from attendees on the talks they would like to see.
# All talks will be recorded at ACL 2014, as in NAACL 2013.
# We scheduled two large poster sessions on two evenings of the conference, to accommodate the large number of poster presentations. Instead of a banquet, we will have is a no-fee light social event, and the president’s talk will be given on the first morning of the conference.
# We grouped oral TACL papers in thematic sessions together with ACL paper presentations, instead of in separate TACL-only sessions.
# We solicited nominations for outstanding reviewers and acknowledged them in the proceedings (around 14% of reviewers were acknowledged).

== Submissions and Presentations ==

ACL 2014 received a total of 1123 submissions, of which 572 were long papers and 551 were short papers. 15 long papers and 19 short papers were rejected without review due to non-anonymity or formatting issues.The remaining submissions were assigned to one of 20 areas, and managed by a program committee of 33 area chairs and 779 reviewers.

146 (26.2%) of the 557 qualifying long papers and 139 (26.1%) of the 532 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 95 were selected for oral presentation, and 51 for poster presentation. Of the accepted short papers, 51 have oral and 88 have poster presentation slots. The oral versus poster decision was made based not on the quality of the work, but the estimated appeal to a wide audience.

In addition, 19 TACL papers will be presented at ACL – 13 as talks and 6 as posters. Including TACL papers, there will be 159 oral and 145 poster presentations at the main ACL conference.
The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area. The table also shows the number of qualifying long and short papers that were withdrawn prior to the completion of the review process (11 long and 34 short papers were withdrawn).

{| class="wikitable" cellpadding="2" border="1"
| Areas||Long Received||Long Accepted||Short Received||Short Accepted||Total Submissions ||Percent of Total Submission||Total Accepts||Percent of Total Accepts||Area Accept Rate
|-
| Cognitive Modeling and Psycholinguistics ||9 ||3 ||14 ||4 ||23 ||2.11% ||7 ||2.46% ||30.43%
|-
| Dialogue and Interactive Systems ||10 ||2 ||8 ||2 ||18 ||1.65% ||4 ||1.40% ||22.22%
|-
| Discourse, Coreference, and Pragmatics ||22 ||5 ||20 ||5 ||42 ||3.86% ||10 ||3.51% ||23.81%
|-
| Document Categorization, Sentiment Analysis, and Topic Models ||53 ||14 ||48 ||13 ||101 ||9.27% ||27 ||9.47% ||26.73%
|-
| Generation ||13 ||6 ||7 ||4 ||20 ||1.84% ||10 ||3.51% ||50.00%
|-
| Information Extraction and Text Mining ||54 ||13 ||49 ||14 ||103 ||9.46% ||27 ||9.47% ||26.21%
|-
| Information Retrieval ||8 ||2 ||9 ||2 ||17 ||1.56% ||4 ||1.40% ||23.53%
|-
| Language Resources and Evaluation ||31 ||8 ||28 ||10 ||59 ||5.42% ||18 ||6.32% ||30.51%
|-
| Lexical Semantics and Ontology ||26 ||7 ||23 ||6 ||49 ||4.50% ||13 ||4.56% ||26.53%
|-
| Machine Learning for Language Processing ||39 ||13 ||15 ||5 ||54 ||4.96% ||18 ||6.32% ||33.33%
|-
| Machine Translation ||76 ||18 ||72 ||19 ||148 ||13.59% ||37 ||12.98% ||25.00%
|-
| Multilinguality and Multimodal NLP ||12 ||3 ||14 ||3 ||26 ||2.39% ||6 ||2.11% ||23.08%
|-
| NLP Applications and NLP-enabled Technology ||32 ||6 ||34 ||9 ||66 ||6.06% ||15 ||5.26% ||22.73%
|-
| NLP for the Web and Social Media ||29 ||5 ||29 ||7 ||58 ||5.33% ||12 ||4.21% ||20.69%
|-
| Question Answering ||6 ||2 ||4 ||0 ||10 ||0.92% ||2 ||0.70% ||20.00%
|-
| Semantics ||53 ||16 ||37 ||11 ||90 ||8.26% ||27 ||9.47% ||30.00%
|-
| Summarization ||19 ||6 ||11 ||4 ||30 ||2.75% ||10 ||3.51% ||33.33%
|-
| Spoken Language Processing ||9 ||2 ||10 ||3 ||19 ||1.74% ||5 ||1.75% ||26.32%
|-
| Tagging, Chunking, Syntax and Parsing ||35 ||12 ||48 ||14 ||83 ||7.62% ||26 ||9.12% ||31.33%
|-
| Phonology, Morphology and Word Segmentation ||10 ||4 ||18 ||4 ||28 ||2.57% ||8 ||2.81% ||28.57%
|-
| Withdrawn ||11 ||0 ||34 ||0 ||45 ||4.13% ||0 ||0% ||0%
|-
| Total ||557 ||146 ||532 ||139 ||1089 ||100% ||285 ||100% ||26.17%
|}

The top five areas for the highest number of submissions this year were Machine Translation, Information Extraction, Document Categorization/ Sentiment analysis/Topic Models, Semantics, and Tagging/Chunking/Syntax/Parsing.

===Review process===
As mentioned above, we used Mark Dredze’s tool for reviewer assignment, which Jiang Guo helped apply to ACL this year. Thanks to the tool and the collaboration among the area chairs, the committee organized a smooth review process with sufficient number of expert reviewers.
The submissions were reviewed under different categories and using different review forms for empirical/data-driven, theoretical, applications/tools, resources/evaluation, and survey papers. For the short papers we additionally used a negative results category and were glad to see that the community is becoming more open to enabling the publication of useful negative results (two out of six short papers submitted under the negative results category were accepted; additional papers reporting negative results but submitted under other categories were also accepted).

We modified the review forms slightly. One modification was to change the final recommendation range from the 1-to-6 range used last year to a 1-to-5 range. We believe this reduces some unnecessary variance in the scores. Another modification was to reword the questions asking reviewers to check whether any submitted code was well structured, because we thought performing a thorough code review would be an unrealistic demand from the reviewers.

We increased the page limit to 9 pages for submitted long papers and 5 pages for submitted short papers and kept the same limits for the camera-ready versions. The change will likely not be implemented next year.

Formatting and anonymity were checked with the help of the area chairs and the student volunteer Jiang Guo. More than 10% of the submissions had author names in the properties of their files – we chose to manually remove this identifying information, but it would be great if that could be automated. Jiang Guo and Jason Eisner helped us apply automated ways to do this, but they did not work in all cases. Papers that had author names listed under the title were rejected without review unless the authors sent us an email realizing their area in the 48 hours following the deadline; in the latter case we removed the author list from the paper for them.

We did not reject papers for non-anonymity of their supplementary data, software, or notes, since the call for papers did not say anything about whether such materials should be anonymized.

===Best paper awards===
The area chairs nominated sixteen papers for the long best paper award. We selected a list of seven candidates which were ranked by a specially formed best paper committee, consisting of seven members --- six area chairs and one external member. Based on the ranking, we selected a long best paper and a long best student paper winner. The selection of a best short paper is in process. The best long papers will be presented in a plenary session, whereas the best short paper will be presented in one of the parallel sessions.

===Presentations===
The oral presentations are arranged in five parallel sessions. There are two large poster sessions including dinner on two evenings of the conference.

We optimized the conference schedule based on feedback from attendees on the talks they would like to see. We collected attendee responses using a scheduling survey developed with the help of David Yarowsky and Svitlana Volkova, and we optimized the conference schedule to assign popular sessions to large conference rooms, and to reduce the chance that two talks that an attendee is interested in are scheduled at the same time. The number of responses collected by the time the schedule was due was 307. The top five topics with highest interest per paper were Semantics, Lexical Semantics, Information Extraction, Question Answering, and Discourse, Dialogue, Coreference and Pragmatics.

We started with a manual grouping of papers in groups for sessions. We then optimized the assignments of groups to rooms/times, and the order of papers within a group automatically. In our initial manually constructed schedule, there were X instances where an attendee is interested in seeing two papers scheduled at the same time and after optimization, this number was reduced to Y.

==Recommendations==
Based on our workthis year, we would like to make the following recommendations for future years:
# Tool for allocating reviewers to areas.
# It would be great if the tool is implemented in START: it would make things easier and less confusing for area chairs and reviewers.
# It would be great if the area chairs can bid on reviewers after the submissions are in since there are some very narrow areas where only a handful of reviewers are experts in.
# It would be good if the Softconf has tools for checking non-anonymity and formatting issues. We rejected 15 long and 19 short papers for such issues which could have been prevented if the system could automatically alert the authors of such issues. If there are large technical problems with implementing this, there could be a student volunteer on the organizing committee who could help alter authors that submit at least a few hours before the deadline.
# Enable merging of the schedules for short and long papers in START, as well as uploading metadata and text of for TACL papers, to simplify the process of deriving the conference program for the website, the conference handbook, and the downloadable proceedings.
# Enable the persistence of reviewer lists in START, as well as persist the completion rate of assignments.
# Issue a policy on whether self-published papers (published on Arxiv or the authors’ web pages) should constitute previously published work. This year we published a policy applicable to ACL-14 only, but this changes from year to year and CL conference to conference.
# Issue a policy on whether supplementary materials should be anonymized.