Difference between revisions of "2017Q3 Reports: Program Chairs"

From Admin Wiki
Jump to navigation Jump to search
 
(13 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
== Innovations ==
 
  
−
As compared to previous ACL conferences, this year's main innovations were:  
+
= '''Program Co-Chairs Report''' =
 +
Regina Barzilay, Massachusetts Institute of Technology <br />
 +
Min-Yen Kan, National University of Singapore
 +
 
 +
The program co-chairs maintained a blog about their process in part to increase transparency and in part to solicit feedback from the community on their new ideas.  I encourage you to read it: https://chairs-blog.acl2017.org
 +
 
 +
The program co-chairs have recruited over 60 area chairs, in part through an open call inviting nominations and self-nominations (https://chairs-blog.acl2017.org/2016/10/24/help-us-recruit-the-best-area-chairs/).
 +
 
 +
Biomedical
 +
* Aurélie Névéol(http://www.limsi.fr/Individu/neveol)
 +
* Karin Verspoor (http://textminingscience.com)
 +
 
 +
Cognitive Modeling and Psycholinguistics
 +
* Roger Levy (http://www.mit.edu/~rplevy/)
 +
* Anders Søgaard (http://cst.dk/anders/)
 +
 
 +
Dialogue and Interactive Systems
 +
* Ron Artstein (http://ron.artstein.org)
 +
* Raquel Fernandez (https://staff.fnwi.uva.nl/r.fernandezrovira/)
 +
* Oliver Lemon (https://www.hw.ac.uk/schools/mathematical-computer-sciences/staff-directory/oliver-lemon.htm)
 +
 
 +
Discourse and Pragmatics
 +
* Yangfeng Ji (http://jiyfeng.github.io)
 +
* Sujian Li (http://123.56.88.210/)
 +
* Bonnie Webber (http://homepages.inf.ed.ac.uk/bonnie/)
 +
 
 +
Information Extraction and Retrieval, Question Answering, Text Mining, Document Analysis and NLP Applications
 +
* Eugene Agichtein (http://www.mathcs.emory.edu/~eugene/)
 +
* Chia-Hui Chang (https://sites.google.com/site/nculab/)
 +
* Jing Jiang (http://www.mysmu.edu/faculty/jingjiang/)
 +
* Sarvnaz Karimi (http://people.csiro.au/K/S/Sarvnaz-Karimi)
 +
* Zornitsa Kozareva (http://www.kozareva.com)
 +
* Kang Liu (http://www.nlpr.ia.ac.cn/cip/~liukang/index.html)
 +
* Tie-Yan Liu (https://www.microsoft.com/en-us/research/people/tyliu/)
 +
* Mausam (http://homes.cs.washington.edu/~mausam/)
 +
* Alessandro Moschitti (https://www.linkedin.com/in/alessandro-moschitti-10999a4)
 +
* Smaranda Muresan (http://www.cs.columbia.edu/~smara/)
 +
 
 +
Machine Learning
 +
* Grzegorz Chrupała (http://grzegorz.chrupala.me/)
 +
* Amir Globerson (http://www.cs.tau.ac.il/~gamir/)
 +
* Tommi Jaakkola (https://people.csail.mit.edu/tommi/)
 +
* Sujith Ravi (http://www.sravi.org/)
 +
* William Yang Wang (https://www.cs.ucsb.edu/~william/)
 +
 
 +
Machine Translation
 +
* Yang Liu (http://nlp.csai.tsinghua.edu.cn/~ly/)
 +
* Minh-Thang Luong (http://stanford.edu/~lmthang/)
 +
* Haitao Mi (http://researcher.watson.ibm.com/researcher/view.php?person=us-hmi)
 +
* Graham Neubig (http://www.phontron.com/)
 +
* Deyi Xiong (http://hlt.suda.edu.cn/~dyxiong/index.html)
 +
 
 +
Multidisciplinary
 +
* Michael Piotrowski (http://dynalabs.de/mxp/)
 +
* Karën Fort (http://www.schplaf.org/kf/index_en.php)
 +
 
 +
Multilinguality
 +
* Omri Abend (http://www.cs.huji.ac.il/~oabend/)
 +
* Mona Diab (http://www.seas.gwu.edu/~mtdiab)
 +
 
 +
Phonology, Morphology and Word Segmentation
 +
* Jason Eisner (http://www.cs.jhu.edu/~jason/)
 +
* Hinrich Schütze (http://www.cis.uni-muenchen.de/schuetze/)
 +
 
 +
Resources and Evaluation
 +
* Sophie Rosset (https://perso.limsi.fr/rosset/)
 +
* Wajdi Zaghouani (http://www.qatar.cmu.edu/~wajdiz/)
 +
 
 +
Semantics
 +
* Manaal Faruqui (http://www.manaalfaruqui.com/)
 +
* Hannaneh Hajishirzi (http://ssli.ee.washington.edu/~hannaneh/)
 +
* Anna Korhonen (https://www.cl.cam.ac.uk/~alk23/)
 +
* Preslav Nakov (http://qcri.org.qa/page?a=117&pid=35&lang=en-CA)
 +
* Mehroosh Sadrzadeh (http://www.eecs.qmul.ac.uk/people/view/33472/dr-mehrnoosh-sadrzadeh)
 +
* Aline Villavicencio (http://www.inf.ufrgs.br/~avillavicencio/)
 +
 
 +
Sentiment Analysis and Opinion Mining
 +
* Alexandra Balahur (https://www.linkedin.com/in/alexandrabalahur)
 +
* Lun-Wei Ku (http://www.lunweiku.com/)
 +
* Saif M Mohammad (http://saifmohammad.com)
 +
 
 +
Social Media
 +
* Zhiyuan Liu (http://nlp.csai.tsinghua.edu.cn/~lzy/)
 +
* Shimei Pan (http://nlp-lab.umbc.edu/home/shimei/)
 +
* Svitlana Volkova (http://www.cs.jhu.edu/~svitlana/)
 +
 
 +
Speech
 +
* Chiori Hori (https://www.linkedin.com/in/chiori-hori-42025823)
 +
* Chia-ying Lee (http://people.csail.mit.edu/chiaying/)
 +
 
 +
Summarization and Generation
 +
* Wenjie Li (http://www4.comp.polyu.edu.hk/~cswjli/)
 +
* Alexander M Rush (http://people.seas.harvard.edu/~srush/)
 +
* Verena Rieser (http://www.macs.hw.ac.uk/~vtr1/)
 +
 
 +
Tagging, Chunking, Syntax and Parsing
 +
* Emily Pitler (http://www.emilypitler.com/)
 +
* Barbara Plank (http://www.let.rug.nl/~bplank/)
 +
* Yue Zhang (http://people.sutd.edu.sg/~yue_zhang/)
 +
* Hai Zhao (http://bcmi.sjtu.edu.cn/~zhaohai/)
 +
 
 +
Vision, Robotics and Grounding
 +
* Mohit Bansal (http://www.cs.unc.edu/~mbansal/)
 +
* Nate Kushman (http://www.kushman.org/)
 +
 
  
−
* In addition to best paper awards, a larger number of outstanding papers were selected.  The original plan was to identify roughly 1-2% of submissions; ultimately 11 papers out of 1290 submissions (0.85%) were identified as outstanding by the awards committee.
+
The program co-chairs attempted to have a balanced set of area chairs in terms of gender, geography and seniority.  Here’s a thoughtful analysis that they did of the makeup of their team: https://chairs-blog.acl2017.org/2017/01/14/on-the-process-of-area-chair-selection/
−
* Instead of 20-minute talks, long papers were presented in 15-minute talks, plus 5 minutes for questions and changing speakers.  Short papers were given 12 minutes plus 4 for questions and transition.
 
−
* Authors were allowed to submit an appendix of any length with their papers. Reviewers were not required to read appendices.
 
−
* Area chairs provided meta-reviews for all papers that were not obvious rejections.
 
−
* The deadline for short papers was ahead of the deadline for long papers.
 
−
* We introduced a new area labeled “Other” to handle papers that did not fit into traditional areas, as well as a few COI papers.
 
  
−
=== Rationale ===
+
In addition to recruiting the 60 area chairs, the program co-chairs invited 1,532 reviewers, explaining their new procedure including a shortened reviewer time and the integration with the Toronto paper matching system (http://torontopapermatching.org/webapp/profileBrowser/about_us/).  This blog post explains the changes to the reviewing process this year: https://chairs-blog.acl2017.org/2016/12/26/changes-to-the-reviewing-process/  1,228 invitees accepted the invitation and 169 that had to decline.  Of the 1,228 that accepted to be reviewers, 893 (72%) have finished the requested survey and we have their area preferences and keywords.  Min and Regina and the area chairs manually assigned the reviewers to corresponding areas for the stragglers.  The program co-chairs also invited PhD students who had previous publications in NLP venues to participate in the reviewing process.
  
−
Outstanding papers were added because the community is growing and so is the number of papers at each conference, including the number of papers that deserve recognition as being of particular importance and quality.
 
  
−
Appendices were added to help address the replication problem. The relevant passage in the Call for Papers reads as follows:
 
  
−
* ''ACL 2016 also encourages the submission of supplementary material to report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.''
+
The deadline for long and short paper submissions was Monday, Feb 6th.
 +
We received 829 long paper submissions, and 590 short paper
 +
submissions (1,359 total). After removing duplicates and erroneous
 +
submissions, 751 long papers and 567 short papers remained (1,318
 +
total). Here are the historical submission rates:
 +
https://www.aclweb.org/aclwiki/index.php?title=Conference_acceptance_rates
 +
. Last year there were 825 long paper submissions (a record) and 463
 +
short paper submissions (1,288 total).
  
−
* ''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''
+
Here is the program co-chairs blog post about the paper submission process and assigning the papers to areas: https://chairs-blog.acl2017.org/2017/02/15/statistics-on-submissions-and-status-update/
  
−
Meta-reviews were strongly recommended by the NAACL 2015 program co-chairs.
+
Paper reviewing and decisions was executed with the usual minor
 +
problems.  The PC chairs operated the areas more such as individual,
 +
parallel conference tracks: areas had a lead meta-AC who was generally
 +
responsible for the track and corresponding with the PC chairs.  This
 +
year, we had a shorter (only 2 weeks) initial review cycle to
 +
facilitate a longer discussion period that followed.  Authors also had
 +
a direct-to-area chair textbox that they could use to write to the
 +
area chairs, which provided additional input to the area chairs in
 +
making their decisions and provided another layer of transparency to
 +
authors (as area chairs must read any input from the authors).
  
−
The deadline for short papers was moved ahead of the deadline for long papers to coordinate deadlines with NAACL, such that rejected NAACL submissions could be reworked into ACL submissions.  
+
ACL 2017 will feature a total of 367 scientific works in the main
 +
conference.  Due to the volume of papers, in coordination with the
 +
local chairs, the PC chairs moved
 +
approximately 200 of these to the poster sessions.  These are broken
 +
down into 195 long papers (25% acceptance rate; 117 and 78, as oral
 +
and poster, respectively) and 107 short papers (18% acceptance rate;
 +
34 and 73, as oral and short, respectively).  The PC chairs also
 +
received a total of 21 TACL papers to accept into the ACL program.
 +
After some consideration, we assigned them to oral presentation slots,
 +
in line with what was done in past years.  There are also 23 papers
 +
from the student research workshop and 21 software demonstrations, all
 +
also assigned to the poster session timings in the evenings.
  
−
The “Other” area was introduced to give papers in non-traditional research areas a better chance at fair review.  Rather than conventional bidding, the “Other” area chairs were given access to the reviewer pool directly (while bidding was taking place for papers in the rest of the areas), so that these papers were given the first chance at the most appropriate reviewers.
+
Even with the shift of works towards the poster sessions burgeoning,
 +
oral presentations will be shorter, with 14+4 minutes and 10+2 for
 +
presentation and Q&A (for long and short, respectively), to
 +
accommodate the volume of the program.
  
−
== Submissions and Presentations ==
+
We solicited from the general public and ACL membership whom to select
 +
for invited speakers.  Unfortunately, the pool of suggestions was not
 +
very rich and we selected and shortlisted over 10 additional
 +
names.  Recruiting an external invited speaker turned out to be
 +
difficult and after several rejections, we decided to go with two
 +
internal-to-the-field candidates, which were Mirella Lapata and Noah
 +
A. Smith.  We look forward to their speeches.
  
−
ACL 2016 received a total of 1288 valid submissions, of which 825 were long papers and 463 were short papers. 21 long papers and 9 short papers were rejected without review due to non-anonymity or formatting issues. The remaining submissions were each assigned to one of 19 areas, and managed by a program committee of 38 area chairs and 884 reviewers (including secondary reviewers indicated on the review forms).  231 (28%) of the 825 qualifying long papers and 97 (21%) of the 463 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 116 were selected for oral presentation, and 115 for poster presentation. Of the accepted short papers, 49 have oral and 48 have poster presentations. The oral versus poster decision was made based on the recommendations of reviewers, which we took as a noisy signal of the intended audience’s preference of format for each paper.
+
The area chairs helped to nominate a pool of best papers from each of
 +
their areas which we further refined with diversity and uniformity
 +
criteria in mind; the shortlisted 15 long papers and 7 short papers
 +
were deemed outstanding papers, and will be presented on the two
 +
parallel sessions on Wednesday.  From this pool, a separate best paper
 +
committee of 5 senior members of the community were recruited to
 +
select and award the best papers.  The committee selected three best
 +
papers through a two stage separate review process: a best long, short
 +
and resource papers.
  
−
In addition, 25 TACL papers will be presented at ACL – 24 as talks and one as a poster. Including TACL papers, there will be 189 oral and 163 poster presentations at the main ACL conference. The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area.  Approximately 59 short and 52 long papers were withdrawn before review was completed; these are not included in the table.
 
  
 +
== Innovations ==
  
−
<table border>
+
As compared to previous ACL conferences, this year's main innovations were:
−
<tr><td>area</td><td>long reviewed</td><td>long accepted</td><td>short reviewed</td><td>short accepted</td><td>total submissions</td><td>percentage of total submissions</td><td>total accepted</td><td>percentage of total accepted</td><td>area acceptance rate</td><td>outstanding papers</td></tr>
 
−
<tr><td>Semantics</td><td>114</td><td>43</td><td>66</td><td>13</td><td>180</td><td>14.0%</td><td>56</td><td>17.1%</td><td>31.1%</td><td>3</td></tr>
 
−
<tr><td>Information Extraction, Question Answering, and Text Mining</td><td>122</td><td>27</td><td>48</td><td>9</td><td>170</td><td>13.2%</td><td>36</td><td>11.0%</td><td>21.2%</td><td>1</td></tr>
 
−
<tr><td>Sentiment Analysis and Opinion Mining</td><td>77</td><td>9</td><td>28</td><td>3</td><td>105</td><td>8.2%</td><td>12</td><td>3.7%</td><td>11.4%</td><td></td></tr>
 
−
<tr><td>Document Analysis</td><td>53</td><td>13</td><td>49</td><td>8</td><td>102</td><td>7.9%</td><td>21</td><td>6.4%</td><td>20.6%</td><td></td></tr>
 
−
<tr><td>Machine Translation</td><td>58</td><td>15</td><td>36</td><td>9</td><td>94</td><td>7.3%</td><td>24</td><td>7.3%</td><td>25.5%</td><td>1</td></tr>
 
−
<tr><td>Tagging, Chunking, Syntax, and Parsing</td><td>48</td><td>18</td><td>33</td><td>9</td><td>81</td><td>6.3%</td><td>27</td><td>8.2%</td><td>33.3%</td><td>2</td></tr>
 
−
<tr><td>Social Media</td><td>39</td><td>6</td><td>30</td><td>8</td><td>69</td><td>5.4%</td><td>14</td><td>4.3%</td><td>20.3%</td><td></td></tr>
 
−
<tr><td>Machine Learning</td><td>46</td><td>15</td><td>22</td><td>6</td><td>68</td><td>5.3%</td><td>21</td><td>6.4%</td><td>30.9%</td><td>1</td></tr>
 
−
<tr><td>Resources and Evaluation</td><td>44</td><td>17</td><td>21</td><td>6</td><td>65</td><td>5.0%</td><td>23</td><td>7.0%</td><td>35.4%</td><td></td></tr>
 
−
<tr><td>Other</td><td>34</td><td>10</td><td>27</td><td>5</td><td>61</td><td>4.7%</td><td>15</td><td>4.6%</td><td>24.6%</td><td></td></tr>
 
−
<tr><td>Discourse and Pragmatics</td><td>42</td><td>15</td><td>18</td><td>4</td><td>60</td><td>4.7%</td><td>19</td><td>5.8%</td><td>31.7%</td><td></td></tr>
 
−
<tr><td>Summarization</td><td>29</td><td>5</td><td>19</td><td>2</td><td>48</td><td>3.7%</td><td>7</td><td>2.1%</td><td>14.6%</td><td></td></tr>
 
−
<tr><td>Multilinguality</td><td>19</td><td>6</td><td>24</td><td>5</td><td>43</td><td>3.3%</td><td>11</td><td>3.4%</td><td>25.6%</td><td></td></tr>
 
−
<tr><td>Phonology, Morphology, and Word Segmentation</td><td>23</td><td>6</td><td>10</td><td>4</td><td>33</td><td>2.6%</td><td>10</td><td>3.0%</td><td>30.3%</td><td>1</td></tr>
 
−
<tr><td>Generation</td><td>20</td><td>8</td><td>9</td><td>3</td><td>29</td><td>2.3%</td><td>11</td><td>3.4%</td><td>37.9%</td><td></td></tr>
 
−
<tr><td>Dialog and Interactive Systems</td><td>20</td><td>7</td><td>8</td><td>0</td><td>28</td><td>2.2%</td><td>7</td><td>2.1%</td><td>25.0%</td><td>1</td></tr>
 
−
<tr><td>Cognitive Modeling and Psycholinguistics</td><td>17</td><td>7</td><td>8</td><td>1</td><td>25</td><td>1.9%</td><td>8</td><td>2.4%</td><td>32.0%</td><td>1</td></tr>
 
−
<tr><td>Vision, Robots, and Grounding</td><td>14</td><td>3</td><td>6</td><td>2</td><td>20</td><td>1.6%</td><td>5</td><td>1.5%</td><td>25.0%</td><td></td></tr>
 
−
<tr><td>Speech</td><td>6</td><td>1</td><td>1</td><td>0</td><td>7</td><td>0.5%</td><td>1</td><td>0.3%</td><td>14.3%</td><td></td></tr>
 
−
<tr><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
 
−
<tr><td>total</td><td>825</td><td>231</td><td>463</td><td>97</td><td>1288</td><td>100.0%</td><td>328</td><td>100.0%</td><td>25.5%</td><td>11</td></tr>
 
−
</table>
 
  
−
The top five areas with the highest number of submissions this year were Semantics; Information Extraction, Question Answering, and Text Mining; Sentiment Analysis and Opinion Mining; Document Analysis (including text categorization, topic models, and retrieval), and Machine Translation.
+
* We used a blog to communicate and interact with the community about a number of important conference and field related issues.
 +
* We opted for a single joint deadline for papers, both long and short.
 +
* We added DOIs into the footers of the papers and changed the style files to incorporate DOI (and as a fallback, ACL Anthology) references to facilitate wayfinding between citing and reference papers. 
 +
* We used the Toronto Paper Matching System (TPMS) to match reviewers to papers.
 +
* We opted to combine a few areas into single, larger areas.
 +
* Through the blog and social media channels, we recruited area chairs and reviewers partially through a crowdsourcing process.  Invited speaker candidates were also nominated through this process.
 +
* We shortened the initial review cycle to two weeks and lengthened the discussion period.  We asked the recruited area chairs to be sensitive to this requirement.
 +
* We added a direct-to-AC communication text box, on the review form, to allow authors to see a higher audience when they felt reviewers misunderstood key aspects of the work.
 +
* We renamed the “Other” area, introduced last year, to "Multidisciplinary" to cater to a wider audience and to carry a more positive sentiment.
  
−
== Review Process ==
+
=== Rationale ===
  
−
The page limit was 8 pages for long paper submissions and 4 pages for short paper submissions (each with unlimited pages for references). Camera-ready versions were given one additional page: 9 plus ∞ for long papers, 5 plus ∞ for short papers. As mentioned above, authors were additionally allowed to submit an appendix of unlimited length.  
+
We wanted to make the process of organizing the conference a transparent one. For this reason we started the blog, which evolved into an online, ongoing dialogue for discussion about certain issues.
 +
We stayed away from using the blog as an announcement channel for conference related issues, as that can better be done through mass email and/or social media channels.
  
−
We changed the review forms slightly. We divided the SOUNDNESS category into one category for theoretical soundness and one for empirical soundness. We also pre-structured the comment box in the review form with three headings, “strengths,” “weaknesses,” and “discussion.” Reviewers were free to delete these headings when authoring their reviews.
+
We felt that the dialogue between authors and reviewers does not always work out well.  A few of our innovations were catered to address this, most notably the shortened initial review period and the longer dialog period. We stressed this when recruiting both area chairs and reviewers. We did not incorporate a formal meta-reviews unlike previous conferences, but worked closely with the area chairs to ensure healthy dialogue between authors and reviewers was maintained all the way until the final acceptance decisions.  Larger areas also facilitate less fracturing among disciplines and allows reviewers and the assignment software better chances to find qualified reviewers.  This meant that large areas needed a "meta area chair" that would oversee the entire process and help facilitate direct communication with the PC chairs to lessen confusion among ACS of big areas.
  
−
Reviewer load balancing continues to be a challenge for this conference.  Reviewers were invited directly by area chairs, then asked later to fill out a survey indicating which areas they were willing to review for.  After papers were submitted and routed to areas, we used a tool provided by Mark Dredze (and used in previous ACL conferences) to improve the load balancing across areas.  This worked well for short papers, but not as well for the much larger set of long papers, and even with considerable manual corrections, there was considerable disparity in the reviewing load across areas.  We suspect the greedy algorithm implemented in this tool is partly to blame, but cannot rule out the possibility that we just needed more reviewers or more flexible reviewers to achieve balance.  We expect that reviewer recruiting and assignment will continue to be a challenge as ACL grows.
+
We wanted to also ensure that the legacy of the conference through its papers are facilitated to the wider, global audience.  This meant adding hyperlinks to references and ensuring that our papers can be easily referenced by other fields.
  
−
There was a large number of papers for which area chairs had conflicts of interest, about 5-10% of submissions, including some papers where one of the co-PCs had a conflict. While the ACL policy at that time stated that “The identity of the reviewers of a paper shall be withheld from  all people who have a conflict of interest in that paper,” we found that this was not doable for this large number of papers. Together with the ACL 2016 coordinating committee, we decided on the following procedure for ACL 2016:
+
== Submissions and Presentations ==
  
−
* As the START tool allows ACL PC co-chairs to see all papers and all reviewers, PC co-chairs did not submit to the conference, but they do handle most aspects of the process for all papers that were submitted to the conference.  When one PC co-chair had a COI with a paper, the other co-chair made final decisions about acceptance and presentation format.
+
Details on the submissions and acceptances by area are detailed on the PC chairs blog:
−
* Papers co-authored by area chairs are assigned to areas other than the one that they chair.
 
−
* If an area chair has a COI with a paper (but is not an author on it), the other area chairs of the same area handle the paper, but the paper remains in the area. START supports this by making such papers invisible to the area chair with the COI. (Note though that there were some glitches where papers were not assigned meta-reviews because of visibility issues.  These were handled informally when discovered.)
 
−
* If all area chairs of an area have a COI with the same paper, the paper is re-assigned to a different area.  The “Other” area was a reasonable last resort.
 
  
−
As mentioned above, area chairs were asked to submit a meta-review (a functionality that is supported by START) for all papers that were not clear rejects. We found this to be extremely helpful in deciding on borderline papers.
+
[[https://chairs-blog.acl2017.org/2017/02/15/statistics-on-submissions-and-status-update/ Statistics on submissions and Status Update]]
  
 
== Outstanding and Best Papers ==
 
== Outstanding and Best Papers ==
−
 
−
From the nominations by t
 
  
 
The area chairs nominated outstanding papers from their areas.  Following this stage, the PC chairs selected a pool of 22 (approximate 1.5%) papers from the areas, using criteria to normalize criteria across areas and diversity.   
 
The area chairs nominated outstanding papers from their areas.  Following this stage, the PC chairs selected a pool of 22 (approximate 1.5%) papers from the areas, using criteria to normalize criteria across areas and diversity.   
Line 115: Line 236:
  
 
We recommend that ACL keep with using TPMS to help assign reviewers to reviews.  However, TPMS is only as good as its profiles.  To benefit from it, ACL needs to support and encourage its use.  There is a difficulty of its potential costs; at the outset we were not informed that TPMS would incur cost, but ACL was billed 2K USD for its use, but this was eventually waived.  A clear agreement needs to be set before its use.  Even though we feel this mitigated assignment difficulties, it is still not a solved matter and needs a lot of care.  Manual intervention and checks are necessary with any amount of automation.
 
We recommend that ACL keep with using TPMS to help assign reviewers to reviews.  However, TPMS is only as good as its profiles.  To benefit from it, ACL needs to support and encourage its use.  There is a difficulty of its potential costs; at the outset we were not informed that TPMS would incur cost, but ACL was billed 2K USD for its use, but this was eventually waived.  A clear agreement needs to be set before its use.  Even though we feel this mitigated assignment difficulties, it is still not a solved matter and needs a lot of care.  Manual intervention and checks are necessary with any amount of automation.
 +
 +
We also had to systematically reject certain submissions that had submission formatting problems.  We encourage future ACL committees to consider building systematic formatting checker to fix.
 +
 +
We also recommend to have secretarial support to the program chairs to help with operational support.  We recommend having long-term support who perform this support over multiple conferences.
  
 
We also detail our actions with respect to the outcomes and recommendations from ACL 2016, on two of their relevant points; see [[https://www.aclweb.org/adminwiki/index.php?title=2016Q3_Reports:_Program_Chairs 2016 Program Chairs' report]].
 
We also detail our actions with respect to the outcomes and recommendations from ACL 2016, on two of their relevant points; see [[https://www.aclweb.org/adminwiki/index.php?title=2016Q3_Reports:_Program_Chairs 2016 Program Chairs' report]].

Latest revision as of 12:45, 30 July 2017

Program Co-Chairs Report

Regina Barzilay, Massachusetts Institute of Technology
Min-Yen Kan, National University of Singapore

The program co-chairs maintained a blog about their process in part to increase transparency and in part to solicit feedback from the community on their new ideas. I encourage you to read it: https://chairs-blog.acl2017.org

The program co-chairs have recruited over 60 area chairs, in part through an open call inviting nominations and self-nominations (https://chairs-blog.acl2017.org/2016/10/24/help-us-recruit-the-best-area-chairs/).

Biomedical

Cognitive Modeling and Psycholinguistics

Dialogue and Interactive Systems

Discourse and Pragmatics

Information Extraction and Retrieval, Question Answering, Text Mining, Document Analysis and NLP Applications

Machine Learning

Machine Translation

Multidisciplinary

Multilinguality

Phonology, Morphology and Word Segmentation

Resources and Evaluation

Semantics

Sentiment Analysis and Opinion Mining

Social Media

Speech

Summarization and Generation

Tagging, Chunking, Syntax and Parsing

Vision, Robotics and Grounding


The program co-chairs attempted to have a balanced set of area chairs in terms of gender, geography and seniority. Here’s a thoughtful analysis that they did of the makeup of their team: https://chairs-blog.acl2017.org/2017/01/14/on-the-process-of-area-chair-selection/

In addition to recruiting the 60 area chairs, the program co-chairs invited 1,532 reviewers, explaining their new procedure including a shortened reviewer time and the integration with the Toronto paper matching system (http://torontopapermatching.org/webapp/profileBrowser/about_us/). This blog post explains the changes to the reviewing process this year: https://chairs-blog.acl2017.org/2016/12/26/changes-to-the-reviewing-process/ 1,228 invitees accepted the invitation and 169 that had to decline. Of the 1,228 that accepted to be reviewers, 893 (72%) have finished the requested survey and we have their area preferences and keywords. Min and Regina and the area chairs manually assigned the reviewers to corresponding areas for the stragglers. The program co-chairs also invited PhD students who had previous publications in NLP venues to participate in the reviewing process.


The deadline for long and short paper submissions was Monday, Feb 6th. We received 829 long paper submissions, and 590 short paper submissions (1,359 total). After removing duplicates and erroneous submissions, 751 long papers and 567 short papers remained (1,318 total). Here are the historical submission rates: https://www.aclweb.org/aclwiki/index.php?title=Conference_acceptance_rates . Last year there were 825 long paper submissions (a record) and 463 short paper submissions (1,288 total).

Here is the program co-chairs blog post about the paper submission process and assigning the papers to areas: https://chairs-blog.acl2017.org/2017/02/15/statistics-on-submissions-and-status-update/

Paper reviewing and decisions was executed with the usual minor problems. The PC chairs operated the areas more such as individual, parallel conference tracks: areas had a lead meta-AC who was generally responsible for the track and corresponding with the PC chairs. This year, we had a shorter (only 2 weeks) initial review cycle to facilitate a longer discussion period that followed. Authors also had a direct-to-area chair textbox that they could use to write to the area chairs, which provided additional input to the area chairs in making their decisions and provided another layer of transparency to authors (as area chairs must read any input from the authors).

ACL 2017 will feature a total of 367 scientific works in the main conference. Due to the volume of papers, in coordination with the local chairs, the PC chairs moved approximately 200 of these to the poster sessions. These are broken down into 195 long papers (25% acceptance rate; 117 and 78, as oral and poster, respectively) and 107 short papers (18% acceptance rate; 34 and 73, as oral and short, respectively). The PC chairs also received a total of 21 TACL papers to accept into the ACL program. After some consideration, we assigned them to oral presentation slots, in line with what was done in past years. There are also 23 papers from the student research workshop and 21 software demonstrations, all also assigned to the poster session timings in the evenings.

Even with the shift of works towards the poster sessions burgeoning, oral presentations will be shorter, with 14+4 minutes and 10+2 for presentation and Q&A (for long and short, respectively), to accommodate the volume of the program.

We solicited from the general public and ACL membership whom to select for invited speakers. Unfortunately, the pool of suggestions was not very rich and we selected and shortlisted over 10 additional names. Recruiting an external invited speaker turned out to be difficult and after several rejections, we decided to go with two internal-to-the-field candidates, which were Mirella Lapata and Noah A. Smith. We look forward to their speeches.

The area chairs helped to nominate a pool of best papers from each of their areas which we further refined with diversity and uniformity criteria in mind; the shortlisted 15 long papers and 7 short papers were deemed outstanding papers, and will be presented on the two parallel sessions on Wednesday. From this pool, a separate best paper committee of 5 senior members of the community were recruited to select and award the best papers. The committee selected three best papers through a two stage separate review process: a best long, short and resource papers.


Innovations

As compared to previous ACL conferences, this year's main innovations were:

  • We used a blog to communicate and interact with the community about a number of important conference and field related issues.
  • We opted for a single joint deadline for papers, both long and short.
  • We added DOIs into the footers of the papers and changed the style files to incorporate DOI (and as a fallback, ACL Anthology) references to facilitate wayfinding between citing and reference papers.
  • We used the Toronto Paper Matching System (TPMS) to match reviewers to papers.
  • We opted to combine a few areas into single, larger areas.
  • Through the blog and social media channels, we recruited area chairs and reviewers partially through a crowdsourcing process. Invited speaker candidates were also nominated through this process.
  • We shortened the initial review cycle to two weeks and lengthened the discussion period. We asked the recruited area chairs to be sensitive to this requirement.
  • We added a direct-to-AC communication text box, on the review form, to allow authors to see a higher audience when they felt reviewers misunderstood key aspects of the work.
  • We renamed the “Other” area, introduced last year, to "Multidisciplinary" to cater to a wider audience and to carry a more positive sentiment.

Rationale

We wanted to make the process of organizing the conference a transparent one. For this reason we started the blog, which evolved into an online, ongoing dialogue for discussion about certain issues. We stayed away from using the blog as an announcement channel for conference related issues, as that can better be done through mass email and/or social media channels.

We felt that the dialogue between authors and reviewers does not always work out well. A few of our innovations were catered to address this, most notably the shortened initial review period and the longer dialog period. We stressed this when recruiting both area chairs and reviewers. We did not incorporate a formal meta-reviews unlike previous conferences, but worked closely with the area chairs to ensure healthy dialogue between authors and reviewers was maintained all the way until the final acceptance decisions. Larger areas also facilitate less fracturing among disciplines and allows reviewers and the assignment software better chances to find qualified reviewers. This meant that large areas needed a "meta area chair" that would oversee the entire process and help facilitate direct communication with the PC chairs to lessen confusion among ACS of big areas.

We wanted to also ensure that the legacy of the conference through its papers are facilitated to the wider, global audience. This meant adding hyperlinks to references and ensuring that our papers can be easily referenced by other fields.

Submissions and Presentations

Details on the submissions and acceptances by area are detailed on the PC chairs blog:

[Statistics on submissions and Status Update]

Outstanding and Best Papers

The area chairs nominated outstanding papers from their areas. Following this stage, the PC chairs selected a pool of 22 (approximate 1.5%) papers from the areas, using criteria to normalize criteria across areas and diversity.

Out of these, the best paper committee of 5 (headed by Min as PC chair) awarded the “best long paper”, "best short paper", and an additional "best resource paper", in a two stage process. Papers were given to the committee in their camera ready form, with author affiliations. Reviews and meta-reviews of the nominated papers were also provided to the committee for reference but members were asked to provide their recommendations and justification (n.b., not review) first without reference to the supplied reviews.

Presentations

The oral presentations are arranged in five parallel sessions. There are two large poster sessions including dinner on the two first evenings of the conference, that include poster presentation space for the system demonstrations and the student research workshop. We manually grouped the papers into sessions largely by area, with TACL papers being manually assigned into the same area hierarchy by us.

We followed the previous guidelines for poster presentations and had 11 m2 available for every poster presented in the poster sessions, to make the space comfortable and easy to move in.

Timeline

In anticipation of a larger pool of submissions, we intentionally scheduled the short and long paper deadlines as a joint, single date deadline..

The complete timeline after submission is given below:

  • Feb 9-12 Paper bidding
  • Feb 13 ACs assign papers to reviewers
  • Feb 13-27 Paper reviewing
  • Feb 28 AC check that all reviews have been received
  • Mar 13-15 Author response period
  • Mar 16-20 Reviewers update reviews after reading author response
  • Mar 25 ACs send preliminary ranking to PCs
  • Mar 28 ACs produce Meta reviews for borderline papers; ACs produce final rankings and Accept / Reject decisions
  • Mar 30 Notification of acceptance
  • April 22 Camera ready deadline

We had one exception to the schedule of completing transmission of the acceptance decisions, being about 12 hours late, due to operations difficulty.

Also, recruitment of the invited speakers started later than we had initially envisioned, which may have led to problems recruiting an appropriate external-to-the-field speaker.

Recommendations

We recommend starting the recruitment of a good external speaker well in advance as possible, as things become busy quite early on in planning the submission process.

We recommend that ACL keep with using TPMS to help assign reviewers to reviews. However, TPMS is only as good as its profiles. To benefit from it, ACL needs to support and encourage its use. There is a difficulty of its potential costs; at the outset we were not informed that TPMS would incur cost, but ACL was billed 2K USD for its use, but this was eventually waived. A clear agreement needs to be set before its use. Even though we feel this mitigated assignment difficulties, it is still not a solved matter and needs a lot of care. Manual intervention and checks are necessary with any amount of automation.

We also had to systematically reject certain submissions that had submission formatting problems. We encourage future ACL committees to consider building systematic formatting checker to fix.

We also recommend to have secretarial support to the program chairs to help with operational support. We recommend having long-term support who perform this support over multiple conferences.

We also detail our actions with respect to the outcomes and recommendations from ACL 2016, on two of their relevant points; see [2016 Program Chairs' report].

> 2. Many reviews were late. At the time that author response started, one third of the papers had at least one review missing, and some papers had all three reviews missing. We recommend leaving a few extra days between the end of reviewing and the start of author response, and starting some way of passing information about delinquent reviewers forward from conference to conference.

We mitigated this somewhat by having a shorter initial review cycle. While certain reviewers were late at this stage, we had a lengthened dialogue period that made it much easier to control for delays in reviews coming in. We recommend also setting an outstanding reviewer recognition award to a somewhat large proportion of reviewers (perhaps 5%) to spur on-time review and the necessary service time to do a good job of reviewing.

> 3. As discussed above, the reviewer load balancing task needs a more principled solution so that enough reviewers are recruited in advance of the deadlines and so that load balancing is handled smoothly with a good outcome.

This was mitigated somewhat by the solution of re-using the previous NAACL and ACL roles for reviewers. However, this has the potential problem (noted by Michael Strube) that personal information of reviewers is circulated to new chairs without explicit permission by the reviewer. Post-conference, we will try to solicit reviewers' explicit permission to have ACL store their personal profile for subsequent program committees.

Retrieved from "https://www.aclweb.org/adminwiki/index.php?title=2017Q3_Reports:_Program_Chairs&oldid=72055"