2016Q3 Reports: Program Chairs - Revision history

ShiqiZhao at 14:23, 13 July 2016

2016-07-13T14:23:36Z

@@ Line 101: / Line 101: @@
 * 9-15 Apr 2016:  program co-chairs finalize short paper decisions [7 days]
 * 19 May 2016:  short paper camera-ready deadline [34 days]
 * 18 Mar 2016:  long paper submission deadline
@@ Line 114: / Line 115: @@
 This timeline left 60 days between the last camera-ready deadline and the first day of the conference (August 7).

ShiqiZhao at 14:17, 13 July 2016

2016-07-13T14:17:10Z

← Older revision		Revision as of 14:17, 13 July 2016
Line 101:		Line 101:
	* 9-15 Apr 2016: program co-chairs finalize short paper decisions [7 days]		* 9-15 Apr 2016: program co-chairs finalize short paper decisions [7 days]
	* 19 May 2016: short paper camera-ready deadline [34 days]		* 19 May 2016: short paper camera-ready deadline [34 days]


	* 18 Mar 2016: long paper submission deadline		* 18 Mar 2016: long paper submission deadline

ShiqiZhao at 14:16, 13 July 2016

2016-07-13T14:16:18Z

← Older revision		Revision as of 14:16, 13 July 2016
Line 92:		Line 92:
	The complete timeline is given below.		The complete timeline is given below.

	29 Feb 2016: short paper submission deadline		* 29 Feb 2016: short paper submission deadline
	1-3 Mar 2016: program co-chairs assign short papers to areas [3 days]		* 1-3 Mar 2016: program co-chairs assign short papers to areas [3 days]
	4-7 Mar 2016: short paper bidding period [4 days]		* 4-7 Mar 2016: short paper bidding period [4 days]
	8-10 Mar 2016: area chairs assign short papers to reviewers [3 days]		* 8-10 Mar 2016: area chairs assign short papers to reviewers [3 days]
	11-27 Mar 2016: short paper review period [17 days]		* 11-27 Mar 2016: short paper review period [17 days]
	28-31 Mar 2016: short paper reviewer discussion [4 days]		* 28-31 Mar 2016: short paper reviewer discussion [4 days]
	1-8 Apr 2016: short paper recommendations and meta-reviews [8 days]		* 1-8 Apr 2016: short paper recommendations and meta-reviews [8 days]
	9-15 Apr 2016: program co-chairs finalize short paper decisions [7 days]		* 9-15 Apr 2016: program co-chairs finalize short paper decisions [7 days]
	19 May 2016: short paper camera-ready deadline [34 days]		* 19 May 2016: short paper camera-ready deadline [34 days]

	18 Mar 2016: long paper submission deadline
	19-24 Mar 2016: program co-chairs assign long papers to areas [6 days]		* 18 Mar 2016: long paper submission deadline
	25 Mar-1 Apr 2016: long paper bidding period [8 days]		* 19-24 Mar 2016: program co-chairs assign long papers to areas [6 days]
	2-5 April 2016: area chairs assign long papers to reviewers [4 days]		* 25 Mar-1 Apr 2016: long paper bidding period [8 days]
	6-27 Apr 2016: long paper review period [22 days]		* 2-5 April 2016: area chairs assign long papers to reviewers [4 days]
	28 Apr-1 May 2016: long paper author response period [4 days]		* 6-27 Apr 2016: long paper review period [22 days]
	2-6 May 2016: long paper reviewer discussion [5 days]		* 28 Apr-1 May 2016: long paper author response period [4 days]
	7-12 May 2016: long paper recommendations and meta-reviews [6 days]		* 2-6 May 2016: long paper reviewer discussion [5 days]
	13-24 May 2016: program co-chairs finalize long paper decisions [12 days]		* 7-12 May 2016: long paper recommendations and meta-reviews [6 days]
	7 June 2016: long paper camera-ready deadline [14 days]		* 13-24 May 2016: program co-chairs finalize long paper decisions [12 days]
			* 7 June 2016: long paper camera-ready deadline [14 days]

	This timeline left 60 days between the last camera-ready deadline and the first day of the conference (August 7).		This timeline left 60 days between the last camera-ready deadline and the first day of the conference (August 7).

ShiqiZhao at 14:14, 13 July 2016

2016-07-13T14:14:45Z

← Older revision		Revision as of 14:14, 13 July 2016
Line 76:		Line 76:

	As mentioned above, area chairs were asked to submit a meta-review (a functionality that is supported by START) for all papers that were not clear rejects. We found this to be extremely helpful in deciding on borderline papers.		As mentioned above, area chairs were asked to submit a meta-review (a functionality that is supported by START) for all papers that were not clear rejects. We found this to be extremely helpful in deciding on borderline papers.

			== Outstanding and Best Papers ==

			The area chairs nominated 18 papers from 13 areas for recognition as outstanding papers. Out of these, the paper awards committee selected 9 outstanding long papers, of which one was awarded “best paper” and another “best student paper.” For short papers, 4 papers were nominated and 2 selected as outstanding. The paper awards committee was formed before the paper deadlines, and papers were given to the committee in their originally submitted, anonymized form. Reviews and meta-reviews of the nominated papers were also provided to the committee on request.

			== Presentations ==

			The oral presentations are arranged in up to seven parallel sessions. There are two large poster sessions including dinner on two evenings of the conference. We manually grouped the papers into sessions, relying more on the topic of each paper than its area. In the end, only 10 out of 44 oral sessions were made up of papers all reviewed in the same area.

			Based on recommendations of past program chairs, we aimed to have at least 11 m2 available for every poster presented in the poster sessions, to make the space comfortable and easy to move in. Additional space was booked at the poster session venue to allow for this; prior to the conference we estimated that more than 12 m2 would be available for every poster at the busier Monday night poster session.

			== Timeline ==
			The timeline this year was complicated to plan because of pressure to keep the deadlines for ACL far away from the deadlines for NAACL and EMNLP, to allow for serial submission of rejected papers.

			The complete timeline is given below.

			29 Feb 2016: short paper submission deadline
			1-3 Mar 2016: program co-chairs assign short papers to areas [3 days]
			4-7 Mar 2016: short paper bidding period [4 days]
			8-10 Mar 2016: area chairs assign short papers to reviewers [3 days]
			11-27 Mar 2016: short paper review period [17 days]
			28-31 Mar 2016: short paper reviewer discussion [4 days]
			1-8 Apr 2016: short paper recommendations and meta-reviews [8 days]
			9-15 Apr 2016: program co-chairs finalize short paper decisions [7 days]
			19 May 2016: short paper camera-ready deadline [34 days]

			18 Mar 2016: long paper submission deadline
			19-24 Mar 2016: program co-chairs assign long papers to areas [6 days]
			25 Mar-1 Apr 2016: long paper bidding period [8 days]
			2-5 April 2016: area chairs assign long papers to reviewers [4 days]
			6-27 Apr 2016: long paper review period [22 days]
			28 Apr-1 May 2016: long paper author response period [4 days]
			2-6 May 2016: long paper reviewer discussion [5 days]
			7-12 May 2016: long paper recommendations and meta-reviews [6 days]
			13-24 May 2016: program co-chairs finalize long paper decisions [12 days]
			7 June 2016: long paper camera-ready deadline [14 days]

			This timeline left 60 days between the last camera-ready deadline and the first day of the conference (August 7).

ShiqiZhao at 14:12, 13 July 2016

2016-07-13T14:12:23Z

@@ Line 1: / Line 1: @@
 == Innovations ==
-As compared to previous ACL conferences, this year’s main innovations were:
+As compared to previous ACL conferences, this year's main innovations were:
 * In addition to best paper awards, a larger number of outstanding papers were selected.  The original plan was to identify roughly 1-2% of submissions; ultimately 11 papers out of 1290 submissions (0.85%) were identified as outstanding by the awards committee.
@@ Line 57: / Line 57: @@
 <tr><td>total</td><td>825</td><td>231</td><td>463</td><td>97</td><td>1288</td><td>100.0%</td><td>328</td><td>100.0%</td><td>25.5%</td><td>11</td></tr>
 </table>

ShiqiZhao at 14:08, 13 July 2016

2016-07-13T14:08:57Z

← Older revision		Revision as of 14:08, 13 July 2016
Line 20:		Line 20:
	* ''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''		* ''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''

			Meta-reviews were strongly recommended by the NAACL 2015 program co-chairs.

			The deadline for short papers was moved ahead of the deadline for long papers to coordinate deadlines with NAACL, such that rejected NAACL submissions could be reworked into ACL submissions.

			The “Other” area was introduced to give papers in non-traditional research areas a better chance at fair review. Rather than conventional bidding, the “Other” area chairs were given access to the reviewer pool directly (while bidding was taking place for papers in the rest of the areas), so that these papers were given the first chance at the most appropriate reviewers.

			== Submissions and Presentations ==

			ACL 2016 received a total of 1288 valid submissions, of which 825 were long papers and 463 were short papers. 21 long papers and 9 short papers were rejected without review due to non-anonymity or formatting issues. The remaining submissions were each assigned to one of 19 areas, and managed by a program committee of 38 area chairs and 884 reviewers (including secondary reviewers indicated on the review forms). 231 (28%) of the 825 qualifying long papers and 97 (21%) of the 463 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 116 were selected for oral presentation, and 115 for poster presentation. Of the accepted short papers, 49 have oral and 48 have poster presentations. The oral versus poster decision was made based on the recommendations of reviewers, which we took as a noisy signal of the intended audience’s preference of format for each paper.

			In addition, 25 TACL papers will be presented at ACL – 24 as talks and one as a poster. Including TACL papers, there will be 189 oral and 163 poster presentations at the main ACL conference. The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area. Approximately 59 short and 52 long papers were withdrawn before review was completed; these are not included in the table.

ShiqiZhao at 14:07, 13 July 2016

2016-07-13T14:07:29Z

← Older revision		Revision as of 14:07, 13 July 2016
Line 16:		Line 16:
	Appendices were added to help address the replication problem. The relevant passage in the Call for Papers reads as follows:		Appendices were added to help address the replication problem. The relevant passage in the Call for Papers reads as follows:

	''ACL 2016 also encourages the submission of supplementary material to report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.''		* ''ACL 2016 also encourages the submission of supplementary material to report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.''

	''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''		* ''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''

ShiqiZhao at 14:05, 13 July 2016

2016-07-13T14:05:59Z

← Older revision	Revision as of 14:05, 13 July 2016
Line 1:	Line 1:
		== Innovations ==

		As compared to previous ACL conferences, this year’s main innovations were:

		* In addition to best paper awards, a larger number of outstanding papers were selected. The original plan was to identify roughly 1-2% of submissions; ultimately 11 papers out of 1290 submissions (0.85%) were identified as outstanding by the awards committee.
		* Instead of 20-minute talks, long papers were presented in 15-minute talks, plus 5 minutes for questions and changing speakers. Short papers were given 12 minutes plus 4 for questions and transition.
		* Authors were allowed to submit an appendix of any length with their papers. Reviewers were not required to read appendices.
		* Area chairs provided meta-reviews for all papers that were not obvious rejections.
		* The deadline for short papers was ahead of the deadline for long papers.
		* We introduced a new area labeled “Other” to handle papers that did not fit into traditional areas, as well as a few COI papers.

		=== Rationale ===

		Outstanding papers were added because the community is growing and so is the number of papers at each conference, including the number of papers that deserve recognition as being of particular importance and quality.

		Appendices were added to help address the replication problem. The relevant passage in the Call for Papers reads as follows:

		''ACL 2016 also encourages the submission of supplementary material to report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.''

		''Nonetheless, supplementary material should be supplementary (rather than central) to the paper. It may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data. The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to reviewers, they will not be asked to review or even download the supplementary material. Authors should refer to the contents of the supplementary material in the paper submission, so that reviewers interested in these supplementary details will know where to look.''

ShiqiZhao: Created page with "

area long reviewed long accepted short reviewed short accepted total submissions percentage of total s..."
2016-07-13T14:01:08Z

Created page with " <table border> <tr><td>area</td><td>long reviewed</td><td>long accepted</td><td>short reviewed</td><td>short accepted</td><td>total submissions</td><td>percentage of total s..."

New page

<table border>
<tr><td>area</td><td>long reviewed</td><td>long accepted</td><td>short reviewed</td><td>short accepted</td><td>total submissions</td><td>percentage of total submissions</td><td>total accepted</td><td>percentage of total accepted</td><td>area acceptance rate</td><td>outstanding papers</td></tr>
<tr><td>Semantics</td><td>114</td><td>43</td><td>66</td><td>13</td><td>180</td><td>14.0%</td><td>56</td><td>17.1%</td><td>31.1%</td><td>3</td></tr>
<tr><td>Information Extraction, Question Answering, and Text Mining</td><td>122</td><td>27</td><td>48</td><td>9</td><td>170</td><td>13.2%</td><td>36</td><td>11.0%</td><td>21.2%</td><td>1</td></tr>
<tr><td>Sentiment Analysis and Opinion Mining</td><td>77</td><td>9</td><td>28</td><td>3</td><td>105</td><td>8.2%</td><td>12</td><td>3.7%</td><td>11.4%</td><td></td></tr>
<tr><td>Document Analysis</td><td>53</td><td>13</td><td>49</td><td>8</td><td>102</td><td>7.9%</td><td>21</td><td>6.4%</td><td>20.6%</td><td></td></tr>
<tr><td>Machine Translation</td><td>58</td><td>15</td><td>36</td><td>9</td><td>94</td><td>7.3%</td><td>24</td><td>7.3%</td><td>25.5%</td><td>1</td></tr>
<tr><td>Tagging, Chunking, Syntax, and Parsing</td><td>48</td><td>18</td><td>33</td><td>9</td><td>81</td><td>6.3%</td><td>27</td><td>8.2%</td><td>33.3%</td><td>2</td></tr>
<tr><td>Social Media</td><td>39</td><td>6</td><td>30</td><td>8</td><td>69</td><td>5.4%</td><td>14</td><td>4.3%</td><td>20.3%</td><td></td></tr>
<tr><td>Machine Learning</td><td>46</td><td>15</td><td>22</td><td>6</td><td>68</td><td>5.3%</td><td>21</td><td>6.4%</td><td>30.9%</td><td>1</td></tr>
<tr><td>Resources and Evaluation</td><td>44</td><td>17</td><td>21</td><td>6</td><td>65</td><td>5.0%</td><td>23</td><td>7.0%</td><td>35.4%</td><td></td></tr>
<tr><td>Other</td><td>34</td><td>10</td><td>27</td><td>5</td><td>61</td><td>4.7%</td><td>15</td><td>4.6%</td><td>24.6%</td><td></td></tr>
<tr><td>Discourse and Pragmatics</td><td>42</td><td>15</td><td>18</td><td>4</td><td>60</td><td>4.7%</td><td>19</td><td>5.8%</td><td>31.7%</td><td></td></tr>
<tr><td>Summarization</td><td>29</td><td>5</td><td>19</td><td>2</td><td>48</td><td>3.7%</td><td>7</td><td>2.1%</td><td>14.6%</td><td></td></tr>
<tr><td>Multilinguality</td><td>19</td><td>6</td><td>24</td><td>5</td><td>43</td><td>3.3%</td><td>11</td><td>3.4%</td><td>25.6%</td><td></td></tr>
<tr><td>Phonology, Morphology, and Word Segmentation</td><td>23</td><td>6</td><td>10</td><td>4</td><td>33</td><td>2.6%</td><td>10</td><td>3.0%</td><td>30.3%</td><td>1</td></tr>
<tr><td>Generation</td><td>20</td><td>8</td><td>9</td><td>3</td><td>29</td><td>2.3%</td><td>11</td><td>3.4%</td><td>37.9%</td><td></td></tr>
<tr><td>Dialog and Interactive Systems</td><td>20</td><td>7</td><td>8</td><td>0</td><td>28</td><td>2.2%</td><td>7</td><td>2.1%</td><td>25.0%</td><td>1</td></tr>
<tr><td>Cognitive Modeling and Psycholinguistics</td><td>17</td><td>7</td><td>8</td><td>1</td><td>25</td><td>1.9%</td><td>8</td><td>2.4%</td><td>32.0%</td><td>1</td></tr>
<tr><td>Vision, Robots, and Grounding</td><td>14</td><td>3</td><td>6</td><td>2</td><td>20</td><td>1.6%</td><td>5</td><td>1.5%</td><td>25.0%</td><td></td></tr>
<tr><td>Speech</td><td>6</td><td>1</td><td>1</td><td>0</td><td>7</td><td>0.5%</td><td>1</td><td>0.3%</td><td>14.3%</td><td></td></tr>
<tr><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr>
<tr><td>total</td><td>825</td><td>231</td><td>463</td><td>97</td><td>1288</td><td>100.0%</td><td>328</td><td>100.0%</td><td>25.5%</td><td>11</td></tr>
</table>