2019Q3 Reports: NAACL 2019 - Revision history

ChristyDoran: /* Detailed statistics by area */

2019-08-06T21:44:06Z

Detailed statistics by area

← Older revision		Revision as of 21:44, 6 August 2019
Line 261:		Line 261:

	{\| class="wikitable"		{\| class="wikitable"
	!
	! style="text-align:left;"\| Area		! style="text-align:left;"\| Area
	! style="text-align:left;"\| Long (%)		! style="text-align:left;"\| Long (%)
	! style="text-align:left;"\| Short (%)		! style="text-align:left;"\| Short (%)
	\|-		\|-
	\|
	\| Bio and clinical NLP		\| Bio and clinical NLP
	\| 7 (57)		\| 7 (57)
	\| 28 (17)		\| 28 (17)
	\|-		\|-
	\|
	\| Question Answering		\| Question Answering
	\| 73 (36)		\| 73 (36)
	\| 41 (17)		\| 41 (17)
	\|-		\|-
	\|
	\| Cognitive modeling		\| Cognitive modeling
	\| 24 (29)		\| 24 (29)
	\| 14 (14)		\| 14 (14)
	\|-		\|-
	\|
	\| Resources and Evaluation		\| Resources and Evaluation
	\| 33 (27)		\| 33 (27)
	\| 20 (20)		\| 20 (20)
	\|-		\|-
	\|
	\| Dialog and Interactive systems		\| Dialog and Interactive systems
	\| 64 (20)		\| 64 (20)
	\| 18 (27)		\| 18 (27)
	\|-		\|-
	\|
	\| Semantics		\| Semantics
	\| 80 (13)		\| 80 (13)
	\| 42 (11)		\| 42 (11)
	\|-		\|-
	\|
	\| Discourse and Pragmatics		\| Discourse and Pragmatics
	\| 38 (21)		\| 38 (21)
	\| 11 (36)		\| 11 (36)
	\|-		\|-
	\|
	\| Sentiment Analysis		\| Sentiment Analysis
	\| 32 (28)		\| 32 (28)
	\| 40 (20)		\| 40 (20)
	\|-		\|-
	\|
	\| Ethics, Bias and Fairness		\| Ethics, Bias and Fairness
	\| 16 (25)		\| 16 (25)
	\| 12 (50)		\| 12 (50)
	\|-		\|-
	Social Media		\| Social Media
	44 (18)		\| 44 (18)
	41 (36)		\| 41 (36)
	\|-		\|-
	Generation		\| Generation
	46 (14)		\| 46 (14)
	19 (23)		\| 19 (23)
	\|-		\|-
	Speech		\| Speech
	19 (31)		\| 19 (31)
	9 (33)		\| 9 (33)
	\|-		\|-
	Information Extraction		\| Information Extraction
	46 (28)		\| 46 (28)
	16 (12)		\| 16 (12)
	~~Style~~
	~~24 ( (25)~~
	~~16 (25~~)
	\|-		\|-
	~~Information Retrieval~~		\| Style
	22 (22)		\| 24 ( (25)
	13 (30)		\| 16 (25)
	\|-		\|-
	~~Summarization~~		\| Information Retrieval
	22 (27)		\| 22 (22)
	28 (28)		\| 13 (30)
	\|-		\|-
	~~Machine Learning for NLP~~		\| Summarization
	~~100~~ (29)		\| 22 (27)
	22 (22)		\| 28 (28)
	\|-		\|-
	~~Syntax~~		\| Machine Learning for NLP
	36 (52)		\| 100 (29)
	54 (13)		\| 22 (22)
	\|-		\|-
	~~Machine Translation~~		\| Syntax
	49 (30)		\| 36 (52)
	53 (18)		\| 54 (13)
	\|-		\|-
	~~Text Mining~~		\| Machine Translation
	~~101~~ (18)		\| 49 (30)
	29 (24)		\| 53 (18)
	\|-		\|-
	Multilingual NLP		\| Text Mining
	43 (25)		\| 101 (18)
	28 (10)		\| 29 (24)
			\|-
			\| Multilingual NLP
			\| 43 (25)
			\| 28 (10)
	\|-		\|-
	Theory and Formalisms		\| Theory and Formalisms
	12 (58)		\| 12 (58)
	12 (16)		\| 12 (16)
	\|-		\|-
	NLP Applications		\| NLP Applications
	60 (30)		\| 60 (30)
	41 (17)		\| 41 (17)
	\|-		\|-
	Vision & Robotics		\| Vision & Robotics
	41 (12)		\| 41 (12)
	22 (36)		\| 22 (36)
	\|-		\|-
	Phonology		\| Phonology
	24 (33)		\| 24 (33)
	24 (25)		\| 24 (25)
	\|}		\|}

ChristyDoran: /* Detailed statistics by area */

2019-08-06T21:40:19Z

Detailed statistics by area

← Older revision		Revision as of 21:40, 6 August 2019
Line 266:		Line 266:
	! style="text-align:left;"\| Short (%)		! style="text-align:left;"\| Short (%)
	\|-		\|-
	Bio and clinical NLP		\|
	7 (57)		\| Bio and clinical NLP
	28 (17)		\| 7 (57)
			\| 28 (17)
	\|-		\|-
	Question Answering		\|
	73 (36)		\| Question Answering
	41 (17)		\| 73 (36)
			\| 41 (17)
	\|-		\|-
	Cognitive modeling		\|
	24 (29)		\| Cognitive modeling
	14 (14)		\| 24 (29)
			\| 14 (14)
	\|-		\|-
	Resources and Evaluation		\|
	33 (27)		\| Resources and Evaluation
	20 (20)		\| 33 (27)
			\| 20 (20)
	\|-		\|-
	Dialog and Interactive systems		\|
	64 (20)		\| Dialog and Interactive systems
	18 (27)		\| 64 (20)
			\| 18 (27)
	\|-		\|-
	Semantics		\|
	80 (13)		\| Semantics
	42 (11)		\| 80 (13)
			\| 42 (11)
	\|-		\|-
	Discourse and Pragmatics		\|
	38 (21)		\| Discourse and Pragmatics
	11 (36)		\| 38 (21)
			\| 11 (36)
	\|-		\|-
	Sentiment Analysis		\|
	32 (28)		\| Sentiment Analysis
	40 (20)		\| 32 (28)
			\| 40 (20)
	\|-		\|-
	Ethics, Bias and Fairness		\|
	16 (25)		\| Ethics, Bias and Fairness
	12 (50)		\| 16 (25)
			\| 12 (50)
	\|-		\|-
	Social Media		Social Media
Line 344:		Line 353:
	101 (18)		101 (18)
	29 (24)		29 (24)
			\|-
	Multilingual NLP		Multilingual NLP
	43 (25)		43 (25)
	28 (10)		28 (10)
			\|-
	Theory and Formalisms		Theory and Formalisms
	12 (58)		12 (58)
	12 (16)		12 (16)
			\|-
	NLP Applications		NLP Applications
	60 (30)		60 (30)
	41 (17)		41 (17)
			\|-
	Vision & Robotics		Vision & Robotics
	41 (12)		41 (12)
	22 (36)		22 (36)
			\|-
	Phonology		Phonology
	24 (33)		24 (33)
	24 (25)		24 (25)
			\|}

	== Conference tracks ==		== Conference tracks ==

ChristyDoran: /* Detailed statistics by area */

2019-08-06T21:35:33Z

Detailed statistics by area

← Older revision		Revision as of 21:35, 6 August 2019
Line 260:		Line 260:
	== Detailed statistics by area ==		== Detailed statistics by area ==

	~~NEED TO FORMAT TABLE~~		{\| class="wikitable"
			!
	Area		! style="text-align:left;"\| Area
	Long (%)		! style="text-align:left;"\| Long (%)
	Short (%)		! style="text-align:left;"\| Short (%)
	~~Area~~		\|-
	~~Long (%)~~
	~~Short (%)~~
	Bio and clinical NLP		Bio and clinical NLP
	7 (57)		7 (57)
	28 (17)		28 (17)
			\|-
	Question Answering		Question Answering
	73 (36)		73 (36)
	41 (17)		41 (17)
			\|-
	Cognitive modeling		Cognitive modeling
	24 (29)		24 (29)
	14 (14)		14 (14)
			\|-
	Resources and Evaluation		Resources and Evaluation
	33 (27)		33 (27)
	20 (20)		20 (20)
			\|-
	Dialog and Interactive systems		Dialog and Interactive systems
	64 (20)		64 (20)
	18 (27)		18 (27)
			\|-
	Semantics		Semantics
	80 (13)		80 (13)
	42 (11)		42 (11)
			\|-
	Discourse and Pragmatics		Discourse and Pragmatics
	38 (21)		38 (21)
	11 (36)		11 (36)
			\|-
	Sentiment Analysis		Sentiment Analysis
	32 (28)		32 (28)
	40 (20)		40 (20)
			\|-
	Ethics, Bias and Fairness		Ethics, Bias and Fairness
	16 (25)		16 (25)
	12 (50)		12 (50)
			\|-
	Social Media		Social Media
	44 (18)		44 (18)
	41 (36)		41 (36)
			\|-
	Generation		Generation
	46 (14)		46 (14)
	19 (23)		19 (23)
			\|-
	Speech		Speech
	19 (31)		19 (31)
	9 (33)		9 (33)
			\|-
	Information Extraction		Information Extraction
	46 (28)		46 (28)
Line 310:		Line 320:
	24 ( (25)		24 ( (25)
	16 (25)		16 (25)
			\|-
	Information Retrieval		Information Retrieval
	22 (22)		22 (22)
	13 (30)		13 (30)
			\|-
	Summarization		Summarization
	22 (27)		22 (27)
	28 (28)		28 (28)
			\|-
	Machine Learning for NLP		Machine Learning for NLP
	100 (29)		100 (29)
	22 (22)		22 (22)
			\|-
	Syntax		Syntax
	36 (52)		36 (52)
	54 (13)		54 (13)
			\|-
	Machine Translation		Machine Translation
	49 (30)		49 (30)
	53 (18)		53 (18)
			\|-
	Text Mining		Text Mining
	101 (18)		101 (18)

ChristyDoran: /* Submissions rates and distributions */

2019-08-06T21:30:29Z

Submissions rates and distributions

← Older revision		Revision as of 21:30, 6 August 2019
Line 225:		Line 225:
	Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.		Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.

	~~NEED TO CONVERT LATEX~~

	{\|		{\| class="wikitable"
			!
	! style="text-align:left;"\| Long		! style="text-align:left;"\| Long
	! Short		! style="text-align:left;"\| Short
	! Total		! style="text-align:left;"\| Total
	! TACL		! style="text-align:left;"\|TACL
	\|- Reviewed \| 1067 \| 666 \| 1733		\|-
	\|- Accepted as talk \| 140 \| 72 \| 212		\| Reviewed
	\|-Accepted as poster \| 141 \| 70 \| 211 \| 5		\| 1067
	\|-Total Accepted \| 281 (26.3\%) \| 142 (21.3\%) \| 423 (24.4\%) \| 9		\| 666
			\| 1733
			\|
			\|-
			\| Accepted as talk
			\| 140
			\| 72
			\| 212
			\| 4
			\|-
			\| Accepted as poster
			\| 141
			\| 70
			\| 211
			\| 5
			\|-
			\| Total Accepted
			\| 281 (26.3%)
			\| 142 (21.3%)
			\| 423 (24.4%)
			\| 9
	\|}		\|}

ChristyDoran: /* Submissions rates and distributions */

2019-08-06T21:22:19Z

Submissions rates and distributions

← Older revision		Revision as of 21:22, 6 August 2019
Line 227:		Line 227:
	NEED TO CONVERT LATEX		NEED TO CONVERT LATEX

	~~Acceptance break~~-~~down~~:		{\|
	~~\begin{table}[h]~~		! style="text-align:left;"\| Long
	~~\centering~~		! Short
	~~\begin{tabular}{~~\|~~l\|l\|l\|l\|l\|}~~		! Total
	~~\hline~~		! TACL
	~~&\textbf{Long}& \textbf{Short} &\textbf{~~Total~~} & \textbf{~~TACL~~}\\ \hline~~		\|- Reviewed \| 1067 \| 666 \| 1733
	Reviewed & 1067 & 666 & 1733 ~~& \\~~		\|- Accepted as talk \| 140 \| 72 \| 212
	Accepted as talk & 140 & 72 & 212 ~~& 4\\~~		\|-Accepted as poster \| 141 \| 70 \| 211 \| 5
	Accepted as poster & 141 & 70 & ~~211 &~~ 5\\		\|-Total Accepted \| 281 (26.3\%) \| 142 (21.3\%) \| 423 (24.4\%) \| 9
	Total Accepted & 281 (26.3\%) & 142 (21.3\%) & 423 (24.4\%) & 9\\		\|}
	~~\hline~~
	~~\end{tabular}~~
	~~\end{table~~}

	== Detailed statistics by area ==		== Detailed statistics by area ==

ChristyDoran: /* Issues and recommendations */

2019-08-06T21:10:36Z

Issues and recommendations

← Older revision		Revision as of 21:10, 6 August 2019
Line 468:		Line 468:
	: Consider moving NAACL to spring so that *ACL timelines are less compressed and NAACL reviewing does not fall over end-of-year holidays		: Consider moving NAACL to spring so that *ACL timelines are less compressed and NAACL reviewing does not fall over end-of-year holidays
	: More automation of format checks in START & better documentation of the ones that are already there (obscure and buried flags) to ease the desk reject process		: More automation of format checks in START & better documentation of the ones that are already there (obscure and buried flags) to ease the desk reject process
	:		: Allow extension of START COI tools to allow authors to list reviewers who should not be assigned to their paper

ChristyDoran: /* Issues and recommendations */

2019-08-06T21:01:27Z

Issues and recommendations

← Older revision		Revision as of 21:01, 6 August 2019
Line 447:		Line 447:
	= Issues and recommendations =		= Issues and recommendations =

	~~Needs editing~~		;Maintaining anonymity
			: Wording of ACL policies invites reinterpretation (e.g. "are asked not to publicize [the paper] further during the anonymity period – the submitted paper should be as anonymous as possible.")
			: Open review from overlapping conferences requires Chairs to make ad hoc decisions about whether de-anonymization as part of the review process does or does not violate ACL policies
			: Expectation for transparency at odds with confidential review process (community wants to discuss all aspects of review process in social media)

	~~Maintaining anonymity is becoming increasingly difficult~~		;Higher volume of papers & participants is straining our infrastructure
	~~Open review from overlapping conferences~~		: START tools struggle to support this volume of papers
	~~Expectation for transparency at odds with confidential review process~~		: Reviewer overload/burnout
	Higher volume of papers is straining our infrastructure		: Challenges in coordinating logistics with the venue (A/V, coffee, recruiting lunch, video release forms, random people jumping into banquet buses) in the absence of a Local Chair
	START tools struggle to support this volume of papers
	Reviewer overload/burnout		; Possible solutions
	Look into sharing reviews for rejected papers with next conferences		: Look into sharing reviews for rejected papers with next conferences
	~~Revist~~ using Open Review for *ACL		: Revisit using Open Review for *ACL
	~~Additional lessons learned~~		: Strict policy on double submissions (like EMNLP)

			; Other recommendations
			: Do not print handbooks for all participants, have a smaller number available by request. Post-conference survey indicated that a majority of participants used only the conference app.
			: Have a Local Arrangements Chair for NAACL
			: Revise ACL anonymity and submission policies to remove alternate interpretations and thereby spare PCs time-consuming negotiations with authors
			: Consider moving NAACL to spring so that *ACL timelines are less compressed and NAACL reviewing does not fall over end-of-year holidays
			: More automation of format checks in START & better documentation of the ones that are already there (obscure and buried flags) to ease the desk reject process
			:

Solorio: /* Timeline */

2019-08-06T20:35:52Z

Timeline

← Older revision		Revision as of 20:35, 6 August 2019
Line 443:		Line 443:
	* March 11: Presentation format recommendations		* March 11: Presentation format recommendations
	* March 18: ACs send best reviewers list		* March 18: ACs send best reviewers list
			* March 20-April 8: Best paper selection period

	= Issues and recommendations =		= Issues and recommendations =

Solorio: /* Timeline */

2019-08-06T20:32:47Z

Timeline

← Older revision		Revision as of 20:32, 6 August 2019
Line 442:		Line 442:
	* Feb 22: Decisions sent to authors		* Feb 22: Decisions sent to authors
	* March 11: Presentation format recommendations		* March 11: Presentation format recommendations
	* March 18: ~~Best~~ reviewers ~~(updated)~~		* March 18: ACs send best reviewers list

	= Issues and recommendations =		= Issues and recommendations =

Solorio: /* Timeline */

2019-08-06T20:31:14Z

Timeline

← Older revision		Revision as of 20:31, 6 August 2019
Line 431:		Line 431:

	* Dec. 10th, 2018: Paper submission deadline (both long and short)		* Dec. 10th, 2018: Paper submission deadline (both long and short)
	* Dec. 14-17: ~~[AC]~~ check papers		* Dec. 14-17: Area chairs check papers
	* Dec~~. 18~~-~~19th~~: ~~Paper bidding window~~		* Dec 20-Jan 2, 2019: Paper bidding window
	* Dec 18-Jan 2nd Paper bidding window		* Jan. 3-8: Area chairs review assignment
	* Jan. ~~3rd~~-~~Jan 8th~~: ~~[AC] Review~~ assignment
	* Jan. 9: Review period starts		* Jan. 9: Review period starts
	* Jan. 29: Reviews due (around 3 weeks for reviewing)		* Jan. 29: Reviews due (around 3 weeks for reviewing)
	* Jan. 30-Feb ~~3th~~: ~~[AC] Chase~~ late reviewers add emergency reviewers		* Jan. 30-Feb 3: Area chairs chase late reviewers add emergency reviewers
	* Feb 4th-~~7th~~: ~~[AC] Discussion~~ period		* Feb 4th-7: Area chairs discussion period
	* Feb 8th-12: ~~[AC] Determine~~ recommendations and enter meta reviews		* Feb 8th-12: Area chairs determine recommendations and enter meta reviews
	* Feb 13-21: Final decisions made		* Feb 13-21: Final decisions made
	* Feb 22: ~~decisions~~ sent to authors		* Feb 22: Decisions sent to authors
	* March 11: Presentation format recommendations		* March 11: Presentation format recommendations
	* March 18: Best reviewers (updated)		* March 18: Best reviewers (updated)