2019Q3 Reports: Program Chairs - Revision history

DavidTraum at 18:22, 22 July 2019

2019-07-22T18:22:02Z

← Older revision	Revision as of 18:22, 22 July 2019
Line 1:	Line 1:
		Our report is here: https://www.aclweb.org/adminwiki/index.php?title=File:ACL_Program_Co-Chairs_Report_July_2019.pdf

ShiqiZhao: Blanked the page

2019-07-22T12:58:44Z

Blanked the page

Show changes

ChristyDoran: /* Issues and recommendations */

2019-07-19T19:52:22Z

Issues and recommendations

← Older revision		Revision as of 19:52, 19 July 2019
Line 439:		Line 439:
	Look into sharing reviews for rejected papers with next conferences		Look into sharing reviews for rejected papers with next conferences
	Revist using Open Review for *ACL		Revist using Open Review for *ACL
			Additional lessons learned

ChristyDoran: /* Issues and recommendations */

2019-07-19T19:51:53Z

Issues and recommendations

← Older revision		Revision as of 19:51, 19 July 2019
Line 429:		Line 429:
	= Issues and recommendations =		= Issues and recommendations =

	~~TBD~~		Needs editing

			Maintaining anonymity is becoming increasingly difficult
			Open review from overlapping conferences
			Expectation for transparency at odds with confidential review process
			Higher volume of papers is straining our infrastructure
			START tools struggle to support this volume of papers
			Reviewer overload/burnout
			Look into sharing reviews for rejected papers with next conferences
			Revist using Open Review for *ACL

ChristyDoran: /* Reviewing */

2019-07-19T19:50:03Z

Reviewing

← Older revision		Revision as of 19:50, 19 July 2019
Line 392:		Line 392:
	SRW papers integrated into sessions: positive feedback from participants, better experience for students		SRW papers integrated into sessions: positive feedback from participants, better experience for students
	Did not repeat Test of Time awards from 2018--should this happen every N years to allow for sliding window?		Did not repeat Test of Time awards from 2018--should this happen every N years to allow for sliding window?

			MORE GRAPHICS TO BE ADDED HERE

	= Best paper awards =		= Best paper awards =

ChristyDoran: /* Deciding on the reject-without-review papers */

2019-07-19T19:14:07Z

Deciding on the reject-without-review papers

← Older revision		Revision as of 19:14, 19 July 2019
Line 363:		Line 363:
	This year we followed a two-stage submission process, in which abstracts were due one week before full papers. Our goal was to get a head start on assigning papers to areas, and recruiting additional area chairs where submissions exceeded our predicted volume. Relative to the projected numbers from NAACL-HLT 2018, several areas received a higher-than-predicted number of submissions: Biomedical/Clinical, Dialogue and Vision. Text Mining ended up with the overall largest number of submissions.		This year we followed a two-stage submission process, in which abstracts were due one week before full papers. Our goal was to get a head start on assigning papers to areas, and recruiting additional area chairs where submissions exceeded our predicted volume. Relative to the projected numbers from NAACL-HLT 2018, several areas received a higher-than-predicted number of submissions: Biomedical/Clinical, Dialogue and Vision. Text Mining ended up with the overall largest number of submissions.

	== ~~Deciding on the reject-without-review papers~~ ==		== Handling desk rejects ==
	Our process for identifying desk rejects has been very similar to what other PCs have done in the past. First, the area chairs check their batch of assigned papers and report any issues to us. As the reviewing begins, reviewers may also identify issues that were not caught by ACs, which they flag up to ACs or directly to PCs. We then review each of these issues and make a final decision, to ensure that papers are handled consistently. This means each paper is reviewed for non-content issues by at least three people.		Our process for identifying desk rejects has been very similar to what other PCs have done in the past. First, the area chairs check their batch of assigned papers and report any issues to us. As the reviewing begins, reviewers may also identify issues that were not caught by ACs, which they flag up to ACs or directly to PCs. We then review each of these issues and make a final decision, to ensure that papers are handled consistently. This means each paper is reviewed for non-content issues by at least three people.
	The major categories for desk rejects are:		The major categories for desk rejects are:

ChristyDoran: /* Reviewing */

2019-07-19T19:13:42Z

Reviewing

← Older revision		Revision as of 19:13, 19 July 2019
Line 373:		Line 373:

	== Review process ==		== Review process ==

	Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant we needed 5 parallel tracks to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consists of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers are integrated into the program, and are marked SRW or TACL accordingly.

	~~INSERT CHART~~

	Assignment to areas used the initial START assignments, followed by load-rebalancing and conflict resolution using keywords and manual inspection of the paper. Authors were blind to Area Chairs		Assignment to areas used the initial START assignments, followed by load-rebalancing and conflict resolution using keywords and manual inspection of the paper. Authors were blind to Area Chairs

ChristyDoran: /* An overview of submissions */

2019-07-19T19:13:09Z

An overview of submissions

← Older revision		Revision as of 19:13, 19 July 2019
Line 219:		Line 219:
	* Full papers available for bidding: reviewers loved it, authors did not		* Full papers available for bidding: reviewers loved it, authors did not

	= ~~An overview of submissions~~ =		= Submissions rates and distributions =
	Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.		Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.

ChristyDoran: /* Main Innovations */

2019-07-19T19:11:31Z

Main Innovations

← Older revision		Revision as of 19:11, 19 July 2019
Line 219:		Line 219:
	* Full papers available for bidding: reviewers loved it, authors did not		* Full papers available for bidding: reviewers loved it, authors did not

	== An overview of submissions ==		= An overview of submissions =
	Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.		Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.

ChristyDoran: /* Main Innovations */

2019-07-19T19:10:48Z

Main Innovations

← Older revision		Revision as of 19:10, 19 July 2019
Line 206:		Line 206:
	: Remote presentations were supported for both talks and posters, via an application form to the committee.		: Remote presentations were supported for both talks and posters, via an application form to the committee.

	* ~~Diversity & Inclusion Organization~~		* The new Diversity & Inclusion team piloted a number of new initiatives including:
	The new Diversity & Inclusion team piloted a number of new initiatives including:
	** additional questions on the registration form to identify any accommodations		** additional questions on the registration form to identify any accommodations
	** preferred pronouns (optionally) added to badges		** preferred pronouns (optionally) added to badges
	** I’m hiring/I’m looking for a job/I’m new badge stickers		** I’m hiring/I’m looking for a job/I’m new badge stickers
	~~<bunch of others, pull from their report>?~~		** INSERT LINK TO THEIR REPORT

	* Submissions		* Two-stage Submissions
	:This year we followed a two-stage submission process, in which abstracts were due one week before full papers. Our goal was to get a head start on assigning papers to areas, and recruiting additional area chairs where submissions exceeded our predicted volume.		:This year we followed a two-stage submission process, in which abstracts were due one week before full papers. Our goal was to get a head start on assigning papers to areas, and recruiting additional area chairs where submissions exceeded our predicted volume.
	** Pro: early response to areas with larger than predicted number of papers		:: Pro: early response to areas with larger than predicted number of papers
	** Con: too much overhead for PCs, as authors repeatedly contacted chairs to request that papers be moved between long and short, or asked about changes to authorship, titles and abstracts.		:: Con: too much overhead for PCs, as authors repeatedly contacted chairs to request that papers be moved between long and short, or asked about changes to authorship, titles and abstracts.

	* Full papers available for bidding: reviewers loved it, authors did not		* Full papers available for bidding: reviewers loved it, authors did not

	== An overview of ~~statistics~~ ==		== An overview of submissions ==
	Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.		Authors were permitted to switch format (long/short) when they submitted the full papers, so the total in the chart below uses 2271 as the total number of submissions, discounting the 103 that never submitted a full paper in the second phase. Seventy nine papers were desk-rejected due to anonymity, formatting, or dual-submission violations; 456 papers withdrawn prior to acceptance decisions being sent, although some were withdrawn part way through the review process; and an additional 11 papers were withdrawn after acceptance notifications had been sent. Keeping the acceptance rate consistent with past years meant 5 parallel tracks were needed to fit more papers into 3 days--as the conference grows, decisions will have to be made about continuing to add more tracks, adding more days to the main conference, or lowering the acceptance rate. The overall technical program consisted of 423 main conference papers, plus 9 TACL papers, 23 SRW papers, 28 Industry papers, and 24 demos. The TACL and SRW papers were integrated into the program, and marked SRW or TACL accordingly.

Line 324:		Line 323:
	24 (33)		24 (33)
	24 (25)		24 (25)


	== Conference tracks ==		== Conference tracks ==