2021Q3 Reports: NAACL-HLT 2021 - Revision history

Kristina.toutanova: /* Kristina Toutanova */

2021-07-26T02:37:10Z

Kristina Toutanova

← Older revision		Revision as of 02:37, 26 July 2021
Line 1:		Line 1:

	~~===Kristina Toutanova ===~~

	== Conference summary ==		== Conference summary ==

Kristina.toutanova: /* Virtual Infrastructure */

2021-07-26T02:36:22Z

Virtual Infrastructure

← Older revision		Revision as of 02:36, 26 July 2021
Line 21:		Line 21:
	<p>		<p>
	* Pre-recorded talks based on Screencast-O-Matic. The paper talk submission deadline was about four weeks before the conference. Underline staff checked each submission and communicated with authors to get any issues resolved.		* Pre-recorded talks based on Screencast-O-Matic. The paper talk submission deadline was about four weeks before the conference. Underline staff checked each submission and communicated with authors to get any issues resolved.
	* Website provided by ~~the~~ Underline, including a conference schedule displayed in a local time zone and organized in tracks.		* Website provided by Underline, including a conference schedule displayed in a local time zone and organized in tracks.
	* Posters, discussion sessions, social lounge, and sponsor spaces in Gather.town. The spaces were configured by Underline. Our virtual infrastructure committee only needed to provide maps for the poster sessions.		* Posters, discussion sessions, social lounge, and sponsor spaces in Gather.town. The spaces were configured by Underline. Our virtual infrastructure committee only needed to provide maps for the poster sessions.
	* Groups of papers organized in sessions, together with their recorded videos, poster pdfs, text-based chat, and video call information (Zoom-based), set-up by Underline.		* Groups of papers organized in sessions, together with their recorded videos, poster pdfs, text-based chat, and video call information (Zoom-based), set-up by Underline.

Kristina.toutanova: /* Virtual Infrastructure */

2021-07-26T02:35:10Z

Virtual Infrastructure

← Older revision		Revision as of 02:35, 26 July 2021
Line 26:		Line 26:
	* Plenary talks which were produced and live-streamed by Underline using StreamYard.		* Plenary talks which were produced and live-streamed by Underline using StreamYard.
	* Sponsor booths including chat, video calls, and spaces in Gather.town (depending on sponsorship tier).		* Sponsor booths including chat, video calls, and spaces in Gather.town (depending on sponsorship tier).
			* The Whova conference app was used for asynchronous person-to-person text chat and threaded discussions on channels. Through Whova, 97 meetups were organized, there were 58 discussion channels (with 1,080 posted messages), 77 job postings, and 246 break-the-ice messages.
	* Recordings of the live sessions and processed pre-recorded talks were provided to us after the conference. Recordings of authors that give permission will be uploaded to the ACL Anthology.		* Recordings of the live sessions and processed pre-recorded talks were provided to us after the conference. Recordings of authors that give permission will be uploaded to the ACL Anthology.

Kristina.toutanova: /* Suggestions for process improvements */

2021-07-26T02:28:15Z

Suggestions for process improvements

← Older revision		Revision as of 02:28, 26 July 2021
Line 37:		Line 37:
	* It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.		* It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.

	* We had several instances of session chairs that did not attend their assigned session. We had both a session chair and a volunteer assigned to each session, and in one instance both were missing. It would be good to have emergency on-call session chairs volunteers.		* We had several instances of session chairs that did not attend their assigned session. We had both a session chair and a volunteer assigned to each session, and in one instance both were missing. It would be good to have emergency on-call session chairs and volunteers.

	* Having a slack channel for communication among conference organizers, session chairs, volunteers, and Underline staff during the conference was very useful. We could have set more clear expectations on when people need to be available on that channel.		* Having a slack channel for communication among conference organizers, session chairs, volunteers, and Underline staff during the conference was very useful. We could have set more clear expectations on when people need to be available on that channel.
Line 43:		Line 43:
	* Some workshops did not finalize their schedules till 1-2 days before the conference. This made it very difficult to coordinate and assign volunteers, and to update the website schedule on time. Workshops should be required to set at least their start and end times (program overview) a week or more before their start day.		* Some workshops did not finalize their schedules till 1-2 days before the conference. This made it very difficult to coordinate and assign volunteers, and to update the website schedule on time. Workshops should be required to set at least their start and end times (program overview) a week or more before their start day.

	* NAACL decided to not provide pre-recorded talks to workshop papers, but some workshop organizers asked their authors to submit pre-recorded talks. The NAACL 2021 workshop co-chairs did a great job and worked very hard to coordinate the diverse workshop requirements. To make the process more efficient, ~~automated checks~~ to ~~make sure the policy is followed~~ would be ~~helpful~~.		* NAACL decided to not provide pre-recorded talks to workshop papers, but some workshop organizers asked their authors to submit pre-recorded talks. The NAACL 2021 workshop co-chairs did a great job and worked very hard to coordinate the diverse workshop requirements. To make the process more efficient, automation to gather requirements and enforce policies would be useful.

Kristina.toutanova: /* Suggestions for process improvements */

2021-07-26T02:25:49Z

Suggestions for process improvements

← Older revision		Revision as of 02:25, 26 July 2021
Line 33:		Line 33:
	* There were data errors on dozens of papers -- missing authors, wrong titles etc. Perhaps it would be possible to specify the metadata automatically though download from the the review system (e.g. START).		* There were data errors on dozens of papers -- missing authors, wrong titles etc. Perhaps it would be possible to specify the metadata automatically though download from the the review system (e.g. START).

	* If configuration of posters in Gather.Town can be programmatic instead of manual, inconsistencies and minute missing posters could perhaps be avoided.		* If configuration of posters in Gather.Town can be programmatic instead of manual, inconsistencies and last minute missing posters could perhaps be avoided.

	* It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.		* It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.

			* We had several instances of session chairs that did not attend their assigned session. We had both a session chair and a volunteer assigned to each session, and in one instance both were missing. It would be good to have emergency on-call session chairs volunteers.

			* Having a slack channel for communication among conference organizers, session chairs, volunteers, and Underline staff during the conference was very useful. We could have set more clear expectations on when people need to be available on that channel.

			* Some workshops did not finalize their schedules till 1-2 days before the conference. This made it very difficult to coordinate and assign volunteers, and to update the website schedule on time. Workshops should be required to set at least their start and end times (program overview) a week or more before their start day.

			* NAACL decided to not provide pre-recorded talks to workshop papers, but some workshop organizers asked their authors to submit pre-recorded talks. The NAACL 2021 workshop co-chairs did a great job and worked very hard to coordinate the diverse workshop requirements. To make the process more efficient, automated checks to make sure the policy is followed would be helpful.

Kristina.toutanova at 02:08, 26 July 2021

2021-07-26T02:08:48Z

← Older revision		Revision as of 02:08, 26 July 2021
Line 13:		Line 13:
	* The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/. The full schedule of social events the committee organized can be seen at https://2021.naacl.org/program/social/; recordings of some of the discussions are available at https://www.youtube.com/channel/UCbdq7M48OgNoPIa9Uj_IBCw.		* The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/. The full schedule of social events the committee organized can be seen at https://2021.naacl.org/program/social/; recordings of some of the discussions are available at https://www.youtube.com/channel/UCbdq7M48OgNoPIa9Uj_IBCw.
	* The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).		* The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).

			== Virtual Infrastructure ==

			We sought a virtual infrastructure solution that would enable pre-recorded talks, interactive poster sessions, live group video calls, text-based asynchronous chat, and live-streaming of plenary sessions. Based on a small scale survey prior to the conference, a large majority of potential attendees and authors were interested in being able to interact in small groups around posters, and it was important to us that we provide that. Other considerations were having flexible video recording software, and sufficient technical support for live presentations.

			The components of the virtual infrastructure were:
			<p>
			* Pre-recorded talks based on Screencast-O-Matic. The paper talk submission deadline was about four weeks before the conference. Underline staff checked each submission and communicated with authors to get any issues resolved.
			* Website provided by the Underline, including a conference schedule displayed in a local time zone and organized in tracks.
			* Posters, discussion sessions, social lounge, and sponsor spaces in Gather.town. The spaces were configured by Underline. Our virtual infrastructure committee only needed to provide maps for the poster sessions.
			* Groups of papers organized in sessions, together with their recorded videos, poster pdfs, text-based chat, and video call information (Zoom-based), set-up by Underline.
			* Plenary talks which were produced and live-streamed by Underline using StreamYard.
			* Sponsor booths including chat, video calls, and spaces in Gather.town (depending on sponsorship tier).
			* Recordings of the live sessions and processed pre-recorded talks were provided to us after the conference. Recordings of authors that give permission will be uploaded to the ACL Anthology.

			We had a positive experience working with Underline as they handled all aspects of the infrastructure and were also very open to our requests to customize the experience (provided changes were feasible in the given timeframe).

			=== Suggestions for process improvements ===
			* There were data errors on dozens of papers -- missing authors, wrong titles etc. Perhaps it would be possible to specify the metadata automatically though download from the the review system (e.g. START).

			* If configuration of posters in Gather.Town can be programmatic instead of manual, inconsistencies and minute missing posters could perhaps be avoided.

			* It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.

Kristina.toutanova: /* Main Innovations */

2021-07-26T01:38:17Z

Main Innovations

← Older revision		Revision as of 01:38, 26 July 2021
Line 11:		Line 11:
	* We followed a more well-defined process for ethics reviews, proposed by the ethics co-chairs Emily Bender and Karën Fort, and refined and implemented in coordination with the program co-chairs and the demo and industry track chairs. Key changes were allowing additional space in submissions to discuss ethical considerations and establishing a category of papers accepted conditionally on addressing ethical concerns together with a timeline and process for an additional stage of review of re-submissions. See the reports of the ethics chairs (https://2021.naacl.org/blog/ethics-review-process-report-back/) and the program co-chairs (https://2021.naacl.org/blog/ethics-review-process/) for more details.		* We followed a more well-defined process for ethics reviews, proposed by the ethics co-chairs Emily Bender and Karën Fort, and refined and implemented in coordination with the program co-chairs and the demo and industry track chairs. Key changes were allowing additional space in submissions to discuss ethical considerations and establishing a category of papers accepted conditionally on addressing ethical concerns together with a timeline and process for an additional stage of review of re-submissions. See the reports of the ethics chairs (https://2021.naacl.org/blog/ethics-review-process-report-back/) and the program co-chairs (https://2021.naacl.org/blog/ethics-review-process/) for more details.
	* There was no distinction between oral and poster presentation at the main conference, and all papers were presented in both assigned video calls for groups of five-six papers, and in poster sessions in Gather.town.		* There was no distinction between oral and poster presentation at the main conference, and all papers were presented in both assigned video calls for groups of five-six papers, and in poster sessions in Gather.town.
	* The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/.		* The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/. The full schedule of social events the committee organized can be seen at https://2021.naacl.org/program/social/; recordings of some of the discussions are available at https://www.youtube.com/channel/UCbdq7M48OgNoPIa9Uj_IBCw.
	* The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).		* The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).

Kristina.toutanova: Created page with " ===Kristina Toutanova === == Conference summary == *We originally planned an in-person NAACL 2021 but switched to a fully virtual format (the decision was finalized less tha..."

2021-07-26T01:31:51Z

Created page with " ===Kristina Toutanova === == Conference summary == *We originally planned an in-person NAACL 2021 but switched to a fully virtual format (the decision was finalized less tha..."

New page

===Kristina Toutanova ===

== Conference summary ==
*We originally planned an in-person NAACL 2021 but switched to a fully virtual format (the decision was finalized less than three months before the conference).
*The NAACL virtual conference hosted 6 tutorials, 17 system demonstrations, 39 industry track papers, 499 main conference and CL/TACL papers, 6 plenary invited talks (4 main track and two industry), 2 panels (Startups in NLP and Careers in NLP), 22 workshops, and a large set of social and thematic gatherings. More details on the conference structure are at https://2021.naacl.org/blog/conference-structure/.
*2,622 participants registered for NAACL. We reduced registration rates further compared to recent virtual conferences: student registration was $25 early/ $50 late, and regular registration was $125 early/ $175 late.
*We chose Underline (https://underline.io/events/122/reception) for our virtual infrastructure, following the experience of COLING 2020. We also used the Whova conference app for text-based chat and based the review process on START.

== Main Innovations ==
* We followed a more well-defined process for ethics reviews, proposed by the ethics co-chairs Emily Bender and Karën Fort, and refined and implemented in coordination with the program co-chairs and the demo and industry track chairs. Key changes were allowing additional space in submissions to discuss ethical considerations and establishing a category of papers accepted conditionally on addressing ethical concerns together with a timeline and process for an additional stage of review of re-submissions. See the reports of the ethics chairs (https://2021.naacl.org/blog/ethics-review-process-report-back/) and the program co-chairs (https://2021.naacl.org/blog/ethics-review-process/) for more details.
* There was no distinction between oral and poster presentation at the main conference, and all papers were presented in both assigned video calls for groups of five-six papers, and in poster sessions in Gather.town.
* The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/.
* The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).