Admin Wiki - User contributions [en]

2020Q3 Reports: Virtual Infrastructure Chairs

2020-07-21T14:08:31Z

Raosudha89: /* Workshops */

The virtual conference infrastructure of ACL2020 consists of 4 major components:
* SlidesLive for pre-recorded talks & video live-streaming
* Zoom for live QA sessions
* RocketChat for async text-based discussions and announcements
* the [https://virtual.acl2020.org virtual conference website] as the central platform to present all information about the conference
Due to limited time, we follow the infrastructure setup of ICLR2020, with some necessary extensions including the plenary sessions, tutorials, virtual sponsor booths, etc. In the remaining sections of this report, we provide a high-level summary of each component and discuss potential improvement can be made in future. At the end, we provide some general advice for future virtual conferences based on our experience.

== SlidesLive ==

SlidesLive is a company that has helped with several previous conferences including ICLR, ICML, etc for recording the videos of the talks, post processing them and putting the slides and the presenter side by side. We initially negotiated with several companies including them for the physical conference in Seattle. The companies we discussed with with their Point Of Contact (POC) were:
* SlidesLive, POC: Katherine (Katie) Millar, katherine@slideslive.com
* Underline, POC: Sol Rosenberg, sol@underline.io
* GloCastLive, POC: Arran Moffat, arran@glocast.com
* BASH FILMS, POC: Mark Bashian, mbashian@bashfilms.com
* Freeman, POC: Brian Holm, Brian.Holm@freemanco.com
We have put all the information we collected from these companies including the quotes [https://drive.google.com/drive/folders/1KYzLT-B5ZVIb3n_KDSRlMQnahoTuxxei?usp=sharing here] for your reference. Please note that these are for the time when the conference was supposed to be physical, nevertheless many of them also offer virtual experience.

After deciding to have the conference fully virtual we signed a contract with SlidesLive and relied on them for our keynotes and workshops besides the post processing of the videos. Below we list “our experience” working with them, noting these before signing a contract with them should be very helpful to have a great experience working with them, if you choose them as the company you want to work with.

=== General suggestions ===
Here we summarize our general suggestions, especially before signing a contract with them.

* We strongly suggest enforcing an early deadline for a finalized program of conference and '''specially workshops'''. This is very important as it helps to reduce communication overhead between SlidesLive and the rest of the conference.
* It is very important to clarify with SlidesLive “before signing the contract” how many live zoom QA sessions they are going to cover. For example, they did not cover “any” of the QA sessions for papers in main conference, demo sessions, sponsors or D&I events, and it was not clearly communicated to us pre-signing the contract. They also did not cover any of the parallel QA sessions happening in our workshops. We ended up creating our own zoom team to handle these. To prevent this from happening we suggest asking them to include a technical staff as well during discussions, marketing representatives might not be aware of these types of details.
* Slideslive does its best to be responsive and usually tries to answer our questions as much as they can. We recommend asking them to assign more number of “technical staff” especially during the conference. We sometimes felt having only two contacts from them is absolutely not enough especially given the fact that one of them was not as technically confident as the other one.
* There was a 8 sec delay in live presentations (the video in zoom vs the one streaming on the SlidesLive website) which our general chair (Dan) brought up during SlidesLive demo session. Please request them to resolve this.

=== Accessibility ===
Please note that they only provide captioning for the videos / live sessions they are handling, i.e., if you end up having to create your own zoom meetings they are not going to provide captioning for them. Another note, when we made the contract with Slideslive, it wasn’t clear in the contract that they only provided automatic captioning service instead of high quality human-edited captioning. Please make sure to go through all these details before signing a contract with them.

=== Non-profit discount ===
SlidesLive provides a 25% discount for Non-Profit Organizations, please make sure to ask for this if you are eligible to use it.

=== SlidesLive demo ===
We “strongly” advise to request a demo session from them “before signing the contract”. You can use part of the ACL 2020 schedule as an example if you do not have the full conference program yet. This is very helpful for you to get a sense of the experience you will have and gives you an opportunity to include your requests from them quite early on. It might take for them some time to include the features you need for your conference and this will give them a reasonable amount of time to address your requests in their framework pre-conference.

== Zoom ==

=== General suggestions ===

* (repeating as it is quite important) We strongly suggest enforcing an early deadline for a finalized program of conference and specially workshops. This is very important as it helps to reduce communication overhead for creating all virtual sessions beforehand.
* We strongly suggest generating Zoom meetings well ahead of time to make sure you have enough time to debug and verify them. We have provided detailed documentation on how to create bulk set of meetings in zoom in [https://docs.google.com/document/d/1EYELyo5Yp3HESllBlqSis4aoAoxU6LQ0sRBwFAc5gMI/edit?usp=sharing this documentation]. We also have provided another step by step guide for users (authors/hosts of the meetings) [https://docs.google.com/document/d/1OWCcxXm9mL7DgcSWgW0FC87KmOEq8lIU3CwcsMEPLiI/edit?usp=sharing here].
* You need to have several volunteers during the conference who are specifically trained for zoom to be on call during the conference in case any incident happens. We recommend at least 2 volunteers per time slot. Note that only these volunteers should have access to your master zoom account.
* Verifying separate accounts need to be crowdsourced pre-conference (it is a pretty quick process), we asked volunteers to do it pre-conference.
* We strongly advise to have a contact person for zoom meetings of each part of conference, e.g., having a couple of POCs for main papers track, demos track, sponsors, etc ahead of time. This will reduce the amount of communication overhead a lot.
We suggest having a separate team for zoom meetings of workshops, there are usually several workshops with a much more diverse set of programs (it is like a mini conference by itself).
* '''If a meeting got hacked in progress''', on-call volunteers should delete the meeting link and create a new link and announce it at rocketchat immediately.
* '''Each meeting should have a unique main host.''' If you are ‘host’ for a meeting and start that meeting, and then hop off and try to start another one, you will get an error message from Zoom stating that you already have a meeting in progress and you need to end the previous one in order to start this one. If you click yes, ZOOM WILL END THE PREVIOUS MEETING AND KICK EVERYONE OFF.
* Attendees can join a meeting on browser '''only when''' the meeting is not requiring registration.
* '''Do not share the master account''' outside of the virtual infrastructure team.
* Note that it is likely that '''API credentials will expire''' after a certain amount of time. If this happens, you need to follow the instruction to regenerate it. (Before you run the script to create Zoom Meeting, you need to login to the zoom account and create JWT credentials in Zoom App Marketplace.)
* Virtual infrastructure team can maintain a spreadsheet to store all the credentials of authors, so on-call volunteers will be able to access every meeting and offer help for all the meetings.
* Paid accounts: you can pay Zoom for a business level account and '''add phone dial-in plans'''. This is a good idea for anyone who may have a spotty internet, so that they can dial in via phone/audio as well as video -- that way, they will always have audio connection for Q&A and it won’t drop them.
* '''When a meeting link is broken''' during the conference, on-call volunteers can login ACL2020 Master Zoom account, and select a specific user from “User Management” -> “Users”, and then create a new meeting for that user (the user will be the host).

=== Zoom attendees statistics ===

We made an analysis of Zoom attendees statistics for the main conference Q&A Session. You can also find more detail and the script in the folder below [https://drive.google.com/drive/folders/1WFU0oml-XUKNzuL_YOBnBJ8wX7BRHJrR?usp=sharing here].

* Total # of Unique Participants in Each Q&A Session:

* Distribution of participants over time 20 minutes after start
* 35 minutes after start
* 50 minutes after start

== RocketChat ==
RocketChat is a platform (similar to Slack) that provides async text-based chat functionality. In our workspace (acl2020.rocket.chat), we can create channels for different purposes.

=== Channels ===
Here we provide a summary of channels we have created for ACL2020.

* '''#announcements''': This is a read-only channel which is used by organizers to make conference-wide announcements. Only authorized users can post messages in this channel. Other users can only react with emoji to individual threads.

* '''#general''': This channel is used to discuss general conference topics. The conversation can be about anything related to ACL2020. During the conference, this channel was used to promote their live sessions by authors, workshop organizers, social events organizers, etc.

* '''#live''': This channel is only used for live plenary sessions. It was read-only for the majority of the time. During the live plenary sessions, we lift the read-only setting and people can use this channel for live QA. People have also used this channel to congratulate award winners, etc.

* '''#firsttimers/#introduction''': We initially only created the #firsttimers for people who attend ACL for the first time to say Hi to each other. At the very beginning of the conference, someone created the #introduction channel and many people joined that channel instead to introduce themselves.

* '''#helpdesk''': This channel was used by all ACL2020 attendees to request tech support for the ACL2020 virtual conference. We ask people to prefix their questions with a topic, e.g., [Zoom], [RocketChat], etc. On-call volunteers always start a thread when responding to a question. And continue the conversation in the thread. Once the question is resolved, they will mark it with :checkmark: emoji. See the screenshot below for a demonstration.

* '''#presenter-helpdesk''': This channel is same as #helpdesk, except that it is dedicated to authors and sponsors. It turns out most attendee still ask questions in #helpdesk so this channel may not be necessary in future.

* '''#incidents''': This channel was used to report incidents in the conference. Conference attendees can request the on-call volunteer in the channel to join your Zoom meetings as a co-host to handle disruptions or delete messages in RocketChat. They can simply click “Reply” button to start a private conversation with the on-call volunteer.

* '''#professional-conduct-committee''': This channel was used to contact the Professional Conduct Committee if you need our assistance (e.g. related to a concern that might fall under the anti-harassment policy). PCC members will post a message in this channel to let people know which of them are available. Then the attendee can contact the PCC member via RocketChat private message (similar to the #incidents).

* '''#social-media-posts''': This channel contains cross-posted social media posts published on various platforms by our official live tweeters and microbloggers, managed by our publicity chairs.

* '''#new-channels''': This channel was used to announce new channels made by conference attendees. It turns out many people have created different channels during the conference, including some topic-based channels (eg., #bio-clinical-nlp), university-based channels (e.g., #uwnlp) to get to know different research groups, location-base channels (e.g., #singapore), etc.

* '''#job-listing''': This channel was used to post job opportunities. We only get 10 posts in this channel throughout the conference though.

* '''#oncall-volunteers''': This channel is dedicated to on-call volunteers for them to coordinate with each other during the conference.

* '''#sponsor-xxx''': For each sponsor, we create a dedicated channel for them to post announcements and interact with conference attendees. These are “broadcast” channels (similar to #incidents, #professional-conduct-committee), i.e., only the channel owner can post messages. Visitors can only start a private conversation with the message author. This setting is due to the privacy concerns -- we do not want to expose who has visited a specific sponsor booth.

* '''#sponsor-registration-desk''': This channel is used as the virtual registration desk for sponsors. We need this channel because we need to set the sponsor account as the owner of their channels, and we didn’t develop an automatic way to do this. Since there are only ~20 sponsors, we just ask them to post a message in this channel when they arrive, and we just manually add them to the #sponsor-xxx channel and set them as the owner. After we add the sponsor account to their channel, we post a message “@username Welcome! I have set you as the owner of the channel. You can delete this message with your owner’s permission” so that they can familiarize themselves with the RocketChat usage.

* '''#organizers-registration-desk''': Similar to #sponsor-registration-desk, this channel is dedicated to organizers who need to be the owner of specific channels.

* '''#paper-xxx''': For each paper, we create a channel where the author can interact with visitors to answer questions.

* '''#keynote-xxx/#workshop-xxx/#tutorial-xxx''': For each keynote/workshop/tutorial, we create a channel where attendees can interact with the keynote speaker and the workshop/tutorial organizers.

* '''#business-meeting/#review-meeting''': These two channels are used for discussions about the business meeting and the review meeting, respectively.

=== Other considerations ===

* Emoji: We figure this is a way to attract more engagement from conference attendees. We uploaded the ACL logo, together with some other emoji. The ACL logo is a must-have one as we (organizers) usually by default add this emoji reaction to an announcement.
* Bad word filtering: We initially enabled the default bad word filtering in RocketChat during the internal test. However, it turns out it blocks many words such as “queer”, “gay”, etc., and we disable this feature during the conference. It seems to be okay as we didn’t receive any report about profanity in RocketChat.
* Permission to @here and @all: We initially allow all users to @here (mention all online users in the channel) and @all (mention all users in the channel). But it turns out this makes the space very noisy as people have extensively used @here/@all to announce their QA sessions. Therefore, we disable this permission and only allow channel owners to @here/@all.

=== Other platforms ===
* Slack might be an alternative to RocketChat, which has a better UI. But it is more expensive. Also, we don’t have the scripts to batch create channels for 800+ papers. Furthermore, we didn’t have time to investigate whether there are some specific features (e.g., broadcast channels) in RocketChat that are not available in Slack. In particular, we need to embed the channel as an iframe on a webpage, which seems to be unsupported natively by Slack.
* [https://www.dory.app/ Dory.app] and [https://www.sli.do/ Slido] could be used in addition to RocketChat for collecting questions & votes. This is perhaps one of the major limitations of RocketChat, i.e., it is not very ergonomic to collect questions/votes, partially because all messages in a thread are also shown in the channel with a smaller font, making it hard to read. AKBC2020 has used Slido, and one of the tutorials in ACL2020 has used Dory.app. Both seem to work well in their scenarios. However, we have tried Dory.app for the live plenary sessions on July 7, and it turns out that their server cannot handle the large amount of traffic and it crashed during the conference. So we have to switch back to the #live channel for collecting questions.

=== Recommendations ===
In general, we have relatively good experience with RocketChat. It does have some issues but the support team is very responsive, so I would recommend it for future conferences. The Slack might be a better alternative if it can meet the necessary requirements.
I would highly recommend finding a better platform for collecting questions & votes to augment RocketChat. The co-founder of Dory.app (Ena ena.zheng@dory-4098177ab9b4.intercom-mail.com) has reached out to me and she explained the situation. They can easily increase the capacity of their server/database for a specific period if we can provide them an early notice. They are very happy to support the conference, so the EMNLP2020 organizers can directly reach out to Ena to discuss further details.

== Virtual Conference Website ==
The [https://virtual.acl2020.org virtual conference website] is the central platform to present all information about the conference. As pointed out by ICLR2020 organizers, a portal bringing together all the different tools needed for the virtual conference was not available, and they have developed their own website. Thus, we adopted their codebase Mini-Conf https://github.com/Mini-Conf/Mini-Conf and made some extensions in our own fork https://github.com/acl-org/acl-2020-virtual-conference to meet our specific needs.
After the conference, we have removed all RocketChat channels from the website, but you can check out the acl2020-v7.0 release of the repo and spin up the website locally to see the version we used during the conference. We also have an introduction [https://slideslive.com/38931552/acl-website-walkthrough video] that explains how to use the website.

=== Overall design ===
The website has 12 top-level pages.
* '''Home''': We explain how to use the website and embed the #announcement channel on this page.
* '''Schedule''': This page shows a calendar view of all events during the conference.
* '''Livestream''': This page is used for the livestream of plenary sessions.
* '''Plenary''': This page contains the information about plenary sessions. Each plenary session also has its own page to display the pre-recorded videos and optionally an associated RocketChat channel.
* '''Papers''': This page displays all main conference papers (including the demo papers and student research workshop papers). Users can browse papers by track, search by author/title, and use the visualization tool to find a cluster of papers. Each paper also has its own page, where we display the detailed information about the paper, the live session slots, the pre-recorded video, and the associated RocketChat channel.
* '''Tutorials''': This page displays all tutorials on July 5, using a calendar view. Each tutorial also has its own page.
* '''Workshops''': This page displays all workshops, and each workshop has its own webpage.
* '''Sponsors''': This page displays all sponsors by their levels. Most sponsors have their own dedicated page to display their video, downloadable content, contact, and a RocketChat channel #sponsor-xxx described in Section 3.
* '''Socials''': This page displays all social events, but each event does not have a dedicated page.
* '''Chat''': Here we simply embed the RocketChat workspace.
* '''Organizers''': Here we display all conference organizers with their photos.
* '''Help''': Here we show the FAQ and Code of Conduct.

=== Tutorials ===
* We found that each tutorial is different. It would be useful to consult with each tutorial organizer beforehand to understand their needs and customize their virtual tutorial page accordingly. This year we had a common template for all tutorials but in hindsight it could have been slightly different for each.
* Some tutorial teachers did not know that tutorial videos can be recorded in parts (using SlidesLive) and had to record their 2+ hours of videos in one go. We should let tutorial teachers know beforehand that video can be recorded in parts.
* Attendees were not clear on what was expected of them before the tutorial. It would have been useful if each tutorial teacher explained what preparation is required in this virtual setup.
* The tutorial chairs posted a great blog on different modalities of tutorials - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It might have been more useful if posted even earlier.

=== Workshops ===
* There should be a separate team of workshop-virtual-chairs dedicated to only thinking about virtual infrastructure needs for workshops. With 20 workshops, some as big as a mini-conference, it becomes important to have someone think about the workshop infrastructure explicitly.
* Each workshop paper should have a page of its own (similar to main conference papers) with link to paper, pre-recorded talk, rocketchat and zoom link for Q&A
* With the new virtual format, each workshop should be asked to write a paragraph explaining their virtual setup. There was a lot of confusion this year since each workshop had a different setup. The paragraph should include
** Whether they have parallel Q&A session on different zoom links
** Whether the pre-recorded talks will be played again on the live stream OR do they expect attendees to watch them beforehand.
** If the pre-recorded talks are going to be played in the livestream, they should ask attendees to stay on the website for the talk part and enter the zoom meeting only after the talk for the Q&A session. This year, many attendees were waiting in the zoom room thinking the pre-recording talks can be seen there as well (not knowing that they should go to the website to watch it)
**Like the main conference, perhaps the keynote of workshops should also allow questions only on rocketchat. Because some workshop organizers had trouble fielding questions from different sources (zoom chat, zoom hand raising, rocketchat channel)
* It seems like workshops that were single track worked better than those that had multiple tracks and this was because with parallel poster sessions, it was confusing how to go to each paper Q&A session separately.
* Workshops organizers should be given a deadline after which they should not be able to update their schedule. Last minute schedule updates this year caused SlidesLive delay in preparing their livestream session.
* Setting such a deadline will also help infra chairs to create all zoom meetings using individual zoom accounts separately as they did for main conference papers.
* The main zoom meeting should have a dedicated moderator who can take care of assigning speakers to be co-hosts.
* Each workshop should have a separate read only rocketchat channel (similar to main conference #announcement channel) where organizers can make important announcements. This year, with a common channel for all communication, the announcements from organizers were getting drowned within other chat.
* Many attendees had questions about what material will be available before/during/after the conference. The tutorial chairs posted a great blog on this - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It would be great if workshop chairs did something similarly.

=== Virtual booths for sponsors ===
* We already have a pretty good template for virtual booths. The hardest part is to communicate with the sponsors and collect information from them.
* In ACL 2020, we only communicate with the sponsors through email. It’s very hard to keep track of the changes sponsors make and sponsors always want to change information at the last minute. It would be great if we could automate some information collection process in the future. Here are some suggestions for future reference:
** Sponsors always want to change Zoom schedules at the last minute, and it would be inconvenient for them to email us every time they change their schedule. So, it would be great if we could ask them to input the information into a shared spreadsheet, so that they can change their schedule by modifying the spreadsheet. We can write a code to load the spreadsheet to the yml file so that we only need to do a refresh before the main conference, which will significantly reduce our effort.
** For all the files and resources, it’s better to ask sponsors to provide us a static url link to the file, so that as long as the link does not change, they can update their file anytime they want. We can also ask sponsors to put the <filename, url link> pair into a shared spreadsheet so that we don’t need to manually input the information based on emails.
** For some files, e.g. logos and pdfs, we can create a shared Google drive for them to update their logos. Again in this ACL we send all files through email which is very inconvenient.
* We should give sponsors a hard deadline on when they can not update the information. For this ACL, we got emails even after the main conference where the sponsors want to change schedules/materials in the virtual booth.
* This year we included the sponsors in the internal dry period, which turns out to be necessary since the sponsors will be able to check and test their virtual booths before the main conference. It’s very important since we can do one round of iteration to fix the information in the virtual booth based on the sponsors’ feedback before the main conference.
* This year we have 19 sponsors and only 2 volunteers to build the virtual booth, so each volunteer needs to build 9-10 sponsors. The workload is too heavy for each of the volunteers (each of them take around 4 days full time). I think the best balance would be 4-5 sponsors per volunteer. There should be one more volunteer that is available during the main conference to fix the website if there are urgent requests from the sponsors.
* We created one admin account for each sponsor that’s valid through the end of the conference, and a test visitor account that’s valid before the conference so that the sponsors can test the interaction between visitors and sponsors (especially on RocketChat). This is very important for the sponsors to learn how to make announcements and how to monitor the channel when visitors send questions.
* It’s important to provide sponsors with a step-by-step instruction on how to login, how to use Zoom, and how to use RocketChat. The direct contact for each module of the virtual booth should also be listed in the instruction, e.g. provide contact for login related issues, Zoom related issues, RocketChat related issues and wrong information in the virtual booth. This way we can significantly reduce the number of repeated questions from sponsors and save a lot of time. Link to the instruction this year: https://docs.google.com/document/d/1paG7hY7E9qKEBZrXZloyfnI3Uxt_723J23JNWBKOMCI/edit
* We should also decide on a date to stop getting new sponsors. It’s not realistic to build new virtual booths at the last minute (which happened a few times for ACL 2020).
* Double check the sponsors’ list with the list on the external website in case there are missed sponsors.

=== User login ===
* We basically follow [https://aws.amazon.com/blogs/networking-and-content-delivery/authorizationedge-using-cookies-protect-your-amazon-cloudfront-content-from-being-downloaded-by-unauthenticated-users/ this AWS solution] to do the login authentication. But we have used a slightly different version [https://github.com/opaquejacob/cloudfront-authorization-at-edge.git here] so we can use the customized auth domain https://signin.acl2020.org. See [https://github.com/aws-samples/cloudfront-authorization-at-edge/compare/master...opaquejacob:master here] for the changes from the original version in the blog post.
* This solution allows us to use OAuth to implement the sign-on for the website and RocketChat. See [https://github.com/acl-org/acl-2020-virtual-conference/issues/53 here] for instructions.
* By default, the AWS Cognito pool can only send 50 emails per day. Need to follow [https://github.com/acl-org/acl-2020-virtual-conference/issues/53#issuecomment-649327363 this instruction] to increase the limit and [https://github.com/acl-org/acl-2020-virtual-conference/issues/104#issuecomment-653592927 this instruction] to switch to use the Amazon SES.
* For people who registered before July 3, we create the accounts on July 3 using the excel file provided by Prisicilla.
* To handle registration during the conference, we use Zapier + AWS Lambda. See https://github.com/acl-org/acl-2020-virtual-conference/issues/55#issue-636817449 for more details.

Although this solution works reasonably well, it does have some issues. Actually, 95% questions sent to acl2020-virtual-helpdesk@googlegroups.com are about login problems, and we probably have handled around 200 such requests.
* The AWS Cognito sent out login emails on behalf of acl2020virtual@gmail.com. It turns out the emails are easily classified as spam. Also some university email server blocks it (e.g., we found consistent issues with @xxx.i.kyoto-u.ac.jp).
* Many people could not login because they have entered the wrong password. We have therefore updated several versions of our instruction https://acl2020.org/_pages/docs/ACL2020_virtual_website_login_steps.pdf with some common errors (see Step 1), which seems to be helpful.
* We announce that our website will be live on July 3, and we send out the login credentials on that day. It turns out to be a not very wise decision because we have to deal with all incoming inquiries about login issues without volunteers on July 3 & 4. (Our on-call volunteers can only start working on July 5).
* Some authors are confused with the account used for the website and the SoftConf account.
* For some unknown reason, there are many users who have registered but were not included in the excel files sent from Priscilla. We may need to check with the company who hosted the registration form about this.
* It would be helpful if we can have the registration company directly handle the login authentication, so that we don’t need to have two separate processes to create user accounts.

== General Advice for Future Virtual Conferences ==
* It would be nice if future conference organizers can find a vendor company to run the virtual conference if possible. Otherwise, the virtual infrastructure committee would definitely need much more people and figure out a way to distribute the work more. Since all organizer committee members are officially volunteers, we cannot rely on people to work full-time or even extra hours on this (which we have to do for ACL2020 given the limited resources and short timeline). Here is a potential organization of the virtual infrastructure committee.
** Pre-recorded videos & livestreaming (1-2 chairs + a vendor company)
** Zoom meetings (1-2 chairs + 5-8 volunteers)
** Captioning (1-2 chairs)
** RocketChat (1 chair + 3-4 code development volunteers)
** Main website development (1-2 chairs + N code development volunteers)
** Virtual sponsor booth (1 chair + 1 code development volunteers per 5 sponsors)
** Tutorials (1 chair + 2-3 code development volunteers)
** Workshops (probably better to have their own virtual infrastructure chairs to customize their webpages)
** Login authentication (1 chair + 2 code development volunteers + 5 helpdesk volunteers; or use a vendor company)
* The repo https://github.com/acl-org/acl-2020-virtual-conference should be a good start for future conferences. Due to the time limit, we didn’t maintain a very high code quality but it should not be very hard to spin up a new website. The most challenging part would be gathering all sitedata, including the paper basic information, QA session schedule, Zoom links, video URLs, etc.
* It would be useful to have a better pipeline for information collection. Currently we have to gather information from different chairs. It would be better if the website, the proceedings, the handbook, and the ACL Anthology all use the same information source such that everything is synchronized.
* GitHub Issue seems to be a good way to manage the tasks and distribute work to volunteers.
* It would be better to have volunteers to be involved earlier, and work with the volunteer coordinator on the timeline. It was not a very smooth transition because most volunteers can work either before or during the conference, not both. And there is a period that our chairs have to handle all emails in acl2020-virtual-helpdesk@googlegroups.com on our own on the first two days when there are lots of inquiries.
* There is many other useful information in the article [https://medium.com/@iclr_conf/gone-virtual-lessons-from-iclr2020-1743ce6164a3 “Gone Virtual: Lessons from ICLR2020"]. The section “Technical Systems and Support” is quite relevant to the virtual conference infrastructure.

2020Q3 Reports: Virtual Infrastructure Chairs

2020-07-21T14:06:25Z

Raosudha89: /* Workshops */

The virtual conference infrastructure of ACL2020 consists of 4 major components:
* SlidesLive for pre-recorded talks & video live-streaming
* Zoom for live QA sessions
* RocketChat for async text-based discussions and announcements
* the [https://virtual.acl2020.org virtual conference website] as the central platform to present all information about the conference
Due to limited time, we follow the infrastructure setup of ICLR2020, with some necessary extensions including the plenary sessions, tutorials, virtual sponsor booths, etc. In the remaining sections of this report, we provide a high-level summary of each component and discuss potential improvement can be made in future. At the end, we provide some general advice for future virtual conferences based on our experience.

== SlidesLive ==

SlidesLive is a company that has helped with several previous conferences including ICLR, ICML, etc for recording the videos of the talks, post processing them and putting the slides and the presenter side by side. We initially negotiated with several companies including them for the physical conference in Seattle. The companies we discussed with with their Point Of Contact (POC) were:
* SlidesLive, POC: Katherine (Katie) Millar, katherine@slideslive.com
* Underline, POC: Sol Rosenberg, sol@underline.io
* GloCastLive, POC: Arran Moffat, arran@glocast.com
* BASH FILMS, POC: Mark Bashian, mbashian@bashfilms.com
* Freeman, POC: Brian Holm, Brian.Holm@freemanco.com
We have put all the information we collected from these companies including the quotes [https://drive.google.com/drive/folders/1KYzLT-B5ZVIb3n_KDSRlMQnahoTuxxei?usp=sharing here] for your reference. Please note that these are for the time when the conference was supposed to be physical, nevertheless many of them also offer virtual experience.

After deciding to have the conference fully virtual we signed a contract with SlidesLive and relied on them for our keynotes and workshops besides the post processing of the videos. Below we list “our experience” working with them, noting these before signing a contract with them should be very helpful to have a great experience working with them, if you choose them as the company you want to work with.

=== General suggestions ===
Here we summarize our general suggestions, especially before signing a contract with them.

* We strongly suggest enforcing an early deadline for a finalized program of conference and '''specially workshops'''. This is very important as it helps to reduce communication overhead between SlidesLive and the rest of the conference.
* It is very important to clarify with SlidesLive “before signing the contract” how many live zoom QA sessions they are going to cover. For example, they did not cover “any” of the QA sessions for papers in main conference, demo sessions, sponsors or D&I events, and it was not clearly communicated to us pre-signing the contract. They also did not cover any of the parallel QA sessions happening in our workshops. We ended up creating our own zoom team to handle these. To prevent this from happening we suggest asking them to include a technical staff as well during discussions, marketing representatives might not be aware of these types of details.
* Slideslive does its best to be responsive and usually tries to answer our questions as much as they can. We recommend asking them to assign more number of “technical staff” especially during the conference. We sometimes felt having only two contacts from them is absolutely not enough especially given the fact that one of them was not as technically confident as the other one.
* There was a 8 sec delay in live presentations (the video in zoom vs the one streaming on the SlidesLive website) which our general chair (Dan) brought up during SlidesLive demo session. Please request them to resolve this.

=== Accessibility ===
Please note that they only provide captioning for the videos / live sessions they are handling, i.e., if you end up having to create your own zoom meetings they are not going to provide captioning for them. Another note, when we made the contract with Slideslive, it wasn’t clear in the contract that they only provided automatic captioning service instead of high quality human-edited captioning. Please make sure to go through all these details before signing a contract with them.

=== Non-profit discount ===
SlidesLive provides a 25% discount for Non-Profit Organizations, please make sure to ask for this if you are eligible to use it.

=== SlidesLive demo ===
We “strongly” advise to request a demo session from them “before signing the contract”. You can use part of the ACL 2020 schedule as an example if you do not have the full conference program yet. This is very helpful for you to get a sense of the experience you will have and gives you an opportunity to include your requests from them quite early on. It might take for them some time to include the features you need for your conference and this will give them a reasonable amount of time to address your requests in their framework pre-conference.

== Zoom ==

=== General suggestions ===

* (repeating as it is quite important) We strongly suggest enforcing an early deadline for a finalized program of conference and specially workshops. This is very important as it helps to reduce communication overhead for creating all virtual sessions beforehand.
* We strongly suggest generating Zoom meetings well ahead of time to make sure you have enough time to debug and verify them. We have provided detailed documentation on how to create bulk set of meetings in zoom in [https://docs.google.com/document/d/1EYELyo5Yp3HESllBlqSis4aoAoxU6LQ0sRBwFAc5gMI/edit?usp=sharing this documentation]. We also have provided another step by step guide for users (authors/hosts of the meetings) [https://docs.google.com/document/d/1OWCcxXm9mL7DgcSWgW0FC87KmOEq8lIU3CwcsMEPLiI/edit?usp=sharing here].
* You need to have several volunteers during the conference who are specifically trained for zoom to be on call during the conference in case any incident happens. We recommend at least 2 volunteers per time slot. Note that only these volunteers should have access to your master zoom account.
* Verifying separate accounts need to be crowdsourced pre-conference (it is a pretty quick process), we asked volunteers to do it pre-conference.
* We strongly advise to have a contact person for zoom meetings of each part of conference, e.g., having a couple of POCs for main papers track, demos track, sponsors, etc ahead of time. This will reduce the amount of communication overhead a lot.
We suggest having a separate team for zoom meetings of workshops, there are usually several workshops with a much more diverse set of programs (it is like a mini conference by itself).
* '''If a meeting got hacked in progress''', on-call volunteers should delete the meeting link and create a new link and announce it at rocketchat immediately.
* '''Each meeting should have a unique main host.''' If you are ‘host’ for a meeting and start that meeting, and then hop off and try to start another one, you will get an error message from Zoom stating that you already have a meeting in progress and you need to end the previous one in order to start this one. If you click yes, ZOOM WILL END THE PREVIOUS MEETING AND KICK EVERYONE OFF.
* Attendees can join a meeting on browser '''only when''' the meeting is not requiring registration.
* '''Do not share the master account''' outside of the virtual infrastructure team.
* Note that it is likely that '''API credentials will expire''' after a certain amount of time. If this happens, you need to follow the instruction to regenerate it. (Before you run the script to create Zoom Meeting, you need to login to the zoom account and create JWT credentials in Zoom App Marketplace.)
* Virtual infrastructure team can maintain a spreadsheet to store all the credentials of authors, so on-call volunteers will be able to access every meeting and offer help for all the meetings.
* Paid accounts: you can pay Zoom for a business level account and '''add phone dial-in plans'''. This is a good idea for anyone who may have a spotty internet, so that they can dial in via phone/audio as well as video -- that way, they will always have audio connection for Q&A and it won’t drop them.
* '''When a meeting link is broken''' during the conference, on-call volunteers can login ACL2020 Master Zoom account, and select a specific user from “User Management” -> “Users”, and then create a new meeting for that user (the user will be the host).

=== Zoom attendees statistics ===

We made an analysis of Zoom attendees statistics for the main conference Q&A Session. You can also find more detail and the script in the folder below [https://drive.google.com/drive/folders/1WFU0oml-XUKNzuL_YOBnBJ8wX7BRHJrR?usp=sharing here].

* Total # of Unique Participants in Each Q&A Session:

* Distribution of participants over time 20 minutes after start
* 35 minutes after start
* 50 minutes after start

== RocketChat ==
RocketChat is a platform (similar to Slack) that provides async text-based chat functionality. In our workspace (acl2020.rocket.chat), we can create channels for different purposes.

=== Channels ===
Here we provide a summary of channels we have created for ACL2020.

* '''#announcements''': This is a read-only channel which is used by organizers to make conference-wide announcements. Only authorized users can post messages in this channel. Other users can only react with emoji to individual threads.

* '''#general''': This channel is used to discuss general conference topics. The conversation can be about anything related to ACL2020. During the conference, this channel was used to promote their live sessions by authors, workshop organizers, social events organizers, etc.

* '''#live''': This channel is only used for live plenary sessions. It was read-only for the majority of the time. During the live plenary sessions, we lift the read-only setting and people can use this channel for live QA. People have also used this channel to congratulate award winners, etc.

* '''#firsttimers/#introduction''': We initially only created the #firsttimers for people who attend ACL for the first time to say Hi to each other. At the very beginning of the conference, someone created the #introduction channel and many people joined that channel instead to introduce themselves.

* '''#helpdesk''': This channel was used by all ACL2020 attendees to request tech support for the ACL2020 virtual conference. We ask people to prefix their questions with a topic, e.g., [Zoom], [RocketChat], etc. On-call volunteers always start a thread when responding to a question. And continue the conversation in the thread. Once the question is resolved, they will mark it with :checkmark: emoji. See the screenshot below for a demonstration.

* '''#presenter-helpdesk''': This channel is same as #helpdesk, except that it is dedicated to authors and sponsors. It turns out most attendee still ask questions in #helpdesk so this channel may not be necessary in future.

* '''#incidents''': This channel was used to report incidents in the conference. Conference attendees can request the on-call volunteer in the channel to join your Zoom meetings as a co-host to handle disruptions or delete messages in RocketChat. They can simply click “Reply” button to start a private conversation with the on-call volunteer.

* '''#professional-conduct-committee''': This channel was used to contact the Professional Conduct Committee if you need our assistance (e.g. related to a concern that might fall under the anti-harassment policy). PCC members will post a message in this channel to let people know which of them are available. Then the attendee can contact the PCC member via RocketChat private message (similar to the #incidents).

* '''#social-media-posts''': This channel contains cross-posted social media posts published on various platforms by our official live tweeters and microbloggers, managed by our publicity chairs.

* '''#new-channels''': This channel was used to announce new channels made by conference attendees. It turns out many people have created different channels during the conference, including some topic-based channels (eg., #bio-clinical-nlp), university-based channels (e.g., #uwnlp) to get to know different research groups, location-base channels (e.g., #singapore), etc.

* '''#job-listing''': This channel was used to post job opportunities. We only get 10 posts in this channel throughout the conference though.

* '''#oncall-volunteers''': This channel is dedicated to on-call volunteers for them to coordinate with each other during the conference.

* '''#sponsor-xxx''': For each sponsor, we create a dedicated channel for them to post announcements and interact with conference attendees. These are “broadcast” channels (similar to #incidents, #professional-conduct-committee), i.e., only the channel owner can post messages. Visitors can only start a private conversation with the message author. This setting is due to the privacy concerns -- we do not want to expose who has visited a specific sponsor booth.

* '''#sponsor-registration-desk''': This channel is used as the virtual registration desk for sponsors. We need this channel because we need to set the sponsor account as the owner of their channels, and we didn’t develop an automatic way to do this. Since there are only ~20 sponsors, we just ask them to post a message in this channel when they arrive, and we just manually add them to the #sponsor-xxx channel and set them as the owner. After we add the sponsor account to their channel, we post a message “@username Welcome! I have set you as the owner of the channel. You can delete this message with your owner’s permission” so that they can familiarize themselves with the RocketChat usage.

* '''#organizers-registration-desk''': Similar to #sponsor-registration-desk, this channel is dedicated to organizers who need to be the owner of specific channels.

* '''#paper-xxx''': For each paper, we create a channel where the author can interact with visitors to answer questions.

* '''#keynote-xxx/#workshop-xxx/#tutorial-xxx''': For each keynote/workshop/tutorial, we create a channel where attendees can interact with the keynote speaker and the workshop/tutorial organizers.

* '''#business-meeting/#review-meeting''': These two channels are used for discussions about the business meeting and the review meeting, respectively.

=== Other considerations ===

* Emoji: We figure this is a way to attract more engagement from conference attendees. We uploaded the ACL logo, together with some other emoji. The ACL logo is a must-have one as we (organizers) usually by default add this emoji reaction to an announcement.
* Bad word filtering: We initially enabled the default bad word filtering in RocketChat during the internal test. However, it turns out it blocks many words such as “queer”, “gay”, etc., and we disable this feature during the conference. It seems to be okay as we didn’t receive any report about profanity in RocketChat.
* Permission to @here and @all: We initially allow all users to @here (mention all online users in the channel) and @all (mention all users in the channel). But it turns out this makes the space very noisy as people have extensively used @here/@all to announce their QA sessions. Therefore, we disable this permission and only allow channel owners to @here/@all.

=== Other platforms ===
* Slack might be an alternative to RocketChat, which has a better UI. But it is more expensive. Also, we don’t have the scripts to batch create channels for 800+ papers. Furthermore, we didn’t have time to investigate whether there are some specific features (e.g., broadcast channels) in RocketChat that are not available in Slack. In particular, we need to embed the channel as an iframe on a webpage, which seems to be unsupported natively by Slack.
* [https://www.dory.app/ Dory.app] and [https://www.sli.do/ Slido] could be used in addition to RocketChat for collecting questions & votes. This is perhaps one of the major limitations of RocketChat, i.e., it is not very ergonomic to collect questions/votes, partially because all messages in a thread are also shown in the channel with a smaller font, making it hard to read. AKBC2020 has used Slido, and one of the tutorials in ACL2020 has used Dory.app. Both seem to work well in their scenarios. However, we have tried Dory.app for the live plenary sessions on July 7, and it turns out that their server cannot handle the large amount of traffic and it crashed during the conference. So we have to switch back to the #live channel for collecting questions.

=== Recommendations ===
In general, we have relatively good experience with RocketChat. It does have some issues but the support team is very responsive, so I would recommend it for future conferences. The Slack might be a better alternative if it can meet the necessary requirements.
I would highly recommend finding a better platform for collecting questions & votes to augment RocketChat. The co-founder of Dory.app (Ena ena.zheng@dory-4098177ab9b4.intercom-mail.com) has reached out to me and she explained the situation. They can easily increase the capacity of their server/database for a specific period if we can provide them an early notice. They are very happy to support the conference, so the EMNLP2020 organizers can directly reach out to Ena to discuss further details.

== Virtual Conference Website ==
The [https://virtual.acl2020.org virtual conference website] is the central platform to present all information about the conference. As pointed out by ICLR2020 organizers, a portal bringing together all the different tools needed for the virtual conference was not available, and they have developed their own website. Thus, we adopted their codebase Mini-Conf https://github.com/Mini-Conf/Mini-Conf and made some extensions in our own fork https://github.com/acl-org/acl-2020-virtual-conference to meet our specific needs.
After the conference, we have removed all RocketChat channels from the website, but you can check out the acl2020-v7.0 release of the repo and spin up the website locally to see the version we used during the conference. We also have an introduction [https://slideslive.com/38931552/acl-website-walkthrough video] that explains how to use the website.

=== Overall design ===
The website has 12 top-level pages.
* '''Home''': We explain how to use the website and embed the #announcement channel on this page.
* '''Schedule''': This page shows a calendar view of all events during the conference.
* '''Livestream''': This page is used for the livestream of plenary sessions.
* '''Plenary''': This page contains the information about plenary sessions. Each plenary session also has its own page to display the pre-recorded videos and optionally an associated RocketChat channel.
* '''Papers''': This page displays all main conference papers (including the demo papers and student research workshop papers). Users can browse papers by track, search by author/title, and use the visualization tool to find a cluster of papers. Each paper also has its own page, where we display the detailed information about the paper, the live session slots, the pre-recorded video, and the associated RocketChat channel.
* '''Tutorials''': This page displays all tutorials on July 5, using a calendar view. Each tutorial also has its own page.
* '''Workshops''': This page displays all workshops, and each workshop has its own webpage.
* '''Sponsors''': This page displays all sponsors by their levels. Most sponsors have their own dedicated page to display their video, downloadable content, contact, and a RocketChat channel #sponsor-xxx described in Section 3.
* '''Socials''': This page displays all social events, but each event does not have a dedicated page.
* '''Chat''': Here we simply embed the RocketChat workspace.
* '''Organizers''': Here we display all conference organizers with their photos.
* '''Help''': Here we show the FAQ and Code of Conduct.

=== Tutorials ===
* We found that each tutorial is different. It would be useful to consult with each tutorial organizer beforehand to understand their needs and customize their virtual tutorial page accordingly. This year we had a common template for all tutorials but in hindsight it could have been slightly different for each.
* Some tutorial teachers did not know that tutorial videos can be recorded in parts (using SlidesLive) and had to record their 2+ hours of videos in one go. We should let tutorial teachers know beforehand that video can be recorded in parts.
* Attendees were not clear on what was expected of them before the tutorial. It would have been useful if each tutorial teacher explained what preparation is required in this virtual setup.
* The tutorial chairs posted a great blog on different modalities of tutorials - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It might have been more useful if posted even earlier.

=== Workshops ===
* There should be a separate team of workshop-virtual-chairs dedicated to only thinking about virtual infrastructure needs for workshops. With 20 workshops, some as big as a mini-conference, I think it is important to have someone think about the workshop infrastructure explicitly.
* Each workshop paper should have a page of its own (similar to main conference papers) with link to paper, pre-recorded talk, rocketchat and zoom link for Q&A
* With the new virtual format, each workshop should be asked to write a paragraph explaining their virtual setup. There was a lot of confusion this year since each workshop had a different setup. The paragraph should include
** Whether they have parallel Q&A session on different zoom links
** Whether the pre-recorded talks will be played again on the live stream OR do they expect attendees to watch them beforehand.
** If the pre-recorded talks are going to be played in the livestream, they should ask attendees to stay on the website for the talk part and enter the zoom meeting only after the talk for the Q&A session. This year, many attendees were waiting in the zoom room thinking the pre-recording talks can be seen there as well (not knowing that they should go to the website to watch it)
**Like the main conference, perhaps the keynote of workshops should also allow questions only on rocketchat. Because I noticed some workshop organizers had trouble fielding questions from different sources (zoom chat, zoom hand raising, rocketchat channel)
* It seems like workshops that were single track worked better than those that had multiple tracks and this was because with parallel poster sessions, it was confusing how to go to each paper Q&A session separately.
* Workshops organizers should be given a deadline after which they should not be able to update their schedule. Last minute schedule updates this year caused SlidesLive delay in preparing their livestream session.
* Setting such a deadline will also help infra chairs to create all zoom meetings using individual zoom accounts separately as they did for main conference papers.
* The main zoom meeting should have a dedicated moderator who can take care of assigning speakers to be co-hosts.
* Each workshop should have a separate read only rocketchat channel (similar to main conference #announcement channel) where organizers can make important announcements. This year, with a common channel for all communication, the announcements from organizers were getting drowned within other chat.
* Many attendees had questions about what material will be available before/during/after the conference. The tutorial chairs posted a great blog on this - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It would be great if workshop chairs did something similarly.

=== Virtual booths for sponsors ===
* We already have a pretty good template for virtual booths. The hardest part is to communicate with the sponsors and collect information from them.
* In ACL 2020, we only communicate with the sponsors through email. It’s very hard to keep track of the changes sponsors make and sponsors always want to change information at the last minute. It would be great if we could automate some information collection process in the future. Here are some suggestions for future reference:
** Sponsors always want to change Zoom schedules at the last minute, and it would be inconvenient for them to email us every time they change their schedule. So, it would be great if we could ask them to input the information into a shared spreadsheet, so that they can change their schedule by modifying the spreadsheet. We can write a code to load the spreadsheet to the yml file so that we only need to do a refresh before the main conference, which will significantly reduce our effort.
** For all the files and resources, it’s better to ask sponsors to provide us a static url link to the file, so that as long as the link does not change, they can update their file anytime they want. We can also ask sponsors to put the <filename, url link> pair into a shared spreadsheet so that we don’t need to manually input the information based on emails.
** For some files, e.g. logos and pdfs, we can create a shared Google drive for them to update their logos. Again in this ACL we send all files through email which is very inconvenient.
* We should give sponsors a hard deadline on when they can not update the information. For this ACL, we got emails even after the main conference where the sponsors want to change schedules/materials in the virtual booth.
* This year we included the sponsors in the internal dry period, which turns out to be necessary since the sponsors will be able to check and test their virtual booths before the main conference. It’s very important since we can do one round of iteration to fix the information in the virtual booth based on the sponsors’ feedback before the main conference.
* This year we have 19 sponsors and only 2 volunteers to build the virtual booth, so each volunteer needs to build 9-10 sponsors. The workload is too heavy for each of the volunteers (each of them take around 4 days full time). I think the best balance would be 4-5 sponsors per volunteer. There should be one more volunteer that is available during the main conference to fix the website if there are urgent requests from the sponsors.
* We created one admin account for each sponsor that’s valid through the end of the conference, and a test visitor account that’s valid before the conference so that the sponsors can test the interaction between visitors and sponsors (especially on RocketChat). This is very important for the sponsors to learn how to make announcements and how to monitor the channel when visitors send questions.
* It’s important to provide sponsors with a step-by-step instruction on how to login, how to use Zoom, and how to use RocketChat. The direct contact for each module of the virtual booth should also be listed in the instruction, e.g. provide contact for login related issues, Zoom related issues, RocketChat related issues and wrong information in the virtual booth. This way we can significantly reduce the number of repeated questions from sponsors and save a lot of time. Link to the instruction this year: https://docs.google.com/document/d/1paG7hY7E9qKEBZrXZloyfnI3Uxt_723J23JNWBKOMCI/edit
* We should also decide on a date to stop getting new sponsors. It’s not realistic to build new virtual booths at the last minute (which happened a few times for ACL 2020).
* Double check the sponsors’ list with the list on the external website in case there are missed sponsors.

=== User login ===
* We basically follow [https://aws.amazon.com/blogs/networking-and-content-delivery/authorizationedge-using-cookies-protect-your-amazon-cloudfront-content-from-being-downloaded-by-unauthenticated-users/ this AWS solution] to do the login authentication. But we have used a slightly different version [https://github.com/opaquejacob/cloudfront-authorization-at-edge.git here] so we can use the customized auth domain https://signin.acl2020.org. See [https://github.com/aws-samples/cloudfront-authorization-at-edge/compare/master...opaquejacob:master here] for the changes from the original version in the blog post.
* This solution allows us to use OAuth to implement the sign-on for the website and RocketChat. See [https://github.com/acl-org/acl-2020-virtual-conference/issues/53 here] for instructions.
* By default, the AWS Cognito pool can only send 50 emails per day. Need to follow [https://github.com/acl-org/acl-2020-virtual-conference/issues/53#issuecomment-649327363 this instruction] to increase the limit and [https://github.com/acl-org/acl-2020-virtual-conference/issues/104#issuecomment-653592927 this instruction] to switch to use the Amazon SES.
* For people who registered before July 3, we create the accounts on July 3 using the excel file provided by Prisicilla.
* To handle registration during the conference, we use Zapier + AWS Lambda. See https://github.com/acl-org/acl-2020-virtual-conference/issues/55#issue-636817449 for more details.

Although this solution works reasonably well, it does have some issues. Actually, 95% questions sent to acl2020-virtual-helpdesk@googlegroups.com are about login problems, and we probably have handled around 200 such requests.
* The AWS Cognito sent out login emails on behalf of acl2020virtual@gmail.com. It turns out the emails are easily classified as spam. Also some university email server blocks it (e.g., we found consistent issues with @xxx.i.kyoto-u.ac.jp).
* Many people could not login because they have entered the wrong password. We have therefore updated several versions of our instruction https://acl2020.org/_pages/docs/ACL2020_virtual_website_login_steps.pdf with some common errors (see Step 1), which seems to be helpful.
* We announce that our website will be live on July 3, and we send out the login credentials on that day. It turns out to be a not very wise decision because we have to deal with all incoming inquiries about login issues without volunteers on July 3 & 4. (Our on-call volunteers can only start working on July 5).
* Some authors are confused with the account used for the website and the SoftConf account.
* For some unknown reason, there are many users who have registered but were not included in the excel files sent from Priscilla. We may need to check with the company who hosted the registration form about this.
* It would be helpful if we can have the registration company directly handle the login authentication, so that we don’t need to have two separate processes to create user accounts.

== General Advice for Future Virtual Conferences ==
* It would be nice if future conference organizers can find a vendor company to run the virtual conference if possible. Otherwise, the virtual infrastructure committee would definitely need much more people and figure out a way to distribute the work more. Since all organizer committee members are officially volunteers, we cannot rely on people to work full-time or even extra hours on this (which we have to do for ACL2020 given the limited resources and short timeline). Here is a potential organization of the virtual infrastructure committee.
** Pre-recorded videos & livestreaming (1-2 chairs + a vendor company)
** Zoom meetings (1-2 chairs + 5-8 volunteers)
** Captioning (1-2 chairs)
** RocketChat (1 chair + 3-4 code development volunteers)
** Main website development (1-2 chairs + N code development volunteers)
** Virtual sponsor booth (1 chair + 1 code development volunteers per 5 sponsors)
** Tutorials (1 chair + 2-3 code development volunteers)
** Workshops (probably better to have their own virtual infrastructure chairs to customize their webpages)
** Login authentication (1 chair + 2 code development volunteers + 5 helpdesk volunteers; or use a vendor company)
* The repo https://github.com/acl-org/acl-2020-virtual-conference should be a good start for future conferences. Due to the time limit, we didn’t maintain a very high code quality but it should not be very hard to spin up a new website. The most challenging part would be gathering all sitedata, including the paper basic information, QA session schedule, Zoom links, video URLs, etc.
* It would be useful to have a better pipeline for information collection. Currently we have to gather information from different chairs. It would be better if the website, the proceedings, the handbook, and the ACL Anthology all use the same information source such that everything is synchronized.
* GitHub Issue seems to be a good way to manage the tasks and distribute work to volunteers.
* It would be better to have volunteers to be involved earlier, and work with the volunteer coordinator on the timeline. It was not a very smooth transition because most volunteers can work either before or during the conference, not both. And there is a period that our chairs have to handle all emails in acl2020-virtual-helpdesk@googlegroups.com on our own on the first two days when there are lots of inquiries.
* There is many other useful information in the article [https://medium.com/@iclr_conf/gone-virtual-lessons-from-iclr2020-1743ce6164a3 “Gone Virtual: Lessons from ICLR2020"]. The section “Technical Systems and Support” is quite relevant to the virtual conference infrastructure.

2020Q3 Reports: Virtual Infrastructure Chairs

2020-07-21T14:05:55Z

Raosudha89: /* Tutorials */

The virtual conference infrastructure of ACL2020 consists of 4 major components:
* SlidesLive for pre-recorded talks & video live-streaming
* Zoom for live QA sessions
* RocketChat for async text-based discussions and announcements
* the [https://virtual.acl2020.org virtual conference website] as the central platform to present all information about the conference
Due to limited time, we follow the infrastructure setup of ICLR2020, with some necessary extensions including the plenary sessions, tutorials, virtual sponsor booths, etc. In the remaining sections of this report, we provide a high-level summary of each component and discuss potential improvement can be made in future. At the end, we provide some general advice for future virtual conferences based on our experience.

== SlidesLive ==

SlidesLive is a company that has helped with several previous conferences including ICLR, ICML, etc for recording the videos of the talks, post processing them and putting the slides and the presenter side by side. We initially negotiated with several companies including them for the physical conference in Seattle. The companies we discussed with with their Point Of Contact (POC) were:
* SlidesLive, POC: Katherine (Katie) Millar, katherine@slideslive.com
* Underline, POC: Sol Rosenberg, sol@underline.io
* GloCastLive, POC: Arran Moffat, arran@glocast.com
* BASH FILMS, POC: Mark Bashian, mbashian@bashfilms.com
* Freeman, POC: Brian Holm, Brian.Holm@freemanco.com
We have put all the information we collected from these companies including the quotes [https://drive.google.com/drive/folders/1KYzLT-B5ZVIb3n_KDSRlMQnahoTuxxei?usp=sharing here] for your reference. Please note that these are for the time when the conference was supposed to be physical, nevertheless many of them also offer virtual experience.

After deciding to have the conference fully virtual we signed a contract with SlidesLive and relied on them for our keynotes and workshops besides the post processing of the videos. Below we list “our experience” working with them, noting these before signing a contract with them should be very helpful to have a great experience working with them, if you choose them as the company you want to work with.

=== General suggestions ===
Here we summarize our general suggestions, especially before signing a contract with them.

* We strongly suggest enforcing an early deadline for a finalized program of conference and '''specially workshops'''. This is very important as it helps to reduce communication overhead between SlidesLive and the rest of the conference.
* It is very important to clarify with SlidesLive “before signing the contract” how many live zoom QA sessions they are going to cover. For example, they did not cover “any” of the QA sessions for papers in main conference, demo sessions, sponsors or D&I events, and it was not clearly communicated to us pre-signing the contract. They also did not cover any of the parallel QA sessions happening in our workshops. We ended up creating our own zoom team to handle these. To prevent this from happening we suggest asking them to include a technical staff as well during discussions, marketing representatives might not be aware of these types of details.
* Slideslive does its best to be responsive and usually tries to answer our questions as much as they can. We recommend asking them to assign more number of “technical staff” especially during the conference. We sometimes felt having only two contacts from them is absolutely not enough especially given the fact that one of them was not as technically confident as the other one.
* There was a 8 sec delay in live presentations (the video in zoom vs the one streaming on the SlidesLive website) which our general chair (Dan) brought up during SlidesLive demo session. Please request them to resolve this.

=== Accessibility ===
Please note that they only provide captioning for the videos / live sessions they are handling, i.e., if you end up having to create your own zoom meetings they are not going to provide captioning for them. Another note, when we made the contract with Slideslive, it wasn’t clear in the contract that they only provided automatic captioning service instead of high quality human-edited captioning. Please make sure to go through all these details before signing a contract with them.

=== Non-profit discount ===
SlidesLive provides a 25% discount for Non-Profit Organizations, please make sure to ask for this if you are eligible to use it.

=== SlidesLive demo ===
We “strongly” advise to request a demo session from them “before signing the contract”. You can use part of the ACL 2020 schedule as an example if you do not have the full conference program yet. This is very helpful for you to get a sense of the experience you will have and gives you an opportunity to include your requests from them quite early on. It might take for them some time to include the features you need for your conference and this will give them a reasonable amount of time to address your requests in their framework pre-conference.

== Zoom ==

=== General suggestions ===

* (repeating as it is quite important) We strongly suggest enforcing an early deadline for a finalized program of conference and specially workshops. This is very important as it helps to reduce communication overhead for creating all virtual sessions beforehand.
* We strongly suggest generating Zoom meetings well ahead of time to make sure you have enough time to debug and verify them. We have provided detailed documentation on how to create bulk set of meetings in zoom in [https://docs.google.com/document/d/1EYELyo5Yp3HESllBlqSis4aoAoxU6LQ0sRBwFAc5gMI/edit?usp=sharing this documentation]. We also have provided another step by step guide for users (authors/hosts of the meetings) [https://docs.google.com/document/d/1OWCcxXm9mL7DgcSWgW0FC87KmOEq8lIU3CwcsMEPLiI/edit?usp=sharing here].
* You need to have several volunteers during the conference who are specifically trained for zoom to be on call during the conference in case any incident happens. We recommend at least 2 volunteers per time slot. Note that only these volunteers should have access to your master zoom account.
* Verifying separate accounts need to be crowdsourced pre-conference (it is a pretty quick process), we asked volunteers to do it pre-conference.
* We strongly advise to have a contact person for zoom meetings of each part of conference, e.g., having a couple of POCs for main papers track, demos track, sponsors, etc ahead of time. This will reduce the amount of communication overhead a lot.
We suggest having a separate team for zoom meetings of workshops, there are usually several workshops with a much more diverse set of programs (it is like a mini conference by itself).
* '''If a meeting got hacked in progress''', on-call volunteers should delete the meeting link and create a new link and announce it at rocketchat immediately.
* '''Each meeting should have a unique main host.''' If you are ‘host’ for a meeting and start that meeting, and then hop off and try to start another one, you will get an error message from Zoom stating that you already have a meeting in progress and you need to end the previous one in order to start this one. If you click yes, ZOOM WILL END THE PREVIOUS MEETING AND KICK EVERYONE OFF.
* Attendees can join a meeting on browser '''only when''' the meeting is not requiring registration.
* '''Do not share the master account''' outside of the virtual infrastructure team.
* Note that it is likely that '''API credentials will expire''' after a certain amount of time. If this happens, you need to follow the instruction to regenerate it. (Before you run the script to create Zoom Meeting, you need to login to the zoom account and create JWT credentials in Zoom App Marketplace.)
* Virtual infrastructure team can maintain a spreadsheet to store all the credentials of authors, so on-call volunteers will be able to access every meeting and offer help for all the meetings.
* Paid accounts: you can pay Zoom for a business level account and '''add phone dial-in plans'''. This is a good idea for anyone who may have a spotty internet, so that they can dial in via phone/audio as well as video -- that way, they will always have audio connection for Q&A and it won’t drop them.
* '''When a meeting link is broken''' during the conference, on-call volunteers can login ACL2020 Master Zoom account, and select a specific user from “User Management” -> “Users”, and then create a new meeting for that user (the user will be the host).

=== Zoom attendees statistics ===

We made an analysis of Zoom attendees statistics for the main conference Q&A Session. You can also find more detail and the script in the folder below [https://drive.google.com/drive/folders/1WFU0oml-XUKNzuL_YOBnBJ8wX7BRHJrR?usp=sharing here].

* Total # of Unique Participants in Each Q&A Session:

* Distribution of participants over time 20 minutes after start
* 35 minutes after start
* 50 minutes after start

== RocketChat ==
RocketChat is a platform (similar to Slack) that provides async text-based chat functionality. In our workspace (acl2020.rocket.chat), we can create channels for different purposes.

=== Channels ===
Here we provide a summary of channels we have created for ACL2020.

* '''#announcements''': This is a read-only channel which is used by organizers to make conference-wide announcements. Only authorized users can post messages in this channel. Other users can only react with emoji to individual threads.

* '''#general''': This channel is used to discuss general conference topics. The conversation can be about anything related to ACL2020. During the conference, this channel was used to promote their live sessions by authors, workshop organizers, social events organizers, etc.

* '''#live''': This channel is only used for live plenary sessions. It was read-only for the majority of the time. During the live plenary sessions, we lift the read-only setting and people can use this channel for live QA. People have also used this channel to congratulate award winners, etc.

* '''#firsttimers/#introduction''': We initially only created the #firsttimers for people who attend ACL for the first time to say Hi to each other. At the very beginning of the conference, someone created the #introduction channel and many people joined that channel instead to introduce themselves.

* '''#helpdesk''': This channel was used by all ACL2020 attendees to request tech support for the ACL2020 virtual conference. We ask people to prefix their questions with a topic, e.g., [Zoom], [RocketChat], etc. On-call volunteers always start a thread when responding to a question. And continue the conversation in the thread. Once the question is resolved, they will mark it with :checkmark: emoji. See the screenshot below for a demonstration.

* '''#presenter-helpdesk''': This channel is same as #helpdesk, except that it is dedicated to authors and sponsors. It turns out most attendee still ask questions in #helpdesk so this channel may not be necessary in future.

* '''#incidents''': This channel was used to report incidents in the conference. Conference attendees can request the on-call volunteer in the channel to join your Zoom meetings as a co-host to handle disruptions or delete messages in RocketChat. They can simply click “Reply” button to start a private conversation with the on-call volunteer.

* '''#professional-conduct-committee''': This channel was used to contact the Professional Conduct Committee if you need our assistance (e.g. related to a concern that might fall under the anti-harassment policy). PCC members will post a message in this channel to let people know which of them are available. Then the attendee can contact the PCC member via RocketChat private message (similar to the #incidents).

* '''#social-media-posts''': This channel contains cross-posted social media posts published on various platforms by our official live tweeters and microbloggers, managed by our publicity chairs.

* '''#new-channels''': This channel was used to announce new channels made by conference attendees. It turns out many people have created different channels during the conference, including some topic-based channels (eg., #bio-clinical-nlp), university-based channels (e.g., #uwnlp) to get to know different research groups, location-base channels (e.g., #singapore), etc.

* '''#job-listing''': This channel was used to post job opportunities. We only get 10 posts in this channel throughout the conference though.

* '''#oncall-volunteers''': This channel is dedicated to on-call volunteers for them to coordinate with each other during the conference.

* '''#sponsor-xxx''': For each sponsor, we create a dedicated channel for them to post announcements and interact with conference attendees. These are “broadcast” channels (similar to #incidents, #professional-conduct-committee), i.e., only the channel owner can post messages. Visitors can only start a private conversation with the message author. This setting is due to the privacy concerns -- we do not want to expose who has visited a specific sponsor booth.

* '''#sponsor-registration-desk''': This channel is used as the virtual registration desk for sponsors. We need this channel because we need to set the sponsor account as the owner of their channels, and we didn’t develop an automatic way to do this. Since there are only ~20 sponsors, we just ask them to post a message in this channel when they arrive, and we just manually add them to the #sponsor-xxx channel and set them as the owner. After we add the sponsor account to their channel, we post a message “@username Welcome! I have set you as the owner of the channel. You can delete this message with your owner’s permission” so that they can familiarize themselves with the RocketChat usage.

* '''#organizers-registration-desk''': Similar to #sponsor-registration-desk, this channel is dedicated to organizers who need to be the owner of specific channels.

* '''#paper-xxx''': For each paper, we create a channel where the author can interact with visitors to answer questions.

* '''#keynote-xxx/#workshop-xxx/#tutorial-xxx''': For each keynote/workshop/tutorial, we create a channel where attendees can interact with the keynote speaker and the workshop/tutorial organizers.

* '''#business-meeting/#review-meeting''': These two channels are used for discussions about the business meeting and the review meeting, respectively.

=== Other considerations ===

* Emoji: We figure this is a way to attract more engagement from conference attendees. We uploaded the ACL logo, together with some other emoji. The ACL logo is a must-have one as we (organizers) usually by default add this emoji reaction to an announcement.
* Bad word filtering: We initially enabled the default bad word filtering in RocketChat during the internal test. However, it turns out it blocks many words such as “queer”, “gay”, etc., and we disable this feature during the conference. It seems to be okay as we didn’t receive any report about profanity in RocketChat.
* Permission to @here and @all: We initially allow all users to @here (mention all online users in the channel) and @all (mention all users in the channel). But it turns out this makes the space very noisy as people have extensively used @here/@all to announce their QA sessions. Therefore, we disable this permission and only allow channel owners to @here/@all.

=== Other platforms ===
* Slack might be an alternative to RocketChat, which has a better UI. But it is more expensive. Also, we don’t have the scripts to batch create channels for 800+ papers. Furthermore, we didn’t have time to investigate whether there are some specific features (e.g., broadcast channels) in RocketChat that are not available in Slack. In particular, we need to embed the channel as an iframe on a webpage, which seems to be unsupported natively by Slack.
* [https://www.dory.app/ Dory.app] and [https://www.sli.do/ Slido] could be used in addition to RocketChat for collecting questions & votes. This is perhaps one of the major limitations of RocketChat, i.e., it is not very ergonomic to collect questions/votes, partially because all messages in a thread are also shown in the channel with a smaller font, making it hard to read. AKBC2020 has used Slido, and one of the tutorials in ACL2020 has used Dory.app. Both seem to work well in their scenarios. However, we have tried Dory.app for the live plenary sessions on July 7, and it turns out that their server cannot handle the large amount of traffic and it crashed during the conference. So we have to switch back to the #live channel for collecting questions.

=== Recommendations ===
In general, we have relatively good experience with RocketChat. It does have some issues but the support team is very responsive, so I would recommend it for future conferences. The Slack might be a better alternative if it can meet the necessary requirements.
I would highly recommend finding a better platform for collecting questions & votes to augment RocketChat. The co-founder of Dory.app (Ena ena.zheng@dory-4098177ab9b4.intercom-mail.com) has reached out to me and she explained the situation. They can easily increase the capacity of their server/database for a specific period if we can provide them an early notice. They are very happy to support the conference, so the EMNLP2020 organizers can directly reach out to Ena to discuss further details.

== Virtual Conference Website ==
The [https://virtual.acl2020.org virtual conference website] is the central platform to present all information about the conference. As pointed out by ICLR2020 organizers, a portal bringing together all the different tools needed for the virtual conference was not available, and they have developed their own website. Thus, we adopted their codebase Mini-Conf https://github.com/Mini-Conf/Mini-Conf and made some extensions in our own fork https://github.com/acl-org/acl-2020-virtual-conference to meet our specific needs.
After the conference, we have removed all RocketChat channels from the website, but you can check out the acl2020-v7.0 release of the repo and spin up the website locally to see the version we used during the conference. We also have an introduction [https://slideslive.com/38931552/acl-website-walkthrough video] that explains how to use the website.

=== Overall design ===
The website has 12 top-level pages.
* '''Home''': We explain how to use the website and embed the #announcement channel on this page.
* '''Schedule''': This page shows a calendar view of all events during the conference.
* '''Livestream''': This page is used for the livestream of plenary sessions.
* '''Plenary''': This page contains the information about plenary sessions. Each plenary session also has its own page to display the pre-recorded videos and optionally an associated RocketChat channel.
* '''Papers''': This page displays all main conference papers (including the demo papers and student research workshop papers). Users can browse papers by track, search by author/title, and use the visualization tool to find a cluster of papers. Each paper also has its own page, where we display the detailed information about the paper, the live session slots, the pre-recorded video, and the associated RocketChat channel.
* '''Tutorials''': This page displays all tutorials on July 5, using a calendar view. Each tutorial also has its own page.
* '''Workshops''': This page displays all workshops, and each workshop has its own webpage.
* '''Sponsors''': This page displays all sponsors by their levels. Most sponsors have their own dedicated page to display their video, downloadable content, contact, and a RocketChat channel #sponsor-xxx described in Section 3.
* '''Socials''': This page displays all social events, but each event does not have a dedicated page.
* '''Chat''': Here we simply embed the RocketChat workspace.
* '''Organizers''': Here we display all conference organizers with their photos.
* '''Help''': Here we show the FAQ and Code of Conduct.

=== Tutorials ===
* We found that each tutorial is different. It would be useful to consult with each tutorial organizer beforehand to understand their needs and customize their virtual tutorial page accordingly. This year we had a common template for all tutorials but in hindsight it could have been slightly different for each.
* Some tutorial teachers did not know that tutorial videos can be recorded in parts (using SlidesLive) and had to record their 2+ hours of videos in one go. We should let tutorial teachers know beforehand that video can be recorded in parts.
* Attendees were not clear on what was expected of them before the tutorial. It would have been useful if each tutorial teacher explained what preparation is required in this virtual setup.
* The tutorial chairs posted a great blog on different modalities of tutorials - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It might have been more useful if posted even earlier.

=== Workshops ===
* There should be a separate team of workshop-virtual-chairs dedicated to only thinking about virtual infrastructure needs for workshops. With 20 workshops, some as big as a mini-conference, I think it is important to have someone think about the workshop infrastructure explicitly.
* Each workshop paper should have a page of its own (similar to main conference papers) with link to paper, pre-recorded talk, rocketchat and zoom link for Q&A
* With the new virtual format, each workshop should be asked to write a paragraph explaining their virtual setup. There was a lot of confusion this year since each workshop had a different setup. The paragraph should include
** Whether they have parallel Q&A session on different zoom links
** Whether the pre-recorded talks will be played again on the live stream OR do they expect attendees to watch them beforehand.
** If the pre-recorded talks are going to be played in the livestream, they should ask attendees to stay on the website for the talk part and enter the zoom meeting only after the talk for the Q&A session. This year, many attendees were waiting in the zoom room thinking the pre-recording talks can be seen there as well (not knowing that they should go to the website to watch it)
**Like the main conference, perhaps the keynote of workshops should also allow questions only on rocketchat. Because I noticed some workshop organizers had trouble fielding questions from different sources (zoom chat, zoom hand raising, rocketchat channel)
* It seems like workshops that were single track worked better than those that had multiple tracks and this was because with parallel poster sessions, it was confusing how to go to each paper Q&A session separately.
* Workshops organizers should be given a deadline after which they should not be able to update their schedule. Last minute schedule updates this year caused SlidesLive delay in preparing their livestream session.
* Setting such a deadline will also help infra chairs to create all zoom meetings using individual zoom accounts separately as they did for main conference papers.
* The main zoom meeting should have a dedicated moderator who can take care of assigning speakers to be co-hosts.
* Each workshop should have a separate read only rocketchat channel (similar to main conference #announcement channel) where organizers can make important announcements. This year, with a common channel for all communication, the announcements from organizers were getting drowned within other chat.
* Many attendees had questions about what material will be available before/during/after the conference. The tutorial chairs posted a great blog on this. It would be great if workshop chairs did something similarly.

=== Virtual booths for sponsors ===
* We already have a pretty good template for virtual booths. The hardest part is to communicate with the sponsors and collect information from them.
* In ACL 2020, we only communicate with the sponsors through email. It’s very hard to keep track of the changes sponsors make and sponsors always want to change information at the last minute. It would be great if we could automate some information collection process in the future. Here are some suggestions for future reference:
** Sponsors always want to change Zoom schedules at the last minute, and it would be inconvenient for them to email us every time they change their schedule. So, it would be great if we could ask them to input the information into a shared spreadsheet, so that they can change their schedule by modifying the spreadsheet. We can write a code to load the spreadsheet to the yml file so that we only need to do a refresh before the main conference, which will significantly reduce our effort.
** For all the files and resources, it’s better to ask sponsors to provide us a static url link to the file, so that as long as the link does not change, they can update their file anytime they want. We can also ask sponsors to put the <filename, url link> pair into a shared spreadsheet so that we don’t need to manually input the information based on emails.
** For some files, e.g. logos and pdfs, we can create a shared Google drive for them to update their logos. Again in this ACL we send all files through email which is very inconvenient.
* We should give sponsors a hard deadline on when they can not update the information. For this ACL, we got emails even after the main conference where the sponsors want to change schedules/materials in the virtual booth.
* This year we included the sponsors in the internal dry period, which turns out to be necessary since the sponsors will be able to check and test their virtual booths before the main conference. It’s very important since we can do one round of iteration to fix the information in the virtual booth based on the sponsors’ feedback before the main conference.
* This year we have 19 sponsors and only 2 volunteers to build the virtual booth, so each volunteer needs to build 9-10 sponsors. The workload is too heavy for each of the volunteers (each of them take around 4 days full time). I think the best balance would be 4-5 sponsors per volunteer. There should be one more volunteer that is available during the main conference to fix the website if there are urgent requests from the sponsors.
* We created one admin account for each sponsor that’s valid through the end of the conference, and a test visitor account that’s valid before the conference so that the sponsors can test the interaction between visitors and sponsors (especially on RocketChat). This is very important for the sponsors to learn how to make announcements and how to monitor the channel when visitors send questions.
* It’s important to provide sponsors with a step-by-step instruction on how to login, how to use Zoom, and how to use RocketChat. The direct contact for each module of the virtual booth should also be listed in the instruction, e.g. provide contact for login related issues, Zoom related issues, RocketChat related issues and wrong information in the virtual booth. This way we can significantly reduce the number of repeated questions from sponsors and save a lot of time. Link to the instruction this year: https://docs.google.com/document/d/1paG7hY7E9qKEBZrXZloyfnI3Uxt_723J23JNWBKOMCI/edit
* We should also decide on a date to stop getting new sponsors. It’s not realistic to build new virtual booths at the last minute (which happened a few times for ACL 2020).
* Double check the sponsors’ list with the list on the external website in case there are missed sponsors.

=== User login ===
* We basically follow [https://aws.amazon.com/blogs/networking-and-content-delivery/authorizationedge-using-cookies-protect-your-amazon-cloudfront-content-from-being-downloaded-by-unauthenticated-users/ this AWS solution] to do the login authentication. But we have used a slightly different version [https://github.com/opaquejacob/cloudfront-authorization-at-edge.git here] so we can use the customized auth domain https://signin.acl2020.org. See [https://github.com/aws-samples/cloudfront-authorization-at-edge/compare/master...opaquejacob:master here] for the changes from the original version in the blog post.
* This solution allows us to use OAuth to implement the sign-on for the website and RocketChat. See [https://github.com/acl-org/acl-2020-virtual-conference/issues/53 here] for instructions.
* By default, the AWS Cognito pool can only send 50 emails per day. Need to follow [https://github.com/acl-org/acl-2020-virtual-conference/issues/53#issuecomment-649327363 this instruction] to increase the limit and [https://github.com/acl-org/acl-2020-virtual-conference/issues/104#issuecomment-653592927 this instruction] to switch to use the Amazon SES.
* For people who registered before July 3, we create the accounts on July 3 using the excel file provided by Prisicilla.
* To handle registration during the conference, we use Zapier + AWS Lambda. See https://github.com/acl-org/acl-2020-virtual-conference/issues/55#issue-636817449 for more details.

Although this solution works reasonably well, it does have some issues. Actually, 95% questions sent to acl2020-virtual-helpdesk@googlegroups.com are about login problems, and we probably have handled around 200 such requests.
* The AWS Cognito sent out login emails on behalf of acl2020virtual@gmail.com. It turns out the emails are easily classified as spam. Also some university email server blocks it (e.g., we found consistent issues with @xxx.i.kyoto-u.ac.jp).
* Many people could not login because they have entered the wrong password. We have therefore updated several versions of our instruction https://acl2020.org/_pages/docs/ACL2020_virtual_website_login_steps.pdf with some common errors (see Step 1), which seems to be helpful.
* We announce that our website will be live on July 3, and we send out the login credentials on that day. It turns out to be a not very wise decision because we have to deal with all incoming inquiries about login issues without volunteers on July 3 & 4. (Our on-call volunteers can only start working on July 5).
* Some authors are confused with the account used for the website and the SoftConf account.
* For some unknown reason, there are many users who have registered but were not included in the excel files sent from Priscilla. We may need to check with the company who hosted the registration form about this.
* It would be helpful if we can have the registration company directly handle the login authentication, so that we don’t need to have two separate processes to create user accounts.

== General Advice for Future Virtual Conferences ==
* It would be nice if future conference organizers can find a vendor company to run the virtual conference if possible. Otherwise, the virtual infrastructure committee would definitely need much more people and figure out a way to distribute the work more. Since all organizer committee members are officially volunteers, we cannot rely on people to work full-time or even extra hours on this (which we have to do for ACL2020 given the limited resources and short timeline). Here is a potential organization of the virtual infrastructure committee.
** Pre-recorded videos & livestreaming (1-2 chairs + a vendor company)
** Zoom meetings (1-2 chairs + 5-8 volunteers)
** Captioning (1-2 chairs)
** RocketChat (1 chair + 3-4 code development volunteers)
** Main website development (1-2 chairs + N code development volunteers)
** Virtual sponsor booth (1 chair + 1 code development volunteers per 5 sponsors)
** Tutorials (1 chair + 2-3 code development volunteers)
** Workshops (probably better to have their own virtual infrastructure chairs to customize their webpages)
** Login authentication (1 chair + 2 code development volunteers + 5 helpdesk volunteers; or use a vendor company)
* The repo https://github.com/acl-org/acl-2020-virtual-conference should be a good start for future conferences. Due to the time limit, we didn’t maintain a very high code quality but it should not be very hard to spin up a new website. The most challenging part would be gathering all sitedata, including the paper basic information, QA session schedule, Zoom links, video URLs, etc.
* It would be useful to have a better pipeline for information collection. Currently we have to gather information from different chairs. It would be better if the website, the proceedings, the handbook, and the ACL Anthology all use the same information source such that everything is synchronized.
* GitHub Issue seems to be a good way to manage the tasks and distribute work to volunteers.
* It would be better to have volunteers to be involved earlier, and work with the volunteer coordinator on the timeline. It was not a very smooth transition because most volunteers can work either before or during the conference, not both. And there is a period that our chairs have to handle all emails in acl2020-virtual-helpdesk@googlegroups.com on our own on the first two days when there are lots of inquiries.
* There is many other useful information in the article [https://medium.com/@iclr_conf/gone-virtual-lessons-from-iclr2020-1743ce6164a3 “Gone Virtual: Lessons from ICLR2020"]. The section “Technical Systems and Support” is quite relevant to the virtual conference infrastructure.

2020Q1 Reports: ACL 2020

2020-02-24T20:59:34Z

Raosudha89: /* Website & Conference App Chairs */

== General Chair ==
Dan Jurafsky, Stanford University

The 58th annual meeting of the Association for Computational Linguistics (ACL) will take place in Seattle, Washington at the Hyatt Regency Seattle in downtown Seattle from July 5th through July 10th, 2020.

We have a great set of chairs! We are continuing 2019's new roles (Diversity and Inclusion chairs, Remote Presentation Chairs, AV Chairs) and adding new ones: (Sustainability chair), and we are doing well in demographic representation among our chairs (gender and region).

Following advice from last year, we have been using Slack for most intra-committee communication (and we put the Slack channel into the ACL pro space, so it can be preserved for future years), and using email only when absolutely necessary.

As usual, the growing size of the conference (both in papers and attendees) is a challenge, but both in papers and space we have been doing well (see the individual chair summaries below).

On Mar 11, we will have a site visit at the hotel in Seattle which besides Priscilla will include the General Chair, and representatives from the Program Chairs, the D&I chairs, and the AV chairs. We will also use that occasion to have a committee mtg including those folks plus the relatively large number of ACL2020 organizing committee members who are local to Seattle.

[some highlights from the below chair summaries to be added here]

== Program Chairs ==
Joyce Chai, University of Michigan

Natalie Schluter, IT University of Copenhagen, Denmark

Joel Tetreault, Dataminr, USA

== Local Organisation Chairs ==
Priscilla Rasmussen, ACL

With advice from:

Jianfeng Gao, Microsoft Research

Luke Zettlemoyer, University of Washington

== Tutorial Chairs ==

Agata Savary, University of Tours, France

Yue Zhang, Westlake University, Hangzhou, China

The call, submission, reviewing and selection of tutorials was coordinated jointly for 4 conferences: ACL, AACL-IJCNLP, COLING and EMNLP.

Before drafting the call, we collected lists of tutorials offered within the past 4 years. We analysed previous calls for tutorials and reports from tutorial chairs (from [https://aclweb.org/adminwiki/index.php?title=2016Q3_Reports:_Tutorial_Chairs 2016], [https://aclweb.org/adminwiki/index.php?title=2017Q3_Reports:_Tutorial_Chairs 2017], [https://aclweb.org/adminwiki/index.php?title=2018Q3_Reports:_Tutorial_Chairs 2018] and [http://aclweb.org/adminwiki/index.php?title=2019Q1_Reports:_ACL_2019 2019]). We consulted previous tutorial chairs with a questionnaire including questions about: the number of submissions, encouraging submissions on specific topics or from specific lecturers, the review procedure, the evaluation criteria, the post-tutorial availability of the slides/codes, and lessons learned from tutorial coordination. We also discussed the publication of slides and video recordings from future tutorials with the persons in charge of the ACL Anthology. As a result of these steps, we created two new sections for the [https://www.aclweb.org/adminwiki/index.php?title=Conference_Handbook ACL Conference Handbook] (future chairs might consider updating these documents yearly):
* the list of [https://www.aclweb.org/adminwiki/index.php?title=Past_tutorials past tutorials] at ACL, COLING, EACL, EMNLP, and NAACL in 2016-2019
* a [https://www.aclweb.org/adminwiki/index.php?title=Tutorial_chair_handbook tutorial chair handbook]

The final [https://www.aclweb.org/portal/content/joint-call-tutorial-proposals-aclaacl-ijcnlpemnlpcoling-2020 call] differs from previous calls in several aspects: (i) the expectations about tutorial proposals were made clearer, (ii) following the central ACL decision, the teachers' payment policy was replaced by a fee-waiving policy, (iii) the required submission details include two new items: diversity considerations and agreement for open access publication of slides, codes, data and video recordings, (iv) the evaluation criteria (see below) are announced.

We recruited a review committee of 19 members, including the 8 tutorial chairs and 11 external members selected for their large understanding of the NLP domain and a good experience in reviewing and/or tutorial teaching:

'''Review Committee'''
* Timothy Baldwin (University of Melbourne, Australia) - AACL-IJCNLP 2020 tutorial chair
* Daniel Beck (University of Melbourne, Australia) - COLING 2020 tutorial chair
* Emily M. Bender (University of Washington, WA, USA)
* Erik Cambria (Nanyang Technological University, Singapore)
* Gaël Dias (University of Caen Normandie, France)
* Stefan Evert (Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany)
* Yang Liu (Tsinghua University, Beijing, China)
* Agata Savary (University of Tours, France) - ACL 2020 tutorial chair
* João Sedoc (Johns Hopkins University, Baltimore, MD, USA)
* Lucia Specia (Sheffield University, UK) - COLING 2020 tutorial chair
* Xu SUN (Peking University, China)
* Yulia Tsvetkov (Carnegie Mellon University, Pittsburgh, PA, USA)
* Benjamin Van Durme (Johns Hopkins University, Baltimore, MD, USA) - EMNLP 2020 tutorial chair
* Aline Villavicencio (University of Sheffield, UK and Federal University of Rio Grande do Sul, Brazil) - EMNLP 2020 tutorial chair
* Taro Watanabe (Google, Inc., Tokyo, Japan)
* Aaron Steven White (University of Rochester, NY, USA)
* Fei Xia (University of Washington, WA, USA) - AACL-IJCNLP 2020 tutorial chair
* Yue Zhang (Westlake University, Hangzhou, China) - ACL 2020 tutorial chair
* Meishan Zhang (Tianjin University, China)

In total, we received 43 submissions for the 4 conferences. Each reviewer was assigned 6-7 proposals and each proposal received 3 reviews. The selection criteria included: clarity and preparedness, novelty or timely character of the topic, lecturers' experience, likely audience interest, open access of the teaching material, diversity aspects (multilingualism, gender, age and country of the lecturers), and compatibility with the preferred venues.
We accepted 31 proposals.

The decision making was handled via an online meeting of the 8 tutorial chairs. In particular, the selection of tutorials for each conference was done via the expression of interest of the tutorial chairs on a round-robin basis. Some slight adjustments were also performed after the meeting to better fit the authors' preferences. In total, 8, 8, 8 and 7 proposals were selected for ACL, AACL-IJCNLP, COLING and EMNLP, respectively. Upon the announcement the results, 2 of the proposals accepted for AACL-IJCNLP were withdrawn.

The submission, review, selection and collection of final material for all tutorials was handled via a dedicated SoftConf space, shared by the 4 coordinating conferences. After the selection of proposals, a separate track was created on SoftConf for each conference. The final submission page (one per conference) was set up so as to collect all the necessary data including notably: the tutorial slides, URLs for course material (if any), printable material (if any) and agreement for open access publication.

The final selection for ACL 2020 consists of the following 8 tutorials of 3 hours each (each of them had ACL as the preferred or the second preferred venue):

'''Morning Tutorials'''

'''T1: Interpretability and Analysis in Neural NLP''' (cutting-edge) 
Yonatan Belinkov, Sebastian Gehrmann and Ellie Pavlick 
While deep learning has transformed the NLP field and impacted the larger computational linguistics community, the rise of neural networks is stained by their opaque nature: It is challenging to interpret the inner workings of neural network models, and explicate their behavior. Therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of neural network models in NLP. 
This body of work is so far lacking a common framework and methodology. Moreover, approaching the analysis of modern neural networks can be difficult for newcomers to the field. This tutorial aims to fill this gap and introduce the nascent field of interpretability and analysis of neural networks in NLP. 
The tutorial covers the main lines of analysis work, such as probing classifier, behavior studies and test suites, psycholinguistic methods, visualizations, adversarial examples, and other methods. We highlight not only the most commonly applied analysis methods, but also the specific limitations and shortcomings of current approaches, in order to inform participants where to focus future efforts.

'''T2: Multi-modal Information Extraction from Text, Semi-structured, and Tabular Data on the Web''' (cutting-edge) 
Xin Luna Dong, Hannaneh Hajishirzi, Colin Lockard and Prashant Shiralkar 
The World Wide Web contains vast quantities of textual information in several forms: unstructured text, template-based semi-structured webpages (which present data in key-value pairs and lists), and tables. Methods for extracting information from these sources and converting it to a structured form have been a target of research from the natural language processing (NLP), data mining, and database communities. While these researchers have largely separated extraction from web data into different problems based on the modality of the data, they have faced similar problems such as learning with limited labeled data, defining (or avoiding defining) ontologies, making use of prior knowledge, and scaling solutions to deal with the size of the Web. 
In this tutorial we take a holistic view toward information extraction, exploring the commonalities in the challenges and solutions developed to address these different forms of text. We will explore the approaches targeted at unstructured text that largely rely on learning syntactic or semantic textual patterns, approaches targeted at semi-structured documents that learn to identify structural patterns in the template, and approaches targeting web tables which rely heavily on entity linking and type information. 
While these different data modalities have largely been considered separately in the past, recent research has started taking a more inclusive approach toward textual extraction, in which the multiple signals offered by textual, layout, and visual clues are combined into a single extraction model made possible by new deep learning approaches. At the same time, trends within purely textual extraction have shifted toward full-document understanding rather than considering sentences as independent units. With this in mind, it is worth considering the information extraction problem as a whole to motivate solutions that harness textual semantics along with visual and semi-structured layout information. We will discuss these approaches and suggest avenues for future work.

'''T3: Reviewing Natural Language Processing Research''' (introductory) 
Kevin Cohen, Karën Fort, Margot Mieskes and Aurélie Névéol 
As the demand for reviewing grows, so must the pool of reviewers. As the [http://www.livecongress.it/aol/indexSA.php?id=E2EAED7D&ticket= survey] presented by Graham Neubig at the 2019 ACL showed, a considerable number of reviewers are junior researchers, who might lack the experience and expertise necessary for high-quality reviews. Some of them might not have the environment or lack opportunities that allow them to learn the skills necessary. A tutorial on reviewing for the NLP community might increase reviewers’ confidence, as well as the quality of the reviews. This introductory tutorial will cover the goals, processes, and evaluation of reviewing research papers in natural language processing.

'''T4: Stylized Text Generation: Approaches and Applications''' (cutting-edge) 
Lili Mou and Olga Vechtomova 
Text generation has played an important role in various applications of natural language processing (NLP), and kn recent studies, researchers are paying increasing attention to modeling and manipulating the style of the generation text, which we call stylized text generation. In this tutorial, we will provide a comprehensive literature review in this direction. We start from the definition of style and different settings of stylized text generation, illustrated with various applications. Then, we present different settings of stylized generation, such as parallel supervised, style label-supervised, and unsupervised. In each setting, we delve deep into machine learning methods, including embedding learning techniques to represent style}, adversarial learning and reinforcement learning with cycle consistency to match content but to distinguish different styles. We also introduce current approaches of evaluating stylized text generation systems. We conclude our tutorial by presenting the challenges of stylized text generation and discussing future directions, such as small-data training, non-categorical style modeling, and a generalized scope of style transfer (e.g., controlling the syntax as a style).

'''Afternoon Tutorials'''

'''T5: Achieving Common Ground in Multi-modal Dialogue''' (cutting-edge) 
Malihe Alikhani and Matthew Stone 
All communication aims at achieving common ground (grounding): interlocutors can work together effectively only with mutual beliefs about what the state of the world is, about what their goals are, and about how they plan to make their goals a reality. Computational dialogue research offers some classic results on grouding, which unfortunately offer scant guidance to the design of grounding modules and behaviors in cutting-edge systems. In this tutorial, we focus on three main topic areas: 1) grounding in human-human communication; 2) grounding in dialogue systems; and 3) grounding in multi-modal interactive systems, including image-oriented conversations and human-robot interactions. We highlight a number of achievements of recent computational research in coordinating complex content, show how these results lead to rich and challenging opportunities for doing grounding in more flexible and powerful ways, and canvass relevant insights from the literature on human--human conversation. We expect that the tutorial will be of interest to researchers in dialogue systems, computational semantics and cognitive modeling, and hope that it will catalyze research and system building that more directly explores the creative, strategic ways conversational agents might be able to seek and offer evidence about their understanding of their interlocutors.

'''T6: Commonsense Reasoning for Natural Language Processing''' (introductory) 
Maarten Sap, Vered Shwartz, Antoine Bosselut, Dan Roth and Yejin Choi 
In our tutorial, we (1) outline the various types of commonsense (e.g., physical, social), and (2) discuss techniques to gather and represent commonsense knowledge, while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). We will then (3) discuss the types of commonsense knowledge captured by modern NLP systems (e.g., large pretrained language models), and (4) present ways to measure systems' commonsense reasoning abilities. We finish with (5) a discussion of various ways in which commonsense reasoning can be used to improve performance on NLP tasks, exemplified by an (6) interactive session on integrating commonsense into a downstream task.

'''T7: Integrating Ethics into the NLP Curriculum''' (introductory) 
Emily M. Bender, Dirk Hovy and Alexandra Schofield 
Our goal in this tutorial is to empower NLP researchers and practitioners with tools and resources to teach others about how to ethically apply NLP techniques. Our tutorial will present both high-level strategies for developing an ethics-oriented curriculum, based on experience and best practices, as well as specific sample exercises that can be brought to a classroom. We plan to make this a highly interactive work session culminating in a shared online resource page that pools lesson plans, assignments, exercise ideas, reading suggestions, and ideas from the attendees. We consider three primary topics with our session that frequently underlie ethical issues in NLP research: Dual use, bias and privacy. 
In this setting, a key lesson is that there is no single approach to ethical NLP: each project requires thoughtful consideration about what steps can be taken to best support people affected by that project. However, we can learn (and teach) what kinds of issues to be aware of and what kinds of strategies are available for mitigating harm. To teach this process, we apply and promote interactive exercises that provide an opportunity to ideate, discuss, and reflect. We plan to facilitate this in a way that encourages positive discussion, emphasizing the creation of ideas for the future instead of negative opinions of previous work.

'''T8: Recent Advances in Open-Domain Question Answering''' (cutting-edge) 
Danqi Chen and Scott Wen-tau Yih 
Open-domain (textual) question answering (QA), the task of finding answers to open-domain questions by searching a large collection of documents, has been a long-standing problem in NLP, information retrieval (IR) and related fields (Voorhees et al., 1999; Moldovan et al., 2000; Brill et al.,2002; Ferrucci et al., 2010). Traditional QA systems were usually constructed as a pipeline, consisting of many different components such as question processing, document/passage retrieval and answer processing. With the rapid development of neural reading comprehension (Chen, 2018), modern open-domain QA systems have been restructured by combining traditional IR techniques and neural reading comprehension models (Chen et al., 2017; Yang et al., 2019) or even implemented in a fully end-to-end fashion (Lee et al., 2019; Seo et al., 2019). While the system architecture has been drastically simplified, two technical challenges remain critical:(1) “Retriever”: finding documents that (might)contain an answer from a large collection of documents; (2) “Reader”: finding the answer in a given paragraph or a document. 
In this tutorial, we aim to provide a comprehensive and coherent overview of recent advances in this line of research. We will start by first giving a brief historical background of open-domain question answering, discussing the basic setup and core technical challenges of the research problem.The focus will then shift to modern techniques and resources proposed for open-domain QA, including the basics of latest neural reading comprehension systems, new datasets and models. The scope will also be broadened to cover the information retrieval component on how to effectively identify passages relevant to the questions. Moreover, in-depth discussions will be given on the use of traditional / neural IR modules, as well as the trade-offs between modular design and end-to-end training. If time permits, we also plan to discuss some hybrid approaches for answering questions using both text and large knowledge bases (e.g. (Sun et al., 2018)) and give a critical review on how structured data complements the information from unstructured text. 
At the end of our tutorial, we will discuss some important questions, including (1) How much progress have we made compared to the QA systems developed in the last decade?(2) What are the main challenges and limitations of cur-rent approaches? (3) How to trade off the efficiency (computational time and memory requirements) and accuracy in the deep learning era? We hope that our tutorial will not only serve as a useful resource for the audience to efficiently acquire the up-to-date knowledge, but also provide new perspectives to stimulate the advances of open-domain QA research in the next phase.

== Workshop Chairs ==

Milica Gašić, Heinrich Heine University Düsseldorf

Dilek Hakkani-Tur, Amazon Alexa AI

Saif M. Mohammad, National Research Council Canada

Ves Stoyanov, Facebook AI

== Student Research Workshop Chairs and Faculty Advisors==

'''Student Research Workshop Co-chairs'''

Rotem Dror, Technion - Israel Institute of Technology

Jiangming Liu, The University of Edinburgh

Shruti Rijhwani, Carnegie Mellon University

'''Student Research Workshop Faculty Advisors'''

Omri Abend, Hebrew University of Jerusalem

Sujian Li, Peking University

Zhou Yu, University of California, Davis

Information about the Student Research Workshop (SRW) has posted on the workshop's website: https://sites.google.com/view/acl20studentresearchworkshop/. The SRW Call for Papers has been distributed to ACL mailing lists, as well as on our official Twitter account (@acl_srw) and the ACL meeting's Twitter account (@acl_meeting).

'''Pre-submission Mentoring Phase (completed mid-February 2020)'''

Before submission to the main deadline, the SRW offered pre-submission mentoring by experienced researchers of the ACL community. The pre-submission mentoring primarily serves to provide feedback on the writing style, readability and presentation of the paper.

We recruited 30 mentors for providing pre-submission feedback. The deadline for the pre-submission phase was January 17, 2020. We had 57 pre-submissions in total.

Mentors were matched to pre-submissions according to their research areas. All mentors have already provided feedback for the submissions and it was sent to the authors mid-February 2020. The majority of mentors have also offered to participate in follow-up discussions with the authors via email until the main submission deadline.

Vouchers for one month's free use of Grammarly Premium have been sent to all the pre-submission authors. These were provided by the ACL 2020 Diversity and Inclusion Committee.

'''Main submission'''

For the main submission, the START (softconf) submission page has been set up. Currently, we have recruited 200 members of the ACL community (both students and senior researchers) to serve as the Program Committee for reviewing submissions to the SRW. We plan on inviting more PC members, as the number of submissions is likely to be larger than originally estimated.

Submission deadlines for the SRW are as follows:

* Paper submission deadline: March 6, 2020

* Review deadline: April 10, 2020

* Acceptance notification: April 15, 2020

* Camera-ready deadline: May 6, 2020

* Travel grant application deadline: to be decided.

* Travel grant notification: to be decided.

We also plan to have a post-acceptance mentoring process, for all papers accepted to the SRW.

'''Funding'''

The SRW has applied for an NSF grant of $18,000. The Don and Betty Walker international fund will also be able to provide student support. The SRW organizers have made contact with a number of industry companies to obtain sponsorship, but not yet secured additional funding. Contact has been made with the ACL 2020 sponsorship chairs and with Priscilla to investigate other funding opportunities, as well as the Student Volunteer Program, which helps students cover registration fee to the main conference.

== Audio-Video Chairs ==

Hamid Palangi, Microsoft Research, Redmond

Lianhui Qin, University of Washington

== Conference Handbook Chair ==

Nanyun Peng, University of Southern California

== Demo Chairs ==

Asli Celikyilmaz, Microsoft Research, Redmond

Shawn Wen, PolyAI

== Diversity & Inclusion (D&I) Chairs ==

Cecilia Ovesdotter Alm, Rochester Institute of Technology

Vinodkumar Prabhakaran, Google

== Local Sponsorship Chairs ==

Hoifung Poon, Microsoft

Kristina Toutanova, Google

== Publication Chairs ==

Steven Bethard, University of Arizona

Ryan Cotterrell, University of Cambridge

Rui Yan, Peking University

Starting from the style files from ACL 2019, we have produced new LaTeX style files for ACL 2020. Most of the description was retained, but the order of sections was overhauled to make sure that important information wasn't scattered so haphazardly across the document. Other improvements were also made, like using the recommended citation style consistently throughout the LaTeX source, and separating out all the LaTeX-specific stuff into clearly marked sections. The MS Word version was derived from these LaTeX versions to match as closely as possible. The LaTeX version was also posted to the Overleaf gallery. The most recent .bib file for the entire ACL Anthology was included in the style file distribution to encourage authors to use the official citations for ACL Anthology publications. All style file changes were merged into https://github.com/acl-org/acl-pub/tree/gh-pages/paper_styles.

== Publicity Chair ==

Emily M. Bender, University of Washington

=== Dissemination ===

Durable accounts for the ACL meeting on Twitter and Facebook have been created:
* https://twitter.com/aclmeeting
* https://www.facebook.com/aclmeeting/

These will be passed along to the ACL 2021 publicity chair(s) so that they don't have to build up followers separately. As of Feb 4, 2020 the Twitter account has 4,061 followers and the Facebook account has 181. We have not yet been making use of the Instagram account, but we have been using the Twitter and Facebook accounts to publicize important dates as well as blog posts. The Twitter account especially has been useful for fielding questions from the community. Calls for papers have also gone out over the ACL member portal and several mailing lists, as well as websites such as WikiCFP. (These are maintained in a spreadsheet which can be handed off to the ACL 2021 publicity chair(s)).

=== Next Steps ===

* Recruit co-chairs, especially to coordinate live-tweeting of the conference
* Contact local media for coverage
* Develop land acknowledgement in consultation with the Duwamish Tribe (on whose land the meeting will take place). The Duwamish publish this information about land acknowledgments: https://www.duwamishtribe.org/land-acknowledgement

== Remote Presentation Chairs ==

Hao Fang, Microsoft Semantic Machines

Yi Luan, Google AI Language

== Sustainability Chairs ==

Ananya Ganesh, Educational Testing Service

Klaus Zechner, Educational Testing Service

Our main goal for this new focus area is to engage the ACL community in discussions about how best to reduce the carbon footprint of future ACL conferences in order to contribute to sustainable and livable conditions on this planet.
One of the main directions we are currently envisioning is to encourage and support conference attendees in virtual participation using live streaming of conference events as air travel is the main contributor to the carbon footprint of international conferences.

== Website & Conference App Chairs ==

Sudha Rao, Microsoft Research, Redmond

Yizhe Zhang, Microsoft Research, Redmond

We are hosting the conference website on GitHub using the easily adaptable website architecture built by Nitin Madnani for NAACL 2019: https://github.com/naacl-org/naacl-hlt-2019. 
We are using the Whova event app for hosting the conference app this year similar to NAACL 2019.

== Business Office ==

Priscilla Rasmussen, ACL

2020Q1 Reports: ACL 2020

2020-02-24T20:58:53Z

Raosudha89: