Ahakim: New page: The ACL workshop co-chairs (John Carroll and Hal Daume III) worked together with the EMNLP workshop chair (Marie Candito) to jointly select workshops for ACL/EMNLP in a single call. The i...

2011-06-14T22:12:32Z

New page: The ACL workshop co-chairs (John Carroll and Hal Daume III) worked together with the EMNLP workshop chair (Marie Candito) to jointly select workshops for ACL/EMNLP in a single call. The i...

New page

The ACL workshop co-chairs (John Carroll and Hal Daume III) worked
together with the EMNLP workshop chair (Marie Candito) to jointly
select workshops for ACL/EMNLP in a single call. The initial call
went out on 16 September 2010, and a final call went out on 4 November
2010. The deadline for submission was 15 Nov 2010. Workshops were
asked to note whether which conferences were acceptable (ACL versus
EMNLP) for their workshop and, if both, which was preferable (if
either). Workshop vetting was through a gmail account we created for
this process.

In all, we received 32 proposals. Of these, 8 exclusively wanted ACL
and 2 exclusively wanted EMNLP. Of those 22 that were okay with
either, 11 expressed a preference for ACL and 6 expressed a preference
for EMNLP (5 did not express a preference). Overall, it was fairly
balanced, which eased the selection process. We made decisions about
how many workshops to accept based on discussion with Priscilla
Rasmussen and the Conference Chair, Dekang Lin. In the end, 15 were
accepted for ACL, of which three were two days long (CoNLL, BioNLP and
Annotation); and 7 for EMNLP). For the most part, these were the
workshops we felt were the strongest. Unlike 2010, we chose to take
into account attendance and did not accept every workshop that would
fit. Each workshop was scored (scale of 1 to 3) independently by the
three workshop co-chairs. For the most part, the highest averaging
workshops were accepted, though some shuffling was done on a phone
call between the three chairs. Having the ability to route them
either to ACL or EMNLP was very helpful to avoid overlapping workshop
topics.

The following workshops were accepted, together with their
preregistration numbers as of 12 June: CoNLL (117); BioNLP (84);
SSST-5 (72); Social Media (70); Educ Applications (60); Comparable
Corpora (50); Text-to-Text (46); Multiword (44); WASSA (43);
TextGraphs-6 (43); The LAW V (35); RELMS (34); DiSCo (31);
Summarization (31); CMCL (29); LaTeCH (29). A post-hoc analysis
suggests that our overall ratings were only moderate predictors of
attendance (rho=0.2626, tau=0.1015). However, our own goal in
acceptance was not just to maximize attendance, but also to encourage
nascent areas.

Overall the selection process worked well. It enabled us to do things
like reduce redundancy, have a bigger pool of proposals, and a bigger
selection committee. As a result, we recommend a joint model for
workshop selection for 2012. However, there were a few issues that
arose that we suggest are considered in the future.

* Involve the main conference chairs to define general guidelines on how to weight the various criteria to use for workshop selection: quality of proposal, originality, expected audience, experience of organizers ... Especially things like "Workshop X ran last year and had good attendance. Workshop Y is new but hard to predict. How do we compare the two?"

* Ensure that all notifications of rejection include a few lines of justification, perhaps with a suggestion for re-submission to future conferences. (We omitted to do this, and Hal got a number of queries from rejected workshops asking for feedback.)

* Finally, there was the issue of a workshop that was rejected, but was still very keen to go ahead and so decided to organize separately as a co-located event. We think this is a fine thing to do, so long as it does not organize during the official workshop session. However, we suggest that the ACL board consider creating a policy that strongly discourages co-located events from overlapping with the workshops. (However, it turned out not to be a problem since the workshop ended up not running anyway.)

One of the biggest headaches for the workshop process had to do with
handling the workshop proceedings. Some of the groups running the
workshops had not previously used ACLPUB and/or the version integrated
into START, and were inadequately prepared to deal with all the
nuances. The workshop co-chairs had to very quickly try to figure out
how to handle this, with numerous emails back and forth to the
publications chair. This problem was exacerbated by several workshop
organizers not having the right tools to spot some types of
incorrectly formatted papers (e.g., not all fonts embedded), and a
short timeline (the publications timeline for workshops was very
compressed, in comparison to the usual ACL timeline). We think the
dividing line for responsibility between the workshop co-chairs and
publications chair needs either to be more tightly defined in the ACL
guidelines, or established by negotiation between the workshop
co-chairs and publications chair in advance. We suggest that since
the publications chair is likely the most knowledgable in this area,
workshop publications be handled entirely by the publications chair.

2011Q3 Reports: Workshop Chairs - Revision history

Ahakim: New page: The ACL workshop co-chairs (John Carroll and Hal Daume III) worked together with the EMNLP workshop chair (Marie Candito) to jointly select workshops for ACL/EMNLP in a single call. The i...