SandraCarberry: Removing all content from page

2010-06-21T22:35:35Z

Removing all content from page

SandraCarberry: New page: ACL Program Chair Report Sandra Carberry and Stephen Clark June 21, 2010 ACL 2010 received 987 submissions, a record number for the conf...

2010-06-21T22:33:32Z

New page: ACL Program Chair Report Sandra Carberry and Stephen Clark June 21, 2010 ACL 2010 received 987 submissions, a record number for the conf...

New page

ACL Program Chair Report
Sandra Carberry and Stephen Clark
June 21, 2010

ACL 2010 received 987 submissions, a record number for the
conference. This was a surprise, since the submission deadline was
identical for both long and short papers. Thus we expected that the
total number of submissions would be lower than for ACL 2009, where
authors of rejected long papers had the opportunity to resubmit them
as revised short papers.

655 papers were submitted to ACL 2010 as long papers, and 332
papers were submitted as short papers. After filtering out those
papers that did not satify the submission requirements, for example
exceeded the length limitations or were not anonymous, 956 papers
were distributed to the Area Chairs for reviewing, of which 646 were
long submissions and 310 were short submissions. Due to the large
number of submissions, a fifth parallel session was added to the
program. We accepted 7 long papers as short papers, since these were
considered worthy of acceptance, but not as long papers, and it was
thought that the content could be presented as short papers. Counting
these 7 submissions as short papers, the overall acceptance rate was
25% for long papers and 22% for short papers. The submissions roughly
fell into the following categories, with an attempt made to prevent
some categories, for example Machine Translation, from becoming too
large (which would have placed an unreasonable burden on the MT Area
Chairs):

Long Paper Submissions:
AREA SUBMITTED ACCEPT-LONG ACCEPT-SHORT %-ACCEPTED
---- --------- ----------- ------------ ----------
Bioinformatics 10 1 10.0%
Discourse 38 11 28.9%
Formal semantics 19 6 31.6%
Generation/summarization 39 10 1 28.2%
Information extraction 44 8 18.2%
Information retrieval 24 6 25.0%
Lexical semantics 59 16 27.1%
Machine learning 57 13 22.8%
Machine translation 64 15 23.4%
Mathematical linguistics 23 10 43.5%
Multimodal 13 4 30.8%
Parsing 68 16 2 26.5%
Psycholinguistics 14 5 35.7%
Question answering 22 5 2 31.8%
Resources and evaluation 28 8 28.6%
Sentiment analysis 46 9 1 21.7%
Speech 18 4 22.2%
Tagging 37 10 27.0%
Text mining 23 7 1 34.8%

Short Paper Submissions:
AREA SUBMITTED ACCEPTED %-ACCEPTED
---- --------- -------- ----------
Bioinformatics 2 0 0.0%
Discourse 21 5 23.8%
Formal semantics 4 1 25.0%
Generation/summarization 17 2 11.8%
Information extraction 11 3 27.3%
Information retrieval 12 1 8.3%
Lexical semantics 30 6 20.0%
Machine learning 30 8 26.7%
Machine translation 38 10 26.3%
Mathematical linguistics 6 0 0.0%
Multimodal 5 1 20.0%
Parsing 24 5 20.8%
Psycholinguistics 5 1 20.0%
Question answering 18 2 11.1%
Resources and evaluation 25 5 20.0%
Sentiment analysis 26 6 23.1%
Speech 15 3 20.0%
Tagging 14 3 21.4%
Text mining 7 1 14.3%

The ACL 2010 program contains a wide variety of papers, ranging
from theoretical papers to analysis papers to empirical papers. Of
particular interest is the presence of 3 challenge papers and 3 survey
papers on the ACL 2010 program; unfortunately, although we sought
position papers, none of the submissions in this category were judged
to warrant acceptance.

ACL 2010 will award 3 best paper prizes: best long paper, best long
paper by a student author, and best short paper. The selection of the
prize recipients was made by a small panel consisting of selected area
chairs and other senior members of the research community. Although
the selection of a single paper in each prize category was difficult,
we chose to award only one prize in each category since we felt that
it lends more prestige to the prize than if the prize were shared by
several papers.

All submissions that were accepted as long papers were allocated 9
pages of content in the proceedings, with the authors being granted an
extra page for the final version compared to the 8 pages of content
allowed at submission time. All submissions that were accepted as
short papers were allocated 5 pages of content in the proceedings,
again with an extra page allowed for the final version compared to the
4 pages at submission time. In both cases, authors were allowed an
unlimited number of extra pages for references. The extra page of
content in the final papers was an experiment this year, in order to
allow authors to better improve their papers by addressing the
comments and suggestions of the reviewers, without having to cut
essential parts of their original submissions. Long papers will be
presented either as 25 minute oral talks or as 10 minute oral talks
followed by a poster presentation. Short papers will be presented
either as 10 minute oral talks followed by a poster presentation or
just as a poster presentation. The decision about presentation mode
was made by the program chairs based on the quality of the paper,
input from the area chairs, and our own judgement about how the paper
might best be presented.

In order to attract and appropriately review a wider variety of
papers, we experimented this year with different review forms for
the different categories of papers. For example, the review criteria
(and thus the questions on the review form) for theoretical papers were
different from those for empirical papers. There were 10 different
review categories (analysis, challenge, empirical, negative result,
paradigms, position, resources, survey, systems, and theoretical), which,
combined with the long/short distinction resulted in 20 different review
forms; the set of review forms can be found at:
http://acl2010.org/reviewforms.html
In retrospect, we believe that the different review forms helped
immensely in obtaining appropriate reviews for the different types
of papers.

However, despite the submission page explicitly directing authors to
examine the review forms posted on the ACL web site before selecting their
review category, it is clear that authors did not do so. Thus the program
chairs and area chairs examined every paper and changed the review
category for papers where an inappropriate category had been selected.
(Note that the Call for Papers stated that the Program Chairs reserved
the right to change the review category.) This was an enormous amount
of work. If this experiment is continued next year, we recommend that
the submission web page do one of the following:
1. contain a question associated with each category (such
as "Does this paper present a system that has been deployed
in an industrial or research setting and includes reports
of tests with actual users?" for a systems paper), where
the user must reply "Yes" in order to select that submission
category.
2. pop up the review form for the category that the author selects,
along with the statement "I have read this entire review form and
believe that it is appropriate for this submission". The author
would need to respond "Yes" in order to finalize the category
selection.
We would like to note that Rich Gerber was extremely helpful
in modifying the START system to accommodate our needs. For example,
the START system has been modified so that when a reviewer clicks
on the review form for a paper, he or she gets the appropriate
review form for that category of paper.

Most of our Area Chairs and reviewers were outstanding. However,
a few reviewers did not provide the high-quality reviews that we
expect for ACL submissions. It appears that these individuals often
review for ACL. Although we do not have a solution, we do believe
that some mechanism should be developed for keeping track of
reviewers whose work is below norm so that future program chairs
and area chairs avoid inviting them to be part of the program
committee.

While we hope that at least one author of each accepted paper will
attend the conference and present the paper, it appears that in a few
cases, the authors may not be making a serious attempt to ensure that
at least one author is present at the conference. Some conferences
require that at least one author register for the conference in order
for a paper to appear in the proceedings; other conferences prohibit
authors from submitting to the conference in the future if at least
one author of an accepted paper does not attend the conference. We
recognize that these are either difficult to enforce or somewhat
draconian measures, we do suggest that ACL adopt the policy that
the submission page contain the following statement:
If this submission is accepted for the ACL conference, we
commit to at least one of the authors attending the conference
and presenting the paper during the main technical session
on <dates>.
with the requirement that the authors click "YES" in order for
the submission to be successful.

We also encountered problems with authors expecting that the
submission deadline would not be enforced. We had several authors
(some who are senior researchers and active in ACL) who were very
unhappy that the START system refused their papers after the
submission deadline. One spoke of a "traditional 1 hour or more grace
period" for submissions. After consultation with the ACL Exec, we did
not accept these late submissions since we felt it was the only way to
be fair to all authors. (We did keep the START system open for 15
minutes after the deadline in order to avoid cutting off authors who
were in the process of submitting prior to the deadline.) We most
strongly recommend that ACL adopt the following (or something similar)
as policy, that it be included in the Call for Papers in future years,
and that it be added to the ACL conference handbook:
"The ACL submission deadline will be extended only in the event
that the START system crashes near the deadline. The START system
will automatically shut down at the deadline, and it is ACL policy
that late submissions will not be allowed."
Without such a policy, exceptions that are informally granted by
Program Chairs are unfair to other authors (who are not aware of the
possibility of an exception) and cause problems for subsequent Program
Chairs.

ACL 2010 will have two outstanding invited talks:
Andrei Broder: vice-president of Yahoo and both an ACM Fellow
and an IEEE Fellow. He will present a talk on
the emerging field of computational advertising,
with an emphasis on issues relevant to computational
linguistics and natural language processing.
Zenzi Griffin: professor of psychology at the University of Texas
at Austin. She will present a talk on the
psycholinguistics of social interaction, with an
emphasis on issues in language processing.
Our goal was to select invited speakers whose talks would be related
to computational linguistics, but would broaden the perspective of
the conference attendees. In addition, we sought individuals who
had a reputation for excellent presentational skills. We believe
that both of these individuals satisfy these criteria and will
present excellent and exciting invited talks at ACL.

2010Q3 Reports: Program Chairs - Revision history

SandraCarberry: Removing all content from page

SandraCarberry: New page: ACL Program Chair Report Sandra Carberry and Stephen Clark June 21, 2010 ACL 2010 received 987 submissions, a record number for the conf...