ACL 2010: The 48th Annual Meeting of the Association for Computational
Linguistics

Review form for SHORT CHALLENGE papers

This review form is appropriate for papers that present a challenge
to the field of computational linguistics.


APPROPRIATENESS (1-5)

Does the paper fit in ACL 2010 as a short paper? Does this paper have
any particular attributes that make it suitable for being a short
paper? (Please answer this question in light of the desire to broaden
the scope of ACL, and with reference to the examples listed in the
Call for Papers as being suitable for short papers: reporting smaller
experiments; describing work-in-progress; single-author position
papers; challenge papers; descriptions of new language resources or
evaluation methodologies; presenting negative results.)

5 = Certainly.
4 = Probably.
3 = Unsure.
2 = Probably not.
1 = Certainly not. 


CLARITY (1-5)

For the reasonably well-prepared reader, is it clear what the
challenge is, why it is important, and what would be regarded as
successfully addressing the challenge? Is the paper well-written and
well-structured?

5 = Very clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing. 


ORIGINALITY / INNOVATIVENESS (1-5)

Does the paper present a challenge that is new to the field?
Does the challenge require innovative research or new approaches?

5 = Seminal: Significant new challenge for the field -- addressing
    the challenge would require a change in research direction or
    resource allocation.
4 = Novel: An intriguing challenge to the field that is substantially 
    different from current research goals.
3 = Respectable: A minor challenge that would require some new
    research effort.
2 = Pedestrian: Current research will likely fulfill the challenge. 
1 = Nothing new: Previous research has already satisfied the challenge.


JUSTIFICATION / FORESIGHT (1-5)

Does the paper present cogent arguments for the importance and
timeliness of the challenge? Does the paper lay out a reasonable
timeframe for satisfying the challenge?

5 = The paper presents solid evidence for the importance of the
    challenge and for current research effort being allocated toward it.
4 = Generally solid arguments for addressing the challenge, but the
    paper does not totally convince me that it is reasonable to focus 
    current research on addressing the challenge.
3 = An interesting challenge that may be important to tackle at some
    time in the future, but seems inappropriate at the current time. 
2 = Troublesome. There are some ideas worth considering here, but the
    presentation of the challenge has serious deficiencies.
1 = Fatally flawed. The challenge would be impossible to address in
    the suggested time-frame given the current state-of-the-art in
    computational linguistics.


MEANINGFUL COMPARISON (1-5)

Does the paper clearly describe the current state-of-the-art relevant
to the challenge? Is it clear how addressing the challenge would
differ from existing research? Are the references adequate?

5 = Precise and complete consideration of existing work. Good job given
    the space constraints.
4 = Mostly solid bibliography and discussion of existing work, but it 
    could be improved.
3 = Bibliography and discussion of existing work are somewhat helpful
    in situating the challenge, but it could be hard for the reader
    to determine exactly how current research efforts are failing
    to address it.
2 = Only partial awareness and understanding of existing work, or a
    flawed consideration of it.
1 = Little awareness of existing work.


IMPACT OF THE CHALLENGE (1-5)

How significant is the challenge? If research is directed toward
tackling the challenge, will it have a significant positive impact
on the field of computational linguistics?

5 = Will affect the field by altering other people's choice of
    research problem or direction.
4 = The challenge will affect some aspects of future research.
3 = Interesting but not too influential. The challenge may be
    mentioned and cited, but it is unlikely to have an impact 
    on research at this time.
2 = Marginally interesting.
1 = Will have no impact on the field.


RECOMMENDATION (1-6)

There are many good submissions competing for slots at ACL 2010; how
important is it to feature this one? Will people learn a lot by
reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all
your scores above. But remember that no paper is perfect, and remember
that we want a conference full of interesting, diverse, and timely
work. If a paper has some weaknesses, but you really got a lot out of
it, feel free to fight for it. If a paper is solid but you could live
without it, let us know that you're ambivalent. Remember also that the
author has a few weeks to address reviewer comments before the
camera-ready deadline.

Should the paper be accepted or rejected?

6 = Exciting: I'd fight to get it accepted; probably would be one
              of the best short papers at the conference.
5 = Strong: I'd like to see it accepted; it will be one of the
            better short papers at the conference.
4 = Worthy: A good short paper that is worthy of being presented at ACL.
3 = Ambivalent: OK but does not seem up to the standards of ACL.
2 = Leaning against: I'd rather not see it in the conference.
1 = Poor: I'd fight to have it rejected.


REVIEWER CONFIDENCE (1-5)

5 = Positive that my evaluation is correct. I read the paper very
    carefully and am familiar with related work.
4 = Quite sure. I tried to check the important points carefully. It's
    unlikely, though conceivable, that I missed something that should
    affect my ratings.
3 = Pretty sure, but there's a chance I missed something. Although I
    have a good feel for this area in general, I did not carefully check
    the paper's details and evidence.
2 = Willing to defend my evaluation, but it is fairly likely that I
    missed some details, didn't understand some central points, or can't
    be sure about the novelty of the challenge.
1 = Not my area, or paper is very hard to understand. My evaluation is
    just an educated guess.


RECOMMENDATION FOR BEST SHORT PAPER AWARD (1-3)

3 = Definitely.
2 = Maybe.
1 = Definitely not.