ACL 2010: The 48th Annual Meeting of the Association for Computational
Linguistics

Review form for LONG CHALLENGE papers

This review form is appropriate for papers that present a challenge
to the field of computational linguistics.


APPROPRIATENESS (1-5)

Does the paper fit in ACL 2010? (Please answer this question in light
of the desire to broaden the scope of ACL.)

5: Certainly.
4: Probably.
3: Unsure.
2: Probably not.
1: Certainly not. 


CLARITY (1-5)

For the reasonably well-prepared reader, is it clear what the
challenge is, why it is important, and what would be regarded as
successfully addressing the challenge? Is the paper well-written and
well-structured?

5 = Very clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing. 


ORIGINALITY / INNOVATIVENESS (1-5)

Does the paper present a challenge that is new to the field?
Does the challenge require innovative research or new approaches?

5 = Seminal: Significant new challenge for the field -- addressing
    the challenge would require a change in research direction or
    resource allocation.
4 = Novel: An intriguing challenge to the field that is substantially 
    different from current research goals.
3 = Respectable: A minor challenge that would require some new
    research effort.
2 = Pedestrian: Current research will likely fulfill the challenge. 
1 = Nothing new: Previous research has already satisfied the challenge.


JUSTIFICATION / FORESIGHT (1-5)

Does the paper present cogent arguments for the importance and
timeliness of the challenge? Does the paper lay out a reasonable
timeframe for satisfying the challenge?

5 = The paper presents solid evidence for the importance of the
    challenge and for current research effort being allocated toward it.
4 = Generally solid arguments for addressing the challenge, but the
    paper does not totally convince me that it is reasonable to focus 
    current research on addressing the challenge.
3 = An interesting challenge that may be important to tackle at some
    time in the future, but seems inappropriate at the current time. 
2 = Troublesome. There are some ideas worth considering here, but the
    presentation of the challenge has serious deficiencies.
1 = Fatally flawed. The challenge would be impossible to address in
    the suggested time-frame given the current state-of-the-art in
    computational linguistics.


MEANINGFUL COMPARISON (1-5)

Does the paper clearly describe the current state-of-the-art relevant
to the challenge? Is it clear how addressing the challenge would
differ from existing research? Are the references adequate?

5 = Precise and complete consideration of existing work. Good job given
    the space constraints.
4 = Mostly solid bibliography and discussion of existing work, but it 
    could be improved.
3 = Bibliography and discussion of existing work are somewhat helpful
    in situating the challenge, but it could be hard for the reader
    to determine exactly how current research efforts are failing
    to address it.
2 = Only partial awareness and understanding of existing work, or a
    flawed consideration of it.
1 = Little awareness of existing work.


SUBSTANCE (1-5)

Does this paper have enough substance, or would it benefit from
further work?

Note that this question mainly concerns the amount of work; its
quality is evaluated in other categories.

5 = Does a better job of laying out the challenge, justifying it,
    and situating it with respect to existing research than would be
    expected of a challenge paper; goes the extra mile.
4 = Represents an appropriate amount of work for a challenge paper at this
    conference. (most submissions)
3 = Leaves open one or two natural questions regarding the challenge 
    that should have been pursued within the paper.
2 = Work in progress. There are some good ideas, but perhaps not
    enough substance in the paper yet.
1 = Seems thin. Not enough ideas here for a challenge paper.


IMPACT OF THE CHALLENGE (1-5)

How significant is the challenge? If research is directed toward
tackling the challenge, will it have a significant positive impact
on the field of computational linguistics?

5 = Will affect the field by altering other people's choice of
    research problem or direction.
4 = The challenge will affect some aspects of future research.
3 = Interesting but not too influential. The challenge may be
    mentioned and cited, but it is unlikely to have an impact 
    on research at this time.
2 = Marginally interesting.
1 = Will have no impact on the field.


RECOMMENDATION (1-6)

There are many good submissions competing for slots at ACL 2010; how
important is it to feature this one? Will people learn a lot by
reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all
your scores above. But remember that no paper is perfect, and remember
that we want a conference full of interesting, diverse, and timely
work. If a paper has some weaknesses, but you really got a lot out of
it, feel free to fight for it. If a paper is solid but you could live
without it, let us know that you're ambivalent. Remember also that the
author has a few weeks to address reviewer comments before the
camera-ready deadline.

Should the paper be accepted or rejected?

6 = Exciting: I'd fight to get it accepted; probably would be one
              of the best papers at the conference.
5 = Strong: I'd like to see it accepted; it will be one of the
            better papers at the conference.
4 = Worthy: A good paper that is worthy of being presented at ACL.
3 = Ambivalent: OK but does not seem up to the standards of ACL.
2 = Leaning against: I'd rather not see it in the conference.
1 = Poor: I'd fight to have it rejected.


REVIEWER CONFIDENCE (1-5)

5 = Positive that my evaluation is correct. I read the paper very
    carefully and am familiar with related work.
4 = Quite sure. I tried to check the important points carefully. It's
    unlikely, though conceivable, that I missed something that should
    affect my ratings.
3 = Pretty sure, but there's a chance I missed something. Although I
    have a good feel for this area in general, I did not carefully check
    the paper's details and evidence.
2 = Willing to defend my evaluation, but it is fairly likely that I
    missed some details, didn't understand some central points, or can't
    be sure about the novelty of the challenge.
1 = Not my area, or paper is very hard to understand. My evaluation is
    just an educated guess.


RECOMMENDATION FOR BEST LONG PAPER AWARD (1-3)

3 = Definitely.
2 = Maybe.
1 = Definitely not.


ACCEPTANCE AS A SHORT PAPER (1-4)

If this submission is rejected as a long paper, could it be turned
into a reasonable short paper? (It is possible that some of the long
paper submissions will be accepted as short papers.) Submissions to
which this might apply include long papers in which the work is too
preliminary, or a challenge paper which really doesn't 
need 8 pages of content, but which would make a good 4-page paper.

In making this judgement please consider only whether the paper could
be reasonably reduced to 4 pages, rather than its quality. (If some
long papers are accepted as short papers, then your evaluation above
will also be taken into account.)

4 = Could easily be turned into an excellent short paper.
3 = Could be turned into a reasonable short paper with some effort.
2 = Could be turned into a short paper with a lot of effort, but key
    points of the paper may be lost.
1 = Not at all suitable for a short paper.