ACL 2010: The 48th Annual Meeting of the Association for Computational
Linguistics

Review form for LONG POSITION papers

This review form is appropriate for papers that take a position on an
issue of importance to computational linguistics research but about
which there is not agreement. A paper can take a single position or
can consist of two parts, each of which takes an opposing position.


APPROPRIATENESS (1-5)

Does the paper fit in ACL 2010? Does the paper take a position on an
issue of importance to computational linguistics? (Please answer this
question in light of the desire to broaden the scope of ACL.) If the
submission is a two-part position paper (one part by each of two
authors), do the two parts take opposing viewpoints?

5: Certainly.
4: Probably.
3: Unsure.
2: Probably not.
1: Certainly not. 


CLARITY (1-5)

For the reasonably well-prepared reader, is it clear what the issue is
and what position the author(s) is(are) taking on the issue? Is the
paper well-written and well-structured?

5 = Very clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing. 


ORIGINALITY / INNOVATIVENESS (1-5)

Does this paper address an issue that has not received sufficient
attention? Or does the paper take a new position with regard to an
issue or present different arguments than have been considered by
others?

5 = Seminal: Significant new arguments or insight -- no prior work
    has presented similar arguments or supporting evidence.
4 = Novel: An intriguing position or interesting arguments that are 
    substantially different from previous work.
3 = Respectable: A nice set of arguments that represents a
    significant extension of prior arguments with respect
    to the position taken by the author(s).
2 = Pedestrian: Obvious arguments or only a minor modification of 
    arguments that have been presented before.
1 = Nothing new: Merely a repetition of well-known arguments.


SOUNDNESS / CORRECTNESS (1-5)

Does the paper present cogent arguments for the position being taken
on the issue? Are the arguments convincing? Is supporting evidence
presented?

5 = The arguments are very convincing and are supported with strong
    evidence or solid examples.
4 = Generally solid arguments, although I am not totally convinced
    and ready to accept the author's viewpoint without reservation.
3 = An interesting position that cannot be immediately dismissed. 
    The arguments are reasonable, but the claims need better support.
2 = Troublesome. There are some ideas worth considering here, but the
    arguments for the author's position lack strength.
1 = Fatally flawed.


MEANINGFUL COMPARISON (1-5)

Does the author take into account other research pertinent to the
issue under consideration? Are the references adequate? In the case of
a two-part paper with opposing positions, do the authors address the
arguments presented for the opposing position?

5 = Precise and complete consideration of other work. In the case of
    a two-part paper, each part addresses all of the arguments made
    by the opposing position. Good job given the space constraints.
4 = Mostly solid bibliography and consideration of other work. In
    the case of as two-part paper, each part addresses some of the
    arguments made by the opposing position, but it could be improved.
3 = Bibliography and comparison are somewhat helpful, but it could be
    hard for a reader to determine exactly how the position taken by
    the author relates to previous work. In the case of a two-part
    paper, only very limited reference is made to the arguments for
    the opposing position.
2 = Only partial awareness and understanding of related work, or a
    flawed consideration of it. In the case of a two-part paper,
    the two parts are independent and fail to address the opposing
    position.
1 = Little awareness of related work.


SUBSTANCE (1-5)

Does this paper have enough substance, or would it benefit from
further work?

Note that this question mainly concerns the amount of work; its
quality is evaluated in other categories.

5 = Contains more arguments and supporting evidence than would be
    expected of a position paper; goes the extra mile.
4 = Represents an appropriate amount of work for a position paper at this
    conference. (most submissions)
3 = Leaves open one or two natural questions that should have been
    pursued within the paper.
2 = Work in progress. There are some good ideas, but perhaps not
    enough substance in the paper yet.
1 = Seems thin. Not enough ideas here for a full-length paper.


IMPACT OF IDEAS OR ANALYSIS (1-5)

How significant is the position taken? If the ideas are novel, will
they also be useful or inspirational? Does the paper bring any new
insights?

5 = Will affect the field by altering other people's choice of
    research approach or methodology.
4 = Some of the ideas or arguments will spark further discussion
    and debate on the issue.
3 = Interesting but not too influential. The work may be cited, but
    is unlikely to strongly impact future work.
2 = Marginally interesting.
1 = Will have no impact on the field.


RECOMMENDATION (1-6)

There are many good submissions competing for slots at ACL 2010; how
important is it to feature this one? Will people learn a lot by
reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all
your scores above. But remember that no paper is perfect, and remember
that we want a conference full of interesting, diverse, and timely
work. If a paper has some weaknesses, but you really got a lot out of
it, feel free to fight for it. If a paper is solid but you could live
without it, let us know that you're ambivalent. Remember also that the
author has a few weeks to address reviewer comments before the
camera-ready deadline.

Should the paper be accepted or rejected?

6 = Exciting: I'd fight to get it accepted; probably would be one
              of the best papers at the conference.
5 = Strong: I'd like to see it accepted; it will be one of the
            better papers at the conference.
4 = Worthy: A good paper that is worthy of being presented at ACL.
3 = Ambivalent: OK but does not seem up to the standards of ACL.
2 = Leaning against: I'd rather not see it in the conference.
1 = Poor: I'd fight to have it rejected.


REVIEWER CONFIDENCE (1-5)

5 = Positive that my evaluation is correct. I read the paper very
    carefully and am familiar with related work.
4 = Quite sure. I tried to check the important points carefully. It's
    unlikely, though conceivable, that I missed something that should
    affect my ratings.
3 = Pretty sure, but there's a chance I missed something. Although I
    have a good feel for this area in general, I did not carefully check
    the paper's details and evidence.
2 = Willing to defend my evaluation, but it is fairly likely that I
    missed some details, didn't understand some central points, or can't
    be sure about the novelty of the work.
1 = Not my area, or paper is very hard to understand. My evaluation is
    just an educated guess.


RECOMMENDATION FOR BEST LONG PAPER AWARD (1-3)

3 = Definitely.
2 = Maybe.
1 = Definitely not.


ACCEPTANCE AS A SHORT PAPER (1-4)

If this submission is rejected as a long paper, could it be turned
into a reasonable short paper? (It is possible that some of the long
paper submissions will be accepted as short papers.) Submissions to
which this might apply include long papers in which the work is too
preliminary, or a single-author position paper which really doesn't
need 8 pages of content, but which would make a good 4-page paper.

In making this judgement please consider only whether the paper could
be reasonably reduced to 4 pages, rather than its quality. (If some
long papers are accepted as short papers, then your evaluation above
will also be taken into account.)

4 = Could easily be turned into an excellent short paper.
3 = Could be turned into a reasonable short paper with some effort.
2 = Could be turned into a short paper with a lot of effort, but key
    points of the paper may be lost.
1 = Not at all suitable for a short paper.