ACL 2010: The 48th Annual Meeting of the Association for Computational
Linguistics

Review form for LONG THEORETICAL research papers

This review form is appropriate for research papers whose primary
contribution is a theoretical one. Examples of such papers might be:
proposal for a new grammar formalism; proof that a particular
linguistic formalism has a certain generative capacity; description of
a new parsing algorithm and analysis of its asymptotic behaviour; a
novel logical analysis of an aspect of discourse.


APPROPRIATENESS (1-5)

Does the paper fit in ACL 2010? (Please answer this question in light
of the desire to broaden the scope of the research areas represented
at ACL.)

5: Certainly.
4: Probably.
3: Unsure.
2: Probably not.
1: Certainly not. 


CLARITY (1-5)

For the reasonably well-prepared reader, is it clear what was done and
why? Is the paper well-written and well-structured?

5 = Very clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing. 


ORIGINALITY / INNOVATIVENESS (1-5)

How original is the approach? Does this paper break new ground in
topic or content? How exciting and innovative is the research it
describes?

Note that a paper could score high for originality even if the results
do not show a convincing benefit.

5 = Seminal: Significant new theoretical advance that addresses a
    limitation of prior theories or formalisms -- no prior research
    has provided something similar.
4 = Creative: An intriguing theoretical result or analysis that is
    substantially different from previous research.
3 = Respectable: A nice research contribution that represents a
    notable extension of prior work.
2 = Pedestrian: Obvious, or a minor improvement on familiar theories
    or formalisms.
1 = Significant portions have actually been done before or done
    better.


SOUNDNESS / CORRECTNESS (1-5)

First, is the theoretical approach sound and well-chosen?  Second, can
one trust the claims of the paper -- for example are they supported by
an appropriate proof or analysis?

5 = The theoretical approach is very apt, and the claims are
    convincingly supported.
4 = Generally solid work, although there are some aspects of the
    approach I am not sure about or the analysis could be stronger.
3 = Fairly reasonable work. The approach is not bad, and at least the
    main claims are probably correct, but I am not entirely ready to
    accept them (based on the material in the paper).
2 = Troublesome. There are some ideas worth salvaging here, but the
    work should really have been done or analysed differently.
1 = Fatally flawed. 


MEANINGFUL COMPARISON (1-5)

Does the author make clear where the work sits with respect to
existing literature? Are the references adequate? Are the benefits of
the theoretical approach well-supported?

5 = Precise and complete comparison with related work.
4 = Mostly solid bibliography and comparison, but there are a few
    additional references that should be included.
3 = Bibliography and comparison are somewhat helpful, but it could be
    hard for a reader to determine exactly how this work relates to
    previous work.
2 = Only partial awareness and understanding of related work, or a
    flawed comparison with other work.
1 = Little awareness of related work.


SUBSTANCE (1-5)

Does this paper have enough substance, or would it benefit from more
ideas, results or analysis?

Note that this question mainly concerns the amount of work; its
quality is evaluated in other categories.

5 = Contains more ideas, results or analysis than most publications in
    this conference; goes the extra mile.
4 = Represents an appropriate amount of work for a publication in this
    conference. (most submissions)
3 = Leaves open one or two natural questions that should have been
    pursued within the paper.
2 = Work in progress. There are enough good ideas, but perhaps not
    enough substance overall.
1 = Seems thin. Not enough ideas here for a full-length paper.


IMPACT OF IDEAS OR RESULTS (1-5)

How significant is the work described? If the ideas are novel, will
they also be useful or inspirational? If the results are sound, are
they also important? Does the paper bring new insights into the nature
of the problem?

5 = Will affect the field by altering other people's choice of
    research topics or basic approach.
4 = Some of the ideas or results will substantially help other
    people's ongoing research.
3 = Interesting but not too influential. The work will be cited, but
    mainly for comparison or as a source of minor contributions.
2 = Marginally interesting. May or may not be cited.
1 = Will have no impact on the field.


RECOMMENDATION (1-6)

There are many good submissions competing for slots at ACL 2010; how
important is it to feature this one? Will people learn a lot by
reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all
your scores above. But remember that no paper is perfect, and remember
that we want a conference full of interesting, diverse, and timely
work. If a paper has some weaknesses, but you really got a lot out of
it, feel free to fight for it. If a paper is solid but you could live
without it, let us know that you're ambivalent. Remember also that the
author has a few weeks to address reviewer comments before the
camera-ready deadline.

Should the paper be accepted or rejected?

6 = Exciting: I'd fight to get it accepted; probably would be one
              of the best papers at the conference.
5 = Strong: I'd like to see it accepted; it will be one of the
            better papers at the conference.
4 = Worthy: A good paper that is worthy of being presented at ACL.
3 = Ambivalent: OK but does not seem up to the standards of ACL.
2 = Leaning against: I'd rather not see it in the conference.
1 = Poor: I'd fight to have it rejected.


REVIEWER CONFIDENCE (1-5)

5 = Positive that my evaluation is correct. I read the paper very
    carefully and am familiar with related work.  
4 = Quite sure. I tried to check the important points carefully. It's
    unlikely, though conceivable, that I missed something that should
    affect my ratings.
3 = Pretty sure, but there's a chance I missed something. Although I
    have a good feel for this area in general, I did not carefully check
    the paper's details, e.g., the math, experimental design, or novelty.
2 = Willing to defend my evaluation, but it is fairly likely that I
    missed some details, didn't understand some central points, or can't
    be sure about the novelty of the work.
1 = Not my area, or paper is very hard to understand. My evaluation is
    just an educated guess.


RECOMMENDATION FOR BEST LONG PAPER AWARD (1-3)

3 = Definitely.
2 = Maybe.
1 = Definitely not.


ACCEPTANCE AS A SHORT PAPER (1-4)

If this submission is rejected as a long paper, could it be turned
into a reasonable short paper? (It is possible that some of the long
paper submissions will be accepted as short papers.) Submissions to
which this might apply include long papers in which the work is too
preliminary, and work in progress; or description of a new formalism
which really doesn't need 8 pages of content, but which would make a
good 4-page paper.

In making this judgement please consider only whether the paper could
be reasonably reduced to 4 pages, rather than its quality. (If some
long papers are accepted as short papers, then your evaluation above
will also be taken into account.)

4 = Could easily be turned into an excellent short paper.
3 = Could be turned into a reasonable short paper with some effort.
2 = Could be turned into a short paper with a lot of effort, but key
    points of the paper may be lost.
1 = Not at all suitable for a short paper.