<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ravichander-etal-2019-equate">
    <titleInfo>
        <title>EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Abhilasha</namePart>
        <namePart type="family">Ravichander</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aakanksha</namePart>
        <namePart type="family">Naik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Carolyn</namePart>
        <namePart type="family">Rose</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Eduard</namePart>
        <namePart type="family">Hovy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hong Kong, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle. We present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quantities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2 %), but has limited verbal reasoning capabilities (-8.1 %). We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.</abstract>
    <identifier type="citekey">ravichander-etal-2019-equate</identifier>
    <identifier type="doi">10.18653/v1/K19-1033</identifier>
    <location>
        <url>https://aclanthology.org/K19-1033</url>
    </location>
    <part>
        <date>2019-11</date>
        <extent unit="page">
            <start>349</start>
            <end>361</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
%A Ravichander, Abhilasha
%A Naik, Aakanksha
%A Rose, Carolyn
%A Hovy, Eduard
%Y Bansal, Mohit
%Y Villavicencio, Aline
%S Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
%D 2019
%8 November
%I Association for Computational Linguistics
%C Hong Kong, China
%F ravichander-etal-2019-equate
%X Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle. We present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quantities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2 %), but has limited verbal reasoning capabilities (-8.1 %). We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.
%R 10.18653/v1/K19-1033
%U https://aclanthology.org/K19-1033
%U https://doi.org/10.18653/v1/K19-1033
%P 349-361

Markdown (Informal)

[EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference](https://aclanthology.org/K19-1033) (Ravichander et al., CoNLL 2019)

• EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference (Ravichander et al., CoNLL 2019)

ACL

• Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. Association for Computational Linguistics.