<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="talman-etal-2021-nli">
    <titleInfo>
        <title>NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Aarne</namePart>
        <namePart type="family">Talman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marianna</namePart>
        <namePart type="family">Apidianaki</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stergios</namePart>
        <namePart type="family">Chatzikyriakidis</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jörg</namePart>
        <namePart type="family">Tiedemann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-may 31–2 jun</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Dobnik</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lilja</namePart>
            <namePart type="family">Øvrelid</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Linköping University Electronic Press, Sweden</publisher>
            <place>
                <placeTerm type="text">Reykjavik, Iceland (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pre-trained neural language models give high performance on natural language inference (NLI) tasks. But whether they actually understand the meaning of the processed sequences is still unclear. We propose a new diagnostics test suite which allows to assess whether a dataset constitutes a good testbed for evaluating the models’ meaning understanding capabilities. We specifically apply controlled corruption transformations to widely used benchmarks (MNLI and ANLI), which involve removing entire word classes and often lead to non-sensical sentence pairs. If model accuracy on the corrupted data remains high, then the dataset is likely to contain statistical biases and artefacts that guide prediction. Inversely, a large decrease in model accuracy indicates that the original dataset provides a proper challenge to the models’ reasoning capabilities. Hence, our proposed controls can serve as a crash test for developing high quality data for NLI tasks.</abstract>
    <identifier type="citekey">talman-etal-2021-nli</identifier>
    <location>
        <url>https://aclanthology.org/2021.nodalida-main.28</url>
    </location>
    <part>
        <date>2021-may 31–2 jun</date>
        <extent unit="page">
            <start>276</start>
            <end>287</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance
%A Talman, Aarne
%A Apidianaki, Marianna
%A Chatzikyriakidis, Stergios
%A Tiedemann, Jörg
%Y Dobnik, Simon
%Y Øvrelid, Lilja
%S Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)
%D 2021
%8 may 31–2 jun
%I Linköping University Electronic Press, Sweden
%C Reykjavik, Iceland (Online)
%F talman-etal-2021-nli
%X Pre-trained neural language models give high performance on natural language inference (NLI) tasks. But whether they actually understand the meaning of the processed sequences is still unclear. We propose a new diagnostics test suite which allows to assess whether a dataset constitutes a good testbed for evaluating the models’ meaning understanding capabilities. We specifically apply controlled corruption transformations to widely used benchmarks (MNLI and ANLI), which involve removing entire word classes and often lead to non-sensical sentence pairs. If model accuracy on the corrupted data remains high, then the dataset is likely to contain statistical biases and artefacts that guide prediction. Inversely, a large decrease in model accuracy indicates that the original dataset provides a proper challenge to the models’ reasoning capabilities. Hence, our proposed controls can serve as a crash test for developing high quality data for NLI tasks.
%U https://aclanthology.org/2021.nodalida-main.28
%P 276-287

Markdown (Informal)

[NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance](https://aclanthology.org/2021.nodalida-main.28) (Talman et al., NoDaLiDa 2021)

• NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance (Talman et al., NoDaLiDa 2021)

ACL

• Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, and Jörg Tiedemann. 2021. NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 276–287, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.