<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="meghwal-etal-2020-wikipedia">
    <titleInfo>
        <title>Can Wikipedia Categories Improve Masked Language Model Pretraining?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Diksha</namePart>
        <namePart type="family">Meghwal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Katharina</namePart>
        <namePart type="family">Kann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iacer</namePart>
        <namePart type="family">Calixto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stanislaw</namePart>
        <namePart type="family">Jastrzebski</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fourth Widening Natural Language Processing Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Rossana</namePart>
            <namePart type="family">Cunha</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Samira</namePart>
            <namePart type="family">Shaikh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Erika</namePart>
            <namePart type="family">Varis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ryan</namePart>
            <namePart type="family">Georgi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alicia</namePart>
            <namePart type="family">Tsai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonios</namePart>
            <namePart type="family">Anastasopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Khyathi</namePart>
            <namePart type="given">Raghavi</namePart>
            <namePart type="family">Chandu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Seattle, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.</abstract>
    <identifier type="citekey">meghwal-etal-2020-wikipedia</identifier>
    <identifier type="doi">10.18653/v1/2020.winlp-1.19</identifier>
    <location>
        <url>https://aclanthology.org/2020.winlp-1.19</url>
    </location>
    <part>
        <date>2020-07</date>
        <detail type="page"><number>78</number></detail>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can Wikipedia Categories Improve Masked Language Model Pretraining?
%A Meghwal, Diksha
%A Kann, Katharina
%A Calixto, Iacer
%A Jastrzebski, Stanislaw
%Y Cunha, Rossana
%Y Shaikh, Samira
%Y Varis, Erika
%Y Georgi, Ryan
%Y Tsai, Alicia
%Y Anastasopoulos, Antonios
%Y Chandu, Khyathi Raghavi
%S Proceedings of the Fourth Widening Natural Language Processing Workshop
%D 2020
%8 July
%I Association for Computational Linguistics
%C Seattle, USA
%F meghwal-etal-2020-wikipedia
%X Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.
%R 10.18653/v1/2020.winlp-1.19
%U https://aclanthology.org/2020.winlp-1.19
%U https://doi.org/10.18653/v1/2020.winlp-1.19
%P 78

Markdown (Informal)

[Can Wikipedia Categories Improve Masked Language Model Pretraining?](https://aclanthology.org/2020.winlp-1.19) (Meghwal et al., WiNLP 2020)

• Can Wikipedia Categories Improve Masked Language Model Pretraining? (Meghwal et al., WiNLP 2020)

ACL

• Diksha Meghwal, Katharina Kann, Iacer Calixto, and Stanislaw Jastrzebski. 2020. Can Wikipedia Categories Improve Masked Language Model Pretraining?. In Proceedings of the Fourth Widening Natural Language Processing Workshop, page 78, Seattle, USA. Association for Computational Linguistics.