<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="aji-etal-2020-neural">
    <titleInfo>
        <title>In Neural Machine Translation, What Does Transfer Learning Transfer?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alham</namePart>
        <namePart type="given">Fikri</namePart>
        <namePart type="family">Aji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikolay</namePart>
        <namePart type="family">Bogoychev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kenneth</namePart>
        <namePart type="family">Heafield</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rico</namePart>
        <namePart type="family">Sennrich</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transfer learning improves quality for low-resource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information transfer, then measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. Word embeddings play an important role in transfer learning, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and randomly generated sequences, finding even randomly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warm-up phase when training transformer models in high resource language pairs.</abstract>
    <identifier type="citekey">aji-etal-2020-neural</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.688</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.688</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>7701</start>
            <end>7710</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T In Neural Machine Translation, What Does Transfer Learning Transfer?
%A Aji, Alham Fikri
%A Bogoychev, Nikolay
%A Heafield, Kenneth
%A Sennrich, Rico
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F aji-etal-2020-neural
%X Transfer learning improves quality for low-resource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information transfer, then measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. Word embeddings play an important role in transfer learning, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and randomly generated sequences, finding even randomly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warm-up phase when training transformer models in high resource language pairs.
%R 10.18653/v1/2020.acl-main.688
%U https://aclanthology.org/2020.acl-main.688
%U https://doi.org/10.18653/v1/2020.acl-main.688
%P 7701-7710

Markdown (Informal)

[In Neural Machine Translation, What Does Transfer Learning Transfer?](https://aclanthology.org/2020.acl-main.688) (Aji et al., ACL 2020)

• In Neural Machine Translation, What Does Transfer Learning Transfer? (Aji et al., ACL 2020)

ACL

• Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. 2020. In Neural Machine Translation, What Does Transfer Learning Transfer?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7701–7710, Online. Association for Computational Linguistics.