<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="jat-etal-2019-relating">
    <titleInfo>
        <title>Relating Simple Sentence Representations in Deep Neural Networks and the Brain</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sharmistha</namePart>
        <namePart type="family">Jat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hao</namePart>
        <namePart type="family">Tang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Partha</namePart>
        <namePart type="family">Talukdar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Mitchell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2019-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Korhonen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Traum</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lluís</namePart>
            <namePart type="family">Màrquez</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Florence, Italy</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>What is the relationship between sentence representations learned by deep recurrent models against those encoded by the brain? Is there any correspondence between hidden layers of these recurrent models and brain regions when processing sentences? Can these deep models be used to synthesize brain data which can then be utilized in other extrinsic tasks? We investigate these questions using sentences with simple syntax and semantics (e.g., The bone was eaten by the dog.). We consider multiple neural network architectures, including recently proposed ELMo and BERT. We use magnetoencephalography (MEG) brain recording data collected from human subjects when they were reading these simple sentences. Overall, we find that BERT’s activations correlate the best with MEG brain data. We also find that the deep network representation can be used to generate brain data from new sentences to augment existing brain data. To the best of our knowledge, this is the first work showing that the MEG brain recording when reading a word in a sentence can be used to distinguish earlier words in the sentence. Our exploration is also the first to use deep neural network representations to generate synthetic brain data and to show that it helps in improving subsequent stimuli decoding task accuracy.</abstract>
    <identifier type="citekey">jat-etal-2019-relating</identifier>
    <identifier type="doi">10.18653/v1/P19-1507</identifier>
    <location>
        <url>https://aclanthology.org/P19-1507</url>
    </location>
    <part>
        <date>2019-07</date>
        <extent unit="page">
            <start>5137</start>
            <end>5154</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Relating Simple Sentence Representations in Deep Neural Networks and the Brain
%A Jat, Sharmistha
%A Tang, Hao
%A Talukdar, Partha
%A Mitchell, Tom
%Y Korhonen, Anna
%Y Traum, David
%Y Màrquez, Lluís
%S Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
%D 2019
%8 July
%I Association for Computational Linguistics
%C Florence, Italy
%F jat-etal-2019-relating
%X What is the relationship between sentence representations learned by deep recurrent models against those encoded by the brain? Is there any correspondence between hidden layers of these recurrent models and brain regions when processing sentences? Can these deep models be used to synthesize brain data which can then be utilized in other extrinsic tasks? We investigate these questions using sentences with simple syntax and semantics (e.g., The bone was eaten by the dog.). We consider multiple neural network architectures, including recently proposed ELMo and BERT. We use magnetoencephalography (MEG) brain recording data collected from human subjects when they were reading these simple sentences. Overall, we find that BERT’s activations correlate the best with MEG brain data. We also find that the deep network representation can be used to generate brain data from new sentences to augment existing brain data. To the best of our knowledge, this is the first work showing that the MEG brain recording when reading a word in a sentence can be used to distinguish earlier words in the sentence. Our exploration is also the first to use deep neural network representations to generate synthetic brain data and to show that it helps in improving subsequent stimuli decoding task accuracy.
%R 10.18653/v1/P19-1507
%U https://aclanthology.org/P19-1507
%U https://doi.org/10.18653/v1/P19-1507
%P 5137-5154

Markdown (Informal)

[Relating Simple Sentence Representations in Deep Neural Networks and the Brain](https://aclanthology.org/P19-1507) (Jat et al., ACL 2019)

• Relating Simple Sentence Representations in Deep Neural Networks and the Brain (Jat et al., ACL 2019)

ACL

• Sharmistha Jat, Hao Tang, Partha Talukdar, and Tom Mitchell. 2019. Relating Simple Sentence Representations in Deep Neural Networks and the Brain. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5137–5154, Florence, Italy. Association for Computational Linguistics.