Similarity-Based Reconstruction Loss for Meaning Representation

Olga Kovaleva, Anna Rumshisky, Alexey Romanov


Abstract
This paper addresses the problem of representation learning. Using an autoencoder framework, we propose and evaluate several loss functions that can be used as an alternative to the commonly used cross-entropy reconstruction loss. The proposed loss functions use similarities between words in the embedding space, and can be used to train any neural model for text generation. We show that the introduced loss functions amplify semantic diversity of reconstructed sentences, while preserving the original meaning of the input. We test the derived autoencoder-generated representations on paraphrase detection and language inference tasks and demonstrate performance improvement compared to the traditional cross-entropy loss.
Anthology ID:
D18-1525
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4875–4880
Language:
URL:
https://aclanthology.org/D18-1525
DOI:
10.18653/v1/D18-1525
Bibkey:
Cite (ACL):
Olga Kovaleva, Anna Rumshisky, and Alexey Romanov. 2018. Similarity-Based Reconstruction Loss for Meaning Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4875–4880, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Similarity-Based Reconstruction Loss for Meaning Representation (Kovaleva et al., EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1525.pdf
Data
SNLI