Paraphrase Generation and Evaluation on Colloquial-Style Sentences

Eetu Sjöblom, Mathias Creutz, Yves Scherrer


Abstract
In this paper, we investigate paraphrase generation in the colloquial domain. We use state-of-the-art neural machine translation models trained on the Opusparcus corpus to generate paraphrases in six languages: German, English, Finnish, French, Russian, and Swedish. We perform experiments to understand how data selection and filtering for diverse paraphrase pairs affects the generated paraphrases. We compare two different model architectures, an RNN and a Transformer model, and find that the Transformer does not generally outperform the RNN. We also conduct human evaluation on five of the six languages and compare the results to the automatic evaluation metrics BLEU and the recently proposed BERTScore. The results advance our understanding of the trade-offs between the quality and novelty of generated paraphrases, affected by the data selection method. In addition, our comparison of the evaluation methods shows that while BLEU correlates well with human judgments at the corpus level, BERTScore outperforms BLEU in both corpus and sentence-level evaluation.
Anthology ID:
2020.lrec-1.224
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
1814–1822
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.224
DOI:
Bibkey:
Cite (ACL):
Eetu Sjöblom, Mathias Creutz, and Yves Scherrer. 2020. Paraphrase Generation and Evaluation on Colloquial-Style Sentences. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1814–1822, Marseille, France. European Language Resources Association.
Cite (Informal):
Paraphrase Generation and Evaluation on Colloquial-Style Sentences (Sjöblom et al., LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.224.pdf