Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution

Yordan Yordanov, Oana-Maria Camburu, Vid Kocijan, Thomas Lukasiewicz


Abstract
Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.
Anthology ID:
2020.emnlp-main.402
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4963–4969
Language:
URL:
https://aclanthology.org/2020.emnlp-main.402
DOI:
10.18653/v1/2020.emnlp-main.402
Bibkey:
Cite (ACL):
Yordan Yordanov, Oana-Maria Camburu, Vid Kocijan, and Thomas Lukasiewicz. 2020. Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4963–4969, Online. Association for Computational Linguistics.
Cite (Informal):
Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution (Yordanov et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.402.pdf
Video:
 https://slideslive.com/38939083
Code
 YDYordanov/WS-training-objectives
Data
WSCWinoGrande