A matter of framing: The impact of linguistic formalism on probing results

Ilia Kuznetsov, Iryna Gurevych


Abstract
Deep pre-trained contextualized encoders like BERT demonstrate remarkable performance on a range of downstream tasks. A recent line of research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. While most work in probing operates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the formalism used to annotate the underlying data. Can the choice of formalism affect probing results? To investigate, we conduct an in-depth cross-formalism layer probing study in role semantics. We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer probing can detect subtle differences between the implementations of the same linguistic formalism. Our results suggest that linguistic formalism is an important dimension in probing studies, along with the commonly used cross-task and cross-lingual experimental settings.
Anthology ID:
2020.emnlp-main.13
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
171–182
Language:
URL:
https://aclanthology.org/2020.emnlp-main.13
DOI:
10.18653/v1/2020.emnlp-main.13
Bibkey:
Cite (ACL):
Ilia Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on probing results. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 171–182, Online. Association for Computational Linguistics.
Cite (Informal):
A matter of framing: The impact of linguistic formalism on probing results (Kuznetsov & Gurevych, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.13.pdf
Video:
 https://slideslive.com/38938722
Data
FrameNetXNLI