Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models

Itzik Malkiel, Lior Wolf


Abstract
Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. We conduct an extensive inter- and intra-dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary.
Anthology ID:
2021.eacl-main.14
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
187–199
Language:
URL:
https://aclanthology.org/2021.eacl-main.14
DOI:
10.18653/v1/2021.eacl-main.14
Bibkey:
Cite (ACL):
Itzik Malkiel and Lior Wolf. 2021. Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 187–199, Online. Association for Computational Linguistics.
Cite (Informal):
Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (Malkiel & Wolf, EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.14.pdf
Data
CoLAGLUEMultiNLIQNLISNLISSTSST-2