Transformer Based Multi-Source Domain Adaptation

Dustin Wright, Isabelle Augenstein


Abstract
In practical machine learning settings, the data on which a model must make predictions often come from a different distribution than the data it was trained on. Here, we investigate the problem of unsupervised multi-source domain adaptation, where a model is trained on labelled data from multiple source domains and must make predictions on a domain for which no labelled data has been seen. Prior work with CNNs and RNNs has demonstrated the benefit of mixture of experts, where the predictions of multiple domain expert classifiers are combined; as well as domain adversarial training, to induce a domain agnostic representation space. Inspired by this, we investigate how such methods can be effectively applied to large pretrained transformer models. We find that domain adversarial training has an effect on the learned representations of these models while having little effect on their performance, suggesting that large transformer-based models are already relatively robust across domains. Additionally, we show that mixture of experts leads to significant performance improvements by comparing several variants of mixing functions, including one novel metric based on attention. Finally, we demonstrate that the predictions of large pretrained transformer based domain experts are highly homogenous, making it challenging to learn effective metrics for mixing their predictions.
Anthology ID:
2020.emnlp-main.639
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7963–7974
Language:
URL:
https://aclanthology.org/2020.emnlp-main.639
DOI:
10.18653/v1/2020.emnlp-main.639
Bibkey:
Cite (ACL):
Dustin Wright and Isabelle Augenstein. 2020. Transformer Based Multi-Source Domain Adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7963–7974, Online. Association for Computational Linguistics.
Cite (Informal):
Transformer Based Multi-Source Domain Adaptation (Wright & Augenstein, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.639.pdf
Video:
 https://slideslive.com/38938766
Code
 copenlu/xformer-multi-source-domain-adaptation