Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation

Maximiliana Behnke, Kenneth Heafield


Abstract
The attention mechanism is the crucial component of the transformer architecture. Recent research shows that most attention heads are not confident in their decisions and can be pruned. However, removing them before training a model results in lower quality. In this paper, we apply the lottery ticket hypothesis to prune heads in the early stages of training. Our experiments on machine translation show that it is possible to remove up to three-quarters of attention heads from transformer-big during early training with an average -0.1 change in BLEU for Turkish→English. The pruned model is 1.5 times as fast at inference, albeit at the cost of longer training. Our method is complementary to other approaches, such as teacher-student, with English→German student model gaining an additional 10% speed-up with 75% encoder attention removed and 0.2 BLEU loss.
Anthology ID:
2020.emnlp-main.211
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2664–2674
Language:
URL:
https://aclanthology.org/2020.emnlp-main.211
DOI:
10.18653/v1/2020.emnlp-main.211
Bibkey:
Cite (ACL):
Maximiliana Behnke and Kenneth Heafield. 2020. Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2664–2674, Online. Association for Computational Linguistics.
Cite (Informal):
Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation (Behnke & Heafield, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.211.pdf
Video:
 https://slideslive.com/38938739