Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

Benjamin Heinzerling, Michael Strube


Abstract
Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it difficult for practitioners to choose between them. In this work, we conduct an extensive evaluation comparing non-contextual subword embeddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. We find that overall, a combination of BERT, BPEmb, and character representations works best across languages and tasks. A more detailed analysis reveals different strengths and weaknesses: Multilingual BERT performs well in medium- to high-resource languages, but is outperformed by non-contextual subword embeddings in a low-resource setting.
Anthology ID:
P19-1027
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
273–291
Language:
URL:
https://aclanthology.org/P19-1027
DOI:
10.18653/v1/P19-1027
Bibkey:
Cite (ACL):
Benjamin Heinzerling and Michael Strube. 2019. Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273–291, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation (Heinzerling & Strube, ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1027.pdf
Video:
 https://aclanthology.org/P19-1027.mp4
Code
 bheinzerling/subword-sequence-tagging
Data
Universal Dependencies