An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios

Andrew Zupon, Faiz Rafique, Mihai Surdeanu


Abstract
Neural networks are a common tool in NLP, but it is not always clear which architecture to use for a given task. Different tasks, different languages, and different training conditions can all affect how a neural network will perform. Capsule Networks (CapsNets) are a relatively new architecture in NLP. Due to their novelty, CapsNets are being used more and more in NLP tasks. However, their usefulness is still mostly untested. In this paper, we compare three neural network architectures—LSTM, CNN, and CapsNet—on a part of speech tagging task. We compare these architectures in both high- and low-resource training conditions and find that no architecture consistently performs the best. Our analysis shows that our CapsNet performs nearly as well as a more complex LSTM under certain training conditions, but not others, and that our CapsNet almost always outperforms our CNN. We also find that our CapsNet implementation shows faster prediction times than the LSTM for Scottish Gaelic but not for Spanish, highlighting the effect that the choice of languages can have on the models.
Anthology ID:
2020.insights-1.10
Volume:
Proceedings of the First Workshop on Insights from Negative Results in NLP
Month:
November
Year:
2020
Address:
Online
Editors:
Anna Rogers, João Sedoc, Anna Rumshisky
Venue:
insights
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
66–70
Language:
URL:
https://aclanthology.org/2020.insights-1.10
DOI:
10.18653/v1/2020.insights-1.10
Bibkey:
Cite (ACL):
Andrew Zupon, Faiz Rafique, and Mihai Surdeanu. 2020. An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 66–70, Online. Association for Computational Linguistics.
Cite (Informal):
An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios (Zupon et al., insights 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.insights-1.10.pdf
Video:
 https://slideslive.com/38940797