A Simple and Effective Usage of Word Clusters for CBOW Model

Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura


Abstract
We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.
Anthology ID:
2020.aacl-main.10
Volume:
Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing
Month:
December
Year:
2020
Address:
Suzhou, China
Editors:
Kam-Fai Wong, Kevin Knight, Hua Wu
Venue:
AACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
80–86
Language:
URL:
https://aclanthology.org/2020.aacl-main.10
DOI:
Bibkey:
Cite (ACL):
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2020. A Simple and Effective Usage of Word Clusters for CBOW Model. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 80–86, Suzhou, China. Association for Computational Linguistics.
Cite (Informal):
A Simple and Effective Usage of Word Clusters for CBOW Model (Feng et al., AACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.aacl-main.10.pdf
Code
 yukunfeng/cluster-cbow