The Lower The Simpler: Simplifying Hierarchical Recurrent Models

Chao Wang, Hui Jiang


Abstract
To improve the training efficiency of hierarchical recurrent models without compromising their performance, we propose a strategy named as “the lower the simpler”, which is to simplify the baseline models by making the lower layers simpler than the upper layers. We carry out this strategy to simplify two typical hierarchical recurrent models, namely Hierarchical Recurrent Encoder-Decoder (HRED) and R-NET, whose basic building block is GRU. Specifically, we propose Scalar Gated Unit (SGU), which is a simplified variant of GRU, and use it to replace the GRUs at the middle layers of HRED and R-NET. Besides, we also use Fixed-size Ordinally-Forgetting Encoding (FOFE), which is an efficient encoding method without any trainable parameter, to replace the GRUs at the bottom layers of HRED and R-NET. The experimental results show that the simplified HRED and the simplified R-NET contain significantly less trainable parameters, consume significantly less training time, and achieve slightly better performance than their baseline models.
Anthology ID:
N19-1402
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4005–4009
Language:
URL:
https://aclanthology.org/N19-1402
DOI:
10.18653/v1/N19-1402
Bibkey:
Cite (ACL):
Chao Wang and Hui Jiang. 2019. The Lower The Simpler: Simplifying Hierarchical Recurrent Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4005–4009, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
The Lower The Simpler: Simplifying Hierarchical Recurrent Models (Wang & Jiang, NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1402.pdf
Video:
 https://vimeo.com/356204320
Data
SQuAD