Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing

Daniel Fried, Dan Klein


Abstract
Dynamic oracles provide strong supervision for training constituency parsers with exploration, but must be custom defined for a given parser’s transition system. We explore using a policy gradient method as a parser-agnostic alternative. In addition to directly optimizing for a tree-level metric such as F1, policy gradient has the potential to reduce exposure bias by allowing exploration during training; moreover, it does not require a dynamic oracle for supervision. On four constituency parsers in three languages, the method substantially outperforms static oracle likelihood training in almost all settings. For parsers where a dynamic oracle is available (including a novel oracle which we define for the transition system of Dyer et al., 2016), policy gradient typically recaptures a substantial fraction of the performance gain afforded by the dynamic oracle.
Anthology ID:
P18-2075
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
469–476
Language:
URL:
https://aclanthology.org/P18-2075
DOI:
10.18653/v1/P18-2075
Bibkey:
Cite (ACL):
Daniel Fried and Dan Klein. 2018. Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 469–476, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing (Fried & Klein, ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-2075.pdf
Note:
 P18-2075.Notes.pdf
Presentation:
 P18-2075.Presentation.pdf
Video:
 https://aclanthology.org/P18-2075.mp4