Deep Learning in Semantic Kernel Spaces

Danilo Croce, Simone Filice, Giuseppe Castellucci, Roberto Basili


Abstract
Kernel methods enable the direct usage of structured representations of textual data during language learning and inference tasks. Expressive kernels, such as Tree Kernels, achieve excellent performance in NLP. On the other side, deep neural networks have been demonstrated effective in automatically learning feature representations during training. However, their input is tensor data, i.e., they can not manage rich structured information. In this paper, we show that expressive kernels and deep neural networks can be combined in a common framework in order to (i) explicitly model structured information and (ii) learn non-linear decision functions. We show that the input layer of a deep architecture can be pre-trained through the application of the Nystrom low-rank approximation of kernel spaces. The resulting “kernelized” neural network achieves state-of-the-art accuracy in three different tasks.
Anthology ID:
P17-1032
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
345–354
Language:
URL:
https://aclanthology.org/P17-1032
DOI:
10.18653/v1/P17-1032
Bibkey:
Cite (ACL):
Danilo Croce, Simone Filice, Giuseppe Castellucci, and Roberto Basili. 2017. Deep Learning in Semantic Kernel Spaces. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 345–354, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Deep Learning in Semantic Kernel Spaces (Croce et al., ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-1032.pdf
Video:
 https://aclanthology.org/P17-1032.mp4