Noun-Modifier Questions (State of the art)
- multiple-choice noun-modifier questions
- 2,180 seven-choice questions for evaluating noun-modifier composition models
- questions derived from WordNet
- split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing
- introduced in Turney (2012)
- see also: Similarity (State of the art)
Stem: coin collector Choices: (a) numismatist (b) coin (c) collector (d) coinage (e) aggregator (f) reflectiveness (g) plasma Solution: (a) numismatist
Table of results
|Algorithm||Reference for algorithm||Reference for experiment||Correct - 7-choices||Correct - 14-choices|
|Vector addition||Landauer and Dumais (1997)||Turney (2012)||50.1||22.5|
|Element-wise multiplication||Mitchell and Lapata (2008)||Turney (2012)||57.5||27.4|
|Dual-Space model||Turney (2012)||Turney (2012)||58.3||41.5|
|SuperSim||Turney (2013)||Turney (2013)||75.9||68.0|
|Holistic model||Turney (2012)||Turney (2012)||81.6||NA|
Landauer, T.K., and Dumais, S.T. (1997). A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211-240.
Mitchell, J., and Lapata, M. (2008). Vector-based models of semantic composition. In Proceedings of ACL-08: HLT, pages 236-244, Columbus, Ohio. Association for Computational Linguistics.
Turney, P.D. (2012). Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585.
Turney, P.D. (2013), Distributional semantics beyond words: Supervised learning of analogy and paraphrase, Transactions of the Association for Computational Linguistics (TACL), 1, 353-366.