Noun-Modifier Questions (State of the art): Difference between revisions
Jump to navigation
Jump to search
Created page with "* [http://jair.org/media/3640/live-3640-6413-jair.txt multiple-choice noun-modifier questions] * 2,180 seven-choice questions for evaluating noun-modifier composition models *..." |
No edit summary |
||
| Line 47: | Line 47: | ||
== Table of results == | == Table of results == | ||
{| border="1" cellpadding="5" cellspacing="1" | |||
|- | |||
! Algorithm | |||
! Reference for algorithm | |||
! Reference for experiment | |||
! Correct - 7-choices | |||
! Correct - 14-choices | |||
|- | |||
| Vector addition | |||
| Landauer and Dumais (1997) | |||
| Turney (2012) | |||
| 50.1 | |||
| 22.5 | |||
|- | |||
| Element-wise multiplication | |||
| | |||
| | |||
| 57.5 | |||
| 27.4 | |||
|- | |||
| Dual-Space model | |||
| | |||
| | |||
| 58.3 | |||
| 41.5 | |||
|- | |||
| SuperSim | |||
| | |||
| | |||
| 75.9 | |||
| 68.0 | |||
|- | |||
| Holistic model | |||
| | |||
| | |||
| 81.6 | |||
| NA | |||
|} | |||
== References == | == References == | ||
Revision as of 12:18, 28 April 2014
- multiple-choice noun-modifier questions
- 2,180 seven-choice questions for evaluating noun-modifier composition models
- questions derived from WordNet
- split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing
- introduced in Turney (2012)
Sample question
Stem: coin collector Choices: (a) numismatist (b) coin (c) collector (d) coinage (e) aggregator (f) reflectiveness (g) plasma Solution: (a) numismatist
Table of results
| Algorithm | Reference for algorithm | Reference for experiment | Correct - 7-choices | Correct - 14-choices |
|---|---|---|---|---|
| Vector addition | Landauer and Dumais (1997) | Turney (2012) | 50.1 | 22.5 |
| Element-wise multiplication | 57.5 | 27.4 | ||
| Dual-Space model | 58.3 | 41.5 | ||
| SuperSim | 75.9 | 68.0 | ||
| Holistic model | 81.6 | NA |
References
Turney, P.D. (2012). Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585.
Turney, P.D. (2013), Distributional semantics beyond words: Supervised learning of analogy and paraphrase, Transactions of the Association for Computational Linguistics (TACL), 1, 353-366.