|
|
| Line 1: |
Line 1: |
| * ESL = English as a Second Language
| | In the fashion life, sweet, pleasant style http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm is a lot of girls to pursue fashion elements, a pleasant little http://www.toptbshop.com/Tory-Burch-Reva-Flats_9_1.htm dress how can a beautiful bag to decorate it! Let me http://www.toptbshop.com/Tory-Burch-New-Arrival_7_1.htm introduce you to several popular http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm bag to match your dress it sweet and pleasant!This D-shaped large bag is very cool fashion, with the gold chain as a decoration. On a simple bag with clothes with http://www.toptbshop.com/Tory-Burch-Flip-Flops_4_1.htm flowers appropriate. |
| * 50 multiple-choice synonym questions; 4 choices per question
| |
| * each question includes a sentence, providing context for the question
| |
| * ESL questions available on request from [http://www.apperceptual.com/ Peter Turney]
| |
| * introduced in Turney (2001) as a way of evaluating algorithms for measuring degree of similarity between words
| |
| * subsequently used by many other researchers
| |
| | |
| | |
| == Sample question ==
| |
| | |
| ::{| border="0" cellpadding="1" cellspacing="1"
| |
| |-
| |
| ! Stem:
| |
| |
| |
| | "A '''rusty''' nail is not as strong as a clean, new one."
| |
| |-
| |
| ! Choices:
| |
| | (a)
| |
| | corroded
| |
| |-
| |
| |
| |
| | (b)
| |
| | black
| |
| |-
| |
| |
| |
| | (c)
| |
| | dirty
| |
| |-
| |
| |
| |
| | (d)
| |
| | painted
| |
| |-
| |
| ! Solution:
| |
| | (a)
| |
| | corroded
| |
| |-
| |
| |}
| |
| | |
| | |
| == Table of results ==
| |
| | |
| {| border="1" cellpadding="5" cellspacing="1" width="100%"
| |
| |-
| |
| ! Algorithm
| |
| ! Reference for algorithm
| |
| ! Reference for experiment
| |
| ! Type
| |
| ! Correct
| |
| ! 95% confidence
| |
| |-
| |
| | Random
| |
| | Random guessing
| |
| | 1 / 4 = 25.00%
| |
| | Random
| |
| | 25.00%
| |
| | 14.63-40.34%
| |
| |-
| |
| | RES
| |
| | Resnik (1995)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Hybrid
| |
| | 32.66%
| |
| | 21.21-48.77%
| |
| |-
| |
| | LC
| |
| | Leacock and Chodrow (1998)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Lexicon-based
| |
| | 36.00%
| |
| | 22.92-50.81%
| |
| |-
| |
| | LIN
| |
| | Lin (1998)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Hybrid
| |
| | 36.00%
| |
| | 22.92-50.81%
| |
| |-
| |
| | JC
| |
| | Jiang and Conrath (1997)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Hybrid
| |
| | 36.00%
| |
| | 22.92-50.81%
| |
| |-
| |
| | HSO
| |
| | Hirst and St.-Onge (1998)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Lexicon-based
| |
| | 62.00%
| |
| | 47.18-75.35%
| |
| |-
| |
| | PMI-IR
| |
| | Turney (2001)
| |
| | Turney (2001)
| |
| | Corpus-based
| |
| | 74.00%
| |
| | 59.66-85.37%
| |
| |-
| |
| | PMI-IR
| |
| | Terra and Clarke (2003)
| |
| | Terra and Clarke (2003)
| |
| | Corpus-based
| |
| | 80.00%
| |
| | 66.28-89.97%
| |
| |-
| |
| | JS
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Jarmasz and Szpakowicz (2003)
| |
| | Lexicon-based
| |
| | 82.00%
| |
| | 68.56-91.42%
| |
| |-
| |
| |}
| |
| | |
| | |
| == Explanation of table ==
| |
| | |
| * '''Algorithm''' = name of algorithm
| |
| * '''Reference for algorithm''' = where to find out more about given algorithm
| |
| * '''Reference for experiment''' = where to find out more about evaluation of given algorithm with ESL questions
| |
| * '''Type''' = general type of algorithm: corpus-based, lexicon-based, hybrid
| |
| * '''Correct''' = percent of 80 questions that given algorithm answered correctly
| |
| * '''95% confidence''' = confidence interval calculated using [http://www.quantitativeskills.com/sisa/statistics/onemean.htm Binomial Exact Test]
| |
| * table rows sorted in order of increasing percent correct
| |
| * several WordNet-based similarity measures are implemented in [http://www.d.umn.edu/~tpederse/ Ted Pedersen]'s [http://www.d.umn.edu/~tpederse/similarity.html WordNet::Similarity] package
| |
| * PMI-IR = Pointwise Mutual Information - Information Retrieval
| |
| * Terra and Clarke (2003) call the ESL Synonym Questions "TS1"
| |
| | |
| == Caveats ==
| |
| | |
| * the performance of a corpus-based algorithm depends on the corpus, so the difference in performance between two corpus-based systems may be due to the different corpora, rather than the different algorithms
| |
| * the ESL questions include nouns, verbs, and adjectives, but some of the WordNet-based algorithms were only designed to work with nouns
| |
| | |
| | |
| == References ==
| |
| | |
| Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332.
| |
| | |
| Jarmasz, M., and Szpakowicz, S. (2003). [http://www.csi.uottawa.ca/~szpak/recent_papers/TR-2003-01.pdf Roget’s thesaurus and semantic similarity], ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, September, pp. 212-219.
| |
| | |
| Jiang, J.J., and Conrath, D.W. (1997). [http://wortschatz.uni-leipzig.de/~sbordag/aalw05/Referate/03_Assoziationen_BudanitskyResnik/Jiang_Conrath_97.pdf Semantic similarity based on corpus statistics and lexical taxonomy]. ''Proceedings of the International Conference on Research in Computational Linguistics'', Taiwan.
| |
| | |
| Leacock, C., and Chodorow, M. (1998). Combining local context and WordNet similarity for word sense identification. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, pp. 265-283.
| |
| | |
| Lin, D. (1998). [http://www.cs.ualberta.ca/~lindek/papers/sim.pdf An information-theoretic definition of similarity]. ''Proceedings of the 15th International Conference on Machine Learning (ICML-98)'', Madison, WI, pp. 296-304.
| |
| | |
| Resnik, P. (1995). [http://citeseer.ist.psu.edu/resnik95using.html Using information content to evaluate semantic similarity]. ''Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95)'', Montreal, pp. 448-453.
| |
| | |
| Terra, E., and Clarke, C.L.A. (2003). [http://acl.ldc.upenn.edu/N/N03/N03-1032.pdf Frequency estimates for statistical word similarity measures]. ''Proceedings of the Human Language Technology and North American Chapter of Association of Computational Linguistics Conference 2003 (HLT/NAACL 2003)'', pp. 244–251.
| |
| | |
| Turney, P.D. (2001). [http://arxiv.org/abs/cs.LG/0212033 Mining the Web for synonyms: PMI-IR versus LSA on TOEFL]. ''Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001)'', Freiburg, Germany, pp. 491-502.
| |
| | |
| == See also ==
| |
| | |
| * [[Attributional and Relational Similarity (State of the art)]]
| |
| * [[SAT Analogy Questions]]
| |
| * [[TOEFL Synonym Questions]]
| |
| * [[State of the art]]
| |
| | |
| | |
| [[Category:State of the art]]
| |