ACL Wiki - User contributions [en]

SAT Analogy Questions (State of the art)

2017-03-22T16:03:37Z

Rspeer: /* References */ make reference style more consistent

* SAT = Scholastic Aptitude Test
* 374 multiple-choice analogy questions; 5 choices per question
* SAT questions collected by [http://www.cs.rutgers.edu/~mlittman/ Michael Littman], available on request from [http://www.apperceptual.com/ Peter Turney]
* introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
* see also: [[Similarity (State of the art)]], [[Analogy (State of the art)]]

== Sample question ==

::{| border="0" cellpadding="1" cellspacing="1"
|-
! Stem:
|
| mason:stone
|-
! Choices:
| (a)
| teacher:chalk
|-
|
| (b)
| carpenter:wood
|-
|
| (c)
| soldier:gun
|-
|
| (d)
| photograph:camera
|-
|
| (e)
| book:word
|-
! Solution:
| (b)
| carpenter:wood
|}

== Table of results ==

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! Algorithm
! Reference for algorithm
! Reference for experiment
! Type
! Correct
! 95% confidence
|-
| Random
| Random guessing
| 1 / 5 = 20.0%
| Random
| 20.0%
| 16.1-24.5%
|-
| JC
| Jiang and Conrath (1997)
| Turney (2006b)
| Hybrid
| 27.3%
| 23.1-32.4%
|-
| LIN
| Lin (1998)
| Turney (2006b)
| Hybrid
| 27.3%
| 23.1-32.4%
|-
| LC
| Leacock and Chodrow (1998)
| Turney (2006b)
| Lexicon-based
| 31.3%
| 26.9-36.5%
|-
| HSO
| Hirst and St.-Onge (1998)
| Turney (2006b)
| Lexicon-based
| 32.1%
| 27.6-37.4%
|-
| RES
| Resnik (1995)
| Turney (2006b)
| Hybrid
| 33.2%
| 28.7-38.5%
|-
| PMI-IR
| Turney (2001)
| Turney (2006b)
| Corpus-based
| 35.0%
| 30.2-40.1%
|-
| LSA+Predication
| Mangalath et al. (2004)
| Mangalath et al. (2004)
| Corpus-based
| 42.0%
| 37.2-47.4%
|-
| KNOW-BEST
| Veale (2004)
| Veale (2004)
| Lexicon-based
| 43.0%
| 38.0-48.2%
|-
| ''k''-means
| Bicici and Yuret (2006)
| Bicici and Yuret (2006)
| Corpus-based
| 44.0%
| 39.0-49.3%
|-
| BagPack
| Herdağdelen and Baroni (2009)
| Herdağdelen and Baroni (2009)
| Corpus-based
| 44.1%
| 39.0-49.3%
|-
| VSM
| Turney and Littman (2005)
| Turney and Littman (2005)
| Corpus-based
| 47.1%
| 42.2-52.5%
|-
| Dual-Space
| Turney (2012)
| Turney (2012)
| Corpus-based
| 51.1%
| 46.1-56.5%
|-
| BMI
| Bollegala et al. (2009)
| Bollegala et al. (2009)
| Corpus-based
| 51.1%
| 46.1-56.5%
|-
| PairClass
| Turney (2008)
| Turney (2008)
| Corpus-based
| 52.1%
| 46.9-57.3%
|-
| PERT
| Turney (2006a)
| Turney (2006a)
| Corpus-based
| 53.5%
| 48.5-58.9%
|-
| SuperSim
| Turney (2013)
| Turney (2013)
| Corpus-based
| 54.8%
| 49.6-59.9%
|-
| ConceptNet
| Speer et al. (2017)
| Speer et al. (2017)
| Hybrid
| 56.1%
| 51.0-61.2%
|-
| LRA
| Turney (2006b)
| Turney (2006b)
| Corpus-based
| 56.1%
| 51.0-61.2%
|-
| Human
| Average US college applicant
| Turney and Littman (2005)
| Human
| 57.0%
| 52.0-62.3%
|-
| Human Voting
| Lofi (2013)
| Lofi (2013)
| Human Voting
| 81.5%
| 77.2-85.4%
|}

== Explanation of table ==

* '''Algorithm''' = name of algorithm
* '''Reference for algorithm''' = where to find out more about given algorithm
* '''Reference for experiment''' = where to find out more about evaluation of given algorithm with SAT questions
* '''Type''' = general type of algorithm: corpus-based, lexicon-based, hybrid
* '''Correct''' = percent of 374 questions that given algorithm answered correctly
* '''95% confidence''' = confidence interval calculated using the [[Statistical calculators|Binomial Exact Test]]
* table rows sorted in order of increasing percent correct
* several WordNet-based similarity measures are implemented in [http://www.d.umn.edu/~tpederse/ Ted Pedersen]'s [http://www.d.umn.edu/~tpederse/similarity.html WordNet::Similarity] package
* KNOW-BEST = KNOWledge-Based Entertainment and Scholastic Testing
* VSM = Vector Space Model
* LRA = Latent Relational Analysis
* PERT = Pertinence
* PMI-IR = Pointwise Mutual Information - Information Retrieval
* LSA+Predication = Latent Semantic Analysis + Predication
* BagPack = Bag of words representation of Paired concept knowledge
* ConceptNet = ConceptNet Numberbatch 2016.09

== References ==

Bicici, E., and Yuret, D. (2006). [http://www.denizyuret.com/pub/tainn-06/LAWSQ-LNCS.pdf Clustering word pairs to answer analogy questions]. ''Proceedings of the Fifteenth Turkish Symposium on Artificial Intelligence and Neural Networks (TAINN 2006)''.

Bollegala D., Matsuo Y., and Ishizuka M. (2009). [http://www2009.org/proceedings/pdf/p651.pdf Measuring the similarity between implicit semantic relations from the web]. ''Proceedings of the 18th International Conference on World Wide Web'', ACM, pages 651–660.

Herdağdelen A. and Baroni M. (2009) [http://clic.cimec.unitn.it/marco/publications/gems-09/herdagdelen-baroni-gems09.pdf BagPack: A general framework to represent semantic relations]. ''Proceedings of the EACL 2009 Geometrical Models for Natural Language Semantics (GEMS) Workshop'', East Stroudsburg PA: ACL, 33-40.

Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332.

Jiang, J.J., and Conrath, D.W. (1997). [http://wortschatz.uni-leipzig.de/~sbordag/aalw05/Referate/03_Assoziationen_BudanitskyResnik/Jiang_Conrath_97.pdf Semantic similarity based on corpus statistics and lexical taxonomy]. ''Proceedings of the International Conference on Research in Computational Linguistics'', Taiwan.

Leacock, C., and Chodorow, M. (1998). [http://books.google.ca/books?id=Rehu8OOzMIMC&lpg=PA265&ots=IpnaLkZUec&lr&pg=PA265#v=onepage&q&f=false Combining local context and WordNet similarity for word sense identification]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, pp. 265-283.

Lin, D. (1998). [http://www.cs.ualberta.ca/~lindek/papers/sim.pdf An information-theoretic definition of similarity]. ''Proceedings of the 15th International Conference on Machine Learning (ICML-98)'', Madison, WI, pp. 296-304.

Lofi, C. (2013). [http://www.ifis.cs.tu-bs.de/sites/default/files/biblio/13cdim_final_pdf_80985.pdf Just ask a human?--Controlling Quality in Relational Similarity and Analogy Processing using the Crowd]. ''Proceedings of the Workshop of the 15th BTW Conference on Database Systems for Business, Technology, and Web (BTW 2013)'', Magdeburg, Germany, pp. 197-210.

Mangalath, P., Quesada, J., and Kintsch, W. (2004). [http://www.cogsci.northwestern.edu/cogsci2004/ma/ma355.pdf Analogy-making as predication using relational information and LSA vectors]. In K.D. Forbus, D. Gentner & T. Regier (Eds.), ''Proceedings of the 26th Annual Meeting of the Cognitive Science Society''. Chicago: Lawrence Erlbaum Associates.

Resnik, P. (1995). [http://citeseer.ist.psu.edu/resnik95using.html Using information content to evaluate semantic similarity]. ''Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95)'', Montreal, pp. 448-453.

Speer, R., Chin, J., and Havasi, C. (2017). [http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972 ConceptNet 5.5: An Open Multilingual Graph of General Knowledge]. ''Proceedings of The 31st AAAI Conference on Artificial Intelligence'', San Francisco, CA.

Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). [http://arxiv.org/abs/cs.CL/0309035 Combining independent modules to solve multiple-choice synonym and analogy problems]. ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, pp. 482-489.

Turney, P.D., and Littman, M.L. (2005). [http://arxiv.org/abs/cs.LG/0508103 Corpus-based learning of analogies and semantic relations]. ''Machine Learning'', 60 (1-3), 251-278.

Turney, P.D. (2001). [http://arxiv.org/abs/cs.LG/0212033 Mining the Web for synonyms: PMI-IR versus LSA on TOEFL]. ''Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001)'', Freiburg, Germany, pp. 491-502.

Turney, P.D. (2006a). [http://arxiv.org/abs/cs.CL/0607120 Expressing implicit semantic relations without supervision]. ''Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (Coling/ACL-06)'', Sydney, Australia, pp. 313-320.

Turney, P.D. (2006b). [http://arxiv.org/abs/cs.CL/0608100 Similarity of semantic relations]. ''Computational Linguistics'', 32 (3), 379-416.

Turney, P.D. (2008). [http://arxiv.org/abs/0809.0124 A uniform approach to analogies, synonyms, antonyms, and associations]. ''Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)'', Manchester, UK, pp. 905-912.

Turney, P.D. (2012). [http://jair.org/papers/paper3640.html Domain and function: A dual-space model of semantic relations and compositions], ''Journal of Artificial Intelligence Research (JAIR)'', 44, 533-585.

Turney, P.D. (2013), [http://aclweb.org/anthology/Q/Q13/Q13-1029.pdf Distributional semantics beyond words: Supervised learning of analogy and paraphrase], ''Transactions of the Association for Computational Linguistics (TACL)'', 1, 353-366.

Veale, T. (2004). [http://afflatus.ucd.ie/Papers/ecai2004.pdf WordNet sits the SAT: A knowledge-based approach to lexical analogy]. ''Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004)'', pp. 606–612, Valencia, Spain.

[[Category:State of the art]]
[[Category:Similarity]]
[[Category:Analogy]]

SAT Analogy Questions (State of the art)

2017-03-22T16:02:45Z

Rspeer: add ConceptNet Numberbatch to analogy evaluation table

* SAT = Scholastic Aptitude Test
* 374 multiple-choice analogy questions; 5 choices per question
* SAT questions collected by [http://www.cs.rutgers.edu/~mlittman/ Michael Littman], available on request from [http://www.apperceptual.com/ Peter Turney]
* introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
* see also: [[Similarity (State of the art)]], [[Analogy (State of the art)]]

== Sample question ==

::{| border="0" cellpadding="1" cellspacing="1"
|-
! Stem:
|
| mason:stone
|-
! Choices:
| (a)
| teacher:chalk
|-
|
| (b)
| carpenter:wood
|-
|
| (c)
| soldier:gun
|-
|
| (d)
| photograph:camera
|-
|
| (e)
| book:word
|-
! Solution:
| (b)
| carpenter:wood
|}

== Table of results ==

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! Algorithm
! Reference for algorithm
! Reference for experiment
! Type
! Correct
! 95% confidence
|-
| Random
| Random guessing
| 1 / 5 = 20.0%
| Random
| 20.0%
| 16.1-24.5%
|-
| JC
| Jiang and Conrath (1997)
| Turney (2006b)
| Hybrid
| 27.3%
| 23.1-32.4%
|-
| LIN
| Lin (1998)
| Turney (2006b)
| Hybrid
| 27.3%
| 23.1-32.4%
|-
| LC
| Leacock and Chodrow (1998)
| Turney (2006b)
| Lexicon-based
| 31.3%
| 26.9-36.5%
|-
| HSO
| Hirst and St.-Onge (1998)
| Turney (2006b)
| Lexicon-based
| 32.1%
| 27.6-37.4%
|-
| RES
| Resnik (1995)
| Turney (2006b)
| Hybrid
| 33.2%
| 28.7-38.5%
|-
| PMI-IR
| Turney (2001)
| Turney (2006b)
| Corpus-based
| 35.0%
| 30.2-40.1%
|-
| LSA+Predication
| Mangalath et al. (2004)
| Mangalath et al. (2004)
| Corpus-based
| 42.0%
| 37.2-47.4%
|-
| KNOW-BEST
| Veale (2004)
| Veale (2004)
| Lexicon-based
| 43.0%
| 38.0-48.2%
|-
| ''k''-means
| Bicici and Yuret (2006)
| Bicici and Yuret (2006)
| Corpus-based
| 44.0%
| 39.0-49.3%
|-
| BagPack
| Herdağdelen and Baroni (2009)
| Herdağdelen and Baroni (2009)
| Corpus-based
| 44.1%
| 39.0-49.3%
|-
| VSM
| Turney and Littman (2005)
| Turney and Littman (2005)
| Corpus-based
| 47.1%
| 42.2-52.5%
|-
| Dual-Space
| Turney (2012)
| Turney (2012)
| Corpus-based
| 51.1%
| 46.1-56.5%
|-
| BMI
| Bollegala et al. (2009)
| Bollegala et al. (2009)
| Corpus-based
| 51.1%
| 46.1-56.5%
|-
| PairClass
| Turney (2008)
| Turney (2008)
| Corpus-based
| 52.1%
| 46.9-57.3%
|-
| PERT
| Turney (2006a)
| Turney (2006a)
| Corpus-based
| 53.5%
| 48.5-58.9%
|-
| SuperSim
| Turney (2013)
| Turney (2013)
| Corpus-based
| 54.8%
| 49.6-59.9%
|-
| ConceptNet
| Speer et al. (2017)
| Speer et al. (2017)
| Hybrid
| 56.1%
| 51.0-61.2%
|-
| LRA
| Turney (2006b)
| Turney (2006b)
| Corpus-based
| 56.1%
| 51.0-61.2%
|-
| Human
| Average US college applicant
| Turney and Littman (2005)
| Human
| 57.0%
| 52.0-62.3%
|-
| Human Voting
| Lofi (2013)
| Lofi (2013)
| Human Voting
| 81.5%
| 77.2-85.4%
|}

== Explanation of table ==

* '''Algorithm''' = name of algorithm
* '''Reference for algorithm''' = where to find out more about given algorithm
* '''Reference for experiment''' = where to find out more about evaluation of given algorithm with SAT questions
* '''Type''' = general type of algorithm: corpus-based, lexicon-based, hybrid
* '''Correct''' = percent of 374 questions that given algorithm answered correctly
* '''95% confidence''' = confidence interval calculated using the [[Statistical calculators|Binomial Exact Test]]
* table rows sorted in order of increasing percent correct
* several WordNet-based similarity measures are implemented in [http://www.d.umn.edu/~tpederse/ Ted Pedersen]'s [http://www.d.umn.edu/~tpederse/similarity.html WordNet::Similarity] package
* KNOW-BEST = KNOWledge-Based Entertainment and Scholastic Testing
* VSM = Vector Space Model
* LRA = Latent Relational Analysis
* PERT = Pertinence
* PMI-IR = Pointwise Mutual Information - Information Retrieval
* LSA+Predication = Latent Semantic Analysis + Predication
* BagPack = Bag of words representation of Paired concept knowledge
* ConceptNet = ConceptNet Numberbatch 2016.09

== References ==

Bicici, E., and Yuret, D. (2006). [http://www.denizyuret.com/pub/tainn-06/LAWSQ-LNCS.pdf Clustering word pairs to answer analogy questions]. ''Proceedings of the Fifteenth Turkish Symposium on Artificial Intelligence and Neural Networks (TAINN 2006)''.

Bollegala D., Matsuo Y., and Ishizuka M. (2009). [http://www2009.org/proceedings/pdf/p651.pdf Measuring the similarity between implicit semantic relations from the web]. ''Proceedings of the 18th International Conference on World Wide Web'', ACM, pages 651–660.

Herdağdelen A. and Baroni M. (2009) [http://clic.cimec.unitn.it/marco/publications/gems-09/herdagdelen-baroni-gems09.pdf BagPack: A general framework to represent semantic relations]. ''Proceedings of the EACL 2009 Geometrical Models for Natural Language Semantics (GEMS) Workshop'', East Stroudsburg PA: ACL, 33-40.

Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332.

Jiang, J.J., and Conrath, D.W. (1997). [http://wortschatz.uni-leipzig.de/~sbordag/aalw05/Referate/03_Assoziationen_BudanitskyResnik/Jiang_Conrath_97.pdf Semantic similarity based on corpus statistics and lexical taxonomy]. ''Proceedings of the International Conference on Research in Computational Linguistics'', Taiwan.

Leacock, C., and Chodorow, M. (1998). [http://books.google.ca/books?id=Rehu8OOzMIMC&lpg=PA265&ots=IpnaLkZUec&lr&pg=PA265#v=onepage&q&f=false Combining local context and WordNet similarity for word sense identification]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, pp. 265-283.

Lin, D. (1998). [http://www.cs.ualberta.ca/~lindek/papers/sim.pdf An information-theoretic definition of similarity]. ''Proceedings of the 15th International Conference on Machine Learning (ICML-98)'', Madison, WI, pp. 296-304.

Lofi, C. (2013). [http://www.ifis.cs.tu-bs.de/sites/default/files/biblio/13cdim_final_pdf_80985.pdf Just ask a human?--Controlling Quality in Relational Similarity and Analogy Processing using the Crowd]. ''Proceedings of the Workshop of the 15th BTW Conference on Database Systems for Business, Technology, and Web (BTW 2013)'', Magdeburg, Germany, pp. 197-210.

Mangalath, P., Quesada, J., and Kintsch, W. (2004). [http://www.cogsci.northwestern.edu/cogsci2004/ma/ma355.pdf Analogy-making as predication using relational information and LSA vectors]. In K.D. Forbus, D. Gentner & T. Regier (Eds.), ''Proceedings of the 26th Annual Meeting of the Cognitive Science Society''. Chicago: Lawrence Erlbaum Associates.

Resnik, P. (1995). [http://citeseer.ist.psu.edu/resnik95using.html Using information content to evaluate semantic similarity]. ''Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95)'', Montreal, pp. 448-453.

Speer, Rob, Joshua Chin and Catherine Havasi. (2017). [http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972 ConceptNet 5.5: An Open Multilingual Graph of General Knowledge]. Proceedings of The 31st AAAI Conference on Artificial Intelligence, San Francisco, CA.

Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). [http://arxiv.org/abs/cs.CL/0309035 Combining independent modules to solve multiple-choice synonym and analogy problems]. ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, pp. 482-489.

Turney, P.D., and Littman, M.L. (2005). [http://arxiv.org/abs/cs.LG/0508103 Corpus-based learning of analogies and semantic relations]. ''Machine Learning'', 60 (1-3), 251-278.

Turney, P.D. (2001). [http://arxiv.org/abs/cs.LG/0212033 Mining the Web for synonyms: PMI-IR versus LSA on TOEFL]. ''Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001)'', Freiburg, Germany, pp. 491-502.

Turney, P.D. (2006a). [http://arxiv.org/abs/cs.CL/0607120 Expressing implicit semantic relations without supervision]. ''Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (Coling/ACL-06)'', Sydney, Australia, pp. 313-320.

Turney, P.D. (2006b). [http://arxiv.org/abs/cs.CL/0608100 Similarity of semantic relations]. ''Computational Linguistics'', 32 (3), 379-416.

Turney, P.D. (2008). [http://arxiv.org/abs/0809.0124 A uniform approach to analogies, synonyms, antonyms, and associations]. ''Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)'', Manchester, UK, pp. 905-912.

Turney, P.D. (2012). [http://jair.org/papers/paper3640.html Domain and function: A dual-space model of semantic relations and compositions], ''Journal of Artificial Intelligence Research (JAIR)'', 44, 533-585.

Turney, P.D. (2013), [http://aclweb.org/anthology/Q/Q13/Q13-1029.pdf Distributional semantics beyond words: Supervised learning of analogy and paraphrase], ''Transactions of the Association for Computational Linguistics (TACL)'', 1, 353-366.

Veale, T. (2004). [http://afflatus.ucd.ie/Papers/ecai2004.pdf WordNet sits the SAT: A knowledge-based approach to lexical analogy]. ''Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004)'', pp. 606–612, Valencia, Spain.

[[Category:State of the art]]
[[Category:Similarity]]
[[Category:Analogy]]

WordSimilarity-353 Test Collection (State of the art)

2017-03-22T15:58:45Z

Rspeer: Add ConceptNet Numberbatch results. (Also, should there be pages for newer/larger/multilingual evaluations?)

* [http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/ WordSimilarity-353 Test Collection]
* contains two sets of English word pairs along with human-assigned similarity judgements
* first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
* second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
* WordSimilarity-353 dataset is available [http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/ here]
* performance is measured by [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rank correlation coefficient]
* introduced by [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Finkelstein et al. (2002)]
* subsequently used by many other researchers
* see also: [[Similarity (State of the art)]]

== Table of results ==

* '''Listed in order of increasing [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rho].'''

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm
! Reference for algorithm
! Reference for reported results
! Type
! Spearman's rho
! Pearson's r
|-
| L&C
| Leacock and Chodorow (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.302
| 0.356
|-
| WNE
| Jarmasz (2003)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.305
| 0.271
|-
| J&C
| Jiang and Conrath 1997
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.318
| 0.354
|-
| L&C
| Leacock and Chodorow (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.348
| 0.341
|-
| H&S
| Hirst and St-Onge (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.302
| 0.356
|-
| Lin
| Lin (1998)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.348
| 0.357
|-
| Resnik
| Resnik (1995)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.353
| 0.365
|-
| ROGET
| Jarmasz (2003)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.415
| 0.536
|-
| C&W
| Collobert and Weston (2008)
| Collobert and Weston (2008)
| Corpus-based
| 0.5
| N/A
|-
| WikiRelate
| Strube and Ponzetto (2006)
| Strube and Ponzetto (2006)
| Corpus-based
| N/A
| 0.48
|-
| LSA
| Landauer et al. (1997)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.581
| 0.492
|-
| LSA
| Landauer et al. (1997)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.581
| 0.563
|-
| simVB+simWN
| Finkelstein et al. (2002)
| Finkelstein et al. (2002)
| Hybrid
| N/A
| 0.55
|-
| SSA
| Hassan and Mihalcea (2011)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.622
| 0.629
|-
| HSMN+csmRNN
| Luong et al. (2013)
| Luong et al. (2013)
| Corpus-based
| 0.65
| N/A
|-
| Multi-prototype
| Huang et al. (2012)
| Huang et al. (2012)
| Corpus-based
| 0.71
| N/A
|-
| Multi-lingual SSA
| Hassan et al. (2011)
| Hassan et al. (2011)
| Corpus-based
| 0.713
| 0.674
|-
| ESA
| Gabrilovich and Markovitch (2007)
| Gabrilovich and Markovitch (2007)
| Corpus-based
| 0.748
| 0.503
|-
| TSA
| Radinsky et al. (2011)
| Radinsky et al. (2011)
| Hybrid
| 0.80
| N/A
|-
| CLEAR
| Halawi et al. (2012)
| Halawi et al. (2012)
| Corpus-based
| 0.81
| N/A
|-
| Y&Q
| Yih and Qazvinian (2012)
| Yih and Qazvinian (2012)
| Hybrid
| 0.81
| N/A
|-
| ConceptNet Numberbatch
| Speer et al. (2017)
| Speer et al. (2017)
| Hybrid
| 0.828
| N/A
|}

== References ==

* '''Listed in alphabetical order.'''

Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Placing Search in Context: The Concept Revisited]. ACM Transactions on Information Systems, 20(1):116-131.

Gabrilovich, Evgeniy, and Shaul Markovitch, [http://www.cs.technion.ac.il/~gabr/papers/ijcai-2007-sim.pdf Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis], Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007.

Halawi, Guy, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. (2012). [http://gabrilovich.com/publications/papers/Halawi2012LSL.pdf Large-scale learning of word relatedness with constraints]. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414. ACM.

Hassan, Samer, and Rada Mihalcea: [http://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/download/3616/3972/ Semantic Relatedness Using Salient Semantic Analysis]. AAAI 2011

Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.

Huang, Eric H., Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1 (ACL '12), Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 873-882.

Islam, A., and Inkpen, D. 2006. [http://www.site.uottawa.ca/~mdislam/publications/LREC_06_242.pdf Second order co-occurrence pmi for determining the semantic similarity of words]. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.

Jarmasz, M. 2003. [http://www.arxiv.org/pdf/1204.0140 Roget’s thesaurus as a Lexical Resource for Natural Language Processing]. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.

Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.

Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.

Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.

Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.

Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). [http://nlp.stanford.edu/~lmthang/data/papers/conll13_morpho.pdf Better word representations with recursive neural networks for morphology]. CoNLL-2013: 104.

Pilehvar, M.T., D. Jurgens and R. Navigli. [http://wwwusers.di.uniroma1.it/~navigli/pubs/ACL_2013_Pilehvar_Jurgens_Navigli.pdf Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity]. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.

Radinsky, Kira, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. (2011). [http://gabrilovich.com/publications/papers/Radinsky2011WTS.pdf A word at a time: computing word relatedness using temporal semantic analysis]. In Proceedings of the 20th international conference on World wide web, pp. 337-346. ACM.

Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.

Speer, Rob, Joshua Chin and Catherine Havasi. (2017). [http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972 ConceptNet 5.5: An Open Multilingual Graph of General Knowledge]. Proceedings of The 31st AAAI Conference on Artificial Intelligence, San Francisco, CA.

Strube, Michael and Simone Paolo Ponzetto. (2006). [http://www.aaai.org/Papers/AAAI/2006/AAAI06-223.pdf WikiRelate! Computing Semantic Relatedness Using Wikipedia]. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.

Yih, W. and Qazvinian, V. (2012). [http://aclweb.org/anthology/N/N12/N12-1077.pdf Measuring Word Relatedness Using Heterogeneous Vector Space Models]. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012).

[[Category:State of the art]]
[[Category:Similarity]]