ACL Wiki - User contributions [en]

SimLex-999 (State of the art)

2015-06-25T12:58:49Z

Minhle: Hill et al. (2014) --> 2014a & 2014b

[http://www.cl.cam.ac.uk/~fh295/simlex.html SimLex-999] aims at a cleaner benchmark of similarity (but not relatedness). Pairs of words were chosen to represent different ranges of similarity and with either high or low association. Subjects were instructed to differentiate between similarity and relatedness and rate regarding the former only.

{|
|-
! Algorithm !! Reference for algorithm !! Reference for reported results !! Type !! Spearman's rho !! Pearson's r !! Notes
|-
| Neural language model
| Collobert & Weston (2008)<ref>R. Collobert and J. Weston. 2008. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In International Conference on Machine Learn- ing, ICML.</ref>
| Hill et al. (2014a)<ref name=simlex/>
| Distributional || 0.268 || - || Trained on Wikipedia
|-
| Neural language model with global context
| Huang et al. (2012)<ref>Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for Computational Linguistics.</ref>
| Hill et al. (2014a)<ref name=simlex/>
| Distributional || 0.098 || - || Trained on Wikipedia
|-
| Word2vec
| Mikolov et al. (2013)<ref>Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.</ref>
| Hill et al. (2014a)<ref name=simlex>Hill, F., Reichart, R., & Korhonen, A. (2014a). SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. Computation and Language.</ref>
| Distributional || 0.414 || - || Trained on Wikipedia
|-
| Lesk
|
| Banjade et al. (2015)<ref name=lemontea>Banjade, R., Maharjan, N., Niraula, N., Rus, V., & Gautam, D. (2015). Lemon and Tea Are Not Similar: Measuring Word-to-Word Similarity by Combining Different Methods. Computational Linguistics and Intelligent Text Processing, 9041, 335–346. doi:10.1007/978-3-319-18111-0_25</ref>
| || 0.404 || 0.347
|-
| UMBC
| Han et al. (2013)<ref>Han, L., Kashyap, A., Finin, T., Mayfield, J., Weese, J.: UMBC EBIQUITY-CORE: Semantic textual similarity systems. In: Proceedings of the Second Joint Conference on Lexical and Computational Semantics, vol. 1, pp. 44–52 (2013)</ref>
| Banjade et al. (2015)<ref name=lemontea/>
| || 0.558 || 0.557 || without using POS information
|-
| SVR4
| Banjade et al. (2015)<ref name=lemontea/>
| Banjade et al. (2015)<ref name=lemontea/>
| Combined || 0.642 || 0.658
|-
| ESA
|
| Banjade et al. (2015)<ref name=lemontea/>
| || 0.271 || 0.145
|-
| RNNenc
| Hill et al. (2014b)<ref name=rnnenc>Hill, F., Cho, K., Jean, S., Devin, C., & Bengio, Y. (2014b). Not All Neural Embeddings are Born Equal, 1–5.</ref>
| Hill et al. (2014b)<ref name=rnnenc/>
| Distributional, multilingual || 0.52 || -
|}

== References ==
<references/>

[[Category:State of the art]]
[[Category:Similarity]]

SimLex-999 (State of the art)

2015-06-25T12:47:04Z

Minhle: Created page with "[http://www.cl.cam.ac.uk/~fh295/simlex.html SimLex-999] aims at a cleaner benchmark of similarity (but not relatedness). Pairs of words were chosen to represent different rang..."

[http://www.cl.cam.ac.uk/~fh295/simlex.html SimLex-999] aims at a cleaner benchmark of similarity (but not relatedness). Pairs of words were chosen to represent different ranges of similarity and with either high or low association. Subjects were instructed to differentiate between similarity and relatedness and rate regarding the former only.

{|
|-
! Algorithm !! Reference for algorithm !! Reference for reported results !! Type !! Spearman's rho !! Pearson's r !! Notes
|-
| Neural language model
| Collobert & Weston (2008)<ref>R. Collobert and J. Weston. 2008. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In International Conference on Machine Learn- ing, ICML.</ref>
| Hill et al. (2014)<ref name=simlex/>
| Distributional || 0.268 || - || Trained on Wikipedia
|-
| Neural language model with global context
| Huang et al. (2012)<ref>Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for Computational Linguistics.</ref>
| Hill et al. (2014)<ref name=simlex/>
| Distributional || 0.098 || - || Trained on Wikipedia
|-
| Word2vec
| Mikolov et al. (2013)<ref>Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.</ref>
| Hill et al. (2014)<ref name=simlex>Hill, F., Reichart, R., & Korhonen, A. (2014). SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. Computation and Language.</ref>
| Distributional || 0.414 || - || Trained on Wikipedia
|-
| Lesk
|
| Banjade et al. (2015)<ref name=lemontea>Banjade, R., Maharjan, N., Niraula, N., Rus, V., & Gautam, D. (2015). Lemon and Tea Are Not Similar: Measuring Word-to-Word Similarity by Combining Different Methods. Computational Linguistics and Intelligent Text Processing, 9041, 335–346. doi:10.1007/978-3-319-18111-0_25</ref>
| || 0.404 || 0.347
|-
| UMBC
| Han et al. (2013)<ref>Han, L., Kashyap, A., Finin, T., Mayfield, J., Weese, J.: UMBC EBIQUITY-CORE: Semantic textual similarity systems. In: Proceedings of the Second Joint Conference on Lexical and Computational Semantics, vol. 1, pp. 44–52 (2013)</ref>
| Banjade et al. (2015)<ref name=lemontea/>
| || 0.558 || 0.557 || without using POS information
|-
| SVR4
| Banjade et al. (2015)<ref name=lemontea/>
| Banjade et al. (2015)<ref name=lemontea/>
| Combined || 0.642 || 0.658
|-
| ESA
|
| Banjade et al. (2015)<ref name=lemontea/>
| || 0.271 || 0.145
|-
| RNNenc
| Hill et al. (2014)<ref name=rnnenc>Hill, F., Cho, K., Jean, S., Devin, C., & Bengio, Y. (2014). Not All Neural Embeddings are Born Equal, 1–5.</ref>
| Hill et al. (2014)<ref name=rnnenc/>
| Distributional, multilingual || 0.52 || -
|}

== References ==
<references/>

[[Category:State of the art]]
[[Category:Similarity]]

WordSimilarity-353 Test Collection (State of the art)

2015-04-29T11:54:52Z

Minhle: replace broken link

* [http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/ WordSimilarity-353 Test Collection]
* contains two sets of English word pairs along with human-assigned similarity judgements
* first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
* second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
* WordSimilarity-353 dataset is available [http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/ here]
* performance is measured by [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rank correlation coefficient]
* introduced by [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Finkelstein et al. (2002)]
* subsequently used by many other researchers
* see also: [[Similarity (State of the art)]]

== Table of results ==

* '''Listed in order of increasing [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rho].'''

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm
! Reference for algorithm
! Reference for reported results
! Type
! Spearman's rho
! Pearson's r
|-
| L&C
| Leacock and Chodorow (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.302
| 0.356
|-
| WNE
| Jarmasz (2003)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.305
| 0.271
|-
| J&C
| Jiang and Conrath 1997
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.318
| 0.354
|-
| L&C
| Leacock and Chodorow (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.348
| 0.341
|-
| H&S
| Hirst and St-Onge (1998)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.302
| 0.356
|-
| Lin
| Lin (1998)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.348
| 0.357
|-
| Resnik
| Resnik (1995)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.353
| 0.365
|-
| ROGET
| Jarmasz (2003)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.415
| 0.536
|-
| C&W
| Collobert and Weston (2008)
| Collobert and Weston (2008)
| Corpus-based
| 0.5
| N/A
|-
| WikiRelate
| Strube and Ponzetto (2006)
| Strube and Ponzetto (2006)
| Corpus-based
| N/A
| 0.48
|-
| LSA
| Landauer et al. (1997)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.581
| 0.492
|-
| LSA
| Landauer et al. (1997)
| Hassan and Mihalcea (2011)
| Corpus-based
| 0.581
| 0.563
|-
| simVB+simWN
| Finkelstein et al. (2002)
| Finkelstein et al. (2002)
| Hybrid
| N/A
| 0.55
|-
| SSA
| Hassan and Mihalcea (2011)
| Hassan and Mihalcea (2011)
| Knowledge-based
| 0.622
| 0.629
|-
| HSMN+csmRNN
| Luong et al. (2013)
| Luong et al. (2013)
| Corpus-based
| 0.65
| N/A
|-
| Multi-prototype
| Huang et al. (2012)
| Huang et al. (2012)
| Corpus-based
| 0.71
| N/A
|-
| Multi-lingual SSA
| Hassan et al. (2011)
| Hassan et al. (2011)
| Corpus-based
| 0.713
| 0.674
|-
| ESA
| Gabrilovich and Markovitch (2007)
| Gabrilovich and Markovitch (2007)
| Corpus-based
| 0.748
| 0.503
|-
| TSA
| Radinsky et al. (2011)
| Radinsky et al. (2011)
| Hybrid
| 0.80
| N/A
|-
| CLEAR
| Halawi et al. (2012)
| Halawi et al. (2012)
| Corpus-based
| 0.81
| N/A
|-
| Y&Q
| Yih and Qazvinian (2012)
| Yih and Qazvinian (2012)
| Hybrid
| 0.81
| N/A
|-
|}

== References ==

* '''Listed in alphabetical order.'''

Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Placing Search in Context: The Concept Revisited]. ACM Transactions on Information Systems, 20(1):116-131.

Gabrilovich, Evgeniy, and Shaul Markovitch, [http://www.cs.technion.ac.il/~gabr/papers/ijcai-2007-sim.pdf Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis], Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007.

Halawi, Guy, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. (2012). [http://gabrilovich.com/publications/papers/Halawi2012LSL.pdf Large-scale learning of word relatedness with constraints]. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414. ACM.

Hassan, Samer, and Rada Mihalcea: [http://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/download/3616/3972/ Semantic Relatedness Using Salient Semantic Analysis]. AAAI 2011

Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.

Islam, A., and Inkpen, D. 2006. [http://www.site.uottawa.ca/~mdislam/publications/LREC_06_242.pdf Second order co-occurrence pmi for determining the semantic similarity of words]. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.

Jarmasz, M. 2003. [http://www.arxiv.org/pdf/1204.0140 Roget’s thesaurus as a Lexical Resource for Natural Language Processing]. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.

Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.

Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.

Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.

Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.

Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). [http://nlp.stanford.edu/~lmthang/data/papers/conll13_morpho.pdf Better word representations with recursive neural networks for morphology]. CoNLL-2013: 104.

Pilehvar, M.T., D. Jurgens and R. Navigli. [http://wwwusers.di.uniroma1.it/~navigli/pubs/ACL_2013_Pilehvar_Jurgens_Navigli.pdf Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity]. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.

Radinsky, Kira, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. (2011). [http://gabrilovich.com/publications/papers/Radinsky2011WTS.pdf A word at a time: computing word relatedness using temporal semantic analysis]. In Proceedings of the 20th international conference on World wide web, pp. 337-346. ACM.

Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.

Strube, Michael and Simone Paolo Ponzetto. (2006). [http://www.aaai.org/Papers/AAAI/2006/AAAI06-223.pdf WikiRelate! Computing Semantic Relatedness Using Wikipedia]. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.

Yih, W. and Qazvinian, V. (2012). [http://aclweb.org/anthology/N/N12/N12-1077.pdf Measuring Word Relatedness Using Heterogeneous Vector Space Models]. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012).

[[Category:State of the art]]

Talk:WordSimilarity-353 Test Collection (State of the art)

2015-02-12T12:46:33Z

Minhle: /* SSA should be corpus-based instead? */ new section

== SSA should be corpus-based instead? ==

[http://web.eecs.umich.edu/~mihalcea/papers/hassan.aaai11.pdf The paper] says:

:In this paper, we introduce a new model called Salient Semantic Analysis (SSA), which incorporates a similar semantic abstraction and interpretation of words, by using salient concepts gathered from encyclopedic knowledge.
:
:The main idea underlying our method is that we can determine the semantic relatedness of words by measuring the distance between their concept-based profiles, where a profile consists of salient concepts '''occurring within contexts across a very large corpus'''. Unlike '''previous corpus-based methods''' of relatedness, which utilize word-word associations to create contextualized profiles, our model utilizes concepts that frequently co-occur with a given word.

[[User:Minhle|Minhle]] ([[User talk:Minhle|talk]]) 05:46, 12 February 2015 (MST)