Difference between revisions of "MC-28 Test Collection (State of the art)"

From ACL Wiki
Jump to navigation Jump to search
(Created page with "* state of the art in Miller & Charles 28 (MC-28) dataset * 28 word pairs of the original Miller & Charles 30 (MC-30) dataset, which is a subset of the RG-65 Test Collection...")
 
m
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
* state of the art in Miller & Charles 28 (MC-28) dataset
+
* state of the art in Miller & Charles 28 (MC-28) dataset [Resnik, 1995]
−
* 28 word pairs of the original Miller & Charles 30 (MC-30) dataset, which is a subset of the [[RG-65 Test Collection (State of the art)|Rubenstein & Goodenough (RG-65) dataset]]; two word pairs have generally been omitted for semantic similarity evaluation, as words in these word pairs have not been included in previous versions of WordNet
+
* 28 word pairs of the original Miller & Charles 30 (MC-30) dataset [Miller and Charles, 1991], which is a subset of the [[RG-65 Test Collection (State of the art)|Rubenstein & Goodenough (RG-65) dataset]]; two word pairs have generally been omitted for semantic similarity evaluation, as words in these word pairs have not been included in previous versions of WordNet
 
* Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
 
* Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
 
* The similarity values in the dataset are the means of judgments made by 38 subjects [Miller and Charles, 1991].
 
* The similarity values in the dataset are the means of judgments made by 38 subjects [Miller and Charles, 1991].
Line 36: Line 36:
 
| Lexicon-based
 
| Lexicon-based
 
| 0.91 [0.813, 0.957]
 
| 0.91 [0.813, 0.957]
 +
|-
 +
| Do19-hybrid
 +
| Dobó (2019)
 +
| Dobó (2019)
 +
| Hybrid
 +
| 0.893 [0.780, 0.949]
 +
|-
 +
| Sp17
 +
| Speer et al. (2017)
 +
| Dobó (2019)
 +
| Hybrid
 +
| 0.892 [0.778, 0.949]
 
|-
 
|-
 
| JS
 
| JS
Line 48: Line 60:
 
| Lexicon-based
 
| Lexicon-based
 
| 0.856 [0.710, 0.931]
 
| 0.856 [0.710, 0.931]
 +
|-
 +
| Do19-corpus
 +
| Dobó (2019)
 +
| Dobó (2019)
 +
| Corpus-based
 +
| 0.853 [0.704, 0.930]
 
|-
 
|-
 
| KC
 
| KC
Line 54: Line 72:
 
| Web-based
 
| Web-based
 
| 0.835 [0.671, 0.921]
 
| 0.835 [0.671, 0.921]
 +
|-
 +
| Pe14
 +
| Pennington et al. (2014)
 +
| Dobó (2019)
 +
| Corpus-based
 +
| 0.832 [0.666, 0.919]
 +
|-
 +
| Sa18
 +
| Salle et al. (2018)
 +
| Dobó (2019)
 +
| Corpus-based
 +
| 0.822 [0.648, 0.914]
 
|-
 
|-
 
| LIN
 
| LIN
Line 67: Line 97:
 
| 0.81 [0.627, 0.908]
 
| 0.81 [0.627, 0.908]
 
|-
 
|-
−
| bnc-bagofwords-num-cos-qw+enwiki-parsed-num-cos-freq
+
| DC13
 
| Dobó and Csirik (2013)
 
| Dobó and Csirik (2013)
 
| Dobó and Csirik (2013)
 
| Dobó and Csirik (2013)
Line 99: Line 129:
  
 
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., Soroa, A. (2009). [http://www.aclweb.org/anthology/N09-1003 A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches]. In: ''10th Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies''. Association for Computa-tional Linguistics, Stroudsburg. pp. 19–27.
 
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., Soroa, A. (2009). [http://www.aclweb.org/anthology/N09-1003 A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches]. In: ''10th Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies''. Association for Computa-tional Linguistics, Stroudsburg. pp. 19–27.
 +
 +
Dobó, A. (2019). [http://doktori.bibl.u-szeged.hu/10120/1/AndrasDoboThesis2019.pdf A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages]. University of Szeged. [https://github.com/doboandras/dsm-parameter-analysis GitHub repository]
  
 
Dobó, A., and Csirik, J. (2013). [http://link.springer.com/chapter/10.1007/978-3-642-35843-2_42 Computing semantic similarity using large static corpora]. In: van Emde Boas, P. et al. (eds.) ''SOFSEM 2013: Theory and Practice of Computer Science. LNCS, Vol. 7741''. Springer-Verlag, Berlin Heidelberg, pp. 491-502.
 
Dobó, A., and Csirik, J. (2013). [http://link.springer.com/chapter/10.1007/978-3-642-35843-2_42 Computing semantic similarity using large static corpora]. In: van Emde Boas, P. et al. (eds.) ''SOFSEM 2013: Theory and Practice of Computer Science. LNCS, Vol. 7741''. Springer-Verlag, Berlin Heidelberg, pp. 491-502.
Line 115: Line 147:
  
 
Patwardhan, S., and Pedersen, T. (2006). [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.145.6642&rep=rep1&type=pdf#page=7 Using WordNet-based Context Vectors to Estimate the Se-mantic Relatedness of Concepts]. In: ''11th Conference of the European Chapter of the Association for Computational Linguistics''. Association for Computational Linguistics, Stroudsbur. pp. 1–8.
 
Patwardhan, S., and Pedersen, T. (2006). [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.145.6642&rep=rep1&type=pdf#page=7 Using WordNet-based Context Vectors to Estimate the Se-mantic Relatedness of Concepts]. In: ''11th Conference of the European Chapter of the Association for Computational Linguistics''. Association for Computational Linguistics, Stroudsbur. pp. 1–8.
 +
 +
Pennington, J., Socher, R., and Manning, C. (2014). [https://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation]. ''EMNLP 2014'', pp. 1532-1543.
  
 
Resnik, P. (1995). [http://arxiv.org/pdf/cmp-lg/9511007 Using information content to evaluate semantic similarity]. In ''Proceedings of the 14th International Joint Conference on Artificial Intelligence'', Montreal, Canada, pages 448–453.
 
Resnik, P. (1995). [http://arxiv.org/pdf/cmp-lg/9511007 Using information content to evaluate semantic similarity]. In ''Proceedings of the 14th International Joint Conference on Artificial Intelligence'', Montreal, Canada, pages 448–453.
  
 
Sahami, M., Heilman, T.D. (2006). [http://robotics.stanford.edu/users/sahami/papers-dir/www2006.pdf A web-based kernel function for measuring the similarity of short text snippets]. In: ''15th international conference on World Wide Web''. ACM Press, New York. pp. 377–386.
 
Sahami, M., Heilman, T.D. (2006). [http://robotics.stanford.edu/users/sahami/papers-dir/www2006.pdf A web-based kernel function for measuring the similarity of short text snippets]. In: ''15th international conference on World Wide Web''. ACM Press, New York. pp. 377–386.
 +
 +
Salle A., Idiart M., and Villavicencio A. (2018) [https://github.com/alexandres/lexvec/blob/master/README.md LexVec]
 +
 +
Speer, R., Chin, J., and Havasi, C. (2017). [https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/download/14972/14051 Conceptnet 5.5: An open multilingual graph of general knowledge]. ''AAAI-17'', pp. 4444-4451.
  
 
Tsatsaronis, G., Varlamis, I., and Vazirgiannis, M. (2010). [http://arxiv.org/abs/1401.5699 Text Relatedness Based on a Word Thesaurus]. ''Journal of Artificial Intelligence Research'' 37, 1–39
 
Tsatsaronis, G., Varlamis, I., and Vazirgiannis, M. (2010). [http://arxiv.org/abs/1401.5699 Text Relatedness Based on a Word Thesaurus]. ''Journal of Artificial Intelligence Research'' 37, 1–39

Latest revision as of 18:34, 15 September 2019

  • state of the art in Miller & Charles 28 (MC-28) dataset [Resnik, 1995]
  • 28 word pairs of the original Miller & Charles 30 (MC-30) dataset [Miller and Charles, 1991], which is a subset of the Rubenstein & Goodenough (RG-65) dataset; two word pairs have generally been omitted for semantic similarity evaluation, as words in these word pairs have not been included in previous versions of WordNet
  • Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
  • The similarity values in the dataset are the means of judgments made by 38 subjects [Miller and Charles, 1991].
  • see also: Similarity (State of the art)


Table of results


Algorithm Reference for algorithm Reference for reported results Type Spearman correlation [with 95% confidence intervals]
Human Human upper bound Resnik (1995) Human 0.934 [0.861, 0.969]
PPR Agirre et al. (2009) Agirre et al. (2009) Hybrid 0.92 [0.833, 0.962]
Gloss Vector Patwardhan and Pedersen (2006) Patwardhan and Pedersen (2006) Lexicon-based 0.91 [0.813, 0.957]
Do19-hybrid Dobó (2019) Dobó (2019) Hybrid 0.893 [0.780, 0.949]
Sp17 Speer et al. (2017) Dobó (2019) Hybrid 0.892 [0.778, 0.949]
JS Jarmasz and Szpakowicz (2003) Jarmasz and Szpakowicz (2003) Lexicon-based 0.87 [0.736, 0.938]
SR Tsatsaronis et al. (2010) Tsatsaronis et al. (2010) Lexicon-based 0.856 [0.710, 0.931]
Do19-corpus Dobó (2019) Dobó (2019) Corpus-based 0.853 [0.704, 0.930]
KC Kulkarni and Caragea (2009) Kulkarni and Caragea (2009) Web-based 0.835 [0.671, 0.921]
Pe14 Pennington et al. (2014) Dobó (2019) Corpus-based 0.832 [0.666, 0.919]
Sa18 Salle et al. (2018) Dobó (2019) Corpus-based 0.822 [0.648, 0.914]
LIN Lin (1998) Lin (1998) Hybrid 0.82 [0.644, 0.913]
RES Resnik (1995) Resnik (1995) Hybrid 0.81 [0.627, 0.908]
DC13 Dobó and Csirik (2013) Dobó and Csirik (2013) Corpus-based 0.773 [0.562, 0.889]
GM Gabrilovich and Markovitch (2007) Tsatsaronis et al. (2010) Corpus-based 0.72 [0.475, 0.861]
WLM Milne and Witten (2008) Milne and Witten (2008) Web-based 0.70 [0.443, 0.850]
SH Sahami and Heilman (2006) Agirre et al. (2009) Web-based 0.618 [0.319, 0.805]


References

  • Listed alphabetically.


Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., Soroa, A. (2009). A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. In: 10th Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies. Association for Computa-tional Linguistics, Stroudsburg. pp. 19–27.

Dobó, A. (2019). A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages. University of Szeged. GitHub repository

Dobó, A., and Csirik, J. (2013). Computing semantic similarity using large static corpora. In: van Emde Boas, P. et al. (eds.) SOFSEM 2013: Theory and Practice of Computer Science. LNCS, Vol. 7741. Springer-Verlag, Berlin Heidelberg, pp. 491-502.

Gabrilovich, E., and Markovitch, S. (2007). Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis, Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India.

Jarmasz, M., and Szpakowicz, S. (2003). Roget’s thesaurus and semantic similarity, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, September, pp. 212-219.

Kulkarni, S., Caragea, D. (2009). Computation of the Semantic Relatedness between Words using Concept Clouds. In: International Conference on Knowledge Discovery and Information Re-trieval. INSTICC Press, Setubal. pp. 183–188.

Lin, D. (1998). An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pp. 296–304.

Miller, G., and Charles, W. (1991). Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1), 1–28.

Milne, D., Witten, I.H. (2008). An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links, In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy, AAAI Press, Chicago, USA pp. 25-30.

Patwardhan, S., and Pedersen, T. (2006). Using WordNet-based Context Vectors to Estimate the Se-mantic Relatedness of Concepts. In: 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsbur. pp. 1–8.

Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. EMNLP 2014, pp. 1532-1543.

Resnik, P. (1995). Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, pages 448–453.

Sahami, M., Heilman, T.D. (2006). A web-based kernel function for measuring the similarity of short text snippets. In: 15th international conference on World Wide Web. ACM Press, New York. pp. 377–386.

Salle A., Idiart M., and Villavicencio A. (2018) LexVec

Speer, R., Chin, J., and Havasi, C. (2017). Conceptnet 5.5: An open multilingual graph of general knowledge. AAAI-17, pp. 4444-4451.

Tsatsaronis, G., Varlamis, I., and Vazirgiannis, M. (2010). Text Relatedness Based on a Word Thesaurus. Journal of Artificial Intelligence Research 37, 1–39

Retrieved from "https://aclweb.org/aclwiki/index.php?title=MC-28_Test_Collection_(State_of_the_art)&oldid=12676"