Analogy (State of the art)

From ACL Wiki
Jump to navigation Jump to search

Analogy task

A proportional analogy holds between two word pairs: a:a* :: b:b* (a is to a* as b is to b*) For example, Tokyo is to Japan as Paris is to France.

With the pair-based methods, given a:a* :: b:?, the task is to find b*.

With set-based methods, the task is to find b* given a set of other pairs (excluding b:b*) that hold the same relation as b:b*.

In NLP analogies (Mikolov's "linguistic regularities"[1]) are interpreted broadly as basically any "similarities between pairs of words" [2], not just semantic.

See Church's (2017)[3] analysis of Word2Vec, which argues that the Google analogy dataset is not as challenging as the SAT (Scholastic Aptitude Test) dataset. Inspection shows that the SAT analogies are all semantic (not syntactic) and involve relatively complex relations. The SemEval-2012 Task 2 website includes a taxonomy of semantic relations derived from human analysis of GRE (Graduate Record Exam) analogies by researchers at ETS (Educational Testing Service).

Available analogy datasets (ordered by date)

  • Listed by date
Dataset Reference Number of questions Number of relations Dataset Link List of state-of-the-art results Comments
SAT Turney et al (2003)[4] 374 misc available on request from Peter Turney SAT Analogy Questions (State of the art) different task formulation: select the correct answer out of 5 proposed alternatives
SemEval 2012 Task 2 Jurgens et al (2012)[5] 3218 79 SemEval2012-Task2 SemEval-2012 Task 2 (State of the art) different task formulation: ranking the degree to which a relation applies.
MSR Mikolov et al. (2013a)[1] 8,000 8 MSR Syntactic Analogies (State of the art) Syntactic (i.e. morphological) questions only
Google Mikolov et al. (2013b)[6] 19544 15 Original link deprecated, copy hosted @TensorFlow Google analogy test set (State of the art) unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; country:capital relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
BATS Gladkova et al. (2016)[7] 99,200 40 BATS Bigger analogy test set (State of the art) balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.

Methods to solve analogies

Pair-based methods for solving analogies

  • vector offset a.k.a. 3CosAdd [8]
  • 3CosMul [2]
  • others discussed by Linzen (2016)[9].

Set-based methods for solving analogies

  • 3CosAvg (vector offset averaged over multiple pairs) [10]
  • LRCos (supervised learning of the target class + cosine similarity to the b word) [10].

Issues with evaluating word embeddings on analogy task

There is interplay between the chosen embedding, its parameters, particular relations [7], and method of solving analogies [9] [10]. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to explore or describe an embedding rather than evaluate it.


Notes

  1. ↑ 1.0 1.1 Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
  2. ↑ 2.0 2.1 Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
  3. ↑ Church, K. W. (2017). Word2Vec. Natural Language Engineering, Volume 23, Issue 1, pp. 155-162, DOI https://doi.org/10.1017/S1351324916000334
  4. ↑ Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366
  5. ↑ Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693
  6. ↑ Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).
  7. ↑ 7.0 7.1 Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf
  8. ↑ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
  9. ↑ 9.0 9.1 Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503
  10. ↑ 10.0 10.1 10.2 Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
Retrieved from "https://aclweb.org/aclwiki/index.php?title=Analogy_(State_of_the_art)&oldid=11745"