ACL Wiki - User contributions [en]

Google analogy test set (State of the art)

2017-01-06T10:07:44Z

Anna gladkova:

* Test set developed by Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
* Contains 19544 question pairs (8,869 semantic and 10,675 syntactic (i.e. morphological) questions)
* 14 types of relations (9 morphological and 5 semantic)
* [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]

This page reports results obtained with the "vanilla" 3CosAdd method, or vector offset<ref name="Mikolov2013"/>. For other methods, see [[Analogy (State of the art)]]

== Table of results ==

* '''Listed in chronological order.'''

{| border="1" cellpadding="5" cellspacing="1"
|-
! Model
! Reference
! Sem
! Syn
! Corpus and window size
|-
| CBOW (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013">Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 24.0
| 64.0
| 6B Google News corpus, window 10
|-
| Skip-Gram (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 55.0
| 59.0
| ibid
|-
| RNNLM (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 9.0
| 36.0
|
|-
| NNLM (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 23.0
| 53.0
|
|-
| GloVe (300 dim)
| Pennington et al (2014) <ref name = "GloVe">Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) (Vol. 12, pp. 1532–1543). Retrieved from http://llcao.net/cu-deeplearning15/presentation/nn-pres.pdf</ref>
| 81.9
| 69.3
| 42 B corpus, window 5
|-
| SVD
| Levy et al (2015) <ref name = "Levy2015">Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3, 211–225.</ref>
| colspan="2" style="text-align:center;"|55.4
| Wikipedia 1.5B, window 2
|-
| PPMI
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|55.3
| ibid
|-
| Skip-Gram
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|67.6
| ibid
|-
| GloVe
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|56.9
| ibid
|-
| Skip-Gram (50 dim)
| Lai et al (2015) <ref name = "Lai2015">Lai, S., Liu, K., Xu, L., & Zhao, J. (2015). How to Generate a Good Word Embedding? arXiv Preprint arXiv:1507.05523. Retrieved from http://arxiv.org/abs/1507.05523</ref>
| 44.8
| 44.43
| W&N 2.8 B corpus, window 5
|-
| CBOW (50 dim)
| Lai et al (2015) <ref name = "Lai2015"/>
| 44.43
| 55.83
| ibid
|-
| DVRS+SG (300 dim)
| Garten et al (2015) <ref>Garten, J., Sagae, K., Ustun, V., & Dehghani, M. (2015). Combining Distributed Vector Representations for Words. In Proceedings of NAACL-HLT (pp. 95–101). Retrieved from http://www.researchgate.net/profile/Volkan_Ustun/publication/277332298_Combining_Distributed_Vector_Representations_for_Words/links/55705a6308aee1eea7586e93.pdf</ref>
| 74.0
| 60.0
| enwiki9, window 10
|}

== Methodological Issues ==

* This test set is not balanced: 20-70 pairs per category, different number of semantic and morphological relations. See other sets at [[Analogy (State of the art)]].
* In the semantic part, ''country:capital'' relation accounts for over 50% of all semantic questions.
* Researchers usually report only the average accuracy for all semantic/syntactic questions, but there is a lot of variation for individual relations - between 10.53% and 99.41% <ref>Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf</ref>, also depending on parameters of the model <ref>Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>. Since the test is not balanced, the above results could be flattering to the embeddings, and averaging the mean scores for each subcategory would yield lower results.
* Accuracy also depends on the method with which analogies are solved <ref>Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref> <ref>Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>. Set-based methods<ref>Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf</ref> considerably outperform pair-based methods, showing that models do in fact encode much "missed" information.

== References ==

[[Category:State of the art]]

Bigger analogy test set (State of the art)

2017-01-06T10:06:22Z

Anna gladkova:

== Dataset description ==
* New dataset proposed by Gladkova et al. (2016) <ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf
</ref>
* available [http://vsm.blackbird.pw/bats here]
* dataset balanced across 4 types of relations (inflectional morphology, derivational morphology, lexicographic semantics, encyclopedic semantics)
* 10 relations of each type, 50 unique pairs per category
* 99,200 questions in total
* more challenging than the Google set because of more diverse relations
* where applicable, more than one correct answer is supplied (e.g. both ''canine'' and ''animal'' are hypernyms of ''dog'').
* comes with a testing script [https://github.com/undertherain/vsmlib/blob/master/scripts/test_analogy.py a testing script] that implements 5 methods of solving analogies (See [[Analogy (State of the art)]])

This page reports results obtained with the "vanilla" 3CosAdd method, or vector offset<ref name = "Mikolov2013">Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>.

== Table of results ==

* '''Listed in chronological order.'''

{| border="1" cellpadding="5" cellspacing="1"
|-
! Model
! Reference
! Inflectional <br/> morphology
! Derivational <br/> morphology
! Lexicographic <br/> semantics
! Encyclopedic <br/> semantics
! Corpus, window size, vector size
|-
| SVD
| Drozd et al. (2016) <ref name = "Drozd2016">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf</ref>
|44.0
|9.8
|10.1
|18.5
| 5B corpus (Araneum + Wikipedia + UkWac), window 3, 1000 dimensions
|-
| GloVe
| Drozd et al. (2016) <ref name = "Drozd2016"/>
| 59.9
| 10.2
| 10.9
|31.5
| 5B corpus (Araneum + Wikipedia + UkWac), window 8, 300 dimensions
|-
| Skip-Gram
| Drozd et al. (2016) <ref name = "Drozd2016"/>
| 61.0
| 11.2
| 9.1
| 26.5
| 5B corpus (Araneum + Wikipedia + UkWac), window 8, 300 dimensions
|}

== Methodological issues ==

* As with other analogy test sets, accuracy depends not only on the embedding and its parameters, but also on the method with which analogies are solved <ref>Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref> <ref>Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>. Set-based methods<ref>Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf</ref> considerably outperform pair-based methods, showing that models do in fact encode much "missed" information.
* Therefore it is more accurate to think of analogy task as a way to describe and characterize an embedding, rather than evaluate it.

== References ==

[[Category:State of the art]]

Bigger analogy test set (State of the art)

2017-01-06T09:57:17Z

Anna gladkova:

Analogy (State of the art)

2017-01-06T09:56:35Z

Anna gladkova:

== Analogy task ==
A proportional analogy holds between two word pairs: ''a'':''a*'' :: ''b'':''b*'' (''a'' is to ''a*'' as ''b'' is to ''b*'')
For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.

With the '''pair-based''' methods, given ''a'':''a*'' :: ''b'':''?'', the task is to find ''b*''.

With '''set-based''' methods, the task is to find ''b*'' given a set of other pairs (excluding ''b'':''b*'') that hold the same relation as ''b'':''b*''.

In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

== Available analogy datasets (ordered by date) ==
* Listed by date

{| border="1" cellpadding="5" cellspacing="1"
|-
! Dataset
! Reference
! Number of questions
! Number of relations
! Dataset Link
! List of state-of-the-art results
!Comments
|-
| SAT
| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
|374
| misc
| available on request from Peter Turney
| [[SAT Analogy Questions (State of the art)]]
| different task formulation: select the correct answer out of 5 proposed alternatives
|-
| SemEval 2012 Task 2
| Jurgens et al (2012)<ref>Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693</ref>
| 3218
|79
|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
| [[SemEval-2012 Task 2 (State of the art)]]
| different task formulation: ranking the degree to which a relation applies.
|-
| MSR
| Mikolov et al. (2013a)<ref name = "Mikolov2013a">Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784</ref>
| 8,000
| 8
|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
| [[Syntactic Analogies (State of the art)]]
|Syntactic (i.e. morphological) questions only
|-
| Google
| Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 19544
| 15
| [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]
| [[Google analogy test set (State of the art)]]
| unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; ''country:capital'' relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
|-
| BATS
| Gladkova et al. (2016)<ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>
| 99,200
| 40
|[http://vsm.blackbird.pw/bats BATS]
| [[Bigger analogy test set (State of the art)]]
|balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
|-
|}

==Methods to solve analogies==

=== Pair-based methods for solving analogies ===
* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
</ref>
* '''3CosMul''' <ref name = "Levy2014">Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>
* others discussed by Linzen (2016)<ref name="Linzen2016">Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>.

=== Set-based methods for solving analogies ===
* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
</ref>
* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

== Issues with evaluating word embeddings on analogy task==
'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''

==Notes==
<references />

[[Category:State of the art]]

Analogy (State of the art)

2017-01-06T09:54:12Z

Anna gladkova:

== Analogy task ==
A proportional analogy holds between two word pairs: ''a'':''a*'' :: ''b'':''b*'' (''a'' is to ''a*'' as ''b'' is to ''b*'')
For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.

With the '''pair-based''' methods, given ''a'':''a*'' :: ''b'':''?'', the task is to find ''b*''.

With '''set-based''' methods, the task is to find ''b*'' given a set of other pairs (excluding ''b'':''b*'') that hold the same relation as ''b'':''b*''.

In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

== Available analogy datasets (ordered by date) ==
* Listed by date

{| border="1" cellpadding="5" cellspacing="1"
|-
! Dataset
! Reference
! Number of questions
! Number of relations
! Dataset Link
! List of state-of-the-art results
!Comments
|-
| SAT
| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
|374
| misc
| available on request from Peter Turney
| [[SAT Analogy Questions (State of the art)]]
| different task formulation: select the correct answer out of 5 proposed alternatives
|-
| SemEval 2012 Task 2
| Jurgens et al (2012)<ref>Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693</ref>
| 3218
|79
|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
| [[SemEval-2012 Task 2 (State of the art)]]
| different task formulation: ranking the degree to which a relation applies.
|-
| MSR
| Mikolov et al. (2013a)<ref name = "Mikolov2013a">Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784</ref>
| 8,000
| 8
|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
| [[Syntactic Analogies (State of the art)]]
|Syntactic (i.e. morphological) questions only
|-
| Google
| Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 19544
| 15
| [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]
| [[Google analogy test set (State of the art)]]
| unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; ''country:capital'' relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
|-
| BATS
| Gladkova et al. (2016)<ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>
| 99,200
| 40
|[https://s3.amazonaws.com/blackbirdprojects/tut_vsm/BATS_3.0.zip BATS]
| [[Bigger analogy test set (State of the art)]]
|balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
|-
|}

==Methods to solve analogies==

=== Pair-based methods for solving analogies ===
* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
</ref>
* '''3CosMul''' <ref name = "Levy2014">Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>
* others discussed by Linzen (2016)<ref name="Linzen2016">Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>.

=== Set-based methods for solving analogies ===
* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
</ref>
* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

== Issues with evaluating word embeddings on analogy task==
'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''

==Notes==
<references />

[[Category:State of the art]]

Bigger analogy test set (State of the art)

2017-01-06T09:52:07Z

Anna gladkova: This page lists published results on Bigger Analogy Test Set (BATS)

Analogy (State of the art)

2017-01-06T07:02:49Z

Anna gladkova:

== Analogy task ==
A proportional analogy holds between two word pairs: ''a'':''a*'' :: ''b'':''b*'' (''a'' is to ''a*'' as ''b'' is to ''b*'')
For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.

With the '''pair-based''' methods, given ''a'':''a*'' :: ''b'':''?'', the task is to find ''b*''.

With '''set-based''' methods, the task is to find ''b*'' given a set of other pairs (excluding ''b'':''b*'') that hold the same relation as ''b'':''b*''.

In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

== Available analogy datasets (ordered by date) ==
* Listed by date

{| border="1" cellpadding="5" cellspacing="1"
|-
! Dataset
! Reference
! Number of questions
! Number of relations
! Dataset Link
! List of state-of-the-art results
!Comments
|-
| SAT
| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
|374
| misc
| available on request from Peter Turney
| [[SAT Analogy Questions (State of the art)]]
| different task formulation: select the correct answer out of 5 proposed alternatives
|-
| SemEval 2012 Task 2
| Jurgens et al (2012)<ref>Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693</ref>
| 3218
|79
|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
| [[SemEval-2012 Task 2 (State of the art)]]
| different task formulation: ranking the degree to which a relation applies.
|-
| MSR
| Mikolov et al. (2013a)<ref name = "Mikolov2013a">Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784</ref>
| 8,000
| 8
|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
| [[Syntactic Analogies (State of the art)]]
|Syntactic (i.e. morphological) questions only
|-
| Google
| Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 19544
| 15
| [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]
| [[Google analogy test set (State of the art)]]
| unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; ''country:capital'' relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
|-
| BATS
| Gladkova et al. (2016)<ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>
| 99,200
| 40
|[https://s3.amazonaws.com/blackbirdprojects/tut_vsm/BATS_3.0.zip BATS]
|
|balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
|-
|}

==Methods to solve analogies==

=== Pair-based methods for solving analogies ===
* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
</ref>
* '''3CosMul''' <ref name = "Levy2014">Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>
* others discussed by Linzen (2016)<ref name="Linzen2016">Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>.

=== Set-based methods for solving analogies ===
* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
</ref>
* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

== Issues with evaluating word embeddings on analogy task==
'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''

==Notes==
<references />

[[Category:State of the art]]

Google analogy test set (State of the art)

2017-01-06T07:01:57Z

Anna gladkova: Created page with "* Test set developed by Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proc..."

* Test set developed by Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
* Contains 19544 question pairs (8,869 semantic and 10,675 syntactic (i.e. morphological) questions)
* 14 types of relations (9 morphological and 5 semantic)
* [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]

This page reports results obtained with the "vanilla" 3CosAdd method, or vector offset<ref name="Mikolov2013"/>. For other methods, see [[Analogy (State of the art)]]

== Table of results ==

* '''Listed in chronological order.'''

{| border="1" cellpadding="5" cellspacing="1"
|-
! Model
! Reference
! Sem
! Syn
! Corpus and window size
|-
| CBOW (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013">Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 24.0
| 64.0
| 6B Google News corpus, window 10
|-
| Skip-Gram (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 55.0
| 59.0
| ibid
|-
| RNNLM (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 9.0
| 36.0
|
|-
| NNLM (640 dim)
| Mikolov et al (2013) <ref name = "Mikolov2013"/>
| 23.0
| 53.0
|
|-
| GloVe (300 dim)
| Pennington et al (2014) <ref name = "GloVe">Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) (Vol. 12, pp. 1532–1543). Retrieved from http://llcao.net/cu-deeplearning15/presentation/nn-pres.pdf</ref>
| 81.9
| 69.3
| 42 B corpus, window 5
|-
| SVD
| Levy et al (2015) <ref name = "Levy2015">Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3, 211–225.</ref>
| colspan="2" style="text-align:center;"|55.4
| Wikipedia 1.5B, window 2
|-
| PPMI
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|55.3
| ibid
|-
| Skip-Gram
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|67.6
| ibid
|-
| GloVe
| Levy et al (2015) <ref name = "Levy2015"/>
| colspan="2" style="text-align:center;"|56.9
| ibid
|-
| Skip-Gram (50 dim)
| Lai et al (2015) <ref name = "Lai2015">Lai, S., Liu, K., Xu, L., & Zhao, J. (2015). How to Generate a Good Word Embedding? arXiv Preprint arXiv:1507.05523. Retrieved from http://arxiv.org/abs/1507.05523</ref>
| 44.8
| 44.43
| W&N 2.8 B corpus, window 5
|-
| CBOW (50 dim)
| Lai et al (2015) <ref name = "Lai2015"/>
| 44.43
| 55.83
| ibid
|-
| DVRS+SG (300 dim)
| Garten et al (2015) <ref>Garten, J., Sagae, K., Ustun, V., & Dehghani, M. (2015). Combining Distributed Vector Representations for Words. In Proceedings of NAACL-HLT (pp. 95–101). Retrieved from http://www.researchgate.net/profile/Volkan_Ustun/publication/277332298_Combining_Distributed_Vector_Representations_for_Words/links/55705a6308aee1eea7586e93.pdf</ref>
| 74.0
| 60.0
| enwiki9, window 10
|}

== Methodological Issues ==

* This test set is not balanced: 20-70 pairs per category, different number of semantic and morphological relations. See other sets at [[Analogy (State of the art)]].
* In the semantic part, ''country:capital'' relation accounts for over 50% of all semantic questions.
* Researchers usually report only the average accuracy for all semantic/syntactic questions, but there is a lot of variation for individual relations - between 10.53% and 99.41% <ref>Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf</ref>, also depending on parameters of the model <ref>Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>. Since the test is not balanced, the above results could be flattering to the embeddings, and averaging the mean scores for each subcategory would yield lower results.
* Accuracy also depends on the method with which analogies are solved <ref>Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>. Set-based methods<ref>Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf</ref> considerably outperform pair-based methods, showing that models do in fact encode much "missed" information.

== References ==

[[Category:State of the art]]

Analogy (State of the art)

2017-01-06T05:31:20Z

Anna gladkova:

== Analogy task ==
A proportional analogy holds between two word pairs: ''a'':''a*'' :: ''b'':''b*'' (''a'' is to ''a*'' as ''b'' is to ''b*'')
For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.

With the '''pair-based''' methods, given ''a'':''a*'' :: ''b'':''?'', the task is to find ''b*''.

With '''set-based''' methods, the task is to find ''b*'' given a set of other pairs (excluding ''b'':''b*'') that hold the same relation as ''b'':''b*''.

In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

== Available analogy datasets (ordered by date) ==
* Listed by date

{| border="1" cellpadding="5" cellspacing="1"
|-
! Dataset
! Reference
! Number of questions
! Number of relations
! Dataset Link
! List of state-of-the-art results
!Comments
|-
| SAT
| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
|374
| misc
| available on request from Peter Turney
| [[SAT Analogy Questions (State of the art)]]
| different task formulation: select the correct answer out of 5 proposed alternatives
|-
| SemEval 2012 Task 2
| Jurgens et al (2012)<ref>Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693</ref>
| 3218
|79
|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
| [[SemEval-2012 Task 2 (State of the art)]]
| different task formulation: ranking the degree to which a relation applies.
|-
| MSR
| Mikolov et al. (2013a)<ref name = "Mikolov2013a">Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784</ref>
| 8,000
| 8
|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
| [[Syntactic Analogies (State of the art)]]
|Syntactic (i.e. morphological) questions only
|-
| Google
| Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 19544
| 15
| [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]
|
| unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; ''country:capital'' relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
|-
| BATS
| Gladkova et al. (2016)<ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>
| 99,200
| 40
|[https://s3.amazonaws.com/blackbirdprojects/tut_vsm/BATS_3.0.zip BATS]
|
|balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
|-
|}

==Methods to solve analogies==

=== Pair-based methods for solving analogies ===
* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
</ref>
* '''3CosMul''' <ref name = "Levy2014">Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>
* others discussed by Linzen (2016)<ref name="Linzen2016">Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>.

=== Set-based methods for solving analogies ===
* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
</ref>
* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

== Issues with evaluating word embeddings on analogy task==
'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''

==Notes==
<references />

[[Category:State of the art]]

Analogy (State of the art)

2017-01-06T05:08:03Z

Anna gladkova: This page lists available datasets and methods for solving analogies with distributional models

== Analogy task ==
A proportional analogy holds between two word pairs: ''a'':''a*'' :: ''b'':''b*'' (''a'' is to ''a*'' as ''b'' is to ''b*'')
For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.

With the '''pair-based''' methods, given ''a'':''a*'' :: ''b'':''?'', the task is to find ''b*''.

With '''set-based''' methods, the task is to find ''b*'' given a set of other pairs (excluding ''b'':''b*'') that hold the same relation as ''b'':''b*''.

In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

== Available analogy datasets (ordered by date) ==
* Listed by date

{| border="1" cellpadding="5" cellspacing="1"
|-
! Dataset
! Reference
! Number of questions
! Number of relations
! Dataset Link
! List of state-of-the-art results
!Comments
|-
| SAT
| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
|374
| misc
| available on request from Peter Turney
| [[SAT Analogy Questions (State of the art)]]
| different task formulation: select the correct answer out of 5 proposed alternatives
|-
| SemEval 2012 Task 2
| Jurgens et al (2012)<ref>Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693</ref>
| 3218
|79
|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
| [[SemEval-2012 Task 2 (State of the art)]]
| different task formulation: ranking the degree to which a relation applies.
|-
| MSR
| Mikolov et al. (2013a)<ref name = "Mikolov2013a">Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784</ref>
| 8,000
| 8
|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
| [[Syntactic Analogies (State of the art)]]
|Syntactic (i.e. morphological) questions only
|-
| Google
| Mikolov et al. (2013b)<ref>Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).</ref>
| 19544
| 15
| [http://download.tensorflow.org/data/questions-words.txt Original link deprecated, copy hosted @TensorFlow]
|
| unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; ''country:capital'' relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
|-
| BATS
| Gladkova et al. (2016)<ref name = "Gladkova2016">Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf</ref>
| 99,200
| 40
|[https://s3.amazonaws.com/blackbirdprojects/tut_vsm/BATS_3.0.zip BATS]
|
|balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
|-
|}

==Methods to solve analogies==

=== Pair-based methods for solving analogies ===
* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
</ref>
* '''3CosMul''' <ref name = "Levy2014">Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
</ref>
* others discussed by Linzen (2016)<ref name="Linzen2016">Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503</ref>.

=== Set-based methods for solving analogies ===
* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
</ref>
* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

== Issues with evaluating word embeddings on analogy task==
'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''

==Notes==
<references />