Analogy (State of the art) - Revision history

Pdturney: /* Methods to solve analogies */

2017-01-25T22:10:45Z

Methods to solve analogies

← Older revision		Revision as of 22:10, 25 January 2017
Line 93:		Line 93:
	</ref>		</ref>
	* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.		* '''LRCos''' (supervised learning of the target class + cosine similarity to the ''b'' word) <ref name="LRCos"/>.

			=== Pair--Pattern matrix for solving analogies ===

			* '''LRA''' (latent relational analysis) <ref>Turney, P.D. (2006), Similarity of semantic relations, Computational Linguistics, 32 (3), 379-416. Retrieved from http://www.mitpressjournals.org/doi/abs/10.1162/coli.2006.32.3.379</ref>.

			=== Dual-Space method for solving analogies ===

			* '''Dual-Space''' <ref>Turney, P.D. (2012), Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585. Retrieved from http://jair.org/papers/paper3640.html</ref>.

	== Issues with evaluating word embeddings on analogy task==		== Issues with evaluating word embeddings on analogy task==

Pdturney: /* Available analogy datasets (ordered by date) */

2017-01-25T21:15:46Z

Available analogy datasets (ordered by date)

← Older revision		Revision as of 21:15, 25 January 2017
Line 39:		Line 39:
	\| Turney (2008)<ref>Turney, P.D. (2008), The latent relation mapping engine: Algorithm and experiments, Journal of Artificial Intelligence Research (JAIR), 33, 615-655. Retrieved from http://jair.org/papers/paper2693.html</ref>		\| Turney (2008)<ref>Turney, P.D. (2008), The latent relation mapping engine: Algorithm and experiments, Journal of Artificial Intelligence Research (JAIR), 33, 615-655. Retrieved from http://jair.org/papers/paper2693.html</ref>
	\| 20 maps; 140 pairs		\| 20 maps; 140 pairs
	\| NA		\| ~140
	\| [http://jair.org/media/2693/live-2693-4214-jair.txt JAIR LRME dataset]		\| [http://jair.org/media/2693/live-2693-4214-jair.txt JAIR LRME dataset]
	\| [[JAIR Analogical Mapping (State of the art)]]		\| [[JAIR Analogical Mapping (State of the art)]]

Pdturney at 21:01, 25 January 2017

2017-01-25T21:01:47Z

← Older revision		Revision as of 21:01, 25 January 2017
Line 104:		Line 104:

	[[Category:State of the art]]		[[Category:State of the art]]
			[[Category:Analogy]]

Pdturney: /* Available analogy datasets (ordered by date) */

2017-01-25T20:00:31Z

Available analogy datasets (ordered by date)

← Older revision		Revision as of 20:00, 25 January 2017
Line 58:		Line 58:
	\|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]		\|[http://research.microsoft.com/en-us/um/people/gzweig/Pubs/myz_naacl13_test_set.tgz MSR]
	\| [[Syntactic Analogies (State of the art)]]		\| [[Syntactic Analogies (State of the art)]]
	\|~~Syntactic~~ (i.e. morphological) questions only		\|syntactic (i.e. morphological) questions only
	\|-		\|-
	\| Google		\| Google

Pdturney at 19:58, 25 January 2017

2017-01-25T19:58:06Z

← Older revision		Revision as of 19:58, 25 January 2017
Line 31:		Line 31:
	\| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>		\| Turney et al (2003)<ref>Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366 </ref>
	\|374		\|374
	\| ~~misc~~		\| ~374
	\| available on request from Peter Turney		\| available on request from Peter Turney
	\| [[SAT Analogy Questions (State of the art)]]		\| [[SAT Analogy Questions (State of the art)]]
Line 38:		Line 38:
	\| LRME		\| LRME
	\| Turney (2008)<ref>Turney, P.D. (2008), The latent relation mapping engine: Algorithm and experiments, Journal of Artificial Intelligence Research (JAIR), 33, 615-655. Retrieved from http://jair.org/papers/paper2693.html</ref>		\| Turney (2008)<ref>Turney, P.D. (2008), The latent relation mapping engine: Algorithm and experiments, Journal of Artificial Intelligence Research (JAIR), 33, 615-655. Retrieved from http://jair.org/papers/paper2693.html</ref>
	\| 20 ~~mappings~~		\| 20 maps; 140 pairs
	\| NA		\| NA
	\| [http://jair.org/media/2693/live-2693-4214-jair.txt JAIR LRME dataset]		\| [http://jair.org/media/2693/live-2693-4214-jair.txt JAIR LRME dataset]

Pdturney at 17:53, 25 January 2017

2017-01-25T17:53:13Z

← Older revision		Revision as of 17:53, 25 January 2017
Line 96:		Line 96:
	== Issues with evaluating word embeddings on analogy task==		== Issues with evaluating word embeddings on analogy task==

	~~'''~~There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.~~'''~~		There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.

Pdturney at 17:52, 25 January 2017

2017-01-25T17:52:05Z

← Older revision		Revision as of 17:52, 25 January 2017
Line 1:		Line 1:
	* see also: [[State of the art]]		* see also: [[State of the art]]

	== Analogy ~~task~~ ==		== Analogy tasks ==

	A proportional analogy holds between two word pairs: ''a'':''a'' :: ''b'':''b'' (''a'' is to ''a'' as ''b'' is to ''b'')		A proportional analogy holds between two word pairs: ''a'':''a'' :: ''b'':''b'' (''a'' is to ''a'' as ''b'' is to ''b'')

Pdturney at 17:50, 25 January 2017

2017-01-25T17:50:58Z

← Older revision		Revision as of 17:50, 25 January 2017
Line 1:		Line 1:
			* see also: [[State of the art]]

	== Analogy task ==		== Analogy task ==

	A proportional analogy holds between two word pairs: ''a'':''a'' :: ''b'':''b'' (''a'' is to ''a'' as ''b'' is to ''b'')		A proportional analogy holds between two word pairs: ''a'':''a'' :: ''b'':''b'' (''a'' is to ''a'' as ''b'' is to ''b'')
	For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.		For example, ''Tokyo'' is to ''Japan'' as ''Paris'' is to ''France''.
Line 12:		Line 15:

	== Available analogy datasets (ordered by date) ==		== Available analogy datasets (ordered by date) ==

	* Listed by date		* Listed by date

Line 30:		Line 34:
	\| available on request from Peter Turney		\| available on request from Peter Turney
	\| [[SAT Analogy Questions (State of the art)]]		\| [[SAT Analogy Questions (State of the art)]]
	\| ~~different task formulation~~: select the correct answer out of 5 proposed alternatives		\| multiple-choice format: select the correct answer out of 5 proposed alternatives
			\|-
			\| LRME
			\| Turney (2008)<ref>Turney, P.D. (2008), The latent relation mapping engine: Algorithm and experiments, Journal of Artificial Intelligence Research (JAIR), 33, 615-655. Retrieved from http://jair.org/papers/paper2693.html</ref>
			\| 20 mappings
			\| NA
			\| [http://jair.org/media/2693/live-2693-4214-jair.txt JAIR LRME dataset]
			\| [[JAIR Analogical Mapping (State of the art)]]
			\| 20 analogical mapping problems, 10 from science and 10 from common metaphors
	\|-		\|-
	\| SemEval 2012 Task 2		\| SemEval 2012 Task 2
Line 38:		Line 50:
	\|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]		\|[https://sites.google.com/site/semeval2012task2/download SemEval2012-Task2]
	\| [[SemEval-2012 Task 2 (State of the art)]]		\| [[SemEval-2012 Task 2 (State of the art)]]
	\| ~~different task formulation~~: ranking the degree to which a relation applies.		\| ranking format: ranking the degree to which a relation applies.
	\|-		\|-
	\| MSR		\| MSR
Line 69:		Line 81:

	=== Pair-based methods for solving analogies ===		=== Pair-based methods for solving analogies ===

	* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784		* '''vector offset''' a.k.a. '''3CosAdd''' <ref>Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
	</ref>		</ref>
Line 76:		Line 89:

	=== Set-based methods for solving analogies ===		=== Set-based methods for solving analogies ===

	* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf		* '''3CosAvg''' (vector offset averaged over multiple pairs) <ref name="LRCos">Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf
	</ref>		</ref>
Line 81:		Line 95:

	== Issues with evaluating word embeddings on analogy task==		== Issues with evaluating word embeddings on analogy task==

	'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''		'''There is interplay between the chosen embedding, its parameters, particular relations <ref name = "Gladkova2016"/>, and method of solving analogies <ref name="Linzen2016"/> <ref name="LRCos"/>. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to ''explore'' or describe an embedding rather than ''evaluate'' it.'''


	==Notes==		==Notes==
	<references />
			<references/>

	[[Category:State of the art]]		[[Category:State of the art]]

Pdturney: /* Analogy task */

2017-01-07T02:57:47Z

Analogy task

← Older revision		Revision as of 02:57, 7 January 2017
Line 9:		Line 9:
	In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.		In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

	See Church's (2017)<ref>Church, K. W. (2017). Word2Vec. Natural Language Engineering, Volume 23, Issue 1, pp. 155-162, DOI https://doi.org/10.1017/S1351324916000334</ref> analysis of Word2Vec, which argues that the Google analogy dataset is not as challenging as the [[SAT Analogy Questions (State of the art)\|SAT]] (Scholastic Aptitude Test) dataset. Inspection shows that the SAT analogies are all semantic (not syntactic) and involve relatively complex relations. The [https://sites.google.com/site/semeval2012task2/home SemEval-2012 Task 2] website includes a [https://sites.google.com/site/semeval2012task2/documentation taxonomy of semantic relations] derived from human analysis of GRE (Graduate Record Exam) analogies by researchers at [https://www.ets.org/ Educational Testing Service].		See Church's (2017)<ref>Church, K. W. (2017). Word2Vec. Natural Language Engineering, Volume 23, Issue 1, pp. 155-162, DOI https://doi.org/10.1017/S1351324916000334</ref> analysis of Word2Vec, which argues that the Google analogy dataset is not as challenging as the [[SAT Analogy Questions (State of the art)\|SAT]] (Scholastic Aptitude Test) dataset. Inspection shows that the SAT analogies are all semantic (not syntactic) and involve relatively complex relations. The [https://sites.google.com/site/semeval2012task2/home SemEval-2012 Task 2] website includes a [https://sites.google.com/site/semeval2012task2/documentation taxonomy of semantic relations] derived from human analysis of GRE (Graduate Record Exam) analogies by researchers at [https://www.ets.org/ ETS (Educational Testing Service)].

	== Available analogy datasets (ordered by date) ==		== Available analogy datasets (ordered by date) ==

Pdturney: /* Analogy task */

2017-01-07T02:57:16Z

Analogy task

← Older revision		Revision as of 02:57, 7 January 2017
Line 9:		Line 9:
	In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.		In NLP analogies (Mikolov's "linguistic regularities"<ref name = "Mikolov2013a"/>) are interpreted broadly as basically any "similarities between pairs of words" <ref name = "Levy2014"/>, not just semantic.

	See Church's (2017)<ref>Church, K. W. (2017). Word2Vec. Natural Language Engineering, Volume 23, Issue 1, pp. 155-162, DOI https://doi.org/10.1017/S1351324916000334</ref> analysis of Word2Vec, which argues that the Google analogy dataset is not as challenging as the [[SAT Analogy Questions (State of the art)\|SAT]] (Scholastic Aptitude Test) dataset. Inspection shows that the SAT analogies are all semantic (not syntactic) and involve relatively complex relations. The [https://sites.google.com/site/semeval2012task2/home SemEval-2012 Task 2] website includes a [https://sites.google.com/site/semeval2012task2/documentation taxonomy of semantic relations] derived from human analysis of GRE (Graduate Record Exam) analogies by researchers at [https://www.ets.org/\|Educational Testing Service].		See Church's (2017)<ref>Church, K. W. (2017). Word2Vec. Natural Language Engineering, Volume 23, Issue 1, pp. 155-162, DOI https://doi.org/10.1017/S1351324916000334</ref> analysis of Word2Vec, which argues that the Google analogy dataset is not as challenging as the [[SAT Analogy Questions (State of the art)\|SAT]] (Scholastic Aptitude Test) dataset. Inspection shows that the SAT analogies are all semantic (not syntactic) and involve relatively complex relations. The [https://sites.google.com/site/semeval2012task2/home SemEval-2012 Task 2] website includes a [https://sites.google.com/site/semeval2012task2/documentation taxonomy of semantic relations] derived from human analysis of GRE (Graduate Record Exam) analogies by researchers at [https://www.ets.org/ Educational Testing Service].

	== Available analogy datasets (ordered by date) ==		== Available analogy datasets (ordered by date) ==