SemEval 2012 versus 2013 - Revision history

Pdturney: Reverted edits by Creek (talk) to last revision by Egiesbre

2012-06-25T11:18:53Z

Reverted edits by Creek (talk) to last revision by Egiesbre

← Older revision		Revision as of 11:18, 25 June 2012
Line 91:		Line 91:

	I have been participating in Senseval/Semeval since the beginning, as a participant and as an organizer. I have two major points to make:		I have been participating in Senseval/Semeval since the beginning, as a participant and as an organizer. I have two major points to make:
	(1) As currently envisaged, the identification of a task occurs in the mind of a single individual or small coordinated group. As a result, the task is somewhat random. This year's organizers have made the strong point about combining tasks, so as to have several subtasks that have some coherence. I would suggest that we all need some overall coherence about outstanding semantic evaluation issues, with the hope that the tasks and subtasks are homing in on those problems. I made a start on this with a quick grouping of task types at [http://~~www~~.~~thai-sbobet~~.~~com sbo]~~. I would like to encourage people to exapnd on this grouping with a greater identification of issues. This might facilitate the development of future tasks.		(1) As currently envisaged, the identification of a task occurs in the mind of a single individual or small coordinated group. As a result, the task is somewhat random. This year's organizers have made the strong point about combining tasks, so as to have several subtasks that have some coherence. I would suggest that we all need some overall coherence about outstanding semantic evaluation issues, with the hope that the tasks and subtasks are homing in on those problems. I made a start on this with a quick grouping of task types at http://aclweb.org/aclwiki/index.php?title=SemEval_Portal. I would like to encourage people to exapnd on this grouping with a greater identification of issues. This might facilitate the development of future tasks.
	(2) With the given structure, a task is defined and then somewhat cast in concrete. As both a participant and an organizer, this results in a major difficulty that the task can't evolve and play out. As an organizer, I've found it necessary to have some back and forth before the final form of a task is crystallized. As a participant, it takes some time to play around with trial data and to wrap oneself around the task. The FrameNet linking task in SemEval-2010 is a case in point. It was sufficiently complex that many of those with an initial interest ended up not participating and the original task was subdivided into a couple of smaller tasks. So, what I'm suggesting is that potential organizers throw out an initial, rough conceptualization with some trial data and that the task not be locked down without a full airing of potential issues and refinements. I don't think it's necessary to have a lot of time between the availability of the training data and submission of results (5 months). Once you're pretty sure that you can get the kind of results you want, it's just a matter of making the final run.		(2) With the given structure, a task is defined and then somewhat cast in concrete. As both a participant and an organizer, this results in a major difficulty that the task can't evolve and play out. As an organizer, I've found it necessary to have some back and forth before the final form of a task is crystallized. As a participant, it takes some time to play around with trial data and to wrap oneself around the task. The FrameNet linking task in SemEval-2010 is a case in point. It was sufficiently complex that many of those with an initial interest ended up not participating and the original task was subdivided into a couple of smaller tasks. So, what I'm suggesting is that potential organizers throw out an initial, rough conceptualization with some trial data and that the task not be locked down without a full airing of potential issues and refinements. I don't think it's necessary to have a lot of time between the availability of the training data and submission of results (5 months). Once you're pretty sure that you can get the kind of results you want, it's just a matter of making the final run.

Creek: /* Reasons for 2013 */

2012-06-25T09:30:43Z

Reasons for 2013

← Older revision		Revision as of 09:30, 25 June 2012
Line 91:		Line 91:

	I have been participating in Senseval/Semeval since the beginning, as a participant and as an organizer. I have two major points to make:		I have been participating in Senseval/Semeval since the beginning, as a participant and as an organizer. I have two major points to make:
	(1) As currently envisaged, the identification of a task occurs in the mind of a single individual or small coordinated group. As a result, the task is somewhat random. This year's organizers have made the strong point about combining tasks, so as to have several subtasks that have some coherence. I would suggest that we all need some overall coherence about outstanding semantic evaluation issues, with the hope that the tasks and subtasks are homing in on those problems. I made a start on this with a quick grouping of task types at http://~~aclweb~~.~~org/aclwiki/index~~.~~php?title=SemEval_Portal~~. I would like to encourage people to exapnd on this grouping with a greater identification of issues. This might facilitate the development of future tasks.		(1) As currently envisaged, the identification of a task occurs in the mind of a single individual or small coordinated group. As a result, the task is somewhat random. This year's organizers have made the strong point about combining tasks, so as to have several subtasks that have some coherence. I would suggest that we all need some overall coherence about outstanding semantic evaluation issues, with the hope that the tasks and subtasks are homing in on those problems. I made a start on this with a quick grouping of task types at [http://www.thai-sbobet.com sbo]. I would like to encourage people to exapnd on this grouping with a greater identification of issues. This might facilitate the development of future tasks.
	(2) With the given structure, a task is defined and then somewhat cast in concrete. As both a participant and an organizer, this results in a major difficulty that the task can't evolve and play out. As an organizer, I've found it necessary to have some back and forth before the final form of a task is crystallized. As a participant, it takes some time to play around with trial data and to wrap oneself around the task. The FrameNet linking task in SemEval-2010 is a case in point. It was sufficiently complex that many of those with an initial interest ended up not participating and the original task was subdivided into a couple of smaller tasks. So, what I'm suggesting is that potential organizers throw out an initial, rough conceptualization with some trial data and that the task not be locked down without a full airing of potential issues and refinements. I don't think it's necessary to have a lot of time between the availability of the training data and submission of results (5 months). Once you're pretty sure that you can get the kind of results you want, it's just a matter of making the final run.		(2) With the given structure, a task is defined and then somewhat cast in concrete. As both a participant and an organizer, this results in a major difficulty that the task can't evolve and play out. As an organizer, I've found it necessary to have some back and forth before the final form of a task is crystallized. As a participant, it takes some time to play around with trial data and to wrap oneself around the task. The FrameNet linking task in SemEval-2010 is a case in point. It was sufficiently complex that many of those with an initial interest ended up not participating and the original task was subdivided into a couple of smaller tasks. So, what I'm suggesting is that potential organizers throw out an initial, rough conceptualization with some trial data and that the task not be locked down without a full airing of potential issues and refinements. I don't think it's necessary to have a lot of time between the availability of the training data and submission of results (5 months). Once you're pretty sure that you can get the kind of results you want, it's just a matter of making the final run.

Egiesbre at 15:57, 3 November 2010

2010-11-03T15:57:21Z

← Older revision		Revision as of 15:57, 3 November 2010
Line 18:		Line 18:
	''Increment the numbers below according to your preference.''		''Increment the numbers below according to your preference.''

	'''For 2012:''' 2		'''For 2012:''' 3

	'''For 2013:''' 9		'''For 2013:''' 9

Kenski at 20:01, 1 November 2010

2010-11-01T20:01:34Z

@@ Line 87: / Line 87: @@
 SAPIENZA Universita' di Roma
 http://www.dsi.uniroma1.it/~navigli
 [[Category:SemEval Portal]]

Kenski at 19:30, 1 November 2010

2010-11-01T19:30:47Z

← Older revision		Revision as of 19:30, 1 November 2010
Line 20:		Line 20:
	'''For 2012:''' 2		'''For 2012:''' 2

	'''For 2013:''' 8		'''For 2013:''' 9

	: '''For 2013 NAACL:''' 0		: '''For 2013 NAACL:''' 0

Navigli: /* Reasons for 2013 */

2010-10-30T16:00:27Z

Reasons for 2013

← Older revision		Revision as of 16:00, 30 October 2010
Line 83:		Line 83:
	As a co-organiser of two Semeval-2007 tasks I can say that you need to a) carefully think about what to do next (you want your task to allow for something new), b) choose the dataset(s), c) build an annotation interface, d) annotate, e) think about how to evaluate and prepare an evaluation software. Especially point (a) needs time, so devising a new task that it not derivative takes time and thinking. Also, I agree that analyzing the results of previous tasks and meeting to talk about that is an important step before developing new tasks and ideas. So, if it wasn't clear before :-), I am all for Semeval 2013!		As a co-organiser of two Semeval-2007 tasks I can say that you need to a) carefully think about what to do next (you want your task to allow for something new), b) choose the dataset(s), c) build an annotation interface, d) annotate, e) think about how to evaluate and prepare an evaluation software. Especially point (a) needs time, so devising a new task that it not derivative takes time and thinking. Also, I agree that analyzing the results of previous tasks and meeting to talk about that is an important step before developing new tasks and ideas. So, if it wasn't clear before :-), I am all for Semeval 2013!

			---
	Roberto Navigli		Roberto Navigli
	SAPIENZA Universita' di Roma		SAPIENZA Universita' di Roma

Navigli: /* Reasons for 2013 */

2010-10-30T15:59:57Z

Reasons for 2013

← Older revision		Revision as of 15:59, 30 October 2010
Line 81:		Line 81:
	---------------------------------------------------------		---------------------------------------------------------

			As a co-organiser of two Semeval-2007 tasks I can say that you need to a) carefully think about what to do next (you want your task to allow for something new), b) choose the dataset(s), c) build an annotation interface, d) annotate, e) think about how to evaluate and prepare an evaluation software. Especially point (a) needs time, so devising a new task that it not derivative takes time and thinking. Also, I agree that analyzing the results of previous tasks and meeting to talk about that is an important step before developing new tasks and ideas. So, if it wasn't clear before :-), I am all for Semeval 2013!

			Roberto Navigli
			SAPIENZA Universita' di Roma
			http://www.dsi.uniroma1.it/~navigli

	[[Category:SemEval Portal]]		[[Category:SemEval Portal]]

Pdturney at 14:24, 30 October 2010

2010-10-30T14:24:02Z

← Older revision		Revision as of 14:24, 30 October 2010
Line 5:		Line 5:
	* 2013 - NAACL		* 2013 - NAACL
	* 2013 - ACL		* 2013 - ACL

	~~The votes for ''2013 NAACL'' and ''2013 ACL'' will be added up. Based on the poll either the 2012 or the 2013 option will be selected.~~

	Related links:		Related links:

Pdturney: /* Votes */

2010-10-30T14:23:09Z

Votes

← Older revision		Revision as of 14:23, 30 October 2010
Line 18:		Line 18:
	'''Deadline for votes:''' November 22, 2010		'''Deadline for votes:''' November 22, 2010

	Increment the numbers below according to your preference.		''Increment the numbers below according to your preference.''

	'''For 2012:''' 2		'''For 2012:''' 2

Pdturney: /* Votes */

2010-10-30T14:22:22Z

Votes

← Older revision		Revision as of 14:22, 30 October 2010
Line 15:		Line 15:

	== Votes ==		== Votes ==

			'''Deadline for votes:''' November 22, 2010

	Increment the numbers below according to your preference.		Increment the numbers below according to your preference.