SemEval 2012 versus 2013
So far, SemEval has been organised in a 3 year cycle. However, many participants feel that this is a long wait. Many other shared tasks such as CoNLL and RTE run annually. For this reason we are giving the opportunity for task organisers to choose between a 2 year or a 3 year cycle. Task proposers will be asked to vote for the date of SemEval-3 and choose amongst:
- 2012 - ACL
- 2013 - NAACL
- 2013 - ACL
The votes for [2013 NAACL] and [2013 ACL] will be added up. Based on the poll either the 2012 or the 2013 option will be selected.
Votes
Increment the numbers below according to your preference.
For 2012: 1
For 2013: 5
- For 2013 NAACL:
- For 2013 ACL:
Reasons for 2012
We still have almost 2 years! If its after 3 years, we probably won't be doing anything in the first year anyway. Hence, I vote for 2012.
--
Naushad UzZaman, University of Rochester (TempEval-2 participant and TempEval-3 co-organizer)
Reasons for 2013
I understand that some participants are eager to run in yet another "competition" sooner rather than later. This is no reason to believe that squeezing the cycle into two years serves a useful purpose. My perspective is that of an organizer. A new task requires much thought and even more legwork. An old task merely repeated is not worth the bytes its data sit in. There must be new elements. To reuse the old data is easier said than done. I could share our experience with tasks 4 (2007) and 8 (2010). There was a markedly higher effort in 2009-2010 than any of us had initially thought. If the community goes with the idea of a common annotation style, well, that alone requires a deeper reflection.
I could go on, but you may already be bored. Let me just make a social observation. What we do is not a spat, a fisticuff or a race. It is a shared evaluation exercise. That many people treat is as a fight is painfully obvious. i suspect that it is not uncommon for someone to use the scores -- especially a showing close to the top -- as an argument in grant applications or requests for promotion. A stimulating intellectual challenge turns into (excusez le mot) a pissing contest. Naturally, it is better to have more chances to win that medal.
I propose to keep the usual pace. A three-year cycle has worked well. It allows organizers to do their work carefully and thoughtfully, without overstraining themselves. Those who run annual events probably survive only because innovation is very incremental.
-- Stan Szpakowicz, PhD, Professor SITE, Computer Science, University of Ottawa
As a task organiser from 2007 (task 10) and 2010 (task 2) I concur with Stan's sentiments. It does all depend on the thought that is required for the new approach/annotation and how much time the organisers have to spare. Another argument for a longer cycle is that it gives some time for analysis of the previous data before implementing the new ideas. I do agree with Suresh and Deniz (the current SemEval co-chairs) that the decision should rest with those who are willing to organise the tasks.
-- Diana McCarthy, (co-) Director Lexical Computing Ltd., Brighton UK
As of the date of this posting (Oct 28, 2010), I would note that about 6? months has already passed since the results were submitted in the last SemEval. I think there is some argument for allowing ourselves a bit of breathing room, and also time to reflect on what happened last time before moving on to the next round.
I particularly like the idea of having a workshop of some kind about one year after the conclusion of a SemEval to allow for more detailed discussion of what happened last time around, and might help to avoid too much repetition in tasks and also provide a nice opportunity for a more in depth discussion of lessons learned. Given that, I tend to prefer a 3 year window (particularly if this enables a lessons learned style workshop after 1 year). There was, for example, an ACL 2002 workshop after Senseval-2 (2001) that focused on "Recent Successes and Future Directions" and featured some papers that did a bit more analysis of results, etc. than is normally possible right after the event. I couldn't find the call for this event, but you can see the proceedings here :
http://aclweb.org/anthology-new/W/W02/#0800
I'd also be concerned that a 2 year window this time around might not be sustainable, and so we'd end up fluctuating a bit on the interval as the years go by. I think having it generally understood that SemEval will happen every 3 years is nice in that folks can generally plan on it. This can be helpful when working with or planning to work with students or others on a fixed time interval.
Ted Pedersen
http://www.d.umn.edu/~tpederse
task participant 2001, 2004, 2007, 2010 and task organizer 2004