ACL Wiki - User contributions [en]

Word Sense Disambiguation (State of the art)

2010-11-30T16:40:41Z

Eneko.agirre: Undo revision 4409 by Alstrup (Talk)

== "Standard" measure: ==

== "Standard" datasets: ==

{{StateOfTheArtTable}}

| SystemName || How does it work? || Author and Article [http://www.bla.com] || Software? || 98% according to... || Any extra comments?
|-

|-

|}

[[Category:State of the art]]

SemEval Portal

2010-11-30T16:38:24Z

Eneko.agirre: /* Tasks in Semantic Evaluation */

__FORCETOC__
This page serves as a community portal for everything related to Semantic Evaluation ('''SemEval''').
==Semantic Evaluation Exercises==

'''SemEval''' (Semantic Evaluation) is an ongoing series of evaluations of [[Semantics|computational semantic analysis]] systems; it evolved from the Senseval [[Word sense disambiguation|Word sense]] evaluation series. The evaluations are intended to explore the nature of [[Semantics|meaning]] in language. While meaning is intuitive to humans, transferring those intuitions to computational analysis has proved elusive.

This series of evaluations is providing a mechanism to characterize in more precise terms exactly what is necessary to compute in meaning. As such, the evaluations provide an emergent mechanism to identify the problems and solutions for computations with meaning. These exercises have evolved to articulate more of the dimensions that are involved in our use of language. They began with apparently simple attempts to identify [[Word sense disambiguation|word senses]] computationally. They have evolved to investigate the interrelationships among the elements in a sentence (e.g., [[semantic role labeling]]), relations between sentences (e.g., [[coreference]]), and the nature of what we are saying ([[semantic relations]] and [[sentiment analysis]]).

The purpose of the SemEval exercises and [http://www.senseval.org SENSEVAL] is to evaluate semantic analysis systems. The first three evaluations, Senseval-1 through Senseval-3, were focused on word sense disambiguation, each time growing in the number of languages offered in the tasks and in the number of participating teams. Beginning with the 4th workshop, SemEval-2007 (SemEval-1), the nature of the tasks evolved to include semantic analysis tasks outside of word sense disambiguation. This portal will be used to provide a comprehensive view of the issues involved in semantic evaluations.

==Upcoming and Past Events==

{| border="1" cellpadding="7" cellspacing="0"
|-
! Event
! Year
! Location
! Notes
|-
| [[SemEval 3]]
| align="center" | to be determined
| to be determined
| - discussion at [http://groups.google.com/group/semeval3 SemEval 3 Group]
|-
| [http://semeval2.fbk.eu/semeval2.php SemEval 2]
| align="center" | 2010
| Uppsala, Sweden
| - [http://aclweb.org/anthology-new/S/S10/ proceedings]
|-
| [http://nlp.cs.swarthmore.edu/semeval/index.php SemEval 1]
| align="center" | 2007
| Prague, Czech Republic
| - [http://aclweb.org/anthology-new/S/S07/ proceedings] - copy of website at [http://web.archive.org/web/20080727062358/http://nlp.cs.swarthmore.edu/semeval/index.php Internet Archive]
|-
| [http://www.senseval.org/senseval3 SENSEVAL 3]
| align="center" | 2004
| Barcelona, Spain
| - [http://aclweb.org/anthology-new/W/W04/#0800 proceedings]
|-
| [http://www.sle.sharp.co.uk/senseval2 SENSEVAL 2]
| align="center" | 2001
| Toulouse, France
| - main link provides links to results, data, system descriptions, task descriptions, and workshop program - copy of website at [http://web.archive.org/web/20050507011044/http://www.sle.sharp.co.uk/senseval2/ Internet Archive]
|-
| [http://www.itri.brighton.ac.uk/events/senseval/ARCHIVE/index.html SENSEVAL 1]
| align="center" | 1998
| East Sussex, UK
| - papers in [http://www.springerlink.com/content/0010-4817/34/1-2/ Computers and the Humanities], subscribers or pay per view
|-
|}

==Overview of Issues in Semantic Analysis==

The SemEval exercises provide a mechanism for examining issues in semantic analysis of texts. The topics of interest fall short of the logical rigor that is found in formal computational semsntics, attempting to identify and characterize the kinds of issues relevant to human understanding of language. The primary goal is to replicate human processing by means of computer systems. The tasks (shown below) are developed by individuals and groups to deal with identifiable issues, as they take on some concrete form.

The first major area in semantic analysis is the identification of the intended meaning at the word level (taken to include idiomatic expressions). This is word-sense disambiguation (a concept that is evolving away from the notion that words have discrete senses, but rather are characterized by the ways in which they are used, i.e., their contexts). The tasks in this area include lexical sample and all-word disambiguation, multi- and cross-lingual disambiguation, and lexical substitution. Given the difficulties of identifying word senses, other tasks relevant to this topic include word-sense induction, subcategorization acquisition, and evaluation of lexical resources.

The second major area in semantic analysis is the understanding of how different sentence and textual elements fit together. Tasks in this area include semantic role labeling, semantic relation analysis, and coreference resolution. Other tasks in this area look at more specialized issues of semantic analysis, such as temporal information processing, metonymy resolution, and sentiment analysis. The tasks in this area have many potential applications, such as information extraction, question answering, document summarization, machine translation, construction of thesauri and semantic networks, language modeling, paraphrasing,
and recognizing textual entailment. In each of these potential applications, the contribution of the types of semantic analysis constitutes the most outstanding research issue.

==Tasks in Semantic Evaluation==

The major tasks in semantic evaluation include:
* '''[[Word sense disambiguation]]''': WSD, lexical sample and all-words, the process of identifying which [[Word sense disambiguation|sense]] of a word (i.e. [[Semantics|meaning]]) is used in a [[Sentence (linguistics)|sentence]], when the word has multiple meanings ([[polysemy]]). The WSD task has two variants: "[[lexical sample task|lexical sample]]" and "[[all-words task|all words]]" task. The former comprises disambiguating the occurrences of a small sample of target words which were previously selected, while in the latter all the words in a piece of running text need to be disambiguated. Tasks have been performed for many languages. Tasks have covered disambiguation of nouns, verbs, adjectives, and prepositions.
* '''Multi-lingual or cross-lingual word-sense disambiguation''': word senses are defined according to translation distinctions, e.g., a polysemous word in Japanese is translated differently in a given context. The WSD task provides texts with target words and requires identification of the appropriate translation. A related task is cross-language information retrieval, where participants disambiguate in one language (e.g., with WordNet synsets) and retrieve documents in another language; standard information retrieval metrics are use to assess the quality of the disambiguation.
* '''Word-sense induction''': comparison of sense-induction and discrimination systems. The task is to cluster corpus instances (word uses, rather than word senses) and to evaluate systems on how well they correspond to pre-existing sense inventories or to various sense mapping systems.
* '''Lexical substitution''': find an alternative substitute word or phrase for a target word in context. The task involves both finding the synonyms and disambiguating the context. It allows the use of any kind of lexical resource or technique, including word sense disambiguation and word sense induction. A cross-lingual task was also defined.
* '''Evaluation of lexical resources''': the task evaluates the submitted lexical resources indirectly, running a simple WSD based on topic signatures (sets of words related to each target sense). A lexical sample tagged with English WordNet senses was used for evaluation.
* '''Subcategorization acquistion''': semantically similar verbs are similar in terms of subcategorization frames. The task is to use any available method for disambiguating verb senses, so that the results can then be fed into automatic methods used for acquiring subcategorization frames, with the hypothesis that the disambiguation will cluster the instances.
* '''Semantic role labeling''': identifying and labeling constituents of sentences with their semantic roles. The basic task began with attempts to replicate FrameNet data, specifically frame elements. This task has expanded to inferring and developing new frames and frame elements, in individual sentences and in full running texts, with identification of intersentential links and coreference chains.
* '''Semantic relation identification''': examining relations between lexical items in a sentence. The task, given a sample of semantic relation types, is to identify and classify semantic relations between nominals(i.e., nouns and base noun phrases, excluding named entities); a main purpose of this task is to assess different classification methods. Another task is, given a sentence and two tagged nominals, to predict the relation between those nominals and the direction of the relation.
* '''Metonymy resolution''': the figurative substitution of an attribute of a name for the thing specified. The task is a lexical sample task (1) to classify preselected expressions of a particular semantic class (such as country names) as having a literal or a metonymic reading, and if so, (2) to identify a further specification into prespecified metonymic patterns (such as place-for-event or company-for-stock) or, alternatively, recognition as an innovative reading. A second task is to identify when the arguments of a specified predicate does not satisfy selectional restrictions, and if not, to identify both the type mismatch and the type shift (coercion).
* '''Temporal information processing''': the temporal location and order of events in newspaper articles, narratives, and similar texts. The task is to identify the events described in a text and locate these in time, i.e., identification of temporal referring expressions, events and temporal relations within a text. A further task requires systems to recognize which of a fixed set of temporal relations holds between (a) events and time expressions within the same sentence (b) events and the document creation time (c) main events in consecutive sentences, and (d) two events where one syntactically dominates the other.
* '''Coreference resolution''': detection and resolution of coreferences. The task is to detect full coreference chains, composed by named entities, pronouns, and full noun phrases and to resolve pronouns, i.e., finding their antecedents.
* '''Sentiment analysis''': emotion annotation, polarity orientation labeling. The task is to classify the titles of newspaper articles with the appropriate emotion label and/or with a valence indication (positive/negative), given a set of predefined six emotion labels (i.e., Anger, Disgust, Fear, Joy, Sadness, Surprise).
This list is expected to grow as the field progresses.

Some tasks are closely related to each other. For instance, word sense disambiguation (monolingual, multi-lingual and cross-lingual), word sense induction task, lexical substitution, subcategorization acquisition and evaluation of lexical resources are all related to word senses.

==Organization==

[http://www.clres.com/siglex.html SIGLEX, the ACL Special Interest Group on the Lexicon] is the umbrella organization for the SemEval semantic evaluations and the SENSEVAL word-sense evaluation exercises. [http://www.senseval.org/ SENSEVAL] is the home page for SENSEVAL 1-3. Each exercise is usually organized by two individuals, who make the call for tasks and handle the overall administration. Within the general guidelines, each task is then organized and run by individuals or groups.

==SemEval on Wikipedia==

On [http://en.wikipedia.org/wiki/Main_Page Wikipedia], a [http://en.wikipedia.org/wiki/SemEval SemEval] page had been created and it is calling for contributions and suggestions on how to improve the Wikipedia page and to further the understanding of computational semantics.

==See also==

* [[Semantics]]
* [[Computational Semantics]]
* [[Statistical Semantics]]
* [[Semantics software for English]]

[[Category:SemEval Portal]]

SemEval Portal

2010-11-30T16:37:30Z

Eneko.agirre: /* Tasks in Semantic Evaluation */

__FORCETOC__
This page serves as a community portal for everything related to Semantic Evaluation ('''SemEval''').
==Semantic Evaluation Exercises==

'''SemEval''' (Semantic Evaluation) is an ongoing series of evaluations of [[Semantics|computational semantic analysis]] systems; it evolved from the Senseval [[Word sense disambiguation|Word sense]] evaluation series. The evaluations are intended to explore the nature of [[Semantics|meaning]] in language. While meaning is intuitive to humans, transferring those intuitions to computational analysis has proved elusive.

This series of evaluations is providing a mechanism to characterize in more precise terms exactly what is necessary to compute in meaning. As such, the evaluations provide an emergent mechanism to identify the problems and solutions for computations with meaning. These exercises have evolved to articulate more of the dimensions that are involved in our use of language. They began with apparently simple attempts to identify [[Word sense disambiguation|word senses]] computationally. They have evolved to investigate the interrelationships among the elements in a sentence (e.g., [[semantic role labeling]]), relations between sentences (e.g., [[coreference]]), and the nature of what we are saying ([[semantic relations]] and [[sentiment analysis]]).

The purpose of the SemEval exercises and [http://www.senseval.org SENSEVAL] is to evaluate semantic analysis systems. The first three evaluations, Senseval-1 through Senseval-3, were focused on word sense disambiguation, each time growing in the number of languages offered in the tasks and in the number of participating teams. Beginning with the 4th workshop, SemEval-2007 (SemEval-1), the nature of the tasks evolved to include semantic analysis tasks outside of word sense disambiguation. This portal will be used to provide a comprehensive view of the issues involved in semantic evaluations.

==Upcoming and Past Events==

{| border="1" cellpadding="7" cellspacing="0"
|-
! Event
! Year
! Location
! Notes
|-
| [[SemEval 3]]
| align="center" | to be determined
| to be determined
| - discussion at [http://groups.google.com/group/semeval3 SemEval 3 Group]
|-
| [http://semeval2.fbk.eu/semeval2.php SemEval 2]
| align="center" | 2010
| Uppsala, Sweden
| - [http://aclweb.org/anthology-new/S/S10/ proceedings]
|-
| [http://nlp.cs.swarthmore.edu/semeval/index.php SemEval 1]
| align="center" | 2007
| Prague, Czech Republic
| - [http://aclweb.org/anthology-new/S/S07/ proceedings] - copy of website at [http://web.archive.org/web/20080727062358/http://nlp.cs.swarthmore.edu/semeval/index.php Internet Archive]
|-
| [http://www.senseval.org/senseval3 SENSEVAL 3]
| align="center" | 2004
| Barcelona, Spain
| - [http://aclweb.org/anthology-new/W/W04/#0800 proceedings]
|-
| [http://www.sle.sharp.co.uk/senseval2 SENSEVAL 2]
| align="center" | 2001
| Toulouse, France
| - main link provides links to results, data, system descriptions, task descriptions, and workshop program - copy of website at [http://web.archive.org/web/20050507011044/http://www.sle.sharp.co.uk/senseval2/ Internet Archive]
|-
| [http://www.itri.brighton.ac.uk/events/senseval/ARCHIVE/index.html SENSEVAL 1]
| align="center" | 1998
| East Sussex, UK
| - papers in [http://www.springerlink.com/content/0010-4817/34/1-2/ Computers and the Humanities], subscribers or pay per view
|-
|}

==Overview of Issues in Semantic Analysis==

The SemEval exercises provide a mechanism for examining issues in semantic analysis of texts. The topics of interest fall short of the logical rigor that is found in formal computational semsntics, attempting to identify and characterize the kinds of issues relevant to human understanding of language. The primary goal is to replicate human processing by means of computer systems. The tasks (shown below) are developed by individuals and groups to deal with identifiable issues, as they take on some concrete form.

The first major area in semantic analysis is the identification of the intended meaning at the word level (taken to include idiomatic expressions). This is word-sense disambiguation (a concept that is evolving away from the notion that words have discrete senses, but rather are characterized by the ways in which they are used, i.e., their contexts). The tasks in this area include lexical sample and all-word disambiguation, multi- and cross-lingual disambiguation, and lexical substitution. Given the difficulties of identifying word senses, other tasks relevant to this topic include word-sense induction, subcategorization acquisition, and evaluation of lexical resources.

The second major area in semantic analysis is the understanding of how different sentence and textual elements fit together. Tasks in this area include semantic role labeling, semantic relation analysis, and coreference resolution. Other tasks in this area look at more specialized issues of semantic analysis, such as temporal information processing, metonymy resolution, and sentiment analysis. The tasks in this area have many potential applications, such as information extraction, question answering, document summarization, machine translation, construction of thesauri and semantic networks, language modeling, paraphrasing,
and recognizing textual entailment. In each of these potential applications, the contribution of the types of semantic analysis constitutes the most outstanding research issue.

==Tasks in Semantic Evaluation==

The major tasks in semantic evaluation include:
* '''[[Word sense disambiguation]]''': WSD, lexical sample and all-words, the process of identifying which [[Word sense disambiguation|sense]] of a word (i.e. [[Semantics|meaning]]) is used in a [[Sentence (linguistics)|sentence]], when the word has multiple meanings ([[polysemy]]). The WSD task has two variants: "[[lexical sample task|lexical sample]]" and "[[all-words task|all words]]" task. The former comprises disambiguating the occurrences of a small sample of target words which were previously selected, while in the latter all the words in a piece of running text need to be disambiguated. Tasks have been performed for many languages. Tasks have covered disambiguation of nouns, verbs, adjectives, and prepositions.
* '''Multi-lingual or cross-lingual word-sense disambiguation''': word senses are defined according to translation distinctions, e.g., a polysemous word in Japanese is translated differently in a given context. The WSD task provides texts with target words and requires identification of the appropriate translation. A related task is cross-language information retrieval, where participants disambiguate in one language (e.g., with WordNet synsets) and retrieve documents in another language; standard information retrieval metrics are use to assess the quality of the disambiguation.
* '''Word-sense induction''': comparison of sense-induction and discrimination systems. The task is to cluster corpus instances (word uses, rather than word senses) and to evaluate systems on how well they correspond to pre-existing sense inventories or to various sense mapping systems.
* '''Lexical substitution''': find an alternative substitute word or phrase for a target word in context. The task involves both finding the synonyms and disambiguating the context. It allows the use of any kind of lexical resource or technique, including word sense disambiguation and word sense induction. A cross-lingual task was also defined.
* '''Evaluation of lexical resources''': the task evaluates the submitted lexical resources indirectly, running a simple WSD based on topic signatures (sets of words related to each target sense). A lexical sample tagged with English WordNet senses was used for evaluation.
* '''Subcategorization acquistion''': semantically similar verbs are similar in terms of subcategorization frames. The task is to use any available method for disambiguating verb senses, so that the results can then be fed into automatic methods used for acquiring subcategorization frames, with the hypothesis that the disambiguation will cluster the instances.
* '''Semantic role labeling''': identifying and labeling constituents of sentences with their semantic roles. The basic task began with attempts to replicate FrameNet data, specifically frame elements. This task has expanded to inferring and developing new frames and frame elements, in individual sentences and in full running texts, with identification of intersentential links and coreference chains.
* '''Semantic relation identification''': examining relations between lexical items in a sentence. The task, given a sample of semantic relation types, is to identify and classify semantic relations between nominals(i.e., nouns and base noun phrases, excluding named entities); a main purpose of this task is to assess different classification methods. Another task is, given a sentence and two tagged nominals, to predict the relation between those nominals and the direction of the relation.
* '''Metonymy resolution''': the figurative substitution of an attribute of a name for the thing specified. The task is a lexical sample task (1) to classify preselected expressions of a particular semantic class (such as country names) as having a literal or a metonymic reading, and if so, (2) to identify a further specification into prespecified metonymic patterns (such as place-for-event or company-for-stock) or, alternatively, recognition as an innovative reading. A second task is to identify when the arguments of a specified predicate does not satisfy selectional restrictions, and if not, to identify both the type mismatch and the type shift (coercion).
* '''Temporal information processing''': the temporal location and order of events in newspaper articles, narratives, and similar texts. The task is to identify the events described in a text and locate these in time, i.e., identification of temporal referring expressions, events and temporal relations within a text. A further task requires systems to recognize which of a fixed set of temporal relations holds between (a) events and time expressions within the same sentence (b) events and the document creation time (c) main events in consecutive sentences, and (d) two events where one syntactically dominates the other.
* '''Coreference resolution''': detection and resolution of coreferences. The task is to detect full coreference chains, composed by named entities, pronouns, and full noun phrases and to resolve pronouns, i.e., finding their antecedents.
* '''Sentiment analysis''': emotion annotation, polarity orientation labeling. The task is to classify the titles of newspaper articles with the appropriate emotion label and/or with a valence indication (positive/negative), given a set of predefined six emotion labels (i.e., Anger, Disgust, Fear, Joy, Sadness, Surprise).
This list is expected to grow as the field progresses.

Some tasks are closely related to each other. For instance, word sense disambiguation (monolingual, multi-lingual and cross-lingual), word sense induction task, lexical substitution and evaluation of lexical resources are all related to word senses.

==Organization==

[http://www.clres.com/siglex.html SIGLEX, the ACL Special Interest Group on the Lexicon] is the umbrella organization for the SemEval semantic evaluations and the SENSEVAL word-sense evaluation exercises. [http://www.senseval.org/ SENSEVAL] is the home page for SENSEVAL 1-3. Each exercise is usually organized by two individuals, who make the call for tasks and handle the overall administration. Within the general guidelines, each task is then organized and run by individuals or groups.

==SemEval on Wikipedia==

On [http://en.wikipedia.org/wiki/Main_Page Wikipedia], a [http://en.wikipedia.org/wiki/SemEval SemEval] page had been created and it is calling for contributions and suggestions on how to improve the Wikipedia page and to further the understanding of computational semantics.

==See also==

* [[Semantics]]
* [[Computational Semantics]]
* [[Statistical Semantics]]
* [[Semantics software for English]]

[[Category:SemEval Portal]]

User:Eneko.agirre

2010-11-27T11:28:57Z

Eneko.agirre: New page: [http://ixa2.si.ehu.es/eneko Homepage]

[http://ixa2.si.ehu.es/eneko Homepage]

SemEval Portal

2010-11-27T11:21:09Z

Eneko.agirre: /* Tasks in Semantic Evaluation */

SemEval 2012 versus 2013

2010-10-28T18:03:45Z

Eneko.agirre:

__FORCETOC__
So far, SemEval has been organised in a 3 year cycle. However, many participants feel that this is a long wait. Many other shared tasks such as CoNLL and RTE run annually. For this reason we are giving the opportunity for task organisers to choose between a 2 year or a 3 year cycle. Task proposers will be asked to vote for the date of SemEval-3 and choose amongst:

* 2012 - ACL
* 2013 - NAACL
* 2013 - ACL

The votes for ''2013 NAACL'' and ''2013 ACL'' will be added up. Based on the poll either the 2012 or the 2013 option will be selected.

Related links:

* [[SemEval 3]]
* [[Draft Schedule for SemEval 3]]

== Votes ==

Increment the numbers below according to your preference.

'''For 2012:''' 1

'''For 2013:''' 6

:: '''For 2013 NAACL:'''
:: '''For 2013 ACL:'''

== Reasons for 2012 ==

We still have almost 2 years! If its after 3 years, we probably won't be doing anything in the first year anyway. Hence, I vote for 2012. 
--
Naushad UzZaman, University of Rochester (TempEval-2 participant and TempEval-3 co-organizer)

== Reasons for 2013 ==

I understand that some participants are eager to run in yet another "competition" sooner rather than later. This is no reason to believe that squeezing the cycle into two years serves a useful purpose. My perspective is that of an organizer. A new task requires much thought and even more legwork. An old task merely repeated is not worth the bytes its data sit in. There must be new elements. To reuse the old data is easier said than done. I could share our experience with tasks 4 (2007) and 8 (2010). There was a markedly higher effort in 2009-2010 than any of us had initially thought. If the community goes with the idea of a common annotation style, well, that alone requires a deeper reflection.

I could go on, but you may already be bored. Let me just make a social observation. What we do is not a spat, a fisticuff or a race. It is a shared evaluation exercise. That many people treat is as a fight is painfully obvious. i suspect that it is not uncommon for someone to use the scores -- especially a showing close to the top -- as an argument in grant applications or requests for promotion. A stimulating intellectual challenge turns into (excusez le mot) a pissing contest. Naturally, it is better to have more chances to win that medal.

I propose to keep the usual pace. A three-year cycle has worked well. It allows organizers to do their work carefully and thoughtfully, without overstraining themselves. Those who run annual events probably survive only because innovation is very incremental.

--
Stan Szpakowicz, PhD, Professor
SITE, Computer Science, University of Ottawa

As a task organiser from 2007 (task 10) and 2010 (task 2) I concur with Stan's sentiments. It does all depend on the thought that is required for the new approach/annotation and how much time the organisers have to spare. Another argument for a longer cycle is that it gives some time for analysis of the previous data before implementing the new ideas. I do agree with Suresh and Deniz (the current SemEval co-chairs) that the decision should rest with those who are willing to organise the tasks.

--
Diana McCarthy, (co-) Director
Lexical Computing Ltd., Brighton UK

---------------------------------------------------------

As of the date of this posting (Oct 28, 2010), I would note that about 6? months has already passed since the results were submitted in the last SemEval. I think there is some argument for allowing ourselves a bit of breathing room, and also time to reflect on what happened last time before moving on to the next round.

I particularly like the idea of having a workshop of some kind about one year after the conclusion of a SemEval to allow for more detailed discussion of what happened last time around, and might help to avoid too much repetition in tasks and also provide a nice opportunity for a more in depth discussion of lessons learned. Given that, I tend to prefer a 3 year window (particularly if this enables a lessons learned style workshop after 1 year). There was, for example, an ACL 2002 workshop after Senseval-2 (2001) that focused on "Recent Successes and Future Directions" and featured some papers that did a bit more analysis of results, etc. than is normally possible right after the event. I couldn't find the call for this event, but you can see the proceedings here :

http://aclweb.org/anthology-new/W/W02/#0800

I'd also be concerned that a 2 year window this time around might not be sustainable, and so we'd end up fluctuating a bit on the interval as the years go by. I think having it generally understood that SemEval will happen every 3 years is nice in that folks can generally plan on it. This can be helpful when working with or planning to work with students or others on a fixed time interval.

Ted Pedersen

http://www.d.umn.edu/~tpederse

task participant 2001, 2004, 2007, 2010 and task organizer 2004

------------------------------------------------------------

[[Category:SemEval Portal]]

Textual Entailment Resource Pool

2010-06-10T23:09:19Z

Eneko.agirre: /* Similarity / Relatedness Tools */

[[Textual Entailment|Textual entailment]] systems rely on many different types of [[Natural Language Processing|NLP]] resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system.

In response, the [[Recognizing Textual Entailment|Recognizing Textual Entailment (RTE)]] shared task community initiated a new activity for building this ''Textual Entailment Resource Pool''. RTE participants and any other member of the NLP community are encouraged to contribute to the pool.

In an effort to determine the relative impact of the resources, RTE participants are strongly encouraged to report, whenever possible, the contribution to the overall performance of each utilized resource. Formal qualitative and quantitative results should be included in a separate section of the system report as well as posted on the talk pages of this Textual Entailment Resource Pool.

'''Adding''' a new resource is very easy. See how to '''use existing templates''' to do this in [[Help:Using Templates]].

== Complete RTE Systems ==

* [http://project.cgm.unive.it/html/venses.html VENSES] (from Ca' Foscari University of Venice, Italy)
* [http://svn.ask.it.usyd.edu.au/trac/candc/wiki/nutcracker Nutcracker] (available for download)
* [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign)
* [http://edits.fbk.eu/ EDITS - Edit Distance Textual Entailment Suite] (open source software developed by [http://hlt.fbk.eu/ Human Language Technology (HLT) group at FBK-Irst])

== RTE data sets ==
* [http://www.coli.uni-saarland.de/projects/salsa/fate FrameNet manually annotated RTE 2006 Test Set.] Provided by [http://www.coli.uni-saarland.de/projects/salsa/ SALSA project, Saarland University.]
* [http://www.cs.biu.ac.il/~nlp/files/RTE_2006_Aligned.zip Manually Word Aligned RTE 2006 Data Sets.] Provided by [http://research.microsoft.com/nlp/ the Natural Language Processing Group, Microsoft Research.]
* [http://www-nlp.stanford.edu/projects/contradiction/ RTE data sets annotated for a 3-way decision: entails, contradicts, unknown.] Provided by Stanford NLP Group.
* [http://www.cs.utexas.edu/~pclark/bpi-test-suite/ BPI RTE data set] - 250 pairs, focusing on world knowledge. Provided jointly by [http://www.boeing.com/phantom/math_ct/index.html Boeing], [http://wordnet.cs.princeton.edu/ Princeton], and [http://www.isi.edu ISI].
* [http://hlt.fbk.eu/en/Technology/TE_Specialized_Data Textual Entailment Specialized Data Sets] - 90 RTE-5 Test Set pairs annotated with linguistic phenomena + 203 monothematic pairs (i.e. pairs where only one linguistic phenomenon is relevant to the entailment relation) created from the 90 annotated pairs. Provided jointly by [http://hlt.fbk.eu/en/home FBK-Irst], and [http://www.celct.it/ CELCT].
* [http://www.nist.gov/tac/data/ RTE-5 Search Pilot Data Set annotated with anaphora and coreference information] - RTE-5 Search Data Set annotated with anaphora/coreference information + Augmented RTE-5 Search Data Set, where all the referring expressions which need to be resolved in the entailing sentences are substituted by explicit expressions on the basis of the anaphora/coreference annotation. Provided by [http://www.celct.it/ CELCT] and distributed by [http://www.nist.gov/index.html NIST] at the [http://www.nist.gov/tac/data/ Past TAC Data] web page (2009 Search Pilot, annotated test/dev data).

== Knowledge Resources ==
[[RTE Knowledge Resources]]

== Tools ==

=== Parsers ===
* [http://svn.ask.it.usyd.edu.au/trac/candc C&C parser for Combinatory Categorial Grammar]
* [[Minipar]]
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SP Shallow Parser] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/shallow_parse_demo.php web demo] of this tool

=== Role Labelling ===
* [http://cemantix.org/assert ASSERT]
* [http://www.coli.uni-saarland.de/projects/salsa/shal/ Shalmaneser]
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SRL Semantic Role Labeler] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/srl-demo.php web demo] of this tool

=== Entity Recognition Tools ===
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=NE Illinois Named Entity Tagger] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_demo.php web demo] of this tool
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=CORANKER Illinois Multi-lingual Named Entity Discovery Tool] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_matcher_demo.php web demo] of this tool

=== Similarity / Relatedness Tools ===
* [http://ixa2.si.ehu.es/ukb UKB]: Open source WordNet-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words

=== Corpus Readers ===
* [http://nltk.org NLTK] provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the [http://nltk.org/doc/guides/corpus.html#rte Corpus Readers] Guide for more information.

=== Related Libraries ===

* [http://www.semantilog.org/pypes.html PyPES] general purpose library containing evaluation environment for RTE and McPIET text inference engine based on the ERG (English Resource Grammar)

== Links ==
* [http://homepages.inf.ed.ac.uk/jbos/rte/ Textual Entailment site by Johan Bos]
* [http://ai-nlp.info.uniroma2.it/te/ Textual Entailment at the University of Rome "Tor Vergata"]
[[Category:Textual Entailment Portal]]
* [http://l2r.cs.uiuc.edu/~cogcomp/entailment-module-demos.php Illinois Textual Entailment System Component demos]

Textual Entailment Resource Pool

2010-06-10T23:08:37Z

Eneko.agirre: /* Tools */

[[Textual Entailment|Textual entailment]] systems rely on many different types of [[Natural Language Processing|NLP]] resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system.

In response, the [[Recognizing Textual Entailment|Recognizing Textual Entailment (RTE)]] shared task community initiated a new activity for building this ''Textual Entailment Resource Pool''. RTE participants and any other member of the NLP community are encouraged to contribute to the pool.

In an effort to determine the relative impact of the resources, RTE participants are strongly encouraged to report, whenever possible, the contribution to the overall performance of each utilized resource. Formal qualitative and quantitative results should be included in a separate section of the system report as well as posted on the talk pages of this Textual Entailment Resource Pool.

'''Adding''' a new resource is very easy. See how to '''use existing templates''' to do this in [[Help:Using Templates]].

== Complete RTE Systems ==

* [http://project.cgm.unive.it/html/venses.html VENSES] (from Ca' Foscari University of Venice, Italy)
* [http://svn.ask.it.usyd.edu.au/trac/candc/wiki/nutcracker Nutcracker] (available for download)
* [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign)
* [http://edits.fbk.eu/ EDITS - Edit Distance Textual Entailment Suite] (open source software developed by [http://hlt.fbk.eu/ Human Language Technology (HLT) group at FBK-Irst])

== RTE data sets ==
* [http://www.coli.uni-saarland.de/projects/salsa/fate FrameNet manually annotated RTE 2006 Test Set.] Provided by [http://www.coli.uni-saarland.de/projects/salsa/ SALSA project, Saarland University.]
* [http://www.cs.biu.ac.il/~nlp/files/RTE_2006_Aligned.zip Manually Word Aligned RTE 2006 Data Sets.] Provided by [http://research.microsoft.com/nlp/ the Natural Language Processing Group, Microsoft Research.]
* [http://www-nlp.stanford.edu/projects/contradiction/ RTE data sets annotated for a 3-way decision: entails, contradicts, unknown.] Provided by Stanford NLP Group.
* [http://www.cs.utexas.edu/~pclark/bpi-test-suite/ BPI RTE data set] - 250 pairs, focusing on world knowledge. Provided jointly by [http://www.boeing.com/phantom/math_ct/index.html Boeing], [http://wordnet.cs.princeton.edu/ Princeton], and [http://www.isi.edu ISI].
* [http://hlt.fbk.eu/en/Technology/TE_Specialized_Data Textual Entailment Specialized Data Sets] - 90 RTE-5 Test Set pairs annotated with linguistic phenomena + 203 monothematic pairs (i.e. pairs where only one linguistic phenomenon is relevant to the entailment relation) created from the 90 annotated pairs. Provided jointly by [http://hlt.fbk.eu/en/home FBK-Irst], and [http://www.celct.it/ CELCT].
* [http://www.nist.gov/tac/data/ RTE-5 Search Pilot Data Set annotated with anaphora and coreference information] - RTE-5 Search Data Set annotated with anaphora/coreference information + Augmented RTE-5 Search Data Set, where all the referring expressions which need to be resolved in the entailing sentences are substituted by explicit expressions on the basis of the anaphora/coreference annotation. Provided by [http://www.celct.it/ CELCT] and distributed by [http://www.nist.gov/index.html NIST] at the [http://www.nist.gov/tac/data/ Past TAC Data] web page (2009 Search Pilot, annotated test/dev data).

== Knowledge Resources ==
[[RTE Knowledge Resources]]

== Tools ==

=== Parsers ===
* [http://svn.ask.it.usyd.edu.au/trac/candc C&C parser for Combinatory Categorial Grammar]
* [[Minipar]]
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SP Shallow Parser] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/shallow_parse_demo.php web demo] of this tool

=== Role Labelling ===
* [http://cemantix.org/assert ASSERT]
* [http://www.coli.uni-saarland.de/projects/salsa/shal/ Shalmaneser]
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SRL Semantic Role Labeler] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/srl-demo.php web demo] of this tool

=== Entity Recognition Tools ===
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=NE Illinois Named Entity Tagger] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_demo.php web demo] of this tool
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=CORANKER Illinois Multi-lingual Named Entity Discovery Tool] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_matcher_demo.php web demo] of this tool

=== Similarity / Relatedness Tools ===
* [http://ixa2.si.ehu.es/ukb Open source WordNet-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words]

=== Corpus Readers ===
* [http://nltk.org NLTK] provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the [http://nltk.org/doc/guides/corpus.html#rte Corpus Readers] Guide for more information.

=== Related Libraries ===

* [http://www.semantilog.org/pypes.html PyPES] general purpose library containing evaluation environment for RTE and McPIET text inference engine based on the ERG (English Resource Grammar)

== Links ==
* [http://homepages.inf.ed.ac.uk/jbos/rte/ Textual Entailment site by Johan Bos]
* [http://ai-nlp.info.uniroma2.it/te/ Textual Entailment at the University of Rome "Tor Vergata"]
[[Category:Textual Entailment Portal]]
* [http://l2r.cs.uiuc.edu/~cogcomp/entailment-module-demos.php Illinois Textual Entailment System Component demos]

Corpora for English

2010-06-03T14:07:08Z

Eneko.agirre:

For languages other than English, see [[List of resources by language]].


*[ftp://ftp.cs.cornell.edu/pub/smart/time/ 1963 Time Magazine corpus]
*[http://www.elda.fr/catalogue/en/speech/S0115.html American English SpeechDat-Car]
*[http://americannationalcorpus.org/ American National Corpus (ANC)]
*[http://americannationalcorpus.org/FirstRelease/ AMERICAN NATIONAL CORPUS FIRST RELEASE]
*[http://compbio.uchsc.edu/ccp/corpora/index.shtml Biomedical corpora]
*[http://homepage.mac.com/bncweb/ BNCweb a web-based interface to the British National Corpus]
*[http://devoted.to/corpora Bookmarks for Corpus-based Linguists]
*[http://info.ox.ac.uk/bnc/ British National Corpus (from Oxford University)]
*[http://www.natcorp.ox.ac.uk/ British National Corpus (BNC)]
*[http://www.comp.lancs.ac.uk/computing/research/ucrel/bnc.html British National Corpus project page (from UCREL)]
*[http://clwww.essex.ac.uk/w3c/corpus_ling/content/corpora/list/private/brown/brown.html Brown Corpus]
*[http://boston.lti.cs.cmu.edu/Data/clueweb09/ ClueWeb]
*[http://www.collins.co.uk/books.aspx?group=154 Collins Wordbanks]
*[http://www.cs.cornell.edu/home/llee/data/convote.html Congressional floor-debate transcripts, with support/oppose labels]
*[http://www.athel.com/corpdes.html Corpus of Spoken Professional English]
*[http://www-rcf.usc.edu/~billmann/diversity/DDivers-site.htm Dialogue Diversity Corpus]
*[http://etext.lib.virginia.edu/ Electronic Text Center -- University of Virginia]
*[http://www.phon.ox.ac.uk/~esther/ivyweb/ English Intonation in the British Isles -The IViE Corpus]
*[http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/theo-11/www/naive-bayes/bow-0.8/stopwords.c English stop words (from SMART)]
*[http://www-personal.umich.edu/~jlawler/levin.html English Verb Classes And Alternations: A Preliminary Investigation (Index)]
*[http://usna.edu/LangStudy/BNC/ Exploring Words and Phrases from the British National Corpus]
*[http://ir.dcs.gla.ac.uk/test_collections/gov2-summary.htm GOV2 Corpus] - 426 gigabytes of text
*[http://www.gutenberg.org/wiki/Main_Page Gutenberg]
*[http://prize.hutter1.net/ Hutter Prize for Lossless Compression of Human Knowledge 100M sample of Wikipedia]
*[http://nora.hd.uib.no/icame.html ICAME]
*[http://www.cs.fit.edu/~mmahoney/compression/text.html Large Text Compression Benchmark's 1G sample of Wikipedia]
*[http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/theo-11/www/naive-bayes/bow-0.8/stopwords.c List of English stopwords]
*[http://www.cs.cornell.edu/People/pabo/movie-review-data/ Movie Review Data]
*[http://www.cs.pitt.edu/mpqa/ Multi-Perspective Question Answering (MPQA)]
*[http://mwe.stanford.edu/resources/ Multiword Expression Resources]
*[http://www.askoxford.com/oec/mainpage/?view=uk Oxford English Corpus]
*[http://pie.usna.edu/ Phrases in English]
*[http://homepages.feis.herts.ac.uk/~comrcml/Lyon-thesis.ps Restricted English Corpus from Dr. Caroline Lyon for PhD]
*[http://www.sketchengine.co.uk/ Sketch Engine]
*[http://www-2.cs.cmu.edu/afs/cs/project/ai-repository/ai/areas/nlp/corpora/susanne/0.html Susanne: Annotated American English Corpus]
*[http://clix.to/davidlee00 The BNC Index (for the BNCWorld Edition)]
*[http://www-users.york.ac.uk/~sp20/corpus.html The Brooklyn-Geneva-Amsterdam-Helsinki Parsed Corpus of Old English]
*[http://www-rcf.usc.edu/~billmann/diversity/DDivers-site.htm The Dialogue Diversity Corpus]
*[http://www.grsampson.net/LucyDoc.html The LUCY Corpus - Documentation]
*[http://www.cs.rochester.edu/research/cisd/resources/trains.html TRAINS Dialogue Corpus]
*[http://wacky.sslmit.unibo.it/ WaCky]
*[http://www.webcorp.org.uk/guide/ WebCorp]

==Link collections==


*[http://www.dcs.gla.ac.uk/idom/ir_resources/ Collections of texts and corpora]
*[http://www.bmanuel.org/clr2_mp.html Manuel Barbera: General Corpora and Corpus Linguistics Resources]
*[http://www.alphabit.net Isabella Chiari: Corpora, Software and Linguistic resources]
*[http://www.sultry.arts.usyd.edu.au/links/statnlp.html Annotated list of resources on statistical NLP and corpus-based CL]

==Corpora tools==


*[http://www.dcs.gla.ac.uk/idom/ir_resources/linguistic_utils/stop_words List of stop words]
*[http://korpus.pl/index.php?page=poliqarp Poliqarp] - open source XML-aware indexer, search engine and concordancer
*[http://www.sketchengine.co.uk/ The Sketch Engine]
*[http://www.cis.upenn.edu/~treebank/tokenization.html Treebank tokenization scheme]

[[Category:Corpora|*]]