Difference between revisions of "WordNet - RTE Users"

From ACL Wiki
Jump to navigation Jump to search
m
m
 
(56 intermediate revisions by one other user not shown)
Line 1: Line 1:
When not otherwise specified, the data about usage of the resource have been gathered from the RTE Knowledge Resources Questionnaire submitted to participants.
+
Given the high number of [[WordNet]] users, the table has been splitted into two: the first containing the users of the last campaign, the second containing the users of previous campaigns.
 +
__TOC__
 
<br>
 
<br>
<br>
+
==RTE5 Users==
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1" style="margin-left: 20px;"
+
 
 +
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
 +
 
 
|- bgcolor="#CDCDCD"
 
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants
+
! nowrap="nowrap"|Participants*
 
! nowrap="nowrap"|Campaign
 
! nowrap="nowrap"|Campaign
 
! nowrap="nowrap"|Version
 
! nowrap="nowrap"|Version
 
! nowrap="nowrap"|Specific usage description
 
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evalutations / Comments
+
! nowrap="nowrap"|Evaluations / Comments
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| AUEB
 +
| RTE5
 +
|
 +
| During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical.
 +
| Ablation test performed. Negative impact of the resource: -2% accuracy on two-way, -2.67% on three-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| BIU
 +
| RTE5
 +
| 3.0
 +
| Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations.
 +
| Ablation test performed. Positive impact of the resource: +2.5% accuracy on two-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| Boeing
 +
| RTE5
 +
|
 +
| The system makes uses of Wordnet synonyms, hypernyms relationships between (senses of) words, "similar" (SIM), "pertains" (PER), and "derivational" (DER) links to recognize equivalence between T and H.
 +
| Ablation test performed. Positive impact of the resource: +4% accuracy on two-way, +5.67% on three-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| DFKI
 +
| RTE5
 +
|
 +
| FIRST USE: Argument alignment between T and H.<br/>SECOND USE: used to change all the nominal predicates into verbs, to calculate relatedness between T and H (using VerbOcean).
 +
| FIRST USE: Ablation test performed. Impact of the resource: -0.17% accuracy/null respectively on two-way and three-way task for run1; +0.16%/+0.34% for run2; +0.17%/+0.17% for run3.<br/>
 +
SECOND USE (WordNet+VerbOcean): null/+0.17% accuracy respectively on two-way and three-way task for run1; +0.5%/+0.67% for run2; +0.17%/+0.17% for run3.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| DirRelCond
 +
| RTE5
 +
|
 +
| Use of many WordNet relations (such as synonymy, hypernymy, hyponymy, meronymy, holonymy etc.) to compute the relatedness between words with the same part of speech in T and H.
 +
| No ablation test performed. The resource cannot be removed without breaking the functionality of the system.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| DLSIUAES
 +
| RTE5
 +
|
 +
| FIRST USE: Similarity between lemmata, computed by WordNet-based metrics.<br/>
 +
SECOND USE: antonymy relations between verbs.<br/>
 +
THIRD USE: synonymy/identity between verb lemmata in T and H.<br/>
 +
FOURTH USE (WordNet+Framenet): WordNet synonym and hyponym relations from T's frame elements to H's frame elements.
 +
| FIRST USE: Ablation test performed. Positive impact of the resource on two-way run: +0.83% accuracy. Negative impact  on three-way run: -0.33% accuracy (-0.5% for two-way derived).<br/>
 +
SECOND USE (WordNet+VerbOcean+DLSIUAES_negation_list): positive impact on two-way run: +0.66% accuracy. Negative impact  on three-way run: -1% (-0.5% for two-way derived).<br/>
 +
THIRD USE: No ablation test performed.<br/>
 +
FOURTH USE (WordNet+Framenet): positive impact on two-way run: +1.16% accuracy. Negative impact  on three-way run: -0.17% (the same for two-way derived).
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| Del Monte
+
| FBKirst
| RTE3
+
| RTE5
 
| 3.0
 
| 3.0
| Semantic relation between words
+
| Extraction of a set of 2698 English entailment rules for terms connected by the hyponymy and synonymy relations
| No evaluation of the resource
+
| No ablation test performed
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| JU_CSE_TAC
 +
| RTE5
 +
|
 +
| WordNet based Unigram match: if any synset for the H unigram matches with any synset of a word in T then the hypothesis unigram is considered as a WordNet based unigram match.
 +
| Ablation test performed. Positive impact of the resource: +0.34% on two-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| PeMoZa
 +
| RTE5
 +
|
 +
| FIRST USE: Derivational Morphology.<br/>
 +
SECOND USE: Verb Entailment.<br/>
 +
| Ablation tests performed.<br/>
 +
FIRST USE. Impact of the resource on two-way task: -0.5%/+1% accuracy respectively on run1 and run2.<br/>
 +
SECOND USE. Impact of the resource on two-way task: +1.33%/-0.33% accuracy respectively on run1 and run2.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| QUANTA
 +
| RTE5
 +
|
 +
| Several relations from wordnet, such as synonyms, hyponym, hypernym et al.
 +
| Ablation test performed. Negative impact of the resource: -0.17% on two-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| Rhodes
 +
| RTE5
 +
|
 +
| FIRST USE: Lexicon based match: we chose a very simple metric: matching between words in T and H based on a path of distance at most 2 in the WordNet graph, using any links (hyponymy, hypernymy, meronymy, pertainymy, etc.)<br/>
 +
SECOND USE: contradiction detection based on antonymy relation
 +
| FIRST USE: Ablation test performed. Positive impact of the resource: +3.17% accuracy on two-way, +4% on three-way task.<br/>
 +
SECOND USE: no ablation test performed.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| Sagan
 +
| RTE5
 +
|
 +
| Used to obtain two features (string similarity based on Levenshtein distance and semantic similarity) in the training and testing steps of the system.
 +
| Ablation test performed. Null/negative (-0.87%) impact of the resource respectively on two-way and three-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| Siel_09
 +
| RTE5
 +
|
 +
| Similarity between nouns using WN tool
 +
| Ablation test performed. Impact of the resource: +0.34% accuracy on two-way, -0.17% on three-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| UAIC
 +
| RTE5
 +
|
 +
| FIRST USE: Antonymy relation to detect contradiction. In order to broaden the domain of the antonymy relation, we consider a combination of synonyms and antonyms. Used in combination with VerbOcean.<br/>
 +
SECOND USE: Synonymy, hyponymy and hypernymy for nouns and adjectives. Used in combination with eXtended WordNet relations.
 +
| FIRST USE: Ablation test performed (Wordnet + VerbOcean). Positive impact of the two resources together: +2% accuracy on two-way, +1.5% on three-way task.<br/>
 +
SECOND USE: Ablation test performed (Wordnet + eXtended WordNet). Positive impact of the two resources together: +1% accuracy on two-way, +1.33% on three-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| UB.dmirg
 +
| RTE5
 +
|
 +
| When using WordNet, we assume that a term is semantically interchangeable with its exact occurrence, its synonyms, and its hypernyms. In extracting hypernyms, we exclude the hypernyms that are more distant than two links to the original terms in WordNet synsets.
 +
| Two ablation tests performed. The first for Wordnet alone, the second for both WordNet and Framenet. Null impact of the resource(s) on two-way task for both ablations.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| UI_ccg
 +
| RTE5
 +
|
 +
| Word similarity == identity
 +
| Ablation test performed. Positive impact of the resource: +4% accuracy on two-way task.
 +
 
 +
|- bgcolor="#ECECEC" align="left"
 +
| Unimelb
 +
| RTE5
 +
|
 +
| Synonyms, derivationally-related forms.
 +
| Used in the search task.
 +
|}
 +
 
 +
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
 +
|-
 +
! align="left"|Subtotal: 17
 +
|}
 +
<br>
 +
 
 +
==Past Campaigns Users==
 +
 
 +
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
 +
 
 +
|- bgcolor="#CDCDCD"
 +
! nowrap="nowrap"|Participants*
 +
! nowrap="nowrap"|Campaign
 +
! nowrap="nowrap"|Version
 +
! nowrap="nowrap"|Specific usage description
 +
! nowrap="nowrap"|Evaluations / Comments
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| AUEB
 
| AUEB
| RTE3
 
| 2.1
 
| Synonymy resolution
 
| Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
 
|- bgcolor="#ECECEC" align="left"
 
| Boeing
 
 
| RTE4
 
| RTE4
| 2.0
+
|  
| Semantic relation between words
+
|  
| No formal evaluation. Plays a role in most entailments found.  
+
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| DFKI
+
| BIU
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Semantic relation between words
+
| Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations. Also used as part of our novel lexical-syntactic resource
| No separate evaluation
+
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]]
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| CERES
+
| Boeing
 
| RTE4
 
| RTE4
| 3.0
+
| 2.0
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv)
+
| Semantic relation between words
| Used, but no evaluation performed
+
| No formal evaluation. Plays a role in most entailments found
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| Cambridge
 
| Cambridge
Line 45: Line 191:
 
| Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x)
 
| Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x)
 
| No systematic evaluation
 
| No systematic evaluation
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| BIU
+
| CERES
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource
+
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv)
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel lexical syntactic rule base
+
| Used, but no evaluation performed
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| FbkIrst
+
| DFKI
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Lexical similarity
+
| Semantic relation between words
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical  similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
+
| No separate evaluation
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| AUEB
+
| DLSIUAES
 
| RTE4
 
| RTE4
 
|  
 
|  
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| DLSIUAES
+
| EMORY
 
| RTE4
 
| RTE4
 
|  
 
|  
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| EMORY
+
| FbkIrst
 
| RTE4
 
| RTE4
|  
+
| 3.0
|  
+
| Lexical similarity
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
+
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical  similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| FSC
 
| FSC
Line 81: Line 233:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| IIT
 
| IIT
Line 87: Line 240:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| IPD
 
| IPD
Line 93: Line 247:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| OAQA
 
| OAQA
Line 99: Line 254:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| QUANTA
 
| QUANTA
Line 105: Line 261:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| SAGAN
 
| SAGAN
Line 111: Line 268:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| Stanford
 
| Stanford
Line 117: Line 275:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| UAIC
 
| UAIC
Line 122: Line 281:
 
|  
 
|  
 
|  
 
|  
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
+
| Ablation test performed: +3% precision on two-way task.
 +
 
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| UMD
 
| UMD
Line 129: Line 289:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| UNED
 
| UNED
Line 135: Line 296:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| Uoeltg
 
| Uoeltg
Line 141: Line 303:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| UPC
 
| UPC
Line 147: Line 310:
 
|  
 
|  
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
 +
 +
|- bgcolor="#ECECEC" align="left"
 +
| AUEB
 +
| RTE3
 +
| 2.1
 +
| Synonymy resolution
 +
| Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
 +
 +
|- bgcolor="#ECECEC" align="left"
 +
| UAIC
 +
| RTE4
 +
|
 +
|
 +
| Ablation test performed: +1.13% precision on two-way task.
 +
 +
|- bgcolor="#ECECEC" align="left"
 +
| UIUC
 +
| RTE3
 +
|
 +
| Semantic distance between words
 +
|
 +
 +
|- bgcolor="#ECECEC" align="left"
 +
| VENSES
 +
| RTE3
 +
| 3.0
 +
| Semantic relation between words
 +
| No evaluation of the resource
 +
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
 
| ''New user''
 
| ''New user''
Line 156: Line 348:
 
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
 
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
 
|-
 
|-
! align="left"|Total: 23
+
! align="left"|Subtotal: 25
 
|}
 
|}
<br>
+
 
 +
 
 +
[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]
 +
 
 
     Return to [[RTE Knowledge Resources]]
 
     Return to [[RTE Knowledge Resources]]

Latest revision as of 04:42, 24 May 2013

Given the high number of WordNet users, the table has been splitted into two: the first containing the users of the last campaign, the second containing the users of previous campaigns.


RTE5 Users

Participants* Campaign Version Specific usage description Evaluations / Comments
AUEB RTE5 During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical. Ablation test performed. Negative impact of the resource: -2% accuracy on two-way, -2.67% on three-way task.
BIU RTE5 3.0 Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations. Ablation test performed. Positive impact of the resource: +2.5% accuracy on two-way task.
Boeing RTE5 The system makes uses of Wordnet synonyms, hypernyms relationships between (senses of) words, "similar" (SIM), "pertains" (PER), and "derivational" (DER) links to recognize equivalence between T and H. Ablation test performed. Positive impact of the resource: +4% accuracy on two-way, +5.67% on three-way task.
DFKI RTE5 FIRST USE: Argument alignment between T and H.
SECOND USE: used to change all the nominal predicates into verbs, to calculate relatedness between T and H (using VerbOcean).
FIRST USE: Ablation test performed. Impact of the resource: -0.17% accuracy/null respectively on two-way and three-way task for run1; +0.16%/+0.34% for run2; +0.17%/+0.17% for run3.

SECOND USE (WordNet+VerbOcean): null/+0.17% accuracy respectively on two-way and three-way task for run1; +0.5%/+0.67% for run2; +0.17%/+0.17% for run3.

DirRelCond RTE5 Use of many WordNet relations (such as synonymy, hypernymy, hyponymy, meronymy, holonymy etc.) to compute the relatedness between words with the same part of speech in T and H. No ablation test performed. The resource cannot be removed without breaking the functionality of the system.
DLSIUAES RTE5 FIRST USE: Similarity between lemmata, computed by WordNet-based metrics.

SECOND USE: antonymy relations between verbs.
THIRD USE: synonymy/identity between verb lemmata in T and H.
FOURTH USE (WordNet+Framenet): WordNet synonym and hyponym relations from T's frame elements to H's frame elements.

FIRST USE: Ablation test performed. Positive impact of the resource on two-way run: +0.83% accuracy. Negative impact on three-way run: -0.33% accuracy (-0.5% for two-way derived).

SECOND USE (WordNet+VerbOcean+DLSIUAES_negation_list): positive impact on two-way run: +0.66% accuracy. Negative impact on three-way run: -1% (-0.5% for two-way derived).
THIRD USE: No ablation test performed.
FOURTH USE (WordNet+Framenet): positive impact on two-way run: +1.16% accuracy. Negative impact on three-way run: -0.17% (the same for two-way derived).

FBKirst RTE5 3.0 Extraction of a set of 2698 English entailment rules for terms connected by the hyponymy and synonymy relations No ablation test performed
JU_CSE_TAC RTE5 WordNet based Unigram match: if any synset for the H unigram matches with any synset of a word in T then the hypothesis unigram is considered as a WordNet based unigram match. Ablation test performed. Positive impact of the resource: +0.34% on two-way task.
PeMoZa RTE5 FIRST USE: Derivational Morphology.

SECOND USE: Verb Entailment.

Ablation tests performed.

FIRST USE. Impact of the resource on two-way task: -0.5%/+1% accuracy respectively on run1 and run2.
SECOND USE. Impact of the resource on two-way task: +1.33%/-0.33% accuracy respectively on run1 and run2.

QUANTA RTE5 Several relations from wordnet, such as synonyms, hyponym, hypernym et al. Ablation test performed. Negative impact of the resource: -0.17% on two-way task.
Rhodes RTE5 FIRST USE: Lexicon based match: we chose a very simple metric: matching between words in T and H based on a path of distance at most 2 in the WordNet graph, using any links (hyponymy, hypernymy, meronymy, pertainymy, etc.)

SECOND USE: contradiction detection based on antonymy relation

FIRST USE: Ablation test performed. Positive impact of the resource: +3.17% accuracy on two-way, +4% on three-way task.

SECOND USE: no ablation test performed.

Sagan RTE5 Used to obtain two features (string similarity based on Levenshtein distance and semantic similarity) in the training and testing steps of the system. Ablation test performed. Null/negative (-0.87%) impact of the resource respectively on two-way and three-way task.
Siel_09 RTE5 Similarity between nouns using WN tool Ablation test performed. Impact of the resource: +0.34% accuracy on two-way, -0.17% on three-way task.
UAIC RTE5 FIRST USE: Antonymy relation to detect contradiction. In order to broaden the domain of the antonymy relation, we consider a combination of synonyms and antonyms. Used in combination with VerbOcean.

SECOND USE: Synonymy, hyponymy and hypernymy for nouns and adjectives. Used in combination with eXtended WordNet relations.

FIRST USE: Ablation test performed (Wordnet + VerbOcean). Positive impact of the two resources together: +2% accuracy on two-way, +1.5% on three-way task.

SECOND USE: Ablation test performed (Wordnet + eXtended WordNet). Positive impact of the two resources together: +1% accuracy on two-way, +1.33% on three-way task.

UB.dmirg RTE5 When using WordNet, we assume that a term is semantically interchangeable with its exact occurrence, its synonyms, and its hypernyms. In extracting hypernyms, we exclude the hypernyms that are more distant than two links to the original terms in WordNet synsets. Two ablation tests performed. The first for Wordnet alone, the second for both WordNet and Framenet. Null impact of the resource(s) on two-way task for both ablations.
UI_ccg RTE5 Word similarity == identity Ablation test performed. Positive impact of the resource: +4% accuracy on two-way task.
Unimelb RTE5 Synonyms, derivationally-related forms. Used in the search task.
Subtotal: 17


Past Campaigns Users

Participants* Campaign Version Specific usage description Evaluations / Comments
AUEB RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
BIU RTE4 3.0 Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations. Also used as part of our novel lexical-syntactic resource 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel lexical-syntactic rule base
Boeing RTE4 2.0 Semantic relation between words No formal evaluation. Plays a role in most entailments found
Cambridge RTE4 3.0 Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x) No systematic evaluation
CERES RTE4 3.0 Hypernyms, antonyms, indexWords (N,V,Adj,Adv) Used, but no evaluation performed
DFKI RTE4 3.0 Semantic relation between words No separate evaluation
DLSIUAES RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
EMORY RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
FbkIrst RTE4 3.0 Lexical similarity No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
FSC RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
IIT RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
IPD RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
OAQA RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
QUANTA RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
SAGAN RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
Stanford RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UAIC RTE4 Ablation test performed: +3% precision on two-way task.
UMD RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UNED RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
Uoeltg RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UPC RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
AUEB RTE3 2.1 Synonymy resolution Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
UAIC RTE4 Ablation test performed: +1.13% precision on two-way task.
UIUC RTE3 Semantic distance between words
VENSES RTE3 3.0 Semantic relation between words No evaluation of the resource
New user Participants are encouraged to contribute.
Subtotal: 25


[*] For further information about participants, click here: RTE Challenges - Data about participants

   Return to RTE Knowledge Resources