Difference between revisions of "WordNet - RTE Users"
Jump to navigation
Jump to search
| Line 8: | Line 8: | ||
! nowrap="nowrap"|Specific usage description | ! nowrap="nowrap"|Specific usage description | ||
! nowrap="nowrap"|Evaluations / Comments | ! nowrap="nowrap"|Evaluations / Comments | ||
| − | + | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| AUEB | | AUEB | ||
| − | | | + | | RTE5 |
| − | | | + | | |
| − | | | + | | During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical. |
| − | + | | Ablation test performed. Negative impact. Relative accuracy: -0.02 on two-way, -0.0267 on three-way task. | |
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| Boeing | | Boeing | ||
| Line 25: | Line 21: | ||
| 2.0 | | 2.0 | ||
| Semantic relation between words | | Semantic relation between words | ||
| − | | No formal evaluation. Plays a role in most entailments found | + | | No formal evaluation. Plays a role in most entailments found |
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| DFKI | | DFKI | ||
| Line 32: | Line 29: | ||
| Semantic relation between words | | Semantic relation between words | ||
| No separate evaluation | | No separate evaluation | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| CERES | | CERES | ||
| Line 38: | Line 36: | ||
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv) | | Hypernyms, antonyms, indexWords (N,V,Adj,Adv) | ||
| Used, but no evaluation performed | | Used, but no evaluation performed | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| Cambridge | | Cambridge | ||
| Line 44: | Line 43: | ||
| Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x) | | Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x) | ||
| No systematic evaluation | | No systematic evaluation | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| BIU | | BIU | ||
| Line 50: | Line 50: | ||
| Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource | | Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource | ||
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]] | | 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]] | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| FbkIrst | | FbkIrst | ||
| Line 56: | Line 57: | ||
| Lexical similarity | | Lexical similarity | ||
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet | | No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet | ||
| − | + | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| AUEB | | AUEB | ||
| Line 68: | Line 64: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| DLSIUAES | | DLSIUAES | ||
| Line 74: | Line 71: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| EMORY | | EMORY | ||
| Line 80: | Line 78: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| FSC | | FSC | ||
| Line 86: | Line 85: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| IIT | | IIT | ||
| Line 92: | Line 92: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| IPD | | IPD | ||
| Line 98: | Line 99: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| OAQA | | OAQA | ||
| Line 104: | Line 106: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| QUANTA | | QUANTA | ||
| Line 110: | Line 113: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| SAGAN | | SAGAN | ||
| Line 116: | Line 120: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| Stanford | | Stanford | ||
| Line 122: | Line 127: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| UAIC | | UAIC | ||
| Line 128: | Line 134: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| UMD | | UMD | ||
| Line 134: | Line 141: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| UNED | | UNED | ||
| Line 140: | Line 148: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| Uoeltg | | Uoeltg | ||
| Line 146: | Line 155: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| UPC | | UPC | ||
| Line 152: | Line 162: | ||
| | | | ||
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | | ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.'' | ||
| + | |||
| + | |- bgcolor="#ECECEC" align="left" | ||
| + | | AUEB | ||
| + | | RTE3 | ||
| + | | 2.1 | ||
| + | | Synonymy resolution | ||
| + | | Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement | ||
| + | |||
| + | |- bgcolor="#ECECEC" align="left" | ||
| + | | VENSES | ||
| + | | RTE3 | ||
| + | | 3.0 | ||
| + | | Semantic relation between words | ||
| + | | No evaluation of the resource | ||
| + | |||
| + | |- bgcolor="#ECECEC" align="left" | ||
| + | | UIUC | ||
| + | | RTE3 | ||
| + | | | ||
| + | | Semantic distance between words | ||
| + | | | ||
| + | |||
|- bgcolor="#ECECEC" align="left" | |- bgcolor="#ECECEC" align="left" | ||
| ''New user'' | | ''New user'' | ||
Revision as of 06:57, 24 November 2009
When not otherwise specified, the data about version, usage and evaluation of the resource have been provided by participants themselves.
| Participants* | Campaign | Version | Specific usage description | Evaluations / Comments |
|---|---|---|---|---|
| AUEB | RTE5 | During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical. | Ablation test performed. Negative impact. Relative accuracy: -0.02 on two-way, -0.0267 on three-way task. | |
| Boeing | RTE4 | 2.0 | Semantic relation between words | No formal evaluation. Plays a role in most entailments found |
| DFKI | RTE4 | 3.0 | Semantic relation between words | No separate evaluation |
| CERES | RTE4 | 3.0 | Hypernyms, antonyms, indexWords (N,V,Adj,Adv) | Used, but no evaluation performed |
| Cambridge | RTE4 | 3.0 | Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x) | No systematic evaluation |
| BIU | RTE4 | 3.0 | Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource | 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel lexical-syntactic rule base |
| FbkIrst | RTE4 | 3.0 | Lexical similarity | No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet |
| AUEB | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| DLSIUAES | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| EMORY | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| FSC | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| IIT | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| IPD | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| OAQA | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| QUANTA | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| SAGAN | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| Stanford | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| UAIC | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| UMD | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| UNED | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| Uoeltg | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| UPC | RTE4 | Data taken from the RTE4 proceedings. Participants are recommended to add further information. | ||
| AUEB | RTE3 | 2.1 | Synonymy resolution | Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement |
| VENSES | RTE3 | 3.0 | Semantic relation between words | No evaluation of the resource |
| UIUC | RTE3 | Semantic distance between words | ||
| New user | Participants are encouraged to contribute. |
| Total: 24 |
|---|
[*] For further information about participants, click here: RTE Challenges - Data about participants
Return to RTE Knowledge Resources