ACL Wiki - User contributions [en]

RTE5 - Ablation Tests

2010-07-23T09:53:16Z

Celct:

The following table lists the results of the ablation tests submitted by participants, which have been introduced as a mandatory track in the RTE5 campaign. 
The first column contains the specific resources which have been ablated. The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way). The third and fourth columns present the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system. The third refers to the score obtained in the 2-way task, the fourth to that obtained in the 3-way task. For all the runs submitted as 3-way, also the 2-way derived accuracy has been calculated. 
Finally, the fifth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the RTE5 proceedings. 

Participants are kindly invited to check if all the inserted information is correct and complete.

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! Ablated Resource
! Team Run<ref>For further information about participants, click here: [[RTE Challenges - Data about participants]]</ref>
! Resource impact - 2way
! Resource impact - 3way
! Resource Usage Description

|- bgcolor="#ECECEC" "align="left"
| Acronym guide
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| The acronyms are expanded using the acronym database, so the acronyms are also matched with the expanded acronyms, and entailment is predicted accordingly

|- bgcolor="#ECECEC" "align="left"
| Acronym guide + UAIC_Acronym_rules
| UAIC20091.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.16
| We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.

|- bgcolor="#ECECEC" "align="left"
| DIRT
| BIU1.2way
| style="text-align: center;"| 1.33
| style="text-align: center;"| —
| Inference rules

|- bgcolor="#ECECEC" "align="left"
| DIRT
| Boeing3.3way
| style="text-align: center;"| -1.17
| style="text-align: center;"| 0
| Verb paraphrases

|- bgcolor="#ECECEC" "align="left"
| DIRT
| UAIC20091.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.33
| We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H

|- bgcolor="#ECECEC" "align="left"
| Framenet+ WordNet
| DLSIUAES1.2way
| style="text-align: center;"| 1.16
| style="text-align: center;"| —
| Frame-to-frame similarity metric

|- bgcolor="#ECECEC" "align="left"
| Framenet+ WordNet
| DLSIUAES1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.17
| Frame-to-frame similarity metric

|- bgcolor="#ECECEC" "align="left"
| Framenet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| If two lexical items are covered in a single FrameNet frame, then the two items are treated as semantically related.

|- bgcolor="#ECECEC" "align="left"
| Grady Ward’s MOBY Thesaurus + Roget's Thesaurus
| Venses2.2way
| style="text-align: center;"| 2.83
| style="text-align: center;"| —
| Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas

|- bgcolor="#ECECEC" "align="left"
| MontyLingua Tool
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form

|- bgcolor="#ECECEC" "align="left"
| NEGATION_rules by UAIC
| UAIC20091.3way
| style="text-align: center;"| 0
| style="text-align: center;"| -1.34
| Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.

|- bgcolor="#ECECEC" "align="left"
| NER (RASP Parser nertag)
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Named Entity match: measure based on the number of Nes in the hypothesis that match in the corresponding text. For named entity recognition, the RASP Parser (Briscoe et al., 2006) nertag component has been used.

|- bgcolor="#ECECEC" "align="left"
| NE component
| UI_ccg1.2way
| style="text-align: center;"| 4.83
| style="text-align: center;"| —
| Named Entity recognition/comparison

|- bgcolor="#ECECEC" "align="left"
| PropBank
| cswhu1.3way
| style="text-align: center;"| 2
| style="text-align: center;"| 3.17
| syntactic and semantic parsing

|- bgcolor="#ECECEC" "align="left"
| Stanford NER
| QUANTA1.2way
| style="text-align: center;"| 0.67
| style="text-align: center;"| —
| We use Named Entity similarity as a feature

|- bgcolor="#ECECEC" "align="left"
| Stopword list
| FBKirst1.2way
| style="text-align: center;"| 1.5
| style="text-align: center;"| —
| A list of the 572 most frequent English words has been collected in order to prevent assigning high costs to the deletion/insertion of terms that are unlikely to bring relevant information to detect entailment,and to avoid substituting these terms with any content word.

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE1, 2, 3
| PeMoZa3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE2
| PeMoZa3.2way
| style="text-align: center;"| 0.66
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE2, 3
| PeMoZa3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.17
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI2.3way
| style="text-align: center;"| 0.33
| style="text-align: center;"| 0.5
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| FBKirst1.2way
| style="text-align: center;"| -0.16
| style="text-align: center;"| —
| Extraction of 18232 entailment rules for all the English verbs connected by the ”stronger-than” relation. For instance, if ”kill [stronger-than] injure”, then the rule ”kill ENTAILS injure” is added to the rules repository.

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| QUANTA1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| We use "opposite-of" relation in VerbOcean as a feature

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| Similarity/anthonymy/unrelatedness between verbs

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| BIU1.2way
| style="text-align: center;"| -1
| style="text-align: center;"| —
| Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| cswhu1.3way
| style="text-align: center;"| 1.33
| style="text-align: center;"| 3.34
| Lexical semantic rules

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| FBKirst1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Rules extracted from WP using Latent Semantic Analysis (LSA)

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| UAIC20091.3way
| style="text-align: center;"| 1.17
| style="text-align: center;"| 1.5
| Relations between named entities

|- bgcolor="#ECECEC" "align="left"
| Wikipedia + NER's (LingPipe, GATE) + Perl patterns
| UAIC20091.3way
| style="text-align: center;"| 6.17
| style="text-align: center;"| 5
| NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text

|- bgcolor="#ECECEC" "align="left"
| WordNet
| AUEBNLP1.3way
| style="text-align: center;"| -2
| style="text-align: center;"| -2.67
| Synonyms

|- bgcolor="#ECECEC" "align="left"
| WordNet
| BIU1.2way
| style="text-align: center;"| 2.5
| style="text-align: center;"| —
| Synonyms, hyponyms (2 levels away from the original term), hyponym_instance and derivations

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Boeing3.3way
| style="text-align: center;"| 4
| style="text-align: center;"| 5.67
| Wordnet synonyms, hypernyms relationships between (senses of) words, "similar" (SIM), "pertains" (PER), and "derivational" (DER) links to recognize equivalence between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| 0
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI2.3way
| style="text-align: center;"| 0.16
| style="text-align: center;"| 0.34
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DLSIUAES1.2way
| style="text-align: center;"| 0.83
| style="text-align: center;"| —
| Similarity between lemmata, computed by WordNet-based metrics

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DLSIUAES1.3way
| style="text-align: center;"| -0.5
| style="text-align: center;"| -0.33
| Similarity between lemmata, computed by WordNet-based metrics

|- bgcolor="#ECECEC" "align="left"
| WordNet
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0.34
| style="text-align: center;"| —
| WordNet based Unigram match: if any synset for the H unigram matches with any synset of a word in T then the hypothesis unigram is considered as a WordNet based unigram match.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Derivational Morphology from WordNet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa1.2way
| style="text-align: center;"| 1.33
| style="text-align: center;"| —
| Verb Entailment from Wordnet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa2.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Derivational Morphology from WordNet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa2.2way
| style="text-align: center;"| -0.33
| style="text-align: center;"| —
| Verb Entailment from Wordnet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| QUANTA1.2way
| style="text-align: center;"| -0.17
| style="text-align: center;"| —
| We use several relations from wordnet, such as synonyms, hyponym, hypernym et al.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Rhodes1.3way
| style="text-align: center;"| 3.17
| style="text-align: center;"| 4
| Lexicon based match: we chose a very simple metric: matching between words in T and H based on a path of distance at most 2 in the WordNet graph, using any links (hyponymy, hypernymy, meronymy, pertainymy, etc.)

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Sagan1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| -0.83
| The system is based on machine learning approach. The ablation test was obtained with 2 less features using WordNet (namely, string similarity based on Levenshtein distance and semantic similarity) in the training and testing steps.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Siel_093.3way
| style="text-align: center;"| 0.34
| style="text-align: center;"| -0.17
| Similarity between nouns using WN tool

|- bgcolor="#ECECEC" "align="left"
| WordNet
| ssl1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.67
| WordNet Analysis

|- bgcolor="#ECECEC" "align="left"
| WordNet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Synonyms, hypernyms (2 levels away from the original term)

|- bgcolor="#ECECEC" "align="left"
| WordNet
| UI_ccg1.2way
| style="text-align: center;"| 4
| style="text-align: center;"| —
| Word similarity == identity

|- bgcolor="#ECECEC" "align="left"
| WordNet + FrameNet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| WN: synonyms, hypernyms (2 levels away from the original term). FN: if two lexical items are covered in a single FrameNet frame, then the two items are treated as semantically related.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.17
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI2.3way
| style="text-align: center;"| 0.5
| style="text-align: center;"| 0.67
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| UAIC20091.3way
| style="text-align: center;"| 2
| style="text-align: center;"| 1.50
| Contradiction identification

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean + DLSIUAES_negation_list
| DLSIUAES1.2way
| style="text-align: center;"| 0.66
| style="text-align: center;"| —
| Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by participant themselves)

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean + DLSIUAES_negation_list
| DLSIUAES1.3way
| style="text-align: center;"| -1
| style="text-align: center;"| -0.5
| Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by participant themselves)

|- bgcolor="#ECECEC" "align="left"
| WordNet + XWordNet
| UAIC20091.3way
| style="text-align: center;"| 1
| style="text-align: center;"| 1.33
| Synonymy, hyponymy and hypernymy and eXtended WordNet relation

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 4.67
| style="text-align: center;"| —
| The ablation test (abl-1) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| -1.5
| style="text-align: center;"| —
| The ablation test (abl-2) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 0.17
| style="text-align: center;"| —
| The ablation test (abl-3) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| -1.16
| style="text-align: center;"| —
| The ablation test (abl-4) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 4.17
| style="text-align: center;"| —
| The ablation test (abl-5) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| Other
| UAIC20091.3way
| style="text-align: center;"| 4.17
| style="text-align: center;"| 4
| Pre-processing module, using MINIPAR, TreeTagger tool and some transformations, e.g. ''hasn't'' > ''has not''

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Everything ablated except lexical-based metrics

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.2way
| style="text-align: center;"| 3.33
| style="text-align: center;"| —
| Everything ablated except semantic-derived inferences

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.33
| Everything ablated except lexical-based metrics

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.3way
| style="text-align: center;"| 2.33
| style="text-align: center;"| 3.17
| Everything ablated except semantic-derived inferences

|- bgcolor="#ECECEC" "align="left"
| Other
| FBKirst1.2way
| style="text-align: center;"| 2.84
| style="text-align: center;"| —
| The automatic estimation of operation costs from run-1 modules was removed: the set of costs were assigned manually.

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Skip bigram match

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Bigram match

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Longest Common Subsequence

|- bgcolor="#ECECEC" "align="left"
| Stemmer
| JU_CSE_TAC1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Stemming, using WordNet stemmer

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| -2.5
| style="text-align: center;"| —
| Idf score

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| -0.66
| style="text-align: center;"| —
| Proper Noun Levenstain Distance

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| 0.34
| style="text-align: center;"| —
| J&C (Jiang and Conrath, 1997) similarity score on nouns, adjectives

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Idf score

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 0.17
| style="text-align: center;"| —
| Proper Noun Levenstain Distance

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 0.5
| style="text-align: center;"| —
| J&C (Jiang and Conrath, 1997) similarity score on nouns, adjectives

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.17
| Acronym match: we match words in all caps against sequences of capitalized words whose initial characters concatenate to form the acronym

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 3.33
| style="text-align: center;"| 1.83
| Proper nouns match: exact string match between T and H, for proper nouns

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 0.33
| style="text-align: center;"| 0.17
| Numbers match: exact string match between T and H, for numbers

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 3.17
| style="text-align: center;"| 4
| Edit-distance-based matching: 2 words match if 80% of the letters of a H word occur in one or more adjacent T words in the same order

|- bgcolor="#ECECEC" "align="left"
| Other
| UI_ccg1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Less sophisticated NE similarity metric: mainly Jaro-Winkler-based

|}
 
==Footnotes==
<references />

Return to [[RTE Knowledge Resources]]

RTE5 - Ablation Tests

2010-07-23T09:52:46Z

Celct:

The following table lists the results of the ablation tests submitted by participants, which have been introduced as a mandatory track in the RTE5 campaign. 
The first column contains the specific resources which have been ablated. The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way). The third and fourth columns present the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system. The third refers to the score obtained in the 2-way task, the fourth to that obtained in the 3-way task. For all the runs submitted as 3-way, also the 2-way derived accuracy has been calculated. 
Finally, the fifth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the RTE5 proceedings. 

Participants are kindly invited to check if all the inserted information is correct and complete.

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! Ablated Resource
! Team Run<ref>For further information about participants, click here: [[RTE Challenges - Data about participants]]</ref>
! Resource impact 2way
! Resource impact 3way
! Resource Usage Description

|- bgcolor="#ECECEC" "align="left"
| Acronym guide
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| The acronyms are expanded using the acronym database, so the acronyms are also matched with the expanded acronyms, and entailment is predicted accordingly

|- bgcolor="#ECECEC" "align="left"
| Acronym guide + UAIC_Acronym_rules
| UAIC20091.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.16
| We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.

|- bgcolor="#ECECEC" "align="left"
| DIRT
| BIU1.2way
| style="text-align: center;"| 1.33
| style="text-align: center;"| —
| Inference rules

|- bgcolor="#ECECEC" "align="left"
| DIRT
| Boeing3.3way
| style="text-align: center;"| -1.17
| style="text-align: center;"| 0
| Verb paraphrases

|- bgcolor="#ECECEC" "align="left"
| DIRT
| UAIC20091.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.33
| We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H

|- bgcolor="#ECECEC" "align="left"
| Framenet+ WordNet
| DLSIUAES1.2way
| style="text-align: center;"| 1.16
| style="text-align: center;"| —
| Frame-to-frame similarity metric

|- bgcolor="#ECECEC" "align="left"
| Framenet+ WordNet
| DLSIUAES1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.17
| Frame-to-frame similarity metric

|- bgcolor="#ECECEC" "align="left"
| Framenet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| If two lexical items are covered in a single FrameNet frame, then the two items are treated as semantically related.

|- bgcolor="#ECECEC" "align="left"
| Grady Ward’s MOBY Thesaurus + Roget's Thesaurus
| Venses2.2way
| style="text-align: center;"| 2.83
| style="text-align: center;"| —
| Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas

|- bgcolor="#ECECEC" "align="left"
| MontyLingua Tool
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form

|- bgcolor="#ECECEC" "align="left"
| NEGATION_rules by UAIC
| UAIC20091.3way
| style="text-align: center;"| 0
| style="text-align: center;"| -1.34
| Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.

|- bgcolor="#ECECEC" "align="left"
| NER (RASP Parser nertag)
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Named Entity match: measure based on the number of Nes in the hypothesis that match in the corresponding text. For named entity recognition, the RASP Parser (Briscoe et al., 2006) nertag component has been used.

|- bgcolor="#ECECEC" "align="left"
| NE component
| UI_ccg1.2way
| style="text-align: center;"| 4.83
| style="text-align: center;"| —
| Named Entity recognition/comparison

|- bgcolor="#ECECEC" "align="left"
| PropBank
| cswhu1.3way
| style="text-align: center;"| 2
| style="text-align: center;"| 3.17
| syntactic and semantic parsing

|- bgcolor="#ECECEC" "align="left"
| Stanford NER
| QUANTA1.2way
| style="text-align: center;"| 0.67
| style="text-align: center;"| —
| We use Named Entity similarity as a feature

|- bgcolor="#ECECEC" "align="left"
| Stopword list
| FBKirst1.2way
| style="text-align: center;"| 1.5
| style="text-align: center;"| —
| A list of the 572 most frequent English words has been collected in order to prevent assigning high costs to the deletion/insertion of terms that are unlikely to bring relevant information to detect entailment,and to avoid substituting these terms with any content word.

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE1, 2, 3
| PeMoZa3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE2
| PeMoZa3.2way
| style="text-align: center;"| 0.66
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| Training data from RTE2, 3
| PeMoZa3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
|

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.17
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI2.3way
| style="text-align: center;"| 0.33
| style="text-align: center;"| 0.5
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| VerbOcean relations are used to calculate relatedness between verbs in T and H

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| FBKirst1.2way
| style="text-align: center;"| -0.16
| style="text-align: center;"| —
| Extraction of 18232 entailment rules for all the English verbs connected by the ”stronger-than” relation. For instance, if ”kill [stronger-than] injure”, then the rule ”kill ENTAILS injure” is added to the rules repository.

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| QUANTA1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| We use "opposite-of" relation in VerbOcean as a feature

|- bgcolor="#ECECEC" "align="left"
| VerbOcean
| Siel_093.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0
| Similarity/anthonymy/unrelatedness between verbs

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| BIU1.2way
| style="text-align: center;"| -1
| style="text-align: center;"| —
| Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| cswhu1.3way
| style="text-align: center;"| 1.33
| style="text-align: center;"| 3.34
| Lexical semantic rules

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| FBKirst1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Rules extracted from WP using Latent Semantic Analysis (LSA)

|- bgcolor="#ECECEC" "align="left"
| WikiPedia
| UAIC20091.3way
| style="text-align: center;"| 1.17
| style="text-align: center;"| 1.5
| Relations between named entities

|- bgcolor="#ECECEC" "align="left"
| Wikipedia + NER's (LingPipe, GATE) + Perl patterns
| UAIC20091.3way
| style="text-align: center;"| 6.17
| style="text-align: center;"| 5
| NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text

|- bgcolor="#ECECEC" "align="left"
| WordNet
| AUEBNLP1.3way
| style="text-align: center;"| -2
| style="text-align: center;"| -2.67
| Synonyms

|- bgcolor="#ECECEC" "align="left"
| WordNet
| BIU1.2way
| style="text-align: center;"| 2.5
| style="text-align: center;"| —
| Synonyms, hyponyms (2 levels away from the original term), hyponym_instance and derivations

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Boeing3.3way
| style="text-align: center;"| 4
| style="text-align: center;"| 5.67
| Wordnet synonyms, hypernyms relationships between (senses of) words, "similar" (SIM), "pertains" (PER), and "derivational" (DER) links to recognize equivalence between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| 0
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI2.3way
| style="text-align: center;"| 0.16
| style="text-align: center;"| 0.34
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| Argument alignment between T and H

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DLSIUAES1.2way
| style="text-align: center;"| 0.83
| style="text-align: center;"| —
| Similarity between lemmata, computed by WordNet-based metrics

|- bgcolor="#ECECEC" "align="left"
| WordNet
| DLSIUAES1.3way
| style="text-align: center;"| -0.5
| style="text-align: center;"| -0.33
| Similarity between lemmata, computed by WordNet-based metrics

|- bgcolor="#ECECEC" "align="left"
| WordNet
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0.34
| style="text-align: center;"| —
| WordNet based Unigram match: if any synset for the H unigram matches with any synset of a word in T then the hypothesis unigram is considered as a WordNet based unigram match.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Derivational Morphology from WordNet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa1.2way
| style="text-align: center;"| 1.33
| style="text-align: center;"| —
| Verb Entailment from Wordnet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa2.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Derivational Morphology from WordNet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| PeMoZa2.2way
| style="text-align: center;"| -0.33
| style="text-align: center;"| —
| Verb Entailment from Wordnet

|- bgcolor="#ECECEC" "align="left"
| WordNet
| QUANTA1.2way
| style="text-align: center;"| -0.17
| style="text-align: center;"| —
| We use several relations from wordnet, such as synonyms, hyponym, hypernym et al.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Rhodes1.3way
| style="text-align: center;"| 3.17
| style="text-align: center;"| 4
| Lexicon based match: we chose a very simple metric: matching between words in T and H based on a path of distance at most 2 in the WordNet graph, using any links (hyponymy, hypernymy, meronymy, pertainymy, etc.)

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Sagan1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| -0.83
| The system is based on machine learning approach. The ablation test was obtained with 2 less features using WordNet (namely, string similarity based on Levenshtein distance and semantic similarity) in the training and testing steps.

|- bgcolor="#ECECEC" "align="left"
| WordNet
| Siel_093.3way
| style="text-align: center;"| 0.34
| style="text-align: center;"| -0.17
| Similarity between nouns using WN tool

|- bgcolor="#ECECEC" "align="left"
| WordNet
| ssl1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.67
| WordNet Analysis

|- bgcolor="#ECECEC" "align="left"
| WordNet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Synonyms, hypernyms (2 levels away from the original term)

|- bgcolor="#ECECEC" "align="left"
| WordNet
| UI_ccg1.2way
| style="text-align: center;"| 4
| style="text-align: center;"| —
| Word similarity == identity

|- bgcolor="#ECECEC" "align="left"
| WordNet + FrameNet
| UB.dmirg3.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| WN: synonyms, hypernyms (2 levels away from the original term). FN: if two lexical items are covered in a single FrameNet frame, then the two items are treated as semantically related.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI1.3way
| style="text-align: center;"| 0
| style="text-align: center;"| 0.17
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI2.3way
| style="text-align: center;"| 0.5
| style="text-align: center;"| 0.67
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| DFKI3.3way
| style="text-align: center;"| 0.17
| style="text-align: center;"| 0.17
| VerbOcean is used to calculate relatedness between nominal predicates in T and H, after using WordNet to change the nouns into verbs.

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean
| UAIC20091.3way
| style="text-align: center;"| 2
| style="text-align: center;"| 1.50
| Contradiction identification

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean + DLSIUAES_negation_list
| DLSIUAES1.2way
| style="text-align: center;"| 0.66
| style="text-align: center;"| —
| Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by participant themselves)

|- bgcolor="#ECECEC" "align="left"
| WordNet + VerbOcean + DLSIUAES_negation_list
| DLSIUAES1.3way
| style="text-align: center;"| -1
| style="text-align: center;"| -0.5
| Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by participant themselves)

|- bgcolor="#ECECEC" "align="left"
| WordNet + XWordNet
| UAIC20091.3way
| style="text-align: center;"| 1
| style="text-align: center;"| 1.33
| Synonymy, hyponymy and hypernymy and eXtended WordNet relation

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 4.67
| style="text-align: center;"| —
| The ablation test (abl-1) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| -1.5
| style="text-align: center;"| —
| The ablation test (abl-2) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 0.17
| style="text-align: center;"| —
| The ablation test (abl-3) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| -1.16
| style="text-align: center;"| —
| The ablation test (abl-4) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| System component
| DirRelCond3.2way
| style="text-align: center;"| 4.17
| style="text-align: center;"| —
| The ablation test (abl-5) was meant to test one component of the most complex condition for entailment used in step 3 of the system

|- bgcolor="#ECECEC" "align="left"
| Other
| UAIC20091.3way
| style="text-align: center;"| 4.17
| style="text-align: center;"| 4
| Pre-processing module, using MINIPAR, TreeTagger tool and some transformations, e.g. ''hasn't'' > ''has not''

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Everything ablated except lexical-based metrics

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.2way
| style="text-align: center;"| 3.33
| style="text-align: center;"| —
| Everything ablated except semantic-derived inferences

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.33
| Everything ablated except lexical-based metrics

|- bgcolor="#ECECEC" "align="left"
| Other
| DLSIUAES1.3way
| style="text-align: center;"| 2.33
| style="text-align: center;"| 3.17
| Everything ablated except semantic-derived inferences

|- bgcolor="#ECECEC" "align="left"
| Other
| FBKirst1.2way
| style="text-align: center;"| 2.84
| style="text-align: center;"| —
| The automatic estimation of operation costs from run-1 modules was removed: the set of costs were assigned manually.

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Skip bigram match

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| 0
| style="text-align: center;"| —
| Bigram match

|- bgcolor="#ECECEC" "align="left"
| Other
| JU_CSE_TAC1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Longest Common Subsequence

|- bgcolor="#ECECEC" "align="left"
| Stemmer
| JU_CSE_TAC1.2way
| style="text-align: center;"| -0.5
| style="text-align: center;"| —
| Stemming, using WordNet stemmer

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| -2.5
| style="text-align: center;"| —
| Idf score

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| -0.66
| style="text-align: center;"| —
| Proper Noun Levenstain Distance

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa1.2way
| style="text-align: center;"| 0.34
| style="text-align: center;"| —
| J&C (Jiang and Conrath, 1997) similarity score on nouns, adjectives

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Idf score

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 0.17
| style="text-align: center;"| —
| Proper Noun Levenstain Distance

|- bgcolor="#ECECEC" "align="left"
| Other
| PeMoZa2.2way
| style="text-align: center;"| 0.5
| style="text-align: center;"| —
| J&C (Jiang and Conrath, 1997) similarity score on nouns, adjectives

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| -0.17
| style="text-align: center;"| -0.17
| Acronym match: we match words in all caps against sequences of capitalized words whose initial characters concatenate to form the acronym

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 3.33
| style="text-align: center;"| 1.83
| Proper nouns match: exact string match between T and H, for proper nouns

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 0.33
| style="text-align: center;"| 0.17
| Numbers match: exact string match between T and H, for numbers

|- bgcolor="#ECECEC" "align="left"
| Other
| Rhodes1.3way
| style="text-align: center;"| 3.17
| style="text-align: center;"| 4
| Edit-distance-based matching: 2 words match if 80% of the letters of a H word occur in one or more adjacent T words in the same order

|- bgcolor="#ECECEC" "align="left"
| Other
| UI_ccg1.2way
| style="text-align: center;"| 1
| style="text-align: center;"| —
| Less sophisticated NE similarity metric: mainly Jaro-Winkler-based

|}
 
==Footnotes==
<references />

Return to [[RTE Knowledge Resources]]

RTE5 - Ablation Tests

2010-07-23T09:51:54Z

Celct:

User talk:Celct

2010-05-07T12:32:36Z

Celct: /* Orphan Pages */

Thank you for this remark.
Is it possible to delete them? I didn't manage to use a sandbox, at the time I created this pages as possible alternatives to other pages I actually published.
Can you help me?

== Orphan Pages ==

Thanks for your many contributions to the ACL Wiki. Did you know that you have three [http://aclweb.org/aclwiki/index.php?title=Special:Lonelypages orphan pages]? --[[User:Pdturney|Pdturney]] 22:24, 21 February 2010 (UTC)

RTE Knowledge Resources

2009-12-22T17:40:25Z

Celct: /* Publicly available Resources */

Knowledge resources have shown their relevance for applied semantic inference, and are extensively used by applied inference systems, such as those developed within the Textual Entailment framework. 
 
This page presents a list of the knowledge resources used by systems that have participated in the last RTE challenges. The first table lists the publicly available resources, the second one lists unpublished resources. Both tables are sortable by Resource name, type, author and number of users. 
 
RTE Participants are encouraged to add information about all kind of knowledge resources used, from standard existing resources (e.g. WordNet) to knowledge collections created for specific purposes, which can be made available to the community. 
 
__TOC__
 
=== Call for Resources ===
[[RTE6 - Call for Resources]]
 

=== Ablation Tests ===
[[RTE5 - Ablation Tests]]
 

=== Publicly available Resources ===
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="150"|Author
! class="unsortable" width="300"|Brief description
! width="45"|PAST Users <ref name:"rtethree">RTE-3 data have been provided by participants by means of a questionnaire.</ref>
! width="45"|RTE4 Users<ref name:"rtefour">RTE-4 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! width="45"|RTE5 Users<ref name:"rtefive">RTE-5 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [[WordNet]]
| Lexical DB
| Princeton University
| Lexical database of English nouns, verbs, adjectives and adverbs
| style="text-align: center;"|3
| style="text-align: center;"|21
| style="text-align: center;"|18
| [[WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://xwn.hlt.utdallas.edu/index.html eXtended Wordnet]
| Lexical DB
| Human Language Technology Research Institute, University of Texas at Dallas
| Extension of WordNet based on the exploitation of the information contained in WordNet definitional glosses: the glosses are syntactically parsed, transformed into logic forms and content words are semantically disambiguated. The Extended Wordnet is an ongoing project.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|2
| [[eXtended WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://ai.stanford.edu/~rion/swn/ Augmented Wordnet]
| Lexical DB
| Stanford University
| The resource is the result of the application of a learning algorithm for inducing semantic taxonomies from parsed text. The algorithm automatically acquires items of world knowledge, and uses these to produce significantly enhanced versions of WordNet (up to 40,000 synsets more).
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Augmented Wordnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/verbnet.html Verbnet]
| Lexical DB
| University of Colorado Boulder
| Lexicon for English verbs organized into classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class
| style="text-align: center;"|2
| style="text-align: center;"|2
| style="text-align: center;"|1
| [[Verbnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[VerbOcean]]
| Lexical DB
| Information Sciences Institute, University of Southern California
| Broad-coverage semantic network of verbs
| style="text-align: center;"|2
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[VerbOcean - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://framenet.icsi.berkeley.edu/ FrameNet]
| Lexical DB
| ICSI (International Computer Science Institute) - Berkley University
| Lexical resource for English words, based on frame semantics (valences) and supported by corpus evidence
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|2
| [[Framenet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/meyers/NomBank.html NomBank]
| Lexical DB
| New York University
| Lexical resource containing syntactic frames for nouns, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[NomBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/ace.html PropBank]
| Lexical DB
| University of Colorado Boulder
| Lexical resource containing syntactic frames for verbs, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[PropBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/nomlex/index.html Nomlex] Plus
| Lexical DB
| New York University
| Dictionary of English nominalizations: it describes the allowed complements for a nominalization and relates the nominal complements to the arguments of the corresponding verb
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Nomlex Plus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.ualberta.ca/~lindek/downloads.htm Dekang Lin’s Thesaurus]
| Thesaurus
| University of Alberta
| Thesaurus automatically constructed using a parsed corpus, based on distributional similarity scores
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Dekang Lin’s Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://icon.shef.ac.uk/Moby/mthes.html Grady Ward's Moby Thesaurus]
| Thesaurus
| University of Sheffield
| Thesaurus containing 30,260 root words, with 2,520,264 synonyms and related terms. Grady Ward placed this thesaurus in the public domain in 1996.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Grady Ward's Moby Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://en.wikipedia.org/wiki/Roget%27s_Thesaurus Roget's Thesaurus]
| Thesaurus
| Peter Mark Roget (Electronic version distributed by University of Chicago)
| Roget's Thesaurus is a widely-used English thesaurus, created by Dr. Peter Mark Roget in 1805. The original edition had 15,000 words, and each new edition has been larger. The electronic edition ([http://machaut.uchicago.edu/rogets version 1.02]) is made available by University of Chicago.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Roget's Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.wikipedia.org/ Wikipedia]
| Encyclopedia
|
| Free encyclopedia. Used for extraction of lexical-semantic rules (from its more structured parts), named entity recognition, geographical information etc.
| style="text-align: center;"|0
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[Wikipedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.umbel.org/ Umbel]
| Ontology
| Structured Dynamics LLC, Coralville, IA
| UMBEL stands for Upper Mapping and Binding Exchange Layer and is a lightweight ontology structure for relating Web content and data to a standard set of subject concepts
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Umbel - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.mpi-inf.mpg.de/yago-naga/yago/ YAGO]
| Ontology
| Max-Planck Institute for Informatics, Saarbrücken, Germany
| Light-weight and extensible ontology. It contains more than 2 million entities and 20 million facts about these entities. The facts have been automatically extracted from Wikipedia and unified with WordNet.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[YAGO - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://dbpedia.org/About DBpedia]
| Ontology
| Open community project
| DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. The DBpedia knowledge base currently describes more than 2.9 million things in 91 different languages and consists of 479 million pieces of information.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DBpedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[DIRT Paraphrase Collection]]
| Collection of paraphrases
| University of Alberta
| DIRT (Discovery of Inference Rules from Text) is both an algorithm and a resulting knowledge collection. The DIRT knowledge collection is the output of the DIRT algorithm over a 1GB set of newspaper text.
| style="text-align: center;"|2
| style="text-align: center;"|4
| style="text-align: center;"|3
| [[DIRT Paraphrase Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[TEASE]] Collection
| Collection of Entailment Rules
| Bar-Ilan University
| Output of the TEASE algorithm
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Tease Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://badc.nerc.ac.uk/help/abbrevs.html BADC Acronym and Abbreviation List]
| Word List
| BADC (British Atmospheric Data Centre)
| Acronym and Abbreviation List
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[BADC Acronym and Abbreviation List - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.acronym-guide.com/ Acronym Guide]
| Word List
| Acronym-Guide.com
| Acronym and Abbreviation Lists for English, branched in thematic directories
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|3
| [[Acronym Guide - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 Web1T 5-grams]
| Word list
| Linguistic Data Consortium, University of Pennsylvania; Google Inc.
| Data set containing English word n-grams and their observed frequency counts. The n-gram counts were generated from approximately 1 trillion word tokens of text from publicly accessible Web pages
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Web1T - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE3_RTE4_NGD.zip Normalized Google Distance (RTE3&RTE4)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE3&RTE4)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE5_NGD.zip Normalized Google Distance (RTE5)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE5)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://geonames.usgs.gov/index.html GNIS - Geographic Names Information System]
| Gazetteer
| USGS (United States Geological Survey)
| Database containing the Federal and national standard toponyms for USA, associated areas and Antarctica
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[GNIS - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.geonames.org/ Geonames]
| Gazetteer
|
| Database containing eight million geographical names. It is integrating geographical data such as names of places in various languages, elevation, population and others from various sources.
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geonames - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/paraphrase/ Sekine's Paraphrase Database]
| Collection of paraphrases
| Department of Computer Science, New York University
| Data-base created using Sekine's method, NOT cleaned up by human. It includes 19,975 sets of paraphrases with 191,572 phrases.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Sekine's Paraphrase Database - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://research.microsoft.com/research/downloads/Details/607D14D9-20CD-47E3-85BC-A2F65CD28042/Details.aspx Microsoft Research Paraphrase Corpus]
| Collection of paraphrases
| Microsoft Research
| Text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Microsoft Research Paraphrase Corpus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.cornell.edu/~cristian/Without_a_doubt_-_Data.html Downward entailing operators]
| Collection of entailing operators
| Department of Computer Science, Cornell University, Ithaca NY
| System output of an unsupervised algorithm recovering many Downward Entailing operators, like 'doubt'.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Downward entailing operators - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
|[http://cs.biu.ac.il/~shey/WikiRules.html WikiRules!]
| Lexical Reference rule-base
| Bar-Ilan University
| Extraction of lexical reference rules from the text body (first sentence) and from metadata (links, redirects, parentheses) of Wikipedia
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[WikiRules! - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''New resource''
|
|
| ''Participants are encouraged to contribute''
| style="text-align: center;"|
| style="text-align: center;"|
| style="text-align: center;"|
| [[New Resource2 - RTE Users|Users]]

|}
 
 

=== Not available Resources ===
The following table lists the unpublished resources used by RTE participants. Some of them have been developed by Users themselves specifically for RTE. Interested people may turn to authors to obtain further information.
 
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="180"|Author
! class="unsortable" width="500"|Brief description
! width="30"|PAST Users
! width="30"|RTE4 Users
! width="30"|RTE5 Users
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [http://www.parc.com/ PARC] Polarity Lexicon
| Lexical DB
| PARC - Palo Alto Research Center
| Verbs classification with respect to semantic polarity
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Parc Polarity Lexicon - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Gazetteer from [http://trec.nist.gov/ TREC]
| Gazetteer
| NIST - National Institute of Standards and Technology
| Cities and other geographical names
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Gazetteer from TREC - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DFKI Geographic Ontology ''(to be released)''
| Ontology
| DFKI - German Research Center for Artificial Intelligence
| Ontology containing geographic terms and two kinds of relations: the directional ''part-of'' relation, and the ''equal'' relation for synonyms and abbreviations of the same geographic area (e.g ''the United Kingdom'', ''the UK'', ''Great Britain'', etc.)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geographic Ontology - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Geo''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Meronymy entailment rules, based on TREC’s TIPSTER gazetteer.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Geo - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Regex''
| Collection of Entailment rules
| Bar-Ilan University; Tel-Aviv University
| Small set of entailment rules based on regular expressions, intended to address lexical variability involving temporal phrases
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Regex - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Syntactic rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which define parse tree transformations. The rules cover generic syntactic phenomena such as appositions, conjunctions, passive, relative clause, etc. (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Polarity rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which detect predicates whose polarity is negative (e.g. didn't dance) or unknown (e.g. plans to dance). The rules capture diverse phenomena that affect polarity, e.g. verbal negation, modal verbs, conditionals, and certain verbs that induce negative or "unknown" polarity context. The latter were taken mainly from VerbNet. Extends a resource described in (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Polarity rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Lexical-Syntactic rule base
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Extract lexical-syntactic entailment rules for predicates (verbal and nominal), including argument mapping. The resource is based on WordNet, Nomlex-Plus and Unary DIRT (Szpektor and Dagan, Coling 08)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Lexical-Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| OPENU Collection
| Collection of Entailment Rules and Patterns
| Open University
| Collections of rules, patterns etc. for RTE purpose, extracted from Reuter corpus parsed using Minipar.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[OPENU Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Abbr''
| Collection of rules for abbreviation
| Bar-Ilan University; Tel-Aviv University
| 2000 Abbreviation rules, extracted from [http://badc.nerc.ac.uk/help/abbrevs.html BADC] and [http://www.acronym-guide.com/ Acronym Guide]
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Abbr - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Negation_list
| Negation rules
| „Al. I. Cuza“ University, Iasi, Romania
| List of negative terms and words (verbs, adjectives, nouns) affecting modality or factuality of a infinitive verb preceded by the particle "to" (e.g. "believe","necessary", "attempt")
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DLSIUAES Negation_list
| List of negative terms
| University of Alicante
| Basic list of negative terms.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DLSIUAES Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Quantifier_list
| List of quantifiers
| „Al. I. Cuza“ University, Iasi, Romania
| List of quantifiers affecting entailment judgment. The quantifiers are taken from a list which contains expressions like “more than”, “less than”, or words such as “over”, “under”, etc.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Quantifier_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| FBKirst StopWord list
| List of frequent words
| FBK-Irst; University of Trento - Italy
| A list of the 572 most frequent English words.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[FBKirst StopWord list - RTE Users|Users]]

|}
 

==Footnotes==
<references/>

Acronym Guide - RTE Users

2009-12-22T17:39:29Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| Used in the search task, together with [[BADC_Acronym_and_Abbreviation_List_-_RTE_Users|BADC]], to create the new resource [[RTE_Knowledge_Resources#Not_available_Resources|Abbr]]
| No ablation test performed

|- bgcolor="#ECECEC" align="left"
| Siel_09
| RTE5
|
| The acronyms are expanded using the acronym database, so the acronyms are also matched with the expanded acronyms, and entailment is predicted accordingly. Ex: UK United Kingdom
| Ablation test performed. Null impact of the resource both on two-way and on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| Acronym resolution. The resource has been expanded by a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form
| Ablation test performed. Positive impact of the acronym module: +0.17% on two-way, +0.16% on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| Ablation test performed: +0.3% precision on two-way task

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE3
|
|
| Ablation test performed: +0.75% precision on two-way task
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 5
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

RTE Knowledge Resources

2009-12-22T17:36:37Z

Celct: /* Publicly available Resources */

Knowledge resources have shown their relevance for applied semantic inference, and are extensively used by applied inference systems, such as those developed within the Textual Entailment framework. 
 
This page presents a list of the knowledge resources used by systems that have participated in the last RTE challenges. The first table lists the publicly available resources, the second one lists unpublished resources. Both tables are sortable by Resource name, type, author and number of users. 
 
RTE Participants are encouraged to add information about all kind of knowledge resources used, from standard existing resources (e.g. WordNet) to knowledge collections created for specific purposes, which can be made available to the community. 
 
__TOC__
 
=== Call for Resources ===
[[RTE6 - Call for Resources]]
 

=== Ablation Tests ===
[[RTE5 - Ablation Tests]]
 

=== Publicly available Resources ===
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="150"|Author
! class="unsortable" width="300"|Brief description
! width="45"|PAST Users <ref name:"rtethree">RTE-3 data have been provided by participants by means of a questionnaire.</ref>
! width="45"|RTE4 Users<ref name:"rtefour">RTE-4 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! width="45"|RTE5 Users<ref name:"rtefive">RTE-5 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [[WordNet]]
| Lexical DB
| Princeton University
| Lexical database of English nouns, verbs, adjectives and adverbs
| style="text-align: center;"|3
| style="text-align: center;"|21
| style="text-align: center;"|18
| [[WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://xwn.hlt.utdallas.edu/index.html eXtended Wordnet]
| Lexical DB
| Human Language Technology Research Institute, University of Texas at Dallas
| Extension of WordNet based on the exploitation of the information contained in WordNet definitional glosses: the glosses are syntactically parsed, transformed into logic forms and content words are semantically disambiguated. The Extended Wordnet is an ongoing project.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|2
| [[eXtended WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://ai.stanford.edu/~rion/swn/ Augmented Wordnet]
| Lexical DB
| Stanford University
| The resource is the result of the application of a learning algorithm for inducing semantic taxonomies from parsed text. The algorithm automatically acquires items of world knowledge, and uses these to produce significantly enhanced versions of WordNet (up to 40,000 synsets more).
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Augmented Wordnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/verbnet.html Verbnet]
| Lexical DB
| University of Colorado Boulder
| Lexicon for English verbs organized into classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class
| style="text-align: center;"|2
| style="text-align: center;"|2
| style="text-align: center;"|1
| [[Verbnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[VerbOcean]]
| Lexical DB
| Information Sciences Institute, University of Southern California
| Broad-coverage semantic network of verbs
| style="text-align: center;"|2
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[VerbOcean - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://framenet.icsi.berkeley.edu/ FrameNet]
| Lexical DB
| ICSI (International Computer Science Institute) - Berkley University
| Lexical resource for English words, based on frame semantics (valences) and supported by corpus evidence
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|2
| [[Framenet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/meyers/NomBank.html NomBank]
| Lexical DB
| New York University
| Lexical resource containing syntactic frames for nouns, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[NomBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/ace.html PropBank]
| Lexical DB
| University of Colorado Boulder
| Lexical resource containing syntactic frames for verbs, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[PropBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/nomlex/index.html Nomlex] Plus
| Lexical DB
| New York University
| Dictionary of English nominalizations: it describes the allowed complements for a nominalization and relates the nominal complements to the arguments of the corresponding verb
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Nomlex Plus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.ualberta.ca/~lindek/downloads.htm Dekang Lin’s Thesaurus]
| Thesaurus
| University of Alberta
| Thesaurus automatically constructed using a parsed corpus, based on distributional similarity scores
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Dekang Lin’s Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://icon.shef.ac.uk/Moby/mthes.html Grady Ward's Moby Thesaurus]
| Thesaurus
| University of Sheffield
| Thesaurus containing 30,260 root words, with 2,520,264 synonyms and related terms. Grady Ward placed this thesaurus in the public domain in 1996.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Grady Ward's Moby Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://en.wikipedia.org/wiki/Roget%27s_Thesaurus Roget's Thesaurus]
| Thesaurus
| Peter Mark Roget (Electronic version distributed by University of Chicago)
| Roget's Thesaurus is a widely-used English thesaurus, created by Dr. Peter Mark Roget in 1805. The original edition had 15,000 words, and each new edition has been larger. The electronic edition ([http://machaut.uchicago.edu/rogets version 1.02]) is made available by University of Chicago.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Roget's Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.wikipedia.org/ Wikipedia]
| Encyclopedia
|
| Free encyclopedia. Used for extraction of lexical-semantic rules (from its more structured parts), named entity recognition, geographical information etc.
| style="text-align: center;"|0
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[Wikipedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.umbel.org/ Umbel]
| Ontology
| Structured Dynamics LLC, Coralville, IA
| UMBEL stands for Upper Mapping and Binding Exchange Layer and is a lightweight ontology structure for relating Web content and data to a standard set of subject concepts
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Umbel - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.mpi-inf.mpg.de/yago-naga/yago/ YAGO]
| Ontology
| Max-Planck Institute for Informatics, Saarbrücken, Germany
| Light-weight and extensible ontology. It contains more than 2 million entities and 20 million facts about these entities. The facts have been automatically extracted from Wikipedia and unified with WordNet.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[YAGO - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://dbpedia.org/About DBpedia]
| Ontology
| Open community project
| DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. The DBpedia knowledge base currently describes more than 2.9 million things in 91 different languages and consists of 479 million pieces of information.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DBpedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[DIRT Paraphrase Collection]]
| Collection of paraphrases
| University of Alberta
| DIRT (Discovery of Inference Rules from Text) is both an algorithm and a resulting knowledge collection. The DIRT knowledge collection is the output of the DIRT algorithm over a 1GB set of newspaper text.
| style="text-align: center;"|2
| style="text-align: center;"|4
| style="text-align: center;"|3
| [[DIRT Paraphrase Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[TEASE]] Collection
| Collection of Entailment Rules
| Bar-Ilan University
| Output of the TEASE algorithm
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Tease Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://badc.nerc.ac.uk/help/abbrevs.html BADC Acronym and Abbreviation List]
| Word List
| BADC (British Atmospheric Data Centre)
| Acronym and Abbreviation List
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[BADC Acronym and Abbreviation List - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.acronym-guide.com/ Acronym Guide]
| Word List
| Acronym-Guide.com
| Acronym and Abbreviation Lists for English, branched in thematic directories
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|3
| [[Acronym Guide - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 Web1T 5-grams]
| Word list
| Linguistic Data Consortium, University of Pennsylvania; Google Inc.
| Data set containing English word n-grams and their observed frequency counts. The n-gram counts were generated from approximately 1 trillion word tokens of text from publicly accessible Web pages
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Web1T - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE3_RTE4_NGD.zip Normalized Google Distance (RTE3&RTE4)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE3&RTE4)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE5_NGD.zip Normalized Google Distance (RTE5)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE5)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://geonames.usgs.gov/index.html GNIS - Geographic Names Information System]
| Gazetteer
| USGS (United States Geological Survey)
| Database containing the Federal and national standard toponyms for USA, associated areas and Antarctica
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[GNIS - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.geonames.org/ Geonames]
| Gazetteer
|
| Database containing eight million geographical names. It is integrating geographical data such as names of places in various languages, elevation, population and others from various sources.
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geonames - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/paraphrase/ Sekine's Paraphrase Database]
| Collection of paraphrases
| Department of Computer Science, New York University
| Data-base created using Sekine's method, NOT cleaned up by human. It includes 19,975 sets of paraphrases with 191,572 phrases.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Sekine's Paraphrase Database - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://research.microsoft.com/research/downloads/Details/607D14D9-20CD-47E3-85BC-A2F65CD28042/Details.aspx Microsoft Research Paraphrase Corpus]
| Collection of paraphrases
| Microsoft Research
| Text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Microsoft Research Paraphrase Corpus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.cornell.edu/~cristian/Without_a_doubt_-_Data.html Downward entailing operators]
| Collection of entailing operators
| Department of Computer Science, Cornell University, Ithaca NY
| System output of an unsupervised algorithm recovering many Downward Entailing operators, like 'doubt'.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Downward entailing operators - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
|[http://cs.biu.ac.il/~shey/WikiRules.html WikiRules!]
| Lexical Reference rule-base
| Bar-Ilan University
| Extraction of lexical reference rules from the text body (first sentence) and from metadata (links, redirects, parentheses) of Wikipedia
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[WikiRules! - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''New resource''
|
|
| ''Participants are encouraged to contribute''
| style="text-align: center;"|
| style="text-align: center;"|
| style="text-align: center;"|
| [[New Resource2 - RTE Users|Users]]

|}
 
 

=== Not available Resources ===
The following table lists the unpublished resources used by RTE participants. Some of them have been developed by Users themselves specifically for RTE. Interested people may turn to authors to obtain further information.
 
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="180"|Author
! class="unsortable" width="500"|Brief description
! width="30"|PAST Users
! width="30"|RTE4 Users
! width="30"|RTE5 Users
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [http://www.parc.com/ PARC] Polarity Lexicon
| Lexical DB
| PARC - Palo Alto Research Center
| Verbs classification with respect to semantic polarity
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Parc Polarity Lexicon - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Gazetteer from [http://trec.nist.gov/ TREC]
| Gazetteer
| NIST - National Institute of Standards and Technology
| Cities and other geographical names
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Gazetteer from TREC - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DFKI Geographic Ontology ''(to be released)''
| Ontology
| DFKI - German Research Center for Artificial Intelligence
| Ontology containing geographic terms and two kinds of relations: the directional ''part-of'' relation, and the ''equal'' relation for synonyms and abbreviations of the same geographic area (e.g ''the United Kingdom'', ''the UK'', ''Great Britain'', etc.)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geographic Ontology - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Geo''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Meronymy entailment rules, based on TREC’s TIPSTER gazetteer.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Geo - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Regex''
| Collection of Entailment rules
| Bar-Ilan University; Tel-Aviv University
| Small set of entailment rules based on regular expressions, intended to address lexical variability involving temporal phrases
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Regex - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Syntactic rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which define parse tree transformations. The rules cover generic syntactic phenomena such as appositions, conjunctions, passive, relative clause, etc. (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Polarity rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which detect predicates whose polarity is negative (e.g. didn't dance) or unknown (e.g. plans to dance). The rules capture diverse phenomena that affect polarity, e.g. verbal negation, modal verbs, conditionals, and certain verbs that induce negative or "unknown" polarity context. The latter were taken mainly from VerbNet. Extends a resource described in (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Polarity rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Lexical-Syntactic rule base
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Extract lexical-syntactic entailment rules for predicates (verbal and nominal), including argument mapping. The resource is based on WordNet, Nomlex-Plus and Unary DIRT (Szpektor and Dagan, Coling 08)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Lexical-Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| OPENU Collection
| Collection of Entailment Rules and Patterns
| Open University
| Collections of rules, patterns etc. for RTE purpose, extracted from Reuter corpus parsed using Minipar.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[OPENU Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Abbr''
| Collection of rules for abbreviation
| Bar-Ilan University; Tel-Aviv University
| 2000 Abbreviation rules, extracted from [http://badc.nerc.ac.uk/help/abbrevs.html BADC] and [http://www.acronym-guide.com/ Acronym Guide]
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Abbr - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Negation_list
| Negation rules
| „Al. I. Cuza“ University, Iasi, Romania
| List of negative terms and words (verbs, adjectives, nouns) affecting modality or factuality of a infinitive verb preceded by the particle "to" (e.g. "believe","necessary", "attempt")
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DLSIUAES Negation_list
| List of negative terms
| University of Alicante
| Basic list of negative terms.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DLSIUAES Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Quantifier_list
| List of quantifiers
| „Al. I. Cuza“ University, Iasi, Romania
| List of quantifiers affecting entailment judgment. The quantifiers are taken from a list which contains expressions like “more than”, “less than”, or words such as “over”, “under”, etc.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Quantifier_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| FBKirst StopWord list
| List of frequent words
| FBK-Irst; University of Trento - Italy
| A list of the 572 most frequent English words.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[FBKirst StopWord list - RTE Users|Users]]

|}
 

==Footnotes==
<references/>

WordNet - RTE Users

2009-12-22T17:36:09Z

Celct: /* Past Campaigns Users */

Given the high number of WordNet users, the table has been splitted into two: the first containing the users of the last campaign, the second containing the users of previous campaigns.
__TOC__
 
==RTE5 Users==

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| AUEB
| RTE5
|
| During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical.
| Ablation test performed. Negative impact of the resource: -2% accuracy on two-way, -2.67% on three-way task.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
| 3.0
| Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations.
| Ablation test performed. Positive impact of the resource: +2.5% accuracy on two-way task.

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE5
|
| The system makes uses of Wordnet synonyms, hypernyms relationships between (senses of) words, "similar" (SIM), "pertains" (PER), and "derivational" (DER) links to recognize equivalence between T and H.
| Ablation test performed. Positive impact of the resource: +4% accuracy on two-way, +5.67% on three-way task.

|- bgcolor="#ECECEC" align="left"
| DFKI
| RTE5
|
| FIRST USE: Argument alignment between T and H. SECOND USE: used to change all the nominal predicates into verbs, to calculate relatedness between T and H (using VerbOcean).
| FIRST USE: Ablation test performed. Impact of the resource: -0.17% accuracy/null respectively on two-way and three-way task for run1; +0.16%/+0.34% for run2; +0.17%/+0.17% for run3. 
SECOND USE (WordNet+VerbOcean): null/+0.17% accuracy respectively on two-way and three-way task for run1; +0.5%/+0.67% for run2; +0.17%/+0.17% for run3.

|- bgcolor="#ECECEC" align="left"
| DirRelCond
| RTE5
|
| Use of many WordNet relations (such as synonymy, hypernymy, hyponymy, meronymy, holonymy etc.) to compute the relatedness between words with the same part of speech in T and H.
| No ablation test performed. The resource cannot be removed without breaking the functionality of the system.

|- bgcolor="#ECECEC" align="left"
| DLSIUAES
| RTE5
|
| FIRST USE: Similarity between lemmata, computed by WordNet-based metrics. 
SECOND USE: antonymy relations between verbs. 
THIRD USE: synonymy/identity between verb lemmata in T and H. 
FOURTH USE (WordNet+Framenet): WordNet synonym and hyponym relations from T's frame elements to H's frame elements.
| FIRST USE: Ablation test performed. Positive impact of the resource on two-way run: +0.83% accuracy. Negative impact on three-way run: -0.33% accuracy (-0.5% for two-way derived). 
SECOND USE (WordNet+VerbOcean+DLSIUAES_negation_list): positive impact on two-way run: +0.66% accuracy. Negative impact on three-way run: -1% (-0.5% for two-way derived). 
THIRD USE: No ablation test performed. 
FOURTH USE (WordNet+Framenet): positive impact on two-way run: +1.16% accuracy. Negative impact on three-way run: -0.17% (the same for two-way derived).

|- bgcolor="#ECECEC" align="left"
| FBKirst
| RTE5
| 3.0
| Extraction of a set of 2698 English entailment rules for terms connected by the hyponymy and synonymy relations
| No ablation test performed

|- bgcolor="#ECECEC" align="left"
| JU_CSE_TAC
| RTE5
|
| WordNet based Unigram match: if any synset for the H unigram matches with any synset of a word in T then the hypothesis unigram is considered as a WordNet based unigram match.
| Ablation test performed. Positive impact of the resource: +0.34% on two-way task.

|- bgcolor="#ECECEC" align="left"
| PeMoZa
| RTE5
|
| FIRST USE: Derivational Morphology. 
SECOND USE: Verb Entailment. 
| Ablation tests performed. 
FIRST USE. Impact of the resource on two-way task: -0.5%/+1% accuracy respectively on run1 and run2. 
SECOND USE. Impact of the resource on two-way task: +1.33%/-0.33% accuracy respectively on run1 and run2.

|- bgcolor="#ECECEC" align="left"
| QUANTA
| RTE5
|
| Several relations from wordnet, such as synonyms, hyponym, hypernym et al.
| Ablation test performed. Negative impact of the resource: -0.17% on two-way task.

|- bgcolor="#ECECEC" align="left"
| Rhodes
| RTE5
|
| FIRST USE: Lexicon based match: we chose a very simple metric: matching between words in T and H based on a path of distance at most 2 in the WordNet graph, using any links (hyponymy, hypernymy, meronymy, pertainymy, etc.) 
SECOND USE: contradiction detection based on antonymy relation
| FIRST USE: Ablation test performed. Positive impact of the resource: +3.17% accuracy on two-way, +4% on three-way task. 
SECOND USE: no ablation test performed.

|- bgcolor="#ECECEC" align="left"
| Sagan
| RTE5
|
| Used to obtain two features (string similarity based on Levenshtein distance and semantic similarity) in the training and testing steps of the system.
| Ablation test performed. Null/negative (-0.87%) impact of the resource respectively on two-way and three-way task.

|- bgcolor="#ECECEC" align="left"
| Siel_09
| RTE5
|
| Similarity between nouns using WN tool
| Ablation test performed. Impact of the resource: +0.34% accuracy on two-way, -0.17% on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| FIRST USE: Antonymy relation to detect contradiction. In order to broaden the domain of the antonymy relation, we consider a combination of synonyms and antonyms. Used in combination with VerbOcean. 
SECOND USE: Synonymy, hyponymy and hypernymy for nouns and adjectives. Used in combination with eXtended WordNet relations.
| FIRST USE: Ablation test performed (Wordnet + VerbOcean). Positive impact of the two resources together: +2% accuracy on two-way, +1.5% on three-way task. 
SECOND USE: Ablation test performed (Wordnet + eXtended WordNet). Positive impact of the two resources together: +1% accuracy on two-way, +1.33% on three-way task.

|- bgcolor="#ECECEC" align="left"
| UB.dmirg
| RTE5
|
| When using WordNet, we assume that a term is semantically interchangeable with its exact occurrence, its synonyms, and its hypernyms. In extracting hypernyms, we exclude the hypernyms that are more distant than two links to the original terms in WordNet synsets.
| Two ablation tests performed. The first for Wordnet alone, the second for both WordNet and Framenet. Null impact of the resource(s) on two-way task for both ablations.

|- bgcolor="#ECECEC" align="left"
| UI_ccg
| RTE5
|
| Word similarity == identity
| Ablation test performed. Positive impact of the resource: +4% accuracy on two-way task.

|- bgcolor="#ECECEC" align="left"
| Unimelb
| RTE5
|
| Synonyms, derivationally-related forms.
| Used in the search task.
|}

{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Subtotal: 17
|}
 

==Past Campaigns Users==

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| AUEB
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
| 3.0
| Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations. Also used as part of our novel lexical-syntactic resource
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]]

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE4
| 2.0
| Semantic relation between words
| No formal evaluation. Plays a role in most entailments found

|- bgcolor="#ECECEC" align="left"
| Cambridge
| RTE4
| 3.0
| Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x)
| No systematic evaluation

|- bgcolor="#ECECEC" align="left"
| CERES
| RTE4
| 3.0
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv)
| Used, but no evaluation performed

|- bgcolor="#ECECEC" align="left"
| DFKI
| RTE4
| 3.0
| Semantic relation between words
| No separate evaluation

|- bgcolor="#ECECEC" align="left"
| DLSIUAES
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| EMORY
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| FbkIrst
| RTE4
| 3.0
| Lexical similarity
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet

|- bgcolor="#ECECEC" align="left"
| FSC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| IIT
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| IPD
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| OAQA
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| QUANTA
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| SAGAN
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| Stanford
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| Ablation test performed: +3% precision on two-way task.

|- bgcolor="#ECECEC" align="left"
| UMD
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UNED
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| Uoeltg
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UPC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| AUEB
| RTE3
| 2.1
| Synonymy resolution
| Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| Ablation test performed: +1.13% precision on two-way task.

|- bgcolor="#ECECEC" align="left"
| UIUC
| RTE3
|
| Semantic distance between words
|

|- bgcolor="#ECECEC" align="left"
| VENSES
| RTE3
| 3.0
| Semantic relation between words
| No evaluation of the resource

|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Subtotal: 25
|}

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

DIRT Paraphrase Collection - RTE Users

2009-12-22T17:32:23Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| Ablation test performed. Positive impact of the resource: +1.33% accuracy on two-way task.

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE5
|
| Verb paraphrases
| Ablation test performed. Negative impact of the resource on two-way task: -1.17% accuracy. Null impact of the resource on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| Use of DIRT relations to map verbs in T with verbs in H
| Ablation test performed. Positive impact of the resource: +0.17% accuracy on two-way, +0.33% on three-way task.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| +0.9% on RTE-4 ablation tests

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE4
| Original DIRT db
| Elaborate T sentence with DIRT-implied entailments
| precision/recall in RTE4: 
boeing run1: 67%/6%; 
boeing run2: 54%/30%

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
| Use of DIRT relations to map verbs in T with verbs in H
| Ablation test performed: +0.7% precision on two-way task.

|- bgcolor="#ECECEC" align="left"
| Uoeltg
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE3
|
| Use of DIRT relations to map verbs in T with verbs in H
| Ablation test performed: +0.37% precision on two-way task.

|- bgcolor="#ECECEC" align="left"
| UIUC
| RTE3
|
| Paired verb/argument patterns
|

|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 9
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

Augmented Wordnet - RTE Users

2009-12-22T16:17:52Z

Celct:

Augmented Wordnet - RTE Users

2009-12-22T15:53:52Z

Celct: New page: {|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1" |- bgcolor="#CDCDCD" ! nowrap="nowrap"|Participants* ! nowrap="nowrap"|Campaign ! nowrap="nowrap"|Version ! nowrap=...

RTE Knowledge Resources

2009-12-22T15:48:48Z

Celct: /* Not available Resources */

Knowledge resources have shown their relevance for applied semantic inference, and are extensively used by applied inference systems, such as those developed within the Textual Entailment framework. 
 
This page presents a list of the knowledge resources used by systems that have participated in the last RTE challenges. The first table lists the publicly available resources, the second one lists unpublished resources. Both tables are sortable by Resource name, type, author and number of users. 
 
RTE Participants are encouraged to add information about all kind of knowledge resources used, from standard existing resources (e.g. WordNet) to knowledge collections created for specific purposes, which can be made available to the community. 
 
__TOC__
 
=== Call for Resources ===
[[RTE6 - Call for Resources]]
 

=== Ablation Tests ===
[[RTE5 - Ablation Tests]]
 

=== Publicly available Resources ===
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="150"|Author
! class="unsortable" width="300"|Brief description
! width="45"|PAST Users <ref name:"rtethree">RTE-3 data have been provided by participants by means of a questionnaire.</ref>
! width="45"|RTE4 Users<ref name:"rtefour">RTE-4 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! width="45"|RTE5 Users<ref name:"rtefive">RTE-5 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [[WordNet]]
| Lexical DB
| Princeton University
| Lexical database of English nouns, verbs, adjectives and adverbs
| style="text-align: center;"|3
| style="text-align: center;"|21
| style="text-align: center;"|17
| [[WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://xwn.hlt.utdallas.edu/index.html eXtended Wordnet]
| Lexical DB
| Human Language Technology Research Institute, University of Texas at Dallas
| Extension of WordNet based on the exploitation of the information contained in WordNet definitional glosses: the glosses are syntactically parsed, transformed into logic forms and content words are semantically disambiguated. The Extended Wordnet is an ongoing project.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|2
| [[eXtended WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://ai.stanford.edu/~rion/swn/ Augmented Wordnet]
| Lexical DB
| Stanford University
| The resource is the result of the application of a learning algorithm for inducing semantic taxonomies from parsed text. The algorithm automatically acquires items of world knowledge, and uses these to produce significantly enhanced versions of WordNet (up to 40,000 synsets more).
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Augmented Wordnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/verbnet.html Verbnet]
| Lexical DB
| University of Colorado Boulder
| Lexicon for English verbs organized into classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class
| style="text-align: center;"|2
| style="text-align: center;"|2
| style="text-align: center;"|1
| [[Verbnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[VerbOcean]]
| Lexical DB
| Information Sciences Institute, University of Southern California
| Broad-coverage semantic network of verbs
| style="text-align: center;"|2
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[VerbOcean - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://framenet.icsi.berkeley.edu/ FrameNet]
| Lexical DB
| ICSI (International Computer Science Institute) - Berkley University
| Lexical resource for English words, based on frame semantics (valences) and supported by corpus evidence
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|2
| [[Framenet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/meyers/NomBank.html NomBank]
| Lexical DB
| New York University
| Lexical resource containing syntactic frames for nouns, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[NomBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/ace.html PropBank]
| Lexical DB
| University of Colorado Boulder
| Lexical resource containing syntactic frames for verbs, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[PropBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/nomlex/index.html Nomlex] Plus
| Lexical DB
| New York University
| Dictionary of English nominalizations: it describes the allowed complements for a nominalization and relates the nominal complements to the arguments of the corresponding verb
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Nomlex Plus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.ualberta.ca/~lindek/downloads.htm Dekang Lin’s Thesaurus]
| Thesaurus
| University of Alberta
| Thesaurus automatically constructed using a parsed corpus, based on distributional similarity scores
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Dekang Lin’s Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://icon.shef.ac.uk/Moby/mthes.html Grady Ward's Moby Thesaurus]
| Thesaurus
| University of Sheffield
| Thesaurus containing 30,260 root words, with 2,520,264 synonyms and related terms. Grady Ward placed this thesaurus in the public domain in 1996.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Grady Ward's Moby Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://en.wikipedia.org/wiki/Roget%27s_Thesaurus Roget's Thesaurus]
| Thesaurus
| Peter Mark Roget (Electronic version distributed by University of Chicago)
| Roget's Thesaurus is a widely-used English thesaurus, created by Dr. Peter Mark Roget in 1805. The original edition had 15,000 words, and each new edition has been larger. The electronic edition ([http://machaut.uchicago.edu/rogets version 1.02]) is made available by University of Chicago.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Roget's Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.wikipedia.org/ Wikipedia]
| Encyclopedia
|
| Free encyclopedia. Used for extraction of lexical-semantic rules (from its more structured parts), named entity recognition, geographical information etc.
| style="text-align: center;"|0
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[Wikipedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.umbel.org/ Umbel]
| Ontology
| Structured Dynamics LLC, Coralville, IA
| UMBEL stands for Upper Mapping and Binding Exchange Layer and is a lightweight ontology structure for relating Web content and data to a standard set of subject concepts
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Umbel - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.mpi-inf.mpg.de/yago-naga/yago/ YAGO]
| Ontology
| Max-Planck Institute for Informatics, Saarbrücken, Germany
| Light-weight and extensible ontology. It contains more than 2 million entities and 20 million facts about these entities. The facts have been automatically extracted from Wikipedia and unified with WordNet.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[YAGO - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://dbpedia.org/About DBpedia]
| Ontology
| Open community project
| DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. The DBpedia knowledge base currently describes more than 2.9 million things in 91 different languages and consists of 479 million pieces of information.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DBpedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[DIRT Paraphrase Collection]]
| Collection of paraphrases
| University of Alberta
| DIRT (Discovery of Inference Rules from Text) is both an algorithm and a resulting knowledge collection. The DIRT knowledge collection is the output of the DIRT algorithm over a 1GB set of newspaper text.
| style="text-align: center;"|1
| style="text-align: center;"|4
| style="text-align: center;"|3
| [[DIRT Paraphrase Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[TEASE]] Collection
| Collection of Entailment Rules
| Bar-Ilan University
| Output of the TEASE algorithm
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Tease Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://badc.nerc.ac.uk/help/abbrevs.html BADC Acronym and Abbreviation List]
| Word List
| BADC (British Atmospheric Data Centre)
| Acronym and Abbreviation List
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[BADC Acronym and Abbreviation List - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.acronym-guide.com/ Acronym Guide]
| Word List
| Acronym-Guide.com
| Acronym and Abbreviation Lists for English, branched in thematic directories
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|3
| [[Acronym Guide - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 Web1T 5-grams]
| Word list
| Linguistic Data Consortium, University of Pennsylvania; Google Inc.
| Data set containing English word n-grams and their observed frequency counts. The n-gram counts were generated from approximately 1 trillion word tokens of text from publicly accessible Web pages
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Web1T - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE3_RTE4_NGD.zip Normalized Google Distance (RTE3&RTE4)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE3&RTE4)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE5_NGD.zip Normalized Google Distance (RTE5)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE5)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://geonames.usgs.gov/index.html GNIS - Geographic Names Information System]
| Gazetteer
| USGS (United States Geological Survey)
| Database containing the Federal and national standard toponyms for USA, associated areas and Antarctica
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[GNIS - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.geonames.org/ Geonames]
| Gazetteer
|
| Database containing eight million geographical names. It is integrating geographical data such as names of places in various languages, elevation, population and others from various sources.
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geonames - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/paraphrase/ Sekine's Paraphrase Database]
| Collection of paraphrases
| Department of Computer Science, New York University
| Data-base created using Sekine's method, NOT cleaned up by human. It includes 19,975 sets of paraphrases with 191,572 phrases.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Sekine's Paraphrase Database - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://research.microsoft.com/research/downloads/Details/607D14D9-20CD-47E3-85BC-A2F65CD28042/Details.aspx Microsoft Research Paraphrase Corpus]
| Collection of paraphrases
| Microsoft Research
| Text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Microsoft Research Paraphrase Corpus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.cornell.edu/~cristian/Without_a_doubt_-_Data.html Downward entailing operators]
| Collection of entailing operators
| Department of Computer Science, Cornell University, Ithaca NY
| System output of an unsupervised algorithm recovering many Downward Entailing operators, like 'doubt'.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Downward entailing operators - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
|[http://cs.biu.ac.il/~shey/WikiRules.html WikiRules!]
| Lexical Reference rule-base
| Bar-Ilan University
| Extraction of lexical reference rules from the text body (first sentence) and from metadata (links, redirects, parentheses) of Wikipedia
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[WikiRules! - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''New resource''
|
|
| ''Participants are encouraged to contribute''
| style="text-align: center;"|
| style="text-align: center;"|
| style="text-align: center;"|
| [[New Resource2 - RTE Users|Users]]

|}
 
 

=== Not available Resources ===
The following table lists the unpublished resources used by RTE participants. Some of them have been developed by Users themselves specifically for RTE. Interested people may turn to authors to obtain further information.
 
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="180"|Author
! class="unsortable" width="500"|Brief description
! width="30"|PAST Users
! width="30"|RTE4 Users
! width="30"|RTE5 Users
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [http://www.parc.com/ PARC] Polarity Lexicon
| Lexical DB
| PARC - Palo Alto Research Center
| Verbs classification with respect to semantic polarity
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Parc Polarity Lexicon - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Gazetteer from [http://trec.nist.gov/ TREC]
| Gazetteer
| NIST - National Institute of Standards and Technology
| Cities and other geographical names
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Gazetteer from TREC - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DFKI Geographic Ontology ''(to be released)''
| Ontology
| DFKI - German Research Center for Artificial Intelligence
| Ontology containing geographic terms and two kinds of relations: the directional ''part-of'' relation, and the ''equal'' relation for synonyms and abbreviations of the same geographic area (e.g ''the United Kingdom'', ''the UK'', ''Great Britain'', etc.)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geographic Ontology - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Geo''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Meronymy entailment rules, based on TREC’s TIPSTER gazetteer.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Geo - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Regex''
| Collection of Entailment rules
| Bar-Ilan University; Tel-Aviv University
| Small set of entailment rules based on regular expressions, intended to address lexical variability involving temporal phrases
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Regex - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Syntactic rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which define parse tree transformations. The rules cover generic syntactic phenomena such as appositions, conjunctions, passive, relative clause, etc. (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Polarity rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which detect predicates whose polarity is negative (e.g. didn't dance) or unknown (e.g. plans to dance). The rules capture diverse phenomena that affect polarity, e.g. verbal negation, modal verbs, conditionals, and certain verbs that induce negative or "unknown" polarity context. The latter were taken mainly from VerbNet. Extends a resource described in (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Polarity rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Lexical-Syntactic rule base
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Extract lexical-syntactic entailment rules for predicates (verbal and nominal), including argument mapping. The resource is based on WordNet, Nomlex-Plus and Unary DIRT (Szpektor and Dagan, Coling 08)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Lexical-Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| OPENU Collection
| Collection of Entailment Rules and Patterns
| Open University
| Collections of rules, patterns etc. for RTE purpose, extracted from Reuter corpus parsed using Minipar.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[OPENU Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Abbr''
| Collection of rules for abbreviation
| Bar-Ilan University; Tel-Aviv University
| 2000 Abbreviation rules, extracted from [http://badc.nerc.ac.uk/help/abbrevs.html BADC] and [http://www.acronym-guide.com/ Acronym Guide]
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Abbr - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Negation_list
| Negation rules
| „Al. I. Cuza“ University, Iasi, Romania
| List of negative terms and words (verbs, adjectives, nouns) affecting modality or factuality of a infinitive verb preceded by the particle "to" (e.g. "believe","necessary", "attempt")
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DLSIUAES Negation_list
| List of negative terms
| University of Alicante
| Basic list of negative terms.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DLSIUAES Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Quantifier_list
| List of quantifiers
| „Al. I. Cuza“ University, Iasi, Romania
| List of quantifiers affecting entailment judgment. The quantifiers are taken from a list which contains expressions like “more than”, “less than”, or words such as “over”, “under”, etc.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Quantifier_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| FBKirst StopWord list
| List of frequent words
| FBK-Irst; University of Trento - Italy
| A list of the 572 most frequent English words.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[FBKirst StopWord list - RTE Users|Users]]

|}
 

==Footnotes==
<references/>

Abbr - RTE Users

2009-12-22T15:47:35Z

Celct:

Regex - RTE Users

2009-12-22T15:46:59Z

Celct:

Regex - RTE Users

2009-12-22T15:44:23Z

Celct:

Abbr - RTE Users

2009-12-22T15:43:19Z

Celct:

Grady Ward's Moby Thesaurus - RTE Users

2009-12-22T15:40:06Z

Celct:

RTE Knowledge Resources

2009-12-22T15:38:21Z

Celct: /* Publicly available Resources */

Knowledge resources have shown their relevance for applied semantic inference, and are extensively used by applied inference systems, such as those developed within the Textual Entailment framework. 
 
This page presents a list of the knowledge resources used by systems that have participated in the last RTE challenges. The first table lists the publicly available resources, the second one lists unpublished resources. Both tables are sortable by Resource name, type, author and number of users. 
 
RTE Participants are encouraged to add information about all kind of knowledge resources used, from standard existing resources (e.g. WordNet) to knowledge collections created for specific purposes, which can be made available to the community. 
 
__TOC__
 
=== Call for Resources ===
[[RTE6 - Call for Resources]]
 

=== Ablation Tests ===
[[RTE5 - Ablation Tests]]
 

=== Publicly available Resources ===
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="150"|Author
! class="unsortable" width="300"|Brief description
! width="45"|PAST Users <ref name:"rtethree">RTE-3 data have been provided by participants by means of a questionnaire.</ref>
! width="45"|RTE4 Users<ref name:"rtefour">RTE-4 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! width="45"|RTE5 Users<ref name:"rtefive">RTE-5 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [[WordNet]]
| Lexical DB
| Princeton University
| Lexical database of English nouns, verbs, adjectives and adverbs
| style="text-align: center;"|3
| style="text-align: center;"|21
| style="text-align: center;"|17
| [[WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://xwn.hlt.utdallas.edu/index.html eXtended Wordnet]
| Lexical DB
| Human Language Technology Research Institute, University of Texas at Dallas
| Extension of WordNet based on the exploitation of the information contained in WordNet definitional glosses: the glosses are syntactically parsed, transformed into logic forms and content words are semantically disambiguated. The Extended Wordnet is an ongoing project.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|2
| [[eXtended WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://ai.stanford.edu/~rion/swn/ Augmented Wordnet]
| Lexical DB
| Stanford University
| The resource is the result of the application of a learning algorithm for inducing semantic taxonomies from parsed text. The algorithm automatically acquires items of world knowledge, and uses these to produce significantly enhanced versions of WordNet (up to 40,000 synsets more).
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Augmented Wordnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/verbnet.html Verbnet]
| Lexical DB
| University of Colorado Boulder
| Lexicon for English verbs organized into classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class
| style="text-align: center;"|2
| style="text-align: center;"|2
| style="text-align: center;"|1
| [[Verbnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[VerbOcean]]
| Lexical DB
| Information Sciences Institute, University of Southern California
| Broad-coverage semantic network of verbs
| style="text-align: center;"|2
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[VerbOcean - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://framenet.icsi.berkeley.edu/ FrameNet]
| Lexical DB
| ICSI (International Computer Science Institute) - Berkley University
| Lexical resource for English words, based on frame semantics (valences) and supported by corpus evidence
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|2
| [[Framenet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/meyers/NomBank.html NomBank]
| Lexical DB
| New York University
| Lexical resource containing syntactic frames for nouns, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[NomBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/ace.html PropBank]
| Lexical DB
| University of Colorado Boulder
| Lexical resource containing syntactic frames for verbs, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[PropBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/nomlex/index.html Nomlex] Plus
| Lexical DB
| New York University
| Dictionary of English nominalizations: it describes the allowed complements for a nominalization and relates the nominal complements to the arguments of the corresponding verb
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Nomlex Plus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.ualberta.ca/~lindek/downloads.htm Dekang Lin’s Thesaurus]
| Thesaurus
| University of Alberta
| Thesaurus automatically constructed using a parsed corpus, based on distributional similarity scores
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Dekang Lin’s Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://icon.shef.ac.uk/Moby/mthes.html Grady Ward's Moby Thesaurus]
| Thesaurus
| University of Sheffield
| Thesaurus containing 30,260 root words, with 2,520,264 synonyms and related terms. Grady Ward placed this thesaurus in the public domain in 1996.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Grady Ward's Moby Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://en.wikipedia.org/wiki/Roget%27s_Thesaurus Roget's Thesaurus]
| Thesaurus
| Peter Mark Roget (Electronic version distributed by University of Chicago)
| Roget's Thesaurus is a widely-used English thesaurus, created by Dr. Peter Mark Roget in 1805. The original edition had 15,000 words, and each new edition has been larger. The electronic edition ([http://machaut.uchicago.edu/rogets version 1.02]) is made available by University of Chicago.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Roget's Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.wikipedia.org/ Wikipedia]
| Encyclopedia
|
| Free encyclopedia. Used for extraction of lexical-semantic rules (from its more structured parts), named entity recognition, geographical information etc.
| style="text-align: center;"|0
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[Wikipedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.umbel.org/ Umbel]
| Ontology
| Structured Dynamics LLC, Coralville, IA
| UMBEL stands for Upper Mapping and Binding Exchange Layer and is a lightweight ontology structure for relating Web content and data to a standard set of subject concepts
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Umbel - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.mpi-inf.mpg.de/yago-naga/yago/ YAGO]
| Ontology
| Max-Planck Institute for Informatics, Saarbrücken, Germany
| Light-weight and extensible ontology. It contains more than 2 million entities and 20 million facts about these entities. The facts have been automatically extracted from Wikipedia and unified with WordNet.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[YAGO - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://dbpedia.org/About DBpedia]
| Ontology
| Open community project
| DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. The DBpedia knowledge base currently describes more than 2.9 million things in 91 different languages and consists of 479 million pieces of information.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DBpedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[DIRT Paraphrase Collection]]
| Collection of paraphrases
| University of Alberta
| DIRT (Discovery of Inference Rules from Text) is both an algorithm and a resulting knowledge collection. The DIRT knowledge collection is the output of the DIRT algorithm over a 1GB set of newspaper text.
| style="text-align: center;"|1
| style="text-align: center;"|4
| style="text-align: center;"|3
| [[DIRT Paraphrase Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[TEASE]] Collection
| Collection of Entailment Rules
| Bar-Ilan University
| Output of the TEASE algorithm
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Tease Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://badc.nerc.ac.uk/help/abbrevs.html BADC Acronym and Abbreviation List]
| Word List
| BADC (British Atmospheric Data Centre)
| Acronym and Abbreviation List
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[BADC Acronym and Abbreviation List - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.acronym-guide.com/ Acronym Guide]
| Word List
| Acronym-Guide.com
| Acronym and Abbreviation Lists for English, branched in thematic directories
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|3
| [[Acronym Guide - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 Web1T 5-grams]
| Word list
| Linguistic Data Consortium, University of Pennsylvania; Google Inc.
| Data set containing English word n-grams and their observed frequency counts. The n-gram counts were generated from approximately 1 trillion word tokens of text from publicly accessible Web pages
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Web1T - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE3_RTE4_NGD.zip Normalized Google Distance (RTE3&RTE4)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE3&RTE4)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE5_NGD.zip Normalized Google Distance (RTE5)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE5)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://geonames.usgs.gov/index.html GNIS - Geographic Names Information System]
| Gazetteer
| USGS (United States Geological Survey)
| Database containing the Federal and national standard toponyms for USA, associated areas and Antarctica
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[GNIS - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.geonames.org/ Geonames]
| Gazetteer
|
| Database containing eight million geographical names. It is integrating geographical data such as names of places in various languages, elevation, population and others from various sources.
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geonames - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/paraphrase/ Sekine's Paraphrase Database]
| Collection of paraphrases
| Department of Computer Science, New York University
| Data-base created using Sekine's method, NOT cleaned up by human. It includes 19,975 sets of paraphrases with 191,572 phrases.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Sekine's Paraphrase Database - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://research.microsoft.com/research/downloads/Details/607D14D9-20CD-47E3-85BC-A2F65CD28042/Details.aspx Microsoft Research Paraphrase Corpus]
| Collection of paraphrases
| Microsoft Research
| Text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Microsoft Research Paraphrase Corpus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.cornell.edu/~cristian/Without_a_doubt_-_Data.html Downward entailing operators]
| Collection of entailing operators
| Department of Computer Science, Cornell University, Ithaca NY
| System output of an unsupervised algorithm recovering many Downward Entailing operators, like 'doubt'.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Downward entailing operators - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
|[http://cs.biu.ac.il/~shey/WikiRules.html WikiRules!]
| Lexical Reference rule-base
| Bar-Ilan University
| Extraction of lexical reference rules from the text body (first sentence) and from metadata (links, redirects, parentheses) of Wikipedia
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[WikiRules! - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''New resource''
|
|
| ''Participants are encouraged to contribute''
| style="text-align: center;"|
| style="text-align: center;"|
| style="text-align: center;"|
| [[New Resource2 - RTE Users|Users]]

|}
 
 

=== Not available Resources ===
The following table lists the unpublished resources used by RTE participants. Some of them have been developed by Users themselves specifically for RTE. Interested people may turn to authors to obtain further information.
 
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="180"|Author
! class="unsortable" width="500"|Brief description
! width="30"|PAST Users
! width="30"|RTE4 Users
! width="30"|RTE5 Users
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [http://www.parc.com/ PARC] Polarity Lexicon
| Lexical DB
| PARC - Palo Alto Research Center
| Verbs classification with respect to semantic polarity
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Parc Polarity Lexicon - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Gazetteer from [http://trec.nist.gov/ TREC]
| Gazetteer
| NIST - National Institute of Standards and Technology
| Cities and other geographical names
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Gazetteer from TREC - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DFKI Geographic Ontology ''(to be released)''
| Ontology
| DFKI - German Research Center for Artificial Intelligence
| Ontology containing geographic terms and two kinds of relations: the directional ''part-of'' relation, and the ''equal'' relation for synonyms and abbreviations of the same geographic area (e.g ''the United Kingdom'', ''the UK'', ''Great Britain'', etc.)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geographic Ontology - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Geo''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Meronymy entailment rules, based on TREC’s TIPSTER gazetteer.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Geo - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Regex''
| Collection of Entailment rules
| Bar-Ilan University; Tel-Aviv University
| Small set of entailment rules based on regular expressions, intended to address lexical variability involving temporal phrases
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Regex - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Syntactic rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which define parse tree transformations. The rules cover generic syntactic phenomena such as appositions, conjunctions, passive, relative clause, etc. (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Polarity rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which detect predicates whose polarity is negative (e.g. didn't dance) or unknown (e.g. plans to dance). The rules capture diverse phenomena that affect polarity, e.g. verbal negation, modal verbs, conditionals, and certain verbs that induce negative or "unknown" polarity context. The latter were taken mainly from VerbNet. Extends a resource described in (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Polarity rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Lexical-Syntactic rule base
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Extract lexical-syntactic entailment rules for predicates (verbal and nominal), including argument mapping. The resource is based on WordNet, Nomlex-Plus and Unary DIRT (Szpektor and Dagan, Coling 08)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Lexical-Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| OPENU Collection
| Collection of Entailment Rules and Patterns
| Open University
| Collections of rules, patterns etc. for RTE purpose, extracted from Reuter corpus parsed using Minipar.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[OPENU Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Abbr''
| Collection of rules for abbreviation
| Bar-Ilan University; Tel-Aviv University
| 2000 Abbreviation rules, extracted from [http://badc.nerc.ac.uk/help/abbrevs.html BADC] and [http://www.acronym-guide.com/ Acronym Guide]
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Abbr - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Negation_list
| Negation rules
| „Al. I. Cuza“ University, Iasi, Romania
| List of negative terms and words (verbs, adjectives, nouns) affecting modality or factuality of a infinitive verb preceded by the particle "to" (e.g. "believe","necessary", "attempt")
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DLSIUAES Negation_list
| List of negative terms
| University of Alicante
| Basic list of negative terms.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DLSIUAES Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Quantifier_list
| List of quantifiers
| „Al. I. Cuza“ University, Iasi, Romania
| List of quantifiers affecting entailment judgment. The quantifiers are taken from a list which contains expressions like “more than”, “less than”, or words such as “over”, “under”, etc.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Quantifier_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| FBKirst StopWord list
| List of frequent words
| FBK-Irst; University of Trento - Italy
| A list of the 572 most frequent English words.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[FBKirst StopWord list - RTE Users|Users]]

|}
 

==Footnotes==
<references/>

RTE Knowledge Resources

2009-12-22T15:37:34Z

Celct: /* Publicly available Resources */

Knowledge resources have shown their relevance for applied semantic inference, and are extensively used by applied inference systems, such as those developed within the Textual Entailment framework. 
 
This page presents a list of the knowledge resources used by systems that have participated in the last RTE challenges. The first table lists the publicly available resources, the second one lists unpublished resources. Both tables are sortable by Resource name, type, author and number of users. 
 
RTE Participants are encouraged to add information about all kind of knowledge resources used, from standard existing resources (e.g. WordNet) to knowledge collections created for specific purposes, which can be made available to the community. 
 
__TOC__
 
=== Call for Resources ===
[[RTE6 - Call for Resources]]
 

=== Ablation Tests ===
[[RTE5 - Ablation Tests]]
 

=== Publicly available Resources ===
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="150"|Author
! class="unsortable" width="300"|Brief description
! width="45"|PAST Users <ref name:"rtethree">RTE-3 data have been provided by participants by means of a questionnaire.</ref>
! width="45"|RTE4 Users<ref name:"rtefour">RTE-4 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! width="45"|RTE5 Users<ref name:"rtefive">RTE-5 data have been provided by participants and have been integrated with information extracted from the related proceedings.</ref>
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [[WordNet]]
| Lexical DB
| Princeton University
| Lexical database of English nouns, verbs, adjectives and adverbs
| style="text-align: center;"|3
| style="text-align: center;"|21
| style="text-align: center;"|17
| [[WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://xwn.hlt.utdallas.edu/index.html eXtended Wordnet]
| Lexical DB
| Human Language Technology Research Institute, University of Texas at Dallas
| Extension of WordNet based on the exploitation of the information contained in WordNet definitional glosses: the glosses are syntactically parsed, transformed into logic forms and content words are semantically disambiguated. The Extended Wordnet is an ongoing project.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|2
| [[eXtended WordNet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://ai.stanford.edu/~rion/swn/ Augmented Wordnet]
| Lexical DB
| Stanford University
| The resource is the result of the application of a learning algorithm for inducing semantic
taxonomies from parsed text. The algorithm automatically acquires items of world knowledge, and uses these to produce significantly enhanced versions of WordNet (up to 40,000 synsets more).
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Augmented Wordnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/verbnet.html Verbnet]
| Lexical DB
| University of Colorado Boulder
| Lexicon for English verbs organized into classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class
| style="text-align: center;"|2
| style="text-align: center;"|2
| style="text-align: center;"|1
| [[Verbnet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[VerbOcean]]
| Lexical DB
| Information Sciences Institute, University of Southern California
| Broad-coverage semantic network of verbs
| style="text-align: center;"|2
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[VerbOcean - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://framenet.icsi.berkeley.edu/ FrameNet]
| Lexical DB
| ICSI (International Computer Science Institute) - Berkley University
| Lexical resource for English words, based on frame semantics (valences) and supported by corpus evidence
| style="text-align: center;"|1
| style="text-align: center;"|1
| style="text-align: center;"|2
| [[Framenet - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/meyers/NomBank.html NomBank]
| Lexical DB
| New York University
| Lexical resource containing syntactic frames for nouns, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[NomBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://verbs.colorado.edu/~mpalmer/projects/ace.html PropBank]
| Lexical DB
| University of Colorado Boulder
| Lexical resource containing syntactic frames for verbs, extracted from annotated corpora
| style="text-align: center;"|2
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[PropBank Resource - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/nomlex/index.html Nomlex] Plus
| Lexical DB
| New York University
| Dictionary of English nominalizations: it describes the allowed complements for a nominalization and relates the nominal complements to the arguments of the corresponding verb
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Nomlex Plus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.ualberta.ca/~lindek/downloads.htm Dekang Lin’s Thesaurus]
| Thesaurus
| University of Alberta
| Thesaurus automatically constructed using a parsed corpus, based on distributional similarity scores
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Dekang Lin’s Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://icon.shef.ac.uk/Moby/mthes.html Grady Ward's Moby Thesaurus]
| Thesaurus
| University of Sheffield
| Thesaurus containing 30,260 root words, with 2,520,264 synonyms and related terms. Grady Ward placed this thesaurus in the public domain in 1996.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Grady Ward's Moby Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://en.wikipedia.org/wiki/Roget%27s_Thesaurus Roget's Thesaurus]
| Thesaurus
| Peter Mark Roget (Electronic version distributed by University of Chicago)
| Roget's Thesaurus is a widely-used English thesaurus, created by Dr. Peter Mark Roget in 1805. The original edition had 15,000 words, and each new edition has been larger. The electronic edition ([http://machaut.uchicago.edu/rogets version 1.02]) is made available by University of Chicago.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Roget's Thesaurus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.wikipedia.org/ Wikipedia]
| Encyclopedia
|
| Free encyclopedia. Used for extraction of lexical-semantic rules (from its more structured parts), named entity recognition, geographical information etc.
| style="text-align: center;"|0
| style="text-align: center;"|3
| style="text-align: center;"|6
| [[Wikipedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.umbel.org/ Umbel]
| Ontology
| Structured Dynamics LLC, Coralville, IA
| UMBEL stands for Upper Mapping and Binding Exchange Layer and is a lightweight ontology structure for relating Web content and data to a standard set of subject concepts
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Umbel - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.mpi-inf.mpg.de/yago-naga/yago/ YAGO]
| Ontology
| Max-Planck Institute for Informatics, Saarbrücken, Germany
| Light-weight and extensible ontology. It contains more than 2 million entities and 20 million facts about these entities. The facts have been automatically extracted from Wikipedia and unified with WordNet.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[YAGO - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://dbpedia.org/About DBpedia]
| Ontology
| Open community project
| DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. The DBpedia knowledge base currently describes more than 2.9 million things in 91 different languages and consists of 479 million pieces of information.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DBpedia - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[DIRT Paraphrase Collection]]
| Collection of paraphrases
| University of Alberta
| DIRT (Discovery of Inference Rules from Text) is both an algorithm and a resulting knowledge collection. The DIRT knowledge collection is the output of the DIRT algorithm over a 1GB set of newspaper text.
| style="text-align: center;"|1
| style="text-align: center;"|4
| style="text-align: center;"|3
| [[DIRT Paraphrase Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [[TEASE]] Collection
| Collection of Entailment Rules
| Bar-Ilan University
| Output of the TEASE algorithm
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Tease Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://badc.nerc.ac.uk/help/abbrevs.html BADC Acronym and Abbreviation List]
| Word List
| BADC (British Atmospheric Data Centre)
| Acronym and Abbreviation List
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[BADC Acronym and Abbreviation List - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.acronym-guide.com/ Acronym Guide]
| Word List
| Acronym-Guide.com
| Acronym and Abbreviation Lists for English, branched in thematic directories
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|3
| [[Acronym Guide - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 Web1T 5-grams]
| Word list
| Linguistic Data Consortium, University of Pennsylvania; Google Inc.
| Data set containing English word n-grams and their observed frequency counts. The n-gram counts were generated from approximately 1 trillion word tokens of text from publicly accessible Web pages
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Web1T - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE3_RTE4_NGD.zip Normalized Google Distance (RTE3&RTE4)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE3&RTE4)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.coli.uni-saarland.de/%7Erwang/resources/RTE5_NGD.zip Normalized Google Distance (RTE5)]
| Word Pair Co-occurrence
| Saarland University
| Co-occurrence of the word pairs in RTE3 and RTE4 using Normalized Google Distance (Cilibrasi and Vitanyi, 2004). The word pairs are all the possible combinations of content words in T and H. In practice, we used [http://m.www.yahoo.com/ Yahoo!] as the search engine.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Normalized Google Distance (RTE5)- RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://geonames.usgs.gov/index.html GNIS - Geographic Names Information System]
| Gazetteer
| USGS (United States Geological Survey)
| Database containing the Federal and national standard toponyms for USA, associated areas and Antarctica
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[GNIS - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.geonames.org/ Geonames]
| Gazetteer
|
| Database containing eight million geographical names. It is integrating geographical data such as names of places in various languages, elevation, population and others from various sources.
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geonames - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://nlp.cs.nyu.edu/paraphrase/ Sekine's Paraphrase Database]
| Collection of paraphrases
| Department of Computer Science, New York University
| Data-base created using Sekine's method, NOT cleaned up by human. It includes 19,975 sets of paraphrases with 191,572 phrases.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Sekine's Paraphrase Database - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://research.microsoft.com/research/downloads/Details/607D14D9-20CD-47E3-85BC-A2F65CD28042/Details.aspx Microsoft Research Paraphrase Corpus]
| Collection of paraphrases
| Microsoft Research
| Text file containing 5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Microsoft Research Paraphrase Corpus - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| [http://www.cs.cornell.edu/~cristian/Without_a_doubt_-_Data.html Downward entailing operators]
| Collection of entailing operators
| Department of Computer Science, Cornell University, Ithaca NY
| System output of an unsupervised algorithm recovering many Downward Entailing operators, like 'doubt'.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Downward entailing operators - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
|[http://cs.biu.ac.il/~shey/WikiRules.html WikiRules!]
| Lexical Reference rule-base
| Bar-Ilan University
| Extraction of lexical reference rules from the text body (first sentence) and from metadata (links, redirects, parentheses) of Wikipedia
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[WikiRules! - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''New resource''
|
|
| ''Participants are encouraged to contribute''
| style="text-align: center;"|
| style="text-align: center;"|
| style="text-align: center;"|
| [[New Resource2 - RTE Users|Users]]

|}
 
 

=== Not available Resources ===
The following table lists the unpublished resources used by RTE participants. Some of them have been developed by Users themselves specifically for RTE. Interested people may turn to authors to obtain further information.
 
{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"

|- bgcolor="#CDCDCD"
! width="80"|Resource
! width="50"|Type
! width="180"|Author
! class="unsortable" width="500"|Brief description
! width="30"|PAST Users
! width="30"|RTE4 Users
! width="30"|RTE5 Users
! class="unsortable" width="30"|Usage info

|- bgcolor="#ECECEC" "align="left"
| [http://www.parc.com/ PARC] Polarity Lexicon
| Lexical DB
| PARC - Palo Alto Research Center
| Verbs classification with respect to semantic polarity
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Parc Polarity Lexicon - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Gazetteer from [http://trec.nist.gov/ TREC]
| Gazetteer
| NIST - National Institute of Standards and Technology
| Cities and other geographical names
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Gazetteer from TREC - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DFKI Geographic Ontology ''(to be released)''
| Ontology
| DFKI - German Research Center for Artificial Intelligence
| Ontology containing geographic terms and two kinds of relations: the directional ''part-of'' relation, and the ''equal'' relation for synonyms and abbreviations of the same geographic area (e.g ''the United Kingdom'', ''the UK'', ''Great Britain'', etc.)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Geographic Ontology - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Geo''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Meronymy entailment rules, based on TREC’s TIPSTER gazetteer.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[Geo - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Regex''
| Collection of Entailment rules
| Bar-Ilan University; Tel-Aviv University
| Small set of entailment rules based on regular expressions, intended to address lexical variability involving temporal phrases
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Regex - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Syntactic rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which define parse tree transformations. The rules cover generic syntactic phenomena such as appositions, conjunctions, passive, relative clause, etc. (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|1
| [[Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Polarity rule base ''(to be released)''
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| A manually-composed collection of entailment rules which detect predicates whose polarity is negative (e.g. didn't dance) or unknown (e.g. plans to dance). The rules capture diverse phenomena that affect polarity, e.g. verbal negation, modal verbs, conditionals, and certain verbs that induce negative or "unknown" polarity context. The latter were taken mainly from VerbNet. Extends a resource described in (Bar-Haim et al., AAAI-07)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Polarity rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| Lexical-Syntactic rule base
| Collection of Entailment Rules
| Bar-Ilan University; Tel-Aviv University
| Extract lexical-syntactic entailment rules for predicates (verbal and nominal), including argument mapping. The resource is based on WordNet, Nomlex-Plus and Unary DIRT (Szpektor and Dagan, Coling 08)
| style="text-align: center;"|0
| style="text-align: center;"|1
| style="text-align: center;"|0
| [[Lexical-Syntactic rule base - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| OPENU Collection
| Collection of Entailment Rules and Patterns
| Open University
| Collections of rules, patterns etc. for RTE purpose, extracted from Reuter corpus parsed using Minipar.
| style="text-align: center;"|1
| style="text-align: center;"|0
| style="text-align: center;"|0
| [[OPENU Collection - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| ''Abbr''
| Collection of rules for abbreviation
| Bar-Ilan University; Tel-Aviv University
| 2000 Abbreviation rules, extracted from [http://badc.nerc.ac.uk/help/abbrevs.html BADC] and [http://www.acronym-guide.com/ Acronym Guide]
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[Abbr - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Negation_list
| Negation rules
| „Al. I. Cuza“ University, Iasi, Romania
| List of negative terms and words (verbs, adjectives, nouns) affecting modality or factuality of a infinitive verb preceded by the particle "to" (e.g. "believe","necessary", "attempt")
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| DLSIUAES Negation_list
| List of negative terms
| University of Alicante
| Basic list of negative terms.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[DLSIUAES Negation_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| UAIC Quantifier_list
| List of quantifiers
| „Al. I. Cuza“ University, Iasi, Romania
| List of quantifiers affecting entailment judgment. The quantifiers are taken from a list which contains expressions like “more than”, “less than”, or words such as “over”, “under”, etc.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[UAIC Quantifier_list - RTE Users|Users]]

|- bgcolor="#ECECEC" "align="left"
| FBKirst StopWord list
| List of frequent words
| FBK-Irst; University of Trento - Italy
| A list of the 572 most frequent English words.
| style="text-align: center;"|0
| style="text-align: center;"|0
| style="text-align: center;"|1
| [[FBKirst StopWord list - RTE Users|Users]]

|}
 

==Footnotes==
<references/>

DIRT Paraphrase Collection - RTE Users

2009-12-22T14:24:15Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| Ablation test performed. Positive impact of the resource: +1.33% accuracy on two-way task.

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE5
|
| Verb paraphrases
| Ablation test performed. Negative impact of the resource on two-way task: -1.17% accuracy. Null impact of the resource on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| Use of DIRT relations to map verbs in T with verbs in H
| Ablation test performed. Positive impact of the resource: +0.17% accuracy on two-way, +0.33% on three-way task.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| +0.9% on RTE-4 ablation tests

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE4
| Original DIRT db
| Elaborate T sentence with DIRT-implied entailments
| precision/recall in RTE4: 
boeing run1: 67%/6%; 
boeing run2: 54%/30%

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| Uoeltg
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UIUC
| RTE3
|
| Paired verb/argument patterns
|

|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 8
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

UAIC Negation list - RTE Users

2009-12-22T12:30:43Z

Celct:

DLSIUAES Negation list - RTE Users

2009-12-22T12:30:22Z

Celct:

UAIC Quantifier list - RTE Users

2009-12-22T12:30:11Z

Celct:

FBKirst StopWord list - RTE Users

2009-12-22T12:29:59Z

Celct:

Polarity rule base - RTE Users

2009-12-22T12:29:21Z

Celct:

Syntactic rule base - RTE Users

2009-12-22T12:29:05Z

Celct:

Syntactic rule base - RTE Users

2009-12-22T12:28:53Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1" style="margin-left: 20px;"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
|
| No ablation test performed.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
|
| +0.4% on RTE-4 ablation tests.

|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 1
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

Polarity rule base - RTE Users

2009-12-22T12:28:43Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1" style="margin-left: 20px;"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments
|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
|
| +0.9% on RTE-4 ablation tests.
|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 1
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

Lexical-Syntactic rule base - RTE Users

2009-12-22T12:28:32Z

Celct:

OPENU Collection - RTE Users

2009-12-22T12:28:20Z

Celct:

Abbr - RTE Users

2009-12-22T12:28:05Z

Celct:

Parc Polarity Lexicon - RTE Users

2009-12-22T12:27:39Z

Celct:

Gazetteer from TREC - RTE Users

2009-12-22T12:27:16Z

Celct:

Geographic Ontology - RTE Users

2009-12-22T12:27:02Z

Celct:

Geo - RTE Users

2009-12-22T12:26:42Z

Celct:

Downward entailing operators - RTE Users

2009-12-22T12:24:19Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| Rhodes
| RTE5
|
| Negation detection
| No ablation test performed

|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 0
|}
 
Return to [[RTE Knowledge Resources]]

WikiRules! - RTE Users

2009-12-22T12:24:10Z

Celct:

Normalized Google Distance (RTE5)- RTE Users

2009-12-22T12:23:38Z

Celct:

GNIS - RTE Users

2009-12-22T12:23:30Z

Celct:

Geonames - RTE Users

2009-12-22T12:23:20Z

Celct:

BADC Acronym and Abbreviation List - RTE Users

2009-12-22T12:22:50Z

Celct:

Acronym Guide - RTE Users

2009-12-22T12:22:38Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| Used in the search task, together with [[BADC_Acronym_and_Abbreviation_List_-_RTE_Users|BADC]], to create the new resource [[RTE_Knowledge_Resources#Not_available_Resources|Abbr]]
| No ablation test performed

|- bgcolor="#ECECEC" align="left"
| Siel_09
| RTE5
|
| The acronyms are expanded using the acronym database, so the acronyms are also matched with the expanded acronyms, and entailment is predicted accordingly. Ex: UK United Kingdom
| Ablation test performed. Null impact of the resource both on two-way and on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| Acronym resolution. The resource has been expanded by a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form
| Ablation test performed. Positive impact of the acronym module: +0.17% on two-way, +0.16% on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 4
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

Web1T - RTE Users

2009-12-22T12:22:29Z

Celct:

Normalized Google Distance (RTE3&RTE4)- RTE Users

2009-12-22T12:22:20Z

Celct:

Umbel - RTE Users

2009-12-22T12:21:55Z

Celct:

YAGO - RTE Users

2009-12-22T12:21:47Z

Celct:

DBpedia - RTE Users

2009-12-22T12:21:37Z

Celct:

DIRT Paraphrase Collection - RTE Users

2009-12-22T12:21:28Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| Ablation test performed. Positive impact of the resource: +1.33% accuracy on two-way task.

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE5
|
| Verb paraphrases
| Ablation test performed. Negative impact of the resource on two-way task: -1.17% accuracy. Null impact of the resource on three-way task.

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| Use of DIRT relations to map verbs in T with verbs in H
| Ablation test performed. Positive impact of the resource: +0.17% accuracy on two-way, +0.33% on three-way task.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
| We used the canonical DIRT rulebase version of Szpektor and Dagan (RANLP 2007), and considered top 25 rules.
| +0.9% on RTE-4 ablation tests

|- bgcolor="#ECECEC" align="left"
| Boeing
| RTE4
| Original DIRT db
| Elaborate T sentence with DIRT-implied entailments
| precision/recall in RTE4: 
boeing run1: 67%/6%; 
boeing run2: 54%/30%

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| Uoeltg
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''

|- bgcolor="#ECECEC" align="left"
| UIUC
| RTE3
|
| Paired verb/argument patterns
|

|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''

|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 8
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]

Tease Collection - RTE Users

2009-12-22T12:21:19Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments
|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 0
|}
 

Return to [[RTE Knowledge Resources]]

Wikipedia - RTE Users

2009-12-22T12:20:53Z

Celct:

{|class="wikitable sortable" cellpadding="3" cellspacing="0" border="1"
|- bgcolor="#CDCDCD"
! nowrap="nowrap"|Participants*
! nowrap="nowrap"|Campaign
! nowrap="nowrap"|Version
! nowrap="nowrap"|Specific usage description
! nowrap="nowrap"|Evaluations / Comments

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE5
|
| Used for the creation of WikiRules!, a resource of lexical reference rules
| Cfr. evaluation of [[WikiRules! - RTE Users|WikiRules!]]

|- bgcolor="#ECECEC" align="left"
| Cswhu
| RTE5
|
| Named entity relations extraction
| Ablation test performed. Positive impact of the resosurce: +1.33% on two-way, +3.34% on three-way task

|- bgcolor="#ECECEC" align="left"
| FBKirst
| RTE5
|
| Rules extracted from WP using Latent Semantic Analysis (LSA)
| Ablation test performed. Positive impact of the resosurce: +1% on two-way task

|- bgcolor="#ECECEC" align="left"
| PeMoZa
| RTE5
| 17 October 2009
| Similarity measure based on WikiPedia, using jLSA (java Latent Semantic Analysis)
| No ablation test performed. 
It is worth mentioning that the similarity measure over Wikipedia, not only covers all the existing pairs, but its computation is also about 100 times faster than the previously used tool based on WordNet

|- bgcolor="#ECECEC" align="left"
| QUANTA
| RTE5
|
| Wikipedia redirection set is employed as a synonymous entity set to detect the entity entailment.
| No ablation test performed. 

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE5
|
| FIRST USE: Extraction of a new resource based on named entity relation mining using Wikipedia (Iftene and Balahur-Dobrescu, 2008) 
SECOND USE: Extraction of a new resource aimed to identify a "distance" between one name entity from hypothesis and name entities from text
| FIRST USE: Ablation test performed. Positive impact of the resource: +0.17% on two-way, +0.50% on three-way task. 
SECOND USE: Ablation test performed (Wikipedia+NER's+Perl patterns). Positive impact of the whole NE module: +6.17% on two-way, +5% on three-way task.

|- bgcolor="#ECECEC" align="left"
| BIU
| RTE4
|
| Used for the creation of WikiRules!, a resource of lexical reference rules
| Cfr. evaluation of [[WikiRules! - RTE Users|WikiRules!]]

|- bgcolor="#ECECEC" align="left"
| UAIC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
|- bgcolor="#ECECEC" align="left"
| UPC
| RTE4
|
|
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
|- bgcolor="#ECECEC" align="left"
| ''New user''
|
|
|
| ''Participants are encouraged to contribute.''
|}
{|class="wikitable" cellpadding="3" cellspacing="0" border="0" style="margin-left: 20px;"
|-
! align="left"|Total: 9
|}
 

[*] For further information about participants, click here: [[RTE Challenges - Data about participants]]

Return to [[RTE Knowledge Resources]]