RTE5 - Ablation Tests
Jump to navigation
Jump to search
Ablated Resource | Team Run | Relative accuracy - 2way | Relative accuracy - 3way | Resource Usage Description |
---|---|---|---|---|
Acronym guide | Siel_093.3way | 0 | 0 | Acronym Resolution |
Acronym guide + UAIC_Acronym_rules |
UAIC20091.3way | 0.0017 | 0.0016 | We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form. |
DIRT | BIU1.2way | 0.0133 | Inference rules | |
DIRT | Boeing3.3way | -0.0117 | 0 | |
DIRT | UAIC20091.3way | 0.0017 | 0.0033 | We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H |
Framenet | DLSIUAES1.2way | 0.0116 | frame-to-frame similarity metric | |
Framenet | DLSIUAES1.3way | -0.0017 | -0.0017 | frame-to-frame similarity metric |
Framenet | UB.dmirg3.2way | 0 | ||
Grady Ward’s MOBY Thesaurus + Roget's Thesaurus |
VensesTeam2.2way | 0.0283 | Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas | |
MontyLingua Tool | Siel_093.3way | 0 | 0 | For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form |
NEGATION_rules by UAIC | UAIC20091.3way | 0 | -0.0134 | Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found. |
NER | UI_ccg1.2way | 0.0483 | Named Entity recognition/comparison | |
PropBank | cswhu1.3way | 0.0200 | 0.0317 | syntactic and semantic parsing |
Stanford NER | QUANTA1.2way | 0.0067 | We use Named Entity similarity as a feature | |
Stopword list | FBKirst1.2way | 0.0150 | -0.1028 | |
Training data from RTE1, 2, 3 | PeMoZa3.2way | 0 | ||
Training data from RTE1, 2, 3 | PeMoZa3.2way | 0 | ||
Training data from RTE2 | PeMoZa3.2way | 0.0066 | ||
Training data from RTE2, 3 | PeMoZa3.2way | 0 | ||
VerbOcean | DFKI1.3way | 0 | 0.0017 | |
VerbOcean | DFKI2.3way | 0.0033 | 0.0050 | |
VerbOcean | DFKI3.3way | 0.0017 | 0.0017 | |
VerbOcean | FBKirst1.2way | -0.0016 | -0.1028 | Rules extracted from VerbOcean |
VerbOcean | QUANTA1.2way | 0 | We use "opposite-of" relation in VerbOcean as a feature | |
VerbOcean | Siel_093.3way | 0 | 0 | Similarity/anthonymy/unrelatedness between verbs |
WikiPedia | BIU1.2way | -0.0100 | Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations | |
WikiPedia | cswhu1.3way | 0.0133 | 0.0334 | Lexical semantic rules |
WikiPedia | FBKirst1.2way | 0.0100 | Rules extracted from WP using Latent Semantic Analysis (LSA) | |
WikiPedia | UAIC20091.3way | 0.0117 | 0.0150 | Relations between named entities |
Wikipedia + NER's (LingPipe, GATE) + Perl patterns |
UAIC20091.3way | 0.0617 | 0.0500 | NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text |