RTE5 - Ablation Tests

From ACL Wiki
Revision as of 09:00, 24 November 2009 by Celct (talk | contribs)
Jump to navigation Jump to search
Ablated Resource Team Run Relative accuracy - 2way Relative accuracy - 3way Resource Usage Description
Acronym guide Siel_093.3way 0 0 Acronym Resolution
Acronym guide +
UAIC_Acronym_rules
UAIC20091.3way 0.0017 0.0016 We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.
DIRT BIU1.2way 0.0133 Inference rules
DIRT Boeing3.3way -0.0117 0
DIRT UAIC20091.3way 0.0017 0.0033 We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H
Framenet DLSIUAES1.2way 0.0116 frame-to-frame similarity metric
Framenet DLSIUAES1.3way -0.0017 -0.0017 frame-to-frame similarity metric
Framenet UB.dmirg3.2way 0
Grady Ward’s MOBY Thesaurus +
Roget's Thesaurus
VensesTeam2.2way 0.0283 Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas
MontyLingua Tool Siel_093.3way 0 0 For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form
NEGATION_rules by UAIC UAIC20091.3way 0 -0.0134 Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.
NER UI_ccg1.2way 0.0483 Named Entity recognition/comparison
PropBank cswhu1.3way 0.0200 0.0317 syntactic and semantic parsing
Stanford NER QUANTA1.2way 0.0067 We use Named Entity similarity as a feature
Stopword list FBKirst1.2way 0.0150 -0.1028
Training data from RTE1, 2, 3 PeMoZa3.2way 0
Training data from RTE1, 2, 3 PeMoZa3.2way 0
Training data from RTE2 PeMoZa3.2way 0.0066
Training data from RTE2, 3 PeMoZa3.2way 0
VerbOcean DFKI1.3way 0 0.0017
VerbOcean DFKI2.3way 0.0033 0.0050
VerbOcean DFKI3.3way 0.0017 0.0017
VerbOcean FBKirst1.2way -0.0016 -0.1028 Rules extracted from VerbOcean
VerbOcean QUANTA1.2way 0 We use "opposite-of" relation in VerbOcean as a feature
VerbOcean Siel_093.3way 0 0 Similarity/anthonymy/unrelatedness between verbs
WikiPedia BIU1.2way -0.0100 Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations
WikiPedia cswhu1.3way 0.0133 0.0334 Lexical semantic rules
WikiPedia FBKirst1.2way 0.0100 Rules extracted from WP using Latent Semantic Analysis (LSA)
WikiPedia UAIC20091.3way 0.0117 0.0150 Relations between named entities
Wikipedia +
NER's (LingPipe, GATE) +
Perl patterns
UAIC20091.3way 0.0617 0.0500 NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text