RTE5 - Ablation Tests

Ablated Resource	Team Run	Relative accuracy - 2way	Relative accuracy - 3way	Resource Usage Description
Acronym guide	Siel_093.3way	0	0	Acronym Resolution
Acronym guide + UAIC_Acronym_rules	UAIC20091.3way	0.0017	0.0016	We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.
DIRT	BIU1.2way	0.0133		Inference rules
DIRT	Boeing3.3way	-0.0117	0
DIRT	UAIC20091.3way	0.0017	0.0033	We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H
Framenet	DLSIUAES1.2way	0.0116		frame-to-frame similarity metric
Framenet	DLSIUAES1.3way	-0.0017	-0.0017	frame-to-frame similarity metric
Framenet	UB.dmirg3.2way	0
Grady Ward’s MOBY Thesaurus + Roget's Thesaurus	VensesTeam2.2way	0.0283		Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas
MontyLingua Tool	Siel_093.3way	0	0	For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form
NEGATION_rules by UAIC	UAIC20091.3way	0	-0.0134	Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.
NER	UI_ccg1.2way	0.0483		Named Entity recognition/comparison
PropBank	cswhu1.3way	0.0200	0.0317	syntactic and semantic parsing
Stanford NER	QUANTA1.2way	0.0067		We use Named Entity similarity as a feature
Stopword list	FBKirst1.2way	0.0150	-0.1028
Training data from RTE1, 2, 3	PeMoZa3.2way	0
Training data from RTE1, 2, 3	PeMoZa3.2way	0
Training data from RTE2	PeMoZa3.2way	0.0066
Training data from RTE2, 3	PeMoZa3.2way	0
VerbOcean	DFKI1.3way	0	0.0017
VerbOcean	DFKI2.3way	0.0033	0.0050
VerbOcean	DFKI3.3way	0.0017	0.0017
VerbOcean	FBKirst1.2way	-0.0016	-0.1028	Rules extracted from VerbOcean
VerbOcean	QUANTA1.2way	0		We use "opposite-of" relation in VerbOcean as a feature
VerbOcean	Siel_093.3way	0	0	Similarity/anthonymy/unrelatedness between verbs
WikiPedia	BIU1.2way	-0.0100		Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations
WikiPedia	cswhu1.3way	0.0133	0.0334	Lexical semantic rules
WikiPedia	FBKirst1.2way	0.0100		Rules extracted from WP using Latent Semantic Analysis (LSA)
WikiPedia	UAIC20091.3way	0.0117	0.0150	Relations between named entities
Wikipedia + NER's (LingPipe, GATE) + Perl patterns	UAIC20091.3way	0.0617	0.0500	NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text
WordNet	AUEBNLP1.3way	-0.0200	-0.0267	Synonyms
WordNet	BIU1.2way	0.0250		Synonyms, hyponyms (2 levels away from the original term), hyponym_instance and derivations
WordNet	Boeing3.3way	0.0400	0.0567
WordNet	DFKI1.3way	-0.0017	0
WordNet	DFKI2.3way	0.0016	0.0034
WordNet	DFKI3.3way	0.0017	0.0017
WordNet	DLSIUAES1.2way	0.0083		Similarity between lemmata, computed by WordNet-based metrics
WordNet	DLSIUAES1.3way	-0.0050	-0.0033	Similarity between lemmata, computed by WordNet-based metrics
WordNet	JU_CSE_TAC1.2way	0.0034		WordNet based Unigram match
WordNet	PeMoZa1.2way	-0.0050		Derivational Morphology from WordNet
WordNet	PeMoZa1.2way	0.0133		Verb Entailment from Wordnet
WordNet	PeMoZa2.2way	0.0100		Derivational Morphology from WordNet
WordNet	PeMoZa2.2way	-0.0033		Verb Entailment from Wordnet
WordNet	QUANTA1.2way	-0.0017		We use several relations from wordnet, such as synonyms, hyponym, hypernym et al.
WordNet	Sagan1.3way	0	-0.0083	The system is based on machine learning approach. The ablation test was obtained with 2 less features using WordNet in the training and testing steps.
WordNet	Siel_093.3way	0.0034	-0.0017	Similarity between nouns using WN tool
WordNet	ssl1.3way	0	0.0067	WordNet Analysis
WordNet	UB.dmirg3.2way	0
WordNet	UI_ccg1.2way	0.0400		word similarity == identity
WordNet + FrameNet	UB.dmirg3.2way	0
WordNet + VerbOcean	DFKI1.3way	0	0.0017
WordNet + VerbOcean	DFKI2.3way	0.0050	0.0067
WordNet + VerbOcean	DFKI3.3way	0.0017	0.0017
WordNet + VerbOcean	UAIC20091.3way	0.0200	0.0150	Contradiction identification
WordNet + VerbOcean + DLSIUAES_negation_list	DLSIUAES1.2way	0.0066		Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by ourselves)
WordNet + VerbOcean + DLSIUAES_negation_list	DLSIUAES1.3way	-0.0100	-0.0050	Antonym relations between verbs (VO+WN); polarity based on negation terms (short list constructed by ourselves)
WordNet + XWordNet	UAIC20091.3way	0.0100	0.0133	Synonymy, hyponymy and hypernymy and eXtended WordNet relation