RTE7 - Ablation Tests - Revision history

Amarchetti at 14:40, 27 March 2012

2012-03-27T14:40:07Z

← Older revision		Revision as of 14:40, 27 March 2012
Line 46:		Line 46:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| WordNet		\| WordNet
	\| ~~DFKI_abl~~-1		\| DFKI1_abl-1
	\| style="text-align: center;"\| -0.14		\| style="text-align: center;"\| -0.14
	\| style="text-align: center;"\| Features based on WordNet similarity measures (JWNL).		\| style="text-align: center;"\| Features based on WordNet similarity measures (JWNL).
Line 52:		Line 52:
	\|- bgcolor="#FFFFF" "align="left"		\|- bgcolor="#FFFFF" "align="left"
	\| Named Entity Recognition		\| Named Entity Recognition
	\| ~~DFKI_abl~~-2		\| DFKI1_abl-2
	\| style="text-align: center;"\| 2.08		\| style="text-align: center;"\| 2.08
	\| style="text-align: center;"\| Features based on WordNet similarity measures (JWNL).		\| style="text-align: center;"\| Features based on WordNet similarity measures (JWNL).
Line 130:		Line 130:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| WordNet		\| WordNet
	\| u_tokyo2_abl-2		\| u_tokyo2_abl-1
	\| style="text-align: center;"\| 0.64		\| style="text-align: center;"\| 0.64
	\| style="text-align: center;"\| Ablated resource is WordNet		\| style="text-align: center;"\| Ablated resource is WordNet
Line 136:		Line 136:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| WordNet		\| WordNet
	\| ~~u_tokyo2_abl~~-3		\| u_tokyo3_abl-1
	\| style="text-align: center;"\| 0.99		\| style="text-align: center;"\| 0.99
	\| style="text-align: center;"\| Ablated resource is WordNet		\| style="text-align: center;"\| Ablated resource is WordNet

Amarchetti at 14:31, 27 March 2012

2012-03-27T14:31:13Z

← Older revision		Revision as of 14:31, 27 March 2012
Line 28:		Line 28:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\|Direct		\|Direct
	\| ~~BIU1_abl~~-2		\| BIU2_abl-2
	\| style="text-align: center;"\| 0.94		\| style="text-align: center;"\| 0.94
	\| style="text-align: center;"\| Without Bap (AKA "Direct"), which is used as a lexical rulebase resource		\| style="text-align: center;"\| Without Bap (AKA "Direct"), which is used as a lexical rulebase resource
Line 34:		Line 34:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| Wikipedia		\| Wikipedia
	\| ~~BIU1_abl~~-3		\| BIU2_abl-3
	\| style="text-align: center;"\| 1.56		\| style="text-align: center;"\| 1.56
	\| style="text-align: center;"\| Without Wikipedia, which is used as a lexical rulebase resource		\| style="text-align: center;"\| Without Wikipedia, which is used as a lexical rulebase resource

Amarchetti at 14:30, 27 March 2012

2012-03-27T14:30:46Z

← Older revision		Revision as of 14:30, 27 March 2012
Line 40:		Line 40:
	\|- bgcolor="#FFFFF" "align="left"		\|- bgcolor="#FFFFF" "align="left"
	\| Coreference resolver		\| Coreference resolver
	\| ~~BIU1_abl~~-4		\| BIU2_abl-4
	\| style="text-align: center;"\| 0.69		\| style="text-align: center;"\| 0.69
	\| style="text-align: center;"\| Without any coreference resolution engine, instead of sing ArkRef to obtain coref information from the text, when preprocessing it		\| style="text-align: center;"\| Without any coreference resolution engine, instead of sing ArkRef to obtain coref information from the text, when preprocessing it

Amarchetti at 14:29, 27 March 2012

2012-03-27T14:29:01Z

← Older revision		Revision as of 14:29, 27 March 2012
Line 141:		Line 141:

	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| ~~WordNet~~		\| UAIC Knowledge Resource
	\| UAIC20112_abl-1		\| UAIC20112_abl-1
	\| style="text-align: center;"\| 0		\| style="text-align: center;"\| 0

Amarchetti at 14:27, 27 March 2012

2012-03-27T14:27:21Z

← Older revision		Revision as of 14:27, 27 March 2012
Line 119:		Line 119:
	\| VerbOcean		\| VerbOcean
	\| SJTU_CIT1_abl-2		\| SJTU_CIT1_abl-2
	\| style="text-align: center;"\| 8.89		\| style="text-align: center;"\| 5.93
	\| style="text-align: center;"\| we removed verbocern resource		\| style="text-align: center;"\| we removed verbocern resource

Dagiampi at 14:00, 27 March 2012

2012-03-27T14:00:36Z

Show changes

Dagiampi at 13:37, 27 March 2012

2012-03-27T13:37:24Z

← Older revision		Revision as of 13:37, 27 March 2012
Line 38:		Line 38:
	\| style="text-align: center;"\| Without Wikipedia, which is used as a lexical rulebase resource		\| style="text-align: center;"\| Without Wikipedia, which is used as a lexical rulebase resource

			\|- bgcolor="#FFFFF" "align="left"
			\| Coreference resolver
			\| BIU1_abl-4
			\| style="text-align: center;"\| 0.69
			\| style="text-align: center;"\| Without any coreference resolution engine, instead of sing ArkRef to obtain coref information from the text, when preprocessing it

	~~**********************************~~		???????????????????????

	\|- bgcolor="#FFFFF" "align="left"		\|- bgcolor="#FFFFF" "align="left"

Dagiampi at 13:35, 27 March 2012

2012-03-27T13:35:37Z

← Older revision		Revision as of 13:35, 27 March 2012
Line 22:		Line 22:
	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| WordNet		\| WordNet
	\| ~~BIU1_abl~~-1		\| BIU2_abl-1
	\| style="text-align: center;"\| 0.9		\| style="text-align: center;"\| -0.05
	\| style="text-align: center;"\| ~~No Word-Net. On Dev set: 39.18% (compared to 40.73% when WN~~ is used)		\| style="text-align: center;"\| Without WordNet, which is used as a lexical rulebase resource

	\|- bgcolor="#ECECEC" "align="left"		\|- bgcolor="#ECECEC" "align="left"
	\| ~~CatVar~~		\|Direct
	\| BIU1_abl-2		\| BIU1_abl-2
	\| style="text-align: center;"\| 0.63		\| style="text-align: center;"\| 0.94
	\| style="text-align: center;"\| ~~No CatVar. On Dev set achieved about 40.20%~~ (~~compared to 40~~.~~73% when CatVar~~ is used)		\| style="text-align: center;"\| Without Bap (AKA "Direct"), which is used as a lexical rulebase resource

			\|- bgcolor="#ECECEC" "align="left"
			\| Wikipedia
			\| BIU1_abl-3
			\| style="text-align: center;"\| 1.56
			\| style="text-align: center;"\| Without Wikipedia, which is used as a lexical rulebase resource


			**********************************

	\|- bgcolor="#FFFFF" "align="left"		\|- bgcolor="#FFFFF" "align="left"
	\| Coreference resolver		\| Coreference resolver
	\| BIU1_abl-3		\| BIU1_abl-3
	\| style="text-align: center;"\| -0.88		\| style="text-align: center;"\| 1.56
	\| style="text-align: center;"\| ~~No coreference resolver~~		\| style="text-align: center;"\| Wikipedia
	On Dev set 41.62% (Compared to 40.73% when Coreference resolver is used).		On Dev set 41.62% (Compared to 40.73% when Coreference resolver is used).
	This ablation test is an unusual ablation test, since it shows that the co-reference resolution component has a negative impact.		This ablation test is an unusual ablation test, since it shows that the co-reference resolution component has a negative impact.

Dagiampi at 13:21, 27 March 2012

2012-03-27T13:21:31Z

← Older revision		Revision as of 13:21, 27 March 2012
Line 6:		Line 6:
	In the table below, the first column contains the specific resources which have been ablated.		In the table below, the first column contains the specific resources which have been ablated.
	<br>The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way).<br>The third column presents the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system.<br>		<br>The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way).<br>The third column presents the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system.<br>
	The fourth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the ~~RTE6~~ proceedings.<br>		The fourth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the RTE7 proceedings.<br>
	If the ablated resource is highlighted in yellow, it is a tool, otherwise is a knowledge resource.<br>		If the ablated resource is highlighted in yellow, it is a tool, otherwise is a knowledge resource.<br>

Dagiampi at 13:20, 27 March 2012

2012-03-27T13:20:22Z

← Older revision		Revision as of 13:20, 27 March 2012
Line 1:		Line 1:
	The following table lists the results of the ablation tests submitted by participants to RTE7 .<br>		The following table lists the results of the ablation tests submitted by participants to RTE7 .<br>
	The exploratory effort started in RTE5 ~~as a mandatory track~~, was ~~not only reiterated~~ in ~~RTE6, but also extended to tools~~.		The exploratory effort about knowledge resources, started in RTE5 and extended to tools in RTE-6, was proposed also in RTE-7.
	~~The~~ first column contains the specific resources which have been ablated.<br>The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way).<br>The third column presents the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system.<br>		<br>
			<br>

			In the table below, the first column contains the specific resources which have been ablated.
			<br>The second column lists the Team Run in the form ''[name_of_the_Team][number_of_the_submitted_run].[submission_task]'' (e.g. BIU1.2way, Boeing3.3way).<br>The third column presents the normalized difference between the accuracy of the complete system run and the accuracy of the ablation run (i.e. the output of the complete system without the ablated resource), showing the impact of the resource on the performance of the system.<br>
	The fourth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the RTE6 proceedings.<br>		The fourth column contains a brief description of the specific usage of the resource. It is based on the information provided both in the "readme" files submitted together with the ablation tests and in the system reports published in the RTE6 proceedings.<br>
	If the ablated resource is highlighted in yellow, it is a tool, otherwise is a knowledge resource.<br>		If the ablated resource is highlighted in yellow, it is a tool, otherwise is a knowledge resource.<br>