MUC-7 (State of the art) - Revision history

Pythonner at 13:51, 7 August 2007

2007-08-07T13:51:41Z

← Older revision		Revision as of 13:51, 7 August 2007
Line 3:		Line 3:
	* '''Recall:''' percentage of named entities defined in the corpus that were found by the program		* '''Recall:''' percentage of named entities defined in the corpus that were found by the program
	* Exact calculation of precision and recall is explained in the [http://www.itl.nist.gov/iad/894.02/related_projects/muc/muc_sw/muc_sw_manual.html MUC scoring software]		* Exact calculation of precision and recall is explained in the [http://www.itl.nist.gov/iad/894.02/related_projects/muc/muc_sw/muc_sw_manual.html MUC scoring software]


	* '''Training data:''' Training section of MUC-7 dataset		* '''Training data:''' Training section of MUC-7 dataset
			* '''Dryrun data:''' Dryrun section of MUC-7 dataset
	* '''Testing data:''' Formal section of MUC-7 dataset		* '''Testing data:''' Formal section of MUC-7 dataset

Pythonner at 19:59, 31 July 2007

2007-07-31T19:59:46Z

← Older revision		Revision as of 19:59, 31 July 2007
Line 66:		Line 66:
	* [[Named Entity Recognition (State of the art)\|Named Entity Recognition]]		* [[Named Entity Recognition (State of the art)\|Named Entity Recognition]]
	* [[State of the art]]		* [[State of the art]]

			[[Category:State of the art]]

Pythonner at 19:53, 31 July 2007

2007-07-31T19:53:21Z

← Older revision		Revision as of 19:53, 31 July 2007
Line 15:		Line 15:
	! System name		! System name
	! Short description		! Short description
	! System type		! System type (1)
	! Main publications		! Main publications
	! Software		! Software
	! Results ~~(F)~~		! Results
	\|-		\|-
	\| Annotator		\| Annotator
Line 33:		Line 33:
	\| -		\| -
	\| 93.39%		\| 93.39%
			\|-
			\| Balie
			\| Unsupervised approach: no prior training
			\| U
			\| Nadeau, Turney and Matwin (2006)
			\| [http://balie.sourceforge.net sourceforge.net]
			\| 77.71% (2)
	\|-		\|-
	\| Baseline		\| Baseline
Line 39:		Line 46:
	\| Whitelaw and Patrick (2003)		\| Whitelaw and Patrick (2003)
	\| -		\| -
	\| 58.89%		\| 58.89% (2)
	\|-		\|-
	\|}		\|}

	* '''System type''': R = hand-crafted rules, S = supervised learning, U = unsupervised learning, H = hybrid		* (1) '''System type''': R = hand-crafted rules, S = supervised learning, U = unsupervised learning, H = hybrid
			* (2) Calculated on Enamex types only.


Line 49:		Line 57:

	Mikheev, A., Grover, C. and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.		Mikheev, A., Grover, C. and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.

			Nadeau, D., Turney, P. D. and Matwin, S. (2006) [http://iit-iti.nrc-cnrc.gc.ca/publications/nrc-48727_e.html Unsupervised Named-Entity Recognition: Generating Gazetteers and Resolving Ambiguity]. ''Proceedings 19th Canadian Conference on Artificial Intelligence''. Québec, Canada.

	Whitelaw, C. and Patrick, J. (2003) [http://www.springerlink.com/content/ju66c6a2734fl20u/ Evaluating Corpora for Named Entity Recognition Using Character-Level Features]. ''Proceeding of the 16th Australian Conference on AI''. Perth, Australia.		Whitelaw, C. and Patrick, J. (2003) [http://www.springerlink.com/content/ju66c6a2734fl20u/ Evaluating Corpora for Named Entity Recognition Using Character-Level Features]. ''Proceeding of the 16th Australian Conference on AI''. Perth, Australia.

Pythonner at 19:35, 31 July 2007

2007-07-31T19:35:20Z

← Older revision		Revision as of 19:35, 31 July 2007
Line 15:		Line 15:
	! System name		! System name
	! Short description		! Short description
			! System type
	! Main publications		! Main publications
	! Software		! Software
	! Results (F)		! Results (F)
	\|-		\|-
	\| ~~Human~~		\| Annotator
	\| Human annotator		\| Human annotator
			\| -
	\| [http://www.itl.nist.gov/iad/894.02/related_projects/muc/proceedings/muc_7_toc.html MUC-7 proceedings]		\| [http://www.itl.nist.gov/iad/894.02/related_projects/muc/proceedings/muc_7_toc.html MUC-7 proceedings]
	\|		\| -
	\| 97.60%		\| 97.60%
	\|-		\|-
	\| LTG		\| LTG
	\| Best MUC-7 participant		\| Best MUC-7 participant
			\| H
	\| Mikheev, Grover and Moens (1998)		\| Mikheev, Grover and Moens (1998)
	\|		\| -
	\| 93.39%		\| 93.39%
			\|-
			\| Baseline
			\| Vocabulary transfer from training to testing
			\| S
			\| Whitelaw and Patrick (2003)
			\| -
			\| 58.89%
	\|-		\|-
	\|}		\|}

			* '''System type''': R = hand-crafted rules, S = supervised learning, U = unsupervised learning, H = hybrid


	== References ==		== References ==

	Mikheev, A., Grover, C., and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.		Mikheev, A., Grover, C. and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.

			Whitelaw, C. and Patrick, J. (2003) [http://www.springerlink.com/content/ju66c6a2734fl20u/ Evaluating Corpora for Named Entity Recognition Using Character-Level Features]. ''Proceeding of the 16th Australian Conference on AI''. Perth, Australia.

	== See also ==		== See also ==

Pythonner at 19:01, 31 July 2007

2007-07-31T19:01:44Z

← Older revision		Revision as of 19:01, 31 July 2007
Line 40:		Line 40:
	== See also ==		== See also ==

			* [[Named Entity Recognition (State of the art)\|Named Entity Recognition]]
	* [[State of the art]]		* [[State of the art]]

Pythonner at 18:58, 31 July 2007

2007-07-31T18:58:41Z

← Older revision		Revision as of 18:58, 31 July 2007
Line 37:		Line 37:

	Mikheev, A., Grover, C., and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.		Mikheev, A., Grover, C., and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.

			== See also ==

			* [[State of the art]]

Pythonner at 18:57, 31 July 2007

2007-07-31T18:57:42Z

New page

* '''Performance measure:''' F = 2 * Precision * Recall / (Recall + Precision)
* '''Precision:''' percentage of named entities found by the algorithm that are correct
* '''Recall:''' percentage of named entities defined in the corpus that were found by the program
* Exact calculation of precision and recall is explained in the [http://www.itl.nist.gov/iad/894.02/related_projects/muc/muc_sw/muc_sw_manual.html MUC scoring software]

* '''Training data:''' Training section of MUC-7 dataset
* '''Testing data:''' Formal section of MUC-7 dataset

== Table of results ==

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! System name
! Short description
! Main publications
! Software
! Results (F)
|-
| Human
| Human annotator
| [http://www.itl.nist.gov/iad/894.02/related_projects/muc/proceedings/muc_7_toc.html MUC-7 proceedings]
|
| 97.60%
|-
| LTG
| Best MUC-7 participant
| Mikheev, Grover and Moens (1998)
|
| 93.39%
|-
|}

== References ==

Mikheev, A., Grover, C., and Moens, M. (1998). [http://www-nlpir.nist.gov/related_projects/muc/proceedings/muc_7_proceedings/ltg_muc7.pdf Description of the LTG system used for MUC-7]. ''Proceedings of the Seventh Message Understanding Conference (MUC-7)''. Fairfax, Virginia.