ACL Wiki - User contributions [en]

Temporal Information Extraction (State of the art)

2013-06-11T10:40:33Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
| rule-based
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
| rule-based
| Chambers, 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| -
| -
|-
| ManTIME (4)
| CRF, probabilistic post-processing pipeline, rule-based normaliser
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
| deterministic rule-based
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
| MaxEnt, third party normalisers
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| -
| -
|-
| ClearTK (1,2)
| SVM, Logistic Regression, third party normaliser
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
| CRF, rule-based normaliser
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| -
| -
|-
| KUL (2)
| Logistic regression, post-processing, rule-based normaliser
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| -
| -
|-
| FSS-TimEx
| rule-based
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| -
| -
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| -
| -
| 70.17
|
|
|-
| ClearTK (4)
|
| Bethard, 2013
| 81.40
| 76.38
| 78.81
| 86.12
| 78.20
| 90.86
| 67.87
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| NavyTime (1)
|
| Chambers, 2013
| 80.73
| 79.87
| 80.30
| 84.03
| 75.79
| 91.26
| 67.48
|
|
|-
| Temp: (ESAfeature)
|
| X, 2013
| 78.33
| 61.61
| 68.97
| 79.09
| -
| -
| 54.55
|
|
|-
| JU_CSE
|
| Kolya et al., 2013
| 80.85
| 76.51
| 78.62
| 67.02
| 74.56
| 91.76
| 52.69
|
|
|-
| FSS-TimeEx
|
| Zavarella et al., 2013
| 63.13
| 67.11
| 65.06
| 66.00
| -
| -
| 42.94
|
|
|-
|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

===Task ABC: Temporal awareness evaluation===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T10:36:02Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
| rule-based
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
| rule-based
| Chambers, 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| -
| -
|-
| ManTIME (4)
| CRF, probabilistic post-processing pipeline, rule-based normaliser
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
| deterministic rule-based
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
| MaxEnt, third party normalisers
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| -
| -
|-
| ClearTK (1,2)
| SVM, Logistic Regression, third party normaliser
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
| CRF, rule-based normaliser
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| -
| -
|-
| KUL (2)
| Logistic regression, post-processing, rule-based normaliser
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| -
| -
|-
| FSS-TimEx
| rule-based
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| -
| -
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| -
| -
| 70.17
|
|
|-
| ClearTK (4)
|
| Bethard, 2013
| 81.40
| 76.38
| 78.81
| 86.12
| 78.20
| 90.86
| 67.87
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| NavyTime (1)
|
| Chambers, 2013
| 80.73
| 79.87
| 80.30
| 84.03
| 75.79
| 91.26
| 67.48
|
|
|-
| Temp: (ESAfeature)
|
| X, 2013
| 78.33
| 61.61
| 68.97
| 79.09
| -
| -
| 54.55
|
|
|-
| JU_CSE
|
| Kolya et al., 2013
| 80.85
| 76.51
| 78.62
| 67.02
| 74.56
| 91.76
| 52.69
|
|
|-
| FSS-TimeEx
|
| Zavarella et al., 2013
| 63.13
| 67.11
| 65.06
| 66.00
| -
| -
| 42.94
|
|
|-
|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T10:17:20Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
|
| Chambers, 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| -
| -
| 70.17
|
|
|-
| ClearTK (4)
|
| Bethard, 2013
| 81.40
| 76.38
| 78.81
| 86.12
| 78.20
| 90.86
| 67.87
|
|
|-
| NavyTime (1)
|
| Chambers, 2013
| 80.73
| 79.87
| 80.30
| 84.03
| 75.79
| 91.26
| 67.48
|
|
|-
| Temp: (ESAfeature)
|
| X, 2013
| 78.33
| 61.61
| 68.97
| 79.09
| -
| -
| 54.55
|
|
|-
| JU_CSE
|
| Kolya et al., 2013
| 80.85
| 76.51
| 78.62
| 67.02
| 74.56
| 91.76
| 52.69
|
|
|-
| FSS-TimeEx
|
| Zavarella et al., 2013
| 63.13
| 67.11
| 65.06
| 66.00
| -
| -
| 42.94
|
|
|-
|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T10:10:50Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
|
| Chambers, 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| 0.0
| 0.0
| 70.17
|
|
|-
| ClearTK (4)
|
| Bethard, 2013
| 81.40
| 76.38
| 78.81
| 86.12
| 78.20
| 90.86
| 67.87
|
|
|-
| NavyTime (1)
|
| Chambers, 2013
| 80.73
| 79.87
| 80.30
| 84.03
| 75.79
| 91.26
| 67.48
|
|
|-
| Temp: (ESAfeature)
|
| X, 2013
| 78.33
| 61.61
| 68.97
| 79.09
| 0.0
| 0.0
| 54.55
|
|
|-
| JU_CSE
|
| Kolya et al., 2013
| 80.85
| 76.51
| 78.62
| 67.02
| 74.56
| 91.76
| 52.69
|
|
|-
|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T10:06:00Z

Filannim: /* Task B: Event extraction and classification */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| 0.0
| 0.0
| 70.17
|
|
|-
| ClearTK (4)
|
| Bethard, 2013
| 81.40
| 76.38
| 78.81
| 86.12
| 78.20
| 90.86
| 67.87
|
|
|-
| NavyTime (1)
|
| XXX, 2013
| 80.73
| 79.87
| 80.30
| 84.03
| 75.79
| 91.26
| 67.48
|
|
|-

|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T10:02:26Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="3" | Identification
! colspan="3" | Attributes
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Accuracy
|-
! Pre.
! Rec.
! F1
! Class
! Tense
! Aspect
|-
| ATT (1)
|
| Jung et al., 2013
| 81.44
| 80.67
| '''81.05'''
| 88.69
| 73.37
| 90.68
| '''71.88'''
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 80.69
| 77.99
| 79.32
| 88.46
| 0.0
| 0.0
| 70.17
|
|
|-
|}

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

State of the art

2013-06-11T09:49:51Z

Filannim:

The purpose of this section of the ACL wiki is to be a repository of ''k''-best state-of-the-art results (i.e., methods and software) for various core natural language processing tasks.

As a side effect, this should hopefully evolve into a knowledge base of standard evaluation methods and datasets for various tasks, as well as encourage more effort into reproducibility of results. This will help newcomers to a field appreciate what has been done so far and what the main tasks are, and will help keep active researchers informed on fields other than their specific research. The next time you need a system for PP attachment, or wonder what is the current state of word sense disambiguation, this will be the place to visit.

Please contribute! (This is also a good place for you to display your results!)

As a historical point of reference, you may want to refer to the [http://web.archive.org/web/20100325144600/http://cslu.cse.ogi.edu/HLTsurvey/ Survey of the State of the Art in Human Language Technology] ([http://www.lt-world.org/hlt_survey/master.pdf also available as PDF]), edited by R. Cole, J. Mariani, H. Uszkoreit, G. B. Varile, A. Zaenen, A. Zampolli, V. Zue, 1996.


* [[Anaphora Resolution (State of the art)|Anaphora Resolution]] (stub)
* [[Attributional and Relational Similarity (State of the art)|Attributional and Relational Similarity]] -- [[ESL Synonym Questions (State of the art)|ESL]], [[SAT Analogy Questions (State of the art)|SAT]], [[TOEFL Synonym Questions (State of the art)|TOEFL]]
* [[Automatic Summarization (State of the art)|Automatic Summarization]]
* [[Chunking (State of the art)|Chunking]] (stub)
* [[Dependency Parsing (State of the art)|Dependency Parsing]] (stub)
* [[Document Classification (State of the art)|Document Classification]] (stub)
* [[Language Identification (State of the art)|Language Identification]] (stub)
* [[Named Entity Recognition (State of the art)|Named Entity Recognition]]
* [[Noun-Modifier Semantic Relations (State of the art)|Noun-Modifier Semantic Relations]]
* [[NP Chunking (State of the art)|NP Chunking]]
* [[Paraphrase Identification (State of the art)|Paraphrase Identification]]
* [[Parsing (State of the art)|Parsing]]
* [[POS Induction (State of the art) |POS Induction]]
* [[POS Tagging (State of the art) |POS Tagging]]
* [[PP Attachment (State of the art)|PP Attachment]] (stub)
* [[Semantic Role Labeling (State of the art)|Semantic Role Labeling]] (stub)
* [[Sentiment Analysis (State of the art)|Sentiment Analysis]] (stub)
* [[Speech Recognition (State of the art)|Speech Recognition]] (article request)
* [[Temporal Expression Recognition and Normalisation (State of the art)|Temporal Expression Recognition and Normalisation]]
* [[Cleaneval (State of the art)| Web Corpus Cleaning]] (stub)
* [[Word Segmentation (State of the art)|Word Segmentation]] (stub)
* [[Word Sense Disambiguation (State of the art)|Word Sense Disambiguation]] (stub)


[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:48:34Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
| [http://www.gnu.org/licenses/gpl.html GNU GPL v3]
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
| [http://www.gnu.org/licenses/gpl-2.0.html GNU GPL v2]
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ Download]
| [http://opensource.org/licenses/BSD-3-Clause BSD-3 Clause]
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:43:02Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
|
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
|
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
| na
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
| [https://code.google.com/p/cleartk/ | Download]
|
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
| na
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
| na
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
| na
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:40:49Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [http://www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
|
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
|
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:39:56Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| '''77.61'''
| [http://dbs.ifi.uni-heidelberg.de/index.php?id=129 Download]
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
| na
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
| [www.cs.man.ac.uk/~filannim/projects/tempeval-3/ Demo & Download]
|
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
| [http://nlp.stanford.edu/software/sutime.shtml Demo & Download]
|
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:36:37Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| '''85.95'''
| 77.61
|
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 78.58
| 70.97
|
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
|
|
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| '''80.43'''
| 79.57
| 89.36
| '''91.30'''
| '''90.32'''
| 88.90
| 74.60
| 67.38
|
|
|-
| ATT (2)
|
| Jung et al., 2013
| '''90.57'''
| 69.57
| 78.69
| '''98.11'''
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| '''82.71'''
| 93.75
| 86.96
| 90.23
| '''93.33'''
| 71.66
| 64.66
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:31:59Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 83.85
| 78.99
| 81.34
| 93.08
| 87.68
| 90.30
| 90.91
| 85.95
| 77.61
|
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 78.72
| 80.43
| 79.57
| 89.36
| 91.30
| 90.32
| 88.90
| 78.58
| 70.97
|
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 78.86
| 70.29
| 74.33
| 95.12
| 84.78
| 89.66
| 86.31
| 76.92
| 68.97
|
|
|-
| SUTime
|
| Chang et al., 2013
| 78.72
| 80.43
| 79.57
| 89.36
| 91.30
| 90.32
| 88.90
| 74.60
| 67.38
|
|
|-
| ATT (2)
|
| Jung et al., 2013
| 90.57
| 69.57
| 78.69
| 98.11
| 75.36
| 85.25
| 91.34
| 76.91
| 65.57
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
| 85.94
| 79.71
| 82.71
| 93.75
| 86.96
| 90.23
| 93.33
| 71.66
| 64.66
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
| 81.51
| 70.29
| 75.49
| 93.28
| 80.43
| 86.38
| 87.39
| 73.87
| 63.81
|
|
|-
| KUL (2)
|
| Kolomiyets et al., 2013
| 76.99
| 63.04
| 69.32
| 92.92
| 76.09
| 83.67
| 88.56
| 75.24
| 62.95
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
| 52.03
| 46.38
| 49.04
| 90.24
| 80.43
| 85.06
| 81.08
| 68.47
| 58.24
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:24:38Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
| 93.08%
| 87.68%
| 90.30%
| 83.85%
| 78.99%
| 81.34%
| 90.91%
| 85.95%
| 77.61%
|
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
| 89.36%
| 91.30%
| 90.32%
| 78.72%
| 80.43%
| 79.57%
| 88.90%
| 78.58%
| 70.97%
|
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
| 95.12%
| 84.78%
| 89.66%
| 78.86%
| 70.29%
| 74.33%
| 86.31%
| 76.92%
| 68.97%
|
|
|-
| SUTime
|
| Chang et al., 2013
|
|
|
|
|
|
| 88.90%
| 74.60%
| 67.38%
|
|
|-
| ATT (2)
|
| Jung et al., 2013
|
|
|
|
|
|
| 91.34%
| 76.91%
| 65.57%
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
|
|
|
|
|
|
| 93.33%
| 71.66%
| 64.66%
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
|
|
|
|
|
|
| 87.39%
| 73.87%
| 63.81%
|
|
|-
| KUL
|
| Kolomiyets et al., 2013
|
|
|
|
|
|
| 88.56%
| 75.24%
| 62.95%
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
|
|
|
|
|
|
| 81.08%
| 68.47%
| 58.24%
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:20:10Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name (best run)
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime (t)
|
| Stro ̈tgen et al., 2013
|
|
|
|
|
|
| 90.91%
| 85.95%
| 77.61%
|
|
|-
| NavyTime (1,2)
|
| Chambers et al., 2013
|
|
|
|
|
|
| 88.90%
| 78.58%
| 70.97%
|
|
|-
| ManTIME (4)
|
| Filannino et al., 2013
|
|
|
|
|
|
| 86.31%
| 76.92%
| 68.97%
|
|
|-
| SUTime
|
| Chang et al., 2013
|
|
|
|
|
|
| 88.90%
| 74.60%
| 67.38%
|
|
|-
| ATT (2)
|
| Jung et al., 2013
|
|
|
|
|
|
| 91.34%
| 76.91%
| 65.57%
|
|
|-
| ClearTK (1,2)
|
| Bethard, 2013
|
|
|
|
|
|
| 93.33%
| 71.66%
| 64.66%
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
|
|
|
|
|
|
| 87.39%
| 73.87%
| 63.81%
|
|
|-
| KUL
|
| Kolomiyets et al., 2013
|
|
|
|
|
|
| 88.56%
| 75.24%
| 62.95%
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
|
|
|
|
|
|
| 81.08%
| 68.47%
| 58.24%
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:18:01Z

Filannim: /* Task A: Temporal expression extraction and normalisation */

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime
|
| Stro ̈tgen et al., 2013
|
|
|
|
|
|
| 90.91%
| 85.95%
| 77.61%
|
|
|-
| NavyTime
|
| Chambers et al., 2013
|
|
|
|
|
|
| 88.90%
| 78.58%
| 70.97%
|
|
|-
| ManTIME
|
| Filannino et al., 2013
|
|
|
|
|
|
| 86.31%
| 76.92%
| 68.97%
|
|
|-
| SUTime
|
| Chang et al., 2013
|
|
|
|
|
|
| 88.90%
| 74.60%
| 67.38%
|
|
|-
| ATT
|
| Jung et al., 2013
|
|
|
|
|
|
| 91.34%
| 76.91%
| 65.57%
|
|
|-
| ClearTK
|
| Bethard, 2013
|
|
|
|
|
|
| 93.33%
| 71.66%
| 64.66%
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
|
|
|
|
|
|
| 87.39%
| 73.87%
| 63.81%
|
|
|-
| KUL
|
| Kolomiyets et al., 2013
|
|
|
|
|
|
| 88.56%
| 75.24%
| 62.95%
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
|
|
|
|
|
|
| 81.08%
| 68.47%
| 58.24%
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:14:50Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime
|
| Stro ̈tgen et al., 2013
|
|
|
|
|
|
|
|
| 77.61%
|
|
|-
| NavyTime
|
| Chambers et al., 2013
|
|
|
|
|
|
|
|
| 70.97%
|
|
|-
| ManTIME
|
| Filannino et al., 2013
|
|
|
|
|
|
|
|
| 68.97%
|
|
|-
| SUTime
|
| Chang et al., 2013
|
|
|
|
|
|
|
|
| 67.38%
|
|
|-
| ATT
|
| Jung et al., 2013
|
|
|
|
|
|
|
|
| 65.57%
|
|
|-
| ClearTK
|
| Bethard, 2013
|
|
|
|
|
|
|
|
| 64.66%
|
|
|-
| JU-CSE
|
| Kolya et al., 2013
|
|
|
|
|
|
|
|
| 63.81%
|
|
|-
| KUL
|
| Kolomiyets et al., 2013
|
|
|
|
|
|
|
|
| 62.95%
|
|
|-
| FSS-TimEx
|
| Zavarella et al., 2013
|
|
|
|
|
|
|
|
| 58.24%
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:09:25Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime
|
| Stro ̈tgen et al., 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| NavyTime
|
| Chambers et al., 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME
|
| Filannino et al., 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| SUTime
|
| Chang et al., 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT
|
| Jung et al., 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK
|
| Bethard, 2013
|
|
|
|
|
|
|
|
|
|
|
|-
| JU-CSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| FSS-TimEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:07:08Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===
The table shows the best result for each system. Different runs per system are not shown.
{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| NavyTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| SUTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| JU-CSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| FSS-TimEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:06:05Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| NavyTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| SUTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| JU-CSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| FSS-TimEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T09:03:28Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime-t
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| HeidelTime-bf
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| HeidelTime-1.2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| NavyTime-1,2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-4
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-6
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| SUTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-1
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-5
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-1
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK-1,2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| JU-CSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL-TE3RunABC
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK-3,4
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| FSS-TimEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. [http://www.aclweb.org/anthology/S/S13/S13-2011.pdf JU_CSE: A CRF based approach to annotation of temporal expression, event and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. [http://www.aclweb.org/anthology/S/S13/S13-2012.pdf Navytime: Event and time ordering from raw text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. [http://www.aclweb.org/anthology/S/S13/S13-2013.pdf SUTime: Evaluation in TempEval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. [http://www.aclweb.org/anthology/S/S13/S13-2014.pdf KUL: Data-driven approach to temporal parsing of newswire articles]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. [http://www.aclweb.org/anthology/S/S13/S13-2015.pdf UTTime: Temporal relation classification using deep syntactic features]. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]

Temporal Information Extraction (State of the art)

2013-06-11T08:58:25Z

Filannim:

==Data sets==

==Performance measures==

==Results==
The following results refers to the TempEval-3 challenge, the last evaluation exercise.

===Task A: Temporal expression extraction and normalisation===

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! rowspan="3" | System name
! rowspan="3" | Short description
! rowspan="3" | Main publication
! colspan="6" | Identification
! colspan="2" | Normalisation
! rowspan="3" | Overall score
! rowspan="3" | Software
! rowspan="3" | License
|-
! colspan="3" | Strict matching
! colspan="3" | Lenient matching
! colspan="2" | Accuracy
|-
! Pre.
! Rec.
! F1
! Pre.
! Rec.
! F1
! Type
! Value
|-
| HeidelTime-t
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| HeidelTime-bf
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| HeidelTime-1.2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| NavyTime-1,2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-4
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-6
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| SUTime
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-1
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-5
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ManTIME-2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-1
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK-1,2
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| JU-CSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| KUL-TE3RunABC
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ClearTK-3,4
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| ATT-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
| FSS-TimEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
|}

===Task B: Event extraction and classification===

===Task C: Annotating relations given gold entities===

===Task C relation only: Annotating relations given gold entities and related pairs===

==Challenges==
* '''TempEval''', ''Temporal Relation Identification'', 2007: [http://www.timeml.org/tempeval/ web page]
* '''TempEval-2''', ''Evaluating Events, Time Expressions, and Temporal Relations'', 2010: [http://www.timeml.org/tempeval2/ web page]
* '''TempEval-3''', ''Evaluating Time Expressions, Events, and Temporal Relations'', 2013: [http://www.cs.york.ac.uk/semeval-2013/task1/ web page]

==References==

* UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. [http://www.aclweb.org/anthology/S/S13/S13-2001.pdf Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 1–9.
* Bethard, S. [http://www.aclweb.org/anthology/S/S13/S13-2002.pdf ClearTK-TimeML: A minimalist approach to tempeval 2013]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), vol. 2, Association for Computational Linguistics, Association for Computational Linguistics, pp. 10–14.
* Stro ̈tgen, J., Zell, J., and Gertz, M. [http://www.aclweb.org/anthology/S/S13/S13-2003.pdf Heideltime: Tuning english and developing spanish resources for tempeval-3]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 15–19.
* Jung, H., and Stent, A. [http://www.aclweb.org/anthology/S/S13/S13-2004.pdf ATT1: Temporal annotation using big windows and rich syntactic and semantic features]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 20–24.
* Filannino, M., Brown, G., and Nenadic, G. [http://www.aclweb.org/anthology/S/S13/S13-2009.pdf ManTIME: Temporal expression identification and normalization in the Tempeval-3 challenge]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evalu- ation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 53–57.
* Zavarella, V., and Tanev, H. [http://www.aclweb.org/anthology/S/S13/S13-2010.pdf FSS-TimEx for tempeval-3: Extracting temporal information from text]. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 58–63.
* Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A., and Bandyopadhyay, S. Ju cse: A crf based approach to annotation of temporal expression, event and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 64–72.
* Chambers, N. Navytime: Event and time ordering from raw text. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 73–77.
* Chang, A., and Manning, C. D. Sutime: Evaluation in tempeval-3. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 78–82.
* Kolomiyets, O., and Moens, M.-F. Kul: Data-driven approach to temporal parsing of newswire ar- ticles. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceed- ings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 83–87.
* Laokulrat, N., Miwa, M., Tsuruoka, Y., and Chikayama, T. Uttime: Temporal relation clas- sification using deep syntactic features. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) (Atlanta, Georgia, USA, June 2013), Association for Computational Linguistics, pp. 88– 92.
*
*
*
*
*
*
*
*

== See also ==
* [[State of the art]]

== External links ==
* [http://timexportal.info TimexPortal]

[[Category:State of the art]]