Difference between revisions of "Textual Entailment Resource Pool"
Jump to navigation
Jump to search
(→RTE data sets: Adding MultiNLI) |
|||
| (90 intermediate revisions by 32 users not shown) | |||
| Line 1: | Line 1: | ||
| + | [[Textual Entailment]] > '''Resources''': | ||
| + | ---- | ||
| + | |||
[[Textual Entailment|Textual entailment]] systems rely on many different types of [[Natural Language Processing|NLP]] resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system. | [[Textual Entailment|Textual entailment]] systems rely on many different types of [[Natural Language Processing|NLP]] resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system. | ||
| Line 9: | Line 12: | ||
== Complete RTE Systems == | == Complete RTE Systems == | ||
| + | * [http://project.cgm.unive.it/html/venses.html VENSES] (from Ca' Foscari University of Venice, Italy) | ||
* [http://svn.ask.it.usyd.edu.au/trac/candc/wiki/nutcracker Nutcracker] (available for download) | * [http://svn.ask.it.usyd.edu.au/trac/candc/wiki/nutcracker Nutcracker] (available for download) | ||
| − | * [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign) | + | * [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign) - INACTIVE (as of 2010-12-22) |
| + | * [http://edits.fbk.eu/ EDITS - Edit Distance Textual Entailment Suite] (open source software developed by [http://hlt.fbk.eu/ Human Language Technology (HLT) group at FBK-Irst]) | ||
| + | * [http://u.cs.biu.ac.il/~nlp/downloads/biutee/protected-biutee.html BIUTEE] - Bar Ilan University Textual Entailment Engine (open source) | ||
| + | * [http://hltfbk.github.io/Excitement-Open-Platform/ EXCITEMENT Open Platform (EOP)] - A generic multi-lingual platform for textual inference made available to the scientific and technological communities by the [https://sites.google.com/site/excitementproject/ EU project EXCITEMENT] | ||
| + | * [http://kmcs.nii.ac.jp/tifmo/ TIFMO] (from National Institute of Informatics, Japan) | ||
| + | |||
| + | == RTE data sets == | ||
| + | === Past campaigns data sets === | ||
| + | * [http://pascallin.ecs.soton.ac.uk/Challenges/RTE/Datasets RTE1 dataset] - provided by [http://pascallin.ecs.soton.ac.uk PASCAL] | ||
| + | * [http://pascallin.ecs.soton.ac.uk/Challenges/RTE2/Datasets RTE2 dataset] - provided by [http://pascallin.ecs.soton.ac.uk PASCAL] | ||
| + | * [http://pascallin.ecs.soton.ac.uk/Challenges/RTE3/Datasets RTE3 dataset] - provided by [http://pascallin.ecs.soton.ac.uk PASCAL] | ||
| + | * [http://www.nist.gov/tac/data/past/2008/RTE-4.html RTE4 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements] | ||
| + | * [http://www.nist.gov/tac/data/past/2009/RTE-5.html RTE5 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements] | ||
| + | * [http://www.nist.gov/tac/data/past/2010/RTE-6_Main_Task.html RTE6 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements] | ||
| + | * [http://www.nist.gov/tac/2011/RTE/index.html RTE7 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements] | ||
| + | * [http://www.cs.york.ac.uk/semeval-2013/task7/ The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge] at SemEval 2013 | ||
| + | * [http://www.nyu.edu/projects/bowman/multinli/ The MultiGenre NLI Corpus] (433k examples, used in the [https://repeval2017.github.io/shared/ RepEval 2017 Shared Task]) | ||
| + | |||
| + | === RTE data sets translated in other languages === | ||
| + | * [http://www.dfki.de/~neumann/resources/RTE3_DE_V1.2_2013-12-02.zip RTE3 dataset translated in German] - provided by [https://sites.google.com/site/excitementproject/ EXCITEMENT] | ||
| + | * [https://sites.google.com/site/excitementproject/results/RTE3-ITA_V1_2012-10-04.zip RTE3 dataset translated in Italian] - provided by [https://sites.google.com/site/excitementproject/ EXCITEMENT] | ||
| − | |||
| − | === | + | === Other data sets === |
| + | * [http://nlp.stanford.edu/projects/snli The Stanford Natural Language Inference (SNLI) corpus], a 570k example manually-annotated TE dataset with accompanying leaderboard. | ||
| + | * [http://www.coli.uni-saarland.de/projects/salsa/fate FrameNet manually annotated RTE 2006 Test Set.] Provided by [http://www.coli.uni-saarland.de/projects/salsa/ SALSA project, Saarland University.] | ||
* [http://www.cs.biu.ac.il/~nlp/files/RTE_2006_Aligned.zip Manually Word Aligned RTE 2006 Data Sets.] Provided by [http://research.microsoft.com/nlp/ the Natural Language Processing Group, Microsoft Research.] | * [http://www.cs.biu.ac.il/~nlp/files/RTE_2006_Aligned.zip Manually Word Aligned RTE 2006 Data Sets.] Provided by [http://research.microsoft.com/nlp/ the Natural Language Processing Group, Microsoft Research.] | ||
| − | *[http:// | + | * [http://www-nlp.stanford.edu/projects/contradiction/ RTE data sets annotated for a 3-way decision: entails, contradicts, unknown.] Provided by Stanford NLP Group. |
| + | * [http://www.cs.utexas.edu/~pclark/bpi-test-suite/ BPI RTE data set] - 250 pairs, focusing on world knowledge. Provided jointly by [http://www.boeing.com/phantom/math_ct/index.html Boeing], [http://wordnet.cs.princeton.edu/ Princeton], and [http://www.isi.edu ISI]. | ||
| + | * [http://hlt.fbk.eu/en/Technology/TE_Specialized_Data Textual Entailment Specialized Data Sets] - 90 RTE-5 Test Set pairs annotated with linguistic phenomena + 203 monothematic pairs (i.e. pairs where only one linguistic phenomenon is relevant to the entailment relation) created from the 90 annotated pairs. Provided jointly by [http://hlt.fbk.eu/en/home FBK-Irst], and [http://www.celct.it/ CELCT]. | ||
| + | * [http://www.nist.gov/tac/data/ RTE-5 Search Pilot Data Set annotated with anaphora and coreference information] - RTE-5 Search Data Set annotated with anaphora/coreference information + Augmented RTE-5 Search Data Set, where all the referring expressions which need to be resolved in the entailing sentences are substituted by explicit expressions on the basis of the anaphora/coreference annotation. Provided by [http://www.celct.it/ CELCT] and distributed by [http://www.nist.gov/index.html NIST] at the [http://www.nist.gov/tac/data/ Past TAC Data] web page (2009 Search Pilot, annotated test/dev data). | ||
| + | * [http://www.investigacion.frc.utn.edu.ar/mslabs/~jcastillo/Sagan-test-suite/ RTE-3-Expanded, RTE-4-Expanded, RTE-5-Expanded.] RTE data set expanded in the two and three way task, at least 2000 pairs in each data set. | ||
| + | * [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf this ACL 2010 paper] | ||
| + | * [http://art.uniroma2.it/zanzotto/resources/WIKI_FINAL_CORPUS_v1.zip Wiki Entailment Corpus] A RTE-like set of entailment pairs extracted from Wikipedia revisions described in [http://aclweb.org/anthology/W/W10/W10-3504.pdf this paper] | ||
| + | * [https://github.com/daoudclarke/rte-experiment The Guardian Headlines Entailment Training Dataset] An automatically generated dataset of 32,000 pairs similar to the RTE-1 dataset. | ||
| + | * [http://nlp.uned.es/clef-qa/ave/ Answer Validation Exercise at CLEF 2006 (AVE 2006)] | ||
| + | * [http://www.evalita.it/2009/tasks/te The Textual Entailment Task for Italian] at [http://www.evalita.it/2009 EVALITA 2009] An evaluation exercise on TE for Italian. | ||
| + | * [http://www.cs.york.ac.uk/semeval-2012/task8/ Cross-Lingual Textual Entailment for Content Synchronization] The Cross-Lingual Textual Entailment task at [http://www.cs.york.ac.uk/semeval-2012/ SemEval 2012]. | ||
| + | * [http://www.cs.york.ac.uk/semeval-2013/task8/ Cross-Lingual Textual Entailment for Content Synchronization] The Cross-Lingual Textual Entailment task at [http://www.cs.york.ac.uk/semeval-2013/ SemEval 2013]. | ||
| + | * [http://nilc.icmc.usp.br/assin/ ASSIN] a shared task on TE for Portuguese with 10,000 pairs. | ||
| + | |||
| + | == Knowledge Resources == | ||
| + | The [[RTE Knowledge Resources]] page presents: | ||
| − | + | * a [[RTE Knowledge Resources#Call for Resources|call for resources]], inviting system developers to share the resources used by their own TE engines, to both help improve the TE technology and further test and evaluate such resources; | |
| − | * [[ | + | * [[RTE Knowledge Resources#Ablation tests|the ablation tests]] carried out in the RTE challenges in order to evaluate the impact of knowledge resources and tools on TE system performances; |
| − | * [ | + | * [[RTE Knowledge Resources#Publicly available Resources|lists of knowledge resources]], both publicly available and unpublished, used by systems participating in the last RTE challenges. |
| − | + | <!-- * [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf this ACL 2010 paper] --> | |
| − | * [ | + | |
| − | * [ | + | == Projects == |
| − | * [ | + | * [http://www.cosyne.eu/ CoSyne EU project] The Cross-Lingual Multilingual Content Synchronization with Wikis. |
| + | * [https://sites.google.com/site/excitementproject/ EXCITEMENT EU project] EXploring Customer Interactions through Textual EntailMENT. | ||
| + | * [http://qallme.fbk.eu/ QALL-ME EU project] Question Answering Learning technologies in a multiLingual and Multimodal Environment. | ||
== Tools == | == Tools == | ||
| Line 31: | Line 73: | ||
* [http://svn.ask.it.usyd.edu.au/trac/candc C&C parser for Combinatory Categorial Grammar] | * [http://svn.ask.it.usyd.edu.au/trac/candc C&C parser for Combinatory Categorial Grammar] | ||
* [[Minipar]] | * [[Minipar]] | ||
| + | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SP Shallow Parser] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/shallow_parse_demo.php web demo] of this tool | ||
| + | |||
| + | === Role Labelling === | ||
| + | * [http://cemantix.org/assert.html ASSERT] | ||
* [http://www.coli.uni-saarland.de/projects/salsa/shal/ Shalmaneser] | * [http://www.coli.uni-saarland.de/projects/salsa/shal/ Shalmaneser] | ||
| − | |||
* [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SRL Semantic Role Labeler] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/srl-demo.php web demo] of this tool | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SRL Semantic Role Labeler] - from the University of Illinois at Urbana-Champaign, see a [http://l2r.cs.uiuc.edu/~cogcomp/srl-demo.php web demo] of this tool | ||
| − | |||
=== Entity Recognition Tools === | === Entity Recognition Tools === | ||
| − | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=NE | + | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=NE Illinois Named Entity Tagger] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_demo.php web demo] of this tool |
| − | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=CORANKER | + | * [http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=CORANKER Illinois Multi-lingual Named Entity Discovery Tool] - see a [http://l2r.cs.uiuc.edu/~cogcomp/ne_matcher_demo.php web demo] of this tool |
| + | |||
| + | === Similarity / Relatedness Tools === | ||
| + | * [http://ixa2.si.ehu.es/ukb UKB]: Open source [[WordNet]]-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words | ||
=== Corpus Readers === | === Corpus Readers === | ||
* [http://nltk.org NLTK] provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the [http://nltk.org/doc/guides/corpus.html#rte Corpus Readers] Guide for more information. | * [http://nltk.org NLTK] provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the [http://nltk.org/doc/guides/corpus.html#rte Corpus Readers] Guide for more information. | ||
| + | |||
| + | === Related Libraries === | ||
| + | |||
| + | * [http://www.semantilog.org/pypes.html PyPES] general purpose library containing evaluation environment for RTE and McPIET text inference engine based on the ERG (English Resource Grammar) | ||
| + | |||
| + | === Text Normalizers === | ||
| + | [http://u.cs.biu.ac.il/~nlp/downloads/normalizer.html Java number normalizer (Beta)] | ||
| + | A tool for converting textual representations of numbers to a standard numerical string. | ||
| + | |||
| + | == References == | ||
| + | |||
| + | *[[Textual Entailment References#Tutorials | Tutorials ]] and [[Textual Entailment References#Workshops | Workshops ]] | ||
| + | *[[Textual Entailment References#Papers in recent conferences and other workshops | Papers in recent conferences and other workshops ]] | ||
| + | *[[Textual Entailment References#Journal papers | Journal papers ]] | ||
== Links == | == Links == | ||
* [http://homepages.inf.ed.ac.uk/jbos/rte/ Textual Entailment site by Johan Bos] | * [http://homepages.inf.ed.ac.uk/jbos/rte/ Textual Entailment site by Johan Bos] | ||
| − | * [http://ai-nlp.info.uniroma2.it/te/ Textual Entailment at the University of Rome "Tor Vergata"] | + | * [http://ai-nlp.info.uniroma2.it/research/te/ Textual Entailment at the University of Rome "Tor Vergata"] |
[[Category:Textual Entailment Portal]] | [[Category:Textual Entailment Portal]] | ||
| − | * [http:// | + | * [http://cogcomp.cs.illinois.edu/page/demo_view/18 Illinois Textual Entailment System Component demos] |
Latest revision as of 09:31, 29 May 2017
Textual Entailment > Resources:
Textual entailment systems rely on many different types of NLP resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system.
In response, the Recognizing Textual Entailment (RTE) shared task community initiated a new activity for building this Textual Entailment Resource Pool. RTE participants and any other member of the NLP community are encouraged to contribute to the pool.
In an effort to determine the relative impact of the resources, RTE participants are strongly encouraged to report, whenever possible, the contribution to the overall performance of each utilized resource. Formal qualitative and quantitative results should be included in a separate section of the system report as well as posted on the talk pages of this Textual Entailment Resource Pool.
Adding a new resource is very easy. See how to use existing templates to do this in Help:Using Templates.
Complete RTE Systems
- VENSES (from Ca' Foscari University of Venice, Italy)
- Nutcracker (available for download)
- Entailment Demo (from the University of Illinois at Urbana-Champaign) - INACTIVE (as of 2010-12-22)
- EDITS - Edit Distance Textual Entailment Suite (open source software developed by Human Language Technology (HLT) group at FBK-Irst)
- BIUTEE - Bar Ilan University Textual Entailment Engine (open source)
- EXCITEMENT Open Platform (EOP) - A generic multi-lingual platform for textual inference made available to the scientific and technological communities by the EU project EXCITEMENT
- TIFMO (from National Institute of Informatics, Japan)
RTE data sets
Past campaigns data sets
- RTE1 dataset - provided by PASCAL
- RTE2 dataset - provided by PASCAL
- RTE3 dataset - provided by PASCAL
- RTE4 dataset - provided by NIST - freely available upon request. For details see TAC User Agreements
- RTE5 dataset - provided by NIST - freely available upon request. For details see TAC User Agreements
- RTE6 dataset - provided by NIST - freely available upon request. For details see TAC User Agreements
- RTE7 dataset - provided by NIST - freely available upon request. For details see TAC User Agreements
- The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge at SemEval 2013
- The MultiGenre NLI Corpus (433k examples, used in the RepEval 2017 Shared Task)
RTE data sets translated in other languages
- RTE3 dataset translated in German - provided by EXCITEMENT
- RTE3 dataset translated in Italian - provided by EXCITEMENT
Other data sets
- The Stanford Natural Language Inference (SNLI) corpus, a 570k example manually-annotated TE dataset with accompanying leaderboard.
- FrameNet manually annotated RTE 2006 Test Set. Provided by SALSA project, Saarland University.
- Manually Word Aligned RTE 2006 Data Sets. Provided by the Natural Language Processing Group, Microsoft Research.
- RTE data sets annotated for a 3-way decision: entails, contradicts, unknown. Provided by Stanford NLP Group.
- BPI RTE data set - 250 pairs, focusing on world knowledge. Provided jointly by Boeing, Princeton, and ISI.
- Textual Entailment Specialized Data Sets - 90 RTE-5 Test Set pairs annotated with linguistic phenomena + 203 monothematic pairs (i.e. pairs where only one linguistic phenomenon is relevant to the entailment relation) created from the 90 annotated pairs. Provided jointly by FBK-Irst, and CELCT.
- RTE-5 Search Pilot Data Set annotated with anaphora and coreference information - RTE-5 Search Data Set annotated with anaphora/coreference information + Augmented RTE-5 Search Data Set, where all the referring expressions which need to be resolved in the entailing sentences are substituted by explicit expressions on the basis of the anaphora/coreference annotation. Provided by CELCT and distributed by NIST at the Past TAC Data web page (2009 Search Pilot, annotated test/dev data).
- RTE-3-Expanded, RTE-4-Expanded, RTE-5-Expanded. RTE data set expanded in the two and three way task, at least 2000 pairs in each data set.
- Explanation-Based Analysis annotation of RTE 5 Main Task subset described in this ACL 2010 paper
- Wiki Entailment Corpus A RTE-like set of entailment pairs extracted from Wikipedia revisions described in this paper
- The Guardian Headlines Entailment Training Dataset An automatically generated dataset of 32,000 pairs similar to the RTE-1 dataset.
- Answer Validation Exercise at CLEF 2006 (AVE 2006)
- The Textual Entailment Task for Italian at EVALITA 2009 An evaluation exercise on TE for Italian.
- Cross-Lingual Textual Entailment for Content Synchronization The Cross-Lingual Textual Entailment task at SemEval 2012.
- Cross-Lingual Textual Entailment for Content Synchronization The Cross-Lingual Textual Entailment task at SemEval 2013.
- ASSIN a shared task on TE for Portuguese with 10,000 pairs.
Knowledge Resources
The RTE Knowledge Resources page presents:
- a call for resources, inviting system developers to share the resources used by their own TE engines, to both help improve the TE technology and further test and evaluate such resources;
- the ablation tests carried out in the RTE challenges in order to evaluate the impact of knowledge resources and tools on TE system performances;
- lists of knowledge resources, both publicly available and unpublished, used by systems participating in the last RTE challenges.
Projects
- CoSyne EU project The Cross-Lingual Multilingual Content Synchronization with Wikis.
- EXCITEMENT EU project EXploring Customer Interactions through Textual EntailMENT.
- QALL-ME EU project Question Answering Learning technologies in a multiLingual and Multimodal Environment.
Tools
Parsers
- C&C parser for Combinatory Categorial Grammar
- Minipar
- Shallow Parser - from the University of Illinois at Urbana-Champaign, see a web demo of this tool
Role Labelling
- ASSERT
- Shalmaneser
- Semantic Role Labeler - from the University of Illinois at Urbana-Champaign, see a web demo of this tool
Entity Recognition Tools
- Illinois Named Entity Tagger - see a web demo of this tool
- Illinois Multi-lingual Named Entity Discovery Tool - see a web demo of this tool
Similarity / Relatedness Tools
- UKB: Open source WordNet-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words
Corpus Readers
- NLTK provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the Corpus Readers Guide for more information.
Related Libraries
- PyPES general purpose library containing evaluation environment for RTE and McPIET text inference engine based on the ERG (English Resource Grammar)
Text Normalizers
Java number normalizer (Beta) A tool for converting textual representations of numbers to a standard numerical string.