Difference between revisions of "Paraphrase Identification (State of the art)"

From ACL Wiki
Jump to navigation Jump to search
(29 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
* [http://research.microsoft.com/en-us/downloads/607D14D9-20CD-47E3-85BC-A2F65CD28042/default.aspx Microsoft Research Paraphrase Corpus] (MSRP)
+
* '''source''': [http://research.microsoft.com/en-us/downloads/607D14D9-20CD-47E3-85BC-A2F65CD28042/default.aspx Microsoft Research Paraphrase Corpus] (MSRP)
−
* see Dolan, Quirk, and Brockett (2004)
+
* '''task''': given a pair of sentences, classify them as paraphrases or not paraphrases
−
* train: 4076 sentence pairs (2753 positive: 67.5%)
+
* '''see''': Dolan et al. (2004)
−
* test: 1725 sentence pairs (1147 positive: 66.5%)
+
* '''train''': 4,076 sentence pairs (2,753 positive: 67.5%)
 +
* '''test''': 1,725 sentence pairs (1,147 positive: 66.5%)
  
  
 
== Sample data ==
 
== Sample data ==
  
−
* Sentence 1: Amrozi accused his brother, whom he called "the witness", of deliberately distorting his evidence.
+
* '''Sentence 1''': Amrozi accused his brother, whom he called "the witness", of deliberately distorting his evidence.
−
* Sentence 2: Referring to him as only "the witness", Amrozi accused his brother of deliberately distorting his evidence.
+
* '''Sentence 2''': Referring to him as only "the witness", Amrozi accused his brother of deliberately distorting his evidence.
−
* Class: 1 (true paraphrase)
+
* '''Class''': 1 (true paraphrase)
  
  
 
== Table of results ==
 
== Table of results ==
 +
 +
* '''Listed in order of increasing F score.'''
 +
  
 
{| border="1" cellpadding="5" cellspacing="1" width="100%"
 
{| border="1" cellpadding="5" cellspacing="1" width="100%"
Line 18: Line 22:
 
! Algorithm
 
! Algorithm
 
! Reference
 
! Reference
−
! Type
+
! Description
 +
! Supervision
 
! Accuracy
 
! Accuracy
 
! F
 
! F
 +
|-
 +
| Vector Based Similarity (Baseline)
 +
| Mihalcea et al. (2006)
 +
| cosine similarity with tf-idf weighting
 +
| unsupervised
 +
| 65.4%
 +
| 75.3%
 +
|-
 +
| KM
 +
| Kozareva and Montoyo (2006)
 +
| combination of lexical and semantic features
 +
| supervised
 +
| 76.6%
 +
| 79.6%
 +
|-
 +
| RMLMG
 +
| Rus et al. (2008)
 +
| graph subsumption
 +
| unsupervised
 +
| 70.6%
 +
| 80.5%
 
|-
 
|-
 
| MCS
 
| MCS
 
| Mihalcea et al. (2006)
 
| Mihalcea et al. (2006)
 
| combination of several word similarity measures
 
| combination of several word similarity measures
 +
| unsupervised
 
| 70.3%
 
| 70.3%
 
| 81.3%
 
| 81.3%
 +
|-
 +
| STS
 +
| Islam and Inkpen (2007)
 +
| combination of semantic and string similarity
 +
| unsupervised
 +
| 72.6%
 +
| 81.3%
 +
|-
 +
| QKC
 +
| Qiu et al. (2006)
 +
| sentence dissimilarity classification
 +
| supervised
 +
| 72.0%
 +
| 81.6%
 +
|-
 +
| matrixJcn
 +
| Fernando and Stevenson (2008)
 +
| JCN WordNet similarity with matrix
 +
| unsupervised
 +
| 74.1%
 +
| 82.4%
 +
|-
 +
| FHS
 +
| Finch et al. (2005)
 +
| combination of MT evaluation measures as features
 +
| supervised
 +
| 75.0%
 +
| 82.7%
 +
|-
 +
| WDDP
 +
| Wan et al. (2006)
 +
| dependency-based features
 +
| supervised
 +
| 75.6%
 +
| 83.0%
 +
|-
 +
| SHPNM
 +
| Socher et al. (2011)
 +
| recursive autoencoder with dynamic pooling
 +
| supervised
 +
| 76.8%
 +
| 83.6%
 +
|-
 +
| MTMETRICS
 +
| Madnani et al. (2012)
 +
| combination of eight machine translation metrics
 +
| supervised
 +
| 77.4%
 +
| 84.1%
 
|-
 
|-
 
|}
 
|}
Line 32: Line 108:
 
== References ==
 
== References ==
  
−
Dolan, B., Quirk, C., and Brockett, C. (2004). [http://acl.ldc.upenn.edu/C/C04/C04-1051.pdf Unsupervised construction of large paraphrase corpora:
+
* '''Listed alphabetically.'''
−
Exploiting massively parallel news sources], ''Proceedings of the 20th international conference on Computational Linguistics (COLING 2004)'', Geneva, Switzerland, pp. 350-356.
+
 
 +
 
 +
Dolan, B., Quirk, C., and Brockett, C. (2004). [http://acl.ldc.upenn.edu/C/C04/C04-1051.pdf Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources], ''Proceedings of the 20th international conference on Computational Linguistics (COLING 2004)'', Geneva, Switzerland, pp. 350-356.
 +
 
 +
Fernando, S., and Stevenson, M. (2008). [http://www.dcs.shef.ac.uk/~samf/clukPaper.pdf A semantic similarity approach to paraphrase detection], ''Computational Linguistics UK (CLUK 2008) 11th Annual Research Colloquium''.
 +
 
 +
Finch, A., and H, Y.S., and Sumita, E. (2005). [http://aclweb.org/anthology/I/I05/I05-5003.pdf Using machine translation evaluation techniques to determine sentence-level semantic equivalence], "Proceedings of the Third International Workshop on Paraphrasing (IWP 2005)", Jeju Island, South Korea, pp. 17-24.
 +
 
 +
Islam, A., and Inkpen, D. (2007). [http://www.site.uottawa.ca/~diana/publications/ranlp_2007_textsim_camera_ready.pdf Semantic similarity of short texts], ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2007)'', Borovets, Bulgaria, pp. 291-297.
 +
 
 +
Kozareva, Z., and Montoyo, A. (2006). [http://www.dlsi.ua.es/~zkozareva/papers/fintalKozareva.pdf Paraphrase identification on the basis of supervised machine learning techniques], ''Advances in Natural Language Processing: 5th International Conference on NLP (FinTAL 2006)'', Turku, Finland, 524-533.
 +
 
 +
Madnani, N., Tetreault, J., and Chodorow, M. (2012). [http://www.aclweb.org/anthology-new/N/N12/N12-1019.pdf Re-examining Machine Translation Metrics for Paraphrase Identification], ''Proceedings of 2012 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2012)'', pp. 182-190.
 +
 
 +
Mihalcea, R., Corley, C., and Strapparava, C. (2006). [http://reference.kfupm.edu.sa/content/c/o/corpus_based_and_knowledge_based_measure_3759629.pdf Corpus-based and knowledge-based measures of text semantic similarity], ''Proceedings of the National Conference on Artificial Intelligence (AAAI 2006)'', Boston, Massachusetts, pp. 775-780.
 +
 
 +
Qiu, L. and Kan, M.Y. and Chua, T.S. (2006). [http://acl.ldc.upenn.edu/W/W06/W06-1603.pdf Paraphrase recognition via dissimilarity significance classification], ''Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006)'', pp. 18-26.
 +
 
 +
Rus, V. and McCarthy, P.M. and Lintean, M.C. and McNamara, D.S. and Graesser, A.C. (2008). [http://csep.psyc.memphis.edu/McNamara/pdf/Paraphrase_Identification.pdf Paraphrase identification with lexico-syntactic graph subsumption], ''FLAIRS 2008'', pp. 201-206.
  
−
Mihalcea, R., Corley, C., and Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity, ''Proceedings of the National Conference on Artificial Intelligence (AAAI 2006)'', Boston, Massachusetts, pp. 775-780.
+
Socher, R. and Huang, E.H., and Pennington, J. and Ng, A.Y., and Manning, C.D. (2011). [http://www.socher.org/uploads/Main/SocherHuangPenningtonNgManning_NIPS2011.pdf Dynamic pooling and unfolding recursive autoencoders for paraphrase detection], "Advances in Neural Information Processing Systems 24"
  
 +
Wan, S., Dras, M., Dale, R., and Paris, C. (2006). [http://www.alta.asn.au/events/altw2006/proceedings/swan-final.pdf Using dependency-based features to take the "para-farce" out of paraphrase], ''Proceedings of the Australasian Language Technology Workshop (ALTW 2006)'', pp. 131-138.
 +
<!-- Please keep this list in alphabetical order -->
  
 
== See also ==
 
== See also ==

Revision as of 12:33, 26 September 2012

  • source: Microsoft Research Paraphrase Corpus (MSRP)
  • task: given a pair of sentences, classify them as paraphrases or not paraphrases
  • see: Dolan et al. (2004)
  • train: 4,076 sentence pairs (2,753 positive: 67.5%)
  • test: 1,725 sentence pairs (1,147 positive: 66.5%)


Sample data

  • Sentence 1: Amrozi accused his brother, whom he called "the witness", of deliberately distorting his evidence.
  • Sentence 2: Referring to him as only "the witness", Amrozi accused his brother of deliberately distorting his evidence.
  • Class: 1 (true paraphrase)


Table of results

  • Listed in order of increasing F score.


Algorithm Reference Description Supervision Accuracy F
Vector Based Similarity (Baseline) Mihalcea et al. (2006) cosine similarity with tf-idf weighting unsupervised 65.4% 75.3%
KM Kozareva and Montoyo (2006) combination of lexical and semantic features supervised 76.6% 79.6%
RMLMG Rus et al. (2008) graph subsumption unsupervised 70.6% 80.5%
MCS Mihalcea et al. (2006) combination of several word similarity measures unsupervised 70.3% 81.3%
STS Islam and Inkpen (2007) combination of semantic and string similarity unsupervised 72.6% 81.3%
QKC Qiu et al. (2006) sentence dissimilarity classification supervised 72.0% 81.6%
matrixJcn Fernando and Stevenson (2008) JCN WordNet similarity with matrix unsupervised 74.1% 82.4%
FHS Finch et al. (2005) combination of MT evaluation measures as features supervised 75.0% 82.7%
WDDP Wan et al. (2006) dependency-based features supervised 75.6% 83.0%
SHPNM Socher et al. (2011) recursive autoencoder with dynamic pooling supervised 76.8% 83.6%
MTMETRICS Madnani et al. (2012) combination of eight machine translation metrics supervised 77.4% 84.1%

References

  • Listed alphabetically.


Dolan, B., Quirk, C., and Brockett, C. (2004). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources, Proceedings of the 20th international conference on Computational Linguistics (COLING 2004), Geneva, Switzerland, pp. 350-356.

Fernando, S., and Stevenson, M. (2008). A semantic similarity approach to paraphrase detection, Computational Linguistics UK (CLUK 2008) 11th Annual Research Colloquium.

Finch, A., and H, Y.S., and Sumita, E. (2005). Using machine translation evaluation techniques to determine sentence-level semantic equivalence, "Proceedings of the Third International Workshop on Paraphrasing (IWP 2005)", Jeju Island, South Korea, pp. 17-24.

Islam, A., and Inkpen, D. (2007). Semantic similarity of short texts, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2007), Borovets, Bulgaria, pp. 291-297.

Kozareva, Z., and Montoyo, A. (2006). Paraphrase identification on the basis of supervised machine learning techniques, Advances in Natural Language Processing: 5th International Conference on NLP (FinTAL 2006), Turku, Finland, 524-533.

Madnani, N., Tetreault, J., and Chodorow, M. (2012). Re-examining Machine Translation Metrics for Paraphrase Identification, Proceedings of 2012 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2012), pp. 182-190.

Mihalcea, R., Corley, C., and Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity, Proceedings of the National Conference on Artificial Intelligence (AAAI 2006), Boston, Massachusetts, pp. 775-780.

Qiu, L. and Kan, M.Y. and Chua, T.S. (2006). Paraphrase recognition via dissimilarity significance classification, Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pp. 18-26.

Rus, V. and McCarthy, P.M. and Lintean, M.C. and McNamara, D.S. and Graesser, A.C. (2008). Paraphrase identification with lexico-syntactic graph subsumption, FLAIRS 2008, pp. 201-206.

Socher, R. and Huang, E.H., and Pennington, J. and Ng, A.Y., and Manning, C.D. (2011). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection, "Advances in Neural Information Processing Systems 24"

Wan, S., Dras, M., Dale, R., and Paris, C. (2006). Using dependency-based features to take the "para-farce" out of paraphrase, Proceedings of the Australasian Language Technology Workshop (ALTW 2006), pp. 131-138.

See also

Retrieved from "https://aclweb.org/aclwiki/index.php?title=Paraphrase_Identification_(State_of_the_art)&oldid=9757"