ACL Wiki - User contributions [en]

Question Answering (State of the art)

2017-02-14T02:28:54Z

Zhiguo Wang:

== Answer Sentence Selection ==

The task of answer sentence selection is designed for the open-domain question answering setting. Given a question and a set of candidate sentences, the task is to choose the correct sentence that contains the exact answer and can sufficiently support the answer choice.

* [http://cs.stanford.edu/people/mengqiu/data/qg-emnlp07-data.tgz QA Answer Sentence Selection Dataset]: labeled sentences using TREC QA track data, provided by [http://cs.stanford.edu/people/mengqiu/ Mengqiu Wang] and first used in [http://www.aclweb.org/anthology/D/D07/D07-1003.pdf Wang et al. (2007)].
* Over time, the original dataset diverged to two versions due to different pre-processing in recent publications: both have the same training set but their development and test sets differ. The Raw version has 82 questions in the development set and 100 questions in the test set; The Clean version (Wang and Ittycheriah et al. 2015, Tan et al. 2015, dos Santos et al. 2016, Wang et al. 2016) removed questions with no answers or with only positive/negative answers, thus has only 65 questions in the development set and 68 questions in the test set.
* Note: MAP/MRR scores on the two versions of TREC QA data (Clean vs Raw) are not comparable according to [https://dl.acm.org/authorize.cfm?key=N27026 Rao et al. (2016)].

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm - Raw Version of TREC QA
! Reference
! [http://en.wikipedia.org/wiki/Mean_average_precision MAP]
! [http://en.wikipedia.org/wiki/Mean_reciprocal_rank MRR]
|-
| Punyakanok (2004)
| Wang et al. (2007)
| 0.419
| 0.494
|-
| Cui (2005)
| Wang et al. (2007)
| 0.427
| 0.526
|-
| Wang (2007)
| Wang et al. (2007)
| 0.603
| 0.685
|-
| H&S (2010)
| Heilman and Smith (2010)
| 0.609
| 0.692
|-
| W&M (2010)
| Wang and Manning (2010)
| 0.595
| 0.695
|-
| Yao (2013)
| Yao et al. (2013)
| 0.631
| 0.748
|-
| S&M (2013)
| Severyn and Moschitti (2013)
| 0.678
| 0.736
|-
| Shnarch (2013) - Backward
| Shnarch (2013)
| 0.686
| 0.754
|-
| Yih (2013) - LCLR
| Yih et al. (2013)
| 0.709
| 0.770
|-
| Yu (2014) - TRAIN-ALL bigram+count
| Yu et al. (2014)
| 0.711
| 0.785
|-
| W&N (2015) - Three-Layer BLSTM+BM25
| Wang and Nyberg (2015)
| 0.713
| 0.791
|-
| Feng (2015) - Architecture-II
| Tan et al. (2015)
| 0.711
| 0.800
|-
| S&M (2015)
| Severyn and Moschitti (2015)
| 0.746
| 0.808
|-
| Yang (2016) - Attention-Based Neural Matching Model
| Yang et al. (2016)
| 0.750
| 0.811
|-
| H&L (2016) - Pairwise Word Interaction Modelling
| He and Lin (2016)
| 0.758
| 0.822
|-
| H&L (2015) - Multi-Perspective CNN
| He and Lin (2015)
| 0.762
| 0.830
|-
| Rao (2016) - PairwiseRank + Multi-Perspective CNN
| Rao et al. (2016)
| 0.780
| 0.834
|}

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm - Clean Version of TREC QA
! Reference
! [http://en.wikipedia.org/wiki/Mean_average_precision MAP]
! [http://en.wikipedia.org/wiki/Mean_reciprocal_rank MRR]
|-
| W&I (2015)
| Wang and Ittycheriah (2015)
| 0.746
| 0.820
|-
| Tan (2015) - QA-LSTM/CNN+attention
| Tan et al. (2015)
| 0.728
| 0.832
|-
| dos Santos (2016) - Attentive Pooling CNN
| dos Santos et al. (2016)
| 0.753
| 0.851
|-
| Wang et al. (2016) - L.D.C Model
| Wang et al. (2016)
| 0.771
| 0.845
|-
| H&L (2015) - Multi-Perspective CNN
| He and Lin (2015)
| 0.777
| 0.836
|-
| Rao et al. (2016) - PairwiseRank + Multi-Perspective CNN
| Rao et al. (2016)
| 0.801
| 0.877
|-
| Wang et al. (2017) - BiMPM
| Wang et al. (2017)
| 0.802
| 0.875
|}

== References ==
* Vasin Punyakanok, Dan Roth, and Wen-Tau Yih. 2004. [http://cogcomp.cs.illinois.edu/papers/PunyakanokRoYi04a.pdf Mapping dependencies trees: An application to question answering]. In Proceedings of the 8th International Symposium on Artificial Intelligence and Mathematics, Fort Lauderdale, FL, USA.
* Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and Tat-Seng Chua. 2005. [http://ws.csie.ncku.edu.tw/login/upload/2005/paper/Question%20answering%20Question%20answering%20passage%20retrieval%20using%20dependency%20relations.pdf Question answering passage retrieval using dependency relations]. In Proceedings of the 28th ACM-SIGIR International Conference on Research and Development in Information Retrieval, Salvador, Brazil.
* Wang, Mengqiu and Smith, Noah A. and Mitamura, Teruko. 2007. [http://www.aclweb.org/anthology/D/D07/D07-1003.pdf What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA]. In EMNLP-CoNLL 2007.
* Heilman, Michael and Smith, Noah A. 2010. [http://www.aclweb.org/anthology/N10-1145 Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions]. In NAACL-HLT 2010.
* Wang, Mengqiu and Manning, Christopher. 2010. [http://aclweb.org/anthology//C/C10/C10-1131.pdf Probabilistic Tree-Edit Models with Structured Latent Variables for Textual Entailment and Question Answering]. In COLING 2010.
* E. Shnarch. 2013. Probabilistic Models for Lexical Inference. Ph.D. thesis, Bar Ilan University.
* Yao, Xuchen and Van Durme, Benjamin and Callison-Burch, Chris and Clark, Peter. 2013. [http://www.aclweb.org/anthology/N13-1106.pdf Answer Extraction as Sequence Tagging with Tree Edit Distance]. In NAACL-HLT 2013.
* Yih, Wen-tau and Chang, Ming-Wei and Meek, Christopher and Pastusiak, Andrzej. 2013. [http://research.microsoft.com/pubs/192357/QA-SentSel-Updated-PostACL.pdf Question Answering Using Enhanced Lexical Semantic Models]. In ACL 2013.
* Severyn, Aliaksei and Moschitti, Alessandro. 2013. [http://www.aclweb.org/anthology/D13-1044.pdf Automatic Feature Engineering for Answer Selection and Extraction]. In EMNLP 2013.
* Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. [http://arxiv.org/pdf/1412.1632v1.pdf Deep Learning for Answer Sentence Selection]. In NIPS deep learning workshop.
* Di Wang and Eric Nyberg. 2015. [http://www.aclweb.org/anthology/P15-2116 A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering]. In ACL 2015.
* Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, Bowen Zhou. 2015. [http://arxiv.org/abs/1508.01585 Applying deep learning to answer selection: A study and an open task]. In ASRU 2015.
* Aliaksei Severyn and Alessandro Moschitti. 2015. [http://disi.unitn.it/~severyn/papers/sigir-2015-long.pdf Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks]. In SIGIR 2015.
* Zhiguo Wang and Abraham Ittycheriah. 2015. [http://arxiv.org/abs/1507.02628 FAQ-based Question Answering via Word Alignment]. In eprint arXiv:1507.02628.
* Ming Tan, Cicero dos Santos, Bing Xiang & Bowen Zhou. 2015. [http://arxiv.org/abs/1511.04108 LSTM-Based Deep Learning Models for Nonfactoid Answer Selection]. In eprint arXiv:1511.04108.
* Cicero dos Santos, Ming Tan, Bing Xiang & Bowen Zhou. 2016. [http://arxiv.org/abs/1602.03609 Attentive Pooling Networks]. In eprint arXiv:1602.03609.
* Zhiguo Wang, Haitao Mi and Abraham Ittycheriah. 2016. [http://arxiv.org/pdf/1602.07019v1.pdf Sentence Similarity Learning by Lexical Decomposition and Composition]. In Coling 2016.
* Hua He, Kevin Gimpel and Jimmy Lin. 2015. [http://aclweb.org/anthology/D/D15/D15-1181.pdf Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks]. In EMNLP 2015.
* Hua He and Jimmy Lin. 2016. [https://cs.uwaterloo.ca/~jimmylin/publications/He_etal_NAACL-HTL2016.pdf Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement]. In NAACL 2016.
* Liu Yang, Qingyao Ai, Jiafeng Guo, W. Bruce Croft. 2016. [http://maroo.cs.umass.edu/pub/web/getpdf.php?id=1240 aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model]. In CIKM 2016.
* Jinfeng Rao, Hua He and Jimmy Lin. 2016. [https://dl.acm.org/authorize.cfm?key=N27026 Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks]. In CIKM 2016.
[[Category:State of the art]]
* Zhiguo Wang, Wael Hamza and Radu Florian. 2017. [https://arxiv.org/pdf/1702.03814.pdf Bilateral Multi-Perspective Matching for Natural Language Sentences]. In eprint arXiv:1702.03814.

Paraphrase Identification (State of the art)

2016-11-26T13:02:14Z

Zhiguo Wang:

* '''source''': [http://research.microsoft.com/en-us/downloads/607D14D9-20CD-47E3-85BC-A2F65CD28042/default.aspx Microsoft Research Paraphrase Corpus] (MSRP)
* '''task''': given a pair of sentences, classify them as paraphrases or not paraphrases
* '''see''': Dolan et al. (2004)
* '''train''': 4,076 sentence pairs (2,753 positive: 67.5%)
* '''test''': 1,725 sentence pairs (1,147 positive: 66.5%)
* '''see also:''' [[Similarity (State of the art)]]

== Sample data ==

* '''Sentence 1''': Amrozi accused his brother, whom he called "the witness", of deliberately distorting his evidence.
* '''Sentence 2''': Referring to him as only "the witness", Amrozi accused his brother of deliberately distorting his evidence.
* '''Class''': 1 (true paraphrase)

== Table of results ==

* '''Listed in order of increasing F score.'''

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! Algorithm
! Reference
! Description
! Supervision
! Accuracy
! F
|-
| Vector Based Similarity (Baseline)
| Mihalcea et al. (2006)
| cosine similarity with tf-idf weighting
| unsupervised
| 65.4%
| 75.3%
|-
| ESA
| Hassan (2011)
| explicit semantic space
| unsupervised
| 67.0%
| 79.3%
|-
| KM
| Kozareva and Montoyo (2006)
| combination of lexical and semantic features
| supervised
| 76.6%
| 79.6%
|-
| LSA
| Hassan (2011)
| latent semantic space
| unsupervised
| 68.8%
| 79.9%
|-
| RMLMG
| Rus et al. (2008)
| graph subsumption
| unsupervised
| 70.6%
| 80.5%
|-
| MCS
| Mihalcea et al. (2006)
| combination of several word similarity measures
| unsupervised
| 70.3%
| 81.3%
|-
| STS
| Islam and Inkpen (2007)
| combination of semantic and string similarity
| unsupervised
| 72.6%
| 81.3%
|-
| SSA
| Hassan (2011)
| salient semantic space
| unsupervised
| 72.5%
| 81.4%
|-
| QKC
| Qiu et al. (2006)
| sentence dissimilarity classification
| supervised
| 72.0%
| 81.6%
|-
| ParaDetect
| Zia and Wasif (2012)
| PI using semantic heuristic features
| supervised
| 74.7%
| 81.8%
|-
| Vector-based similarity
| Milajevs et al. (2014)
| Additive composition of vectors and cosine distance
| unsupervised
| 73.0%
| 82.0%
|-
| SDS
| Blacoe and Lapata (2012)
| simple distributional semantic space
| supervised
| 73.0%
| 82.3%
|-
| matrixJcn
| Fernando and Stevenson (2008)
| JCN WordNet similarity with matrix
| unsupervised
| 74.1%
| 82.4%
|-
| FHS
| Finch et al. (2005)
| combination of MT evaluation measures as features
| supervised
| 75.0%
| 82.7%
|-
| PE
| Das and Smith (2009)
| product of experts
| supervised
| 76.1%
| 82.7%
|-
| WDDP
| Wan et al. (2006)
| dependency-based features
| supervised
| 75.6%
| 83.0%
|-
| SHPNM
| Socher et al. (2011)
| recursive autoencoder with dynamic pooling
| supervised
| 76.8%
| 83.6%
|-
| MTMETRICS
| Madnani et al. (2012)
| combination of eight machine translation metrics
| supervised
| 77.4%
| 84.1%
|-
| L.D.C Model
| Wang et al. (2016)
| Sentence Similarity Learning by Lexical Decomposition and Composition
| supervised
| 78.4%
| 84.7%
|-
| Multi-Perspective CNN
| He et al. (2015)
| Multi-perspective Convolutional NNs and structured similarity layer
| supervised
| 78.6%
| 84.7%
|-
| SAMS-RecNN
| Cheng and Kartsaklis (2015)
| Recursive NNs using syntax-aware multi-sense word embeddings
| supervised
| 78.6%
| 85.3%
|-
| TF-KLD
| Ji and Eisenstein (2013)
| Matrix factorization with supervised reweighting
| supervised
| 80.4%
| 85.9%
|}

== References ==

* '''Listed alphabetically.'''

Blacoe, W. and Lapata, M. (2012). [http://newdesign.aclweb.org/anthology/D/D12/D12-1050.pdf A comparison of vector-based representations for semantic composition], ''Proceedings of EMNLP'', Jeju Island, Korea, pp. 546-556.

Cheng, J. and Kartsaklis, D. (2015). [http://www.aclweb.org/anthology/D/D15/D15-1177.pdf Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning], ''Proceedings of EMNLP 2015'', Lisbon, Portugal, pp. 1531-1542.

Das, D., and Smith, N. (2009). [http://www.aclweb.org/anthology-new/P/P09/P09-1053.pdf Paraphrase identification as probabilistic quasi-synchronous recognition]. ''Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP'', pp. 468-476, Suntec, Singapore.

Dolan, B., Quirk, C., and Brockett, C. (2004). [http://acl.ldc.upenn.edu/C/C04/C04-1051.pdf Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources], ''Proceedings of the 20th international conference on Computational Linguistics (COLING 2004)'', Geneva, Switzerland, pp. 350-356.

Fernando, S., and Stevenson, M. (2008). [http://staffwww.dcs.shef.ac.uk/people/S.Fernando/pubs/clukPaper.pdf A semantic similarity approach to paraphrase detection], ''Computational Linguistics UK (CLUK 2008) 11th Annual Research Colloquium''.

Finch, A., and H, Y.S., and Sumita, E. (2005). [http://aclweb.org/anthology/I/I05/I05-5003.pdf Using machine translation evaluation techniques to determine sentence-level semantic equivalence], "Proceedings of the Third International Workshop on Paraphrasing (IWP 2005)", Jeju Island, South Korea, pp. 17-24.

Hassan, Samer. [http://samerhassan.com/images/0/01/Dissertation.pdf Measuring Semantic Relatedness Using Salient Encyclopedic Concepts]. Doctor of Philosophy, August 2011

He, Hua, Gimpel K. and Lin J. (2015). [http://aclweb.org/anthology/D/D15/D15-1181.pdf Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks], ''Proceedings of EMNLP 2015'', Lisbon, Portugal, pp. 1576-1586.

Islam, A., and Inkpen, D. (2007). [http://www.site.uottawa.ca/~diana/publications/ranlp_2007_textsim_camera_ready.pdf Semantic similarity of short texts], ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2007)'', Borovets, Bulgaria, pp. 291-297.

Ji, Y. and Eisenstein, J. (2013) [http://www.aclweb.org/anthology/D/D13/D13-1090.pdf Discriminative Improvements to Distributional Sentence Similarity],
''Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2013)'', Seattle, Washington, USA, pp. 891--896

Kozareva, Z., and Montoyo, A. (2006). [http://www.dlsi.ua.es/~zkozareva/papers/fintalKozareva.pdf Paraphrase identification on the basis of supervised machine learning techniques], ''Advances in Natural Language Processing: 5th International Conference on NLP (FinTAL 2006)'', Turku, Finland, 524-533.

Madnani, N., Tetreault, J., and Chodorow, M. (2012). [http://www.aclweb.org/anthology-new/N/N12/N12-1019.pdf Re-examining Machine Translation Metrics for Paraphrase Identification], ''Proceedings of 2012 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2012)'', pp. 182-190.

Mihalcea, R., Corley, C., and Strapparava, C. (2006). [http://www.cse.unt.edu/~rada/papers/mihalcea.aaai06.pdf Corpus-based and knowledge-based measures of text semantic similarity], ''Proceedings of the National Conference on Artificial Intelligence (AAAI 2006)'', Boston, Massachusetts, pp. 775-780.

Milajevs, D., Kartsaklis, D., Sadrzadeh, M. and Purver, M. (2014). [https://aclweb.org/anthology/D/D14/D14-1079.pdf Evaluating Neural Word Representations in Tensor-Based Compositional Settings], ''Proceedings of EMNLP 2014'', Doha, Qatar, pp. 708–719.

Qiu, L. and Kan, M.Y. and Chua, T.S. (2006). [http://acl.ldc.upenn.edu/W/W06/W06-1603.pdf Paraphrase recognition via dissimilarity significance classification], ''Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006)'', pp. 18-26.

Rus, V. and McCarthy, P.M. and Lintean, M.C. and McNamara, D.S. and Graesser, A.C. (2008). [http://csep.psyc.memphis.edu/McNamara/pdf/Paraphrase_Identification.pdf Paraphrase identification with lexico-syntactic graph subsumption], ''FLAIRS 2008'', pp. 201-206.

Socher, R. and Huang, E.H., and Pennington, J. and Ng, A.Y., and Manning, C.D. (2011). [http://www.socher.org/uploads/Main/SocherHuangPenningtonNgManning_NIPS2011.pdf Dynamic pooling and unfolding recursive autoencoders for paraphrase detection], "Advances in Neural Information Processing Systems 24"

Wan, S., Dras, M., Dale, R., and Paris, C. (2006). [http://www.alta.asn.au/events/altw2006/proceedings/swan-final.pdf Using dependency-based features to take the "para-farce" out of paraphrase], ''Proceedings of the Australasian Language Technology Workshop (ALTW 2006)'', pp. 131-138.

Zia Ul-Qayyum and Wasif Altaf, (2012). [http://maxwellsci.com/print/rjaset/v4-4894-4904.pdf Paraphrase Identification using Semantic Heuristic Features], ''Research Journal of Applied Sciences, Engineering and Technology'', 4(22): 4894-4904.

Zhiguo Wang, Haitao Mi and Abraham Ittycheriah. 2016. [http://arxiv.org/pdf/1602.07019v1.pdf Sentence Similarity Learning by Lexical Decomposition and Composition]. In Coling 2016.



[[Category:State of the art]]
[[Category:Similarity]]

Paraphrase Identification (State of the art)

2016-11-26T13:00:39Z

Zhiguo Wang:

* '''source''': [http://research.microsoft.com/en-us/downloads/607D14D9-20CD-47E3-85BC-A2F65CD28042/default.aspx Microsoft Research Paraphrase Corpus] (MSRP)
* '''task''': given a pair of sentences, classify them as paraphrases or not paraphrases
* '''see''': Dolan et al. (2004)
* '''train''': 4,076 sentence pairs (2,753 positive: 67.5%)
* '''test''': 1,725 sentence pairs (1,147 positive: 66.5%)
* '''see also:''' [[Similarity (State of the art)]]

== Sample data ==

* '''Sentence 1''': Amrozi accused his brother, whom he called "the witness", of deliberately distorting his evidence.
* '''Sentence 2''': Referring to him as only "the witness", Amrozi accused his brother of deliberately distorting his evidence.
* '''Class''': 1 (true paraphrase)

== Table of results ==

* '''Listed in order of increasing F score.'''

{| border="1" cellpadding="5" cellspacing="1" width="100%"
|-
! Algorithm
! Reference
! Description
! Supervision
! Accuracy
! F
|-
| Vector Based Similarity (Baseline)
| Mihalcea et al. (2006)
| cosine similarity with tf-idf weighting
| unsupervised
| 65.4%
| 75.3%
|-
| ESA
| Hassan (2011)
| explicit semantic space
| unsupervised
| 67.0%
| 79.3%
|-
| KM
| Kozareva and Montoyo (2006)
| combination of lexical and semantic features
| supervised
| 76.6%
| 79.6%
|-
| LSA
| Hassan (2011)
| latent semantic space
| unsupervised
| 68.8%
| 79.9%
|-
| RMLMG
| Rus et al. (2008)
| graph subsumption
| unsupervised
| 70.6%
| 80.5%
|-
| MCS
| Mihalcea et al. (2006)
| combination of several word similarity measures
| unsupervised
| 70.3%
| 81.3%
|-
| STS
| Islam and Inkpen (2007)
| combination of semantic and string similarity
| unsupervised
| 72.6%
| 81.3%
|-
| SSA
| Hassan (2011)
| salient semantic space
| unsupervised
| 72.5%
| 81.4%
|-
| QKC
| Qiu et al. (2006)
| sentence dissimilarity classification
| supervised
| 72.0%
| 81.6%
|-
| ParaDetect
| Zia and Wasif (2012)
| PI using semantic heuristic features
| supervised
| 74.7%
| 81.8%
|-
| Vector-based similarity
| Milajevs et al. (2014)
| Additive composition of vectors and cosine distance
| unsupervised
| 73.0%
| 82.0%
|-
| SDS
| Blacoe and Lapata (2012)
| simple distributional semantic space
| supervised
| 73.0%
| 82.3%
|-
| matrixJcn
| Fernando and Stevenson (2008)
| JCN WordNet similarity with matrix
| unsupervised
| 74.1%
| 82.4%
|-
| FHS
| Finch et al. (2005)
| combination of MT evaluation measures as features
| supervised
| 75.0%
| 82.7%
|-
| PE
| Das and Smith (2009)
| product of experts
| supervised
| 76.1%
| 82.7%
|-
| WDDP
| Wan et al. (2006)
| dependency-based features
| supervised
| 75.6%
| 83.0%
|-
| SHPNM
| Socher et al. (2011)
| recursive autoencoder with dynamic pooling
| supervised
| 76.8%
| 83.6%
|-
| MTMETRICS
| Madnani et al. (2012)
| combination of eight machine translation metrics
| supervised
| 77.4%
| 84.1%
-
| L.D.C Model
| Wang et al. (2016)
| Sentence Similarity Learning by Lexical Decomposition and Composition
| supervised
| 78.4%
| 84.7%
|-
| Multi-Perspective CNN
| He et al. (2015)
| Multi-perspective Convolutional NNs and structured similarity layer
| supervised
| 78.6%
| 84.7%
|-
| SAMS-RecNN
| Cheng and Kartsaklis (2015)
| Recursive NNs using syntax-aware multi-sense word embeddings
| supervised
| 78.6%
| 85.3%
|-
| TF-KLD
| Ji and Eisenstein (2013)
| Matrix factorization with supervised reweighting
| supervised
| 80.4%
| 85.9%
|}

== References ==

* '''Listed alphabetically.'''

Blacoe, W. and Lapata, M. (2012). [http://newdesign.aclweb.org/anthology/D/D12/D12-1050.pdf A comparison of vector-based representations for semantic composition], ''Proceedings of EMNLP'', Jeju Island, Korea, pp. 546-556.

Cheng, J. and Kartsaklis, D. (2015). [http://www.aclweb.org/anthology/D/D15/D15-1177.pdf Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning], ''Proceedings of EMNLP 2015'', Lisbon, Portugal, pp. 1531-1542.

Das, D., and Smith, N. (2009). [http://www.aclweb.org/anthology-new/P/P09/P09-1053.pdf Paraphrase identification as probabilistic quasi-synchronous recognition]. ''Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP'', pp. 468-476, Suntec, Singapore.

Dolan, B., Quirk, C., and Brockett, C. (2004). [http://acl.ldc.upenn.edu/C/C04/C04-1051.pdf Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources], ''Proceedings of the 20th international conference on Computational Linguistics (COLING 2004)'', Geneva, Switzerland, pp. 350-356.

Fernando, S., and Stevenson, M. (2008). [http://staffwww.dcs.shef.ac.uk/people/S.Fernando/pubs/clukPaper.pdf A semantic similarity approach to paraphrase detection], ''Computational Linguistics UK (CLUK 2008) 11th Annual Research Colloquium''.

Finch, A., and H, Y.S., and Sumita, E. (2005). [http://aclweb.org/anthology/I/I05/I05-5003.pdf Using machine translation evaluation techniques to determine sentence-level semantic equivalence], "Proceedings of the Third International Workshop on Paraphrasing (IWP 2005)", Jeju Island, South Korea, pp. 17-24.

Hassan, Samer. [http://samerhassan.com/images/0/01/Dissertation.pdf Measuring Semantic Relatedness Using Salient Encyclopedic Concepts]. Doctor of Philosophy, August 2011

He, Hua, Gimpel K. and Lin J. (2015). [http://aclweb.org/anthology/D/D15/D15-1181.pdf Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks], ''Proceedings of EMNLP 2015'', Lisbon, Portugal, pp. 1576-1586.

Islam, A., and Inkpen, D. (2007). [http://www.site.uottawa.ca/~diana/publications/ranlp_2007_textsim_camera_ready.pdf Semantic similarity of short texts], ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2007)'', Borovets, Bulgaria, pp. 291-297.

Ji, Y. and Eisenstein, J. (2013) [http://www.aclweb.org/anthology/D/D13/D13-1090.pdf Discriminative Improvements to Distributional Sentence Similarity],
''Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2013)'', Seattle, Washington, USA, pp. 891--896

Kozareva, Z., and Montoyo, A. (2006). [http://www.dlsi.ua.es/~zkozareva/papers/fintalKozareva.pdf Paraphrase identification on the basis of supervised machine learning techniques], ''Advances in Natural Language Processing: 5th International Conference on NLP (FinTAL 2006)'', Turku, Finland, 524-533.

Madnani, N., Tetreault, J., and Chodorow, M. (2012). [http://www.aclweb.org/anthology-new/N/N12/N12-1019.pdf Re-examining Machine Translation Metrics for Paraphrase Identification], ''Proceedings of 2012 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2012)'', pp. 182-190.

Mihalcea, R., Corley, C., and Strapparava, C. (2006). [http://www.cse.unt.edu/~rada/papers/mihalcea.aaai06.pdf Corpus-based and knowledge-based measures of text semantic similarity], ''Proceedings of the National Conference on Artificial Intelligence (AAAI 2006)'', Boston, Massachusetts, pp. 775-780.

Milajevs, D., Kartsaklis, D., Sadrzadeh, M. and Purver, M. (2014). [https://aclweb.org/anthology/D/D14/D14-1079.pdf Evaluating Neural Word Representations in Tensor-Based Compositional Settings], ''Proceedings of EMNLP 2014'', Doha, Qatar, pp. 708–719.

Qiu, L. and Kan, M.Y. and Chua, T.S. (2006). [http://acl.ldc.upenn.edu/W/W06/W06-1603.pdf Paraphrase recognition via dissimilarity significance classification], ''Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006)'', pp. 18-26.

Rus, V. and McCarthy, P.M. and Lintean, M.C. and McNamara, D.S. and Graesser, A.C. (2008). [http://csep.psyc.memphis.edu/McNamara/pdf/Paraphrase_Identification.pdf Paraphrase identification with lexico-syntactic graph subsumption], ''FLAIRS 2008'', pp. 201-206.

Socher, R. and Huang, E.H., and Pennington, J. and Ng, A.Y., and Manning, C.D. (2011). [http://www.socher.org/uploads/Main/SocherHuangPenningtonNgManning_NIPS2011.pdf Dynamic pooling and unfolding recursive autoencoders for paraphrase detection], "Advances in Neural Information Processing Systems 24"

Wan, S., Dras, M., Dale, R., and Paris, C. (2006). [http://www.alta.asn.au/events/altw2006/proceedings/swan-final.pdf Using dependency-based features to take the "para-farce" out of paraphrase], ''Proceedings of the Australasian Language Technology Workshop (ALTW 2006)'', pp. 131-138.

Zia Ul-Qayyum and Wasif Altaf, (2012). [http://maxwellsci.com/print/rjaset/v4-4894-4904.pdf Paraphrase Identification using Semantic Heuristic Features], ''Research Journal of Applied Sciences, Engineering and Technology'', 4(22): 4894-4904.

Zhiguo Wang, Haitao Mi and Abraham Ittycheriah. 2016. [http://arxiv.org/pdf/1602.07019v1.pdf Sentence Similarity Learning by Lexical Decomposition and Composition]. In Coling 2016.



[[Category:State of the art]]
[[Category:Similarity]]

Question Answering (State of the art)

2016-11-20T15:43:09Z

Zhiguo Wang:

== Answer Sentence Selection ==

The task of answer sentence selection is designed for the open-domain question answering setting. Given a question and a set of candidate sentences, the task is to choose the correct sentence that contains the exact answer and can sufficiently support the answer choice.

* [http://cs.stanford.edu/people/mengqiu/data/qg-emnlp07-data.tgz QA Answer Sentence Selection Dataset]: labeled sentences using TREC QA track data, provided by [http://cs.stanford.edu/people/mengqiu/ Mengqiu Wang] and first used in [http://www.aclweb.org/anthology/D/D07/D07-1003.pdf Wang et al. (2007)].
* Over time, the original dataset diverged to two versions due to different pre-processing in recent publications: both have the same training set but their development and test sets differ. The Raw version has 82 questions in the development set and 100 questions in the test set; The Clean version (Wang and Ittycheriah et al. 2015, Tan et al. 2015, dos Santos et al. 2016, Wang et al. 2016) removed questions with no answers or with only positive/negative answers, thus has only 65 questions in the development set and 68 questions in the test set.
* Note: MAP/MRR scores on the two versions of TREC QA data (Clean vs Raw) are not comparable according to [http://www.cs.umd.edu/~jinfeng/publications/PairwiseNeuralNetwork_CIKM2016.pdf Rao et al. (2016)].

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm - Raw Version of TREC QA
! Reference
! [http://en.wikipedia.org/wiki/Mean_average_precision MAP]
! [http://en.wikipedia.org/wiki/Mean_reciprocal_rank MRR]
|-
| Punyakanok (2004)
| Wang et al. (2007)
| 0.419
| 0.494
|-
| Cui (2005)
| Wang et al. (2007)
| 0.427
| 0.526
|-
| Wang (2007)
| Wang et al. (2007)
| 0.603
| 0.685
|-
| H&S (2010)
| Heilman and Smith (2010)
| 0.609
| 0.692
|-
| W&M (2010)
| Wang and Manning (2010)
| 0.595
| 0.695
|-
| Yao (2013)
| Yao et al. (2013)
| 0.631
| 0.748
|-
| S&M (2013)
| Severyn and Moschitti (2013)
| 0.678
| 0.736
|-
| Shnarch (2013) - Backward
| Shnarch (2013)
| 0.686
| 0.754
|-
| Yih (2013) - LCLR
| Yih et al. (2013)
| 0.709
| 0.770
|-
| Yu (2014) - TRAIN-ALL bigram+count
| Yu et al. (2014)
| 0.711
| 0.785
|-
| W&N (2015) - Three-Layer BLSTM+BM25
| Wang and Nyberg (2015)
| 0.713
| 0.791
|-
| Feng (2015) - Architecture-II
| Tan et al. (2015)
| 0.711
| 0.800
|-
| S&M (2015)
| Severyn and Moschitti (2015)
| 0.746
| 0.808
|-
| H&L (2016) - Pairwise Word Interaction Modelling
| He and Lin (2016)
| 0.758
| 0.822
|-
| H&L (2015) - Multi-Perspective CNN
| He and Lin (2015)
| 0.762
| 0.830
|-
| Rao (2016) - PairwiseRank + Multi-Perspective CNN
| Rao et al. (2016)
| 0.780
| 0.834
|}

{| border="1" cellpadding="5" cellspacing="1"
|-
! Algorithm - Clean Version of TREC QA
! Reference
! [http://en.wikipedia.org/wiki/Mean_average_precision MAP]
! [http://en.wikipedia.org/wiki/Mean_reciprocal_rank MRR]
|-
| W&I (2015)
| Wang and Ittycheriah (2015)
| 0.746
| 0.820
|-
| Tan (2015) - QA-LSTM/CNN+attention
| Tan et al. (2015)
| 0.728
| 0.832
|-
| dos Santos (2016) - Attentive Pooling CNN
| dos Santos et al. (2016)
| 0.753
| 0.851
|-
| Wang et al. (2016) - L.D.C Model
| Wang et al. (2016)
| 0.771
| 0.845
|-
| H&L (2015) - Multi-Perspective CNN
| He and Lin (2015)
| 0.777
| 0.836
|-
| Rao et al. (2016) - PairwiseRank + Multi-Perspective CNN
| Rao et al. (2016)
| 0.801
| 0.877
|}

== References ==
* Vasin Punyakanok, Dan Roth, and Wen-Tau Yih. 2004. [http://cogcomp.cs.illinois.edu/papers/PunyakanokRoYi04a.pdf Mapping dependencies trees: An application to question answering]. In Proceedings of the 8th International Symposium on Artificial Intelligence and Mathematics, Fort Lauderdale, FL, USA.
* Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and Tat-Seng Chua. 2005. [http://ws.csie.ncku.edu.tw/login/upload/2005/paper/Question%20answering%20Question%20answering%20passage%20retrieval%20using%20dependency%20relations.pdf Question answering passage retrieval using dependency relations]. In Proceedings of the 28th ACM-SIGIR International Conference on Research and Development in Information Retrieval, Salvador, Brazil.
* Wang, Mengqiu and Smith, Noah A. and Mitamura, Teruko. 2007. [http://www.aclweb.org/anthology/D/D07/D07-1003.pdf What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA]. In EMNLP-CoNLL 2007.
* Heilman, Michael and Smith, Noah A. 2010. [http://www.aclweb.org/anthology/N10-1145 Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions]. In NAACL-HLT 2010.
* Wang, Mengqiu and Manning, Christopher. 2010. [http://aclweb.org/anthology//C/C10/C10-1131.pdf Probabilistic Tree-Edit Models with Structured Latent Variables for Textual Entailment and Question Answering]. In COLING 2010.
* E. Shnarch. 2013. Probabilistic Models for Lexical Inference. Ph.D. thesis, Bar Ilan University.
* Yao, Xuchen and Van Durme, Benjamin and Callison-Burch, Chris and Clark, Peter. 2013. [http://www.aclweb.org/anthology/N13-1106.pdf Answer Extraction as Sequence Tagging with Tree Edit Distance]. In NAACL-HLT 2013.
* Yih, Wen-tau and Chang, Ming-Wei and Meek, Christopher and Pastusiak, Andrzej. 2013. [http://research.microsoft.com/pubs/192357/QA-SentSel-Updated-PostACL.pdf Question Answering Using Enhanced Lexical Semantic Models]. In ACL 2013.
* Severyn, Aliaksei and Moschitti, Alessandro. 2013. [http://www.aclweb.org/anthology/D13-1044.pdf Automatic Feature Engineering for Answer Selection and Extraction]. In EMNLP 2013.
* Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. [http://arxiv.org/pdf/1412.1632v1.pdf Deep Learning for Answer Sentence Selection]. In NIPS deep learning workshop.
* Di Wang and Eric Nyberg. 2015. [http://www.aclweb.org/anthology/P15-2116 A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering]. In ACL 2015.
* Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, Bowen Zhou. 2015. [http://arxiv.org/abs/1508.01585 Applying deep learning to answer selection: A study and an open task]. In ASRU 2015.
* Aliaksei Severyn and Alessandro Moschitti. 2015. [http://disi.unitn.it/~severyn/papers/sigir-2015-long.pdf Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks]. In SIGIR 2015.
* Zhiguo Wang and Abraham Ittycheriah. 2015. [http://arxiv.org/abs/1507.02628 FAQ-based Question Answering via Word Alignment]. In eprint arXiv:1507.02628.
* Ming Tan, Cicero dos Santos, Bing Xiang & Bowen Zhou. 2015. [http://arxiv.org/abs/1511.04108 LSTM-Based Deep Learning Models for Nonfactoid Answer Selection]. In eprint arXiv:1511.04108.
* Cicero dos Santos, Ming Tan, Bing Xiang & Bowen Zhou. 2016. [http://arxiv.org/abs/1602.03609 Attentive Pooling Networks]. In eprint arXiv:1602.03609.
* Zhiguo Wang, Haitao Mi and Abraham Ittycheriah. 2016. [http://arxiv.org/pdf/1602.07019v1.pdf Sentence Similarity Learning by Lexical Decomposition and Composition]. In Coling 2016.
* Hua He, Kevin Gimpel and Jimmy Lin. 2015. [http://aclweb.org/anthology/D/D15/D15-1181.pdf Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks]. In EMNLP 2015.
* Hua He and Jimmy Lin. 2016. [https://cs.uwaterloo.ca/~jimmylin/publications/He_etal_NAACL-HTL2016.pdf Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement]. In NAACL 2016.
* Jinfeng Rao, Hua He and Jimmy Lin. 2016. [http://www.cs.umd.edu/~jinfeng/publications/PairwiseNeuralNetwork_CIKM2016.pdf Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks]. In CIKM 2016
[[Category:State of the art]]

Question Answering (State of the art)

2016-02-24T15:36:34Z

Zhiguo Wang:

Question Answering (State of the art)

2016-01-21T16:01:44Z

Zhiguo Wang:

Question Answering (State of the art)

2016-01-21T16:01:05Z

Zhiguo Wang:

Question Answering (State of the art)

2015-07-10T13:24:43Z

Zhiguo Wang:

Question Answering (State of the art)

2015-07-10T12:52:08Z

Zhiguo Wang:

User:Zhiguo Wang

2015-06-04T14:42:11Z

Zhiguo Wang:

I'm a research staff member at IBM Watson research center. My research interests include statistical parsing, question answering and machine learning.