Know What You Don’t Know: Unanswerable Questions for SQuAD

Pranav Rajpurkar, Robin Jia, Percy Liang


Abstract
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.
Anthology ID:
P18-2124
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
784–789
Language:
URL:
https://aclanthology.org/P18-2124
DOI:
10.18653/v1/P18-2124
Bibkey:
Cite (ACL):
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Know What You Don’t Know: Unanswerable Questions for SQuAD (Rajpurkar et al., ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-2124.pdf
Note:
 P18-2124.Notes.pdf
Video:
 https://aclanthology.org/P18-2124.mp4
Code
 worksheets/0x9a15a170 +  additional community code
Data
SQuAD