A Sentence Cloze Dataset for Chinese Machine Reading Comprehension

Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, Guoping Hu


Abstract
Owing to the continuous efforts by the Chinese NLP community, more and more Chinese machine reading comprehension datasets become available. To add diversity in this area, in this paper, we propose a new task called Sentence Cloze-style Machine Reading Comprehension (SC-MRC). The proposed task aims to fill the right candidate sentence into the passage that has several blanks. We built a Chinese dataset called CMRC 2019 to evaluate the difficulty of the SC-MRC task. Moreover, to add more difficulties, we also made fake candidates that are similar to the correct ones, which requires the machine to judge their correctness in the context. The proposed dataset contains over 100K blanks (questions) within over 10K passages, which was originated from Chinese narrative stories. To evaluate the dataset, we implement several baseline systems based on the pre-trained models, and the results show that the state-of-the-art model still underperforms human performance by a large margin. We release the dataset and baseline system to further facilitate our community. Resources available through https://github.com/ymcui/cmrc2019
Anthology ID:
2020.coling-main.589
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6717–6723
Language:
URL:
https://aclanthology.org/2020.coling-main.589
DOI:
10.18653/v1/2020.coling-main.589
Bibkey:
Cite (ACL):
Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2020. A Sentence Cloze Dataset for Chinese Machine Reading Comprehension. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6717–6723, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
A Sentence Cloze Dataset for Chinese Machine Reading Comprehension (Cui et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.589.pdf
Code
 ymcui/cmrc2019
Data
CMRC 2019CMRCCMRC 2017CMRC 2018DRCDDuReaderSQuAD