Benchmarking Multimodal Regex Synthesis with Complex Structures

Xi Ye, Qiaochu Chen, Isil Dillig, Greg Durrett


Abstract
Existing datasets for regular expression (regex) generation from natural language are limited in complexity; compared to regex tasks that users post on StackOverflow, the regexes in these datasets are simple, and the language used to describe them is not diverse. We introduce StructuredRegex, a new regex synthesis dataset differing from prior ones in three aspects. First, to obtain structurally complex and realistic regexes, we generate the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts. Second, to obtain linguistically diverse natural language descriptions, we show crowdworkers abstract depictions of the underlying regex and ask them to describe the pattern they see, rather than having them paraphrase synthetic language. Third, we augment each regex example with a collection of strings that are and are not matched by the ground truth regex, similar to how real users give examples. Our quantitative and qualitative analysis demonstrates the advantages of StructuredRegex over prior datasets. Further experimental results using various multimodal synthesis techniques highlight the challenge presented by our dataset, including non-local constraints and multi-modal inputs.
Anthology ID:
2020.acl-main.541
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6081–6094
Language:
URL:
https://aclanthology.org/2020.acl-main.541
DOI:
10.18653/v1/2020.acl-main.541
Bibkey:
Cite (ACL):
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2020. Benchmarking Multimodal Regex Synthesis with Complex Structures. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6081–6094, Online. Association for Computational Linguistics.
Cite (Informal):
Benchmarking Multimodal Regex Synthesis with Complex Structures (Ye et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.541.pdf
Video:
 http://slideslive.com/38929291