Neural Semantic Parsing

Semantic parsing, the study of translating natural language utterances into machine-executable programs, is a well-established research area and has applications in question answering, instruction following, voice assistants, and code generation. In the last two years, the models used for semantic parsing have changed dramatically with the introduction of neural encoder-decoder methods that allow us to rethink many of the previous assumptions underlying semantic parsing. We aim to inform those already interested in semantic parsing research of these new developments in the field, as well as introduce the topic as an exciting research area to those who are unfamiliar with it. Current approaches for neural semantic parsing share several similarities with neural machine translation, but the key difference between the two fields is that semantic parsing translates natural language into a formal language, while machine translation translates it into a different natural language. The formal language used in semantic parsing allows for constrained decoding, where the model is constrained to only produce outputs that are valid formal statements. We will describe the various approaches researchers have taken to do this. We will also discuss the choice of formal languages used by semantic parsers, and describe why much recent work has chosen to use standard programming languages instead of more linguistically-motivated representations. We will then describe a particularly challenging setting for semantic parsing, where there is additional context or interaction that the parser must take into account when translating natural language to formal language, and give an overview of recent work in this direction. Finally, we will introduce some tools available in AllenNLP for doing semantic parsing research.


Introduction
Semantic parsers translate natural language utterances into machine-executable logical forms or programs, and are thus key components in natural language understanding systems. Semantic parsing is a well-established research area, with application in tasks such as question answering, instruction following, voice assistants, and code generation. In the last two years, the models used for semantic parsing have changed dramatically, with the introduction of neural methods that allow us to rethink many of the previous assumptions underlying semantic parsing.
Traditionally, the executable formalisms and models used in semantic parsing research have been heavily reliant on notions of formal semantics in linguistics, such as λ-calculus generated by a CCG parser. However, recent work with neural encoder-decoder semantic parsers allow for more accessible formalisms, such as standard programming languages, and NMT-style models that are much more approachable to a broader NLP audience. We will present an overview of modern neural methods for semantic parsing and how they have changed semantic parsing research.

Description
This tutorial will cover how the transition to neural encoder-decoder models has changed semantic parsing research. We aim to both inform those already interested in semantic parsing research of new developments in the field, as well as introduce the topic as an exciting research area to those who are unfamiliar with it.
Current semantic parsing research uses encoder-decoder models that are very similar to machine translation systems. The key difference between these two fields is that semantic parsing translates natural language into a formal language, while machine translation translates natural language into a different natural language. The formal language used in semantic parsing research allows for constrained decoding, where the model is constrained to only produce outputs that are valid formal statements. We will describe how this is done, and the various approaches researchers have taken to model this constrained decoding.
Encoder-decoder semantic parsing models also allow us to drop our reliance on linguistic formalisms, and much recent work has explored replacing λ-calculus and λ-DCS with standard programming languages like SQL, python, or java. This has the promise of dramatically decreasing annotation costs, allowing researchers to collect much larger and more varied semantic parsing datasets than have previously been available. In our tutorial, we will describe recent efforts in this direction and why programming languages are a natural target for future semantic parsing research.
Neural models also allow representation of continuous, diverse, and less well-defined contexts (e.g., photographs), with methods for representing these contexts that generalize better to new environments (e.g., they don't necessarily require symbolic representations of the environments). The last section of our tutorial will cover recent work on these more complex semantic parsing tasks.
Much of the content covered in this tutorial will have corresponding implementations in the Al-lenNLP toolkit for NLP research. We will provide a brief overview at the end of the tutorial outlining how to use this toolkit to get started with semantic parsing research. decoder models have changed semantic parsing research. We will briefly discuss the complexity of prior systems, and how new models can be seen as very similar to neural machine translation models, with the addition of constrained decoding.

2.
Datasets: Before talking about modern methods, we will spend some time discussing what you can do with semantic parsing, and which datasets and tasks are most exciting for current research.
3. Constrained Decoding: Current semantic parsing models use an encoder-decoder architecture with constrained decoding. This section will first describe the basic encoderdecoder architecture, then describe how constrained decoding works. There are many ways to parameterize the decoder; we will discuss a simple method in-depth, to give the audience a detailed understanding of the basic model architecture, then describe several other model structures and how they relate to the simple architecture.

Semantic Parsing as Code Generation:
This section will discuss the choice of formal languages used by semantic parsers, and describe why much recent work has chosen to use standard programming languages instead of more linguistically-motivated representations.
5. Grounded and Context-Dependent Semantic Parsing: This section will describe a particularly challenging setting for semantic parsing: where there is additional context or interaction that the parser must take into account when translating natural language to formal language. Neural models provide a natural way to include this context, and we will give an overview of recent work in this direction.
6. Building Semantic Parsers with Al-lenNLP: A brief demonstration of the tools available in the AllenNLP toolkit for doing semantic parsing research.

Instructors
Matt Gardner is a research scientist at the Allen Institute for Artificial Intelligence. His research focuses on question answering and semantic parsing. He is the lead maintainer of the AllenNLP toolkit and a host of the NLP Highlights podcast. Pradeep Dasigi is a PhD student at the Language Technologies Institute in Carnegie Mellon University. His research interest lies in building knowledge-aware language understanding systems, with a recent focus on neural semantic parsing. Srinivasan Iyer is a graduate student in the Natural Language Processing group at the University of Washington, Seattle. His main research area is context dependent semantic parsing directly from natural language to general purpose programming source code. Other aspects of his research are learning semantic parsers from massive online resources and incorporating user feedback for model improvement.
Alane Suhr is a PhD student in Computer Science at Cornell University. Alane's research interests include developing machine learning methods for understanding natural language grounded in complex environments and interactions. She is a recipient of an NSF Graduate Research Fellowship, the Best Resource Paper award at ACL 2017, and an Outstanding Paper Award at NAACL 2018. Luke Zettlemoyer Luke Zettlemoyer is an Associate Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. He has a been doing research in semantic parsing for many years, and recently shifted to studying neural models for this problem. Luke's honors include multiple best paper awards, a PECASE award, and an Allen Distinguished Investigator award.