CoSaTa: A Constraint Satisfaction Solver and Interpreted Language for Semi-Structured Tables of Sentences

This work presents CoSaTa, an intuitive constraint satisfaction solver and interpreted language for knowledge bases of semi-structured tables expressed as text. The stand-alone CoSaTa solver allows easily expressing complex compositional “inference patterns” for how knowledge from different tables tends to connect to support inference and explanation construction in question answering and other downstream tasks, while including advanced declarative features and the ability to operate over multiple representations of text (words, lemmas, or part-of-speech tags). CoSaTa also includes a hybrid imperative/declarative interpreted language for expressing simple models through minimally-specified simulations grounded in constraint patterns, helping bridge the gap between question answering, question explanation, and model simulation. The solver and interpreter are released as open source. Screencast Demo: https://youtu.be/t93Acsz7LyE


Introduction
Performing inference for complex question answering typically requires combining multiple facts from a knowledge base to arrive at a correct answer, where this set of facts can then be used to generate detailed human-readable explanations for the reasoning behind those answers. Combining multiple facts to perform natural language inference is extremely challenging, with contemporary methods generally unable to reliably combine more than two facts together. This is a significant limitation, as even elementary science questions require an average of six (and, as many as 16) atomic facts to answer and explain Xie et al., 2020) -particularly when those explanations include detailed world knowledge. For example,  explaining the common process of an ice cube melting in sunlight (see Figure 1) can require a large number of facts, especially when those facts are expressed at a fine level of granularity. Compositional (or "multi-hop") inference solving methods tend to exist on a formality continuum. At one end of the continuum, logical or declarative methods (e.g. Lenat et al., 1990;Forbus, 2019) model a knowledge base as a set of assertions, and inference as sets of axioms and combinatorial rules acting on those assertions. While logical methods provide provably correct inference and detailed explanations, these methods tend to be brittle in practice (MacCartney and Manning, 2007). At the other end of the formality continuum are inference methods that use unstructured text as knowledge (modeled at the level of the word (Fried et al., 2015), sentence (Valentino et al., 2020), or paragraph (Yang et al., 2018)), which is typically combined using connectivity (e.g. Jansen et al., 2017), embedding (e.g. Tu et al., 2020), or other features. Due to the difficulty of combining free text (Jansen, 2018), these methods typically reach peak performance when combining only a small number of facts together -typically two or three.
A middle-ground exists between these two extremes, where semi-structured knowledge bases of text (such as tables) are used to support multi-hop inference (e.g. Sun et al., 2016). This approach offers many practical benefits, such as ease of knowledge base creation (over logical decomposition methods), and providing structure to help infer when combining facts is appropriate (versus free text methods). In spite of these benefits, it is often still challenging to implement inference models that compose (or "hop" between) facts expressed in tables of language data in practice, and practitioners tend to resort to using complex models (such as integer linear programming (e.g. TableILP; Khashabi et al., 2016)) that significantly increase development time and limit interpretability, maintainability, and reuse.
This work presents an easy-to-use scripting language paired with an open source solver and interpreter designed to make compositional inference over semi-structured knowledge bases of tables easy, particularly when those tables express knowledge as lightly structured sentences. The contributions of this work are: 1. The COSATA SOLVER, a stand-alone solver for Constraint Satisfaction over Tables of text.
The COSATA language allows easily expressing large multi-hop "inference patterns" that describe how facts typically connect across tables in a knowledge base to express a larger compositional solution. The optimized multithreaded solver supports advanced features for dealing with text, including enumerative variable span detection, and robustness to surface form variations with patterns that can match words, lemmas, or parts-of-speech.
2. The COSATA INTERPRETER, a hybrid imperative/declarative interpreted language for modeling simple inferences through minimallyspecified simulations grounded in constraint patterns.

Language Description
The COSATA language includes declarative features for performing constraint satisfaction over tables, and imperative features for expressing and executing models. The fundamental unit is the pattern, analagous to a class in object oriented programming, which (at a minimum) contains a declarative constraint pattern of table rows. The declarative features are sufficient for easily expressing variablized compositional patterns over collections of table rows, and the output of the stand-alone constraint satisfaction solver (e.g. solutions in JSON format) can serve as input to further processing. The language also supports a suite of imperative features for expressing and executing models. In this paradigm, each pattern is considered a process that, if executed, imparts some change upon a small model of the world (such as an object warming from heat transfer) by executing a pattern code block. Agents, physical objects, and environments in the model are represented as objects, here sets of property-value pairs. A control script imports a library of patterns, initializes objects, and executes a small subset of patterns in a particular order to create a simulation. A state space keeps a log of each object, it's properties, the patterns executed, and their resulting changes, to form a detailed and human-readable record of a simulation performed to arrive at a particular inference.   Row Name Table Row  som1 a <solid> is a kind of "state of matter" som2 a <liquid> is a kind of "state of matter" cos <melting> is a kind of "change of state" somprop "state of matter" is a property of a <substance> point a <melting point> is a kind of "phase transition point" change <melting> means the "state of matter" of <substance> changes from a <solid> into a <liquid> by <increasing> "heat energy" thresh <melting> occurs when the temperature of a <substance> is <increased> <above> the substance 's <melting point> heatcool <heating> means the "heat energy" of a substance is <increased> (b) An example enumerated solution of the above constraint pattern.

Declarative: Constraints over tables
The variable values from this solution. strings (words, lemmas, or part-of-speech tags) and variables. Elements can be combined with simple boolean operations, as well as advanced booleans (e.g. optional elements, enumerative ANDs that automatically determine variable spans). Example constraint expressions are shown in Table 1.
Inheritance and Composite Patterns: Similar to object oriented programming, patterns can contain their own  Figure 4 changes the state of matter of an object (for example from a solid to a liquid).
Model Control Script: Models take the form of small, easily-composed control scripts that import a library of patterns, initialize the objects required for a model, then sequentially execute a series of patterns that impart changes on those objects. For example, the control script in Figure 3 initializes three objects: an ice cube, freezer, and outside environment. The ice cube begins in the freezer, and is then moved to the outside environment. Heat transfer happens between the ice cube and outside environment until the ice cube meets the conditions for a Change of State. The Change of State then happens, in this case melting, changing the ice cube's state of matter property from solid to liquid.  Figure 3: An example micro-model control script (numbered states and instructions) with three objects (ice cube, outside environment, freezer), and the resulting state space changes (red highlights). This control script simulates the result of leaving an ice cube outside of a freezer. The full script and COSATA interpreter output is provided in the distribution. each pattern, then implements a backtrack search (e.g. Davis et al., 1962) to exhaustively find all combinations of table rows that meet the constraints for a given pattern (each unique collection of table rows that satisfies a given pattern is termed a solution of that pattern). 2 While some powerful declarative features (such as enumerative ANDs with automatic variable span detection) are expensive to evaluate, nearly all stages of evaluation are multi-threaded for speed, and rely heavily on precomputed look-up tables for evaluating constraint expressions. In practice, the patterns presented in Section 4 below are typically evaluated in between several seconds to a few minutes each.
// Code: Run the imperative code below if a given enumeration of this pattern is executed. // If the object (sub1) recently changed temperature above/below <pointTemp>, and is in state of matter <SOM1> if ((sub1."temperature" CHANGE [direction:COS.<tempDir> threshold:fromSOM.<pointTemp>]) && (sub1."state of matter" == COS. <SOM1>)) then // Set the object (sub1)'s new state of matter to be <SOM2> sub1."state of matter" = COS.<SOM2> // Add a human-readable explanation to the state space describing what this inference pattern did. addExplanationText("Substance (" + sub1."name" + ") made of (" + sub1."material" + ") is within the temperature range to change from a (" + COS.<SOM1> + ") to a (" + COS.<SOM2> + ").") endif endinferencepattern Inference Pattern: ChangeOfStateWithSubstanceFromTo Inherited Pattern 1: COS:changeStateOfMatter Row Name Table Row  som1 a <solid> is a kind of "state of matter" som2 a <liquid> is a kind of "state of matter" cos <melting> is a kind of "change of state" somprop "state of matter" is a property of a <substance> point a <melting point> is a kind of "phase transition point" change <melting> means the "state of matter" of <substance> changes from a <solid> into a <liquid> by <increasing> "heat energy" thresh <melting> occurs when the temperature of a <substance> is <increased> <above> the substance 's <melting point> heatcool <heating> means the "heat energy" of a substance is <increased> Inherited Pattern 2: fromSOM:substanceInSOMWithPhaseTransitionPoint Row Name Table Row  somHasPoint a <solid> has a <melting point> point1 the <melting point> of <water> is <0.0> <C> fromSOM.subSOM:substanceInSOM (nested) Row Name Table Row  som a <solid> is a kind of "state of matter" propSomTemp the <water> is in the <solid> state, called <ice>, for temperatures below 0.0 C Inherited Pattern 3: toSOM:substanceInSOM Row Name Table Row  som a <liquid> is a kind of "state of matter" propSomTemp the <water> is in the <liquid> state, called <water>, for temperatures between 0.0 C and 100.0 C Figure 4: (Top) An example composite pattern that (i) inherits row constraints from three other simpler patterns, and (ii) includes imperative code that effects the change described by the constraint satisfaction pattern. (Bottom) One example solution of this pattern, melting ice into liquid water. All other combinations of state changes (e.g. freezing, boiling) for all substances described in the knowledge base of semi-structured tables are also enumerated, but not shown here for space.

Example Solutions
Here the feasibility of generating constraint patterns (for downstream processing) or executable patterns (for modeling) is empirically demonstrated in the context of generating detailed multi-hop explanations to standardized elementary and middle school science exam questions drawn from the AI2 Aristo Reasoning Challenge (Clark et al., 2018).

Explanation Regeneration
The explanation regeneration task (Jansen and Ustalov, 2019) requires models to reconstruct large multi-fact gold explanations by selecting a set of interconnected facts from a knowledge base that match gold explanations provided in an explanation corpus. The task is very challenging, and current state-of-the-art models (e.g. Das et al., 2019) achieve nearly all of their performance by evaluat-  ing facts independently rather than jointly. The Worldtree V2 corpus (Xie et al., 2020) includes detailed multi-fact explanations for 4,400 standardized science exam questions grounded in a knowledge base of 63 tables and approximately 10k table rows, as well as a set of 353 semi-automatically authored collections of facts surrounding specific subtopics, such as changes of state, inherited characteristics, or seasonal changes in daylight. Here, those 353 inference patterns were converted to the COSATA constraint language using a prototype automatic converter, and all solutions to each pattern were enumerated with the COSATA solver. 42 of the automatically converted patterns were selected based on frequency of use for manual curation, where they were further abstracted, decomposed, and debugged. Ceiling performance on the explanation regeneration task was calculated for a shortlist of ranked solutions, in terms of both single solutions, and combinations of up to 3 solutions, with results shown in Table 2. Performance is evaluated in terms of accuracy (proportion of gold rows included in the explanation) and the average number of "extra" facts included in the solutions but not included in the gold explanation. The results show that the pattern solutions enumerated by COSATA have a ceiling performance of regenerating up to 58% of gold explanations when using a single solution, and up to 78% when combining up to three solutions. This empirically demonstrates the potential utility of using COSATA patterns as input to downstream inference models that are able to accurately select which patterns to combine to generate an explanation. The inference patterns in this experiment and their solutions are included as examples in the distribution.

Micro-models and Interpreter
Constructing micro-models from scratch requires (i) authoring a knowledge base of semi-structured tables, (ii) authoring patterns that reference those tables (such as those in Figures 2 and 4), and (iii) constructing micro-model control scripts (such as the example in Figure 3) that describe, in a short series of steps, how processes interact with objects and agents to reach a given outcome.
To demonstrate this workflow, a series of 23 patterns including imperative code were authored for topics in heat transfer and changes of state, as well as a supporting semi-structured knowledge base containing several hundred facts across 21 tables including taxonomic relations, locations of common objects, processes causing discrete changes, and physical properties of substances. To support this effort, a prototype IDE called Procession (shown in Figure 5) was implemented using ELECTRON that integrates a table-editor (including D3-based visual taxonomy editor), MONACO-based code editor, and side-to-side debugger/editor for micro-model control scripts that enables fast debug cycles. These example imperative patterns and the resulting interpreter output of the control scripts are included as examples in the distribution.

Conclusion
COSATA is an open-source constraint satisfaction solver for easily expressing and evaluating multifact compositional patterns in semi-structured tables of text, paired with an interpreted language that allows expressing micro-models. The tool, source, examples, and documentation are available at http://www.github.com/clulab/cosata/ .