Enhancing PTB Universal Dependencies for Grammar-Based Surface Realization

Grammar-based surface realizers require inputs compatible with their reversible, constraint-based grammars, including a proper representation of unbounded dependencies and coordination. In this paper, we report on progress towards creating realizer inputs along the lines of those used in the ﬁrst surface realization shared task that satisfy this requirement. To do so, we augment the Universal Dependencies that result from running the Stanford Dependency Converter on the Penn Tree-bank with the unbounded and coordination dependencies in the CCGbank, since only the latter takes the Penn Treebank’s trace information into account. An evaluation against gold standard dependencies shows that the enhanced dependencies have greatly enhanced recall with moderate precision. We conclude with a discussion of the implications of the work for a second realization shared task.


Introduction
Surface realization systems employing reversible, broad coverage constraint-based grammars together with statistical ranking mod-els have achieved impressive results in multiple languages, using a variety of formalisms (HPSG, TAG, LFG, CCG). However, these systems all require somewhat different inputs, making comparative evaluation difficult. In the first surface realization shared task (Belz et al., 2011, henceforth SR-11), which aimed to ameliorate these difficulties, attempts to use grammar-based realizers were unsuccessful, as converting shared task inputs to systemnative inputs turned out to be more difficult than anticipated. Subsequently, Narayan & Gardent (2012) demonstrated that grammarbased systems can be substantially improved with error mining techniques, and Gardent and Narayan (2013) showed that augmenting the (shallow) SR-11 representation of coordination to include shared dependencies can benefit grammar-based realizers. White (2014) then showed that even better results can be achieved by inducing a grammar (Kwiatkowski et al., 2011;Artzi and Zettlemoyer, 2013) that is directly compatible with (an enhanced version of) the SR-11 inputs. However, as explained below, subsequent analysis revealed substantial remaining issues with the data, which this paper takes a step towards addressing.
A common thread in work on reversible, constraint-based grammars is emphasis on properly representing unbounded dependencies and coordination. For parsing, this emphasis has been shown to pay off in improved recall of unbounded dependencies (Rimell et al., 2009;Nguyen et al., 2012;Oepen et al., 2014). For realization, however, it remains an open question as to whether approaches based on constraintbased grammars can likewise yield an empirical payoff, given the continuing lack of a common input representation that adequately treats unbounded dependencies and coordination, as these grammars require.
With this issue in mind, White (2014) experimented with a version of the shallow SR-11 inputs (created by Richard Johansson) which included extra dependencies for unbounded dependencies and coordination, yielding dependency graphs extending core dependency trees. Unlike the rewrite rules employed by Gardent and Narayan (2013), the extra dependencies were derived from the gold traces in the Penn Treebank (Marcus et al., 1993, PTB), which is necessary to adequately handle right node raising and relativization. However, this version was still found to be incomplete, in particular because it was missing cases where the extra dependencies are encoded structurally in the PTB.
Since then, Universal Dependencies (Nivre et al., 2016, UDs), which aim to represent syntactic dependencies similarly across languages, have become increasingly prominent. Building on the enhanced Stanford dependencies for English (de Marneffe et al., 2013)which were designed to properly represent unbounded dependencies in dependency graphsenhanced UDs for English have been partially implemented in the Stanford Dependency Converter (Schuster and Manning, 2016, SDC). The SDC transforms automatic or gold PTB-style trees into UDs; unfortunately, however, it was not designed to take traces into account, and thus the treatment of unbounded dependencies and coordination is only heuristic. To address this impasse, in this paper we report on progress towards creating SR-11-style realizer inputs that are both based on enhanced UDs and which accurately represent unbounded dependencies and coordination. To do so, we augment the UDs that result from running the SDC on the PTB with the dependencies in the CCGbank (Hockenmaier and Steedman, 2007), since the latter includes lexicalized dependencies derived from gold PTB traces.

Background
Figures 1-2 show an example where the CCGbank preserves the information provided by the trace in a free relative clause along with a crucial structurally encoded dependency. In Figure 1 (left), the unbounded dependency between what and achieve is annotated via a trace in the PTB. Figure 1 (right) shows the SDC output for the sentence. While the SDC manages to capture the unbounded dependency in this case, what is not recognized as the head of the free relative clause and there is no direct dependency from the copula to what, contrary to de Marneffe et al.'s (2013) specifications. The inadequacy of the representation here-which is essentially the same as the SR-11 representation for the sentence-has serious implications for realization, as it will be difficult for any realizer to determine that what should appear at the start of the free relative clause rather than following achieve, where direct objects would normally appear (or perhaps sentence initially). By contrast, Figure 2 shows how the Combinatory Categorial Grammar (Steedman, 2000;Steedman and Baldridge, 2011, CCG) derivation yields ( (S (NP-SBJ (PRP It) ) (VP (VBZ is) (SBAR-PRD (WHNP-1 (WP what) ) (S (NP-SBJ-2 (JJ federal) (NN support) ) (VP (MD should) (VP (VB try) (ADVP-MNR (RBS hardest) ) (S (NP-SBJ (-NONE-* -2) ) (VP (TO to) (VP (VB achieve) (NP (-NONE-* T * -1) ))))))))) (.

.) ))
It is what federal support should try hardest to achieve .   a structural dependency between what and is along with the unbounded dependency between what and achieve. (More commonly, the SDC captures the structural dependency but not the unbounded one, especially in right node raising and object relatives.)

Using the CCGbank to Augment PTB Universal Dependencies
Unlike UDs, CCGbank dependencies are numeric and depend on the lexical category of the functor (e.g. what fills the second argument of the category for achieve in Figure 2). To determine UD labels, we employ a maxent classifier taking information from CCGbank as in-put. Comparing the CCGbank and SDC output, the classifier is trained where their dependencies overlap and predicts both a label and headdependent direction. Features used are functor and argument categories; functor and argument tokens; functor and argument POS tags; and functor and argument relative directionality.
Our system for augmenting the SDC's PTB output begins by combining the SDC basic and enhanced output, since the basic representation does not skip words while the enhanced representation already includes many correct extra dependencies. The system then scans the SDC output and CCGbank for 3 triggers: (i) shared arguments in coordination (e.g. shared objects in right node raising), (ii) CCGbank unbounded dependency annotations, and (iii) underspecified SDC dep relations (i.e. instances where the SDC cannot determine the appropriate dependency relation). In each case, the maxent classifier is used to predict UD labels for the CCGbank dependencies in question. Predictions are only added to the corpus if there is no (nondep) SDC dependency already present. In addition, ccomp and csubj relations that co-occur with free relatives are remapped to make the relative the head of the clause. Finally, structural changes for coordination and compounding along SR-11 lines are carried out.

Evaluation
The system's recall was evaluated on Rimell et al.'s (2009) unbounded dependency corpus, a hand-curated corpus with gold annotations for constructions including object free relatives, right node raising, subject extraction, and object extraction. During the creation of CCGbank, some problematic sentences involving gapping were left out of the CCGbank. As a result, we evaluate the system using four different criteria: with and without the skipped CCG sentences, and with both exact and unlabeled matches. Tables 1 and 2 show significant improvements across the board over the SDC.
Precision was evaluated by manually examining 401 predictions from the system's output to see whether the proposed edits adhered to UD specifications. Precision from the converter is 70% for exact label matches and 91% for unlabeled matches.

Discussion and Future Work
We have adapted and extended White's (2014) CCG induction algorithm to work with the augmented UDs that our system produces. White's algorithm assumed CCG phrases are only rarely projected from a dependent rather than a heade.g., where an NP is projected from a determiner, which is a dependent of the head nounand thus could be easily handled by handcrafted lexical entries. Since such cases are very common in UDs, the algorithm needed to be extended to induce such categories automatically. Once this was done, the algorithm yielded complete derivations in most cases (approx. 94%). In particular, derivations were induced that captured all but one of the extra dependencies in Table 1 that appear in the CCGbank dev section, and realization experiments with the UD-based representations are underway.
With the augmented UD reported in this paper, we expect the resulting dependency graphs to serve as a promising basis for a second surface realization challenge (with using just the basic dependency trees as an option). A remaining obstacle, however, are the dependent cluster and gapping cases in the PTB, for which the SDC produces rather degenerate output. A promising avenue here would be to adapt Gardent and Narayan's (2013) method of enhancing the SR-11 representations for these cases.