Dependency Graph-to-String Translation

Compared to tree grammars, graph grammars have stronger generative capacity over structures. Based on an edge replacement grammar, in this paper we pro-pose to use a synchronous graph-to-string grammar for statistical machine translation. The graph we use is directly converted from a dependency tree by labelling edges. We build our translation model in the log-linear framework with standard features. Large-scale experiments on Chinese–English and German–English tasks show that our model is signiﬁcantly better than the state-of-the-art hierarchical phrase-based (HPB) model and a recently improved dependency tree-to-string model on BLEU, METEOR and TER scores. Experiments also suggest that our model has better capability to perform long-distance reordering and is more suitable for translating long sentences.


Introduction
Compared to trees, which have dominated the field of natural language processing (NLP) for decades, graphs are more general for modelling natural languages. The corresponding grammars for recognizing and producing graphs are more flexible and powerful than tree grammars. However, because of their high complexity, graph grammars have not been widely used in NLP.
Recently, along with progress on graph-based meaning representation, hyperedge replacement grammars (HRG) (Drewes et al., 1997) have been revisited, explored and used for semantic-based machine translation (Jones et al., 2012). However, the translation process is rather complex and the resources it relies on, namely abstract meaning corpora, are limited as well.
As most available syntactic resources and tools are tree-based, in this paper we propose to convert dependency trees, which are usually taken as a kind of shallow semantic representation, to dependency graphs by labelling edges. We then use a synchronous version of edge replacement grammar (ERG) (Section 2), a special case of HRG, to translate these graphs. The resulting translation model has the same order of magnitude in terms of time complexity with the hierarchical phrasebased model (HPB) (Chiang, 2005) under a certain restriction (Section 3).
Compared to dependency tree-to-string models, using ERG for graph-to-string translation brings some benefits (Section 3). Thanks to the stronger generative capacity of the grammar, our model can naturally translate siblings in a tree structure, which are usually treated as non-syntactic phrases and handled by other techniques (Huck et al., 2014;. Furthermore, compared to the known treelet approach  and Dep2Str (Xie et al., 2011), our method not only uses treelets but also has a full capacity of reordering.
We define our translation model (Section 4) in the log-linear framework (Och and Ney, 2002). Large-scale experiments (Section 5) on Chinese-English and German-English, two language pairs that have a high degree of syntactic reordering, show that our method significantly improves translation quality over both HPB and Dep2Str, as measured by BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and METEOR (Denkowski and Lavie, 2011). We also find that the rules in our model are more suitable for long-distance reordering and translating long sentences.

Edge Replacement Grammar
As a special case of HRG, ERG is also a contextfree rewriting grammar to recognize and produce graphs. Following HRG, the graph we use in this paper is connected, nodes ordered, acyclic and has edge labels but no node labels (Chiang et al., 2013). We provide some formal definitions on ERG.
• V is a finite set of nodes.
• E ⊆ V 2 is a finite set of edges.
• φ : E → C assigns a label (drawn from C) to each edge.
In ERG, the elementary unit is a graph fragment, which is also the right-hand side of a production in the grammar. Its definition is as follows.
Definition 2. A graph fragment is a tuple H = V, E, φ, X , where V, E, φ is a graph and X ∈ (V ∪ V 2 ) is a list of distinct nodes. Following Chiang et al. (2013), we call these external nodes.
The external nodes indicate how to integrate a graph into another one during a derivation. Different to HRG, ERG limits the number of external nodes to 2 at most to make sure hyperedges do not exist during a derivation. Now we define the ERG.
Definition 3. An edge replacement grammar is a tuple N, T, P, S , where • N and T are disjoint finite sets of nonterminal symbols and terminal symbols, respectively.
• P is a finite set of productions of the form A → R, where A ∈ N and R is a graph fragment, where edge-labels are from N T .
• S ∈ N is the start symbol. Figure 1 shows an example of a derivation in an ERG to produce a graph. Starting from the start symbol S, when a rule (A → R) is applied to an edge e, the edge is replaced by the graph fragment R. Just like in HRG, the ordering of nodes V e in e and external nodes X R in R implies the mapping from V e to X R (Chiang et al., 2013).

Graph-to-String Grammar
In SMT, we need a synchronous grammar to simultaneously parse an input graph and produce translations. The graph we use in this paper is from a dependency structure which is capable of modelling long-distance relations in a sentence.

The Grammar
Before defining the synchronous grammar, we firstly define a dependency graph which is a special case of a graph. Definition 4. A dependency graph is a tuple V, E, φ, ∆ , where V, E, φ is a graph and ∆ is a restriction: edges are ordered.
A dependency graph is directly derived from a dependency tree by labeling edges with words, as shown in Figure 2. Although in general graph edges are unordered, in Definition 4 we keep word order by ordering edges, because the word order is an important piece of information for translation.
Similar to the graph fragment, a dependencygraph fragment is defined as below. Definition 5. A dependency-graph fragment is a tuple V, E, φ, ∆, X , where V, E, φ, ∆ is a dependency graph, X ∈ (V ∪V 2 ) is a list of external nodes. In this paper, we define a synchronous ERG over dependency graphs as a dependency graphto-string grammar, which can be used for MT.
Definition 6. A dependency graph-to-string grammar (DGSG) is a tuple N, T, T , P, S , where • N is a finite set of non-terminal symbols.
• T and T are finite sets of terminal symbols.
• S ∈ N is the start symbol.
• P is a finite set of productions of the form R is a dependency-graph fragment over N T and R is a string over N T . ∼ is a one-toone mapping between non-terminal symbols in R and R . Figure 3 shows a derivation simultaneously producing a Chinese dependency graph and an English string using a DGSG. Each time a rule is applied, the dependency-graph fragment in the rule replaces an edge in the source graph, and the string in the rule replaces a non-terminal in the target string.
Proposition 1. DGSG has stronger generative capacity over graph-string pairs than both SCFG and synchronous tree substitution grammar (STSG).
Proof. STSG has stronger generative capacity over structures than SCFG (Chiang, 2012). 1 Any STSG can easily be converted into a DGSG by labelling edges in tree structures. 1 The following STSG generates a trivial example of a tree-string pair that no SCFG can generate, as SCFG must always have an equal number of non-terminal symbols.

X
| : The following DGSG generates a trivial example of a graph-string pair, which no STSG can generate, as the left-head side has no head nodes while STSG always requires one to form a tree.

c : a b
This proof is also verified in Figure 3 where the third rule is used to translate a non-syntactic phrase, which can be a problem for dependency tree-to-string methods. In addition, the second rule translates a treelet and the first rule encodes reordering information inside. All these three aspects are uniformly modeled in our grammar, which makes it more powerful than other methods, such as the treelet approach and the Dep2Str.

Time Complexity and a Restriction
Given a dependency graph, training and decoding time using DGSG depends on the number of dependency-graph fragments. For example, for a graph where the degree of a node is k, the number of all possible fragments starting from the node is O(2 k ). Therefore, the time complexity would be exponential if we consider them all.
It is easy to find that the high complexity of DGSG comes from the free combination of edges. That means that a dependency-graph fragment can cover discontinuous words of an input sentence. However, this is not the convention in the field of SMT.
For efficient training and decoding, we add a restriction to DGSG: each dependency-graph fragment covers a continuous span of the source sentence. This reduces the complexity from exponential time to cubic time.

Non-terminal Symbols
In this paper we build a dependency graph-tostring model, so we only use one non-terminal symbol X as in HPB on the target side. However, on the source side we define non-terminal symbols over Part-of-Speech (POS) tags, which can be easily obtained as a by-product of dependency parsing.
We define the head of a dependency-graph fragment H as a list of edges, the dependency head of each of which is not in this fragment. Then the   Figure 4: An example inducing a non-terminal symbol (left side) for a dependency-graph fragment (right side). Each edge is labeled by a word associated with its POS tag. The head of this fragment includes three edges which are in the rectangle.
non-terminal symbol for H is defined as the joining of POS tags of its head (Li et al., 2012). Figure  4 shows an example.

Rule Extraction
As well as the restriction defined in Section 3.2 making the grammar much smaller, it also results in a similar way of extracting rules as in HPB. Inspired by HPB, we define the rule set over initial pairs.
Given a word-aligned dependency graph-string pair P = G, e, ∼ , let G j i stand for the sub-graph (it may not be connected) covering words from position i to position j. Then a rule G j i , e j i is an initial pair of P , iff: 1. G j i is a dependency-graph fragment. That means it is a connected sub-graph and has at most two external nodes, nodes which connect with nodes outside or are the root.
2. It is consistent with the word alignment ∼ (Och and Ney, 2004).
The set of rules from P satisfies the following: 1. If G j i , e j i is an initial pair, then is a rule, where N (G) defines the nonterminal symbol for G.
2. If N (R) → R, X → R is a rule of P and G j i , e j i is an initial pair such that G j i is a sub-graph of R and R = r 1 e j i r 2 , then is a rule of P , where \ means replacing G j i in R with an edge labelled with N (G j i ) and k is a unique index for a pair of non-terminal symbols.
As in HPB, in addition to rules extracted from the parallel corpus, we also use glue rules to combine fragments and translations when no matched rule can be found.
Furthermore, we can use the same rule extraction algorithm as that in HPB, except that we need to check if a span of a source sentence indicates a dependency-graph fragment, in which case we keep the dependency structure and induce a nonterminal for the fragment.

Model and Decoding
We define our model in the log-linear framework over a derivation d, as in Equation (1): where φ i are features defined on derivations and λ i are feature weights. In our experiments, we use 9 features: • translation probabilities P (s|t) and P (t|s), where s is the source graph fragment and t is the target string.
• lexical translation probabilities P lex (s|t) and P lex (t|s).
• language model lm(e) over translation e.
• unknown words penalty exp(u(g)), where u(g) is the number of unknown words in a source graph g.
Our decoder is based on the conventional chart parsing CYK algorithm (Kasami, 1965;Younger, 1967;Cocke and Schwartz, 1970). It searches for the best derivation d * among all possible derivations D, as in Equation (2): For each span of an input graph, the decoder checks if it is a dependency-graph fragment. Then for each fragment, the decoder finds rules to translate it. The translation of a large span can be obtained by combining translations from its sub-span using rules which have non-terminals. Finally, glue rules are used to make sure that at least one translation is produced.

Experiment
We conduct experiments on Chinese-English and German-English translation tasks.

Datasets
The  Table 1 provides a summary of this corpus. We use mate-tools 2 to perform morphological analysis and parse German sentences (Bohnet, 2010). Then MaltParser 3 converts a parse result into a projective dependency tree (Nivre and Nilsson, 2005).

Settings
In this paper, we mainly compare our system (DGST) with HPB in Moses (Koehn et al., 2007). We implement our model in Moses and take the same settings as Moses HPB in all experiments. In addition, translation results from a recently open-source dependency tree-to-string system, Dep2Str 4 (Li et al., 2014), which is implemented in Moses and improves the dependencybased model in Xie et al. (2011), are also reported. All systems use the same sets of features defined in Section 4.
In all experiments, word alignment is performed by GIZA++ (Och and Ney, 2003) with the heuristic function grow-diag-final-and. We use SRILM (Stolcke, 2002) to train a 5-gram language model on the Xinhua portion of the English Gigaword corpus 5th edition with modified Kneser-Ney discounting (Chen and Goodman, 1996). Minimum Error Rate Training (MERT) (Och, 2003) is used to tune weights.
To obtain more reliable results, in each experiment, we run MERT three times and report average scores. These scores are calculated by three widely used automatic metrics in case-insensitive mode: BLEU, METEOR and TER. Table 2 shows the scores of all three metrics on all systems. Similar to Li et al. (2014), in our experiments Dep2Str has on average a comparable result with Moses HPB in terms of BLEU and METEOR scores. However, it obtains a significantly higher (i.e. worse) TER score on the Chinese-English task. This may suggest that translations produced by Dep2Str need more post-editing effort (He et al., 2010).

Results
By contrast, on all test sets, measured by all metrics, our system is significantly better than Moses HPB. On the Chinese-English task, our system achieves an average gain of 1.25 (absolute, 3.6% relative) BLEU score and 0.55 (absolute, 1.7% relative) METEOR score while also ob-   taining a reduction of 1.1 (absolute, 1.91% relative) TER score on average. On the German-English task, our system achieves an average gain of 0.55 (absolute, 2.56% relative) BLEU score and 0.1 (absolute, 0.35% relative) METEOR score and also obtains a reduction of 0.55 (absolute, 0.89% relative) TER score on average.

Analysis
As shown in Table 2, compared to Moses HPB and Dep2Str, our system achieves higher translation quality as measured by three automatic metrics. In this section, we investigate whether dependency structures bring benefits as expected on long-distance reordering. Table 3 provides the statistics on sentence length of our four test sets.
In both HPB and our model, the length range of a reordering performed on an input sentence is related to the use of glue grammars which bring two benefits during decoding. When no matched rule is found in the models, glue grammars are applied to make sure a translation is produced. In addition, because of the generalization capability of rules, which typically are learned under a length limitation, using them on long sentences could cause translation quality to deteriorate. Therefore, when the length of a phrase is greater than a certain value, glue grammars are also applied. Therefore, our experiment of analysis is based on the length limitation that a rule can cover (max. phrase length) during decoding.
We set this max. phrase length to different values, including 10, 20 (default), 30, 40 and 50. Figure 5 gives the BLEU scores on all test sets. We find that on all different values, our system achieves higher BLEU scores than Moses HPB. In addition, when the max. phrase length becomes larger, Moses HPB shows a declining trend in most cases, especially on the German-English task (WMT12 and WMT13). However, our system is less sensitive to this value. We hypothesize that this is because rules from dependency graphs have better generalization for translating longer phrases and are more suitable for translating long sentences.

Case Study
On a manual check, we find that translations produced by our system are more fluent than those of both Moses HPB and Dep2Str. Figure 6 gives an example comparing translations produced by three systems on the Chinese-English task.
We first find a case of long-distance relation, i.e. the subject-verb-object (SVO) structure in the source sentence. In this example, this relation implies a long-distance reordering, which moves the translation of the object to the front of its modifiers, as shown in the given reference.

Ref:
The two sides welcomed the decision by the Iraqi Interim Governing Council to establish a special court to try the murderers.
HPB: the two sides welcomed the interim iraqi authority on establishing a special court, trial of the murderer.
Dep2Str: the two sides welcomed the decision on the Establishment of a special court, justice murderers of the provisional governing council of iraq.
DGST: the two sides welcomed the decision of the iraqi interim governing council on the establishment of a special court, justice murderers. Figure 6: An example of comparing translations produced by three systems on the Chinese-English task. The source sentence is parsed into a dependency structure. Each source word is annotated by a corresponding English word (or phrase). Figure 7: An example of inducing a dependency structure in Figure 6 to "X 的(of) X" structure in our system by using treelets and non-syntactic phrases. µ denotes one or more steps. All non-terminals are simply represented by X. pared to Moses HPB, both Dep2Str and our system, which rely on dependency structures, are capable of dealing with this. This also suggests that dependency structures are useful for long-distance reordering.
Furthermore, compared to Dep2Str, our system produces a better translation for the "X 的(of) X" expression, which is not explicitly represented in the dependency structure and thus results in a wrong translation in Dep2Str. After looking into the details of the translation process, we find that our system induces the dependency structure to the "X 的(of) X" structure by handling both treelets and non-syntactic phrases. Figure 7 shows the process of this induction.

Related Work
Dependency structures have been used in SMT for a few years. Because of its better inter-lingual phrasal cohesion properties (Fox, 2002), it is believed to be beneficial to translation.
Researchers have tried to use dependency structures on both target and source sides. Shen et al. (2010) propose a string-to-dependency model by using dependency fragments of neighbouring words on the target side, which makes the model easier to include a dependency-based language model.  and  propose the treelet approach which uses dependency structures on the source side. Xiong et al. (2007) extend this approach by allowing gaps in rules. However, their methods need a separate reordering model to decide the position of translated words (insertion problem). To avoid this problem, Xie et al. (2011) propose to use full head-dependent structures of a dependency tree and build a new dependency-to-string model. However, this model has difficulties in handling non-syntactic phrasal rules and ignores treelets. Meng et al. (2013) and  further augment this model by incorporating constituent phrases and integrating fix/float structures (Shen et al., 2010), respectively, to allow phrasal rules. Li et al. (2014) extend this model by decomposing head-dependent structures into treelets.
Different from these methods, by labelling edges and using the ERG, our model considers the three aspects in a unified way: treelet, reordering and non-syntactic phrase. In addition, the ERG also naturally provides a decision on what kind of treelets and phrases should be used.

Conclusion
In this paper, we present a dependency graph-tostring grammar based on a graph grammar, which we call edge replacement grammar. This grammar can simultaneously produce a pair of dependency graph and string. With a restriction of using contiguous edges, our translation model built using this grammar can decode an input dependency graph, which is directly converted from a dependency tree, in cubic time using the CYK algorithm.
Experiments on Chinese-English and German-English tasks show that our model is significantly better than the hierarchical phrase-based model and a recent dependency tree-to-string model (Dep2Str) in Moses. We also find that the rules used in our model are more suitable for longdistance reordering and translating long sentences.
Although experiments show significant improvements over baselines, our model has limitations that can be avenues for future work. The restriction used in this paper reduces the time complexity but at the same time reduces the generative capacity of graph grammars. Without allowing hyperedges or only using at most two external nodes reduces the phrase coverage in our model as well.