Encoding Source Language with Convolutional Neural Network for Machine Translation

The recently proposed neural network joint model (NNJM) (Devlin et al., 2014) augments the n-gram target language model with a heuristically chosen source context window, achieving state-of-the-art performance in SMT. In this paper, we give a more systematic treatment by summarizing the relevant source information through a convolutional architecture guided by the target information. With different guiding signals during decoding, our specifically designed convolution+gating architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of entire source sentence to form a unified representation. This representation, together with target language words, are fed to a deep neural network (DNN) to form a stronger NNJM. Experiments on two NIST Chinese-English translation tasks show that the proposed model can achieve significant improvements over the previous NNJM by up to +1.01 BLEU points on average


Introduction
Learning of continuous space representation for source language has attracted much attention in both traditional statistical machine translation (SMT) and neural machine translation (NMT).Various models, mostly neural network-based, have been proposed for representing the source sentence, mainly as the encoder part in an encoder-decoder framework (Bengio et al., 2003;Auli et al., 2013;Kalchbrenner and Blunsom, 2013;Cho et al., 2014;Sutskever et al., 2014).There has been some quite recent work on encoding only "relevant" part of source sentence during the decoding process, most notably neural network joint model (NNJM) in (Devlin et al., 2014), which extends the n-grams target language model by additionally taking a fixed-length window of source sentence, achieving state-ofthe-art performance in statistical machine translation.
In this paper, we propose novel convolutional architectures to dynamically encode the relevant information in the source language.Our model covers the entire source sentence, but can effectively find and properly summarize the relevant parts, guided by the information from the target language.With the guiding signals during decoding, our specifically designed convolution architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of entire source sentence to form a unified representation.This representation, together with target words, are fed to a deep neural network (DNN) to form a stronger NNJM.Since our proposed joint model is purely lexicalized, it can be integrated into any SMT decoder as a feature.
Two variants of the joint model are also proposed, with coined name tagCNN and inCNN, with different guiding signals used from the decoding process.We integrate the proposed joint models into a state-of-the-art dependency-to-string translation system (Xie et al., 2011) to evaluate their effectiveness.Experiments on NIST Chinese-English translation tasks show that our model is able to  RoadMap: In the remainder of this paper, we start with a brief overview of joint language model in Section 2, while the convolutional encoders, as the key component of which, will be described in detail in Section 3. Then in Section 4 we discuss the decoding algorithm with the proposed models.The experiment results are reported in Section 5, followed by Section 6 and 7 for related work and conclusion.

Joint Language Model
Our joint model with CNN encoders can be illustrated in Figure 1 (a) & (b), which consists 1) a CNN encoder, namely tagCNN or inCNN, to represent the information in the source sentences, and 2) an NN-based model for predicting the next words, with representations from CNN encoders and the history words in target sentence as inputs.
In the joint language model, the probability of the target word e n , given previous k target words {e n−k , • • •, e n−1 } and the representations from CNN-encoders for source sentence S are tagCNN: p(e n |φ 1 (S, index of tagged word)) where φ 1 (S, index of tagged word) stands for the representation given by tagCNN with the set of indexes of source words aligned to the target word e n , and φ 2 (S, h({e} n−1 n−k )) stands for the representation from inCNN with the attention signal h({e} n−1 n−k ).
Let us use the example in Figure 1, where the task is to translate the Chinese sentence into English.

Convolutional Models
We start with the generic architecture for convolutional encoder, and then proceed to tagCNN and inCNN as two extensions.

Generic CNN Encoder
The basic architecture is of a generic CNN encoder is illustrated in Figure 2 (a), which has a fixed architecture consisting of six layers: Layer-0: the input layer, which takes words in the form of embedding vectors.In our work, we set the maximum length of sentences to 40 words.For sentences shorter than that, we put zero padding at the beginning of sentences.
Layer-1: a convolution layer after Layer-0, with window size = 3.As will be discussed in Section 3.2 and 3.3, the guiding signal are injected into this layer for "guided version".
Layer-2: a local gating layer after Layer-1, which simply takes a weighted sum over feature-maps in non-adjacent window with size = 2.
Layer-3: a convolution layer after Layer-2, we perform another convolution with window size = 3.
Layer-4: we perform a global gating over feature-maps on Layer-3.
Layer-5: fully connected weights that maps the output of Layer-4 to the this layer as the final representation.

Convolution
As shown in Figure 2 (a), the convolution in Layer-1 operates on sliding windows of words (width k 1 ), and the similar definition of windows carries over to higher layers.Formally, for source sentence input where gives the output of feature map of type-f for location i in Layer-; • w ( ,f ) is the parameters for f on Layer-; • σ(•) is the Sigmoid activation function; Figure 2: Illustration for the CNN encoders.
• ẑ( −1) i denotes the segment of Layer-−1 for the convolution at location i , while concatenates the vectors for 3 words from sentence input x.

Gating
Previous CNNs, including those for NLP tasks (Hu et al., 2014;Kalchbrenner et al., 2014), take a straightforward convolution-pooling strategy, in which the "fusion" decisions (e.g., selecting the largest one in max-pooling) are based on the values of feature-maps.This is essentially a soft template matching, which works for tasks like classification, but harmful for keeping the composition functionality of convolution, which is critical for modeling sentences.In this paper, we propose to use separate gating unit to release the score function duty from the convolution, and let it focus on composition.
We take two types of gating: 1) for Layer-2, we take a local gating with non-overlapping windows (size = 2) on the feature-maps of convolutional Layer-1 for representation of segments, and 2) for Layer-4, we take a global gating to fuse all the segments for a global representation.We found that this gating strategy can considerably improve the performance of both tagCNN and inCNN over pooling.
• Local Gating: On Layer-1, for every gating window, we first find its original input (before convolution) on Layer-0, and merge them for the input of the gating network.For example, for the two windows: word (3,4,5) and word (4,5,6) on Layer-0, we use concatenated vector consisting of embedding for word (3,4,5,6) as the input of the local gating network (a logistic regression model) to determine the weight for the convolution result of the two windows (on Layer-1), and the weighted sum are the output of Layer-2.
• Global Gating: On Layer-3, for feature-maps at each location i, denoted z (3) i , the global gating network (essentially soft-max,parameterized w g ), assigns a normalized weight ω(z and the gated representation on Layer-4 is given by the weighted sum i ω(z (3) i .

Training of CNN encoders
The CNN encoders, including tagCNN and inCNN that will be discussed right below, are trained in a joint language model described in Section 2, along with the following parameters • the embedding of the words on source and the proceeding words on target; • the parameters for the DNN of joint language model, include the parameters of soft-max for word probability.
The training procedure is identical to that of neural network language model, except that the parallel corpus is used instead of a monolingual corpus.We seek to maximize the log-likelihood of training samples, with one sample for every target word in the parallel corpus.Optimization is performed with the conventional back-propagation, implemented as stochastic gradient descent (LeCun et al., 1998) with mini-batches.

tagCNN
tagCNN inherits the convolution and gating from generic CNN (as described in Section 3.1), with the only modification in the input layer.As shown in Figure 2 (b), in tagCNN, we append an extra tagging bit (0 or 1) to the embedding of words in the input layer to indicate whether it is one of affiliated words Those extended word embedding will then be treated as regular word-embedding in the convolutional neural network.This particular encoding strategy can be extended to embed more complicated dependency relation in source language, as will be described in Section 5.4.
This particular "tag" will be activated in a parameterized way during the training for predicting the target words.In other words, the supervised signal from the words to predict will find, through layers of back-propagation, the importance of the tag bit in the "affiliated words" in the source language, and learn to put proper weight on it to make tagged words stand out and adjust other parameters in tagCNN accordingly for the optimal predictive performance.In doing so, the joint model can pinpoint the parts of a source sentence that are relevant to predicting a target word through the already learned word alignment.

inCNN
Unlike tagCNN, which directly tells the location of affiliated words to the CNN encoder, inCNN sends the information about the proceeding words in target side to the convolutional encoder to help retrieve the information relevant for predicting the next word.This is essentially a particular case of attention model, analogous to the automatic alignment mechanism in (Bahdanau et al., 2014), where the attention signal is from the state of a generative recurrent neural network (RNN) as decoder.
Basically, the information from proceeding words, denoted as h({e} n−1 n−k ), is injected into every convolution window in the source language sentence, as illustrated in Figure 2 (c).More specifically, for the window indexed by t, the input to convolution is given by the concatenated vector In this work, we simply use the vectors concatenated from word-embedding for words , although h(•) can be non-trivially realized with another neural network.Through layers of convolution and gating, inCNN can 1) retrieve the relevant segments of Figure 3: Illustration for a dependency tree (a) with three head-dependents relations in shadow, an example of head-dependents relation rule (b) for the top level of (a), and an example of head rule (c)."X 1 :NN" indicates a substitution site that can be replaced by a subtree whose root has part-of-speech "NN".The underline denotes a leaf node.source sentences, and 2) compose and transform the retrieved segments into representation recognizable by the DNN in predicting the words in target language.Different from that of tagCNN, inCNN uses information from proceeding words, hence provides complementary information in the augmented joint language model of tagCNN.This has been empirically verified when using feature based on tagCNN and that based on inCNN in decoding with greater improvement.

Decoding with the Joint Model
Our joint model is purely lexicalized, and therefore can be integrated into any SMT decoders as a feature.For a hierarchical SMT decoder, we adopt the integrating method proposed by Devlin et al. (2014).As inherited from the n-gram language model for performing hierarchical decoding, the leftmost and rightmost n − 1 words from each constituent should be stored in the state space.We extend the state space to also include the indexes of the affiliated source words for each of these edge words.For an aligned target word, we take its aligned source words as its affiliated source words.And for an unaligned word, we use the affiliation heuristic adopted by Devlin et al. (2014).In this paper, we integrate the joint model into the state-of-the-art dependency-to-string machine translation decoder as a case study to test the efficacy of our proposed approaches.We will briefly describe the dependency-to-string translation model and then the description of MT system.

Dependency-to-String Translation
In this paper, we use a state-of-the-art dependency-to-string (Xie et al., 2011) decoder (Dep2Str), which is also a hierarchical decoder.This dependency-to-string model employs rules that represent the source side as head-dependents relations and the target side as strings.A head-dependents relation (HDR) is composed of a head and all its dependents in dependency trees.Figure 3 shows a dependency tree (a) with three HDRs (in shadow), an example of HDR rule (b) for the top level of (a), and an example of head rule (c).HDR rules are constructed from head-dependents relations.HDR rules can act as both translation rules and reordering rules.And head rules are used for translating source words.
We adopt the decoder proposed by Meng et al. (2013) as a variant of Dep2Str translation that is easier to implement with comparable performance.Basically they extract the HDR rules with GHKM (Galley et al., 2004) algorithm.For the decoding procedure, given a source dependency tree T , the decoder transverses T in post-order.The bottom-up chart-based decoding algorithm with cube pruning (Chiang, 2007;Huang and Chiang, 2007) is used to find the k-best items for each node.

MT Decoder
Following Och and Ney (2002), we use a general loglinear framework.Let d be a derivation that convert a source dependency tree into a target string e.The probability of d is defined as: where φ i are features defined on derivations and λ i are the corresponding weights.Our decoder contains the following features: Baseline Features: • translation probabilities P (t|s) and P (s|t) of HDR rules; • lexical translation probabilities P LEX (t|s) and P LEX (s|t) of HDR rules; • rule penalty exp(−1); • pseudo translation rule penalty exp(−1); • target word penalty exp(|e|); • n-gram language model P LM (e); Proposed Features: • n-gram tagCNN joint language model P TLM (e); • n-gram inCNN joint language model P ILM (e).
Our baseline decoder contains the first eight features.The pseudo translation rule (constructed according to the word order of a HDR) is to ensure the complete translation when no matched rules is found during decoding.The weights of all these features are tuned via minimum error rate training (MERT) (Och, 2003).For the dependency-to-string decoder, we set rule-threshold and stack-threshold to 10 −3 , rule-limit to 100, stack-limit to 200.

Experiments
The experiments in this Section are designed to answer the following questions: 1. Are our tagCNN and inCNN joint language models able to improve translation quality, and are they complementary to each other?
2. Do inCNN and tagCNN benefit from their guiding signal, compared to a generic CNN?
3. For tagCNN, is it helpful to embed more dependency structure, e.g., dependency head of each affiliated word, as additional information?

Setup
Data: Our training data are extracted from LDC data2 .We only keep the sentence pairs that the length of source part no longer than 40 words, which covers over 90% of the sentence.The bilingual training data consist of 221K sentence pairs, containing 5.0 million Chinese words and 6.8 million English words.The development set is NIST MT03 (795 sentences) and test sets are MT04 (1499 sentences) and MT05 (917 sentences) after filtering with length limit.
Preprocessing: The word alignments are obtained with GIZA++ (Och and Ney, 2003) on the corpora in both directions, using the "grow-diag-final-and" balance strategy (Koehn et al., 2003).We adopt SRI Language Modeling Toolkit (Stolcke and others, 2002) to train a 4-gram language model with modified Kneser-Ney smoothing on the Xinhua portion of the English Gigaword corpus (306 million words).We parse the Chinese sentences with Stanford Parser into projective dependency trees.

Optimization of NN:
In training the neural network, we limit the source and target vocabulary to the most frequent 20K words for both Chinese and English, covering approximately 97% and 99% of two corpus respectively.All the out-of-vocabulary words are mapped to a special token UNK.We used stochastic gradient descent to train the joint model, setting the size of minibatch to 500.All joint models used a 3-word target history (i.e., 4-gram LM).The dimension of word embedding is 100.Metric: We use the case-insensitive 4-gram NIST BLEU3 as our evaluation metric, with statistical significance test with sign-test (Collins et al., 2005) between the proposed models and two baselines.

Setting for Model Comparisons
We use the tagCNN and inCNN joint language models as additional decoding features to a dependency-to-string baseline system (Dep2Str), and compare them to the neural network joint model with 11 source context words (Devlin et al., 2014).We use the implementation of an open source toolkit4 with default configuration except the global settings described in Section 5.1.Since our tagCNN and inCNN models are source-to-target and left-to-right (on target side), we only take the source-to-target and left-to-right type NNJM in (Devlin et al., 2014) in comparison.We call this type NNJM as BBN-JM hereafter.Although the BBN-JM in (Devlin et al., 2014) is originally tested in the hierarchical phrase-based (Chiang, 2007) SMT and string-to-dependency (Shen et al., 2008) SMT, it is fairly versatile and can be readily integrated into Dep2Str.

The Main Results
The main results of different models are given in Table 1.Before proceeding to more detailed comparison, we first observe that • the baseline Dep2Str system gives BLEU 0.5+ higher than the open-source phrase-based system Moses (Koehn et al., 2007); • BBN-JM can give about +0.92 BLEU score over Dep2Str, a result similar as reported in (Devlin et al., 2014).
Clearly than the affiliated words suggested by the word alignment (GIZA++).We conjecture that this is due to the following two facts • inCNN avoids the propagation of mistakes and artifacts in the already learned word alignment; • the guiding signal in inCNN provides complementary information to evaluate the translation.
Moreover, when tagCNN and inCNN are both used in decoding, it can further increase its winning margin over BBN-JM to +1.01 BLEU points (in the last row of Table 1), indicating that the two models with different guiding signals are complementary to each other.
The Role of Guiding Signal It is slight surprising that the generic CNN can also achieve the gain on BLEU similar to that of BBN-JM, since intuitively generic CNN encodes the entire sentence and the representations should in general far from optimal representation for joint language model.The reason, as we conjecture, is CNN yields fairly informative summarization of the sentence (thanks to its sophisticated convolution and gating architecture), which makes up some of its loss on resolution and relevant parts of the source senescence.That said, the guiding signal in both tagCNN and inCNN are crucial to the power of CNN-based encoder, as can be easily seen from the difference between the BLEU scores achieved by generic CNN, tagCNN, and inCNN.Indeed, with the signal from the already learned word alignment, tagCNN can gain +0.31 BLEU over its generic counterpart, while for inCNN with the guiding signal from the proceeding words in target, the gain is more saliently +0.66 BLEU.

Dependency Head in tagCNN
In this section, we study whether tagCNN can further benefit from encoding richer dependency structure in source language in the input.More specifically, we dependency head words can be used to further improve tagCNN model.As described in Section 3.2, in tagCNN, we append a tagging bit (0 or 1) to the embedding of words in the input layer as tags on whether they are affiliated source words.To incorporate dependency head information, we extend the tagging rule in Section 3.2 to add another tagging bit (0 or 1) to the word-embedding for original tagCNN to indicate whether it is part of dependency heads of the affiliated words.For example, if x i is the embedding of an affiliated source word and x j the dependency head of word x i , the extended input of tagCNN would contain If the affiliated source word is the root of a sentence, we only append 0 as the second tagging bit since the root has no dependency head.From Table 2, with the help of dependency head information, we can improve tagCNN by +0.23 BLEU points averagely on two test sets.

Related Work
The seminal work of neural network language model (NNLM) can be traced to Bengio et al. (2003) on monolingual text.It is recently extended by Devlin et al. (2014) to include additional source context (11 source words) in modeling the target sentence, which is clearly most related to our work, with however two important differences: 1) instead of the ad hoc way of selecting a context window in (Devlin et al., 2014), our model covers the entire source sentence and automatically distill the context relevant for target modeling; 2) our convolutional architecture can effectively leverage guiding signals of vastly different forms and nature from the target.Prior to our model there is also work on representing source sentences with neural networks, including RNN (Cho et al., 2014;Sutskever et al., 2014) and CNN (Kalchbrenner and Blunsom, 2013).These work typically aim to map the entire sentence to a vector, which will be used later by RNN/LSTM-based decoder to generate the target sentence.As demonstrated in Section 5, the representation learnt this way cannot pinpoint the relevant parts of the source sentences (e.g., words or phrases level) and therefore is inferior to be directly integrated into traditional SMT decoders.
Our model, especially inCNN, is inspired by is the automatic alignment model proposed in (Bahdanau et al., 2014).As the first effort to apply attention model to machine translation, it sends the state of a decoding RNN as attentional signal to the source end to obtain a weighted sum of embedding of source words as the summary of relevant context.In contrast, inCNN uses 1) a different attention signal extracted from proceeding words in partial translations, and 2) more importantly, a convolutional architecture and therefore a highly nonlinear way to retrieve and summarize the relevant information in source.

Conclusion and Future Work
We proposed convolutional architectures for obtaining a guided representation of the entire source sentence, which can be used to augment the n-gram target language model.With different guiding signals from target side, we devise tagCNN and inCNN, both of which are tested in enhancing a dependency-to-string SMT with +1.93 BLEU points over baseline and +1.01 BLEU points over the state-of-the-art in (Devlin et al., 2014).For future work, we will consider encoding more complex linguistic structures to further enhance the joint model.

Figure 1 :
Figure 1: Illustration for joint LM based on CNN encoder.

Table 1 :
from Table1, tagCNN and inCNN improve upon the Dep2Str baseline by +1.28 and +1.63 BLEU, outperforming BBN-JM in the same setting by respectively +0.36 and +0.71 BLEU, averaged on NIST MT04 and MT05.These indicate that tagCNN and inCNN can individually provide discriminative information in decoding.It is worth noting that inCNN appears to be more informative BLEU-4 scores (%) on NIST MT04-test and MT05-test, of Moses (default settings), dependency-to-string baseline system (Dep2Str), and different features on top of Dep2Str: neural network joint model (BBN-JM), generic CNN, tagCNN, inCNN and the combination of tagCNN and inCNN.The boldface numbers and superscript * indicate that the results are significantly better (p<0.01)than those of the BBN-JM and the Dep2Str baseline respectively."+" stands for adding the corresponding feature to Dep2Str.