Probabilistic Graph-based Dependency Parsing with Convolutional Neural Network

This paper presents neural probabilistic parsing models which explore up to third-order graph-based parsing with maximum likelihood training criteria. Two neural network extensions are exploited for performance improvement. Firstly, a convolutional layer that absorbs the inﬂuences of all words in a sentence is used so that sentence-level information can be effectively captured. Secondly, a linear layer is added to integrate different order neural models and trained with perceptron method. The proposed parsers are evaluated on English and Chinese Penn Tree-banks and obtain competitive accuracies.


Introduction
Neural network methods have shown great promise in the field of parsing and other related natural language processing tasks, exploiting more complex features with distributed representation and non-linear neural network (Wang et al., 2013;Wang et al., 2014;Cai and Zhao, 2016;Wang et al., 2016). In transition-based dependency parsing, neural models that can represent the partial or whole parsing histories have been explored (Weiss et al., 2015;Dyer et al., 2015). While for graphbased parsing, on which we focus in this work, Pei et al. (2015) also show the effectiveness of neural methods.
The graph-based parser generally consists of two components: one is the parsing algorithm for inference or searching the most likely parse tree, the other is the parameter estimation approach for the machine learning models. For the former, classical dynamic programming algorithms are usually adopted, while for the latter, there are various solutions. Like some previous neural methods (Socher et al., 2010;Socher et al., 2013), to tackle the structure prediction problems, Pei et al. (2015) utilize a max-margin training criterion, which does not include probabilistic explanations. Re-visiting the traditional probabilistic criteria in log-linear models, this work utilizes maximum likelihood for neural network training. Durrett and Klein (2015) adopt this method for constituency parsing, which scores the anchored rules with neural models and formalizes the probabilities with tree-structured random fields. Motivated by this work, we utilize the probabilistic treatment for dependency parsing: scoring the edges or high-order sub-trees with a neural model and calculating the gradients according to probabilistic criteria. Although scores are computed by a neural network, the existing dynamic programming algorithms for gradient calculation remain the same as those in log-linear models.
Graph-based methods search globally through the whole space for trees and get the highestscored one, however, the scores for the sub-trees are usually locally decided, considering only surrounding words within a limited-sized window. Convolutional neural network (CNN) provides a natural way to model a whole sentence. By introducing a distance-aware convolutional layer, sentence-level representation can be exploited for parsing. We will especially verify the effectiveness of such representation incorporated with window-based representation.
Graph-based parsing has a natural extension through raising its order and higher-order parsers usually perform better. In previous work on highorder graph-parsing, the scores of high-order subtrees usually include the lower-order parts in their high-order factorizations. In traditional linear models, combining scores can be implemented by including low-order features. However, for neural models, this is not that straightforward because of nonlinearity. A straightforward strategy is simply adding up all the scores, which in fact works well; another way is stacking a linear layer on the top of the representation from various already-trained neural parsing models of different orders. This paper presents neural probabilistic models for graph-based projective dependency parsing, and explores up to third-order models. Here are the three highlights of the proposed methods: • Probabilistic criteria for neural network training. (Section 2.2) • Sentence-level representation learned from a convolutional layer. (Section 3.2) • Ensemble models with a stacked linear output layer. (Section 3.3) Our main contribution is exploring sub-tree scoring models which combine local features with a window-based neural network and global features from a distance-aware convolutional neural network. A free distribution of our implementation is publicly available 1 . The remainder of the paper is organized as follows: Section 2 explains the probabilistic model for graph-based parsing, Section 3 describes our neural network models, Section 4 presents our experiments and Section 5 discusses related work, we summarize this paper in Section 6.
2 Probabilistic Graph-based Dependency Parsing

Graph-based Dependency Parsing
Dependency parsing aims to predict a dependency tree, in which all the edges connect head-modifier pairs. In graph-based methods, a dependency tree is factored into sub-trees, from single edge to multiple edges with different patterns; we will call these specified sub-trees factors in this paper. According to the sub-tree size of the factors, we can  define the order of the graph model. Three different ordered factorizations considered in this work and their sub-tree patterns are shown in Figure 1.
The score for a dependency tree (T ) is defined as the sum of the scores of all its factors (p): In this way, the dependency parsing task is to find a max-scoring tree. For projective dependency parsing considered in this work, this searching problem is conquered by dynamic programming algorithms with the key assumption that the factors are scored independently. Previous work (Eisner, 1996;McDonald et al., 2005;McDonald and Pereira, 2006;Koo and Collins, 2010;Ma and Zhao, 2012) explores ingenious algorithms for decoding ranging from first-order to higher-orders. Our proposed parsers also take these algorithms as backbones and use them for inference.

Probabilistic Model
With the graph factorization and inference, the remaining problems are how to obtain the scores and how to train the scoring model. For the scoring models, traditional linear methods utilize manually specified features and linear scoring models, while we adopt neural network models, which may exploit better feature representations.
For the training methods, in recent neural graph-based parsers, non-probabilistic marginbased methods are usually used. However, following the maximum likelihood criteria in traditional log-linear models, we can treat it in a probabilistic way. In fact, the probabilistic treatment still utilizes the scores of sub-tree factors in graph models. As in log-linear models like Conditional Random Field (CRF) (Lafferty et al., 2001), the exponentials of scores are taken before re-normalizing, and the probability distribution over trees condi-tioned on a sentence X is defined as follows: where θ represents the parameters and Z(X) is the re-normalization partition function. The intuition is that the higher the score is, the more potential or mass it will get, leading to higher probability. The training criteria will be log-likelihood in the classical setting of maximum likelihood estimation, and we define the loss for a parse tree as negative log-likelihood: where T g stands for the golden parse tree. Now we need to calculate the gradients of θ according to gradient-based optimization. Focusing on the second term, we have (some conditions are left out for simplicity):

Pr(T )
Here, T (p) is the set of trees that contain the factor p, and the inner summation is defined as the marginal probability m(p): which can be viewed as the mass of all the trees containing the specified factor p. The calculation of m(p) (Paskin, 2001;Ma and Zhao, 2015) is solved by a variant of inside-outside algorithm, which is of the same complexity compared with the corresponding inference algorithms. Finally, the gradients can be represented as: where [p ∈ T g ] is a binary value which indicates whether p is in tree T g . Traditional models usually utilize linear functions for the Score function, which might need carefully feature engineering such as (Zhao et al., 2009a;Zhao et al., 2009b;Zhao et al., 2009c;Zhao, 2009;, while we adopt neural models with the probabilistic training criteria unchanged.

Training Criteria
We take a further look between the maximumlikelihood criteria and the max-margin criteria. For the max-margin method, the loss is the difference between the scores of the golden tree and a predicted tree, and its sub-gradient can be written in a similar form: Here, the predicted tree T b is the best-scored tree with a structured margin loss in the score.
Comparing the derivatives, we can see that the one of probabilistic criteria can be viewed as a soft version of the max-margin criteria, and all the possible factors are considered when calculating gradients for the probabilistic way, while only wrongly predicted factors have non-zero subgradients for max-margin training. This observation is not new and Gimpel and Smith (2010) provide a good review of several training criteria. It might be interesting to explore the impacts of different training criteria on the parsing performance, and we will leave it for future research.

Labeled Parsing
In a dependency tree, each edge can be given a label indicating the type of the dependency relation, this labeling procedure can be integrated directly into the parsing task, instead of a second pass after obtaining the structure. For the probabilistic model, integrating labeled parsing only needs some extensions for the inference procedure and marginal probability calculations. For the simplicity, we only consider a single label for each factor (even for high-order ones) which corresponds to Model 1 in (Ma and Hovy, 2015): the label of the edge between head and modifier word, which will only multiply O(l) to the complexity. We find this direct approach not only achieves labeled parsing in one pass, but also improves unlabeled attachment accuracies (see Section 4.3), which may benefit from the joint learning with the labels.

Neural Model
The task for the neural models is computing the labeled scores of the factors. The inputs are the words in a factor with contexts, and the outputs are the scores for this factor to be valid in the dependency tree. We propose neural models to in- tegrate features from both local word-neighboring windows and the entire sentence, and furthermore explore ensemble models with different orders.

Basic Local Model
Architecture The basic model uses a windowbased approach, which includes only surrounding words for the contexts. Figure 2 illustrates a second-order sibling model and models of other orders adopt similar structures. It is simply a standard feed-forward neural network with two hidden layers (h 1 and h 2 ) above the embedding layer (h 0 ), the hidden layers all adopt tanh activation function, and the output layer (noted as s) directly represents the scores for different labels.
Feature Sets All the features representing the input factor are atomic and projected to embeddings, then the embedding layer is formed by concatenating them. There are three categories of features: word forms, POS (part-of-speech) tags and distances. For each node in the factor, word forms and POS tags of the surrounding words in a specified window are also considered. Special tokens for start or end of sentences, root node and unknown words are added for both word forms and POS tags. Distances can be negative or positive to represent the relative positions between the factor nodes in surface string. Take the situation for the second-order model as an example, there are three nodes in a factor: h for head, m for modifier and s for sibling. When considering three-word windows, there will be three word forms and three tags for each node and its surrounding context. m and s both have one distance feature while h does not have one as its parent does not exist in the factor.
Training As stated in Section 2.2, we use the maximum likelihood criteria. Moreover, we add two L2-regularizations: one is for all the weights θ (biases and embeddings not included) to avoid over-fitting and another is for preventing the final output scores from growing too large. The former is common practice for neural network, while the latter is to set soft limits for the norms of the scores. Although the second term is not usually adopted, it directly puts soft constraints on the scores and improves the accuracies (about 0.1% for UAS/LAS overall) according to our primary experiments. So the final loss function will be: where λ m and λ s respectively represent regularization parameters for model and scores. The training process utilizes a mini-batched stochastic gradient descent method with momentum.
Comparisons Our basic model resembles the one of Pei et al. (2015), but with some major differences: probabilistic training criteria are adopted, the structures of the proposed networks are different and direction information is encoded in distance features. Moreover, they simply average embeddings in specified regions for phraseembedding, while we will include sentenceembedding in convolutional model as follows.

Convolutional Model
To encode sentence-level information and obtain sentence embeddings, a convolutional layer of the whole sentence followed by a max-pooling layer is adopted. However, we intend to score a factor in a sentence and the position of the nodes should also be encoded. The scheme is to use the distance embedding for the whole convolution window as the position feature. We will take the second-order model as an example to introduce the related operations. Figure  3 shows the convolution operation for a convolution window, the input atomic features are the word forms and POS tags for each word inside the window, and the distances of only the center word (assuming an odd-sized window) to specified nodes in the factor are adopted as position features. In the example, "game-good-a" is to be scored as a second-order sibling factor, and for a convolution window of "This is a", word forms and corresponding POS tags are projected to embeddings and concatenated as the lexical vector v l , the distances of the center word "is" to all the three nodes in the factor are also projected to embeddings and concatenated as the distance vector v d , then these two vectors go through difference linear transformations into the same dimension and are combined together through element-wise addition or multiplication.
In general, assuming after the projection layer, embeddings of the word forms and POS tags of the sentence are represented as [w 0 , w 1 , ..., w n−1 ] and [p 0 , p 1 , ..., p n−1 ]. Those embeddings in the basic model may be reused here by sharing the embedding look-up table. The second-order sibling factor to be scored has nodes with indexes of m (modifier), h (head) and s (sibling). The distance embeddings are denoted by d, which can be either negative or positive. These distance embeddings are different from the ones in the basic model, because here we measure the distances between the convolution window (its center word) and factor nodes, while the distances between nodes inside the factors are measured in the basic model.
For a specified window [i : j], always assuming an odd number sized window, and the center token is indexed to c = i+j 2 , the v l and v d are obtained through simple concatenation: then v l and v d go through difference linear transformations into same dimension space: v l , v d ∈ R n , where n is also the dimension of the output vector v o for the window. The linear operations can be expressed as: The final vector v o is obtained by element-wise operations of v l and v d . We consider two strategies: (1) add: simple element-wise addition, (2) mul: element-wise multiplication with v d activated by tanh. They can be formalized as: All the windows whose center-located word is valid (exists) in the sentence are considered and we will get a sequence of convolution outputs whose number is the same as the sentence length. The convolution outputs (all v o ) are collapsed into one global vector v g using a standard max-pooling operation. Finally, for utilizing the sentence-level representation in the basic model, we can either replace the original first hidden layer h 1 with v g or concatenate v g to h 1 for combining local and global features.

Ensemble Models
For higher-order dependency parsing, it is a standard practice to include the impact of lower-order parts in the scoring of higher-order factors, which actually is an ensemble method of different order models for scoring.
A simple adding scheme is often used. For nonlinear neural models, we use an explicit adding method. For example, in third-order parsing, the final score for the factor (g, h, m, s) will be: s add (g, h, m, s) = s o3 (g, h, m, s) + s o2 (h, m, s) Here, g, h, m and s represent the grandparent, head, modifier and sibling nodes in the grandsibling third-order factor; s o1 , s o2 and s o3 stand for the corresponding lower-order scores from first, second and third order models, respectively. We notice that ensemble or stacking methods for dependency parsing have explored in previous work (Nivre and McDonald, 2008;Torres Martins et al., 2008). Recently, Weiss et al. (2015) stack a linear layer for the final scoring in a single model, and we extend this method to combine multiple models by stacking a linear layer on their output and hidden layers. The simple adding scheme can be viewed as adopting a final layer with specially fixed weights.
For each model to be combined, we concatenate the output layer and all hidden layers (except embedding layer h 0 ): All v all from different models are again concatenated to form the input for the final linear layer and the final scores are obtained through a linear transformation (no bias adding): We no longer update weights for the underlying neural models, and the learning of the final layer is equally training a linear model, for which structured average perceptron (Collins, 2002;Collins and Roark, 2004) is adopted for simplicity.
This ensemble scheme can be extended in several ways which might be explored in future work: (1) feed-forward network can be stacked rather than a single linear layer, (2) traditional sparse features can also be concatenated to v combine to combine manually specified representations with distributed neural representations as in .

Experiments
The proposed parsers are evaluated on English Penn Treebank (PTB) and Chinese Penn Treebank (CTB). Unlabeled attachment scores (UAS), labeled attachment scores (LAS) and unlabeled complete matches (CM) are the metrics. Punctuations 2 are ignored as in previous work (Koo and Collins, 2010;Zhang and Clark, 2008).
For English, we follow the splitting convention for PTB3: sections 2-21 for training, 22 for developing and 23 for test. We prepare three datasets of PTB, using different conversion tools: (1) Penn2Malt 3 and the head rules of Yamada and Matsumoto (2003), noted as PTB-Y&M; (2) dependency converter in Stanford parser v3.3.0 with Stanford Basic Dependencies (De Marneffe et al., 2006), noted as PTB-SD; (3) LTH Constituentto-Dependency Conversion Tool 4 (Johansson and Nugues, 2007), noted as PTB-LTH. We use Stanford POS tagger (Toutanova et al., 2003) to get predicted POS tags for development and test sets, and the accuracies for their tags are 97.2% and 97.4%, respectively.
For Chinese, we adopt the splitting convention for CTB5 described in (Zhang and Clark, 2008). The dependencies (noted as CTB), are converted with the Penn2Malt converter. Gold segmentation and POS tags are used as in previous work.

Settings
Settings of our models will be described in this sub-section, including pre-processing and initializations, hyper-parameters, and training details.
We ignore the words that occur less than 3 times in the training treebank and use a special token to replace them. For English parsing, we initialize word embeddings with word vectors trained on Wikipedia using word2vec (Mikolov et al., 2013); all other weights and biases are initialized randomly with uniform distribution.
For the structures of neural models, all the embeddings (word, POS and distances) have dimensions of 50. For basic local models, h 1 and h 2 are set to 200 and 100, and the local window size is set to 7. For convolutional models, a three-word-sized window for convolution is specified, and convolution output dimension (number of filters) is 100. When concatenating the convolution vector (after pooling) to h 1 , it will make the first hidden layer's dimension 300.
For the training of neural network, we set the initial learning rate to 0.1 and the momentum to 0.6. After each iteration, the parser is tested on the development set and if the accuracy decreases, the learning rate will be halved. The learning rate will also be halved if no decreases of the accuracy for three epochs. We train the neural models for 12 epochs and select the one that performs best on the development set. The regularization parameters λ m and λ s are set to 0.0001 and 0.001. For the perceptron training of the ensemble model, only one epoch is enough based on the results of the development set.
The runtime of the model is influenced by the hyper-parameter setting. According to our experiments, using dual-core on 3.0 GHz i7 CPU, the training costs 6 to 15 hours for different-order models and the testing is comparably efficient as recent neural graph-parsers. The calculation of the convolution model approximately takes up 40% of all computations. The convolution operation indeed costs more, but the lexical parts v l of the convolution do not concern the factors and are computed only once for one sentence, which makes it less computationally expensive.

Pruning
For high-order parsing, the computation cost rises in proportion to the length of the sentence, and it will be too expensive to calculate scores for all the factors. Fortunately, many edges are quite unlikely to be valid and can be pruned away using low-order models. We follow the method of Koo and Collins (2010) and directly use the first-order probabilistic neural parser for pruning. We compute the marginal probability m(h, m) for each edge and prune away the edges whose marginal probability is below × max h m(h , m). means the pruning threshold that is set to 0.0001 for second-order. For third-order parsing, considering the computational cost, we set it to 0.001.

Model Analysis
This section presents experiments to verify the effectiveness of the proposed methods and only the PTB-SD development set will be used in these experiments, which fall into three groups concerning basic models, convolutional models and ensemble ones, as shown in Table 1. The first group focuses on the basic local models of first order. The first two, Unlabeled and Labeled, do not use pre-training vectors for initialization, while the third, Labeled+pre-training, utilizes them. The Unlabeled does not utilize the labels in training set and its model only gives one dependency score (we do not train a second stage labeling model, so the LAS of the unlabeled one is not available) and the Labeled directly predicts the scores for all labels. We can see that labeled parsing not only demonstrates the convenience of outputting dependency relations and labels for once, but also obtains better parsing performances. Also, we observe that pre-trained word vectors bring slight improvements. Pre-trained initialization and labeled parsing will be adopted for the next two groups and the rest experiments.
Next, we explore the effectiveness of the CNN enhancement. In the four entries of this group, concatenate or replace means whether to concatenate the sentence-level vector v g to the first hidden layer h 1 or just replace it (just throw away the representation from basic models), add or mul means to use which way for attaching distance information. Surprisingly, simple adding method surpasses the more complex multiplication-withactivation method, which might indicate that the direct activation operation may not be suitable for encoding distance information. With no surprises, the concatenating method works better because it combines both the local window-based and global sentence-level information. We also explore the influences of the convolution operations on dependencies of different lengths, as shown in Figure  4, the convolutional methods help the decisions of long-range dependencies generally. For the highorder parsing in the rest of this paper, we will all adopt the concatenate-add setting.
In the third group, we can see that high-order parsing brings significant performance improvement. For high-order parsing, three ensemble schemes are examined: no combination, adding  (2015) ------91.6-88.9-----Zhang and Zhao (2015) ------92.52 -41.10 86.01 -31.88 Graph-Linear Koo and Collins (2010) 93  and stacking another linear perceptron layer (with the suffixes of -nope, -adding and -perceptron respectively). The results show that model ensemble improves the accuracies quite a few. For thirdorder parsing, the no-combination method performs quite poorly compared to the others, which may be caused by the relative strict setting of the pruning threshold. Nevertheless, with model ensemble, the third-order models perform better than the second-order ones. Though the perceptron strategy does not work well for third-order parsing in this dataset, it is still more general than the simple adding method, since the latter can be seen as a special parameter setting of the former.

Results
We show the results of two of the best proposed parsers: third-order adding (o3-adding) and thirdorder perceptron (o3-perceptron) methods, and compare with the reported results of some previous work in Table 2. We compare with three categories of models: other Graph-based NN (neural network) models, traditional Graph-based Linear models and Transition-based NN models. For PTB, there have been several different dependency converters which lead to different sets of dependencies and we choose three of the most popular ones for more comprehensive comparisons. Since not all work report results on all of these dependencies, some of the entries might be not available.
From the comparison, we see that the proposed parser has output competitive performance for different dependency conversion conventions and treebanks. Compared with traditional graph-based linear models, neural models may benefit from better feature representations and more general non-linear transformations.
The results and comparisons in Table 2 demonstrate the proposed models can obtain comparable accuracies, which show the effectiveness of combining local and global features through windowbased and convolutional neural networks.

Related Work
CNN has been explored in recent work of relation classification (Zeng et al., 2014;Chen et al., 2015), which resembles the task of deciding dependency relations in parsing. However, relation classification usually involves labeling for given arguments and seldom needs to consider the global structure. Parsing is more complex for it needs to predict structures and the use of CNN should be incorporated with the searching algorithms.
Neural network methods have been proved effective for graph-based parsing. Lei et al. (2014) explore a tensor scoring method, however, it needs to combine scores from linear models and we are not able to compare with it because of different datasets (they take datasets from CoNLL shared task). Zhang and Zhao (2015) also explore a probabilistic treatment, but its model may give mass to illegal trees or non-trees. Fonseca and Aluísio (2015) utilize CNN for scoring edges, though only explore first-order parsing. Its model is based on head selection for each modifier and might be difficult to be extended to high-order parsing. Recently, several neural re-ranking models, like Inside-Outside Recursive Neural Network (Le and Zuidema, 2014) and Recursive CNN (Zhu et al., 2015), are utilized for capturing features with more contexts. However, re-ranking models depend on the underlying base parsers, which might already miss the correct trees. Generally, the re-ranking techniques play a role of additional enhancement for basic parsing models, and therefore they are not included in our comparisons.
The conditional log-likelihood probabilistic criterion utilized in this work is actually a (conditioned) Markov Random Field for tree structures, and it has been applied to parsing since long time ago. Johnson et al. (1999) utilize the Markov Random Fields for stochastic grammars and gradient based methods are adopted for parameter estimations, and Geman and Johnson (2002) extend this with dynamic programming algorithms for inference and marginal-probability calculation. Collins (2000) uses the same probabilistic treatment for re-ranking and the denominator only includes the candidate trees which can be seen as an approximation for the whole space of trees. Finkel et al. (2008) utilize it for feature-based parsing. The probabilistic training criterion for linear graphbased dependency models have been also explored in (Li et al., 2014;Ma and Zhao, 2015). However, these previous methods usually exploit loglinear models utilizing sparse features for input representations and linear models for score calculations, which are replaced by more sophisticated distributed representations and neural models, as shown in this work.

Conclusions
This work presents neural probabilistic graphbased models for dependency parsing, together with a convolutional part which could capture the sentence-level information. With distributed vectors for representations and complex non-linear neural network for calculations, the model can effectively capture more complex features when deciding the scores for sub-tree factors and experiments on standard treebanks show that the proposed techniques improve parsing accuracies.