Morphological Segmentation Inside-Out

Morphological segmentation has traditionally been modeled with non-hierarchical models, which yield flat segmentations as output. In many cases, however, proper morphological analysis requires hierarchical structure -- especially in the case of derivational morphology. In this work, we introduce a discriminative, joint model of morphological segmentation along with the orthographic changes that occur during word formation. To the best of our knowledge, this is the first attempt to approach discriminative segmentation with a context-free model. Additionally, we release an annotated treebank of 7454 English words with constituency parses, encouraging future research in this area.


Introduction
In NLP, supervised morphological segmentation has typically been viewed as either a sequence-labeling or a segmentation task (Ruokolainen et al., 2016).
In contrast, we consider a hierarchical approach, employing a context-free grammar (CFG). CFGs provide a richer model of morphology: They capture (i) the intuition that words themselves have internal constituents, which belong to different categories, as well as (ii) the order in which affixes are attached. Moreover, many morphological processes, e.g., compounding and reduplication, are best modeled as hierarchical; thus, context-free models are expressively more appropriate.
The purpose of morphological segmentation is to decompose words into smaller units, known as morphemes, which are typically taken to be the smallest meaning-bearing units in language. This work concerns itself with modeling hierarchical structure over these morphemes. Note a simple flat morphological segmentation can also be straightforwardly derived from the CFG parse tree. Segmentations have found use in a diverse set of NLP applications, e.g., automatic speech recognition (Afify et al., 2006), keyword spotting (Narasimhan et al., 2014), machine translation (Clifton and Sarkar, 2011) and parsing (Seeker and Ç etinoglu, 2015). In contrast to prior work, we focus on canonical segmentation, i.e., we seek to jointly model orthographic changes and segmentation. For instance, the canonical segmentation of untestably is un+test+able+ly, where we map ably to able+ly, restoring the letters le.
We make two contributions: (i) We introduce a joint model for canonical segmentation with a CFG backbone. We experimentally show that this model outperforms a semi-Markov model on flat segmentation. (ii) We release the first morphology treebank, consisting of 7454 English word types, each annotated with a full constituency parse.

The Case For Hierarchical Structure
Why should we analyze morphology hierarchically?
It is true that we can model much of morphology with finite-state machinery (Beesley and Karttunen, 2003), but there are, nevertheless, many cases where hierarchical structure appears requisite. For instance, the flat segmentation of the word untestably →un+test+able+ly is missing important information about how the word was derived.
The correct parse [[un[[test]able]]ly], on the other hand, does tell us that this is the order in which the complex form was derived: This gives us insight into the structure of the lexicon-we expect that the segment testable exists as an independent word, but ably does not.
Moreover, a flat segmentation is often semantically ambiguous. There are two potentially valid readings of untestably depending on how the negative prefix un scopes. The correct tree (see Arguably, (a) is the only correct parse tree for untestably; the reading associated with (b) is hard to get. On the other hand, unlockable is truly ambiguous between "able to be unlocked" (c) and "unable to be locked" (d).
able to be tested." A second-likely infelicitous reading-where the segment untest forms a constituent yields the reading "in a manner of being able to untest." Recovering the hierarchical structure allows us to select the correct reading; note there are even cases of true ambiguity; e.g., unlockable has two readings: "unable to be locked" and "able to be unlocked. " We also note that theoretical linguists often implicitly assume a context-free treatment of word formation, e.g., by employing brackets to indicate different levels of affixation. Others have explicitly modeled word-internal structure with grammars (Selkirk, 1982;Marvin, 2002).

Parsing the Lexicon
A novel component of this work is the development of a discriminative parser (Finkel et al., 2008;Hall et al., 2014) for morphology. The goal is to define a probability distribution over all trees that could arise from the input word, after reversal of orthographic and phonological processes. We employ the simple grammar shown in Table 1. Despite its simplicity, it models the order in which morphemes are attached.
More formally, our goal is to map a surface form w (e.g., w=untestably) into its underlying canonical form u (e.g., u=untestablely) and then into a parse tree t over its morphemes. We assume u, w ∈ Σ * , for some discrete alphabet Σ. 1 Note that a parse tree over the string implicitly defines a flat segmentation given our grammar-one can simply extract the characters spanned by all preterminals in the resulting tree. Before describing the joint model in detail, we first consider its pieces individually.

Restoring Orthographic Changes
To extract a canonical segmentation (Naradowsky and Goldwater, 2009;Cotterell et al., 2016), we restore orthographic changes that occur during word formation. To this end, we define the score function where a is a monotonic alignment between the strings u and w. The goal is for score η to assign higher values to better matched pairs, e.g., (w=untestably, u=untestablely). We refer to Dreyer et al. (2008) for a thorough exposition. For ease of computation, we can encode this function as a weighted finite-state machine (WFST) (Mohri et al., 2002). This requires, however, that the feature function g factors over the topology of the finite-state encoding. Since our model conditions on the word w, the feature function g can extract features from any part of this string. Features on the output string, u, however, are more restricted. In this work, we employ a bigram model over output characters. This implies that each state remembers exactly one character, the previous one. See Cotterell et al. (2014) for details. We can compute the score for two strings u and w using a weighted generalization of the Levenshtein algorithm. Computing the partition function requires a different dynamic program, which runs in O(|w| 2 · |Σ| 2 ) time. Note that since |Σ| ≈ 26 (lower case English letters), it takes roughly 26 2 = 676 times longer to compute the partition function than to score a pair of strings.
Our model includes several simple feature templates, including features that fire on individual edit actions as well as conjunctions of edit actions and characters in the surrounding context. See Cotterell et al. (2016) for details.

Morphological Analysis as Parsing
Next, we need to score an underlying canonical form (e.g., u=untestablely) together with a parse tree (e.g., t= [[un[[test]able]]ly]). Thus, we define the parser score with the following function where Π(t) is the set of anchored productions in the tree t. An anchored production π is a grammar rule in Chomsky normal form attached to a span, e.g., A i,k → B i,j C j+1,k . Each π is then assigned a weight by the linear function f (π, u) ⊤ ω, where the function f extracts relevant features from the anchored production as well as the corresponding span of the underlying form u. This model is typically referred to as a weighted CFG (WCFG) (Smith and  or a CRF parser. For f , we define three span features: (i) indicator features on the span's segment, (ii) an indicator feature that fires if the segment appears in an external corpus 2 and (iii) the conjunction of the segment with the label (e.g., PREFIX) of the subtree root. Following Hall et al. (2014), we employ an indicator feature for each production as well as production backoff features.

A Joint Model
Our We have adjusted the definition of the model from the original paper to directly introduce the alignment a between the strings u and w.
complete model is a joint CRF (Koller and Friedman, 2009) where each of the above scores are factors. We define the following probability distribution over trees, canonical forms and their alignments to the original word score ω (t, u) · score η (u, a, w) 2 We use the Wikipedia dump from 2016-05-01.
where θ = {ω, η} is the parameter vector and the normalizing partition function as where T (u) is the set of all parse trees for the string u. This involves a sum over all possible underlying orthographic forms and all parse trees for those forms.
The joint approach has the advantage that it allows both factors to work together to influence the choice of the underlying form u. This is useful as the parser now has access to which words are attested in the language; this helps guide the relatively weak transduction model. On the downside, the partition function Z θ now involves a sum over both all strings in Σ |w|+k and all possible parses of each string! Inference in this joint model is intractable, so we resort to approximate methods.
Finally, we define the marginal distribution where A(u, w) is the set of all monotonic alignments between u and w. This will be our model of morphological segmentation since we are not interested in the latent alignments a.

Learning and Inference
We use stochastic gradient descent to optimize the log-probability of the training data N i=1 log p θ (t (i) , u (i) | w (i) ); this requires the computation of the gradient of the partition function ∇ θ log Z θ , which is intractable. We may view this gradient as an expectation: For any given w, a, u and t, the gradients ∇ ω score ω (t, u) and ∇ η score η (u, a, w) may each be computed in linear time. However, the sum over all underlying forms and trees in eq. (6) is still intractable, so we resort to the importance-sampling estimator derived by   Cotterell et al. (2016). Roughly speaking, we approximate the hard-to-sample-from distribution p θ by taking samples from an easy-to-sample-from proposal distribution q. Specifically, we employ a pipeline model for q consisting of WFST and then a WCFG sampled from consecutively. We then reweight the samples using the unnormalized score from p θ . Importance sampling has found many uses in NLP ranging from language modeling (Bengio et al., 2003) and neural MT (Jean et al., 2015) to parsing (Dyer et al., 2016). Due to a lack of space, we omit the derivation of the importance-sampled approximate gradient.

Decoding
We also decode by importance sampling. Given w, we sample canonical forms u and then run the CKY algorithm to get the highest scoring tree.

Related Work
We believe our attempt to train discriminative grammars for morphology is novel. Nevertheless, other researchers have described parsers for morphology. Most of this work is unsupervised:  applied a Bayesian PCFG to unsupervised morphological segmentation. Similarly, Adaptor Grammars (Johnson et al., 2006), a non-parametric Bayesian generalization of PCFGs, have been applied to the unsupervised version of the task (Botha and Blunsom, 2013;Sirts and Goldwater, 2013). Relatedly, Schmid (2005) performed unsupervised disambiguation of a German morphological analyzer (Schmid et al., 2004) using a PCFG, using the inside-outside algorithm (Baker, 1979). Also, discriminative parsing approaches have been applied to the related problem of Chinese word segmentation (Zhang et al., 2014).

Morphological Treebank
Supervised morphological segmentation has historically been treated as a segmentation problem, devoid of hierarchical structure. A core reason behind this is that-to the best of our knowledgethere are no hierarchically annotated corpora for the task. To remedy this, we provide tree annotations for a subset of the English portion of CELEX (Baayen et al., 1993). We reannotated 7454 English types with a full constituency parse. 3 The resource will be freely available for future research.

Annotation Guidelines
The annotation of the morphology treebank was guided by three core principles. The first principle concerns productivity: we exclusively annotate productive morphology. In the context of morphology, productivity refers to the degree that native speakers actively employ the affix to create new words (Aronoff, 1976). We believe that for NLP applications, we should focus on productive affixation. Indeed, this sets our corpus apart from many existing morphologically annotated corpora such as CELEX. For example, CELEX contains warmth →warm+th, but th is not a productive suffix and cannot be used to create new words. Thus, we do not want to analyze hearth →hear+th or, in general, allow wug →wug+th. Second, we annotate for semantic coherence. When there are several candidate parses, we choose the one that is best compatible with the compositional semantics of the derived form.
Interestingly, multiple trees can be considered valid depending on the linguistic tier of interest. Consider the word unhappier. From a semantic perspective, we have the parse [[un [happy]] er] which gives us the correct meaning "not happy to a greater degree." However, since the suffix er only attaches to mono-and bisyllabic words, we get [un [[happy] er]] from a phonological perspective. In the linguistics literature, this problem is known as the bracketing paradox (Pesetsky, 1985;Embick, 2015). We annotate exclusively at the syntactic-semantic tier.
Thirdly, in the context of derivational morphology, we force spans to be words themselves.
Since derivational morphology-by definition-forms new words from existing words (Lieber andŠtekauer, 2014), it follows that each span rooted with WORD or ROOT in the correct parse corresponds to a word in the lexicon. For example, consider unlickable.

Experiments
We run a simple experiment to show the empirical utility of parsing words-we compare a WCFG-based canonical segmenter with the semi-Markov segmenter introduced in Cotterell et al. (2016). We divide the corpus into 10 distinct train/dev/test splits with 5454 words for train and 1000 for each of dev and test. We report three evaluation metrics: full form accuracy, morpheme F 1 (Van den Bosch and Daelemans, 1999) and average edit distance to the gold segmentation with boundaries marked by a distinguished symbol. For the WCFG model, we also report constituent F 1typical for sentential constituency parsing-as a baseline for future systems. This F 1 measures how well we predict the whole tree (not just a segmentation). For all models, we use L 2 regularization and run 100 epochs of ADAGRAD (Duchi et al., 2011) with early stopping. We tune the regularization coefficient by grid search considering λ ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5}. Table 2 shows the results. The hierarchical WCFG model outperforms the flat semi-Markov model on all metrics on the segmentation task. This shows that modeling structure among the morphemes, indeed, does help segmentation. The largest improvements are found under the morpheme F 1 metric (≈ 6.5 points). In contrast, accuracy improves by < 1%. Edit distance is in between with an improvement of 0.2 characters. Accuracy, in general, is an all or nothing metric since it requires getting every canonical segment correct.

Results and Discussion
Morpheme F 1 , on the other hand, gives us partial credit. Thus, what this shows us is that the WCFG gets a lot more of the morphemes in the held-out set correct, even if it only gets a few more complete forms correct. We provide additional results evaluating the entire tree with constituency F 1 as a future baseline.

Conclusion
We presented a discriminative CFG-based model for canonical morphological segmentation and showed empirical improvements on its ability to segment words under three metrics.
We argue that our hierarchical approach to modeling morphemes is more often appropriate than the traditional flat segmentation. Additionally, we have annotated 7454 words with a morphological constituency parse. The corpus is available online at http://ryancotterell.github.io/data/morphological-tree to allow for exact comparison and to spark future research.