Conversation Trees: A Grammar Model for Topic Structure in Forums

Online forum discussions proceed differ-ently from face-to-face conversations and any single thread on an online forum contains posts on different subtopics. This work aims to characterize the content of a forum thread as a conversation tree of topics. We present models that jointly perform two tasks: segment a thread into sub-parts, and assign a topic to each part. Our core idea is a deﬁnition of topic structure using probabilistic grammars. By leveraging the ﬂexibility of two grammar formalisms, Context-Free Grammars and Linear Context-Free Rewriting Systems, our models create desirable structures for forum threads: our topic segmentation is hierarchical, links non-adjacent segments on the same topic, and jointly labels the topic during segmentation. We show that our models outperform a number of tree generation baselines.


Introduction
Online forums are commonplace today and used for various purposes: product support and troubleshooting, opining about events and people, and student interaction on online course platforms. Threads in these forums become long, involve posts from multiple users, and the chronological order of the posts in a thread does not represent a continuous flow of dialog. Adding structure to these threads is important for tasks such as information extraction, search, and summarization.
One such aspect of structure is topic. Figure 1 shows a computer-troubleshooting related thread with six posts. The first post is the troubleshooting question and the remaining posts can be seen as focusing on either of two topics, the driver software (posts p 1 , p 2 , p 5 ) or the speaker hardware p0 Bob: When I play a recorded video on my camera, it looks and sounds fine. On my computer, it plays at a really fast rate and sounds like Alvin and the Chipmunks! p1 Kate: I'd find and install the latest audio driver. p2 Mary: The motherboard supplies the clocks for audio feedback. So update the audio and motherboard drivers. p3 Chris: Another fine mess in audio is volume and speaker settings. You checked these? p4 Jane: Yes, under speaker settings, look for hardware acceleration. Turning it off worked for me. p5 Matt: Audio drivers are at this link. Rather than just audio drivers, I would also just do all drivers. (p 3 , p 4 ). By categorizing posts into such topics, we can provide a useful division of content in a thread and even across multiple threads. Note that the driver topic is not a contiguous sequence but present in non-adjacent parts, (p 1 , p 2 ) and (p 5 ).
We tackle the problem of joint topic segmentation and topic labeling of forum threads. Given a thread's posts in chronological order (the order in which they were posted), we create a phrase structure tree indicating how the posts are grouped hierarchically into subtopics and super-topics. In these conversation trees, leaves span entire posts. Each non-terminal identifies the topic characterizing the posts in its span. Topics are concepts or themes which summarize the content of a group of posts. Specifically, a topic is a set of words which frequently co-occur in posts which are similar in content and other conversation regularities.
Our key insight in this work is to formalize topic structure using probabilistic grammars. We define a base grammar for topic structure of forum threads and refine it to represent finer topics and subtrees. We learn to predict trees under our grammar based on two formalisms: Probabilistic Context-Free Grammars (PCFG) and Probabilistic Linear Context-Free Rewriting Systems (PLCFRS). In the PCFG model, a non-terminal spans a contiguous sequence of posts. In the PLCFRS model, non-terminals are allowed to span discontinuous segments of posts. We leverage algorithms from probabilistic parsing of natural language sentences and modify them for our domain. We show that our model performs well and sidesteps a number of limitations of prior topic segmentation approaches. In particular: • Our models perform joint topic segmentation and topic labeling while most existing models identify unlabeled segments. Labeling topics on segments creates richer annotation, and links nonadjacent segments on the same topic.
• Our grammar-based probabilistic models have two key benefits. They naturally create tree structures which are considered linguistically suitable for topic segmentation but were difficult to create under previous approaches. Second, the flexibility of grammars such as PLCFRS allow our models to seamlessly learn to produce trees where nonadjacent segments on the same topic are explicitly linked, an issue that was not addressed before.
We present large-scale experiments on a collection of forum threads from the computertroubleshooting domain. 1 We show that our grammar models achieve a good balance between identifying when posts should be in the same topic versus a different topic. These grammar models outperform other tree generation baselines by a significant margin especially on short threads.

Related work
The ideas in this paper are related to three areas of prior research. Forum thread analysis. Finding structure in forum threads has been previously addressed in two ways. The first is reply structure prediction where a parent post is linked to its children (replies) which were posted later in time Cong et al., 2008). Reply links are sometimes augmented with a dialog act label indicating whether the child post is a question, answer, or confirmation to the parent post (Kim et al., 2010;Wang et al., 2011). The second set of methods partition sentences in emails or blog comments into topical clusters and then show salient words per cluster as topic tags (Joty et al., 2013).
We focus on producing rich hierarchical segmentation going beyond clusters which do not contain any cluster-internal structure. We also be-1 Our corpus is available from http:// kinloch.inf.ed.ac.uk/public/CTREES/ ConversationTrees.html lieve that topic structure is complementary to dialog act and reply link annotations. Tasks on forum data such as user expertise (Lui and Baldwin, 2009) and post quality prediction (Agichtein et al., 2008), and automatic summarization (Nenkova and Bagga, 2003) can be carried out on a finegrained level using topic information. Conversation disentanglement. A related problem of clustering utterances is defined specifically for Internet Relay Chat (IRC) and speech conversations. In this case, multiple conversations, on different topics, are mixed in and systems extract threads which separate out individual conversations (Shen et al., 2006;Adams and Martell, 2008;Elsner and Charniak, 2010).
Disentanglement is typically applied for the coarse-level problem of identifying a coherent conversation. In addition, these methods do not create any structure upon the clustered utterances in contrast to the focus of this work. Topic Segmentation. is a task which directly focuses on the topic aspect of text and speech. This task is of greater importance for speech which lacks explicit structure such as paragraphs and sections. Many approaches perform linear segmentation where boundaries are inserted in the text or speech to divide it into a flat set of topic segments (Hearst, 1994;Utiyama and Isahara, 2001;Galley et al., 2003;Malioutov and Barzilay, 2006;Eisenstein and Barzilay, 2008). Very few methods recursively combine smaller segments into larger ones. Such hierarchical models (Eisenstein, 2009) have been applied at a coarse level for segmenting very long texts such as books into sections. Other work has focused on linear segmentation for documents within the same domain and having a regular structure (Chen et al., 2009;Jeong and Titov, 2010;Du et al., 2015). These latter approaches rely on three assumptions: that the documents contain a regular set of topics, that these topics are discussed in a fairly regular consensus order, and that the same topic does not recur in the same document.
Our models address two deficiencies in these approaches. First, text is commonly understood to have a hierarchical structure (Grosz and Sidner, 1986) and our grammar model is an ideal framework for this goal. Tree structures also have other advantages, for example, we do not predefine the number of expected topic segments in a conversation tree, a requirement posed by many prior seg-mentation algorithms. The second limitation of prior studies is assuming that topics do not recur in the same document. But linguistic theories allow for non-adjacent utterances to belong to the same topic segment (Grosz and Sidner, 1986) and this fact is empirically true in chat and forum conversations (Elsner and Charniak, 2010;Wang et al., 2011). Our models can flexibly handle and link recurring topics within and across threads.
As a final note, because of the annotations required, most prior work on forums or IRC chats have typically used few hundred threads. We present a heuristically derived large corpus of topic structure on which we evaluate our models.

Background
Our topic discovery methods are based on two constituency grammar formalisms.

Probabilistic Context-Free Grammars
A PCFG is defined by a 5-tuple G C =(N , T , P , S, D) where N is a set of non-terminal symbols, T a set of terminal symbols, and S is the start symbol. P is a set of production rules of the form A → β where A is a non-terminal and β ∈ {N ∪ T } * . D is a function that associates each production rule with a conditional probability of the form p(A → β|A). This probability indicates how often the non-terminal A expands into β. The probabilities of all the rules conditioned on a particular non-terminal should sum to 1.
The joint probability of a tree T with yield Y , P (T, Y ), is the product of the probabilities of all the productions used to construct T . The parsing problem is to find the treeT which is most likely given the yield Y .T = arg max T P (T |Y ) = arg max T P (T, Y ).
Given training trees, we can enumerate the productions and compute their probabilities using maximum likelihood estimates (MLE): which is the fraction of the times the non-terminal A expands into β.
The most likely parse tree can be found using a number of algorithms. In this work, we use the CYK algorithm for PCFGs in Chomsky Normal Form. This algorithm has complexity O(n 3 ) where n is the length of the yield.
PCFGs do not capture a frequently occurring property of forum threads, discontinuous seg-ments on the same topic. Indirectly however, a PCFG may assign the same non-terminal for each of these segments. To model these discontinuities more directly, we present a second model based on PLCFRS where non-terminals are allowed to span discontinuous yield strings.

Probabilistic Linear Context-Free Rewriting Systems
LCFRS grammars (Vijay-Shanker et al., 1987) generalize CFGs, where non-terminals can span discontinuous constituents. Formally, the span of an LCFRS non-terminal is a tuple, with size k ≥ 1, of strings, where k is the non-terminal "fan-out". As such, the fan-out of a CFG nonterminal is 1.
where N is the set of non-terminals, T the terminals and S is the start symbol. A function f : N → N gives the fan-out of each non-terminal. P is the set of productions or otherwise called rewriting rules of the LCFRS. V is a set of variables used to indicate the spans of each non-terminal in these rules. A rewriting rule has the form: Here A,A 1 ,. . .,A m ∈ N . Since there are m non-terminals on the RHS, this rule has rank m.
A rewriting rule explains how the left-hand side (LHS) non-terminal's span can be composed from the yields of the right-hand side (RHS) non-terminals. For example, in the rule , A and C have fan-out 2, B has fan-out 1. The two spans of A, x 1 x 2 and x 3 , are composed from the spans of B and C. For comparison, the productions of a CFG take the single spans of each non-terminal on the RHS and concatenate them in the same order to yield a single span of the LHS non-terminal.
A Probabilistic LCFRS (PLCFRS) (Levy, 2005) also contains D, a function which assigns conditional probabilities p(A( x) → φ|A( x)) to the rules. The probabilities conditioned on a particular non-terminal and span configuration, A( x) should sum to 1. Given a training corpus, LCFRS rules can be read out and probabilities computed similar to CFG rules.
To find the most likely parse tree, we use the parsing algorithm proposed by Kallmeyer and Maier (2013) for binary PLCFRS. The approach uses weighted deduction rules (Shieber et al., 1995;Nederhof, 2003), which specify how to compute a new item from other existing items. Each item is of the form [A, ρ] where A is a non-terminal and ρ is a vector indicating the spans dominated by A. A weight w is attached to each item which gives the |log| of the Viterbi inside probability of the subtree under that item. A set of goal items specify the form of complete parse trees. By using the Knuth's generalization (Knuth, 1977) of the shortest paths algorithm, the most likely tree can be found without exhaustive parsing as in Viterbi parsing of CFGs. The complexity of parsing is O(n 3k ) where k is the fan-out of the grammar (the maximum fan-out of its rules).

Problem Formulation
Given a thread consisting of a sequence of posts (p 1 , p 2 , . . . , p n ) in chronological order, the task is to produce a constituency tree with yield (p 1 , p 2 . . . p n ). A leaf in this tree spans an entire post. Non-terminals identify the topic of the posts within their span. Non-terminals at higher levels of the tree represent coarser topics in the conversation (the span covered by these nodes contain more posts) than those lower in the tree. The root topic node indicates the overall topic of the thread.
Below we define a Context-Free Grammar (CFG) for such trees.

A Grammar for Conversation Trees
G B is our base grammar which is context-free and has four non-terminals {S, X, T , C}. Each post p in the corpus is a terminal symbol (i.e. a terminal symbol is a bag of words). The productions in G B are: S → T X * , X → T X * and T → p.
G B generates trees with the following structure. A root-level topic S characterizes the content of the entire thread. Thread-starting rules are of the form, S → T X * , where X * indicates a sequence of zero or more X non-terminals. T nodes are pre-terminals analogous to part-of-speech tags in the case of syntactic parsing. In our grammar, the T → p rule generates a post in the thread. In the thread-starting rules, T generates the first post of the thread which poses the query or comment that elicits the rest of the conversation. The X * sequence denotes topic branches, the subtree under each X is assumed to correspond to a dif-  Figure 1: Example conversation tree for the thread in Table 1 ferent topic. These X's characterize all but the first post of the thread. The continuation rules, X → T X * , recursively subdivide the X subtrees into topics spanning fewer posts. In each case, the T node on the right-hand side of these rules generates the first post (in terms of posting time) in that subtree. Therefore posts made earlier in time always dominate (in the tree structure) those which come later in the thread. We define the head of a non-terminal as the first post as per the chronological order of posts in the span of the non-terminal.
This grammar does not generate binary trees. We binarize the tree with C nodes to obtain an equivalent grammar in Chomsky Normal Form (CNF) (CNF yields parsing algorithms with lower complexity) 2 : S → T C | T , X → T C | T , T → p and C → X | X X| X C. The C nodes can be collapsed and its daughters attached to the parent of C to revert back to the non-binary tree.
While this CFG defines the structure of conversation trees, by itself this grammar is insufficient for our task. In particular, it contains a single nonterminal of each type (S, X, T , C) and so does not distinguish between topics. We extend this grammar to create G E which has a set of nonterminals corresponding to each non-terminal in G B , these fine-grained non-terminals correspond to different topics. G E is created using latent annotations (Matsuzaki et al., 2005) on the X, T , S and C non-terminals from G B . The resulting non-terminals for i, j, k and l identify specific topics attached to a particular node type.
Our output trees are created with G E to depict the topic segmentation of the thread and are nonbinary. The binary trees produced by our algorithms are converted by collapsing the C. As a result, conversation trees have S[i], X[j] and T [k] nodes but no C[l] nodes.
An example conversation tree for the thread in Table 1 is shown in Figure 1. At level 1, T [3] describes the topic of the first post while the remaining posts are under X[5] which may indicate a driver topic, and X[12], a speaker hardware topic. Note how X[5] may re-occur in the conversation to accommodate post p 5 on the driver topic.

Supervised learning framework
We use a supervised framework for learning the models. We assume that we have training trees according to the base grammar, G B . The following section describes our data and how we obtain these G B -based trees. In Section 6, we present a method for creating G E -type trees with nonterminal refinements. Estimates of rule probabilities from this augmented training corpus are used to develop the parsers for topic segmentation.

Data
We collected 13,352 computer-troubleshooting related threads from http://forums.cnet. com/. The number of posts per thread varies greatly between 1 and 394, and the average is around 5 posts. We divide these threads into training, development and test sets. The most frequent 100 words from the training set are used as stopwords. After filtering stopwords, a post contains 39 tokens on average and the vocabulary size of our corpus is 81,707. For development and testing, we only keep threads with a minimum of 3 posts (so that the problem is non-trivial) and a maximum of 50 posts (due to complexity of parsing). We have 9,243 training threads, 2,014 for development, and 2,071 for testing.
A particular feature of the forums on cnet. com is the explicit reply structure present in the threads. The forum interface elicits these reply relationships as users develop a thread. When a user replies in a particular thread, she has to choose (only) one of the earlier posts in the thread (including the question post) to attach her reply to. In this way, each post is linked to a unique post earlier in time in the same thread. This reply structure forms a dependency tree. Figure 2 (a) is a possible reply tree for the thread in Table 1.

Deriving conversation trees
Next we convert these reply-link trees into phrasestructure conversation trees. We developed a deterministic conversion method that uses the gen- Figure 2: (a) A reply structure tree for the thread in Table 1 and (b) the derived conversation tree erative process defined by the base grammar G B .
The key idea is to track when the conversation branches into sub-topics and when the replies are proceeding within the same topic. The algorithm traverses the nodes of the dependency tree D in breadth-first order, starting at the root (first) post. We create a root S node in the phrase structure tree H. Then the thread-starting rule from G B , S → T X * , is used to create one T and k X nodes as children of S. The first post p 0 is attached as a child of the T node. k is equal to the number of replies to p 0 (children of p 0 in D). For each of these k X nodes, we instantiate a X → T X * rule in H. The k replies of p 0 are attached one each as a child of the T nodes in these rules. Any set of children are always instantiated in chronological order. So the span of a non-terminal in H always contains posts in nondecreasing time order. We continue the procedure with the next post from D in the traversal order.
This procedure converts the reply tree of Figure  2 (a) into the conversation tree (b). Note that (a) is a possible reply structure for our example thread in Table 1. The conversation tree (b) derived according to this reply structure has a non-projective structure where p 1 , p 2 and p 5 are linked under one X node (at level 1). Such a tree can be produced by our LCFRS model. The ideal PCFG tree will repeat the topic branch as in Figure 1.
The derived trees at this stage follow G B and contain only the S, X, T non-terminals (without any latent annotations). This tree is converted into Chomsky Normal Form using C nodes.

Discontinuous topic segments
As in our example above, non-projective edges in the reply structure are rather frequent. Of the total threads in our corpus 14.5% contain a nonprojective edge. A thread should have a minimum of four posts to have the possibility of nonprojective edges. Among the 7,691 threads with at least four posts, the percentage of non-projective trees is even higher, 25%. This finding suggests that in any thread of reasonable size which we wish to summarize or categorize, non-projective edges will be common. Hence a direct approach for addressing discontinuous segments such as our PLCFRS model is important for this domain.

Parsers for Conversation Trees
The training data are conversation trees with rules from G B . We refine the non-terminals to create G E , extract PCFG or PLCFRS rules from the training trees, and build a CYK parser that predicts the most likely tree according to G E .

Refining the non-terminals
We use a clustering approach, akin to the spectral algorithm of Cohen et al. (2013) and Narayan and Cohen (2015), 3 to create finer grained categories corresponding to G B 's non-terminals: S, X, C and T . Each node in each tree in the training data is associated with a feature vector, which is a function of the tree and the anchor node. These vectors are clustered (for each of the non-terminals separately) and then, each node is annotated with the corresponding cluster. This process gives us the non-terminals S[i], X[j], T [k] and C[l] of G E .
The features for a node n l are: depth of n l in the tree, root is at depth 0; maximum depth of the subtree under n l ; number of siblings of n l ; number of children of n l ; number of posts in the span of n l ; average length (in terms of tokens) of the posts in the span of n l ; average similarity of the span of n l with the span of n l 's siblings 4 ; similarity of n l 's span with the span of its left-most sibling; elapsed time between the first and last posts in n l 's span.
We use CLUTO toolkit (Karypis, 2002) to perform clustering. The algorithm maximizes the pairwise cosine similarity between the feature vectors of nodes within the same cluster. The 3 The main difference between our algorithm and the algorithm by Narayan and Cohen (2015) is that we do not decompose the trees into "inside" trees and "outside" trees, or use a singular value decomposition step before clustering the features. 4 The span of n l and that of a sibling are each represented by binary vectors indicating the presence and absence of a term in the span. The similarity value is computing using cosine overlap between the vectors and the average across all siblings is recorded. best number of clusters for the four non-terminal node types are tuned jointly to give the best performance on our final topic segmentation task.

Learning rule probabilities
As mentioned previously, each terminal in our grammar is an entire post's text. For the preterminal to terminal productions in our grammar ) as the probability under a unigram language model L j which is trained on the collection of the posts from the training corpus which are dominated by are the tokens in post p i . The rest of the production probabilities are learned using MLE on the training trees. In the case of LCFRS rules, the gap information is also obtained during the extraction.

CYK parsing
For both PCFG and LCFRS we use CYK style algorithms, as outlined in §3, to obtain the most likely tree. For the more computationally complex LCFRS model, we make a number of additions to improve speed. First, we restrict the fanout of the grammar to 2, i.e. any non-terminal can only span a maximum of two discontinuous segments. 97% of the productions in fact have only non-terminals with fan-out ≤ 2. Second, we use A * search (Maier et al., 2012) to prioritize our agenda. Last, we reduce the number of items added to the agenda. An item has the form [A, ρ], A is a non-terminal and ρ is the spans covered by A. For every span, we only keep the top 5 non-terminal items according to the score. In addition, we only allow spans with a gap of at most 2 since 77% of all gaps (dominated by fan-out ≤ 2) non-terminals are ≤ 2 posts. Moreover, after a certain number of items (10,000) are added to the chart, we only allow the creation of new items which have a contiguous span.

Systems for comparison
We compare our models to two types of systems.

STRUCTURE ONLY
The first type generate tree structures without considering the content of the threads.
Right-branching tree (RBT). produces a strictly right branching tree where each post is dominated by the immediately previous (according to time) post in the thread. It uses the grammar with the rules {S → T X, X → T X, X → T , T → p}. This method does not perform useful topic segmentation as it produces only a single topic branch containing all the posts.
Attach-to-root tree (ART). attaches each post to the root of the tree. The grammar rules are {S → T X 1 ...X n , X → T , T → p}, where n is the number of posts in the thread. This approach assumes each post belongs to a different topic in the thread. In contrast to RBT, ART contains too many topic branches, one per post in the thread.
Random tree (RAND). mixes decisions to create a new topic branch or continue in the same branch. The generation process is top down, at each step, the algorithm chooses a certain number of topic branches (Xs) to create (≤ number of posts left to add to the tree). Then, the number of posts under each branch is sampled (such that each branch has at least one post). This process is then recursively done at the new topic branches.

STRUCTURE AND CONTENT
These approaches produce tree structures informed by content. We build these parsers by modifying prior models for chat disentanglement and linear topic segmentation of documents.
Similarity tree (SIM). produces trees by attaching each post as a child of the most similar of the previous (by time) posts . We use cosine similarity between vector representations of two posts in order to compute similarity. When the similarity exceeds a threshold value, the post is added under the topic branch of the prior post. Otherwise the post is under a new topic branch attached to the root of the tree. A threshold of 0.15 was chosen after tuning on the development data.
Cluster tree (CLUS). uses an approach related to chat disentanglement (Elsner and Charniak, 2010). The posts within each thread are clustered separately into k l clusters where k l = l/h depends on the number of posts in the thread, l. h = 6 was chosen by tuning. The posts in each cluster are ordered by time and a right branching tree is created over them. These k l cluster-level trees are then attached as children of a new node to create a thread-level tree. The cluster-trees are ordered left to right in the thread-tree according to the time of the earliest post in each cluster.
Linear segmentation tree (LSEG). is based on postprocessing the output of a Bayesian linear topic segmentation model (Eisenstein and Barzi-lay, 2008). Each post's content is treated as a sentence and a document is created for each thread by appending its posts in their time order. The model is then used to group consecutive sentences into k l segments. For each thread of length l, k l = l/h, h = 6 was chosen by tuning. For each segment, a right branching tree is created and these segmentlevel trees are made siblings in a thread-level tree. The segments are added left to right in the threadtree as per their order in the text.
All STRUCTURE trees contain thread structure but no topic labels. In other words, they have coarse non-terminals (X, T and S) only. The STRUCTURE AND CONTENT trees, LSEG and CLUS contain topics or groups but only at one top level, and further the number and labels of these topics are different per thread. Hence there is no linking across threads. Within a thread, the SIM and CLUS tree can link non-adjacent posts under the same topic. These links are also not available from a LSEG tree.

Evaluation metrics
To evaluate the topic segmentation, we develop a node-governance based measure. Our score compares two conversation trees g and h, where g is the gold-standard tree and h is the hypothesized one. We assume that g and h are in dependency format, reversing the transformation from §5.1.
We break g (and h) into a set of pairs, for each pair of nodes in the tree (each node is a post in the thread). For each such pair, p and q, we find their least common ancestor, ℓ(p, q|g) (or ℓ(p, q|h). If these nodes are in a governing relation (p dominates q or vice versa), then ℓ(p, q) is the dominating node. We then define the following sets and quantities for · ∈ {g, h}: s 1 (·) and s 2 (·) are defined as the size of S 1 (·) and S 2 (·), repsectively. Let g 1 , . . . , g n and h 1 , . . . , h n be a corpus of gold-standard conversation trees and their corresponding hypothesized conversation trees. Then the evaluation metric we compute is the harmonic mean (Fscore) of the micro-average of the precision for governing (Gp) and non-governing (NG-p) pairs, and recall for governing (G-r) and non-governing (NG-r) pairs. For example, G-p is calculated as Traditional parsing evaluation measures such as constituency bracketting and dependency attachment scores were too local for our purpose. For example, if a long chain of posts is placed in a different topic but their local dependencies are maintained, we only penalize one constituent and one node's parent in the constituency and dependency scores respectively. But the topic segmentation created by this change has several posts placed in the wrong topic branch. Our scores overcome this problem by considering the relationship between all pairs of posts and also dividing the relationship in the pair as governing or non-governing.

Results and discussion
We tune the number of latent topic annotations for the non-terminals using grid search on the development set. The best settings are 40 S, 100 X, 20 C, 80 T clusters for PCFG and 10 S, 5 X, 15 C, 40 T for LCFRS.
Below we show an example non-projective tree created by our LCFRS parser. The topics are indicated with the most frequent 5 words in the matching cluster. Here post p 4 though later in posting time is predicted to be on the same topic as p 1 .
The non-terminals in our trees enable useful topic segmentation and we found that performance is extremely sensitive to the number of non-terminals of each type S, X, C and T . Currently, we do not have a direct method to evaluate the non-terminals in our tree but we plan to use the information in other applications as an evaluation. Table 2 and 3 shows the segmentation performance of the models (as percentages). The performance varied greatly depending on the length of the threads and hence we show the results separately for threads with up to 15 posts (SHORT) and those with 16 to 50 posts (LONG). The results are divided into sections based on the subset of test data on which the evaluation is performed. The first section (R1.) is performance on all threads, (R2.) only on the projective threads in the test data, and (R3.) only on the non-projective threads.
Among the baselines, the Right-branching trees (RBT) or Attaching to the root (ART) have some advantages: the RBT receives 100% recall of the governing pairs and the ART tree has high recall of the non-governing pairs. However, their Fscores are 0. Recall that the RBT contains a single topic branch and hence no useful segmentation is done; ART is the other extreme where every post is put in a separate topic branch. RAND is the average performance of 3 randomly generated trees for each thread. This method has a better balance between branching and depth leading to 33.4 Fscore for SHORT and 21.5 for LONG threads.
The PCFG and the LCFRS models clearly outperform these baselines. The Fscore improves up to 15% over RAND on SHORT and LONG threads. The grammar models also consistently outperform SIM systems.
With regard to CLUS and LSEG, there is a difference in performance between SHORT and LONG threads and based on whether the desired structure was projective or non-projective. On SHORT threads, the grammar models outperform LSEG and CLUS particularly on the projective threads (the LCFRS model has a 22% higher Fscore). On the longer threads however, the CLUS and LSEG models perform best overall and for non-projective threads. CLUS and LSEG directly model the content similarity of posts while the grammar models make many decisions at level of topic nodes. Remember that the clustering is done per thread in CLUS and LSEG compared to using a common set of topics across all threads. Making such fine-grained similarity comparison appears to be helpful especially for longer threads and even though LSEG does not make non-projective decisions, its accuracy is high on the attachments it makes leading to good performance on nonprojective threads too. In future work, we plan to explore how we can combine the advantages of direct similarity with the grammar models.
Between the two grammar models, the LCFRS model is better than PCFG, even on projective threads, and can produce non-projective trees. Part of this improvement on projective trees could be due to more data being available in the LCFRS model since all the data can be used for training it.  For the PCFG model, only the projective data can be used for training.
Overall, the LCFRS model is powerful on projective threads and SHORT non-projective threads. Compared to PCFG, the LCFRS model has a number of advantages: we can use more data, can predict non-projective trees. Some of the constraints we imposed on the LCFRS parser, such as restricting the gap degree are likely to have limited the ability of the model to generate more flexible nonprojective edges. We believe that as we figure out how to make these parsers faster, we will see even more improvements from the LCFRS models.

Conclusions
This work represents a first approach to learn discourse structure of forum threads within an explicit grammar framework. We show that a coarse  Table 3: Results on threads with > 15 posts grammar for structure can be refined using latent annotations to indicate the finer topic differences. Our trees have good segmentation performance and provide useful summaries of the thread content at the non-terminal nodes. A main goal for future work is to incorporate further domain specific constraints on the models to improve parsing speed and at the same time allow more flexible trees. We also plan to evaluate the usefulness of conversation trees in tasks such as predicting if a thread is resolved, and user expertise.