Japanese Word Reordering Executed Concurrently with Dependency Parsing and Its Evaluation

This paper proposes a method for re-ordering words in a Japanese sentence based on concurrent execution with dependency parsing so that the sentence becomes more readable. Our contributions are summarized as follows: (1) we extend a probablistic model used in the previous work which concurrently performs word reordering and dependency parsing; (2) we conducted an evaluation experiment using our semi-automatically constructed evaluation data so that sentences in the data are more likely to be spontaneously written by natives than the automatically constructed evaluation data in the previous work.


Introduction
Although Japanese has relatively free word order, Japanese word order is not completely arbitrary and has some sort of preference. Since such preference is incompletely understood, even Japanese natives often write Japanese sentences which are grammatically well-formed but not easy to read. For example, in Figure 1, the word order of S1 is less readable than that of S2 because the distance between the bunsetsu "Suzuki-san-ga (Mr. Suzuki)" and its modified bunsetsu "toi-te-shimatta (solved)" is large and thus the loads on working memory become large (Nihongo Kijutsu Bunpo Kenkyukai, 2009;Uchimoto et al., 2000) There have been some conventional researches for reordering words in a sentence so that the sentence becomes easier to read (Belz et al., 2011;Filippova and Strube, 2007;Harbusch et al., 2006;Kruijff et al., 2001;Ringger et al., 2004;Shaw and Hatzivassiloglou, 1999;Uchimoto et al., 2000;Yokobayashi et al., 2004). Most of the conventional researches used syntactic information by assuming that an input sentence for word reordering^Ă Note: A box and an arrow express a bunsetsu 1 and a dependency relation, respectively. Both the sentences S1 and S2 have the same meaning which is translated as "Mr. Suzuki instantly solved the problem that Mr. Sato could not possibly solve." in English. The difference between S1 and S2 is just in their word orders in Japanese. Figure 1: Example of less-readable/readable word order has been already parsed. There is a problem that the errors of dependency parsing increase when an input sentence is less-readable, and the parsing errors cause negative effects on word reordering. To solve the problem, we previously proposed a method for concurrently performing word reordering and dependency parsing and confirmed the effectiveness of their proposed method using evaluation data created by randomly changing the word order in newspaper article sentences (Yoshida et al., 2014). However, since some of the just automatically created sentences are unlikely to be spontaneously written by a native, the evaluation is thought to be not enough. In addition, the probablistic model has room for improvement in targeting at sentences which a native is likely to spontaneously write.
This paper proposes a new method on Japanese word reordering based on concurrent execution with dependency parsing by extending the probablistic model proposed by Yoshida et al. (2014), and describes an evaluation experiment using our 1 Bunsetsu is a linguistic unit in Japanese that roughly corresponds to a basic phrase in English. A bunsetsu consists of one independent word and zero or more ancillary words. A dependency relation in Japanese is a modification relation in which a modifier bunsetsu depends on a modified bunsetsu. That is, the modifier bunsetsu and the modified bunsetsu work as modifier and modifyee, respectively. evaluation data semi-automatically constructed by adding human judgement after automatically changing word order in newspaper article sentences. The experimental results showed the effectiveness of our method.

Word Order and Dependency
In this section, we discuss the relation between word order and dependency in a Japanese sentence using the example shown in Figure 1.
On the ground that dependency is one of fundamental contributing factors which decide the appropriate word order (Nihongo Kijutsu Bunpo Kenkyukai, 2009), the conventional method (Uchimoto et al., 2000) reordered words using syntactic information obtained by dependency parsing which was assumed to be beforehand performed. However, the accuracy of dependency parsing decreases when an input sentence has lessreadable word order such as S1 because dependency parsers are usually trained on syntactically annotated corpora in which sentences have the readable word order such as S2.
On the other hand, if word reordering is performed before dependency parsing, the accuracy of the word reordering is thought to decrease because syntactic information can not be utilized. In fact, to change the word order in S1 to the appropriate one such as S2, it is necessary to comprehend the dependency structure of S1.
The above discussion indicates that word reordering and dependency parsing depend on each other. Therefore, we can consider it is more desirable to concurrently perform the two processes than to sequentially perform them.

Word Reordering Method
In our method, a sentence, on which morphological analysis and bunsetsu segmentation have been performed, is considered as the input. We assume that the input sentence might have word order which is not easy to read but grammatically well-formed. Our method identifies the suitable word order which is easy to read by being executed concurrently with dependency parsing.
We realize the concurrent execution of dependency parsing and word reordering by searching for the maximum-likelihood pattern of word order and dependency structure for an input sentence. We use the same search algorithm as one proposed by Yoshida et al. (2014), which can effi-ciently find the approximate solution from a huge number of candidates of the pattern by extending CYK algorithm used in conventional dependency parsing. In this paper, we refine the probabilistic model proposed by Yoshida et al. (2014) to improve the accuracy. Note our method reorders bunsetsus in a sentence without paraphrasing and does not reorder morphemes within a bunsetsu. In addition, we assume there are not any inverted structures and commas in an input sentence.

Probabilistic Model for Word Reordering
When a sequence of bunsetsus in an input sentence B = b 1 · · ·b n is provided, our method identifies the structure S which maximizes P (S|B). The structure S is defined as a tuple S = ⟨O, D⟩ where O = {o 1,2 , o 1,3 ,· · ·, o 1,n ,· · ·, o i,j ,· · ·, o n−2,n−1 , o n−2,n , o n−1,n } is the word order pattern after reordering and D = {d 1 , · · · , d n−1 } is dependency structure. Here, o i,j (1 ≤ i < j ≤ n) expresses the order between b i and b j after reordering. o i,j is 1 if b i is located before b j , and is 0 otherwise. In addition, d i expresses the dependency relation whose modifier bunsetsu is b i .
In the probablistic model proposed by Yoshida et al. (2014), P (S|B) was calculated as follows: We extend the above model and calculate P (S|B) as follows: where α is a weight and 0 ≤ α ≤ 1. Formula (2) is obtained for the weighted geometric average 2 between the following two Formulas (3) and (4).
Here, Formulas (3) and (4)   represent the inverse flow. According to the probability theory, the calculated result of Formula (2) is equal to those of Formulas (3) and (4). However, in practice, since each factor in the formulas is estimated based on a training corpus, the results of these formulas are different from each other. Figure 2 shows a conceptual diagram which represents relations among Fomulas (2) -(4). If an input sentence has low adequacy of word order, it is thought that performing word reordering before dependency parsing enables S = ⟨O, D⟩ to be identified with higher accuracy, and thus, we can conceive an idea of calculating P (O, D|B) by Fomula (3). Conversely, if an input sentence has high adequacy of word order, it is probably better to perform word reordering after dependency parsing, and thus, we can think of calculating P (O, D|B) by Fomula (4). Therefore, we mix Formulas (3) and (4) by adjusting the weight α depending on the adequacy of word order in an input sentence, instead of using the constant 0.5 in the previous model proposed by Yoshida et al. (2014). (2) is estimated by the maximum entropy method in the same approximation procedure as that of Yoshida et al. (2014).

Experiment
To evaluate the effectiveness of our method, we applied our method to less-readable sentences artificially created by changing the word order of Japanese newspaper article sentences, and evaluated how much our method could reproduce the word order of the original sentences.

Construction of Evaluation Data
From a viewpoint of utilizing our method for support revision, it is desirable to use less-readable sentences spontaneously written by Japanese natives in the experiment. However, it is not easy to collect a large amount of pairs composed of such a sentence and the corresponding sentence which was modified by hand so that the word order becomes readable, and also, such data is unavailable. In addition, since spontaneously written sentences have many factors other than word order which decrease the readability, it is difficult to conduct the evaluation with a focus solely on word order. Therefore, our previous work (Yoshida et al., 2014) artificially generated sentences which were not easy to read, by just automatically changing the word order of newspaper article sentences in Kyoto Text Corpus 3 based on the dependency structure. However, just automatically changing the word order may create sentences which are unlikely to be written by a native. To solve the problem, we semi-automatically constructed the evaluation data by adding human judgement. That is, if a subject judges that a sentence generated by automatically changing the word order in the same way as the previous work (Yoshida et al., 2014) may have spontaneously written by a native. Our constructed data has 552 sentences including 4,906 bunsetsus.

Outline of Experiment
Since our method needs to decide the weight α in Formula (2) in advance, we conducted 5-fold cross validation using the evaluation data constructed in Section 4.1. Concretely, we divided 552 sentences into 5 sets, and then, we repeated an experiment 5 times, in which we used one set from among 5 sets as the test data and the others as the held-out data to decide α. As the training data to estimate each probability in Formula (2), we used 7,976 sentences in Kyoto Text Corpus, which were different from the 552 sentences. Here, we used the Maximum Entropy Modeling Toolkit for Python and C++ 4 with the default options except "-i (iteration) 1000." In the evaluation of word reordering, we obtained the complete agreement (the percentage of the sentences in which all words' order completely agrees with that of the original sentence) and pair agreement (the percentage of the pairs of bunsetsus whose word order agrees with that in the original sentence), which are defined by Uchimoto et al. (2000). Here, when deciding α using the held-out data, we calculate the α to two places of decimals which maximizes the pair agreement.
In the evaluation of dependency parsing, we obtained the dependency accuracy (the percentage of correctly analyzed dependencies out of all dependencies) and sentency accuracy (the percentage of the sentences in which all the dependencies are analyzed correctly), which were defined by Uchimoto et al. (1999). We compared our method to Yoshida's method (Yoshida et al., 2014) and two conventional sequential methods. Both the sequential methods execute the dependency parsing primarily, and then, perform the word reordering by using the conventional word reordering method (Uchimoto et al., 1999). The difference between the two is the method of dependency parsing. The sequential methods 1 and 2 use the dependency parsing method proposed by Uchimoto et al. (2000) and the dependency parsing tool CaboCha 5 , respectively. All of the methods used the same training features as those described in Yoshida et al. (2014). Table 1 shows the experimental results on word reordering of each method. Here, the last row shows the agreements measured by comparing the input word order with the correct word order. The agreements mean the values which can be achieved with no reordering. The both agreements of our method are micro averages for the agreements of each of the 5 sets. As the result of decision of α by using the held-out data, the α for 3 sets was 0.66, and the α for the other two sets was 0.75. The both agreements of our method were highest among all. We can confirm the effectiveness of our method. Although the purpose of our method is reordering to improve readability, our method generates a dependency structure as a by-product. Here, for reference, we show the experimental results on dependency parsing in Table 2. The dependency accuracy of our method was significantly lower than that of the two sequential methods, and was higher than that of Yoshida's method although there was no significant difference. On the other hand, the sentence accuracy of our method was highest among all the methods although there were no significant differences in them. As a result of analysis, especially, our method and Yoshida's method tended to improve the sentence accuracy very well in case of short sentences. On the other hand, CaboCha, which is a dependency parser in sequential 2, tended not to depend very well on the length of sentences.

Conclusion
This paper proposed the method for reordering bunsetsus in a Japanese sentence based on executing concurrently with dependency parsing. Especially, we extended the probablistic model proposed by Yoshida et al. (2014) to deal with sentences spontaneously written by a native. In addition, we conducted the experiment using our semiautomatically constructed evaluation data so that the sentences are likely to be spontaneously written by a native. The experimental results showed the effectiveness of our method.
In the future, we would like to develop a word reordering method which can take account of comma positions by integrating our method with a method for identifying proper comma positions (for example, Murata et al., 2010).