Tense Manages to Predict Implicative Behavior in Verbs

Implicative verbs (e.g. manage) entail their complement clauses, while non-implicative verbs (e.g. want) do not. For example, while managing to solve the problem entails solving the problem, no such inference follows from wanting to solve the problem. Differentiating between implicative and non-implicative verbs is therefore an essential component of natural language understanding, relevant to applications such as textual entailment and summarization. We present a simple method for predicting implicativeness which exploits known constraints on the tense of implicative verbs and their complements. We show that this yields an effective, data-driven way of capturing this nuanced property in verbs.


Overview
Understanding language requires the ability to perform basic inference-to make conclusions about what is likely true or false based on what is said. For example, given the sentence She fixed the bug, we should almost certainly infer that the bug is fixed. However, rather than stating plainly that She fixed the bug, one might instead say: (1a) She managed to fix the bug before midnight.
(1b) She happened to fix the bug while refactoring.
In either case, the hearer should still infer that the bug is fixed. But it is not as easy as always inferring that embedded clauses are true. By changing only one word, these sentence no longer give a clear indication as to whether or not the bug has been fixed: (2a) She wanted to fix the bug before midnight.
(2b) She planned to fix the bug while refactoring.
Implicative verbs, like those in (1), give rise to entailments, while non-implicative verbs, like those in (2), do not. It is therefore vital to natural language understanding to differentiate between clauses that are embedded under implicatives, which we can often infer to be either true or false, and those which are embedded under non-implicatives, for which such inferences cannot be made. In this paper, we exploit a known linguistic property of implicative verbs-that their complement clause is constrained to be in the same tense as the main clause-in order to predict the tendency of verbs to behave implicatively. We show that our method almost perfectly separates non-implicatives from implicatives in a small hand-labeled dataset, and that it provides strong signal for predicting entailments in sentences involving implicative verbs.

Implicative Verbs
Some English verbs can take infinitival complements, meaning they can appear in constructions of the form VB * 1 to VB 2 , where VB * 1 is the "main" verb (which can be conjugated 1 ) and VB 2 is the "complement" verb (which is in infinitive form). Examples (1a)-(2b) illustrate verbs taking infinitive complements.
Implicative verbs are a special subclass 2 of such verbs which give rise to entailments involving their 1 Here, * indicates that VB1 can match any verb form, e.g. VB, VBD, VBP, etc. VB2 can only match the base form VB. 2 We note that factive verbs represent another special class of verbs which can take infinitival complements. Unlike implica-Is the main verb Is the complement Example negated? entailed? Implicative − Yes I managed to solve the problem. ⇒ I solved the problem. Implicative + No I did not manage to solve the problem. ⇒ I did not solve the problem. Implicative − No I failed to solve the problem. ⇒ I did not solve the problem. Implicative + Yes I did not fail to solve the problem. ⇒ I solved the problem. Non-Impl.
− Unknown I wanted to solve the problem. ⇒ I solved the problem. Non-Impl. + Unknown I did not want to solve the problem. ⇒ I did not solve the problem. Table 1: Implicative verbs give rise to entailments involving their complement clauses. Non-implicatives entail neither the truth nor the falsity of their complements, and thus the truth/falsity of the complement is unaffected by negation of the main clause. complement clauses. Individual implicatives can differ in the entailments they generate: e.g. while manage entails the truth of its complement, fail entails the falsity of its complement (failed to solve the problem ⇒ didn't solve the problem). Despite these differences, however, implicatives represent a coherent class of verbs in that they permit some inference to be made about their complements, and this inference is sensitive to the context (positive/negated) of the main clause. This contrasts with non-implicative verbs, like want, which do not permit any inference regarding their complements, and for which the truth of the complement is unaffected by negation in the main clause (Table 1). The method described in this paper aims to separate implicatives from non-implicatives (manage vs. want), rather than to differentiate between types implicatives (manage vs. fail). Making this implicative/non-implicative distinction is a necessary first step toward handling inferences involving embedded clauses, and one that, to date, has only been performed using manually-constructed word lists (MacCartney, 2009;Recasens et al., 2013).

Tense Constraints on Complement Clauses
Karttunen (1971) observed that, in sentences involving implicatives, the tense of the main verb must necessarily match the tense of the complement clause. For example, (3), in which the main clause and the complement are both in the past tense, is acceptable but (4), in which the complement is in the future, is clearly not. For non-implicatives, however, tives, factives presuppose, rather than entail, their complements. E.g. both I was/was not glad to solve the problem entail I solved the problem. We do not address factives here, as factives rarely take infinitival complements: more often, they take "that" complements (e.g. know that, realize that). Factives that do take infinitival complements are mostly phrasal (e.g. be glad to). no such constraint exists: (6) is perfectly felicitous.
(3) I managed to solve the problem last night.
(4) #I managed to solve the problem tomorrow.
(5) I planned to solve the problem last night.
(6) I planned to solve the problem tomorrow.
We exploit this property to predict implicativenesswhether the truth of a verb's complement can be inferred-by observing the verb's usage in practice.

Method
We hypothesize that, given a large corpus, we should be able to distinguish implicative verbs from nonimplicative verbs by observing how often the main verb tense agrees/disagrees with the tense of the complement clause. Unfortunately, verbs in infinitival complement clauses are not conjugated, and so are not necessarily marked for tense. We therefore use the Stanford Temporal Tagger (TT) (Chang and Manning, 2012) in order to identify time-referring expressions (e.g. tomorrow or last night) and resolve them to either past, present, or future tense.
We find all sentences containing VB * 1 to VB 2 constructions in the Annotated Gigaword corpus (Napoles et al., 2012). We run the the TT over all of the sentences in order to identify time-referring expressions. We only consider sentences in which a time-referring expression appears and is in a direct dependency relationship with the complement verb (VB 2 ). We provide the TT with the document publication dates, 3 which are used to resolve each time mention to a specific calendar date and time. We then map these time expressions coarsely to either past, present, or future tense by comparing the resolved time to the document creation time. Because of the fact that days were often not resolved correctly, or at all, we eventually throw away sentences in which the complement clause is labeled as present tense, as these are rarely true references to the present, and rather the result of incorrect time resolution, or implicit future references (e.g. I am going to solve the problem today implies the future as in later today, but this is not captured by the TT). We also assign the main clause to past, present, or future tense by using the fine-grained POS tag and a set of heuristics (for example, to check for modals). 4 We assign a tense agreement score to each verb v as follows. Let S be the set of all VB * 1 to VB 2 constructions in which VB * 1 = v. Then tense agreement is simply 1 |S| × |{s ∈ S | complement tense = main tense}|, i.e. the fraction of constructions in S in which the tenses of the main and complement clauses agree. We expect implicative verbs to occur mostly in agreeing constructions, and thus have high tense agreement, while nonimplicatives may occur in both agreeing and nonagreeing constructions, and thus should have lower tense agreement. Note that while in theory, implicatives should never appear in non-agreeing constructions, the time annotation process is very imprecise, and thus we do not expect perfect results.

Evaluation
Recreating list from Karttunen (1971). Karttunen (1971) provides a short illustrative list of 7 known implicatives 5 and 8 non-implicatives (shown in Table 2). As a first evaluation, we test whether tense agreement can accurately separate the verbs in this list, such that the implicatives are assigned higher agreement scores than the non-implicatives. Table 2 shows that this is indeed the case. Tense agreement almost perfectly divides the list, with implicative verbs appearing above non-implicative verbs in all cases. The one exception is decide (reportedly non-implicative), which appears above dare (reportedly implicative). This error, however, seems understandable: while decide is not strictly implicative in the way manage is, it is often used as an implicative. E.g. the sentence I decided to leave would likely be taken to mean I left.
venture to 1.00 try to 0.42 forget to 0.80 agree to 0.34 manage to 0.79 promise to 0.22 bother to 0.61 want to 0.14 happen to 0.59 intend to 0.12 get to 0.52 plan to 0.10 decide to 0.45 hope to 0.03 dare to 0.44 and non-implicatives listed in Karttunen (1971). Ranking by tense agreement almost perfectly divides the two classes.
Predicting Entailment. Our interest is not in distinguishing implicatives from non-implicatives for its own sake, but rather to predict, based on the main verb, whether the truth of the complement can be inferred. We therefore conduct a second evaluation to assess how well tense agreement predicts this entailment property. We design our evaluation following the recognizing textual entailment (RTE) task (Dagan et al., 2006), in which two sentences are given, a premise p and a hypothesis h, and the goal is to determine whether p reasonably entails h. To construct our p/h pairs, we take all the verbs extracted in Section 3 which appear in at least 50 tense-labeled sentences. For each of these verbs, we choose 3 random sentences in which the verb appears as VB * 1 in a VB * 1 to VB 2 construction. 6 From each sentence, we extract the complement clause by deleting VB * 1 to from the sentence, and conjugating VB 2 to match the tense of VB * 1 . We then use the original sentence as p and the extracted complement as h: e.g. a p/h pair might look like I get to interact with fellow professors/I interact with fellow professors. We ask 5 independent annotators on Amazon Mechanical Turk to read each p and then determine whether h is true, false, or unclear given p. 7 We take the majority answer as the true label. We expect that implicative verbs should lead to judgements which are decidedly true or false while non-implicatives should lead (0.14) UFJ wants to merge with Mitsubishi, a combination that'd surpass Citigroup as the world's biggest bank.
⇒ The merger of Japanese Banks creates the world's biggest bank. (0.55) After graduating, Gallager chose to accept a full scholarship to play football for Temple University.
⇒ Gallager attended Temple University. (0.68) Wilkins was allowed to leave in 1987 to join French outfit Paris Saint-Germain.
⇒ Wilkins departed Milan in 1987.  (Giampiccolo et al., 2007) which require recognizing implicative behavior, even in verbs that are not implicative by definition. The tendency of certain verbs (e.g. be allowed) to behave as de facto implicatives is captured surprisingly well by the tense agreement score (shown in parentheses).
to mostly judgements of unclear. Figure 1 shows that these expectations hold. When a verb with low tense agreement appeared as the main verb of a sentence, the truth of the complement could only be inferred 30% of the time. When a verb with high tense agreement appeared as the main verb, the truth of the complement could be inferred 91% of the time. This difference is significant at p < 0.01. That is, tense agreement provides a strong signal for identifying non-implicative verbs, and thus can help systems avoid false-positive entailment judgements, e.g. incorrectly inferring that wanting to merge ⇒ merging (Table 3). Interestingly, tense agreement accurately models verbs that are not implicative by definition, but which nonetheless tend to behave implicatively in practice. For example, our method finds high tense agreement for choose to and be allowed to, which are often used to communicate, albeit indirectly, that their complements did in fact happen. To convince ourselves that treating such verbs as implicatives makes sense in practice, we manually look through the RTE3 dataset (Giampiccolo et al., 2007) for examples containing high-scoring verbs according to our method. Table 3 shows some example inferences that hinge precisely on recognizing these types of de facto implicatives.

Discussion and Related Work
Language understanding tasks such as RTE (Clark et al., 2007;MacCartney, 2009) and bias detection (Recasens et al., 2013) have been shown to require knowledge of implicative verbs, but such knowledge has previously come from manually-built word lists rather than from data. Nairn et al. (2006) and Martin et al. (2009) describe automatic systems to handle implicatives, but require hand-crafted rules for each unique verb that is handled. The tense agreement method we present offers a starting point for acquiring such rules from data, and is well-suited for incorporating into statistical systems. The clear next step is to explore similar data-driven means for learning the specific behaviors of individual implicative verbs, which has been well-studied from a theoretical perspective (Karttunen, 1971;Nairn et al., 2006;Amaral et al., 2012;Karttunen, 2012). Another interesting extension concerns the role of tense in word representations. While currently, tense is rarely built directly into distributional representations of words (Mikolov et al., 2013;Pennington et al., 2014), our results suggest it may offer important insights into the semantics of individual words. We leave this question as a direction for future work.

Conclusion
Differentiating between implicative and nonimplicative verbs is important for discriminating inferences that can and cannot be made in natural language. We have presented a data-driven method that captures the implicative tendencies of verbs by exploiting the tense relationship between the verb and its complement clauses. This method effectively separates known implicatives from known non-implicatives, and, more importantly, provides good predictive signal in an entailment recognition task.