Recognizing Counterfactual Thinking in Social Media Texts

Counterfactual statements, describing events that did not occur and their consequents, have been studied in areas including problem-solving, affect management, and behavior regulation. People with more counterfactual thinking tend to perceive life events as more personally meaningful. Nevertheless, counterfactuals have not been studied in computational linguistics. We create a counterfactual tweet dataset and explore approaches for detecting counterfactuals using rule-based and supervised statistical approaches. A combined rule-based and statistical approach yielded the best results (F1 = 0.77) outperforming either approach used alone.


Introduction
Counterfactuals describe events that did not occur, and what would have happened (or not happened), had the event occurred (e.g., "If I hadn't broken my arm, I never would have met her."). More precisely, counterfactual conditionals have the form "If it had been the case that A (or not A), it would have been the case that B (or not B)." Counterfactuals have been studied in many different domains. Logicians and philosophers focus on literally logical relations between the antecedent and consequent of counterfactual forms and the outcomes (Goodman, 1947). In contrast, political scientists usually conduct counterfactual thought experiments for hypothetical tests on historical events, policies, or other aspects of a society and assess them (Tetlock, 1996).
Counterfactual thoughts are defined, especially in psychology, as mental representations of alternatives to past events, actions, or states. Their use has been explored for correlations with many different demographics (age, gender) and psychological variables (depression, religiosity) (Kray et al., 2010;Markman and Miller, 2006).
Counterfactual thinking has been linked to perceiving life events as more meaningful, fated, and even as influenced by the divine (Kray et al., 2010;Buffone et al., 2016), as well as with problemsolving, because imagining alternate outcomes can easily bring to mind the steps needed for improvement (Epstude and Roese, 2008;Roese, 1994). It has also been shown to be associated with affect management, particularly when imagining realities that are worse than what actually happened (Epstude and Roese, 2008;Roese, 1994) Despite the extensive research on counterfactual thinking, counterfactual language forms have not been studied in computational linguistics. Language-based models to recognize counterfactual thinking in social media would potentially allow for psychological analysis on users based on their everyday language, avoiding the high expense of capturing counterfactual thinking at a large scale using traditional psychological assessments.
Therefore, in this paper, we build a languagebased model to recognize counterfactual forms in social media texts of Twitter and Facebook. There are many challenges for this task. First, counterfactual statements have a low base rate; we found only 2% of status updates on Facebook and 1% of tweets contain counterfactual statements. Secondly, counterfactual statements can take on many forms in natural language. 1 For example, they may or may not use explicit if-or then-clauses (e.g, consider "If I had not met him then I would be better off" versus "I wish I had not met him").
The low base rate and high variability of natural language counterfactuals in social media texts make them difficult to recognize using simple linguistic or statistical features. We address these challenges by using a combined rule-based and statistical approach. Key to our success is defining seven sub-types of counterfactuals, allowing better coverage of rarer sub-types.

Related Work
Identifying counterfactuals is in many ways similar to identifying discourse relations. In terms of relation classification, the counterfactual conditionals can be viewed as a subset of Condition type of Contingency class in the Penn Discourse Treebank (PDTB) (Prasad et al., 2008) or the Condition relation of Rhetorical Structure Theory (RST) (Mann and Thompson, 1987). Also, like all discourse relations in the PDTB, counterfactuals have implicit and explicit forms, and so cannot by uniquely identified by the presence of specific words.
There have been many researchers who have tried end-to-end discourse relation parsing with the PDTB and RST (Biran and McKeown, 2015;Lin et al., 2014;Ji and Eisenstein, 2014). Many of them used dependency parsing or constituency parsing for argument detection or elementary discourse unit (EDU) segmentation to infer the relation between them. However, the short lengths and poor quality of parses of social media texts make dependency constituents unreliable. 2 For example, posters frequently drop the subject of a sentence.
Other work mostly focuses on relation classification with an assumption that arguments of the given relations are already identified (Park and Cardie, 2012;Pitler et al., 2009). They explore various learning algorithms and types of features in the given arguments of discourse relations. Then, they report which combinations give the best performance of each discourse relation.
Our work, while possible to view as a task in discourse relation classification, focuses on critical features of counterfactuals rather than on accurate demarcation of each argument of the relation. Most downstream applications, such as

Method
We use a combination of a rule-based approach and a supervised classifier to capture counterfactual statements from Twitter.

Data Set
No existing corpus of counterfactual statements was available, so we collected our own data set, starting from a random set of tweets from May 2014 and July 2014. As noted previously, couterfactual statements are rare, so we first limited the random tweet set to 1,637 containing keywords 3 that can signal counterfactuals (Train and Test row from Table 1). Keywords were in part based on prior literature on spontaneous counterfactual generation, such as should have, could have, at least, if only, or next time (Sanna and Turley, 1996). We identified further counterfactual forms (e.g. wish) based on visual inspection of the data. Next we used the overall list of keywords to draw samples of 500 tweets for further visual inspection. Words or phrases which had an unreasonably high false positive rate for containing counterfactuals were eliminated. Well-trained annotators then manually labeled each of the 1,637 tweets for counterfactuals with a 9% postive rate, results in 153 counterfactuals and 1,484 negative samples. A random set of 500 of these instances were used in training and the rest were reserved for testing.
To build out our training set to capture examples of all forms of counterfactuals, we added a train supplement from random tweets from 2012 and 2014 -at least thirty tweets from each of seven counterfactual forms we defined for our statistical model using the regular expressions 4 with brownclusters and the tweet PTB tagging model (described next). With this process, we enabled the model to be less biased towards only the samples with the counterfactual cue phrases used for data collection. Additionally, the model learned syntactically different forms of counterfactuals identified in prior work. To evaluate counterfactual form annotation, inter-annotator agreement was established on 1,637 tweets with a second rater with

Classification
We first use a rule-based model to capture counterfactual patterns from social media texts. We then use a statistical model (Linear SVM) to increase precision by identifying tricky false positives with forms similar to counterfactuals (e.g., "wish you the best").
Rule-based Classification. Our rule-based approach is based on seven forms of counterfactuals (  (Table 2). First, we remove sentences ending in question marks predicted as 'end of sentence' by the tweet part-of-speech (POS) tagger (Gimpel et al., 2011). We then use pattern matching with regular expression using a combination of cue phrases (bold), POS tags, and word clusters. The word clusters, based on a set of Twitter Brown clusters 5 are used to capture the numerous variations of words in social media texts (e.g., 'shuldve' for 'should have'). This approach requires matching both the token and its part-of-speech, since the POS tag of each token is important for counterfactual form.
The rule-based approach is also useful in that it allows us to detect the arguments for counterfactual relations; conditional statement and consequent statement from Conjunctive Normal/ Converse form and Verb Inversion form, one counterfactual statement from Wish Verb and Could / Would / Should have. We customized Biran's demarcation methods using the first verb phrase or the connective as a boundary to capture the more informative argument of the statement: For one argument detection, we demarcate from the cue phrase (e.g., would have) to the end of sentence. For two arguments, we demarcate from condi-tional word (e.g., if, unless) to the end of statement or before the start of the second verb phrase.

Part of Speech Tagging
We use the Penn Treebank (PTB)-style Tweet POS tags 6 instead of Tweet POS tags (Gimpel et al., 2011) as it contains more fine-grained categories and yields higher accuracy of pattern matching. For instance, Tweet POS tags do not differentiate modal verbs, past tense verbs, and other types of verbs, but categorize all of them as 'V'. However, in many forms of counterfactuals, the distinction between modal verbs and past particles from other types of verbs are critical (e.g., in Should / Could / Would Have forms). Finally, we conduct a postprocessing on the Tweet POS parsing results for the more accurate prediction. First, we delete RT tags along with the token since it is not informative for our task. Then, we convert 'USR' to nouns because the word token tagged as 'user' usually plays the role of a common noun from the discourse relation perspective. Additionally, in order to enhance the POS tagging, we use the brown clusters to tag empirical variations of modal verbs as 'MD' and we define 'CCJ', a new tag to distinguish conditional conjunctions (i.e. Brown clusters for 'if') from other types of conjunctions.
Statistical Modeling. Each counterfactual form has a different number of arguments for the relation, and different types of features that cause the most errors. Therefore, we analyze the errors of each form separately and use different approaches expected to ensure the best performance.
If a tweet matches rules for counterfactual forms 1, 2, 3, 4, or 5, it is further classified using a statistical model trained with features of sequential words (n-gram) and POS tags of demarcated arguments and the whole sentence.
A statistical model is expected to capture some implicit relations between arguments as well as lexical and part-of-speech patterns, but may also hurt performance in situations where the rulebased approach achieves high precision. Therefore, we applied statistical approaches to counterfactual forms which cannot be easily differentiated by their superficial patterns. These forms were selected by both theoretical and empirical analysis; we discuss these forms further in our evaluation section.

Evaluation
As discussed, counterfactuals are not easily identified by rules or specific words. Given their low base rate and multiplicity of forms, traditional machine learning approaches trained on a random tweet sample tend to label all tweets as the most frequent class (non-counterfactual). Use of a counterfactual-enriched training set increases the performance, but still gives a low F1 on the imbalanced test set. Thus, in order to make the classifier robust to the imbalanced dataset, we designed a rule-based model with counterfactual forms, which resulted in significantly higher F1 than statistical model. Moreover, the rule-based model captures positive samples of all possible forms which might not exist in the training set. A combined approach gives the best result. As Table 3 shows, our whole pipeline ('CF Parser' in Table 3) obtained the best overall performance with the combination of both approaches.
For Wish Verb form prediction gets a big performance boost from the statistical model because of highly frequent false positives which have counterfactual-like forms such as birthday wishes or new year's day wishes. Among samples classified as Wish Verb form the counterfactual prediction F1 increased from 0.82 to 0.90 after the final prediction by the statistical model. Finally, we conducted an ablation test to analyze how each process of the pipeline affects the overall performance of the classifier ( Table 4). The argument detection was less effective (F1 0.01 drop) than we expected due to the relatively simple and concise structure of tweets in general (Args in Table 4).
Using only n-grams as features for the statistical model without PTB-style Tweet POS tags gives a relatively large drop (0.02) from F1. From the grammatical perspective, n-grams are less informative than POS tags for counterfactuals especially considering that there are so many variations of each word token in social media (e.g., 'clda', 'coulda', and 'couldve' for 'could have').
We examined how the statistical model affected the final performance of each counterfactual form. The model we used for filtering out frequent false positives (e.g., birth day wishes) of Wish Verb form caused 0.03 F1 drop when it is removed. Also, the models trained with twoargument-relation forms (Conjunctive Normal / Converse, Modal Normal, and Verb Inversion) caused 0.04 F1 drop when they are removed from the pipeline, since the classifier cannot use subtle relations between arguments for its counterfactual prediction.

Conclusion
This is the first work to identify counterfactuals in social media, a task we hope more people will address. Our best results came from combining rulebased methods that exploit a theory of the different forms of counterfactual with focused statistical methods for reclassification of challenging forms. Our counterfactual predictor can now be applied to large collections of tweets and Facebook posts from people of known education, religiosity, political orientation, well-being, and other attributes of interest to psychologists and political scientists, allowing further study of their theories of counterfactual use.