Ideological Phrase Indicators for Classification of Political Discourse Framing on Twitter

Politicians carefully word their statements in order to influence how others view an issue, a political strategy called framing. Simultaneously, these frames may also reveal the beliefs or positions on an issue of the politician. Simple language features such as unigrams, bigrams, and trigrams are important indicators for identifying the general frame of a text, for both longer congressional speeches and shorter tweets of politicians. However, tweets may contain multiple unigrams across different frames which limits the effectiveness of this approach. In this paper, we present a joint model which uses both linguistic features of tweets and ideological phrase indicators extracted from a state-of-the-art embedding-based model to predict the general frame of political tweets.


Introduction
Social media platforms have played an increasingly important role in U.S. presidential elections, beginning in 2008. Among these, microblogs such as Twitter have a special role, as they allow politicians to react quickly to events as they unfold and to shape the discussion of current political issues according to their views.
Framing is an important tool used by politicians to bias the discussion towards their stance. Framing contextualizes the discussion by emphasizing specific aspects of the issue, which creates an association between the issue and a specific frame of reference. Research on issue framing in political discourse is rooted in social science research (Entman, 1993;Chong and Druckman, 2007) and recently has attracted growing interest in the natural language processing community (Tsur et al., 2015;Card et al., 2015;Baumer et al., 2015) as a way to automatically analyze political discourse in congressional speeches and political news articles. Contrary to these sources, Twitter requires politicians to compress their ideas and reactions into 140 character long tweets. As a result, politicians have to cleverly choose how to frame controversial issues, as well as react to events and each other (Mejova et al., 2013;Tumasjan et al., 2010).
Framing decisions can be used to build support for political stances and they often reflect ideological differences between politicians. For example, in debates concerning the issue of abortion, the stance opposing abortion is framed as "pro-life", which reflects a moral or religious-based ideology. Correctly identifying how issues are framed can help reveal the ideological base of the speaker. However, in many cases framing abstracts this information and groups content reflecting differing ideologies together under the same frame. As a concrete example consider the following tweets: 1. POTUS exec. order on guns is a gross overreach of power that tramples on the rights of law abiding Americans and our Constitution 2. With this ruling #SCOTUS has upheld a critical freedom for women to make their own decisions about their bodies In both tweets, the same frame (Legality, Constitutionality, & Jurisdiction) is used to discuss two different issues: guns and abortion, respectively. Despite the use of a similar frame, the two tweets reflect opposing ideologies.
A straight-forward approach for identifying these differences would be to refine the issueindependent general frames into more specific categories. However, this would limit their generalization and considerably increase the difficulty of analysis, both for human annotators and for automated techniques. Instead, we suggest to aug-ment the frame analysis with additional information. Our modeling approach is based on the observation that politicians often use slogans in both their tweets and speeches. These are key phrases used to indirectly indicate the political figures' core beliefs and ideological stances. Identification of these phrases automatically decomposes the frames into more specific categories.
Consider the two tweets in the example above. In the first tweet, several phrases indicate the frame: "exec. order", "overreach of power", "rights of law abiding Americans", "our constitution". In the second tweet, the relevant phrases are "this ruling" and "upheld a critical freedom". All of these phrases indicate that the same frame is being used in both tweets. However, analyzing the specific terminology in each case and the context in which it appears helps capture the ideological similarities and differences. For example, in the context of gun-rights debates, phrases highlighting "law and order" and references to the constitution tend to reflect a conservative ideology, while phrases highlighting upholding of freedoms in the abortion debate tend to reflect a liberal ideology.
Given the rapidly changing nature of trending issues and political discourse on Twitter, our key technical challenge is to relay these ideological dimensions to an automated model, such that it will be able to easily adapt to new issues and language. Our model consists of two components combined together: frame identification and ideological-indicators identification. For the first piece we use a structured probabilistic model to capture general framing dimensions by combining content and political context analysis. For the second task, we employ a state-of-the-art textual similarity model which captures and generalizes over lexical indicators of key phrases that identify the politicians' ideology. More details of both components are described in Section 4.
In this paper we take a first step towards connecting these two dimensions of analysis: issue framing and ideology identification. We lay the foundation for more advanced research by identifying this connection, analyzing tweets authored by U.S congressional representatives, and extracting ideological phrase indicators. We build and analyze a joint model which combines the two dimensions. Our experiments in Section 5 quantitatively compare the differences in frame prediction performance when using ideological phrase indi-cators. We also include a qualitative analysis in Section 6 of several examples in which ideological phrase indicators can help differentiate between tweets with similar frame predictions that reflect different ideologies.
Several previous works have explored framing in public statements, congressional speeches, and news articles (Fulgoni et al., 2016;Tsur et al., 2015;Card et al., 2015;Baumer et al., 2015). Framing is further related to works which analyze biased language (Recasens et al., 2013;Choi et al., 2012;Greene and Resnik, 2009) and subjectivity (Wiebe et al., 2004). Important to the language analysis of our work, Tan et al. (2014) have shown how wording choices can affect message propagation on Twitter. The study of political sentiment analysis (Pla and Hurtado, 2014;Bakliwal et al., 2013), ideology measurement and prediction (Iyyer et al., 2014;Bamman and Smith, 2015;Sim et al., 2013;Djemili et al., 2014), policies (Nguyen et al., 2015, voting patterns (Gerrish and Blei, 2012), and polls based on Twitter political sentiment (Bermingham and Smeaton, 2011;O'Connor et al., 2010;Tumasjan et al., 2010) are also related to the study of framing on Twitter.  Political and social science works have studied the role of Twitter and framing in molding public opinion of events and issues (Burch et al., 2015;Harlow and Johnson, 2011;Meraz and Papacharissi, 2013;Jang and Hart, 2015), as well as sentiment analysis and network agenda modeling of the 2012 U.S. presidential election (Groshek and Al-Rawi, 2013). Boydstun et al. (2014) composed a Policy Frames Codebook for use in labeling general, issue-independent frames of longer texts. These frames were extended for Twitter and studied in a computational setting by Johnson et al. (2017b,a). Our approach builds upon these findings by identifying phrases which are relevant for determining ideology and increasing prediction accuracy of frames.

Data and Problem Setting
Dataset: In this work, we use the Congressional Tweets Dataset of Johnson et al. (2017b,a) which consists of the tweets of members of the 114 th U.S. Congress. These tweets discuss six current political issues: (1) abortion, (2) the Affordable Care Act (i.e., the ACA or Obamacare), (3) gun ownership, (4) immigration, (5) terrorism, and (6) the LGBTQ community. The dataset provides a labeled portion of 2,050 tweets, which are labeled using 17 possible frames. A brief description of each frame is shown in Table 1.
Frame Overlap: Johnson et al. (2017b,a) found that for most tweets, one or two frames were used. Additionally, in many cases, tweets authored by Republican and Democratic politicians use similar frames, both when discussing similar and different issues. For example, consider the following two tweets concerning the shooting of the Emanuel African Methodist Episcopal Church in 2015.
1. Our thoughts and prayers must be with 9 innocent men and women murdered in Charleston, SC. Every effort must be made to capture the killer. RIP 2. My thoughts are with those impacted by the #CharlestonShooting. I pray that the perpetrator is brought to justice soon.
Both tweets frame the shooting using two frames: Frame 6 (Crime & Punishment) and Frame 17 (Personal Sympathy & Support). In Tweet (1) the politician states that the killer must be captured. Similarly, in Tweet (2) the politician hopes for the perpetrator of the crime to be brought to justice. These phrases indicate that Frame 6 is being used. Additionally, in both tweets the politicians express that their thoughts are with those affected by the crime, indicating the use of Frame 17. Despite the use of the same frames by both tweets, there are very subtle differences between the two tweets, indicated by the specific phrase choices. For example, in Tweet (1) the politician uses the phrase "men and women murdered" to specifically reference the victims, while in Tweet (2) the politician uses "those impacted", a more inclusive definition.
Phrase Identification: Using the labeled tweets of the dataset, we extracted lists of short phrases which frequently appear in each frame, for all frames. 1 All of these phrases can be further grouped into a more general phrase, which we term an ideological phrase indicator. For example, sub-phrases such as rates will increase, increasing the rates this year, and premiums skyrocket can be grouped into the more general ideological phrase indicator Increase of Frame 1 (Economic). From our observations, Democrats tend to  use more phrase indicators (with more sub-phrases each) than Republicans for each frame. Finally, while the general phrase indicator name may be similar for both parties, the sub-phrases that are grouped under the general phrase may overlap, but are often different. For example, Frame 12 (Political Factors & Implications) has the general phrase indicator Refers to POTUS for both parties. However, the sub-phrases under this general phrase can differ across the parties, e.g. Republicans use phrases like "Obama admin" or "commander in chief", while Democrats use phrases like "the administration", "the president", or "thank you PO-TUS". Sub-phrases can also be similar across parties, e.g., both parties use "President Obama" in Frame 12. The general ideological phrase indica-tors for each frame are listed in Table 2. 2

PSL Models of Language on Twitter
Weakly Supervised Models with PSL: In order to model the dependencies between politicians and the language of their tweets, we design models with PSL, a declarative modeling language (Bach et al., 2015). PSL allows the user to specify first-order logic rules using domain knowledge. Weights for these rules are learned in either a supervised or unsupervised fashion and each weight indicates the importance of its associated rule. These rules are compiled into a hinge-loss Markov random field which defines a probability distribution over continuous value assignments to random variables of the model. For more details To evaluate if modeling ideological phrase indicators can increase the F 1 score of frame prediction, we use the most indicative features for predicting a tweet's frame (as determined by Johnson et al. (2017b)): unigrams, word similarity to unigrams, bigrams, and trigrams. In addition, we add tweet similarity to phrases (SIMPHRASE(T,P F ) described below) as a feature. These features are extracted using weakly supervised models and represented as the following predicates in PSL notation: UNIGRAM F (T,U), SIMUNIGRAM(T,F), BIGRAM P (T,B), TRIGRAM P (T,TR). Each predicate indicates that the tweet T has that unigram U, a word similar to that unigram, a bigram B, or a trigram TR, respectively. Finally, the party of the politician who authored the tweet (PARTY(T,P)) is also used. These predicates are combined into the probabilistic rules of the PSL model as shown in Table 3.

PSL MODEL RULES UNIGRAMF (T, U) ∧ SIMPHRASE(T,PF ) → FRAME(T, F) UNIGRAMF (T, U) ∧ PARTY(T, P) ∧ SIMPHRASE(T,PF ) → FRAME(T, F) UNIGRAMF (T, U) ∧ SIMUNIGRAM(T, F) ∧ SIMPHRASE(T,PF ) → FRAME(T, F) UNIGRAMF (T, U) ∧ PARTY(T, P) ∧ BIGRAMP (T, B) ∧ SIMPHRASE(T,PF ) → FRAME(T, F) UNIGRAMF (T, U) ∧ PARTY(T, P) ∧ TRIGRAMP (T, TR) ∧ SIMPHRASE(T,PF ) → FRAME(T, F)
Incorporating Phrase Similarity: Due to the dynamic nature of language and trending political issues on Twitter, it is infeasible to construct a list of all possible phrases one can expect politicians to use when framing an issue. Therefore, we use the embedding-based model of Lee et al. (2017) to determine which tweets contain phrases that are similar to our initial list of phrases. For example, given the phrase insurance rates will increase, we want to find all tweets which contain similar phrases, e.g., rising insurance premiums.
The phrase similarity model was trained on the Paraphrase Database (PPDB) (Ganitkevitch et al., 2013) and incorporates a Convolutional Neural Network (CNN) to capture sentence structures. This model generates the embeddings of our phrases and computes the cosine similarities between phrases and tweets as the scores. The input tweets and phrases are represented as the average word embeddings in the input layer, which are then projected into a convolutional layer, a maxpooling layer, and finally two fully-connected lay-ers. The embeddings are thus represented in the final layer. The learning objective of this model is: where X is all the positive input pairs, δ is the margin, g(·) represents the network, λ c and λ w are the weights for L2-regularization, W c is the network parameters, W w is the word embeddings, W init is the initial word embeddings, and t 1 and t 2 are negative examples that are randomly selected.
All tweet-phrase pairs with a cosine similarity over a given threshold are used as input to the PSL model via the predicate SIMPHRASE(T,P F ), which indicates that tweet T contains a phrase that is similar to the phrases for a certain frame (P F ). Table 3 presents examples of the rules used in our modeling procedure.

Experiments
Analysis of Supervised Experiments: Since each tweet can be classified as having more than one frame, the prediction task becomes a multilabel classification task. Therefore, we use the standard measurements for precision and recall of a multilabel task. The F 1 score is the harmonic mean of these two measures. We conducted supervised experiments using five-fold cross validation with randomly chosen splits on the labeled portion of the dataset. Table 4 shows the results of our supervised experiments. The first column lists the frame number. The second column presents the results of the baseline model, which includes all of the rules listed in Table 3 without the SIMPHRASE(T,P F ) predicate. The third FRAME NO.   Johnson et al. (2017b). The phrases column indicates the scores for the best model when combined with our proposed phrases. Items in bold are the highest score. The weighted average is the micro-weighted average of the F 1 scores.
column lists the results of our model which consists of the baseline model with the addition of the SIMPHRASE(T,P F ) predicate. From these results we can see that the joint model that uses both language features (i.e., unigrams, bigrams, and trigrams) and phrase indicators (shown in Table 2) is able to improve performance in 9 out of the 17 frames. The most likely cause for the decrease in score for the other 8 frames is that it is possible that there are too many overlapping sub-phrases within the general phrases of these 8 frames. This would introduce extra noise into the probabilistic model and result in lower scores. The 9 frames which improve have either 1 or no overlapping sub-phrases across parties for each general phrase category. Further refinement of the sub-phrases is left for future work.
Ablation Case Study: To investigate the usefulness of ideological phrase indicators, we conducted an ablation study on the results of Frame 12. Frame 12 is used when a politician references other political entities (e.g., the House, Senate, former presidents, etc.) as well as political actions (e.g., filibusters or lobbying). For our dataset, we used the following general phrases for Frame 12 which include references to: Democrats, Republicans, the President (POTUS), the Supreme Court (SCOTUS), and Congress. We ran our model through an ablation study, in which each pair of phrases is removed one at a time to study their overall effect on the final prediction.  Table 5: F 1 Scores of Ablation Experiments. All Phrases represents our score for Frame 12 when using all possible phrases. The remaining rows indicate which general phrase indicators have been removed from the comprehensive model. Column 2 presents the F 1 score. Column 3 indicates the increase or decrease in score after the respective phrases are removed.
From these initial results, it appears that the way politicians refer to Democrats and Congress are the most important phrase indicators for predicting Frame 12. When these two phrase groups are removed, there is a large decrease in F 1 score. Additionally, removing references to the president has a slight increase, while removing references to Republicans and the Supreme Court has a larger increase. Therefore, references to Republicans and the Supreme Court are likely to be the least useful for predicting this frame. We leave finding the best combinations of phrases for each frame as future work, as described in Section 7.

Qualitative Analysis
The supervised experiments of the previous section allow us to analyze the effects of phrases as features for frame prediction. In this section, we explore the predictions of the phrase-based model to locate framing trends of a real world event. We first learned the weights of each model using the labeled data and then performed MPE inference on the unlabeled tweets to obtain their predicted frames. We used these predictions to analyze the political discourse on Twitter by focusing on tweets concerning the shooting of the Pulse Nightclub in Orlando, Florida (June 12, 2016). Table 6 presents the frame predictions and example tweets for this event.
Frame 17 reflects politicians tweeting that their  "thoughts and prayers" are with the community, as seen in the first line of Table 6. Offers of prayers and sympathy are used by both parties as the initial response the day this (and most other) shootings occur. This can be considered both as a reflection of the politicians' immediate emotional reaction to the shooting but also to support other agendas, as Frame 17 also appears in tweets that use other frames, specifically Frames 9 and 3. Interestingly, Republicans and Democrats use these frames in nuanced ways to promote different agendas, which are identifiable by the presence (or lack thereof) of different key phrases.
Republicans used Frame 3, often in combination with Frame 17, to discuss the shooting as an act of evil or terrorism as well as to suggest links between the shooter and ISIS (examples of these tweets are shown in rows three and four of Table 6). Democrats, however, used Frame 3 to express a sense of responsibility on their part to take actions to prevent gun violence (e.g., row five of Table 6) or refer to the shooting as a hate crime or act of terror (e.g., row six of Table 6). All of these examples are expressed with Frame 3, however, the different phrases indicate differing underlying ideologies. For example, referring to the shooting as an "act of evil" indicates a religiousbased ideology, which also limits possible ways to combat the problem. However, by associating the cause with hatred or terror instead, there is a subtle implication that measures can be taken to prevent future violence with similar causes. Democrats go one step further by using this frame to transition into calls for increased gun legislation, which would be a concrete step towards preventing future shootings.
On June 15 th , three days after the shooting, Democrats held a filibuster to push for a vote on gun control. The top frame that day for both parties is Frame 7 (Security & Defense), however different phrases represent different ideologies in this example as well. Democrats frame the need for gun control laws as a preemptive measure that will prevent gun violence (e.g., row seven of Table 6). Republicans use Frame 7 to discuss the need to prevent threats posed by ISIS (possibly due to the shooter's association with ISIS) as shown in row eight of Table 6. Additionally, some Republicans promote bipartisan efforts to stop the sale of guns to known terrorists (row eight). While all examples use Frame 7 to support gun control, this support is limited depending on party and identifiable by different key phrases, e.g. the general goal of "reforms 2 prevent gun violence" versus the specific target to "keep guns out of the hands of suspected terrorists".
Lastly, the impacts of the shooting on the quality of life of the community (or nation as a whole) are discussed in tweets having Frame 9. For example, row two of Table 6 shows a Democrat's tweet calling for action to keep gun violence tragedies from affecting communities. For this event, Republicans are more likely to refer to the "Orlando community" while Democrats are more likely to reference the "LGBT community", indicating that national versus specific-group phrases are useful in identifying Frame 9.

Future Work
Currently, this work requires human knowledge and engineering to compile the sub-phrases by party. Additionally, for computational simplicity all phrases are currently added to the baseline model for evaluation. Since frames can overlap and politicians can use the talking points of other parties, we hypothesize that frame prediction can be further improved by automatically testing all possible phrases with the baseline model.
For future work, we are building an automatic search over all possible phrase indicators, designed to choose the most indicative phrases for predicting each frame. We hope this tool will be useful for scientists from other fields, allowing them to compile their expert knowledge of a domain into many rules, which can then be analyzed to indicate the most useful features for further study of a subject.

Conclusion
In this paper we present an analysis of the usefulness of ideological phrases as a feature for predicting the frame of a political tweet. By compiling a list of common phrases and computing their similarity to tweets, we are able to increase the F 1 scores for half of the frames over a simpler language based model. We provide an analysis of our joint model in a supervised setting and show interesting real world examples. Finally, we propose the automation of phrase searching as a future work to improve the usefulness of this technique in other scientific communities.