Exploring Optimism and Pessimism in Twitter Using Deep Learning

Identifying optimistic and pessimistic viewpoints and users from Twitter is useful for providing better social support to those who need such support, and for minimizing the negative influence among users and maximizing the spread of positive attitudes and ideas. In this paper, we explore a range of deep learning models to predict optimism and pessimism in Twitter at both tweet and user level and show that these models substantially outperform traditional machine learning classifiers used in prior work. In addition, we show evidence that a sentiment classifier would not be sufficient for accurately predicting optimism and pessimism in Twitter. Last, we study the verb tense usage as well as the presence of polarity words in optimistic and pessimistic tweets.


Introduction
"You know, ever since we were little, I would get this feeling like... Like I'm floating outside of my body, looking down at myself... And I hate what I see... How I'm acting, the way I sound. And I don't know how to change it. And I'm so scared... That the feeling is never gonna go away." The Edge of Seventeen 1 Much has been written about optimism and pessimism in psychological studies for decades (Scheier and Carver, 1992). These feelings are affected by one's personality from an early age (as pinpointed above) and can strongly impact one's psychological and physical health. For example, pessimism and negative attitudes impact negatively one's mental health, can induce suicidal thoughts, and affect negatively not only the person in question, but also their family and friends (Peterson and Bossio, 2001;Achat et al., 2000; 1 https://www.imdb.com/title/tt1878870/ Scheier et al., 2001). On the other hand, optimism reduces stress and promotes better physical health and overall well-being (Carver et al., 2010).
Despite that optimism and pessimism are under the scrutiny of many researchers (Rasmussen et al., 2009;Kumar et al., 2017), large scale analyses that explore optimism and pessimism in social media have just started to emerge (Ruan et al., 2016). However, Ruan et al. (2016) focused on identifying optimism and pessimism in Twitter using a simple "bag of words" representation with no emphasis on incorporating semantic information hidden in text. Often, a deeper understanding of the text that accounts for textual semantic similarities and the writer's intention are required in order to correctly detect the characteristics of optimistic and pessimistic feelings in tweets. Towards this end, our contributions in this paper are as follows. First, we focus on the question: "Would a deep learning approach help to discover these characteristics better than traditional machine learning classifiers used in prior work?" To our knowledge, we take the first step towards exploring the performance of deep learning models for optimism/pessimism prediction in Twitter and identify the most promising deep learning models for this task. Identifying optimism and pessimism in Twitter has many applications including identifying suicidal/depressive people and providing better social support (e.g., emotional/empathetic support) that can improve people's moods and attitudes (Yan and Tan, 2014;Biyani et al., 2014;Khanpour et al., 2018Khanpour et al., , 2017Qiu et al., 2011).
Second, since it may seem intuitive that a positive sentiment is associated with optimism and a negative sentiment with pessimism, we address the question: "Would a sentiment classifier be sufficient to correctly identify optimism and pessimism in social media?" Figure 1 shows evidence that a sentiment tool would not suffice on accurately predicting tweets with pessimistic and optimistic connotations (left and right side of the figure, respectively). We answer the above question by investigating a spectrum of sentiment analysis tools and datasets for optimism/pessimism prediction.
Third, we perform a linguistic analysis, first of its kind, and study the usage of verb tenses (past, present, future) in optimistic and pessimistic tweets, as well as the presence of polarity words associated with both types of tweets.

Datasets
In this section, we first describe the optimism/ pessimism Twitter dataset and then present two datasets used for sentiment analysis.
The Optimism/Pessimism Twitter dataset (OPT) was made available to us by Ruan et al. (2016). The total number of tweets in the dataset is 7,475. These tweets were sampled from data corresponding to 500 optimist and 500 pessimist users, and were manually annotated using Amazon Mechanical Turk. Precisely, each tweet was manually annotated by five independent annotators using a score between −3 (very pessimistic) and 3 (very optimistic). For our evaluation, we consider two different thresholds (0 and 1/-1) on the above score and create two settings as follows. In the first evaluation setting, a tweet is labeled as pessimistic if its score is smaller than or equal to 0, and optimistic, otherwise. In the second evaluation setting, a tweet is labeled as pessimistic if its score is smaller than or equal to −1, and optimistic if its score is greater than or equal to 1. A summary of this dataset is given in Table 1.  The Stanford Sentiment Treebank (SST) (Socher et al., 2013) is a corpus for sentiment analysis that capture complex linguistic patterns. This dataset 2 is based on a dataset originally introduced by Pang and Lee (2005) and consists of 10,662 sentences from movie reviews downloaded from rottentomatoes.com. From these sentences, 215,154 phrases were extracted using the Stanford Parser (Klein and Manning, 2003) and labeled using Amazon Mechanical Turk such that each phrase was annotated by 3 human judges.
The Twitter Sentiment Analysis (TSA) dataset, 3 available online for download, contains 1,578,627 tweets that are classified as 1 for positive sentiment and 0 for negative sentiment.

Experiments and Results
In experiments, we explore several deep learning models for optimism/pessimism prediction. The general training strategy is as follows: sentence embeddings are fed into a sentence encoder to obtain the sentence representation. The classifier consists of three fully connected layers topped by a softmax layer. Dropout was applied to the first layer only. We used several encoders as follows, based on: (1) Bidirectional Long Short Term Memory networks (BiLSTMs), which are a special type of Recurrent Neural Networks (RNNs) (Hochreiter and Schmidhuber, 1997); (2) Convolutional Neural Networks (CNNs), which consist of convolution and max pooling (Kim, 2014); and (3) Stacked Gated RNNs (Chung et al., 2015).
We used SGD optimizer (Goodfellow et al., 2016) with a learning rate of 0.1 and no weight decay. At every tenth epoch we decreased the learn-  ing rate by half. We used mini-batches of 40 samples. Dropout rate was set to 0.5 and the classifier's last three layers have 300, 200, and 100 neurons. We used GloVe vectors (Pennington et al., 2014) trained on Common Crawl 840B 4 with 300 dimensions as fixed word embeddings. For sentence embedding, after a cleanup process, sentences were transformed into a list of words, then words were replaced with word embeddings (GloVe) and padding was used to align batch sentences to the same size.

Optimism/Pessimism Prediction
In our first experiment, we explore the above deep learning models on the OPT dataset and compare their performance with that of two traditional machine learning classifiers, Naïve Bayes (NB) and Support Vector Machines (SVM), which were used in the previous work for this task by Ruan et al. (2016). In this experiment, the OPT dataset is split in train-dev-test as 80-10-10(%), respectively. We repeated each experiment 5 times and averaged the results. Our deep learning implementation is built on top of TensorFlow (Abadi et al., 2015). For NB and SVM, we used their implementation available in scikit-learn (Pedregosa et al., 2011). Table 2 shows the accuracy of all these models at tweet and user level for the two thresholds 0 and 1/-1 (as discussed in Section 2).
We can see that overall, the deep learning models achieve a much higher performance compared with the work by Ruan et al. (2016), i.e., the NB and SVM classifiers on "bag of words," for both tweet and user level with both thresholds, yielding an improvement in performance between 5%-10%. For example, at tweet level and 1/-1 threshold, CNN yields an accuracy of 90.32% as compared with NB, which achieves an accuracy of 84.10%. At user level and 1/-1 threshold, GRUStack yields an accuracy of 92.24%, as compared 4 https://nlp.stanford.edu/projects/glove/ with 81.80% achieved by SVM. Not surprising, for both tweet and user level, when we use a threshold of 0, the performance of all models is smaller compared with that of models obtained when we use a 1/-1 threshold. Intuitively, this is true since most of the tweets with a humanannotated score between -1 and 1 are in the "gray" area that is harder to classify. Note that Ruan et al. (2016) considered the tweets with a score between -1 and 1 as being neutral.

Sentiment vs. Optimism/Pessimism
In our second experiment, we investigate the correlation between sentiment and optimism / pessimism, and argue that sentiment analyzers, that are trained to predict sentiment (Liu, 2012;Pang and Lee, 2008), fail to detect optimism and pessimism. Specifically, we train several sentiment classifiers on the large SST and TSA sentiment datasets (described in Section 2) and evaluate the performance of these classifiers on the optimism/pessimism categories from the OPT dataset.   Table 3 shows the performance of several deep learning models trained on either SST or TSA datasets and evaluated on the OPT dataset. Note that the Dev set was used for model selection. As can be seen from the table, the models trained on the sentiment datasets perform poorly on the optimism/pessimism dataset. For example, there is a drop in performance from 80.19% to 67.60% when training on TSA (with an even larger decrease when we train on SST).
The SST/TSA sentiment classifiers are trained to predict the sentiment as negative, neutral, or positive. To calculate the accuracy in Table 3, an optimistic tweet predicted as positive by the sentiment classifier counts as a correct prediction, whereas an optimistic tweet predicted as either neutral or negative by the sentiment classifier counts as an incorrect prediction (similarly for pessimistic tweets). This analysis is done at tweet level for the threshold of 0.  Figure 2 shows the normalized number of examples from optimism and pessimism categories classified as positive, negative and neutral, using the CNN model trained on TSA. Precisely, we show how many tweets from the set of optimistic (or pessimistic) tweets in the OPT dataset are predicted as negative, neutral or positive by the TSA sentiment classifier. The numbers on each row sum up to 1. As we can see from the figure, although pessimism is more correlated with a negative sentiment, 13% of the pessimistic tweets are classified as positive (with similar results on the optimism category).

Linguistic Analysis
In this section, we perform a linguistic analysis and study the usage of verb tenses in optimistic and pessimistic tweets, as well as the presence of polarity words associated with both types of tweets. This analysis is done at tweet level with 1/-1 threshold. The reason for using the 1/-1 threshold is that we wanted to study the usage of verb tenses and polarity words in tweets that are clear optimistic or clear pessimistic (far from the decision boundary).

Verb Tenses in Optimism/Pessimism
For this analysis, we used the part of speech tagger spaCy 5 and assigned the verbs to their corresponding tenses according to the Penn Treebank Project; that is, the tags VBD and VBN correspond to past tense, VBG, VBZ , VBP correspond to present tense, whereas an MD tag followed by VB (possibly with a negation between them) corresponds to the future tense.
As mentioned, a tweet was considered optimist if its manually annotated score was above 1 and pessimist if the score was below −1. The numbers of tweets with past, present, and future tenses in the optimistic category are: 1,474, 7,444, and 561, respectively, whereas these numbers in the pessimistic category are: 1,276, 5,311, and 325, respectively.  Figure 3 shows the normalized verb occurrences at past, present and future tenses in optimistic and pessimistic tweets. As can be seen from the figure, the present tense is the most prevalent for both categories, although there are more present tense verbs in the optimistic category compared with the pessimistic one. We can also observe that more past tense verbs occur in the pessimistic category and less future tense verbs in the pessimistic one.
While there are some common verbs such as "be," "have," and "do," that appear most frequently in both optimistic and pessimistic categories at all three tenses, there are some verbs that are more specific to one category than the other. Examples of such verbs and their frequencies from both categories at the present tense are shown in Table 4. As we can see, optimism is characterized more by verbs with a positive connotation. [optimism] [pessimism] ('be ', 1662) (

Polarity Words in Optimism/Pessimism
Next, we analyze the association of polarity words from the positive and negative lexicons constructed by Hu and Liu (2004), in both tweet categories: optimism and pessimism. Instead of using the presence or absence of the words from the two lexicons in tweets, we calculated the cosine similarity between the word embeddings of the words in the two lexicons with the words in the tweets. If the similarity is above 0.8, then we consider the word from the corresponding lexicon to be present in the tweet (or synonym with a word in tweet). Using the cosine similarity between the word embeddings of words in lexicons with words in tweets captures not only the exact match between the words (a cosine similarity of 1 for exact match), but also incorporates the semantic information that exists in the text. Although this word similarity computation relaxes the exact match/presence of a word in a tweet and aims at incorporating semantic similarity, a high similarity between antonyms may occur since word embeddings are known to not differentiate well between synonyms and antonyms, which tend to appear in similar contexts.  Figure 4 shows the number of polarity words in optimistic and pessimistic tweets. As shown in the figure, more positive words appear in optimistic tweets compared with negative words (1,242 vs. 71), while there is not a substantial difference between the numbers of positive and negative words in pessimistic tweets (118 vs. 210). Table 5 shows the top most frequent polarity words associated with optimism and pessimism. As we can see, words with a negative polarity (e.g., bad) although not very frequent, still appear in optimistic tweets. This supports our intuition that a sentiment model is not enough to accurately predict pessimism and optimism in Twitter.

Concluding Remarks
In this paper, we explored deep learning models for optimism and pessimism prediction in Twitter and showed that these models substantially outperform traditional classifiers such as Naïve Bayes and Support Vector Machines. To our knowledge, this work is the first computational study that explores optimism and pessimism using deep learning. We also showed that a sentiment classifier would not be sufficient for accurately predicting optimism and pessimism. This topic is less explored despite its importance in many applications such as identifying suicidal/depressive people.
Interesting future directions are: understanding how one's age is correlated with optimism / pessimism; if one user is characterized by a mixture of topics, is that user optimist (pessimist) across all these topics? Thus, decomposing a user's textual data into topic and correlating this with optimism and pessimism may be interesting to explore; last, studying how optimism and pessimism are affected by sarcasm.
As we started our study with a pessimistic quote from the movie "The Edge of Seventeen," we end our study with a quote from the same movie, with a positive sentiment and full of optimism: "Life's about taking risks. Don't be afraid to put yourself out there."