Did you ever read about Frogs drinking Coffee? Investigating the Compositionality of Multi-Emoji Expressions

In this work, we present a first attempt to investigate multi-emoji expressions and whether they behave similarly to multiword expressions in terms of non-compositionality. We focus on the combination of the frog and the hot beverage emoji, but also show some preliminary results for other non-compositional emoji combinations. We use off-the-shelf sentiment analysers as well as manual classifications to approach the compositionality of these emoji combinations.


Introduction
Emojis do not only represent faces, but also concepts and ideas such as weather, objects, or activities (Pavalanathan and Eisenstein, 2016). We assume that these "concept" emojis are not always used as literal representations, but that there are many instances of their use as indicators of a certain emotion or intention. In this paper, we focus on two such "concept" emojis: the frog face emoji and the hot beverage emoji. We want to analyse the sentiment value of these emojis when used together and when used separately. By looking up tweets that contain the emojis and classifying them as positive, neutral or negative, we show that the meaning of the combination of these emojis is non-compositional, and could not be inferred from the meaning of its components. This is in-line with the behaviour of multi-word expressions and motivates for further study of the phe- nomenon. The choice of emojis is based on what has been observed in social media sites such as Facebook, Twitter and Tumblr. The users of these sites seem to use these emojis with a definite emotion and intention in mind. This is likely to have its origin in a popular internet meme known as "But That's None of My Business".
According to Know Your Meme 1 , this meme represents "a sarcastic expression used as a postscript to an insult or disrespectful remark said towards a specific individual or group". Thus, this is the kind of sentiment we expect to find in tweets that include both the frog face and the hot beverage emoji. When analysing tweets that only include the frog face emoji, we expect the sentiment to be neutral. For the hot beverage emoji, we assume a neutral sentiment too. When it comes to the frequency distribution of words, the separate meanings of the hot beverage emoji and the frog face emoji are expected to be quite literal. It is expected to find that words related to frogs and beverages are very common in these tweets, but not in tweets where both emojis appear.

Background
In recent years, sentiment analysis has found an invaluable source of material and information in social media. Because of their prevalence in social media, emoticons became the focus of many sentiment analysis studies. Kouloumpis et al. (2011) found that, in the microblogging domain, emoticons were more useful than part-of-speech features for training data collection. Research shows that emoticons do not only represent affective stances, but also intention or identity and can be used to strengthen a message (Derks et al., 2008). Because of their similarities, it is likely that this is true also of emojis.
Nowadays, emojis have started substituting emoticons for conveying emotions (Pavalanathan and Eisenstein, 2016) and they are becoming an important part of internet language. Chin et al. (2016) have used emojis to expand on positive or negative sentiment and classified tweets into five emotions. This shows that emojis can help us be more precise in our sentiment analysis. Moreover, Kelly and Watts (2015) investigate the "appropriation" of emojis. In other words, appropriation is the usage in a way that was not intended or envisaged by the designer (Dix, 2007). Even though there has not been much previous work on emojis, they have certain interesting characteristics that make them worthy of being investigated and they could become a useful feature when analysing the sentiment of social media text in the future.

Collecting tweets
In order to collect tweets, we went through iEmoji's archive of tweets 2 and manually retrieved 1,000 tweets that contained the hot beverage emoji and not the frog face emoji, 1,000 that had the frog face but not the hot beverage and 1,000 that had both. All tweets were published between 2013 and 2016 and were chosen randomly from random pages from the archive. They were pre-processed before the analysis: all hash characters, usernames and emojis were deleted.

Sentiment analysis and frequency distribution
The three tweet sets were analysed using two different analysers: i) TEXTBLOB (Smedt and Daelemans, 2012) and ii) VADER (Hutto and Gilbert, 2014). TEXTBLOB is based on a pattern library and returns a tweet's polarity and subjectivity. For the present work, we focused on polarity, which goes from -1 (negative) to 1 (positive), 0 being neutral. VADER was trained to be used on social media and microblogging texts. It returns a dictionary such as: "''pos': 0.446, 'neg': 0.0, 'neu': 0.554, 'compound': 0.6166". We used the "compound" value, since it represents the overall sentiment value of the tweet and uses the same scale as TEXTBLOB.
To evaluate their performance, we manually classified the 100 first tweets from each set and

Results
We evaluated our hypothesis using three different approaches: off-the-shelf sentiment analysers (4.1), manual classification (4.2) and finally we report on the most frequent words occurring in the context of the investigated emojis and their combination (4.3).

Sentiment Analysis
Both FrogFace Hot Beverage TEXTBLOB 0,048 0,11 0,17 VADER 0,011 0,16 0,18 After automatically analysing each of the 3000 tweets and obtaining its polarity score, the average score for every 1000 tweet sets was calculated. The results are shown in Table 1. The sentiment analysis shows a small but clear difference between the sentiment that each emoji has separately and the sentiment they have when used together. TEXTBLOB and VADER give very similar scores to the frog face tweets (0,11 and 0,16) and to the hot beverage tweets (0,17 and 0,18), and these two sets are at the same time quite similar to each other in their scores. Since they seem to have a similar sentiment, if their meaning were still literal when combined, we would expect the combination to have the same score, but from our results we can see that the score for the combination is lower. Even though we first assumed a more negative sentiment for the combination, the results make sense: tweets with both emojis are supposed to have a subtle, sarcastic tone, which is not easily recognised by sentiment analysers yet. Nevertheless, the distinction is there, and we will have a closer look at it in an additional evaluation using manual classification.

Manual Classification
We manually classified 100 tweets for each tweet set as negative, positive or neutral. An overview is given in  beverage set have a similar distribution and are in general fairly positive, but the combination of both is clearly negative, which supports the hypothesis in question. The meaning in this case is non-compositional and non-literal. While classifying the tweets, it was obvious that they had the same sarcastic, disrespectful tone as the "But That's None of My Business" meme, and some even used the same words. For example, one Twitter user wrote "I can solve all your problems, yet you do stupid shit. But that's none of my business." 3 , followed by a hot beverage and a frog face.
We also used this manual classification to evaluate the performance of both classifiers. The results are given in Table 2. Precision and recall are lower for both analysers when it comes to the combination of emojis. This is not surprising, since they were expected to be more difficult to analyse due to the sarcasm and the subtlety of their sentiment. The analysers performed best when classifying positive tweets. VADER performed sightly better for the frog face tweets, and TEXTBLOB performed better for the hot beverage tweets. We attribute the overall low performance to the fact that there was a lot of sarcasm in the tweets, and for this to be understood one needs some realworld knowledge which is almost impossible for a classifier to make use of.

Word count
For the word count-based evaluation, we looked at the 15 most common words in each set of 1000 tweets. If a word is repeated in the same tweet, it is 3 https://twitter.com/rai close/status/557368926506864640 like (97) frogs (85) morning (73)  7 don't (97) my (77) day (72)  8 all (91) have (66) my (65)  9 are (88) a (56) need (60)  10 a (86) but (55) good (59) (71) love (52) cup (57)  14 if (70) be (49) hot (56)  15 have (65) no (48) starbucks (55) counted as many times as it appears. In Table 4 we see the hot beverage set's most common word is "coffee", and the second one is "tea". Most of the words are semantically related and show a literal use of the emoji. For the frog face tweets, the most common word is "frog", and the plural "frogs" is on 6th place, which also points to a literal use. In the combined set the most common word is "but". Even though it is a function word and would usually be ignored, in this case it is not, because it is the first word in the meme ("But That's None of My Business") and it can indicate an objection. In this last case, the words do not tell us much about the sentiment or meaning of the emoji combination. This suggests that the emojis have a nonliteral meaning. The results also show emojis are often used in addition to the concept they represent, without substituting it. This could be an interesting direction for future work.

Frog face plus other beverages
While retrieving tweets we noticed a few tweets that expressed the same sentiment as the frog face and hot beverage tweets, but had a different beverage emoji, e.g. the tropical drink, cocktail glass or teacup without handle emojis. We found 18 such tweets. For example, a user wrote "Got a pic of u that could expose u so keep saying stuff" 4 and added a frog face and a teacup without handle. This sample is too small to confirm that the sentiment and meaning of these combinations will remain the same regardless of the drink emoji. However, nothing similar was observed when retrieving tweets containing the hot beverage. The frog face was never substituted by another face. This might indicate that the frog face is the main element of the combination and the one which brings the sarcastic, negative tone to the expression.

Clapping Hands Sign emoji
For the clapping hands sign emoji, we expect a change of meaning whenever it is repeated or placed in certain parts of the sentence. On its own, it is mostly used to show excitement or appreciation. However, when placed between each word in a sentence (sometimes two or three words), the connotation turns negative. In this case, the user is aggressively sharing their opinion, telling off the readers or correcting them. In order to investigate this hypothesis, we used iEmoji to retrieve 50 tweets in which the emoji appeared several times between the words and 50 tweets where the emoji appeared differently. We found a significant change in meaning and sentiment in all of them. For example, one user wrote "@Michael Sanchez I didn't expect this to happen. So happy rn. Good luck sir" 5 , with a clapping hands sign at its end. On the other hand, another user wrote "PLANNED PARENTHOOD DOESN'T JUST DO ABORTIONS" 6 . After every word, there was a clapping hands sign emoji, emphasising the message and seemingly trying to correct a false idea. We investigated only a small sample, but this use of the emojis was consistent 4 https://twitter.com/jc00003333/status/525776951966183 424 5 https://twitter.com/BrokenHarmonyML/status/78953474 3117205504 6 https://twitter.com/Jeeennnnaa/status/788912245975748 608 throughout all tweets, which indicates there is a pattern worth studying.

Nail Polish emoji
The nail polish emoji shares the sarcastic subtext of the emoji combination we extensively studied in this paper, but its meaning changes and becomes literal when used in combination with the lipstick emoji. In this case, the lipstick emoji retains its literal meaning and turns the nail polish emoji into a literal representation of nail polish. We retrieved 50 tweets with the nail polish emoji appearing next to the lipstick emoji, and 50 tweets with only the nail polish emoji. In 47 out of the 50 tweets, the difference was clear. For example, one user tweeted "@TAMU sorry my tampons and wallet are so threatening," 7 adding a nail polish at the end. This tweet is clearly sarcastic and has nothing to do with makeup. A counter example is this tweet: "I need a makeup advent calendar in my life" 8 with a nail polish next to a lipstick and a clearly literal meaning. This is a small sample, but the pattern is found in most of the tweets, which suggests it is not accidental.

Conclusion
We showed that there is a distinct change between the literal meaning the frog face and the hot beverage emojis have when used separately and the more subtle, non-literal meaning their combination has. This is shown in the frequency distribution of the words and in the sentiment analysis of the tweets. Both sentiment analysers showed a small yet observable difference in sentiment, and while this was not enough to reach any conclusions, the difference became clearer after the manual classification of the tweets.
This kind of multi-emoji expression is not unique, since there are similar cases that have also been examined, but it is uncommon and each case has its own characteristics. Even though the case that has been studied here is not a widespread phenomenon, emojis are a very recent addition to our communication methods. In the future we expect to see more transformations in the way emoji are used, which would be worth being researched in depth. 7 https://twitter.com/ShelbyLCole/status/80002227867936 7680 8 https://twitter.com/sophiaguyy/status/801143183157686 272