Jujeop: Korean Puns for K-pop Stars on Social Media

Jujeop is a type of pun and a unique way for fans to express their love for the K-pop stars they follow using Korean. One of the unique characteristics of Jujeop is its use of exaggerated expressions to compliment K-pop stars, which contain or lead to humor. Based on this characteristic, Jujeop can be separated into four distinct types, with their own lexical collocations: (1) Fragmenting words to create a twist, (2) Homophones and homographs, (3) Repetition, and (4) Nonsense. Thus, the current study first defines the concept of Jujeop in Korean, manually labels 8.6K comments and annotates the comments to one of the four Jujeop types. With the given annotated corpus, this study presents distinctive characteristics of Jujeop comments compared to the other comments by classification task. Moreover, with the clustering approach, we proposed a structural dependency within each Jujeop type. We have made our dataset publicly available for future research of Jujeop expressions.


Introduction
With the rapid improvement of information and telecommunication technologies, people have become not only consumers, but also producers of media content (Jenkins and Deuze, 2008). With this trend, there are a number of online media platforms that allow people to interact with other users anywhere and anytime (Burgess and Green, 2018). On these platforms, users actively create and share their contents, and express their thoughts and opinions on other users' contents (Van Dijck, 2013). In particular, online fan communities, where fans interact with each other, tend to use such platforms to share their contents and opinions on their favorite stars (e.g., Ariana Grande 1 , BTS 2 ; Baym (2007); Littlejohn and Foss (2009)).
With this vitalization of the communities on the platforms, several novel interaction patterns have been observed among South Korean users. Among these patterns, Jujeop in online environments is one of the notable phenomena presented by South Korean fans (Figure 1). Although the dictionary definition of the Korean word Jujeop refers to a disgraceful or silly behavior of a person, the term has evolved into a facetious expression with an implicit sense of humor in the online K-pop community; in South Korean culture, Jujeop is a punning activity that makes conversations enjoyable and allows users to engage on platforms (Yu et al., 2018). Miller and his colleagues (Miller et al., 2017) defined a pun as "a form of wordplay in which a word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another word, for an intended humorous or rhetorical effect." Based on this definition, the majority of recent studies have proposed several pun generation models using machine learning approaches (He et al., 2019;Luo et al., 2019).
However, compared to a huge body of prior research on English puns (Yu et al., 2018(Yu et al., , 2020, only a few studies have been conducted on Korean puns in online environments. Because of some obstacles including the unique linguistic and cultural aspects of South Korea, there are several limitations in studying users' punning activities (Choi, 2018).
Thus, we propose the first Korean corpus, annotated for Jujeop comments, and categorize them into four different types. We have made the dataset publicly available. 3 2 Jujeop Data

Data Collection
As Jujeop comments are frequently observed in Youtube channels of K-pop stars, we assumed that high number of views in a channel guarantees the presence of the Jujeop comments. Based on this assumption, we collected 281,968 users' comments on K-pop stars from 285 Youtube channels 4 , which have the number of views between 5,177 and 38,039,597. Then, we conducted the pre-processing procedures for the remaining Korean words (i.e., excluding words used for commercial purposes).
We sorted the comments based on the number of likes a Jujeop comment received. The comments that had more than the average number of likes in the collected comments (i.e., 167) were employed. With this approach, 8,650 comments were selected for annotation.

Annotation
Ten annotators who has been enthusiastic fans of their K-pop stars for at least 2 to 15 years (Mean: 9.3 SD: 4.2) and has been frequently exposed to Jujeop comments were employed for the annotation process. After explaining the definition and examples of Jujeop comments, each annotator was asked to respond to the following question to classify, whether each comment is a Jujeop comment: • Is this a Jujeop comment, which has a sense of humor by praising K-pop stars with exaggerations and flashy modifiers?
Then, each annotator was asked to classify the Jujeop comment into one of the following types.

Fragmenting words to create a twist
The comments in this type intentionally fragment a specific word and extract/concentrate a single character from the word to disguise the word's full meaning (e.g., 'pretty' to 't'), in order to create a twist in the sentence meaning.
When one of the characters is included in both a specific word and sentence with the same pronunciation, the word and sentence are linked. This means that there are two steps in a Jujeop comment. After the sentence with hidden or sarcastic meanings is first presented, the word with complimentary meanings is then provided. For instance, 't' can mean 'tee' (t-shirt) as it has the same Korean pronunciation. Moreover, the fragmented word (e.g., 'T') usually carries a neutral connotation, while the complete word (e.g., 'Pretty') carries a positive connotation.
Because two words are linked and combined to make a sentence ('t' (t-shirt) and 'pretty'), it creates a pun in Korean: 언니. 왜 맨날 똑같은 티만 입어? 프리티! Sis, Why do you always wear the same Tee? pretTee! The first sentence asks why she always wears the same t-shirt, which is pronounced [ti:]. Then, the following word changes the whole sentence meaning, which makes the initial meaning of the sentence a compliment about her prettiness [prti], thus creating a humorous twist.

Homophones and Homographs
Both homophones and homographs are sometimes employed to create pun expressions.
Homophones are defined as follows: "when two or more words, different in origin and signification, are pronounced alike, whether they are alike or not in their spelling, they are said to be homophones" (Bridges, 2018). The definition of homographs is "words that have more than one meaning but share the same orthography" (Twilley et al., 1994).
Users can employ specific lexical features of homophones and homographs to make a Jujeop comment. After a user makes his/her first sentence with the original meanings of words, they employ other word meanings in the second sentence to compliment the K-pop stars while allowing other users to enjoy the fun.
For example, George Bush, the former US president, has the same pronunciation in Korean and English (Korean: '조지 부시'), when George Bush is employed as a big name. The South Korean pronunciations of George is identical to the phrase 'to beat somebody/something' (Korean: '조지(다)'), while the pronunciation of Bush is identical to 'to break something' (Korean: '부시(다)'). Thus, the pronunciations of George Bush and 'to beat somebody/something + to break something' can be the same in Korean, although the meanings of the words differ depending on whether they are employed as a big name or as verbs.
내 마음을 조지고 부시니까. Change your English name to George Bush... because you beat and break my heart.

Repetition
This is a type of repetition of the same phrase. As presented in the following example, the comments in this type employ repetition to emphasize the complimentary meanings on the K-pop stars.

Nonsense
The comments in this type include the K-pop stars within fictions. The majority of such comments flatter the stars by using exaggerated and almost nonsensical, over-the-top expressions. One representative example is presented below: 그녀가 예쁘다고 생각하는 사람 일어나! 라고 했더니 지구가 일어나서 태양계 순서가 바뀌었잖아. I said, Anybody who thinks she's pretty, get up! and then the whole Earth got up and the order of the solar system changed.
There is no way that the Earth can 'get up' like a human being, nor could the order of the solar system change due to a person's prettiness. Such ridiculous and exaggerated expressions create humor and a profound expression with which fans can express admiration for their favorite celebrities.

Experiments
We conducted two NLP tasks to investigate whether the labeled data can be significant in understand-ing Jujeop comments. First, we proposed several deep learning models to verify the annotated Jujeop comments. Then, we clustered Jujeop comments to figure out specific linguistic structures.

Jujeop Classification
At first, for the Jujeop classification, we applied three baseline classifiers for the experiment: Convolutional Neural Network (CNN; Kalchbrenner et al. (2014)), Bidirectional Long Short-Term Memory (BiLSTM; Schuster and Paliwal (1997)), and KoBERT 5 . All model configurations are presented in Appendix A. Because more than 80% of the annotated comments in the dataset are non-Jujeop comments, we randomly selected 1,573 non-Jujeop comments, which is the same number of Jujeop comments to address the data imbalance issue. Then, we randomly divided the collected comments into training (2,256, 72%), validation (260, 8%), and testing (630, 20%) sets. We tokenized each comment with the Mecab tokenizer of KoNLPy package 6 . The maximum word counts of the comments and total vocabulary size are 58 and 6,536, respectively.    Table 2 presents the classification results with four evaluation metrics. In general, the KoBERT showed the greatest levels of all evaluation metrics. In particular, the accuracy of the KoBERT (73.65%) was higher than those of the CNN (69.05%) and BiLSTM (70.79%). In case of the recall level of Jujeop comments, it can be explained by the potentiality of misclassifying Jujeop to non-Jujeop comments. Moreover, we measured macro F1-score for the binary classification task (Table 3). Compared to the other benchmark models, KoBERT showed the best performance (72.91%).
Furthermore, we computed macro F1-score for the Jujeop classification task as each type of comment had a skewed distribution (Tran et al., 2018). The details of configurations are attached on Appendix A. Table 3 shows KoBERT with the highest performance of 77.18% followed by CNN (62.63%) and BiLSTM (56.96%). The implemented models are publicly available 7 .

Jujeop Clustering
Pun usually relies on specific linguistic structure that can be classified based to patterns of the syllable, word, or phrase similarity (Binsted and Ritchie, 1997;Ritchie et al., 2007). Since, Jujeop comments share the characteristic of the pun, we assumed that Jujeop comments within the same type would share similar dependency relations.
Based on the assumption, we employed partof-speech (pos) tagging to analyze the distinctive linguistic structure of each Jujeop type. Then, the tagged sentences were used as the input for the unsupervised learning algorithm, which allows identification of data into similar groups or clusters (Likas et al., 2003).
We utilized Okt pos tagger, which is commonly used to analyze the social media data analyses (Park and Cho, 2014). First, to balance the number of each type in Jujeop comments, we randomly selected 50 samples from type 4. Then, we vectorized each pos tag of the sentence as an input to the K-means clustering with K as 4, which represents 4 types of Jujeop comments. Figure 2 represents the confusion matrix of the true and the predicted data points. The total accuracy of the K-means clustering was 32%, where the most correctly predicted type was type 2 with the 34 out of 57 correct predictions (59.65%).
Whereas most of type 1 were classified into type 3 (23 out of 39), which indicates that two types might share similar dependency relations. The single word appeared at the beginning of the sentence that was used again at the later part might have been characterized as a repetition. Type 3 was clas- Moreover, type 4 showed the lowest clustering accuracy with 10% (5 out of 50). This indicates that nonsense might be interpreted as semantic feature rather than syntactic feature. The further explanations and visual supplements are attached in Appendix B.

Conclusion
The current study first conceptualized the construct of Jujeop, which is one of the Korean pun interaction patterns on social media and annotated 8,650 comments. To provide a better understanding of Jujeop comments, four separate Jujeop types were proposed and labeled. Then, the presented NLP tasks results imply that Jujeop comments and each type of Jujeop has semantic and syntactic distinctiveness compared to the other comments. To employ a CNN-based classifier, we created a sequence of the tokenized words by embedding a layer with 128 units. The sequence was then sent to the CNN layer with 64 units. The max pooling layer was used to extract the prominent features of the given data. The final output was computed by sigmoid function to classify whether or not the given comment is a Jujeop comment. Ten epochs were employed in the training sessions with 32 batch size.

A.1.2 Quaternary classification
We used the the same configurations with the binary classification task except optimizer, loss and activation functions of the last layer. For the multiclass classifiction task, we employed the softmax activation function for the last layer and sparse categorical crossentropy for the loss function with adam optimizer. Also, we compiled the model with class weights by scikit-learn package 8 to handle the class imbalance problem.

A.2.1 Binary classification
The tokenized words of the comments were outputted to the embedding layer with 128 units. The representation of the input data was then sent to the bi-directional LSTM layer with 64 units. The final output of the BiLSTM was calculated through sigmoid function. We trained the model with 10 epochs with 256 batch size.

A.2.2 Quaternary classification
We changed the optimizer, loss and activation functions of the last layer as in a CNN classifier for the multi-class classification. We also compiled the model with same class weights as in the CNN classifier.

A.3.1 Binary classification
To employ a KoBERT model, we adopted a built in SentencePiece tokenizer. We set embedding size as 128 and trained the model with 10 epochs. We set the batch size as 32 and learning rate as 0.00002.