Label Propagation-Based Semi-Supervised Learning for Hate Speech Classification

Research on hate speech classification has received increased attention. In real-life scenarios, a small amount of labeled hate speech data is available to train a reliable classifier. Semi-supervised learning takes advantage of a small amount of labeled data and a large amount of unlabeled data. In this paper, label propagation-based semi-supervised learning is explored for the task of hate speech classification. The quality of labeling the unlabeled set depends on the input representations. In this work, we show that pre-trained representations are label agnostic, and when used with label propagation yield poor results. Neural network-based fine-tuning can be adopted to learn task-specific representations using a small amount of labeled data. We show that fully fine-tuned representations may not always be the best representations for the label propagation and intermediate representations may perform better in a semi-supervised setup.


Introduction
Online hate speech is anti-social communicative behavior and targets minority sections of the society based on religion, ethnicity, gender, etc. (Delgado and Stefancic, 2014). It leads to threat, fear, and violence to an individual or a group. As monitoring these contents by humans is expensive and timeconsuming, machine learning-based classification techniques can be used. The last few years have seen a tremendous increase in research towards hate speech classification (Badjatiya et al., 2017;Nobata et al., 2016;Del Vigna et al., 2017;Malmasi and Zampieri, 2018). The performance of these classifiers depends on the amount of available labeled data. However, in many real life scenarios there is a limited amount of labeled data and abundant unlabeled data. In need of data, different data augmentation techniques based on synonym replacement (Rizos et al., 2019), text generation (Rizos et al., 2019;Wullach et al., 2020), back translation (Aroyehun and Gelbukh, 2018), knowledge graphs (Sharifirad et al., 2018), etc, have been employed for up-sampling the training data in the field of hate speech classification. The performance gain by these techniques is small, and they fail to take advantage of the available unlabeled data.
Semi-supervised learning is a technique to combine a small amount of labeled data with a large amount of unlabeled data during training (Abney, 2007), intending to improve the performance of the classifiers. Label propagation (Xiaojin and Zoubin, 2002) is a graph-based semi-supervision technique analogous to the k-Nearest-Neighbours algorithm. It assumes that data points close to each other tend to have a similar label. These algorithms rely on the representation of data points to create a distance graph which captures their proximity.
Recently, pre-trained word embeddings such as Word2Vec, fastText, Global Vectors for Word Representation (GloVe) have been used for representing words for hate speech classification (Waseem et al., 2017;Badjatiya et al., 2017). Furthermore, pre-trained sentence embeddings such as InferSent, Universal Sentence Encoder, Embeddings from Language Models (ELMo) have been used for the task of hate speech classification (Indurthi et al., 2019;Bojkovskỳ and Pikuliak, 2019). These pretrained sentence embeddings are generic representations and are unaware of task-specific classes. Transforming the pre-trained sentence embeddings to task-specific representations can be helpful for label propagation, and our work explores this direction. The contributions of this article are: • evaluation of label propagation based semisupervised learning for hate speech classification; • comparison of label propagation on pretrained and task-specific representations learned from a small labeled corpus.

55
The rest of the paper is organized as follows: Section 2 describes label-propagation based semisupervised learning for hate speech classification. Section 3 describes the experimental setup. Results and discussions are presented in Section 4.

Semi-supervised Learning
In this section, we briefly describe sentence embeddings and the label propagation algorithm. This is followed by our methodology for semi-supervised training for hate speech classification.

Sentence Embeddings
Sentence embeddings are fixed-length vector representations that capture the semantics of the sentence. These embeddings are learned from large unlabeled corpora. Similar sentences are close to each other in this vector space and hence they are used as an input representation for various downstream tasks. We use the pre-trained Universal Sentence Encoder (USE) (Cer et al., 2018) to represent the tweets.

Label Propagation
Label propagation is a graph-based semisupervised learning technique that uses the labels from the labeled data to transduce the labels to unlabeled data. Label propagation considers two sets: (x 1 , y 1 ) . . . (x l , y l ) ∈ L as labeled set and Here, C is the number of classes, and the labeled set L consists of all the classes. The algorithm constructs a graph G = (V, E), where V is the set of vertices representing set L and U , and the edges in set E represents the similarity between two nodes i and j with weight w ij . The weight w ij is computed such that nodes with smaller distances (similar nodes) will have larger weights. The algorithm uses a probabilistic transition matrix T : The algorithm iteratively updates the labels Y ← T Y , by clamping the labels of the labeled set, and until Y converges. Figure 1 outlines the semi-supervised learning setup adopted in our study. We use a Multilayer Perceptron (MLP) for two purposes: (a) to learn task-specific representations; (b) to perform multiclass classification. Task-specific representations: First, the pretrained representation S ∈ R D are transformed to task-specific representationŜ ∈ RD with the MLP classifier trained using a small amount of available labeled set L. After training the MLP classifier, we pass the label agnostic pre-trained representation S of a given sample from the labeled set L and unlabeled set U as input to the MLP. We consider the activation from outputs of the hidden layers (h1 and h2) as two different task-specific transformed rep-resentationsŜ. Since the MLP classifier is trained with labeled data, we expect the representations h1 and h2 to capture task-specific label information.

Proposed Methodology
Semi-supervised training: The pre-trained representations S or task-specific representationsŜ will be used to represent data points in label propagation. We perform label propagation using the labeled set L and unlabeled set U , to obtain the labels for the samples in U . Finally, the pre-trained embeddings of set L and set U , along with original labels for L and labels obtained from label propagation for U will be used to train an MLP for hate speech classification.

Data Description
We consider two datasets, by Founta et al. (2018) and Davidson et al. (2017), containing tweets sampled from Twitter. Table 1 shows the statistics of these datasets. Both datasets have an imbalanced class distribution with 'hateful' as a minority class.   (2018) and Davidson et al. (2017) Twitter data by Founta et al. (2018): A large part of this dataset is collected using random sampling. Since hate and abusive speech occurs in a very small percentage, the authors chose to perform boosted sampling. The dataset has four classes, namely 'normal','spam','abusive', and 'hateful'. We exclude the samples from the 'spam' class. This brings down the total number of samples in the dataset from 100K to 86.9K.
Twitter data by Davidson et al. (2017): It is collected based on the keywords from the hatebase 1 lexicon. The dataset is annotated into three classes, namely 'hate speech', 'offensive language' and 'neither'. Further in this paper, to map the class labels to Founta et al. (2018) dataset, we consider 'neither' class as 'normal' and 'offensive language' as 'abusive'. Since 'abusive speech' and 'offensive language' are strongly correlated (Founta et al., 2018), we use the terms interchangeably.

Text Preprocessing
We remove all the numbers and punctuations except '.', ',', '!', '-', and apostrophe. The repeated occurrence of the same punctuation is changed to a single one. Hashtags are preprocessed by removing the '#' symbol and those with multiple words are split based on uppercase letters. For example, "#getBackHome" is processed into "get Back Home". This is done to ensure that the text tokenizer treats multiple words as a sequence of distinct words. However, the hashtags with multiple words without any uppercase is left as it is. Twitter user handles and the symbol 'RT' which indicates re-tweet are removed.

Data Split
We split the datasets randomly into three portions 'training', 'validation', and 'test' sets, each containing 60%, 20%, and 20% respectively. To simulate the semi-supervised learning setup, we partition the training set into labeled and unlabeled set with a ratio of 1:4. The final labeled set consists of 12% of the entire dataset.

Sentence Embeddings
We obtain pre-trained sentence embeddings from the transformer-based model of USE 2 . This model is trained on data from Wikipedia, web news, web question-answer pages, and discussion forums. Each sentence embedding is of 512 dimension.

Label Propagation
The label propagation API provided by scikit-learn library 3 is used. Euclidean distance is used to compute the distances between two data points. Based on the validation set, we have chosen 80 nearest neighbors and a maximum of 4 iterations.

Model and Representation Setup
We use an MLP with two hidden layers to perform classification and derive transformed representations. As shown in the Figure 1, the MLP model always has pre-trained representations S as its input. The hidden layers have ReLU activation. The transformed representationŜ are taken from the 1st or 2nd hidden layers, referred as 'h1-representation' and 'h2-representation' in our experiments, respectively. We use Adam optimizer, early-stopping based on validation set, and maximum of 10 epochs. Both the hidden layers have 50 units. The model weights learnt while training the system with only the labeled data is used to initialize the classifier before training with labeled and unlabeled data. data is fixed, but the amount of labeled data is varied to 10%, 20%, 30%, 50% and 100% of the available labeled set. The 'Baseline' is obtained by training the MLP classifier with only the labeled set, and without using label propagation. The 'Pre-trained', 'h1', and 'h2' show the results of the classifier trained using labeled and unlabeled set, the respective representation was used to label the unlabeled set using label propagation. In all the four cases, pre-trained embeddings were used as input to the MLP classifier.

Results and Discussion
Results in Figure 2 and Figure 3 show that semisupervised training using label propagation on pretrained representations performs worse than the 'Baseline' classifier. This implies that label propagation to the unlabeled sets using pre-trained representations introduces significant noise in the classifier. To support this hypothesis, we analyse the intra-class and inter-class separations.   Table 2 shows the average intra-class and interclass euclidean distance across different representations on the training set, containing the labeled and unlabeled set. Ground truth labels are used for this analysis. From Table 2, we observe that  the intra-class and inter-class distance are similar for pre-trained embeddings. This implies that the representations belonging to the samples from the same class were not close to each other and those from different classes were not far from each other. As hypothesised, h1 and h2 capture class information and hence have lower intra-class and higher inter-class distances.
Further, we qualitatively analyze the euclidean distances across pre-trained representation and h1 representation using color maps, shown in Figure  4 and Figure 5 respectively. We randomly consider 300 samples from Founta et al. (2018) dataset, of which the 1 st set of 100 samples belongs to the labeled set of 'normal' class, the 2 nd and the 3 rd set of 100 samples belong to the labeled sets of 'abusive', and 'hate' classes respectively. From Figure 4, we observe that the pre-trained representation has similar distance across all the samples, thus appearing in a similar color. This infers that pre-trained representation does not contain any task-specific label information, and are generic. However, as shown in Figure 5, the intra-class distance of the samples within 'normal' and 'abusive' classes is smaller than their inter-class distance of h1 representation, hence appearing as lighter-colored squares. From these figures, we observe that h1 representation captures task-specific label information. Figure 2 and Figure 3 further show that semisupervised training using label propagation on transformed representations ('h1' and 'h2') performs better than the 'Baseline' when amount of labeled data is small. The performance is comparable for larger labeled data. Thus, semi-supervised learning with label propagation using task-specific representations can have significant advantages when the available labeled samples are very few.
Furthermore, in a few cases, label propagation using the representations from the intermediate hidden layer 'h1' performed slightly better than label propagation using the representations from the final hidden layer 'h2'. This hints that fully finetuned representation may not always be the best performing representations in the k-Nearest Neighbors space.

Conclusion
In this article, we have explored label propagationbased semi-supervised learning for hate speech classification. We evaluated our approach on two datasets of hate speech on the multi-class classification task. We showed that label propagation using the pre-trained sentence embeddings reduces the performance achieved with only the labeled data. This is because pre-trained embeddings do not contain any task-specific information. We validated this by comparing the average intra-class and interclass distances. Further, training an MLP classifier with a small amount of labeled data and using the activations of its hidden layers as task aware representations improved the performance of the label propagation and semi-supervised training. It also appears that the fully fine-tuned representations from the MLP may not be the best representations for the label propagation. We can conclude that semi-supervised learning based on label propagation helps to improve hate speech classification in very low resource scenarios and that the performance gain reduces with more amount of labeled data.