MayAnd at SemEval-2016 Task 5: Syntactic and word2vec-based approach to aspect-based polarity detection in Russian

This paper describes aspect-based polarity detection system for Russian, used in aspect-based sentiment analysis task (ABSA) of SemEval-2016 (Task 5, subtask 1, slot 3). The system consists of two independent classiﬁers: for opinion target expressions and for implicit opinion target mentions. We introduce a set of standard unigram features together with more sophisticated ones: based on sentence syntactic structure and based on lemmas vector representation. Being applied to Russian restaurant reviews, our system achieved best quality among four participants.


Introduction
Aspect-based sentiment analysis task (ABSA) drew much attention with the growth of Internet retail sites and facilities for customers to make a free text product review. ABSA systems allow businesses to track reputation of their products even more granularly than before: per aspect. At the same time, customers can be provided with per-aspect sentiment summaries of reviews about concurrent products to simplify their choice.
SemEval-2016 Task 5 (ABSA) organizers (Pontiki et al., 2016) proposed two subtasks: Sentencelevel ABSA and Text-level ABSA. First subtask was split to three slots: Aspect Category Detection, Opinion Target Expression, Sentiment Polarity. Training data for the task were available for various languages (English, Russian, etc.) and domains (restaurants, laptops, etc.). Slot 3 can be described as follows: given a set of reviews labeled with OTEs (opinion target expressions: start-end-entity-attribute quadruples) determine a polarity label for each OTE: positive, negative, neutral (mildly positive or mildly negative sentiment). Additionally polarity labels should be computed for entity-attribute pairs assigned to reviews sentences.
This paper describes our team participation in slot 3 evaluation for Russian language Restaurant domain. The rest of the article is structured as follows: section 2 mentions some recent approaches to aspect-based polarity detection task, section 3 contains our system description, section 4 provides our system evaluation results.

System Description
Our aspect-based polarity detection system consists of two independent parts: OTE polarity classifier and implicit entity mention polarity classifier. Both parts are based on linear SVM with L2 regularization for three classes: positive, negative, neutral (available training set contains the only conflict polarity label, thus it is ignored).
At first, we preprocess review using Texterra (Turdakov et al., 2014) system. We perform tokenization, part of speech tagging, lemmatization and syntactic parsing for each review sentence.
All used features can be grouped to several classes: • opinion target features; • sentence-level lexical features; • syntactic-level features.
Some most valuable features are based on word vector representation obtained by word2vec. For word2vec training we used 40140 unlabeled reviews from restoclub 1 resource. All word2vec models were trained with same dimensionality (100) and same window size (10). We tried word2vec with different input data preprocessing approaches: without preprocessing, with lowercased words and with lemmatized words. The last one demonstrates the best result on 8-fold cross-validation (see section 4 for details).
The rest of this section describes each group of features in details.

Opinion Target Features
Initially we tried to train separate SVM classifiers for each entity-aspect pair. Unfortunately training data contains just a few examples of certain entityattribute pairs (e.g., DRINKS#PRICES are presented in training data only 17 times). Moreover, we suspect that different entities can share most part of sentiment vocabulary for some attributes (e.g., quality).
So, we decided to use a single SVM classifier for all entity-aspect labels and to extract two additional features: opinion target entity label and opinion target attribute label. These features are used in both system parts. 1 www.restoclub.ru

Sentence-level Features
As implicit entity mention has no boundaries, we use the whole sentence to extract its features. We use a number of common features, usually exploited in NLP tasks: sentence word unigrams, sentence lemma unigrams, part of speech tag of each word in sentence. Moreover we use frequency of punctuation token unigrams in sentence as a feature.
Described features group is used for implicit entity mention polarity classifier as well as for OTE polarity classifier, because of slight quality improvement.
For OTE polarity classification information about entity mention is also used: we utilize vector representations of lemmatized sentence words. We extract lemmas in window of size 5 (5 words to the left and 5 words to the right from OTE), map them to vector representations and concatenate these representations. OTE polarity classifier exploits a number of syntactic-level features. Our basic assumption is that OTE polarity is closely related to some dependent words (especially adjectives), which may express sentiment. Our system uses Texterra syntactic dependency parser to produce syntactic tree of input sentence.

Syntactic-level features
Each OTE should match subtree in syntactic tree, which is treated as a single node. In case OTE is not a subtree, syntactic-level features for classifi-  Figure 1 depicts syntactic tree extracted from review sentence "I would like to mention very nice table appointments.". Opinion target expression "Table appointments" is a subtree, so it is treated as a single node. We start from using OTE parent words as features: we extract a chain of parent words for given OTE (up to 3 words), and use their lemmas vector representations as features. In example syntactic tree ( Figure 1) there are two parent words: "otmetit"' at depth 1 and "hochu" at depth 2.
Additionally we use syntactic children chains to extract some relevant features. For each word dependent on OTE (in parent-child chains of length up to 2) we use syntactic relations path conjoint with word form, lemma and lemma vector representation as features. In example (Figure 1) syntactic children words are "krasivuju" with relations path "attributive" and "ochen"' with relations path "attributiverestrictive".

word2vec Training Settings
During system development we tried several word2vec models trained with different settings. We varied data sources (Russian Wikipedia and restaurant reviews obtained from restoclub.ru) and data preprocessing (without preprocessing, lowercased words and lemmatized words). We experimented with 8-fold cross-validation on training data. The best result was obtained with restoclub.ru data source and lemmatization preprocessing (see Table 1).

Features Impact
In this section we try to evaluate each feature group impact. We test quality of our system with all de-scribed features on test data (using gold labels provided by organizers after evaluation) and using 8fold cross-validation on train data (as we did before evaluation). Then we consecutively test our system without some group of features. Test results are presented in Table 2.  In spite of the fact that word lemmas vector representation features are present in sentence-level and syntactic features groups, we additionally test these features impact separately.
As it could be seen, most significant feature groups are sentence level features and lemma vector representation features. However, large impact of sentence-level features group is explained with the fact that implicit entity mention polarity classifier uses a few additional features, and when we don't use sentence-level features it uses only opinion target features. If we test OTE polarity classifier separately, we obtain 0.8115 accuracy score without sentence-level features versus 0.8287 with all features on cross-validation (0.0172 impact) and 0.7746 versus 0.7860 on test data (0.0114 impact).

Evaluation Results
The official results are presented in Table 3. Our system (MayAnd) official result differs from result presented in Table 2 due to regrettable inadvertence with training dataset. Our system was trained on ini-tially published data instead of revised dataset that had been published later.

Conclusion
Aspect-based sentiment analysis task and polarity detection subtask are still at their early days. Current methods are not able to significantly overcome trivial baselines. The following directions of future work can be recommended: collecting more reviews for various languages and domains to improve quality of unsupervised techniques like word2vec; development of sentiment lexicons for non-English languages and various domains.