Position-Aware Tagging for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) is the task of extracting the triplets of target entities, their associated sentiment, and opinion spans explaining the reason for the sentiment. Existing research efforts mostly solve this problem using pipeline approaches, which break the triplet extraction process into several stages. Our observation is that the three elements within a triplet are highly related to each other, and this motivates us to build a joint model to extract such triplets using a sequence tagging approach. However, how to effectively design a tagging approach to extract the triplets that can capture the rich interactions among the elements is a challenging research question. In this work, we propose the first end-to-end model with a novel position-aware tagging scheme that is capable of jointly extracting the triplets. Our experimental results on several existing datasets show that jointly capturing elements in the triplet using our approach leads to improved performance over the existing approaches. We also conducted extensive experiments to investigate the model effectiveness and robustness.


Introduction
Designing effective algorithms that are capable of automatically performing sentiment analysis and opinion mining is a challenging and important task in the field of natural language processing (Pang and Lee, 2008;Liu, 2010;Ortigosa et al., 2014;Smailović et al., 2013;Li and Wu, 2010). Recently, Aspect-based Sentiment Analysis (Pon- * Equal contribution. Lu Xu is under the Joint PhD Program between Alibaba and Singapore University of Technology and Design. The work was done when Hao Li was a PhD student in Singapore University of Technology and Design. Accepted as a long paper in EMNLP 2020 (Conference on Empirical Methods in Natural Language Processing). 1 We release our code at https://github.com/ xuuuluuu/Position-Aware-Tagging-for-ASTE 0 + food was so so but excited to see many vegan options Figure 1: ASTE with targets in bold in solid squares, their associated sentiment on top, and opinion spans in dashed boxes. The arc indicates connection between a target and the corresponding opinion span. tiki et al., 2014) or Targeted Sentiment Analysis (Mitchell et al., 2013) which focuses on extracting target phrases as well as the sentiment associated with each target, has been receiving much attention. In this work, we focus on a relatively new task -Aspect Sentiment Triplet Extraction (ASTE) proposed by Peng et al. (2019). Such a task is required to extract not only the targets and the sentiment mentioned above, but also the corresponding opinion spans expressing the sentiment for each target. Such three elements: a target, its sentiment and the corresponding opinion span, form a triplet to be extracted. Figure 1 presents an example sentence containing two targets in solid boxes. Each target is associated with a sentiment, where we use + to denote the positive polarity, 0 for neutral, and − for negative. Two opinion spans in dashed boxes are connected to their targets by arcs. Such opinion spans are important, since they largely explain the sentiment polarities for the corresponding targets (Qiu et al., 2011;Yang and Cardie, 2012).
This ASTE problem was basically untouched before, and the only existing work that we are aware of (Peng et al., 2019) employs a 2-stage pipeline approach. At the first stage, they employ a unified tagging scheme which fuses the target tag based on the BIOES 2 tagging scheme, and sentiment tag together. Under such a uni-0 + food was so so but excited to see many vegan options fied tagging scheme, they proposed methods based on Long Short-Term Memory networks (LSTM) (Hochreiter and Schmidhuber, 1997), Conditional Random Fields (CRF) (Lafferty et al., 2001) and Graph Convolutional Networks (GCN) (Kipf and Welling, 2017) to perform sequence labeling to extract targets with sentiment as well as opinion spans. At the second stage, they use a classifier based on Multi-Layer Perceptron (MLP) to pair each target (containing a sentiment label) with the corresponding opinion span to obtain all the valid triplets.
One important observation is that the three elements in a triplet are highly related to each other. Specifically, sentiment polarity is largely determined by an opinion span as well as the target and its context, and an opinion span also depends on the target phrase in terms of wording (e.g., an opinion span "fresh" usually describes food targets instead of service). Such an observation implies that jointly capturing the rich interaction among three elements in a triplet might be a more effective approach. However, the BIOES tagging scheme on which the existing approaches based comes with a severe limitation for this task: such a tagging scheme without encoding any positional information fails to specify the connection between a target and its opinion span as well as the rich interactions among the three elements due to the limited expressiveness. Specifically, BIOES uses the tag B or S to represent the beginning of a target. For example, in the example sentence in Figure 1, "vegan" should be labeled with B, but the tagging scheme does not contain any information to specify the position of its corresponding opinion "excited". Using such a tagging scheme inevitably leads to an additional step to connect each target with an opinion span as the second stage in the pipeline approach. The skip-chain sequence models (Sutton and McCallum, 2004;Galley, 2006) are able to capture interactions between given input tokens which can be far away from each other. However, they are not suitable for the ASTE task where the positions of targets and opinion spans are not explicitly provided but need to be learned.
Motivated by the above observations, we present a novel approach that is capable of predicting the triplets jointly for ASTE. Specifically, we make the following contributions in this work: • We present a novel position-aware tagging scheme that is capable of specifying the structural information for a triplet -the connection among the three elements by enriching the label semantics with more expressiveness, to address the above limitation. • We propose a novel approach, JET, to Jointly Extract the Triplets based on our novel position-aware tagging scheme. Such an approach is capable of better capturing interactions among elements in a triplet by computing factorized features for the structural information in the ASTE task. • Through extensive experiments, the results show that our joint approach JET outperforms baselines significantly.

Our Approach
Our objective is to design a model JET to extract the triplet of Target, Target Sentiment, and Opinion Span jointly. We first introduce the new position-aware tagging scheme, followed by the model architecture. We next present our simple LSTM-based neural architecture for learning feature representations, followed by our method to calculate factorized feature scores based on such feature representations for better capturing the interactions among elements in a triplet. Finally, we also discuss a variant of our model.

Position-Aware Tagging Scheme
To address the limitations mentioned above, we propose our position-aware tagging scheme by enriching expressiveness to incorporate position in-formation for a target and the corresponding opinion span. Specifically, we extend the tag B and tag S in the BIOES tagging scheme to new tags respectively: where B ǫ j,k with the sub-tag 3 B still denotes the beginning of a target, and S ǫ j,k with the sub-tag S denotes a single-word target. Note that ǫ ∈ {+, 0, −} denotes the sentiment polarity for the target, and j, k indicate the position information which are the distances between the two ends of an opinion span and the starting position of a target respectively. Here, we use the term "offset" to denote such position information for convenience. We keep the other tags I, E, O as is. In a word, we use sub-tags BIOES for encoding targets, ǫ for sentiment, and offsets for opinion spans under the new position-aware tagging scheme for the structural information.
For the example in Figure 1, under the proposed tagging scheme, the tagging result is given in Figure 2. The single-word target "food" is tagged with S 0 2,3 , implying the sentiment polarity for this target is neutral (0). Furthermore, the positive offsets 2, 3 indicate that its opinion span is on the right and has distances of 2 and 3 measured at the left and right ends respectively, (i.e., "so so"). The second target is "vegan options" with its first word tagged with B + −4,−4 and the last word tagged with E, implying the sentiment polarity is positive (+). Furthermore, the negative offsets −4, −4 indicate that the opinion span "excited" appears on the left of the target, and has distances of 4 and 4 measured at the left and right ends respectively, (i.e., "vegan").
Our proposed position-aware tagging scheme has the following theoretical property: Theorem 2.1. There is a one-to-one correspondence between a tag sequence and a combination of aspect sentiment triplets within the sentence as long as the targets do not overlap with one another, and each has one corresponding opinion span. 4 Proof. For a given triplet, we can use the following process to construct the tag sequence. First note that the sub-tags of our proposed tags B ǫ j,k , I, O, E, S ǫ j,k , are B, I, O, E, S. The standard BIOES tagset is capable of extracting all possible targets when they do not overlap with one another. Next, for each specified target, the position information j, k that specifies the position of its corresponding opinion span can be attached to the B (or S) tag, resulting in B j,k (or S j,k ). Note that the opinion span can be any span within the sentence when j, k are not constrained. Finally, we assign each extracted target its sentiment polarity ǫ by attaching it to the tag B (or S), resulting in B ǫ j,k (or S ǫ j,k ). This construction process is unique for each combination of triplets. Similarly, given a tag sequence, we can reverse the above process to recover the combination of triplets.
We would like to highlight that our proposed position-aware tagging scheme is capable of handling some special cases where the previous approach is unable to. For example, in the sentence "The salad is cheap with fresh salmon", there are two triplets, ("salad", "cheap with fresh salmon", positive) 5 and ("salmon", "fresh", positive). The previous approach such as (Peng et al., 2019), which was based on a different tagging scheme, will not be able to handle such a case where the two opinion spans overlap with one another.

Our JET Model
We design our novel JET model with CRF (Lafferty et al., 2001) and Semi-Markov CRF (Sarawagi and Cohen, 2004) based on our position-aware tagging scheme. Such a model is capable of encoding and factorizing both token-level features for targets and segment-level features for opinion spans.
Given a sentence x with length n, we aim to produce the desired output sequence y based on the position-aware tagging scheme. The probability of y is defined as: where s(x, y) is a score function defined over the sentence x and the output structure y, and Y x,M represents all the possible sequences under our position-aware tagging scheme with the offset constraint M , indicating the maximum absolute value of an offset. The score s(x, y) is defined as: whereȳ i ∈ {B, I, O, E, S} returns the sub-tag Figure 3: Neural Module for Feature Score of y i , ψȳ i ,ȳ i+1 represents the transition score: the weight of a "transition feature" -a feature defined over two adjacent sub-tagsȳ i andȳ i+1 , and Φ y i (x, i) represents the factorized feature score with tag y i at position i. In our model, the calculation of transition score ψȳ i ,ȳ i+1 is similar to the one in CRF 6 . For the factorized feature score Φ y i (x, i), we will explain computation details based on a simple LSTM-based neural network in the following two subsections. Such a factorized feature score is able to encode both token-level features as in standard CRF, segmentlevel features as in Semi-Markov CRF as well as the interaction among a target, its sentiment and an opinion span in a triplet.

Neural Module
We deploy a simple LSTM-based neural architecture for learning features. Given an input token sequence x = {x 1 , x 2 , · · · , x n } of length n, we first obtain the embedding sequence {e 1 , e 2 , · · · , e n }. As illustrated in Figure 3, we then apply a bidirectional LSTM on the embedding sequence and obtain the hidden state h i for each position i, which could be represented as: where − → h i and ← − h i are the hidden states of the forward and backward LSTMs respectively.
Motivated by (Wang and Chang, 2016;Stern et al., 2017), we calculate the segment representation g a,b for an opinion span with boundaries of a and b (both inclusive) as follows: where

Factorized Feature Score
We explain how to compute the factorized feature scores (the second part of Equation 2) for the position-aware tagging scheme based on the neural architecture described above. Such factorized feature scores involve 4 types of scores, as illustrated in the solid boxes appearing in Figure 3 (top). Basically, we calculate the factorized feature score for the tag y i as follows: where the linear layer f t is used to calculate the score for local context for targets. Such a linear layer takes the hidden state h i as the input and returns a vector of length 5, with each value in the vector indicating the score of the corresponding sub-tag among BIOES. The subscriptȳ i indicates the index of such a sub-tag. When y i ∈ {B ǫ j,k , S ǫ j,k }, we need to calculate 3 additional factorized feature scores for capturing structural information by adding them to the basic score as follows: Note that the subscript of the variable g is represented as i + j, i + k which are the absolute positions since j, k are the offsets. We explain such 3 additional factorized scores appearing in Equation 6.
• f s ([g i+j,i+k ; ← − h i ]) ǫ calculates the score for the sentiment. A linear layer f s takes the concatenation of the segment representation g i+j,i+k for an opinion span and the local context ← − h i for a target, since we believe that the sentiment is mainly determined by the opinion span as well as the target phrase itself. Note that we only use the backward hidden state ← − h i here, because the end position of a target is not available in the tag and the target phrase appears on the right of this position i. The linear layer f s returns a vector of length 3, with each value representing the score of a certain polarity of +, 0, −. The subscript ǫ indicates the index of such a polarity.
• f o (g i+j,i+k ) is used to calculate a score for an opinion span. A linear layer f o takes the segment representation g i+j,i+k of an opinion span and returns one number representing the score of an opinion span.   Figure 4: The gold tagging sequence of JET o for the example sentence.
• f r (j, k) is used to calculate a score for offsets, since we believe the offset is an important feature. A linear layer f r returns one number representing the score of offsets j, k which again are the distances between a target and two ends of the opinion span. Here, we introduce the offset embedding w r randomly initialized for encoding different offsets. Specifically, we calculate the score as follows 7 : where W r and b r are learnable parameters.

One Target for Multiple Opinion Spans
The approach JET described above allows multiple targets to point to the same opinion span. One potential issue is that such an approach is not able to handle the case where one target is associated with multiple opinion spans. To remedy such an issue, we could swap a target and an opinion span to arrive at a new model as a model variant, since they are both text spans which are characterized by their boundaries. Specifically, in such a model variant, we still use the extended tags B ǫ j,k and S ǫ j,k , where we use sub-tags BIOES to encode an opinion span, the offsets j, k for the target and ǫ for the sentiment polarity. We use a similar procedure for the feature score calculation.
To differentiate with our first model, we name our first model as JET t and such a model variant as JET o . The superscripts t and o indicate the use of the sub-tags B and S to encode a target and an opinion span respectively. Figure 4 presents the gold tagging sequence of JET o . 7 We use min (j, k) since we care the offset between the starting positions of an opinion span and a target.

Training and Inference
The loss function L for the training data D is defined as: The overall model is analogous to that of a neural CRF (Peng et al., 2009;Do et al., 2010;Lample et al., 2016); hence the inference and decoding follow standard marginal and MAP inference 8 procedures. For example, the prediction of y follows the Viterbi-like MAP inference procedure during decoding. Notice that the number of labels at each position under the position-aware tagging scheme is O(M 2 ), since we need to compute segment representation for text spans of lengths within M . Hence, the time complexity for inference is O(nM 2 ). When M ≪ n (empirically, we found n can be up to 80 in our datasets, and we set M ∈ [2, 6]), this complexity is better than the existing work with complexity O(n 2 ) (Peng et al., 2019).

Data
We refine the dataset previously created by Peng et al. (2019) 9 . We call our refined dataset ASTE-Data-V2, and the original version as ASTE-Data-V1 10 . Note that ASTE-Data-V1 does not contain cases where one opinion span is associated with multiple targets. For example, there are two targets, "service" and "atmosphere", in the sentence "Best service and atmosphere". The opinion span "Best" is associated with such two targets, resulting in two triplets. However, we found that not all such triplets are explicitly annotated in ASTE-Data-V1. We refine the dataset with these additional missing triplets in our dataset ASTE-Data-V2 11 . Table 1 presents the detailed statistics for 4 datasets. 12 14Rest, 15Rest, 16Rest are the datasets of restaurant domain and 14Lap is of laptop domain. Such datasets were all created based on the datasets originally released by Se-mEval (Pontiki et al., 2014(Pontiki et al., , 2015(Pontiki et al., , 2016.

Baselines
Our JET approaches are compared with the following baselines using pipeline.
• RINANTE+ ( (2019) proposed an approach motivated by Li-unified-R to co-extract targets with sentiment, and opinion spans simultaneously. Such an approach also fuses GCN to capture dependency information to facilitate the co-extraction. At the second stage, it uses the same method to obtain all the valid triplets as RINANTE+.

Experimental Setup
Following the previous work (Peng et al., 2019), we use pre-trained 300d GloVe (Pennington et al., 2014) to initialize the word embeddings. We use 100 as the embedding size of w r (offset embedding). We use the bi-directional LSTM with the hidden size 300. For experiments with contextualised representation, we adopt the pre-trained language model BERT (Devlin et al., 2019). Specifically, we use bert-as-service (Xiao, 2018) to generate the contextualized word embedding without fine-tuning. We use the representation from the last layer of the uncased version of BERT base model for our experiments. Before training, we discard any instance from the training data that contains triplets with offset larger than M . We train our model for a maximal of 20 epochs using Adam (Kingma and Ba, 2014) as the optimizer with batch size 1 and dropout rate 0.5 13 . We select the best model parameters based on the best F 1 score on the development data and apply it to the test data for evaluation.
Following the previous works, we report the precision (P.), recall (R.) and F 1 scores for the correct triplets. Note that a correct triplet requires the boundary 14 of the target, the boundary of the opinion span, and the target sentiment polarity to be all correct at the same time. Table 2 presents the main results, where all the baselines as well as our models with different maximum offsets M are listed. In general, our joint models JET t and JET o , which are selected based on the best F 1 score on the dev set, are able to outperform the most competitive baseline of Peng et al. (2019) on the 4 datasets 14Rest, 15Rest, 16Rest, and 14Lap. Specifically, the best models selected from JET t and JET o outperform Peng et al. (2019) significantly 15 on 14Rest and 16Rest datasets with p < 10 −5 respectively. Such results imply that our joint models JET t and JET o are more capable of capturing interactions among the elements in triplets than those pipeline approaches. In addition, we observe a general trend from the results that the F 1 score increases as M increases on the 4 datasets when M ≤ 5. We observe that the performance of JET t and JET o on the dev set of 14Lap drops when M = 6.   also achieves a good performance with 56.58 in terms of F 1 score. Comparing results of our models to baselines, the reason why ours have better F 1 scores is that our models JET t (M ≥ 4) and JET o (M ≥ 4) both achieve improvements of more than 15 precision points, while we maintain acceptable recall scores. Similar patterns of results on the datasets 14Lap, 15Rest and 16Rest are observed, except that JET t (M = 5) and JET o (M = 5) achieves the best F 1 score on the dev set of 14Lap. Furthermore, we discover that the performance of both JET o and JET t on 14Rest and 16Rest datasets is better than on14Lap and 15Rest datasets. Such a behavior can be explained by the large distribution differences of positive, neutral and negative sentiment between the train and test set of the 14Rest and 16Rest datasets, shown in Table 1.

Main Results
Furthermore, we also conduct additional experiments on our proposed model with the contextualized word representation BERT. Both JET t (M = 6) + BERT and JET o (M = 6) + BERT achieve new stateof-the-art performance on the four datasets.

Robustness Analysis
We analyze the model robustness by assessing the performance on targets, opinion spans and offsets of different lengths for two models JET t (M =  Figure 5 shows the results on the 14Rest dataset 16 . As we can see, JET o (M = 6) + BERT is able to better extract triplets with targets of lengths ≤ 3 than JET t (M = 6) + BERT . Furthermore, JET o (M = 6) + BERT achieves a better F 1 score for triplets whose opinion spans are of length 1 and 4. However, JET o (M = 6) + BERT performs comparably to JET t (M = 6) + BERT for triplets whose opinion spans are of length 2 and 3. In addition, JET o (M = 6) + BERT is able to outperform JET t (M = 6) + BERT with offset of length 4 and above. We also observe that the performance drops when the lengths of targets, opinion spans and offsets are longer. This confirms that modeling the boundaries are harder when their lengths are longer. Similar patterns of results are observed on 14Lap, 15Rest, and 16Rest 17 . We also investigate the robustness on different evaluation methods, as presented in Figure 6. T Gold Peng et al. (2019) JET t JET o + Food is fresh and hot ready to eat + Food is fresh and hot ready to eat + Food is fresh and hot ready to eat + Food is fresh and hot ready to eat 0 + with a quaint bar and good food 0 + with a quaint bar and good food 0 + with a quaint bar and good food 0 + with a quaint bar and good food

Qualitative Analysis
To help us better understand the differences among these models, we present two example sentences selected from the test data as well as predictions by Peng et al. (2019), JET t and JET o in Table 3   an incorrect opinion span "hot ready" in the second triplet. JET t only predicts 1 triplet due to the model's limitation (JET t is not able to handle the case of one target connecting to multiple opinion spans). JET o is able to predict 2 triplets correctly. In the second example, the gold data contains two triplets. Peng et al. (2019) is able to correctly predict all the targets and opinion spans. However, it incorrectly connects each target to both two opinion spans. Our joint models JET t and JET o are both able to make the correct prediction.

Ablation Study
We also conduct an ablation study for JET t (M = 6) + BERT and JET o (M = 6) + BERT on dev set of the 4 datasets, presented in Table 4. "+char embedding" denotes concatenating character embedding into word representation. The results show that concatenating character embedding mostly has no much positive impact on the performance, which we believe is due to data sparsity. "−offset features" denotes removing f r (j, k) in the feature score calculation, Equation 6. F 1 scores drop more on the JET t (M = 6) + BERT , this further confirms that modeling the opinion span is more difficult than target. "−opinion features" denotes removing f o (g i+j,i+k ) in the feature score calculation in Equation 6. F 1 scores drop consistently, implying the importance of such features for opinion spans.

Ensemble Analysis
As mentioned earlier, JET o is proposed to overcome the limitation of JET t , and vice versa. We  Table 5: Results for Ensemble. We use the models JET t and JET o (with BERT, M = 6) as base models for building two ensemble models on 4 datasets. believe that such two models complement each other. Hence, we propose two ensemble models JET o→t and JET t→o to properly merge the results produced by JET t and JET o . JET o→t merges results of JET o towards JET t by adding distinct triplets from JET o to JET t , and analogously for JET t→o . We discuss how we build the ensemble models based on the two models JET t and JET o (with BERT, M = 6). First we call two triplets are overlap with one another if two targets overlap and any of their opinions overlap with one another. The ensemble model JET o→t merges results from JET o towards JET t . Specifically, within the same instance, if a triplet produced by JET o does not overlap with any triplet produced by JET t , we augment the prediction space with such an additional triplet. After going through each triplet produced by JET o , we regard the expanded predictions as the output of the ensemble model JET o→t . Similarly, we merge the result from JET t towards JET o to obtain the result for the ensemble model JET t→o .
We report results for ensemble models JET o→t and JET t→o presented in Table 5. As we can see, on 14Rest, 14Lap and 15Rest, the ensemble model JET t→o is able to achieve better F 1 score than JET t and JET o . However, such a simple ensemble approach appears to be less effective on 16Rest. It is worth highlighting that the ensemble models have significant improvements in terms of recall score. Note that the recall score reflects the number of gold triplets extracted. Such improvement confirms our earlier hypothesis that the two models largely complement each other.

Related Work
ASTE is highly related to another research topic -Aspect Based Sentiment Analysis (ABSA) (Pontiki et al., 2014(Pontiki et al., , 2016. Such a research topic focuses on identifying aspect categories, recognizing aspect targets as well as the associated sentiment. There exist a few tasks derived from ABSA. Target extraction (Chernyshevich, 2014;San Vicente et al., 2015;Yin et al., 2016;Lample et al., 2016;Li et al., 2018b;Ma et al., 2019) is a task that focuses on recognizing all the targets which are either aspect terms or named entities. Such a task is mostly regarded as a sequence labeling problem solvable by CRF-based methods. Aspect sentiment analysis or targeted sentiment analysis is another popular task. Such a task either refers to predicting sentiment polarity for a given target (Dong et al., 2014;Chen et al., 2017;Xue and Li, 2018;Wang and Lu, 2018;Li et al., 2018a;Peng et al., 2018;Xu et al., 2020) or joint extraction of targets as well as sentiment associated with each target (Mitchell et al., 2013;Zhang et al., 2015;Li and Lu, 2017;Ma et al., 2018;Li and Lu, 2019;Li et al., 2019). The former mostly relies on different neural networks such as self-attention (Liu and Zhang, 2017) or memory networks (Tang et al., 2016) to generate an opinion representation for a given target for further classification. The latter mostly regards the task as a sequence labeling problem by applying CRF-based approaches. Another related task -target and opinion span co-extraction (Qiu et al., 2011;Liu et al., 2013Liu et al., , 2014Liu et al., , 2015Wang et al., 2017;Xu et al., 2018;Dai and Song, 2019) is also often regarded as a sequence labeling problem.

Conclusion
In this work, we propose a novel position-aware tagging scheme by enriching label expressiveness to address a limitation associated with existing works. Such a tagging scheme is able to specify the connection among three elements -a target, the target sentiment as well as an opinion span in an aspect sentiment triplet for the ASTE task. Based on the position-aware tagging scheme, we propose a novel approach JET that is capable of jointly extracting the aspect sentiment triplets. We also design factorized feature representations so as to effectively capture the interaction. We conduct extensive experiments and results show that our models outperform strong baselines significantly with detailed analysis. Future work includes finding applications of our novel tagging scheme in other tasks involving extracting triplets as well as extending our approach to support other tasks within sentiment analysis.

A More Data Statistics
We present the statistics of accumulative percentage of different lengths for targets, opinion spans and offsets in the training data on 4 datasets 14Rest, 14Rest, 15Rest and 16Rest in Figure 7. As we mentioned in the main paper, similar patterns are observed on accumulative statistics on these 4 datasets. We also present the statistics of the number of targets with a single opinion span and with multiple opinion spans, and the number of opinion associated with a single target span and with multiple target spans, shown in Table 6.

B Experimental Details
We test our model on Intel ( 15Rest and 16Rest datasets respectively when M = 6. The total number of parameters is 2.5M. For hyper-parameter, we use pre-trained 300d GloVe (Pennington et al., 2014) to initialize the word embeddings. We use 100 as the embedding size of w r (offset embedding). For out-ofvocabulary words as well as w r , we randomly sample their embeddings from the uniform distribution U (−0.1, 0.1), as done in (Kim, 2014). We use the bi-directional LSTM with the hidden size 300. We train our model for a maximal of 20 epochs using Adam (Kingma and Ba, 2014) as the optimizer with batch size 1 and dropout rate 0.5 for datasets in restaurant domain and 0.7 for laptop domain. We manually tune the dropout rate from 0.4 to 0.7, and select the best model parameters based on the best F 1 score on the development data and apply it to the test data for evaluation. For experiments with contextualised representation, we adopt the pre-trained language model BERT (Devlin et al., 2019). Specifically, we use bert-as-service (Xiao, 2018) to generate the contextualized word embedding without fine-tuning. We use the representation from the last layer of the uncased version of BERT base model for our experiments. Table 7 presents the experimental result on the previous released dataset by (Peng et al., 2019).

D Decoding based on Viterbi
Let T = {B ǫ j,k , S ǫ j,k , I, E, O} as the new tag set under our position-aware tagging scheme, where ǫ denotes the sentiment polarity for the target, and j, k indicate the position information which are the distances between the two ends of an opinion span and the starting position of a target respectively.
We define the sub-tags of B ǫ j,k , S ǫ j,k as B and S respectively, and the sub-tags of I, O, E as themselves. We use the bar on top to denote the sub-tag. For example,ū is the subtag of u ∈ T .
We use π(i, v) to denote the score for the optimal sequence {y * 1 · · · y * i } among all the possible sequences whose last tag is v.
Given the input x of length n, we aim to obtain the optimal sequence y * = {y * 1 · · · y * n }.  • Base Case for all the v ∈ T If v ∈ {I, E, O}: 1+k ), and f r (j, k) are the factorized feature score mentioned in the section 2.2.2.
• Loop forward for i ∈ {2, · · · , n} and all the v ∈ T If v ∈ {I, E, O}: Note that ST ART appears before the start of the input sentence and ST OP appears after the end of the input sentence.

E Analysis E.1 Robustness Analysis
We present the performance on targets, opinion spans and offsets of different lengths for two models JET t (M = 6) and JET o (M = 6) with BERT on 3 datasets 14Lap,15Rest and 16Rest in Figure 8, Figure 9 and Figure 10 respectively.

E.2 Qualitative Analysis
We present one additional example sentence selected from the test data as well as predictions by Peng et al. (2019), JET t and JET o in Table 8.
As we can see, the gold data contains two triplets. Peng et al. (2019) only predicts 1 opinion span, and therefore incorrectly assigns the opinion span "Good" to the target "price". JET t is able to make the correct predictions. JET o only predicts 1 triplet correctly. The qualitative analysis helps us to better understand the differences among these models.

F More Related Work
The task of joint entity and relation extraction is also related to joint triplet extraction. Different from our task, such a relation extraction task aims to extract a pair of entities (instead of a target and an opinion span) and their relation as a triplet in a joint manner. Miwa and Sasaki (2014) and Li and Ji (2014) used approaches motivated by a tablefilling method to jointly extract entity pairs as well as their relations. The tree-structured neural networks (Miwa and Bansal, 2016) and CRF-based approaches (Adel and Schütze, 2017) were also adopted to capture rich context information for triplet extraction. Recently, Bekoulis et al. (2018) used adversarial training (Goodfellow et al., 2015) for this task and results show that it performs more robustly in different domains. Although these approaches may not be applied to our task ASTE, they may provide inspirations for future work.
Algorithm 1 Decoding based on Viterbi Backward for the optimal sequence y * = {y * 1 · · · y * n } for i ∈ {n, · · · , 1} do if i = n then y * n = arg max v∈T {π(n, v) + ψv ,ST OP } end else y * i = arg max v∈T {π(i, v) + ψv ,ȳ * i+1 } end end Gold Peng et al. (2019) JET t JET o 0 + Good food at the right price , 0 + Good food at the right price , 0 + Good food at the right price , 0 Good food at the right price ,