Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization

There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition. However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption. To address these concerns, we present a regularization method based on tensor rank minimization. Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations. However, the presence of noise or incomplete values breaks these correlations and results in tensor representations of higher rank. We design a model to learn such tensor representations and effectively regularize their rank. Experiments on multimodal language data show that our model achieves good results across various levels of imperfection.


Introduction
Analyzing multimodal language sequences spans various fields including multimodal dialog (Das et al., 2017;Rudnicky, 2005), question answering (Antol et al., 2015;Tapaswi et al., 2015;Das et al., 2018), sentiment analysis (Morency et al., 2011), and speech recognition (Palaskar et al., 2018).Generally, these multimodal sequences contain heterogeneous sources of information across the language, visual and acoustic modalities.For example, when instructing robots, these machines have to comprehend our verbal instructions and interpret our nonverbal behaviors while grounding these inputs in their visual sensors (Schmerling et al., 2017;Iba et al., 2005).Likewise, comprehending human intentions requires integrating human language, speech, * first two authors contributed equally clean multimodal data < l a t e x i t s h a 1 _ b a s e 6 4 = " h 1 p z H R z B A J 7 X L u y 3 k D 5 g z y U F T l U k U K o l B 0 h g Q B u 1 y x a / 6 I 9 B p E h S k Q g p c t s t f r T h h m e I a m Q R r m 4 G f Y p i D Q e E 2 D E q t z P I U 2 B 1 0 e d N R D Y r b M B 9 9 M a D 7 T o l p J z G u N N K R + n s i B 2 V t X 0 W u U w H 2 7 K Q 3 F P / z m h l 2 T s N c 6 D R D r t l 4 U S e T F B M 6 j I T G w n C G s u 8 I M C P c r Z T 1 w A B D F 1 z J h R B M v j x N G o f V 4 K j q X x 1 X a m d F H E t k h + y R A x K Q E 1 I j F + S S 1 A k j j + S Z v J I 3 7 8 l 7 8 d 6 9 j 3 H r j F f M b J M / 8 D 5 / A B 0 o m P M = < / l a t e x i t > imperfect multimodal data < l a t e x i t s h a 1 _ b a s e 6 4 = " I 3 w m M C k k o G + g n 0 8 p u Q Z q e M 8 e R b Q = " > A A A C C X i c b V C 7 S g N B F J 3 1 G e M r a m k z G A S r s K u C l k E b y w h G h S S E u 7 N 3 4 + D M 7 j J z V w x L W h t / x c Z C E V v / w M 6 / c f I o 1 H h g 4 H D O v d w 5 J 8 y U t O T 7 X 9 7 M 7 N z 8 w m J p q b y 8 s r q 2 X t n Y v L R p b g Q 2 R a p S c x 2 C R S U T b J I k h d e Z Q d C h w q v w 9 n T o X 9 2 h s T J N L q i f Y U d D L 5 G x F E B O 6 l Z 4 m / C e C q k z N D E K 4 j p X J H U a g e I R E A y 6 l a p f 8 0 f g 0 y S Y k C q b o N G t f L a j V O Q a E x I K r G 0 F f k a d A g x J o X B Q b u c W M x C 3 0 M O W o w l o t J 1 i l G T A d 5 0 S 8 T g 1 7 i X E R + r P j Q K 0 t X 0 d u k k N d G P / e k P x P 6 + V U 3 z c K W S S 5 Y S J G B + K c 8 U p 5 c N a e C S N S 6 / 6 j o A w 0 v 2 V i x s w I M i V V 3 Y l B H 8 j T 5 P L / V p w U P P P D 6 v 1 k 0 k d J b b N d t g e C 9 g R q 7 M z 1 m B N J t g D e 2 I v 7 N V 7 9 J 6 9 N + 9 9 P D r j T X a 2 2 C 9 4 H 9 + G K Z r X < / l a t e x i t > tensor rank regularization Figure 1: Clean multimodal time series data (in shades of green) exhibits correlations across time and across modalities, leading to redundancy in low rank tensor representations.On the other hand, the presence of imperfect entries (in gray, blue, and red) breaks these correlations and leads to higher rank tensors.In these scenarios, we use tensor rank regularization to learn tensors that more accurately represent the true correlations and latent structures in multimodal data.facial behaviors, and body postures (Mihalcea, 2012;Rossiter, 2011).However, as much as more modalities are required for improved performance, we now face a challenge of imperfect data where data might be 1) incomplete due to mismatched modalities or sensor failure, or 2) corrupted with random or structured noise.As a result, an important research question involves learning robust representations from imperfect multimodal data.
Recent research in both unimodal and multimodal learning has investigated the use of tensors for representation learning (Anandkumar et al., 2014).Given representations h 1 , ..., h M from M modalities, the order-M outer product tensor T = h 1 ⊗ h 2 ⊗ ... ⊗ h M is a natural representation for all possible interactions between the modality dimensions (Liu et al., 2018).In this paper, we propose a model called the Temporal Tensor Fusion Network (T2FN) that builds tensor representations from multimodal time series data.T2FN arXiv:1907.01011v1[cs.LG] 1 Jul 2019 learns a tensor representation that captures multimodal interactions across time.A key observation is that clean data exhibits tensors that are lowrank since high-dimensional real-world data is often generated from lower dimensional latent structures (Lakshmanan et al., 2015).Furthermore, clean multimodal time series data exhibits correlations across time and across modalities (Yang et al., 2017;Hidaka and Yu, 2010).This leads to redundancy in these overparametrized tensors which explains their low rank (Figure 1).On the other hand, the presence of noise or incomplete values breaks these natural correlations and leads to higher rank tensor representations.As a result, we can use tensor rank minimization to learn tensors that more accurately represent the true correlations and latent structures in multimodal data, thereby alleviating imperfection in the input.With these insights, we show how to integrate tensor rank minimization as a simple regularizer for training in the presence of imperfect data.As compared to previous work on imperfect data (Sohn et al., 2014;Srivastava and Salakhutdinov, 2014;Pham et al., 2019), our model does not need to know which of the entries or modalities are imperfect beforehand.Our model combines the strength of temporal non-linear transformations of multimodal data with a simple regularization technique on tensor structures.We perform experiments on multimodal video data consisting of humans expressing their opinions using a combination of language and nonverbal behaviors.Our results back up our intuitions that imperfect data increases tensor rank.Finally, we show that our model achieves good results across various levels of imperfection.

Related Work
Tensor Methods: Tensor representations have been used for learning discriminative representations in unimodal and multimodal tasks.Tensors are powerful because they can capture important higher order interactions across time, feature dimensions, and multiple modalities (Kossaifi et al., 2017).For unimodal tasks, tensors have been used for part-of-speech tagging (Srikumar and Manning, 2014), dependency parsing (Lei et al., 2014), word segmentation (Pei et al., 2014), question answering (Qiu and Huang, 2015), and machine translation (Setiawan et al., 2015).For multimodal tasks, Huang et al. (2017) used tensor products between images and text features for image captioning.A similar approach was proposed to learn representations across text, visual, and acoustic features to infer speaker sentiment (Liu et al., 2018;Zadeh et al., 2017).Other applications include multimodal machine translation (Delbrouck and Dupont, 2017), audio-visual speech recognition (Zhang et al., 2017), and video semantic analysis (Wu et al., 2009;Gao et al., 2009).
Imperfect Data: In order to account for imperfect data, several works have proposed generative approaches for multimodal data (Sohn et al., 2014;Srivastava and Salakhutdinov, 2014).Recently, neural models such as cascaded residual autoencoders (Tran et al., 2017), deep adversarial learning (Cai et al., 2018), or translation-based learning (Pham et al., 2019) have also been proposed.However, these methods often require knowing which of the entries or modalities are imperfect beforehand.While there has been some work on using low-rank tensor representations for imperfect data (Chang et al., 2017;Fan et al., 2017;Chen et al., 2017;Long et al., 2018;Nimishakavi et al., 2018), our approach is the first to integrate rank minimization with neural networks for multimodal language data, thereby combining the strength of non-linear transformations with the mathematical foundations of tensor structures.

Proposed Method
In this section, we present our method for learning representations from imperfect human language across the language, visual, and acoustic modalities.In §3.1, we discuss some background on tensor ranks.We outline our method for learning tensor representations via a model called Temporal Tensor Fusion Network (T2FN) in §3.2.In §3.3, we investigate the relationship between tensor rank and imperfect data.Finally, in §3.4,we show how to regularize our model using tensor rank minimization.We use lowercase letters x ∈ R to denote scalars, boldface lowercase letters x ∈ R d to denote vectors, and boldface capital letters X ∈ R d 1 ×d 2 to denote matrices.Tensors, which we denote by calligraphic letters X , are generalizations of matrices to multidimensional arrays.An order-M tensor has M dimensions, X ∈ R d 1 ×...×d M .We use ⊗ to denote outer product between vectors.

Background: Tensor Rank
The rank of a tensor measures how many vectors are required to reconstruct the tensor.Simple tensors that can be represented as outer products of < l a t e x i t s h a 1 _ b a s e 6 4 = " z c V G z 7 + t 4 Z I b w 5 R / j z r 8 x 0 8 5 C W w 8 E D u f c y z 0 5 Q c y Z N q 7 7 7 a y s r q 1 v b B a 2 i t s 7 u 3 v 7 p Y P D p p a J I r R B J J e q H W B

acoustic LSTM
< l a t e x i t s h a 1 _ b a s e 6 4 = " x 1 F k 1 e U X + I l 9 e X i 9 b o f g q l + E z X q y X q x 3 6 2 P S m r O m M 7 v o D 6 z P H 5 P / l U k = < / l a t e x i t > vectors have lower rank, while complex tensors have higher rank.To be more precise, we define the rank of a tensor using Canonical Polyadic (CP)-decomposition (Carroll and Chang, 1970).
For an order-M tensor X ∈ R d 1 ×...×d M , there exists an exact decomposition into vectors w: The minimal r for exact decomposition is called the rank of the tensor.The vectors {{w i m } M m=1 } r i=1 are called the rank r decomposition factors of X .

Multimodal Tensor Representations
Our model for creating tensor representations is called the Temporal Tensor Fusion Network (T2FN), which extends the model in Zadeh et al. (2017) to include a temporal component.We show that T2FN increases the capacity of TFN to capture high-rank tensor representations, which itself leads to improved prediction performance.More importantly, our knowledge about tensor rank properties allows us to regularize our model effectively for imperfect data.
We begin with time series data from the language, visual and acoustic modalities, denoted as [x 1 , ..., x T ], [x 1 v , ..., x T v ], and [x 1 a , ..., x T a ] respectively.We first use Long Short-term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) to encode the temporal information from each modality, resulting in a sequence of hidden representations [h 1 , ..., h T ], [h 1 v , ..., h T v ], and [h 1 a , ..., h T a ].Similar to prior work which found tensor representations to capture higher-order interactions from multimodal data (Liu et al., 2018;Zadeh et al., 2017;Fukui et al., 2016), we form tensors via outer products of the individual representations through time (as shown in Figure 2): where we append a 1 so that unimodal, bimodal, and trimodal interactions are all captured as described in Zadeh et al. (2017).M is our multimodal representation which can then be used to predict the label y using a fully connected layer.
Observe how the construction of M closely resembles equation ( 1) as the sum of vector outer products.As compared to TFN which uses a single outer product to obtain a multimodal tensor of rank one, T2FN creates a tensor of high rank (upper bounded by T ).As a result, the notion of rank naturally emerges when we reason about the properties of M.

How Does Imperfection Affect Rank?
We first state several observations about the rank of multimodal representation M: 1) r noisy : The rank of M is maximized when data entries are sampled from i.i.d.noise (e.g.Gaussian distributions).This is because this setting leads to no redundancy at all between the feature dimensions across time steps.
2) r clean < r noisy : Clean real-world data is often generated from lower dimensional latent structures (Lakshmanan et al., 2015).Furthermore, multimodal time series data exhibits correlations across time and across modalities (Yang et al., 2017;Hidaka and Yu, 2010).This redundancy leads to low-rank tensor representations.
3) r clean < r imperf ect < r noisy : If the data is imperfect, the presence of noise or incomplete values breaks these natural correlations and leads to higher rank tensor representations.These intuitions are also backed up by several experimental results which are presented in §4.2.

Tensor Rank Regularization
Given our intuitions above, it would then seem natural to augment the discriminative objective function with a term to minimize the rank of M.
In practice, the rank of an order-M tensor is computed using the nuclear norm X * which is defined as (Friedland and Lim, 2014), When M = 2, this reduces to the matrix nuclear norm (sum of singular values).However, computing the rank of a tensor or its nuclear norm is NP-hard for tensors of order ≥ 3 (Friedland and Lim, 2014).Fortunately, there exist efficiently computable upper bounds on the nuclear norm and minimizing these upper bounds would also minimize the nuclear norm M * .We choose the upper bound as presented in Hu (2014), which upper bounds the nuclear norm with the tensor Frobenius norm scaled by the tensor dimensions: where the Frobenius norm M F is defined as the sum of squared entries in M which is easily computable and convex.Since M F is easily computable and convex, including this term adds negligible computational cost to the model.We will use this upper bound as a surrogate for the nuclear norm in our objective function.Our objective function is therefore a weighted combination of the prediction loss and the tensor rank regularizer in equation ( 4).

Experiments
Our experiments are designed with two research questions in mind: 1) What is the effect of various levels of imperfect data on tensor rank in T2FN?
2) Does T2FN with rank regularization perform well on prediction with imperfect data?We answer these questions in §4.2 and §4.3 respectively.

Datasets
We experiment with real video data consisting of humans expressing their opinions using a combination of language and nonverbal behaviors.We use the CMU-MOSI dataset which contains 2199 videos annotated for sentiment in the range [−3, +3] (Zadeh et al., 2016).CMU-MOSI and related multimodal language datasets have been studied in the NLP community (Gu et al., 2018;Liu et al., 2018;Liang et al., 2018) from fully supervised settings but not from the perspective of supervised learning with imperfect data.We use 52 segments for training, 10 for validation and 31 for testing.GloVe word embeddings (Pennington et al., 2014), Facet (iMotions, 2017), and COVAREP (Degottex et al., 2014) features are extracted for the language, visual and acoustic modalities respectively.Forced alignment is performed using P2FA (Yuan and Liberman, 2008) to align visual and acoustic features to each word, resulting in a multimodal sequence.Our data splits, features, alignment, and preprocessing steps are consistent with prior work on the CMU-MOSI dataset (Liu et al., 2018).

Rank Analysis
We first study the effect of imperfect data on the rank of tensor M. We introduce the following types of noises parametrized by noise level = [0.0,0.1, ..., 1.0].Higher noise levels implies more imperfection: 1) clean: no imperfection, 2) random drop: each entry is dropped independently with probability p ∈ noise level, and 3) structured drop: independently for each modality, each time step is chosen with probability p ∈ noise level.If a time step is chosen, all feature dimensions at that time step are dropped.For all imperfect settings, features are dropped during both training and testing.We would like to show how the tensor ranks vary under different imperfection settings.However, as is mentioned above, determining the exact rank of a tensor is an NP-hard problem (Friedland and Lim, 2014).In order to analyze the effect of imperfections on tensor rank, we perform CP decomposition (equation ( 5)) on the tensor representations under different rank settings r and measure the reconstruction error , = min Given the true rank r * , will be high at ranks r < r * , while will be approximately zero at ranks r ≥ r * (for example, a rank 3 tensor would display a large reconstruction error with CP decomposition at rank 1, but would show almost zero error with CP decomposition at rank 3).By analyzing the effect of r on , we are then able to derive a surrogate r to the true rank r * .Using this approach, we experimented on CMU-MOSI and the results are shown in Figure 3(a).We observe that imperfection leads to an increase in (approximate) tensor rank as measured  by reconstruction error (the graph shifts outwards and to the right), supporting our hypothesis that imperfect data increases tensor rank ( §3.3).

Prediction Results
Our next experiment tests the ability of our model to learn robust representations despite data imperfections.We use the tensor M for prediction and report binary classification accuracy on CMU-MOSI test set.We compare to several baselines: Early Fusion (EF)-LSTM, Late Fusion (LF)-LSTM, TFN, and T2FN without rank regularization.These results are shown in Figure 3(b) for random drop and Figure 3(c) for structured drop.T2FN with rank regularization maintains good performance despite imperfections in data.
We also observe that our model's improvement is more significant on random drop settings, which results in a higher tensor rank as compared to structured drop settings (from Figure 3(a)).This supports our hypothesis that our model learns robust representations when imperfections that increase tensor rank are introduced.On the other hand, the existing baselines suffer in the presence of imperfect data.
5 Discussion and Future Work We acknowledge that there are other alternative methods to upper bound the true rank of a tensor (Alexeev et al., 2011;Atkinson and Lloyd, 1980;Ballico, 2014).From a theoretical perspective, there exists a trade-off between the cost of computation and the tightness of approximation.
In addition, the tensor rank can (far) exceed the maximum dimension, and a low-rank approximation for tensors may not even exist (de Silva and Lim, 2008).While our tensor rank regularization method seems to work well empirically, there is definitely room for a more thorough theoretical analysis of constructing and regularizing tensor representations for multimodal learning.

Conclusion
This paper presented a regularization method based on tensor rank minimization.We observe that clean multimodal sequences often exhibit correlations across time and modalities which leads to low-rank tensors, while the presence of imperfect data breaks these correlations and results in tensors of higher rank.We designed a model, the Temporal Tensor Fusion Network, to learn such tensor representations and effectively regularize their rank.Experiments on multimodal language data show that our model achieves good results across various levels of imperfections.We hope to inspire future work on regularizing tensor representations of multimodal data for robust prediction in the presence of imperfect data.
r H e r Y 9 p a 8 7 K Z n b R H 1 i f P 3 B 7 l T I = < / l a t e x i t > visual LSTM < l a t e x i t s h a 1 _ b a s e 6 4 = " P k d T A P 5

Figure 2 :
Figure 2: The Temporal Tensor Fusion Network (T2FN) creates a tensor M from temporal data.The rank of M increases with imperfection in data so we regularize our model by minimizing an upper bound on the rank of M.
CP decomposition error of M under random and structured dropping of features.Imperfect data leads to an increase in decomposition error and an increase in (approximate) tensor rank.(b)Sentiment classification accuracy under random drop (i.e.dropping entries randomly with probability p ∈ noise level).T2FN with rank regularization (green) performs well.(c)Sentiment classification accuracy under structured drop (dropping entire time steps randomly with probability p ∈ noise level).T2FN with rank regularization (green) performs well.

Figure 3 :
Figure 3: (a) Effect of imperfect data on tensor rank.(b) and (c): CMU-MOSI test accuracy under imperfect data.