Recently, a set of tensor norms known as coupled norms has been proposed as a convex solution to coupled tensor completion. Coupled norms have been designed by combining low-rank inducing tensor norms with the matrix trace norm. Though coupled norms have shown good performances, they have two major limitations: they do not have a method to control the regularization of coupled modes and uncoupled modes, and they are not optimal for couplings among higher-order tensors. In this letter, we propose a method that scales the regularization of coupled components against uncoupled components to properly induce the low-rankness on the coupled mode. We also propose coupled norms for higher-order tensors by combining the square norm to coupled norms. Using the excess risk-bound analysis, we demonstrate that our proposed methods lead to lower risk bounds compared to existing coupled norms. We demonstrate the robustness of our methods through simulation and real-data experiments.
In recent years, learning from multiple data sources has gained considerable interest. One such field that has gained interest is coupled tensor completion (also referred to as collective tensor completion), where we impute missing elements of a partially observed tensor by sharing information from its coupled tensors (Acar, Papalexakis et al., 2014; Acar, Nilsson, & Saunders, 2014; Acar, Bro, & Smilde, 2015; Bouchard, Yin, & Guo, 2013). A coupling between two tensors occurs when they share a common mode, where one tensor can be side information (Narita, Hayashi, Tomioka, & Kashima, 2011) to the other or both mutually share information (Acar, Papalexakis et al., 2014). Coupled tensor completion has been useful in several real-world applications such as link prediction (Ermis, Acar, & Cemgil, 2015), recommendation systems (Acar, Kolda, & Dunlavy, 2011; Acar, Papalexakis et al., 2014; Acar, Nilsson et al., 2014; Acar et al., 2015; Jeon, Jeon, Sael, & Kang, 2016), and computer vision (Li, Zhao, Li, Cichocki, & Guo, 2015; Zhou, Qian, Shen, Zhang, & Xu, 2017).
Recently, Wimalawarne, Yamada, and Mamitsuka (2018) proposed a set of norms known as coupled norms to solve coupled completion. One of the main advantages of these norms is that they are convex and lead to global solutions, while many of the existing coupled completion models are nonconvex factorization methods (Acar, Nilsson et al., 2014; Ermis et al., 2015). Furthermore, most factorization-based methods are restricted to the CANDECOMP/PARAFAC (CP) rank (Acar, Nilsson et al., 2014) of tensors while others are restricted to nonnegative factorization (Ermis et al., 2015). Coupled norms are able to learn using the multilinear rank of tensors and are applicable to heterogeneous tensor data. Theoretical analysis on completion with coupled norms has shown that proper regularization of low-rankness along the coupled modes leads to better performance compared to individual tensor completion. Except for the computational challenge of using trace norm regularization, these norms can be easily extended to multiple couplings of multiple tensors, thus making them a promising approach to coupled tensor learning.
Although coupled norms have several favorable qualities, we find that they have two limitations. One limitation is that there is no control on regularization of coupled modes with respect to uncoupled modes. In the existing design of coupled norms, there is always an assumption that all modes are low-ranked. This is not an optimal design since the the shared low-rankness induced by the concatenation of the tensors on the coupled modes could be different compared to low-rankness induced by the tensors independently. Another limitation with coupled norms is that their design is limited to coupling of three-mode tensors. A naive application of these coupled norms to higher-order tensors (tensors with more than three dimensions) may not be optimal since these norms have been designed using norms robust for three-mode tensors such as the overlapped trace norm (Liu, Musialski, Wonka, & Ye, 2009; Tomioka & Suzuki, 2013) and scaled latent trace norm (Wimalawarne, Sugiyama, & Tomioka, 2014). The recently proposed square norm (Mu, Huang, Wright, & Goldfarb, 2014) has been shown to be better for completion of higher-order tensors, which would be more suitable for coupled higher-order tensors.
In this letter, we propose extensions to coupled norms to overcome the limitations we have noted. We introduce scaling of the regularization on the coupled mode with respect to uncoupled modes. Additionally, we integrate the square norm to coupled tensors to create extensions for higher-order tensors. We derive excess risk bounds for coupled completion based on regularization by scaled coupled norms and their higher-order extensions. We provide details of simulation and real-data experiments and show that our methods lead to better performance for coupled completion.
Before we move on to the core concepts of our research, we introduce some notations that we use in this letter. Following Kolda and Bader (2009), we write a -mode tensor as , and its mode- unfolding is obtained by , which is obtained by concatenating all slices along mode-. Given two matrices and , the notation represents their concatenation on the common mode-1.
2 A Short Review on Completion with Coupled Norms
3 Limitations of Coupled Norms
Though coupled norms have favorable properties such as convexity and better performance in coupled tensor completion (Wimalawarne et al., 2018) compared to individual tensor completion, they are not optimal for coupled tensor completion. We identify two major limitations with coupled norms.
3.1 Lack of Control on Shared Low-Rankness
The basic design principle of coupled norms is to combine two tensor norms by having a single trace norm regularization on the concatenated unfolding of tensors along the coupled mode. The underlying assumption with this formulation is that the concatenation of unfolded tensors is low rank; in other words, we can have a low-rank factorization of the concatenated matrix. This indicates that both tensors have a common left component matrix indicating shared low-rankness along the coupled modes. Though this is a reasonable assumption, in practice, the degree of shared low-rankness needs to be controlled when regularizing a learning model. Since the coefficient in front of the regularization of concatenation of unfolded tensors is equal to one, similar to other trace norm regularizations for other modes in a coupled norm, it induces an equal amount of regularization for both coupled and uncoupled components. This makes existing coupled norms suboptimal. A better design of coupled norms with theoretical guarantees should be developed.
3.2 Inefficiency with Higher-Order Tensors
The coupled norms proposed by Wimalawarne et al. (2018) are confined to the overlapped trace norm and latent trace norms. Though these norms can be applied as low-rank-inducing norms for any tensor, they may not be efficient with higher-order tensors. The square norm proposed by Mu et al. (2014) has been shown to be more efficient as a low-rank-inducing norm for higher-order tensors. More specifically, for a higher-order tensor with modes each of dimensions with a multilinear rank of , the excess risk bound using the overlapped norm is bounded as (Wimalawarne et al., 2018), while the use of the square norm leads to an excessive risk bound of (see theorem 11 in the appendix and Mu et al., 2014). Thus, the existing coupled norms would not give the best performance for coupled higher order tensors, which creates a need to incorporate the square norm in coupled norms.
4 Proposed Methods
In this section, we propose new approaches to overcome the limitations we have described. We propose a new coupled completion model and discuss extensions to coupled norms.
4.1 Scaled Coupled Norms
We propose to explicitly control the regularization on concatenated components on the coupled mode. We achieve this by introducing a scaling factor for the coupled regularization of the coupled norm. To include the scaling parameter, we extend the existing definition of the coupled norm as , which we hereafter refer to as a scaled coupled norm, where the superscript indicates that the regularization on components on coupled mode is scaled by .
We can also regularize each of the trace norms separately (e.g., the right-hand side of equation 4.2 can be ), which would add more computational costs to solve the completion model. Our definition of scaled coupled norms is more convenient during optimization due to fewer parameters and also helps in theoretical analysis and interpretation, as we show in section 5.
4.2 Completion of Coupled Higher-Order Tensors
Now we propose coupled norms for higher-order tensors by combining the square norm with the coupled norms.
4.2.1 Coupled Norms for a Higher-Order Tensor and a Matrix
4.2.2 Coupled Norms for a Higher-Order Tensor and a Three-Way Tensor
4.2.3 Coupled Norms for Two Higher-Order Tensors
4.2.4 Coupled Norms for Tensors Coupled on Multiple Modes
4.3 Optimization of the Coupled Completion Model
The objective function in equation 4.1 can be solved using convex optimization methods. We have used the alternating direction method of multipliers (ADMM) method (Boyd, Parikh, Chu, Peleato, & Eckstein, 2011) to solve the above objective function. We do not give the details of the optimization procedure since it is similar to the optimization of coupled norms given in Wimalawarne et al. (2018).
5 Theoretical Analysis
In this section, we present a theoretical analysis of the proposed coupled completion models regularized by the scaled coupled norms and the higher-order coupled norms.
Taking a similar approach to that of Wimalawarne et al. (2018), we derive excess risk bounds (El-Yaniv & Pechyony, 2007) for coupled completion using the Rademacher complexity. For our analysis, we consider two tensors and coupled on their first mode. We represent indexes of observed elements of by the set , where refers to the element . Similarly, the set represents the set of observed elements in . Further, we consider observed elements separated as training and test sets as , , , and , such that and .
5.1 Excess Risk of Scaled Three-Mode Couple Norms
Excess risk bounds for three-mode tensors coupled with a matrix based on unscaled coupled norms have been derived in Wimalawarne et al. (2018). Since the scaling parameter affects only the concatenated components of the coupled norm, the excess risk bounds in Wimalawarne et al. (2018) can be easily updated for scaled norms (see appendix A). The updated Rademacher complexities ( in equation 5.5 for completion of a coupled three-mode tensor and a matrix are shown in Table 1. We show only the , , and norms due to space limitations.
|Norm .||Rademacher Complexity .|
|Norm .||Rademacher Complexity .|
Notes: Coupled completion of and resulting in a hypothesis class where is any of the three-mode tensor-based coupled norms. The multilinear rank of is , and the rank on coupled mode unfolding is . , , , and are constants.
In Table 1, the parameter scales the induced low-rankness related to the rank of the coupled unfolding . Note that the parameter inversely scales the components and . This behavior of scaling with tells us that if , then the shared low-rankness among the two tensors on the coupled mode is less, and excess risk would be bounded with larger terms of and . On the other hand, if , it indicates more shared low-rankness among coupled tensors, and the maximum term would select a smaller value. This analysis allows us to conclude that in order to obtain less excess risk, coupled tensor should share an adequate amount of low-rankness on the coupled mode.
5.2 Excess Risk of Coupled Higher-Order Tensors
Now we look into excess risk bounds for completion models regularized by the proposed higher-order coupled norms. Due to the large number of coupled norms we can define using the proposed norm, we analyze excess risk bounds for only a few coupled norms in this section.
The following theorem gives the Rademacher complexity for a coupling between a -mode tensor with and a matrix .
Next, three theorems consider a coupling between a -mode tensor with and a three-mode tensor .
Inspection of the bounds of theorems 1 to 3 leads us to similar conclusions as in section 5.1 that more shared low-rankness among the coupled tensors leads to lower excess risks due to the scaling of . In order to make easier comparisons, we consider a tensor with and matrix with . Assuming that the multilinear rank of is , the rank of is , and the rank on the concatenation of unfolded tensors is also , then the Rademacher complexity of is bounded by given that is sufficiently large ().
Theorems 1, 2, and 3 also show that combining the square norm with the overlapped trace norm and the scaled latent norm leads to a lower Rademacher complexity. Again, if we consider the special case of and , we have a Rademacher complexity of . However, if we apply coupled norm in Wimalawarne et al. (2018), we end up with a larger Rademacher complexity of . Hence, our proposed method leads to a better theoretical guarantee.
Finally, we consider the coupling of two higher-order tensors and where .
We can draw a similar conclusion for coupling two higher-order tensors as in previous theorems that the proposed extension leads to a lower Rademacher complexity compared to applying coupled norms in Wimalawarne et al. (2018). If we extend norms from Wimalawarne et al. (2018) or section 4.1 for higher-order tensors (e.g., ), the excess risk will be bounded by a larger term, such as , which is larger than the excess risk bounds achievable from theorem 5. This indicates that the integration of the square norm to coupled norms leads to better performance for coupled higher-order tensors.
Finally, we point out that Rademacher complexities for all coupled norms are bounded by , where is the total number of observed elements of both coupled tensors. If completion of tensors were performed separately, the resulting Rademacher complexities for each tensor would be bounded with respect to the number of observed elements of that tensor. Since the Rademacher complexity is bounded by , it may lead to lower bounds compared to the sum of individual Rademacher complexities for each tensor. Furthermore, since we used the transductive Rademacher complexity analysis, we obtained faster rates of decrease by compared to an analysis under inductive settings (Shamir & Shalev-Shwartz, 2014), which could lead to a bounding by .
In this section, we present details of simulation experiments that we carried out for coupled tensor completion.
6.1 Simulation Experiments
We organized our simulation experiments into two sections. In the first section, we give a simulation experiment based on scaled coupled norm regularized coupled completion models for a coupled three-mode tensor and a matrix. In the following section, we give simulation experiments to evaluate the proposed higher-order coupled norms for coupled higher-order tensor completion.
6.1.1 Experiment with Coupled a Three-Mode Tensor and a Matrix
To create coupled tensors for our simulations, we used a similar approach as in Wimalawarne et al. (2018). All our coupled tensors were created using multilinear ranks. To generate a -mode tensor with multilinear rank , we created a core tensor sampled from a Normal distribution and orthogonal component matrices and compute where is the -mode product (Kolda & Bader, 2009). We coupled two tensors and along a mode by sharing left singular vectors on their mode-, unfolding and with . We added noise sampled from a gaussian distribution of zero mean and variance of 0.01 to all elements of the tensors. We randomly sampled training sets of percentages of 30, 50, and 70 from the total number of elements of each tensor and another 10% as validation sets. The remaining elements were taken as test sets. We repeated the experiments with three random selections and calculated the mean squared error (MSE) on the test data.
For our simulation experiment in this section, we created a three-mode tensor and a matrix coupled on their first modes. We specified the multilinear rank of as (15, 5, 5) and the rank of as 5. We explicitly shared five left singular vectors among the tensor and the matrix on the coupled mode such that all the left singular vectors of the matrix are shared with the tensor. We cross-validated the regularization parameters from the range of 0.01 to 1 in intervals of 0.05 and the scaling parameters from the set .
Figure 1 shows the performance of the simulation experiment. We experimented with all the coupled norms for a three-mode tensor (Wimalawarne et al., 2018) and their scaled norms; however, for clear plotting, we show only the coupled norms and scaled coupled norms that gave the best performances for the coupled matrix and the tensor. As baseline methods, we used individual completion models regularized by the overlapped trace norm (OTN), the scaled latent trace norm (SLTN), and the matrix trace norm (MTN). As a further baseline method we used MTCSS proposed by Li et al. (2015).
Figure 1 shows that for matrix completion, none of the coupled norms outperformed individual matrix completion. The scaled coupled norm has performance equal to that of the matrix trace norm, while its unscaled norm has given a poor performance. For tensor completion, several norms, such as , , and , performed better than individual tensor completion by the overlapped trace norm and the scaled latent trace norm. In addition, the coupled norm had weaker performance than individual tensor completion, while its scaled norm had the best performance, and the MTCSS method had poor performance compared to coupled norms.
6.1.2 Experiments with Higher-Order Coupled Norms
In this section, we consider four-mode tensors of dimensions coupled to other tensors. We used the same procedure to create coupled tensors as in the section 6.1.1.
For all the experiments in this section, we used the coupled completion models regularized by higher-order norms introduced in section 4.2. To evaluate individual completion of the higher-order tensors, we used the square norm (SN; Mu et al., 2014). Further, we used the OTN and the SLTN for individual three-mode tensor completion and the matrix trace norm for individual matrix completion. For all models, we used regularization parameters from a range of 0.01 to 2 in intervals of 0.025 and scaling parameter set .
For our first simulation experiment with coupled higher-order tensors, we designed a coupled tensor with a four-mode tensor and a matrix coupled on their first modes. We specified the multilinear rank of the to be (3, 6, 6, 6) and the rank of to be 3 where all the left singular vectors along mode-1 were shared among both the tensor and the matrix. For this experiment, we used the coupled norm for coupled completion. From Figure 2, we can see that both the scaled and unscaled norms of gave the best performances for both the matrix and the tensor completion.
Next, we look into a coupled tensor consisting of a four-mode tensor and a three-mode tensor . We specified the multilinear rank of to be (3, 6, 6, 6) and the multilinear rank of to be (3, 6, 6) with all the left singular vectors along mode-1 shared among the tensors. As baseline methods, we used the square norm for and the overlapped trace norm and the scaled latent trace norm for . We experimented with different coupled norms that can be applied for coupled four-mode and three-mode tensors; however, for convenience, we plot in Figure 3 only results from the norms that gave the best performance. We observe that for , the best performance is given by the scaled and unscaled norms of and . For the tensor , the coupled norms , , and outperformed the OTN and the SLTN for individual tensor completion.
Finally, we look into two coupled four-mode tensors. Here, we considered two tensors and . We constrained the multilinear ranks of and to be (3, 6, 6, 6) and (3, 8, 8, 8), respectively, and coupled them on their first modes by making all the left singular vectors common to both the tensors. We used the coupled norms and for coupled completion. Additionally, we used the scaled overlapped norm extended from Wimalawarne et al. (2018) as , which indicates that both tensors are regularized with respect to each mode unfolding and the concatenated tensor unfolding on mode-1.
Figure 4 shows that both higher-order coupled norms outperformed individual tensor learning with the square norm. For the tensor , the scaled higher-order norm further improved the performance compared to the unscaled norm. We can also see that gave a weaker performance compared to coupled higher-order norms, agreeing with our theoretical analysis in section 5.
6.2 Multiview Video Completion Experiment
As a real-data experiment we applied our proposed methods for multiview video completion using the EPFL data set: multicamera pedestrian Videos data (Berclaz, Fleuret, Turetken, & Fua, 2011). The data set consists of movements of four people in a room captured from synchronized cameras. For our experiments, we used two videos; one video was considered to be corrupted and the other not. To create the video data, we sampled 50 frames with equal time splits from each video. We then downsampled each frame to a height and width of 76 and 102, respectively. The dimensions of both videos were the same , where , representing RBG color channels, , and . Since frames and RGB channels are common to both videos, we considered that the two videos are coupled on both of these modes. We considered the video to be corrupted and sampled percentages of 10, 30, 50, and 70 of the total number of elements as its observed elements (training sets). From the remaining elements, we considered 10% of the total number of elements as validation sets; the rest were taken as test sets. To recover missing elements of the corrupted video, we completed it coupled to the uncorrupted video as side information using our proposed completion models regularized using higher-order coupled norms.
We completed coupled videos using the coupled norms in equation 4.7 and in equation 4.8. We performed individual completion of using the square norm similar to the coupled norm using the first two modes to reshape the tensor as a matrix. We applied cross-validation on regularization parameters of ranging from 0 to 7, with intervals of 0.25. We experimented with different values for ranging from 0.1 to 1, with intervals of 0.1. We found that best performance for was given, with and for with . Figure 5 shows that the proposed coupled norms gave better performance compared to the individual tensor completion using square norm regularization. We provide further experimental results to compare the performances of the proposed methods with baseline methods in appendix D.
In this letter, we have investigated two limitations of coupled norms and proposed scaled coupled norms and coupled norms for higher-order tensors. Through theoretical analysis and experiments, we demonstrated that our proposed methods are more robust for coupled completion compared to existing coupled norms. However, we feel that coupled norms should be further investigated to be used widely in real-world applications.
One drawback of the scaling of coupled norms is that it requires more computations to find the optimal scaling parameters (). Though cross-validation can be employed to find the optimal scaling parameter, it can become computationally infeasible in real-world applications, especially with tensors with large dimensions. Future research in coupled norms should be directed toward finding better optimization strategies and parameter selection methods to overcome these computational issues. Further, our theoretical analysis was focused on excess risk bounds for tensor completion. In future research, a more suitable yet rigorous theoretical analysis would be to derive exact recovery bounds (Yuan & Zhang, 2016) for coupled completion.
Appendix A: Dual Norms for Scaled Coupled Norms
Dual norms of coupled higher-order tensors can also be derived by using a similar approach as in Wimalawarne et al. (2018). We give a brief overview of how to derive dual norms for higher-order coupled norms, starting with the dual norm of , which we derive next.
Appendix B: Excess Risk Bounds for Coupled Three-Mode Tensor Completion
The excess risk bounds for coupled completion using scaled coupled norms given in section 5.1 can be derived in an identical way, as Wimalawarne et al. (2018) proved for unscaled norms. As a guide for proof and completeness, we give the detailed proof of excess risk bounds for the norm .
Appendix C: Excess Risk Bounds for Coupled Higher-Order Tensor Completion
In this section, we give the proofs for the theorems in section 5.2.
Let us denote and corresponding Rademacher variables corresponding to and in equation 5.5.
From Shamir and Shalev-Shwartz (2014), we know that .
Next, we look at the excess risk for the coupled norm .
Let and be the completed tensors for and . Also let us denote and consisting of corresponding Rademacher variables of and .
To derive the bounds for we use a similar approach as for theorem 4.
Next, we give proofs for the mixed norms among higher-order tensors and three-mode tensors. First, we give the proof of theorem 4 for .
Following theorem 6, we can derive the bounds for other mixed norms as well. Next, we give the bounds for and without proofs.
Next we give the bound for the coupled norm .
Given higher-order tensor , the excess risk bound with the square reshaping norm.
The proof is direct and can be derived similarly to theorem 3 without considering the matrix coupling.
Appendix D: Further Experiments for Multiview Video Completion
We give further results for baseline methods for the multiview video completion experiment in section 6.4. We performed individual tensor completion of corrupted video data using the overlapped trace norm (OTN) and scaled latent trace norm (SLTN). We performed coupled completion using the coupled norm by extending the coupled norms in Wimalawarne et al. (2018). In Figure 6, we compare these baseline methods with the proposed norms and . We observed that the baseline methods performed poorly compared to proposed methods.
M.Y. was supported by the JST PRESTO program JPMJPR165A and partly supported by MEXT KAKENHI 16K16114. H.M. has been supported in part by JST ACCEL (grant JPMJAC1503), MEXT Kakenhi (grants 16H02868 and 19H04169), FiDiPro by Tekes (currently Business Finland), and AIPSE by Academy of Finland.