Supervised Learning by Low Rank Estimation on Tensor Data

Abstract

Data tend to show up in tensor format more frequently in real life such as videos and magnetic resonance imaging (MRI). However, classical regression and classification methods are not designed capture the information in tensor formatted data and analyze them efficiently. This thesis consists of two main parts, tensor regression and tensor classification. In the first part, we consider the low rank tensor regression framework proposed in the paper \cite{paper}, which protects the natural form and spatial structure of tensor covariates in data. We propose and study the regularized low rank tensor regression by employing shrinkage-type penalty functions, including the ridge and the lasso, and compare with the unpenalized low rank tensor regression. Extensive numerical experiments are conducted to evaluate the performance of the new methods, by varying tensor ranks, tensor dimensions, sample sizes, and noise variance. In addition, we compare the regularized and unregularized low rank tensor regression in high-order tensor settings. Lastly, we apply the tensor regression models to hand-written digit classification problems. In the second part, we consider binary classification with high-dimensional tensors as covariates. We are motivated by the least squares formulation of linear discriminant analysis (LDA), which is extensively studied for classification with vector and matrix inputs \cite{sda}. However, extending it to high-dimensional tensors introduces unique theoretical and computational challenges. Classical LDA depends heavily on estimating mean tensors and the inverse of the sample covariance matrix, which becomes computationally costly and even infeasible in high-dimensional settings. The problem of inverting the covariance matrix becomes ill-posed when the number of parameters far exceeds the number of training samples. We introduce a generalized LDA framework for classifying tensors of arbitrary dimensions, which is reformulated as a regularized least squares problem to avoid estimating high-dimensional tensor variances. This new framework incorporates existing methods for vector and matrix inputs \cite{stat, matrix2, tda}, and addresses the curse of dimensionality while enhancing computational efficiency through its low-rank sparse feature. We investigate the theoretical properties of the estimators, establishing their optimality and Fisher consistency. Additionally, we offer a closed-form formula for computing the optimal intercept, specially adapted for tensor inputs from vector inputs \cite{sda}. The effectiveness of the new methods is demonstrated through both simulations and real-world applications in 3D biomedical imaging data.