Solution Path Algorithms for Distributionally Robust Regression and Classification

Abstract

Data uncertainty is a challenging problem in machine learning. Distributionally robust optimization (DRO) is a popular technique to address the data uncertainty problem and it minimizes the worst-case expected loss function under the ambiguity set. It has been shown that various DRO formulations can be reformulated as regularized machine learning models. The regularization hyperparameters control the bias-variance trade-off and will influence the generalization performance. However, existing works mainly focus on modeling part and solve the model by off-the-shelf solvers. To perform the hyperparameter tuning, we need to train the model multiple times and hence is computationally prohibitive for large-scale data and high-dimensional data. Solution path algorithms provide a new approach to speed up the hyperparameter tuning. It can be proven that the optimal solutions of some machine learning models change piecewise linearly with respect to hyperparameters. Hence, we can obtain the entire solution path with respect to different hyperparameters by monitoring the breakpoints only. This piecewise linear property inspires us to develop efficient algorithms to speed up the hyperparameter tuning. For the regression problems, we propose a general distributionally robust regression model based on DRO. The proposed model has piecewise linear loss function and elastic net penalty term. We prove the piecewise linear property of the optimal solutions to this model, which enables us to develop a solution path algorithm for the hyperparameter tuning. Doubly regularized least absolute deviations (DrLAD) regression model is proposed based this framework. A solution path algorithm is developed to speed up the tuning of two hyperparameters in this model. For the classification problems, a new support vector machine (SVM) with double regularization terms and double margins is derived based on DRO. The proposed model can explain the data uncertainty in probabilistic way as well as perform automatic feature selection for high-dimensional data. We prove that the optimal solutions of this model change piecewise linearly with respect to the hyperparameters. Based on this piecewise linear property, a solution path algorithm is proposed to efficiently obtain the optimal solutions and thus speed up the hyperparameter tuning process. We also extend the DRO SVM to kernel space to deal with the data that are not linearly separable. A solution path algorithm is presented to efficiently solve the proposed model.