Title:
Explicit Group Sparse Projection for Machine Learning
Explicit Group Sparse Projection for Machine Learning
Author(s)
Ohib, Riyasat
Advisor(s)
Calhoun, Vince D.
Plis, Sergey
Plis, Sergey
Editor(s)
Collections
Supplementary to
Permanent Link
Abstract
The concept of sparse solutions in classical machine learning is noted for its efficiency and has parallels in the natural world, such as in the mammalian visual cortex. This biological basis provides an inspiration for the importance of sparsity in computational models. Sparsity is increasingly relevant in machine learning, especially in non-negative matrix factorization (NMF), where it aids in interpretability and efficiency. NMF involves breaking down a non-negative matrix into simpler components, with sparsity ensuring these components distinctly represent data features, simplifying interpretation. In deep learning, sparse model parameters lead to more efficient computation, quicker training and inference, and in some cases, more robust models. As models grow in size, the role of inducing sparsity becomes even more crucial.
In this thesis, we design a new sparse projection method for a set of vectors that guarantees a desired average sparsity level measured leveraging the popular Hoyer measure. Existing approaches either project each vector individually or require the use of a regularization parameter which implicitly maps to the average $\ell_0$-measure of sparsity. Instead, in our approach we set the \revise{Hoyer} sparsity level for the whole set explicitly and simultaneously project a group of vectors with the \revise{Hoyer} sparsity level of each vector tuned automatically. Hence, we call this the Group Sparse Projection (GSP).
We show that the computational complexity of our projection operator is linear in the size of the problem. GSP can be used in particular to sparsify the columns of a matrix, which we use to compute sparse low-rank matrix approximations (namely, sparse NMF).
We showcase the efficacy of our approach in both supervised and unsupervised learning tasks on image datasets including MNIST and CIFAR10. In non-negative matrix factorization, our approach yields competitive reconstruction errors against state-of-the-art algorithms. In neural network pruning, the sparse models produced by our method have competitive accuracy at corresponding sparsity values compared to existing methods.
Sponsor
Date Issued
2023-12-14
Extent
Resource Type
Text
Resource Subtype
Thesis