Combining Structural Knowledge with Sparsity in Machine Learning and Signal Processing

Abstract

With the advancement of technology, we are able to collect, store, and transmit an ever-increasing volume of data. However, our ability to extract useful patterns from this massive amount of data is still lagging behind. Machine learning and signal processing research aims to address fundamental issues in discovering the hidden structure in large-scale data, and to develop practical algorithms for real-world applications.

In this thesis, we take a deep look at two fundamental elements in machine learning and signal processing research. The first element is sparsity. The sparsity principle emphasizes the importance of having simple representations of patterns. The second element is structural knowledge, which emphasizes the importance of respecting structure in the patterns. In this thesis, we argue that although sparsity leads to many effective machine learning and signal processing algorithms, simply considering sparsity is not enough, and that combining structural knowledge with sparsity leads to better algorithm performance and deeper theoretical understanding. By designing and analyz- ing machine learning and signal processing algorithms that utilize both structural knowledge and sparsity, we demonstrate that combining structural knowledge with sparsity is a useful strategy in various signal and data representation, denoising and classification problems.

Under the unifying theme of combining structural knowledge with sparsity, this thesis takes us on a tour of a variety of problems in machine learning and signal processing, including boost- ing classification algorithms, image denoising methods, wavelet transforms, dictionary learning and solving lasso problems. We will study how structural knowledge and sparsity interact with each other in these different contexts, and demonstrate the importance of combining structural knowl- edge with sparsity. The work in this thesis helps to strengthen our understanding of the role that structural knowledge and sparsity play in machine learning and signal processing and to improve various sparsity inspired data analysis algorithms.