Pseudo-Random Weight Initialization in Deep Neural Networks

Abstract

Deep neural networks are incredibly complex learning models whose performance is greatly dependent on the random initialization of their connecting weights. Training two neural networks of the same architecture with different weight initializations will almost inevitably lead to two completely different networks, each learning to perform its task in a unique way.

This thesis investigates the use of an elliptical sampling technique to produce pseudo-random initializations for a rudimentary deep neural network. By using this technique to gradually change the weight initialization of our network, we hope to gain a better understanding of the complicated mechanics underlying its learning process.

Additionally, we present the method of iterative weight refinement, which takes advantage of elliptical sampling techniques to optimize the performance of a given metric by repeatedly improving the choice of weight initialization. While this technique has certain limitations, it offers a clear and systematic method for manipulating weight initializations to improve the performance of a neural network.