Effects of Input-Dependent Data Augmentation on Model Generalization
Author(s)
Jin, Matthew Y.
Advisor(s)
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
Data augmentation is a technique used by many machine learning models to reduce overfitting on the training data. Common augmentations include gaussian noise injections and dropout/input masking. However, determining the hyperparameters for these augmentations often involve conducting a parameter sweep over the parameter values and choosing the best performing one which comes with significant computational cost. In this thesis, we explore the possibility of using statistics from the training data itself to inform the parameterization of these two augmentations. For gaussian noise injection, we parameterize the noise covariance with the data covariance by class. For dropout, we define the dropout probability to be inversely proportional to the variance of each input feature. These input-dependent augmentations are tested in classification experiments with three handwritten digit datasets: the noiseless Optdigits dataset and two noisy MNIST variations, the first with background noise sampled from a uniform distribution and the second with the MNIST digits pasted on top of natural images. We find that these input-dependent augmentations provide equivalent or better accuracies compared to the best performing isotropic noise injection and fixed dropout. We also find that the input-dependent dropout performs exceptionally well on the MNIST with background images dataset. In addition to improving generalization, these augmentation methods automatically determine the scale of the noise injected or dropout probability, forgoing the need for hyperparameter tuning. We also visualize the effect the augmentations have on the input to provide some insight on how they affect the features.
Sponsor
Date
Extent
Resource Type
Text
Resource Subtype
Undergraduate Research Option Thesis