Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination

Researchers have made new findings in understanding how deep learning networks perform hierarchical feature learning across layers, with implications for transfer learning.

A recent study published on arXiv^[1] has shed light on the mystery of how deep networks learn meaningful features from raw data. The researchers found that linear layers mimic the roles of deep layers in nonlinear networks for feature learning. They discovered that the evolution of features follows a simple and quantitative pattern from shallow to deep layers in linear networks, with within-class features being compressed at a geometric rate and between-class features being discriminated at a linear rate. This pattern was also observed in deep nonlinear networks, aligning with recent empirical studies. The study's findings have practical implications for transfer learning. Another study published on arXiv^[2] found that constraining the optimization space through a pre-trained base model and low-rank adaptation (LoRA) can induce structure in weight space, with multiplicative LoRA weights achieving high representation quality and enabling higher-quality generation when used with latent diffusion models.