The role of class encoding in neural collapse

Researchers have explored the concept of neural collapse in neural network classification models, revealing its structural properties and the role of label encoding.

A recent study^[1] examined the last-hidden-layer activations in neural network classification models, finding that neural collapse is a structural property when trained beyond zero classification error. The researchers used the unrestricted feature model with mean squared error training loss to investigate the impact of label encoding on neural collapse. They demonstrated that one-hot encoded labels and balanced data lead to a transition from a simplex equiangular tight frame to an orthogonal frame when increasing the bias regularization coefficient. The final classifier's bias was found to center the labels, compensating for the discrepancy between the global mean of the labels and the origin. Meanwhile, another research team^[2] proposed a new framework, Optical-Guided Neural Collapse, for Synthetic Aperture Radar (SAR) few-shot class-incremental learning. This framework utilizes data-rich optical ATR datasets to guide SAR feature learning and projects SAR features onto orthogonal subspaces via principal angle constraints. The approach achieved the highest final accuracy and a favorable trade-off between final performance and performance degradation. Neural collapse metrics showed improved intra-class compactness and inter-class separability.