Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning

Continual learning systems often forget previously acquired knowledge, but a new study sheds light on the mechanisms driving this forgetting using a controlled, synthetic framework^[1].

Researchers have been studying ways to mitigate 'catastrophic forgetting' in sequential tasks, where AI models forget previously learned information^[2]. A recent study used a toy-world framework to make the mechanisms driving forgetting observable and testable. The study found that 'superposition' - or the overlap between different features - tends to increase over time, with transient dips at task boundaries^[1]. Higher feature sparsity was shown to induce more superposition, yet did not inevitably cause forgetting. In fact, when representations remain strong, forgetting can be reduced despite overlap. The study also found that task-level effective rank grows with sparsity, indicating broader capacity usage under sparse regimes. Another study proposed a new method called 'Experience Blending', which jointly trains on exemplars and 'Support Boundary Data' generated via differential-privacy-inspired noise into latent features. This approach was shown to lead to more stable and robust continual learning compared to standard 'experience replay' methods^[2].