Energy-Structured Low-Rank Adaptation for Continual Learning

A new method called Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E^2-LoRA) aims to improve how neural networks learn tasks sequentially without forgetting previous ones, according to a paper published on arXiv. The approach addresses a limitation in existing continual learning techniques by concentrating knowledge into specific parameter ranks. [1] Continual learning, the process by which a single neural network model is trained on a stream of different tasks, often faces the problem of catastrophic forgetting. While orthogonal subspace methods attempt to mitigate task interference, they suffer from energy diffusion across the basis, which hinders knowledge compaction and exhausts capacity for future tasks. [1][2] Neural networks, which consist of layers of connected artificial neurons that process signals via weighted connections, are foundational to modern machine learning applications from image generation to large language models. [3] The researchers behind E^2-LoRA observed that the output feature drift caused by parameter updates is inherently low-rank. They theoretically proved that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. [1][2] By explicitly ordering and concentrating knowledge into leading ranks, the method frees capacity for subsequent tasks. [1] To manage the trade-off between retaining old knowledge and learning new information, the team designed a dynamic rank allocation strategy. This strategy jointly optimizes energy retention and model plasticity. [1][2] The paper reports that extensive experiments across multiple benchmarks demonstrate E^2-LoRA achieves state-of-the-art performance. [1][2] The specific datasets used in these benchmarks were not detailed in the abstract, though machine learning research commonly relies on curated, high-quality datasets for training and evaluation. [4]