Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning

A technique originally developed to shrink the memory footprint of large language models has been successfully transferred to robotics, allowing multiple specialist robot policies to be stored on-device without sacrificing performance, according to a paper submitted June 24, 2026 [1]. The study, posted to the arXiv preprint server, applies Low Rank Adaptation (LoRA) — a form of Parameter-Efficient Fine-Tuning (PEFT) — to reinforcement learning for robotic manipulation tasks [1]. The researchers used a Proximal Policy Optimization (PPO) algorithm and fine-tuned a baseline model to different tasks using LoRA [1]. Their experiments show that LoRA can reduce memory usage by a factor of 20 to 160 compared with updating every layer of a neural network [1]. For a library of 10 to 50 specialized policies, that translates to a 90 to 95 percent storage saving [1]. The authors note this can be “the differentiating factor between being able to store the entire library in memory or having to use swap-memory in an applied robotics setting” [1]. LoRA works by freezing a model’s pre-trained weights and inserting small, trainable low-rank matrices into the architecture [3]. Only those compact adapters are updated during fine-tuning, which constrains learning to a low-dimensional subspace [5]. The approach has already been explored as a structural regularizer for critic learning in off-policy reinforcement learning, where it was found to reduce critic loss and improve policy performance across algorithms including SAC and FastTD3 [5]. The new work extends LoRA into online RL with PPO, a combination the authors say has not been demonstrated before [3]. In the robotics experiments, the team trained a base policy on a general task and then created specialist policies via LoRA fine-tuning [3]. The resulting policies performed with no significant difference in success rate compared to fully fine-tuned models [1]. Already with 10 specialist policies, the memory saving reached 85 percent compared with saving fully fine-tuned versions [3]. The researchers suggest this efficiency could have a significant effect when switching between policies on resource-constrained systems such as mobile robots [3]. Large language models, which are typically based on transformer architectures, have driven recent advances in chatbots and text generation [8]. The broader field of machine learning relies on neural networks that learn from data without explicit programming [6]. Training these networks is compute-intensive and often accelerated by graphics processing units [7]. The robotics study indicates that memory-saving techniques from the language domain can be adapted to physical systems where onboard storage and compute are limited [1].