Beyond LoRA: Can you beat the most popular fine-tuning technique?

Low-Rank Adaptation, or LoRA, dominates parameter-efficient fine-tuning, appearing on 98.4% of Hugging Face Hub model cards that cite a single PEFT method, but new benchmarks from Hugging Face suggest alternative techniques can outperform it on both accuracy and memory usage [1]. Parameter-efficient fine-tuning, known as PEFT, allows developers to adapt large open models to custom tasks using a fraction of the memory required for full fine-tuning [1]. The Hugging Face PEFT library unifies more than 40 distinct techniques behind a single API, integrating with the Transformers and Diffusers ecosystems [1]. Despite this variety, LoRA accounts for 20,509 of 20,834 model cards that mention exactly one PEFT technique on the Hub, and 7,111 of 10,000 image-generation checkpoints sampled from an external site [1]. On GitHub, 71.3% of code snippets importing a PEFT configuration reference LoRA [1]. Hugging Face researchers argue that LoRA’s dominance may be self-reinforcing rather than purely merit-based, driven by early visibility, abundant tutorials, and broad downstream support [1]. To test that hypothesis, the team built standardized benchmarks for large language models and image generation, running every technique under identical conditions: same base model, dataset, training code, and hardware [1]. The benchmarks track test performance alongside VRAM usage, runtime, checkpoint size, and forgetting metrics, with results designed to be reproducible on consumer hardware [1]. On a math reasoning task using MetaMathQA, LoRA with rank-stabilized initialization reached 53.2% accuracy at 22.6 GB peak VRAM, placing it on the Pareto frontier [1]. However, BEFT achieved 32.9% accuracy while using only 20.2 GB, and Lily reached 54.9% accuracy at 25.6 GB, offering different trade-offs depending on whether memory or accuracy is the priority [1]. Standard LoRA without specialized initialization or optimizers scored 48.1% at 22.5 GB, a result the authors suggest should be avoided in favor of the enhanced variants [1]. For image generation, the benchmark tested whether a model could learn a new concept—a cat plushy—and generalize it to unseen prompts [1]. Orthogonal Fine-Tuning, or OFT, achieved a DINO similarity score of 0.708 using 9.01 GB of memory, while LoRA scored 0.697 at 9.97 GB, meaning OFT strictly dominated LoRA on both metrics [1]. The platform hosting these models has also drawn attention for its governance challenges; a recent study noted that Hugging Face, GitHub, and Civitai face complex moderation demands because AI systems can both contain content and function as open-ended tools [5]. The PEFT library’s benchmarks are open to community contributions, with instructions provided for adding new experiments or hyperparameter configurations [1]. The authors caution that no single benchmark can capture every use case and encourage users to inspect generated samples and explore additional metrics such as runtime and checkpoint size before selecting a technique [1].