Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization

Researchers have shown that small language models, fine-tuned with a fraction of the data and cost required by large commercial systems, can outperform GPT-4o and GPT-5 on biomedical claim verification tasks, according to a new study [1]. The study, posted to arXiv on June 11, 2026, fine-tuned three small large language models (LLMs)—Phi-3-mini (3.8B parameters), Qwen2.5-3B, and Mistral-7B—using the parameter-efficient QLoRA method on the SciFact and HealthVer datasets [1]. Large language models are neural networks trained on vast text corpora for tasks including text generation and analysis, with generative pre-trained transformers (GPTs) representing a prominent class of such models [3]. The authors provide the first direct comparison of QLoRA-tuned models against the zero-shot performance of GPT-4o, GPT-5, and fine-tuned BioLinkBERT encoders [1]. Mistral-7B, fine-tuned with QLoRA on just 1,008 training examples, achieved an F1 score up to 12% higher than both GPT-4o and GPT-5, while operating at a fraction of the computational cost [1]. The work addresses a key limitation of large proprietary models, which the paper notes suffer from cost and opacity that limit scalable use in biomedical settings [2]. The investigation also uncovered a previously unreported structural artifact in the SciFact dataset that artificially inflates in-domain evaluation scores [1]. To test the robustness of their findings, the team conducted bidirectional cross-domain evaluations, training models on SciFact and testing on HealthVer, and vice versa, while matching model sizes to isolate the effect of dataset structure from data quantity [1]. The results demonstrated that training on structurally sound data enables more reliable cross-domain transfer [1]. The researchers plan to release all code and adapter checkpoints to facilitate further work in cost-effective biomedical claim verification [1].