Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

Fine-tuning large language models on vulnerability data does not improve their security reasoning for systems software, according to a study that introduced a new evaluation framework called CWE-Trace. The research found models merely adapt their output distributions to training data without developing genuine comprehension of software flaws [1]. The study, submitted to arXiv on 18 June 2026, evaluated eight vanilla LLMs and 15 LoRA fine-tuned variants across three tasks: non-targeted detection, targeted detection, and CWE classification [1][2]. The CWE-Trace framework was built from 834 manually curated Linux kernel samples spanning 74 Common Weakness Enumeration categories [1][2]. It enforces a strict temporal split between pre-2025 historical data and post-cutoff leakage-free data, preserving context-aware vulnerable-to-patched pairs [2]. Two diagnostic metrics were introduced: the Directional Failure Index and Hierarchical Distance and Direction [2]. The DFI measures systematic failure modes, with values ranging from -85.5 to +94.8 percentage points across models [1][2]. These failure patterns persisted from historical to post-cutoff data and resisted correction through fine-tuning [2]. Data contamination provided no measurable advantage, the researchers found. Function-level analysis showed that 84% of nominally contaminated samples carried no usable memorization signal, as vulnerable functions were absent or cross-mapped across datasets [1][2]. Approximately 31% of contaminated samples carried CWE misclassification [1][2]. The best detection score reached only 52.1%, just 2.1 percentage points above chance, while exact CWE ranking remained below 1.3% Top-1 accuracy [1][2]. The weakest backbone at binary detection, DeepSeek-R1, gained the most in coarse CWE classification, revealing that detection and understanding are decoupled capabilities [1][2]. DeepSeek-R1, developed by the Chinese AI company DeepSeek founded in July 2023, is an open-weight model that provided responses comparable to OpenAI's GPT-4 and o1 at reportedly lower training costs [8]. "Fine-tuning shifts the output threshold without changing the decision policy," the authors wrote. "This is calibration without comprehension: output distributions adapt to training data while the underlying security reasoning remains absent" [2]. The findings confirm that current LLMs lack reliable security reasoning for systems software, regardless of fine-tuning strategy [1][2].