The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages

A study posted to arXiv on 14 June 2026 finds that large language models inherit specific internal components, termed truthful heads, from their foundational models, a trait that persists through fine-tuning and multimodal adaptation [1]. The research examined model lineages built on Vicuna, Qwen2.5, LLaMA2, and Mistral architectures [1]. By quantifying head-level context-truthfulness scores, the authors determined that these scores are strongly preserved within model families, even after instruction tuning or conversion into multimodal systems [1]. The inheritance pattern was found to be consistent with attention-head weight preservation, and the identified context-truthful heads were shown to attend to query-relevant evidence [1]. Building on this observation, the team proposed TruthProbe, a soft-gating strategy designed to amplify the contribution of context-truthful heads while maintaining the function of other attention heads [1]. The approach was evaluated on the HaluEval dataset for contextual truthfulness and on the POPE and CHAIR benchmarks for multimodal hallucination, where it delivered improvements [1]. The base-model Truth Scores transferred effectively to fine-tuned language models and their multimodal descendants, according to the paper [1]. The concept of inheriting functional components through model lineages draws a loose parallel to biological systems, where transcription factors regulate gene expression by binding to specific DNA sequences and are preserved across cell lineages to maintain function [7]. In machine learning, transfer learning has previously been used to carry knowledge between related tasks or datasets, as seen in catalyst informatics where models trained on large datasets like OC20 have improved performance on smaller, specialized datasets [4]. The TruthProbe work extends this principle to the internal structure of attention heads, suggesting that foundational model properties can serve as a stable substrate for downstream applications [1][2]. The authors have released the code for TruthProbe on GitHub, allowing other researchers to test the soft-gating method on additional model families [2]. The paper was submitted under the Computation and Language category on arXiv [1].