The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages

The study introduces TruthProbe, a soft-gating mechanism designed to enhance contextual truthfulness in large language models (LLMs) and multimodal LLMs (MLLMs) by amplifying context-truthful attention heads. It demonstrates that Truth Scores, which quantify head-level context-truthfulness, are preserved across model lineages, including Vicuna, Qwen2.5, LLaMA2, and Mistral, even after adaptations like instruction tuning. This approach significantly improves performance on benchmarks such as HaluEval, POPE, and CHAIR, making it a valuable tool for practitioners focused on reducing hallucinations in AI outputs.