Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions

This paper introduces a diagnostic protocol for evaluating evidence utilization in long-context and retrieval-augmented language models, assessing their performance across four conditions: no-evidence, full-context, retrieved-evidence, and oracle-evidence. The study evaluates five models from the Qwen, Gemma, Llama, and Mistral families on 18,000 predictions, revealing that full-context inputs generally outperform retrieved inputs in terms of answer accuracy and evidence recovery, particularly in multi-hop scenarios. This work is significant for practitioners as it provides a structured methodology to assess model reliance on evidence, informing better model design and evaluation strategies in AI applications.