From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

The paper presents a study on hallucination detection in the Whisper large v3 ASR model, focusing on fluent transcriptions that lack audio basis. It evaluates three detection paradigms—text-based classifiers, LLM-based methods with domain-specific prompts, and probing of internal decoder states—finding that the latter provides the highest performance. A late-fusion meta-classifier combining these approaches achieves optimal detection, highlighting the importance of understanding internal model representations for practitioners addressing ASR hallucinations.