Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

The paper introduces AgenticInterpBench, a benchmark comprising 84 semi-synthetic transformer circuits with 163 component-level annotations, aimed at assessing language model (LM) agents' ability to explain identified circuits in mechanistic interpretability. It presents HyVE (Hypothesize, Validate, Explain), an agentic explainer that utilizes an iterative process to produce detailed explanations, demonstrating that while various LM backbones can generate useful insights, challenges in the validation phase hinder consistent performance. This work is significant for practitioners as it highlights the potential of LMs in circuit explanation while emphasizing the need for robust validation mechanisms to enhance interpretability in AI systems.