The Query Channel: Information-Theoretic Limits of Masking-Based Explanations

This paper presents a novel framework for analyzing masking-based post-hoc explanation methods like KernelSHAP and LIME through the lens of information theory, treating the explanation process as communication over a query channel. It derives a strong converse and achievability results related to the identification capacity of the channel, demonstrating that reliable explanations can be achieved under certain query budgets, while standard techniques may fail. The findings highlight the importance of understanding the entropy of explanations and the impact of noise and model architecture on the effectiveness of feature importance estimations, which is crucial for practitioners developing interpretable AI systems.