SANA: What Matters for QA Agents over Massive Data Lakes?

The article introduces SANA (Search Agent Navigation Ablation framework), a diagnostic framework designed to evaluate exploratory question answering (EQA) agents operating over large data lakes. SANA transforms EQA tasks into runtime profiles that allow for the ablation of components like search, planning, and data analysis, revealing specific areas of failure in agent performance. This framework was applied to the LakeQA and KramaBench benchmarks, highlighting that data analysis is a consistent bottleneck and search capabilities are particularly limited in larger data lake contexts, providing practitioners with insights for improving agent design and performance evaluation.