(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable

The paper presents the Human-in-the-Loop Economic Research (HLER) framework, which enhances the reliability of AI-assisted social science research by structuring cognitive labor between humans and LLMs. In a factorial experiment involving 280 research runs, HLER reduced critical failure rates from 72% to 16% by implementing architectural commitments such as restricting LLMs from executing data work and incorporating human decision gates. This approach not only improves reliability but also highlights residual weaknesses in AI outputs, thus preventing the advancement of unreliable claims in research publications.