AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable

The study evaluates the performance of LLM-based coding agents, specifically Claude Code and Codex, in social science research, focusing on their methodological diversity and bias in interpretation. The findings indicate that while Codex matches human methodological diversity and Claude Code exceeds it, both agents maintain alignment with human consensus estimates. Notably, the agents exhibit vulnerability in their interpretative processes, as demonstrated by a prompt that significantly alters Claude Code's conclusions without affecting its estimation distribution, highlighting the need for careful consideration of AI biases in research interpretations.