Beyond Simpson's Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship

The study published in arXiv examines the impact of human co-authorship on the merge rates of pull requests (PRs) across five AI coding agents, revealing a case of Simpson's Paradox. Analyzing 33,596 PRs from the AIDev dataset, it finds that while aggregate merge rates for human co-authored PRs are lower (53.8% vs. 79.8%), stratification by agent identity shows that agents like Copilot and Devin actually benefit from co-authorship, highlighting the importance of controlling for agent composition and PR structure in evaluations. This research underscores the necessity for practitioners to avoid relying on pooled statistics without stratification, as they may lead to misleading conclusions regarding the efficacy of AI agents in collaborative coding environments.