Confident and Wrong: Silent Semantic Failures in Coding Agents

The study introduces the concept of "silent semantic failure" in coding agents, revealing that models like GPT-5 and Llama 4 exhibit high submission rates but low resolution rates for software engineering tasks. Specifically, GPT-5 submits patches 100% of the time but only resolves 44% of tasks, while Llama 4 resolves 18% of tasks despite a 99% submission rate. This highlights a critical gap in evaluating AI models, as current metrics based on submission rates do not accurately reflect their trustworthiness; thus, the authors advocate for new evaluation criteria that prioritize test-verified correctness and the ability to recognize when no action is warranted.