AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems

AIChilles is a new framework designed to automatically identify hidden weaknesses in AI-evolved systems by comparing baseline programs with their AI-generated counterparts. It employs techniques such as deterministic workload-parameter extraction and differential oracles to uncover regressions in correctness, runtime, memory usage, and output quality, revealing 49 distinct weaknesses across 30 AI-evolved programs. This tool is significant for practitioners as it enhances the reliability of AI-driven development by systematically detecting flaws that could arise from automated code generation.