SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

SEVRA-BENCH is a newly introduced benchmark designed to evaluate the effectiveness of large language model (LLM) reviewers in detecting adversarial pull requests (PRs) that reintroduce vulnerabilities. It consists of 1,062 malicious PRs derived from real project commits that fixed vulnerabilities listed in the CVE database, framed with varying social-engineering tactics. The benchmark highlights significant performance disparities between closed- and open-source models, emphasizing the need for improved security capabilities in open-source LLMs to better safeguard code review processes.