Evaluating LLMs for Real-World Web Vulnerability Detection

This study benchmarks six large language models (LLMs) for their ability to detect web vulnerabilities in WordPress plugins, focusing on SQL injection, stored cross-site scripting, path traversal, and remote code execution. Notably, Claude Opus 4.6 achieved the highest detection rate at 63%, while open-weight MiniMax M2.5 performed comparably at 48%, and self-hosted Qwen 3.5 lagged at 35%. The findings highlight the impact of prompt design on detection efficacy and reveal that no model achieved consistent reporting across iterations, underscoring the challenges of using LLMs for real-world vulnerability detection and providing valuable insights for security practitioners.