Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

This paper evaluates the performance of open-source GenAI-based agents for cybersecurity, specifically comparing them to the established Static Application Security Testing (SAST) tool, Bandit. Using three different Ollama-hosted models, the study measures precision, recall, and false positive rates, ultimately concluding that these GenAI agents are not yet adequate for effective SAST scanning in practical scenarios. This assessment highlights the limitations of current LLMs in specialized security applications, informing practitioners about the challenges of integrating AI into security workflows.