Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware

This study benchmarks four open-source local LLMs (Qwen 2.5 Coder 7B, Llama 3.1 8B, Mistral 7B, and Meditron 7B) for natural-language-to-SQL query generation in biopharmaceutical manufacturing, utilizing a synthetic database with 63,000 records. The evaluation, conducted via a FastAPI platform, revealed that Llama 3.1 8B achieved the highest SQL compliance, while Qwen 2.5 Coder 7B excelled in text similarity and factual consistency, indicating that general-purpose LLMs can outperform specialized models in this domain. These findings suggest that while local, GxP-aligned systems are viable on consumer-grade hardware, they still necessitate human oversight for compliance in regulated environments.