FactoryLLM: A Safe and Open-Source AI Playground for Evaluating LLMs in Smart Factories

FactoryLLM is an open-source AI platform designed for evaluating retrieval-augmented generation (RAG) models in smart factory environments, focusing on fault diagnostics and recovery across interconnected machines. It allows users to configure LLMs and assess their performance through a dual evaluation setup using RAGAS and NVIDIA's LLM-as-a-Judge metrics, ensuring safety by enabling local execution without exposing sensitive data. A case study demonstrated its effectiveness, with three LLMs achieving groundedness scores above 0.88 on 30 maintenance queries from about 600 pages of documentation, making it a valuable tool for practitioners seeking to enhance document reasoning in industrial applications.