A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

The paper introduces a reproducible semantic benchmark for translating high-level network intents into multivendor configurations, addressing the challenge of operational correctness in network automation. It evaluates five cloud LLMs across three vendors and five use cases, revealing that semantic quality and operational reliability are independent, with vendor effects significantly influencing outcomes. This benchmark is crucial for practitioners as it enables rigorous comparison of LLM-based network configuration systems, highlighting the importance of repeated-execution metrics in assessing model performance.