NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

NL2Scratch is introduced as an executable benchmark for natural-language-to-Scratch code generation, featuring 311,648 parser-valid program pairs derived from actual Scratch projects paired with aligned natural language descriptions. It employs a new metric, Semantic Alignment Consistency (SAC), to evaluate the semantic agreement between descriptions and programs, revealing significant discrepancies between lexical similarity and semantic alignment in instruction-tuned and fine-tuned LLMs. This benchmark is crucial for practitioners as it highlights the limitations of traditional evaluation metrics and provides insights into common failure modes in NL2Code tasks, particularly in handling operational slots.