Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code

The paper presents a survey on the limitations of autoregressive (AR) models in automated software engineering and explores alternative paradigms such as Diffusion Models, Code World Models (CWMs), and State Space Models (SSMs). Diffusion Models address the shortcomings of AR by enabling holistic denoising for long-range syntactic constraints, while CWMs and SSMs enhance reasoning and efficiency in code generation. This research is significant for practitioners as it highlights potential architectural advancements that could improve code intelligence and reasoning capabilities in AI systems.