Generalized Discrete Diffusion with Self-Correction

The paper introduces the Self-Correcting Discrete Diffusion (SCDD) model, which enhances discrete diffusion models by applying pretraining-based self-correction with explicit state transitions and a simplified training noise schedule. It eliminates the need for a remasking step and employs uniform transitions to improve parallel decoding efficiency at the GPT-2 scale without sacrificing generation quality. This advancement is significant for practitioners as it addresses challenges in hyperparameter tuning and reasoning performance in existing self-correction techniques.