TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment

The article presents Trajectory Matching Policy Optimization (TMPO), a novel approach to align diffusion models with downstream tasks, addressing issues of reward hacking and mode collapse in reinforcement learning. TMPO replaces traditional scalar reward maximization with a Softmax Trajectory Balance objective, enabling trajectory-level reward distribution matching to enhance generative diversity by 9.1% compared to existing methods. Additionally, it employs Dynamic Stochastic Tree Sampling to optimize training efficiency by reducing redundant computations, making it a significant advancement for practitioners seeking to improve generative model performance and diversity.