Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

The article introduces STREAM (Structural-Temporal Rhythmic Energy-based Attention for Motion), a novel diffusion transformer designed for choreographic motion generation that effectively separates conditioning pathways for text and music. It utilizes Adaptive Layer Normalization (AdaLN) for kinematic structure control and a Bimodal Energy-Based Attention Module (BEAM) to align musical beats without compromising semantic integrity. The accompanying Motorica++ dataset enhances training with domain-specific vocabulary and annotations, while the Exchange Evaluation Protocol and Editable Dance Score (EDS) provide metrics for evaluating zero-shot editability, making STREAM a significant advancement for practitioners focused on controllable AI in artistic applications.