OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data

OmniDirector is a new framework for multi-shot camera cloning in video generation that utilizes a novel camera motion representation, encoding cameras as grid motion videos. This approach allows for the integration of diverse trajectories and is trained on a million-scale dataset of camera grid-video pairs, enhancing control over characters, actions, and camera movements. Its hierarchical prompt expansion agent improves the integration of control signals, significantly boosting performance and controllability in multimodal diffusion transformers, which is crucial for practitioners developing sophisticated video generation systems.