Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

The paper argues for a paradigm shift in token reduction within Transformer architectures, advocating that it should extend beyond mere efficiency to become a fundamental principle in generative modeling across vision, language, and multimodal systems. It highlights the potential benefits of token reduction in enhancing multimodal integration, reducing hallucinations, maintaining coherence in long inputs, and improving training stability. The authors propose future directions for research, including algorithm design and reinforcement learning-guided token optimization, which are crucial for practitioners aiming to improve the performance and reliability of generative models.