Chiaroscuro Attention: Spending Compute in the Dark

The article introduces CHIAR-Former, a transformer architecture that utilizes Chiaroscuro Attention to dynamically route tokens between DCT spectral mixing and full self-attention based on their spectral entropy. This model, with 400M parameters, achieves a 35-40% reduction in FLOPs while demonstrating improved perplexity on the WikiText-103 dataset (27.51 vs. 23.58), highlighting its efficiency in handling varying complexity in token embeddings. The findings suggest that the MetaRouter effectively balances computational efficiency and representational capacity, making it a significant advancement for practitioners optimizing LLM performance.