SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

SpotAttention is a novel plug-in block-sparse routing mechanism for pretrained long-context transformers, designed to enhance computational efficiency by selecting a relevant subset of past tokens for attention. It utilizes a lightweight selector that learns attention distribution through KL distillation, achieving competitive accuracy with dense attention models like Qwen3 and Qwen3.5 while processing contexts up to 128K tokens, resulting in decoding speeds 3.9x faster than FlashAttention. Additionally, quantization of the selector's K-cache to INT4 or FP4 reduces its size by 3.5x without sacrificing accuracy, making it a valuable tool for practitioners aiming to optimize LLM performance in resource-constrained environments.