Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test

The paper introduces a new architecture called Ledger Residuals, which separates the residual stream into a mutable scratch stream and a protected decode-only accumulator to test the hypothesis regarding massive activations in transformers. Experiments with models of 160M and 290M parameters show that massive activations persist even when the architecture is modified, suggesting that these activations are functionally necessary rather than mere artifacts. This finding is significant for AI practitioners as it indicates that architectural changes may not eliminate massive activations, which could impact model performance and efficiency in language tasks.