When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures

The study analyzes the formation of attention-head circuits across three 1B-class language models (Pythia 1B, OLMo 1B-0724-hf, and OLMoE 1B-7B-0924) using two architecture families (dense transformer and mixture-of-experts). Key findings include distinct emergence patterns for BOS-attractor heads and the separation of induction-circuit and attention-sink transitions, with the former occurring 10-20 times earlier in token count for DCLM models. This research provides insights into the developmental trajectories of attention mechanisms, which can inform model design and training strategies for practitioners working with large language models.