Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Families

This study analyzes the impact of Rotary Position Embeddings (RoPE) on retrieval heads in various open-weight models (7-8B parameters) by testing their functionality against varying frequencies defined by a hyperparameter theta. The findings reveal that retrieval heads are essential for long-context recall, with masking leading to a complete loss of recall, and that increasing theta does not hinder the formation of retrieval heads, contradicting previous hypotheses. The research provides insights into the nuanced relationship between RoPE frequency and retrieval head performance, which is crucial for practitioners optimizing LLM architectures for better context recall. Code and a paired-seed testing framework have been made available for further experimentation.