Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

The paper investigates the in-context retrieval capabilities of Transformers, State Space Models (SSMs), and hybrid architectures, focusing on two tasks: n-gram retrieval and position retrieval. Hybrid models demonstrate superior data efficiency and extrapolation capabilities compared to SSMs and match or exceed Transformers in specific retrieval tasks, although Transformers retain an edge in position retrieval. The study highlights the differences in how these architectures learn positional associations, with SSMs developing locality-aware embeddings, while Transformers leverage causal attention and positional encodings for improved data efficiency.