Abstract representational geometry supports inference in large language models

The paper presents research demonstrating that large language models (LLMs) exhibit abstract representational geometry akin to that found in the human hippocampus, which aids in inference tasks. The study reveals that LLMs, while less frequently generalizing reasoning than humans, show hierarchical organization in their internal states, with lower layers encoding stimulus identity and higher layers forming abstract context geometry. This geometric structure is shown to be crucial for reasoning, as interventions like task-sequence language modeling and geometric regularization enhance inference capabilities, highlighting the importance of representational geometry in LLM functionality.