Words as Difference Makers: How Large Language Models Determine Causal Structure in Text

The paper discusses how large language models (LLMs) utilize a difference-making logic, also known as variational induction, to infer causal structures from text. It critiques traditional causal inference frameworks, highlighting that LLMs require extensive and diverse text data to identify causal relationships effectively. The study also examines specific architectural features, such as token embeddings and self-attention mechanisms, which facilitate this inductive reasoning process, providing insights valuable for practitioners aiming to enhance LLM capabilities in causal reasoning tasks.