Beyond Memorization: Distinguishing Between Pattern-Based and Epistemic Reasoning in LLMs Using Epistemic Puzzles

The paper introduces a two-dimensional benchmark for evaluating LLMs on DEL-style epistemic puzzles, distinguishing between pattern-based reasoning and true epistemic reasoning. The findings indicate that while models exhibit robustness to surface form changes, they struggle significantly in asymmetric scenarios requiring the tracking of fragmented epistemic states. This distinction is crucial for practitioners as it highlights the limitations of current LLMs in handling complex reasoning tasks, informing future improvements in model architectures and evaluation methodologies.