Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning

The paper introduces Coherence Under Commitment (CUC), a dual-query evaluation paradigm aimed at addressing the issue of vacuous coherence in large language models (LLMs) during logical reasoning tasks. Key innovations include a commitment score to quantify decisiveness, a deterministic elicitation protocol to reduce sampling variance, and a three-way decision framework that integrates coherence and commitment metrics. Experiments on four open-weight LLMs (1B-3B) reveal significant discrepancies between coherence and utility, highlighting the need for evaluations that reflect both aspects, along with the release of a toolkit for standardized assessment.