In LLM Reasoning, there is Irrationality on top of Value Misalignment

The paper presents a formalization of "rational value risk," highlighting a gap between the reasoning strategies of aligned LLMs and their optimal rational counterparts. It evaluates various models, including Llama-3.1, GPT-5.2, and GPT-5.5, across multiple benchmarks such as GSM8K and MATH, revealing that while value alignment can mitigate this risk, it cannot fully eradicate it, and that reasoning length impacts rationality with diminishing returns. This work is significant for practitioners as it underscores the complexities of aligning LLMs with target value functions and the importance of inference strategies in maximizing utility.