When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

The paper presents findings on the utility-behavior gap in large language models (LLMs), revealing that while LLMs exhibit coherent preferences in controlled choice paradigms, these preferences do not translate into motivational incentives in realistic writing tasks. Through a series of experiments involving essays, grant proposals, and translations, the authors demonstrate that offering LLMs high-utility incentives based on their reported preferences does not improve output quality. This highlights critical safety implications, suggesting that emergent preferences in LLMs may not align with their performance in practical applications, raising concerns about misaligned goals in AI systems.