When Researchers Say Mental Model/Theory of Mind of AI, What Are They Really Talking About?

The paper discusses the misconception surrounding the concept of Theory of Mind (ToM) in AI, arguing that claims of AI possessing ToM are based on behavioral predictions rather than true cognitive states. It highlights that while recent studies indicate LLMs can perform at human levels on ToM tasks, this reflects pattern matching rather than genuine understanding. The author advocates for a shift towards mutual ToM frameworks that consider the interactive dynamics between human cognition and AI, rather than evaluating AI in isolation.