Inverting the Bellman Equation: From $Q$-Values to World Models

The paper presents a novel approach called $P$-learning, which serves as an inverse analogue to $Q$-learning, enabling the extraction of a world model from value-based agents trained on diverse reward functions. It establishes that agents can implicitly encode the transition kernel $P$ under certain conditions, demonstrating this with empirical results on environments like $\texttt{Reacher}$ and $\texttt{MountainCar}$. This work bridges model-based and model-free reinforcement learning, revealing hidden generalization capabilities in agents and suggesting new avenues for improving goal-conditioned RL strategies.