UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

The paper introduces Uncertainty-Balanced Preference Planning (UBP2), a model-based approach for preference-based reinforcement learning that improves sample efficiency by actively directing exploration through uncertainty reasoning in reward, dynamics, and value functions. UBP2 employs ensembles to evaluate candidate trajectories based on a unified score that incorporates expected reward and epistemic uncertainty, achieving sublinear regret guarantees in both finite and infinite horizons. Empirical results demonstrate that UBP2 significantly outperforms existing model-free and non-optimistic model-based methods on the Meta-World benchmark, making it a valuable tool for practitioners focused on efficient exploration in reinforcement learning tasks.