Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

The paper presents a reinforcement learning framework for Computer-Use Agents (CUAs) that utilizes autonomous vision-language evaluation as a scalable supervision signal, addressing the challenge of sparse reward signals in open-ended desktop environments. By modeling the imperfect feedback from a Vision-Language Model as a noisy binary reward channel, the authors implement a noise-corrected reward estimator for Proximal Policy Optimization, resulting in an average improvement of 12.6 percentage points in success rates over zero-shot performance. This approach highlights the potential of autonomous evaluation as a viable reward mechanism for training RL agents in graphical user interfaces, particularly when noise is accounted for in the reward estimation process.