Grounding Computer Use Agents on Human Demonstrations

The article introduces GroundCUA, a large-scale desktop grounding dataset created from expert human demonstrations, encompassing 56K screenshots and over 3.56M annotations across 87 applications. The GroundNext model family, available in 3B and 7B parameter sizes, achieves state-of-the-art performance on five benchmarks with significantly less training data than previous models, and further refinement through reinforcement learning enhances its capabilities. This work underscores the importance of high-quality, expert-annotated datasets for developing effective computer-use agents that can accurately translate natural language instructions into UI interactions.