Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?

The article introduces AgentCIBench, a new evaluation framework designed to assess the privacy risks posed by computer-use agents (CUAs) when accessing multiple personal applications. It identifies three critical failure modes leading to privacy leaks and reports that 11 out of 15 evaluated agents exhibited over 50% leakage in these scenarios, with an average leakage rate of 67.9%. This framework aims to promote safer design practices for CUAs by integrating contextual disclosure testing as a necessary pre-deployment safety measure, which is crucial for developers focused on enhancing privacy in AI applications.