PrivacyAlign: Contextual Privacy Alignment for LLM Agents

The article introduces PrivacyAlign, a dataset designed to improve the privacy alignment of AI agents by centering human judgment in training and evaluation. It comprises 1,350 samples and 3,516 annotations from 599 annotators, focusing on scenarios where LLMs may leak private information. The study demonstrates that conditioning LLMs on these human annotations enhances the reliability of privacy judgments and presents a novel annotation-conditioned reward modeling approach, resulting in better alignment with human privacy norms and improved performance on privacy benchmarks. This work is significant for practitioners as it provides a framework to build more trustworthy AI agents that respect user privacy in decision-making processes.