MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations

MuPPET (Multi-Party Privacy Exposure Testing) is a newly introduced benchmark that evaluates contextual privacy risks for LLM agents in multi-party conversations, addressing a gap in existing benchmarks that focus solely on single-interlocutor settings. The findings indicate that models exhibit significantly higher privacy leakage in multi-party contexts compared to one-on-one interactions, with both advanced and smaller models showing vulnerabilities. This benchmark is crucial for practitioners as it highlights the inadequacy of current privacy defenses in multi-party environments and underscores the need for improved strategies to manage sensitive data exposure in group settings.