DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

DynamicMem is a newly introduced benchmark designed to evaluate the memory capabilities of LLM agents over long periods, simulating 15 months of user activity across 16 applications with an average of 2.2 million tokens and 1,772 grounded events per user. It addresses critical shortcomings in existing benchmarks by incorporating heterogeneous profiles that evolve over time due to external contexts, requiring systems to infer user attributes, habits, and preferences from dispersed signals. The evaluation of five systems revealed that while service-task accuracy remained stable, profile reconstruction suffered as history length increased, highlighting significant challenges in memory retrieval that need to be addressed for improved long-term user interaction.