EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

The article introduces EMAgnet, a novel regularization method for policy gradient self-play in large games, which uses an exponential moving average (EMA) of the last-iterate policy's parameters as a dynamic target for regularization. This approach adapts to the agent's evolving strategy, leading to improved performance over traditional uniform distribution targets, particularly in two-player zero-sum games with exploration challenges and dominated strategies. EMAgnet demonstrates lower exploitability compared to PPO with uniform regularization, making it a significant advancement for practitioners working on reinforcement learning in complex game environments.