Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

The article introduces the "Age of LLM," a new turn-based benchmark designed for evaluating the reasoning, diplomacy, and reliability of large language models (LLMs) in a competitive setting on a 13x7 grid. It emphasizes the impact of fog of war and strict JSON schema adherence, benchmarking 15 models across 54 matches and 5,258 actions, revealing insights such as the dominance of nuclear strategies and the relationship between reliability and winning outcomes. This benchmark provides a unique framework for practitioners to analyze LLM behavior under adversarial conditions, particularly in terms of belief tracking and cognitive strategies, with resources available for further exploration.