ForecastBench-Sim: A Simulated-World Forecasting Benchmark

ForecastBench-Sim is a new simulated-world forecasting benchmark that utilizes game rollouts from Freeciv, allowing forecasters to analyze hidden future states based on a structured snapshot of the game. This benchmark enables the generation of continuous or binary forecasting questions, including rare or disruptive outcomes, and features a comprehensive scoring protocol. It is designed to complement real-world benchmarks by facilitating controlled experiments in probabilistic reasoning, which is crucial for practitioners developing AI systems that require dynamic decision-making capabilities.