Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

The article introduces RV-Bench, a new evaluation methodology designed to assess large language models' mathematical reasoning capabilities using random variable questions (RVQs). By generating questions with randomized variable combinations that are "unseen" to the models, RV-Bench aims to measure genuine reasoning skills rather than memorization. Experiments conducted on over 30 LLMs with more than 1,000 RVQs indicate a significant proficiency gap between familiar and unseen data distributions, emphasizing the need for improved generalization techniques in mathematical reasoning tasks for LLMs.