LLM Parameters for Math Across Languages: Shared or Separate?

The study presents a cross-lingual analysis of mathematical reasoning in large language models (LLMs), revealing that mathematical reasoning parameters exhibit partial overlap across languages, primarily concentrated in intermediate layers of the models. Notably, English LLMs show a larger set of math-relevant parameters compared to lower-resource languages, indicating that multilingual LLMs have a hybrid mechanism for mathematical reasoning that is neither entirely language-invariant nor fully language-specific. This insight is crucial for practitioners as it informs the design of LLMs for improved performance in multilingual contexts, particularly in mathematical tasks.