Randomized YaRN Improves Length Generalization for Long-Context Reasoning

The article introduces Randomized YaRN, a novel training method that enhances length generalization in large language models (LLMs) by integrating YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. Evaluated on the BABILong and Multi-Round Coreference Resolution benchmarks, Randomized YaRN demonstrates significant improvements in reasoning performance on context lengths ranging from 16K to 128K when trained on data with less than 8K context, outperforming traditional fine-tuning methods. This approach highlights the importance of exposing models to out-of-distribution positional representations to achieve effective long-context reasoning, which is crucial for practitioners developing LLMs for tasks requiring extensive context.