Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

The paper introduces KDoS (Knowledge Distribution-optimized Synthesis), a framework for enhancing knowledge injection in Large Language Models (LLMs) by optimizing knowledge distribution during synthetic data generation. It employs a three-stage feedback mechanism to shift from traditional synthesis methods to a distribution-aware approach, demonstrating that an optimal knowledge distribution can significantly expand knowledge boundaries across models ranging from 0.6B to 16B parameters (including Qwen, Ling, and LLaMA) and varying data scales of 1B to 5B tokens. This methodology consistently outperforms existing baselines across six knowledge benchmarks, providing a new practical framework for practitioners focused on improving LLM performance through synthetic data.