L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling

The L20-Edu-135M model, a 134.5M-parameter language model, has been released, trained on approximately 13 billion tokens using a single NVIDIA L20 GPU. Key technical features include data-efficient training techniques like cross-source MinHash/LSH near-deduplication and supervised fine-tuning with weight interpolation, achieving a mean score of 0.4150 on the GSM8K benchmark, which is 87.1% of the performance of larger models SmolLM-135M and SmolLM2-135M. This work highlights the potential for effective small language models in resource-constrained environments, offering insights into their performance relative to larger counterparts.