A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget

This study presents an analysis of training dynamics in a small Llama-style language model with 4.26 million parameters, trained under a compute-constrained token budget of approximately 20 million tokens using the TinyStories corpus. Key findings include significant changes in validation loss and perplexity across training intervals, with initial improvements followed by later degradation, indicating that traditional endpoint metrics may obscure important stability issues and diminishing returns in constrained compute environments. These insights highlight the need for practitioners to adopt a more nuanced approach to evaluating language model training by focusing on interval-level telemetry rather than solely on final performance metrics.