Tapered Language Models

The paper introduces Tapered Language Models (TLMs), which propose a non-uniform allocation of parameters across model depth, focusing more capacity on earlier layers. Experiments demonstrate that tapering the width of Multi-Layer Perceptrons (MLPs) using a cosine schedule improves perplexity and downstream performance across various architectures, including Transformers and Gated Attention, without increasing parameter count or computational cost. This approach highlights the potential for depth-aware capacity allocation as a significant design consideration in language model development.