Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey

The survey presents a constraint-centric framework for improving efficiency in large language model (LLM) training by addressing the interconnected challenges of data efficiency, memory efficiency, and compute budget awareness. It reviews advanced techniques for data selection and pruning, highlights the dominance of GPU memory as a bottleneck, and emphasizes the need for coordinated optimization of weight storage, optimizer states, and activation memory. This unified approach offers practitioners guidelines for resource-conditioned decision-making, optimizing training and inference processes within finite computational budgets.