The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model

A new framework for modeling the energy consumption of Transformer training on multiple GPUs has been introduced, focusing on BERT models. This framework utilizes architectural sweeps to correlate energy usage with proxies for compute, memory traffic, and hardware efficiency, incorporating a roofline-inspired model that accounts for tensor parallelism and fully sharded data parallelism. This approach is significant for practitioners as it provides a predictive model for energy costs, aiding in the design of sustainable and cost-effective AI systems.