A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction

The article presents a framework for pre-hoc performance prediction in fine-tuning large language models (LLMs), addressing the economic challenges associated with fine-tuning costs. It formulates the prediction as a stochastic estimation problem, decomposing risk into intrinsic limits and optimization variance, and establishes a lower bound on the decay rate of uncertainty. The introduction of a budget-optimal probing principle and a predictability phase diagram categorizes tasks into three regimes, validated through extensive experiments, providing practitioners with insights to optimize fine-tuning strategies effectively.