When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model

The study evaluates the efficacy of large language models (LLMs), specifically LLM-OptFlow, as hyperparameter optimization (HPO) advisors on tabular data across eight benchmarks. It finds that the initial strong performance of the LLM is primarily due to a fixed default configuration rather than the model's outputs, yielding only marginal improvements in cross-validation accuracy. For practitioners, the results suggest that classical search methods seeded with sensible defaults may be more effective than LLM-based approaches for HPO in tabular settings, as LLMs do not provide significant generalization benefits and can be outperformed in a limited number of evaluations.