Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers

The article presents a survey on the use of Large Language Models (LLMs) in optimization tasks, categorizing their applications into three paradigms: direct optimization, tool-augmented optimization, and tool-creating optimization. It discusses the performance frontiers of these approaches based on existing benchmarks and highlights a critical reasoning gap in current architectures, emphasizing the trade-offs between the efficiency of direct optimization and the auditability offered by tool-augmented methods. This analysis is significant for practitioners as it informs the choice of optimization strategies when integrating LLMs into real-world problem-solving scenarios.