Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters

The article introduces EAGER, a novel code execution framework for LLMs that allows for parallel execution of code generation and execution, significantly reducing end-to-end latency. By implementing a three-stage pipeline of generation, detection, and execution, EAGER utilizes AST-based chunking and dynamic batching to achieve up to 99.8% reduction in non-overlapped execution time and up to 37.3% reduction in overall latency across various benchmarks and LLMs. This advancement is crucial for practitioners aiming to enhance the efficiency of LLM-based applications that require real-time code execution.