Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference

The paper presents a novel approach to speculative decoding for low-latency inference of large language models (LLMs) in edge-cloud environments, introducing a method called UCB-SpecStop that dynamically adjusts the draft length based on communication delays. The study formulates the draft length tradeoff as an optimal stopping problem and establishes a state-dependent threshold policy for varying network conditions, demonstrating that UCB-SpecStop can reduce per-token latency by up to 22.4% compared to existing methods. This advancement is significant for practitioners as it enhances the efficiency of LLM inference in real-time applications by optimizing communication resources and adapting to network variability.