Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

Nightjar introduces a dynamic adaptive speculative decoding framework for large language models, addressing the limitations of existing speculative decoding methods that struggle under varying load conditions. By dynamically adjusting speculative lengths and disabling speculation when it becomes counterproductive, Nightjar enhances throughput by up to 14.76% and reduces latency by 20.18% in real-time serving scenarios. This approach allows practitioners to optimize resource utilization in memory-bound environments, improving the efficiency of LLM inference under diverse workloads.