Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

This paper evaluates the inference performance of large language models (LLMs), specifically Llama2-7B, across GPUs and emerging AI accelerators by analyzing Prefill and Decode phases separately. The study reveals that GPUs outperform in the compute-intensive Prefill phase, while GroqRack offers lower time per output token during the Decode phase, although it lacks batching support. These insights are crucial for practitioners as they highlight the importance of phase-dependent performance characteristics, guiding the selection of hardware for specific LLM workloads.