SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

SMEPilot is a newly introduced LLM inference engine that optimizes the execution of matrix operations on CPUs with Arm Scalable Matrix Extensions (SME). It employs a roofline-based characterization to determine optimal execution strategies—selecting between CPU-only, SME-only, or a cooperative SME+CPU approach for different operator shapes, resulting in up to 3.94x improvements in inference performance across models like Llama-3.2-3B and Qwen3-4B on various platforms. This optimization is crucial for practitioners as it enhances the efficiency of LLM inference on hardware that integrates SME, addressing the specific arithmetic and memory bandwidth challenges encountered in LLM workloads.