Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

The study benchmarks the Mixture-of-Experts (MoE) model OLMoE-1B-7B (1.3B active of 6.9B total parameters) against dense models on consumer and edge hardware, specifically an Apple M2 Pro and an NVIDIA Jetson Orin Nano. Results indicate that while MoE models theoretically reduce per-token compute costs, in practice, they underperform compared to dense models due to factors like total memory footprint and expert dispatch, with OLMoE being 10% slower on the laptop and 31% slower on the edge device. This research highlights that on bandwidth-constrained hardware, inference costs are more influenced by total parameters rather than active ones, suggesting that sparse activation may not significantly improve efficiency in such environments.