Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

A new empirical study finds that Mixture-of-Experts (MoE) language models do not always deliver faster or cheaper inference than comparable dense models on consumer and edge hardware, challenging a common assumption about their efficiency [1]. The research, conducted by Alfarizy Alfarizy and posted to arXiv, benchmarks OLMoE-1B-7B, a model with 1.3 billion active parameters out of a total of 6.9 billion, against three dense baselines [1]. Tests were run on an Apple M2 Pro laptop and an NVIDIA Jetson Orin Nano 8 GB edge device using the llama.cpp inference framework, measuring throughput, memory consumption, and on-device energy [1]. The performance gap was device-dependent. On the laptop, OLMoE-1B-7B ran roughly 10% behind the similarly sized dense model Llama-3.2-1B [1]. On the edge device, the gap widened significantly, with the MoE model falling 31% behind the dense baselines while consuming 2.1 times the energy per token and hitting the device's 8 GB memory ceiling [1]. To isolate the cause, the researchers patched llama.cpp to time the decode graph node-by-node. They found that the routing mechanism itself accounted for under 9% of the MoE-block compute on the edge device's backend [1]. The primary bottlenecks were instead the total-parameter memory footprint, expert dispatch overhead, and key-value cache pressure [1]. The study concludes that on bandwidth-bound edge hardware, inference cost tracks total parameters, not active ones, and that sparse activation does not compensate for the device's primary constraints [1]. These findings are bounded to a single MoE model at this parameter scale and two specific devices, and the authors have released their full measurement harness and per-run data [1]. The results add a practical counterpoint to the theoretical efficiency of MoE architectures, which are designed to activate only a small subset of experts per token to reduce floating-point operations [2]. The work also arrives amid broader scrutiny of GPU and accelerator behavior in AI workloads, including recent research into hardware-level vulnerabilities on Apple, NVIDIA, and Qualcomm products that can leak intermediate neural network data [4]. Nvidia, whose Jetson Orin Nano was used in the study, remains the dominant supplier of GPUs for AI training and deployment, controlling more than 80% of that market as of 2025 [5].