MoE models decouple capacity from compute by activating only a fraction of parameters per token, but when the expert parameters live on host memory and the GPU is waiting on PCIe, you have traded a compute bottleneck for a memory bus bottleneck. SPICE is a speculative prefetching framework that treats expert routing prediction as a branch predictor problem: build a lightweight draft model aligned with the target MoE architecture, use confidence-aware lookahead to prefetch high-confidence experts before they are needed, and when the prediction misses, patch the gap with low-rank expert (LoRE) surrogates executing asynchronously on CPU while the GPU continues with committed work.
On DeepSeek-V2-Lite and Qwen2-57B-A14B across consumer and data center GPU platforms, SPICE delivers up to 3.12x speedup in Time Per Output Token with minimal quality loss. The LoRE fallback is the design choice that distinguishes this from prior prefetching work: instead of stalling on a miss, the system approximates with a rank-reduced variant and offloads the residual computation. That keeps the GPU execution unit busy even when the speculation is wrong.
For teams running large MoE models on hardware where GPU HBM cannot hold the full expert set, this is now the architecture to evaluate. The approach is model-agnostic and platform-agnostic; the draft model is aligned once to the target MoE structure and reused. The real question is where LoRE approximation quality degrades enough to matter at production inference thresholds, and the paper does not bury that number: quality loss on the benchmarks is minimal. Vendors selling HBM capacity as the answer to MoE inference cost have a 12-18 month window before speculative offloading becomes the cheaper counter-argument.