Mixture-of-experts models are cheap to run per token but expensive to hold in memory, because every expert has to sit on the GPU even though most are idle at any moment. Qwen3.5-122B, for example, has 256 experts in each of its 48 MoE layers. Amazon researchers now describe a way to delete half of those experts while keeping agentic coding scores intact, which matters to anyone paying for the hardware to host a large open model.
Most pruning methods score each expert on its own and drop the weakest. The Amazon team argues that experts work in stable groups that fire together, so removing one that looks unimportant can break a partner that matters. HOPE, short for higher-order pruning of experts, builds a matrix of how strongly each pair of experts contributes when both are active, then solves a quadratic program to pick which set to remove. The diagonal of that matrix almost exactly reproduces the rankings of REAP, the leading single-expert method, with a rank correlation of 0.988, so any gain over REAP comes from the pair terms.
The team tested three models, Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, with two calibration sets and pruning rates from 10% to 50%, against four simpler baselines. HOPE had the best average rank, 2.07 among five methods, and finished first in 39% of conditions. At 50% pruning its average rank was 1.58 against 2.42 for REAP. The headline case is Qwen3.5-35B-A3B with half its experts gone, scoring 42.3 on SWE-Bench Pro where the full model scored 41.8 and the REAP version fell to 36.2. The solve step takes a few seconds per layer and the output is a standard Hugging Face checkpoint that runs in vLLM or SGLang without changes; code is on GitHub.
For teams self-hosting Qwen or GLM models, this is the kind of change that can move a deployment down a GPU tier. A pruned model that edges past its parent is less surprising than it sounds; the authors note that dropping low-relevance experts can cut noise on a target task, and other pruning methods show the same effect.
The limits are stated plainly. The gains are largest at 40 to 50% pruning and shrink at 10 to 20%, where REAP and others are sometimes competitive. Each layer is pruned on its own, with cross-layer interactions left as preliminary theory. The results come from a company blog post and an arXiv paper, not peer-reviewed work, and the coding calibration means anyone with a different workload should test before trusting the numbers.