Your Consultants Have Companions. There’s HOPE for Protecting Them Collectively

Larger-Order Pruning of Consultants in Combination-of-Consultants Language Fashions

Introduction

Combination-of-Consultants (MoE) fashions have enabled state-of-the-art LLMs to be scaled to tons of of billions of parameters, however they arrive with a basic and dear caveat: each professional MLP should occupy reminiscence at inference time, regardless that the overwhelming majority are idle for any given token. For instance, Qwen3.5-122B has 256 specialists in every of its 48 MoE layers, and 95% of its ~250 GB parameter reminiscence footprint is taken up fully by these professional MLPs alone. Nevertheless, for any given token, solely 8 specialists in every layer are literally used.

Knowledgeable pruning is likely one of the most direct fixes to this reminiscence bottleneck. That’s, completely delete the least helpful specialists. Pruning avoids the latency points which make dynamic loading/off-loading of specialists impractical; it doesn’t require any coaching; and it composes with quantization. Given these enticing advantages and the simplicity of the thought, it is simple to see why professional pruning has grow to be a go-to method for shrinking MoEs. The issue lies in deciding which specialists are most secure to take away.

Current pruning strategies rating every professional independently, assigning each professional a scalar significance rating (primarily based on a calibration dataset), and slicing out the least necessary specialists (in every layer independently). These are first-order strategies, as a result of they assume every professional’s contribution is impartial of all others. In actuality, specialists kind cooperative teams. We observe constant clusters of co-activated specialists in our personal experiments, and impartial work has proven that professional cooperation is a steady property of MoEs and is necessary for mannequin efficiency. Ignoring these interactions results in suboptimal pruning, particularly at aggressive charges.

Our latest paper derives HOPE (Larger-Order Pruning of Consultants), a second-order methodology that accounts for pairwise professional interactions. At 50% pruning, HOPE achieves the most effective common rank of any methodology, with positive factors of as much as +6.1% on agentic coding duties over the next-best pruning methodology.

On this submit, we clarify how HOPE works, present the place it outperforms present strategies, and stroll by way of how you can use it in your personal fashions. For readers unfamiliar with MoE architectures, see Combination of Consultants (MoEs) in Transformers.

Consultants do not work alone

In a normal MoE, every layer has $EE$ specialists, and every enter token $xx$ is routed independently to $\mathrm{Okay}Okay$ such specialists ( $\mathrm{Okay}=8Okay\; =\; 8$

Left: typical MoE layer, the place every token is routed to the top-Okay specialists. Proper: first-order strategies rank and delete specialists primarily based on a scalar significance rating, however the most effective pruning choices will even think about the power of interactions between specialists.

Importantly, specialists within the chosen set $T(x)T(x)$ aren’t impartial. Consultants kind steady cooperative teams that get persistently co-activated collectively. This has been noticed earlier than within the literature, as nicely: professional co-selection patterns emerge early in coaching and stabilize all through.

Because of this after we prune specialists from an MoE layer, pruning choices want to contemplate the higher-order results between these specialists. Eradicating professional A may appear nice by itself, but when professional B—which ceaselessly cooperates with and depends on A—is extremely necessary, then eradicating A would dismantle a cooperative group simply by way of a single deletion. First-order strategies miss this fully, as a result of they rating A and B individually.

This impact compounds at excessive pruning charges. After we take away as much as 50% of specialists, we danger breaking many cooperative hyperlinks. That is the place second-order pruning is most advantageous.

How HOPE identifies a prune-set

(See Appendix A in our corresponding paper for the complete derivation of the HOPE goal)

After we delete a set of $PP$ specialists from some MoE layer, the chosen set of specialists $T(x)T(x)$ (for any token $xx$) can change. The pruned specialists that would have been chosen are changed by the specialists with the next-highest gate weights. The ensuing change in layer output is the pruning error. Our aim is to seek out the prune-set $PP$ (of some predefined pruning funds $\mathrm{\mid }P\mathrm{\mid }vert\; Pvert$) that minimizes the ensuing pruning error (over tokens $xx$ from our calibration set).

Sadly, minimizing the pruning error over $PP$ straight is intractable. The pruning error is determined by the identities of the alternative specialists, which in flip rely upon the prune-set $PP$ itself. HOPE sidesteps this by defining a tractable surrogate $ZZ$, which strictly upper-bounds the dominant part of the pruning error:

$$Z=[\sum _{j\in P\cap T(x)}{g}_j(x)\mathrm{\parallel }{f}_j(x){\mathrm{\parallel }}_2{]}^2\ge \text{true error}Z\; =\; bigl[sumlimits\_\{j\; in\; Pcap\; T(x)\}\; g\_\{j\}(x)Vert\; f\_j(x)Vert\_\{2\}bigr]^\{2\}\; geq\; textual\; content\{true\; error\}$$

As a result of $ZZ$ is a squared sum, increasing it naturally produces cross-terms $g_i(x)g_j(x)\mathrm{\parallel }{f}_i(x){\mathrm{\parallel }}_2\mathrm{\parallel }{f}_j(x){\mathrm{\parallel }}_{2}g\_\{i\}(x)g\_\{j\}(x)Vert\; f\_\{i\}(x)Vert\_\{2\}Vert\; f\_\{j\}(x)Vert\_\{2\}$

To seek out the optimum prune-set, HOPE minimizes $\mathbb{E}[Z]mathbb\{E\}[Z]$ over the calibration set, which could be reformulated as the next quadratic program:

$$P^{\ast }=\underset{p}{\mathrm{arg\hspace{0.17em}min}}{p}^{\mathrm{\top }}Fp\text{s.t.}p\in \{0,1{\}}^E,\sum _{\mathrm{ok}=1}^E{p}_{\mathrm{ok}}=\mathrm{\mid }P\mathrm{\mid }P^\{*\}\; =\; argmin\_\{p\};\; p^\{high\}Fp\; quad\; textual\; content\{s.t.\}\; quad\; p\; in\; \{0,1\}^\{E\},\; sumlimits\_\{ok=1\}^\{E\}p\_\{ok\}\; =\; vert\; Pvert$$

the place $F\in {\mathbb{R}}^{E\times E}Finmathbb\{R\}^\{Etimes\; E\}$

$$F_{i,j}=\frac{1}{\mathrm{\mid }{\mathcal{X}}_{i,j}\mathrm{\mid }}\sum _{x\in {\mathcal{X}}_{i,j}}{g}_i(x)g_j(x)\mathrm{\parallel }{f}_i(x){\mathrm{\parallel }}_2\mathrm{\parallel }{f}_j(x){\mathrm{\parallel }}_2\text{the place}{\mathcal{X}}_{i,j}=\{x:i,j\in T(x)\}F\_\{i,j\}\; =\; frac\{1\}\{vert\; mathcal\{X\}\_\{i,j\}vert\}\; sumlimits\_\{xinmathcal\{X\}\_\{i,j\}\}\; g\_\{i\}(x)g\_\{j\}(x)Vert\; f\_\{i\}(x)Vert\_\{2\}Vert\; f\_\{j\}(x)Vert\_\{2\}quadtext\{the\; place\}quad\; mathcal\{X\}\_\{i,j\}\; =\; \{x:\; i,j\; in\; T(x)\}$$

This F-matrix is collected on the calibration dataset, and the optimum prune-set is obtained by fixing the ensuing quadratic program (QP) with a steady leisure (then rounding to binary options). Every entry $F_{i,j}F\_\{i,j\}$

What does $FF$ appear to be in apply? It reveals the construction of professional cooperation, which is determined by layer:

Visualization of the interplay matrix F for GLM-4.5-Air (128 specialists per layer, 106B parameters whole) at three consultant layers (early, center, late). Consultants have been clustered to disclose block construction. Vivid blocks alongside the diagonal point out teams of specialists with sturdy pairwise interplay scores.

By way of sensible value, HOPE requires the identical calibration move as every other expert-pruning methodology (one ahead move over calibration knowledge). The one addition is fixing the quadratic program, which is non-convex, however could be solved effectively by way of a normal QP solver (1–2 seconds every layer). The whole extra overhead is negligible in comparison with first-order strategies, because the time for calibration dominates.

Connection to REAP

Notably, HOPE has a powerful connection to Router-weighted Knowledgeable Activation Pruning (REAP), a state-of-the-art first-order expert-pruning methodology. HOPE’s F-matrix encodes particular person professional significance scores on the diagonal, and this diagonal successfully recovers the scores utilized by REAP to independently rank specialists for pruning (differing solely by the variance of professional contributions). The diagonal of $FF$ recovers REAP’s professional rankings nearly completely (Spearman $\rho =0.988rho\; =\; 0.988$

Outcomes

We evaluated HOPE as an professional pruning methodology utilizing 3 fashions (Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air), with 2 calibration datasets (Evol-CodeAlpaca and SWE-Bench Verified trajectories). We used the Tulu3 suite and SWE-Bench Professional (agentic coding) to check the ensuing pruned fashions, evaluating HOPE to 4 first-order baselines (REAP, MAN, EAN, and Frequency) throughout pruning charges spanning from 10% to 50%.

HOPE achieves the most effective general rank throughout circumstances and dominates at aggressive pruning charges. Left: common rank (decrease = higher) throughout all fashions, calibration units, and benchmarks. Proper: SWE-Bench Professional efficiency on Qwen3.5-35B.

Throughout all circumstances, HOPE achieved the most effective common rank (2.07 out of 5 strategies), with the very best fee of top-1 finishes (39% of circumstances) and top-2 finishes (67%). In a head-to-head comparability, HOPE beats each different baseline within the majority of circumstances.

As anticipated, HOPE’s benefit grows at increased pruning charges. As extra specialists are eliminated, extra cooperative hyperlinks are damaged, and the off-diagonal phrases of $FF$ grow to be extra necessary. At 50% pruning, HOPE’s common rank improves to 1.58 (vs. 2.42 for the next-best methodology, REAP). That is additionally essentially the most deployment-relevant setting, the place reminiscence financial savings are massive sufficient to accommodate longer contexts, deeper reasoning, and extra device calls.

HOPE’s benefit is most pronounced on SWE-Bench Professional, an agentic coding benchmark the place fashions should resolve actual GitHub points autonomously. These issues contain compositionally various duties (e.g. coding, device calls, reasoning, and many others.) over many turns and certain train various professional combos—precisely the regime the place preserving cooperative construction issues most. On Qwen3.5-35B-A3B at 50% pruning, HOPE achieved 42.3 on SWE-Bench Professional, matching the unpruned mannequin’s 41.8 (whereas REAP dropped to 36.2, a 13% regression).

Notably, pruned fashions can sometimes match or exceed the unpruned baseline, probably as a result of the removing of low-relevance specialists reduces noise on the goal duties. This impact just isn’t distinctive to HOPE and has been noticed throughout pruning strategies. Moreover, our experiments confirmed that code-focused calibration with any pruning methodology (together with HOPE) didn’t severely degrade non-coding benchmarks, and HOPE carried out nicely throughout the Tulu3 suite no matter calibration set.

HOPE retains extra cooperating specialists

Common power of cooperation (measured by F-matrix worth) with retained specialists, break up by specialists on which HOPE and REAP disagree.

HOPE’s benefit stems from optimizing over your entire interplay matrix $FF$. To know how this impacts pruning choices in apply, we look at the specialists the place HOPE and REAP disagree. For every disputed professional, we compute its cooperative rating (i.e. its common F-value with different retained specialists). Consultants retained solely by HOPE persistently have increased cooperative scores than specialists retained solely by REAP. In different phrases, as a part of the QP optimization, HOPE sacrifices some individually high-scoring specialists in favor of preserving cooperative clusters. The positive factors in benchmark efficiency (proven above) validate that this trade-off is advantageous.

How one can run HOPE

We created a primary bundle to run HOPE. It has three steps: 1) calibrate; 2) resolve; 3) prune. Our bundle has been examined on Qwen3, Qwen3-Subsequent, Qwen3.5, and GLM-4.5.

Step 1: Calibrate

hope calibrate 
    --model-path Qwen/Qwen3.5-35B-A3B 
    --prompts calibration_prompts.txt 
    --out-path observations.h5

The calibration step runs a single ahead move over calibration knowledge, whether or not it’s agentic rollouts or single-turn traces. This makes use of HuggingFace transformers to hook into the ahead move of every particular person MoE layer. For every token, this information which specialists are chosen, their gate weights, output norms, and pairwise co-activation statistics. The output of this step is the F-matrix, saved in an HDF5 file.

Step 2: Resolve QP

hope resolve 
    --obs-path observations.h5 
    --prune-frac 0.50 
    --out-path pruneset.json

This command materializes the F-matrix and solves the binary quadratic program by way of a steady leisure. This takes just a few seconds per layer, and the output is a JSON file which accommodates the prune-set. That is technically the ultimate output of the HOPE algorithm.

Step 3: Prune

hope prune 
    --model-path Qwen/Qwen3.5-35B-A3B 
    --pruneset-path pruneset.json 
    --out-path ./Qwen3.5-35B-A3B-pruned-50

To finish the pipeline, we run this command to really create the smaller mannequin checkpoint from the prune-set. The pruned mannequin is a normal HuggingFace checkpoint which is smaller on disk and in reminiscence (no modifications to vLLM or SGLang mannequin definitions wanted).

We additionally launched a extra compact Python API:

from hope.calibrate import calibrate
from hope.resolve import resolve
from hope.prune import prune_model

calibrate("Qwen/Qwen3.5-35B-A3B", calibration_prompts, "observations.h5")
resolve("observations.h5", prune_frac=0.50, out_path="pruneset.json")
prune_model("Qwen/Qwen3.5-35B-A3B", "pruneset.json", "./Qwen3.5-35B-A3B-pruned-50")

Limitations of HOPE

HOPE’s benefit is most pronounced at increased pruning charges (40–50%). At decrease pruning charges (10–20%), first-order strategies like REAP and MAN are generally aggressive, and HOPE’s benefit shrinks. Moreover, as with most expert-pruning algorithms, HOPE prunes every layer independently and doesn’t account for cross-layer interactions. In our paper, we offered a theoretical framework for cross-layer HOPE, however the outcomes stay preliminary. Lastly, HOPE captures pairwise interactions, however even higher-order interactions (triplets and past) might present extra sign, though they’re combinatorially troublesome to estimate.

Extra data

For extra data, see our corresponding paper and codebase.

Paper: https://arxiv.org/abs/2609.18916

Code: https://github.com/awslabs/HOPE

To see how HOPE could be utilized to next-generation hybrid architectures, see our corresponding repository.

Quotation:

@article{tseng2026hope,
    title={Larger-Order Pruning of Consultants in Combination-of-Consultants Language Fashions},
    creator={Tseng, Alex M. and Kaul, Prannay and Zancato, Luca and Xia, Wei and Soatto, Stefano},
    journal={arXiv preprint arXiv:2609.18916},
    yr={2026},
}