acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Expert Pruning for Multi-Agent Systems

14 views

Abstract

Mixture-of-Experts (MoE) architectures scale language models efficiently by activating only a few experts per token, but the saving is confined to computation: every expert must stay resident on the accelerator, so memory bounds where these models can be deployed. Expert pruning reduces this footprint, yet existing methods are static — a single mask, calibrated offline, is applied to every subsequent request. This assumption can fail when the workload is heterogeneous, most prominently in multi-agent systems, where one backbone serves many tasks and roles at once: our analysis shows that different tasks and roles recruit different experts, while static methods assign one fixed subset to all of them. We therefore propose Dynamic Expert Pruning (DEP), which rests on a finding we establish here: an agent's system and task prompts are by themselves sufficient to identify the experts that agent and its task require, since that text already describes what the agent will do. A lightweight predictor, trained on workflow transcripts, turns those prompts into a specialized per-request mask in a single forward pass, with no per-configuration calibration. Across diverse tasks and roles, model scales, and MoE architectures, DEP achieves better overall accuracy than static pruning and merging baselines, and generalizes to workflows unseen in training without retraining. Its margin over those baselines is largest when few experts are retained, suggesting that the role specialization inherent to multi-agent systems permits sparser serving than static pruning allows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Multi-Agent System, Large-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_852NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to prune the experts in the MoE models to reduce the memory footprint. Instead of finding a fixed pruning mask, the authors propose training a mask predictor that dynamically prunes the experts for different tasks and agent roles. The paper starts with preliminary experiments on how the agent's role affects the operations it uses and how those operations affect the experts activated, motivating dynamic expert pruning. The training of the mask predictor includes an impact-coverage target that estimates the individual impact of each expert, and a Self-distillation mechanism that jointly optimizes the mask prediction of all experts under the objective of minimizing the output distribution change. Experiments show that the proposed method outperforms static pruning mask baselines and generalizes to various setups and generalizes to unseen workflows.

Critical strengths
  • The paper is well-motivated, with the preliminary results showing the necessity and potential of dynamic expert pruning.
  • The empirical results look promising, outperforming the static pruning baselines, especially at high pruning rates.
Critical weaknesses
  • My main concern is the batching efficiency in practice. Under the proposed serving setup, which maintains only k resident experts per layer, requests with different expert masks cannot be directly batched together without exceeding the fixed memory budget. Although the authors mention batching requests by agent, the expert masks also depend on the task input and may differ even for the same agent. This limits batching to requests with the same agent and task input, which makes batching very difficult, if possible at all, in practice. And this further leads to low GPU utilization rate and low throughput.
  • The paper proposes prefetching experts to hide loading latency. However, overlapping expert loading with the current agent's generation may require additional GPU memory, potentially compromising the memory savings that motivate expert pruning.
Questions for authors
  • Could the authors discuss how the proposed method can be deployed in practice, especially whether batch inference is possible?
  • Could the authors provide the peak memory footprint of the proposed method during inference with the prefetching mechnism?
4 comments

We appreciate the careful reading. Both of the reviewer's points go to how DEP would actually be served, and both turned out to be answerable with measurements rather than argument, so we have run them. Here, we present these additional experimental results and explanations for each point raised.

W1 — "per-input masks make batching very difficult, if possible at all"

The reviewer raises an important point. DEP can be batched by agent, our implementation already batches, and every accuracy and timing number in the submission was produced with batched generation rather than one request at a time.

The predictor is supervised per input during training, but at serving time a mask is not computed per request and does not change during one. One mask is produced for an (agent system prompt, task input) pair before the turn begins and is then held fixed: it does not change from token to token, and it does not change across the repeated calls a workflow makes on the same problem. Requests that share an agent and a problem therefore run under one mask and are batched together, with no mask switch between them. This is the common case rather than a corner case, because workflows call one agent several times on one problem — parallel samples feeding a self-consistency ensemble, a debate agent across rounds, a verify-then-retry loop. Across the six workflows of our suite, 96% of agent calls share their mask with at least one other call, and a mask carries 3.24 calls on average. A batch that mixes different agents of a task needs the union of their masks resident: in our workflows, this is 39–49 of the 128 experts per layer at ρ=25%, against 32 for a single agent's mask.

We recognise that this part of the submission can be confusing, since the task input is part of the predictor's input while Section 3.4 speaks of "batches grouped by agent". We have clarified the granularity at which a mask is produced, cached, and batched in Section 3.4 of the revised paper.

W2 / Q2 — peak memory with prefetching

This is an important question, and the short answer is that DEP can prefetch but deliberately does not. No result in the submission uses prefetching, and no memory figure in it includes residency caused by prefetching: the footprint we report is the delta-loading runtime with the expert swap fully exposed, charged at its analytic worst case at every agent hand-off while the static baselines are charged nothing.

We left prefetching out because its trade runs against the purpose of the method. As reported in the submission, the expert swap is a small share of the generation wall even at worst case, so hiding it can buy back only a small share; and hiding it means holding room on the accelerator for the next agent's experts while the current agent is still using its own. Prefetching does minimise the Δ-load time, and that is precisely why it collides with the design goal of expert pruning: it spends the memory the method saved to remove a cost that was already small.

Since the reviewer asks for the number, we have now implemented prefetching and measured it on the three workflows of Table 3, reserving spare expert slots per layer sized either for an average hand-off or for the largest one we observe.

configurationpeak memory, ρ=50%peak memory, ρ=25%
all experts resident (no delta loading)56.9 GB56.9 GB
delta loading, as submitted29.9 GB (−47%)16.4 GB (−71%)
+ prefetch, headroom for an average hand-off34.6–35.8 GB (−39 to −37%)17.7–19.8 GB (−69 to −65%)
+ prefetch, headroom for the largest hand-off40.0 GB (−30%)19.8–23.6 GB (−65 to −59%)

(Expert residency, measured at batch 1; a larger batch's KV cache adds equally in every row.)

The wall-time side, extending Table 3 (mean over the three workflows at ρ=50%; generation is the submission's measured three-seed wall, Δ-load the measured swap charged over a 50-problem run, in place of the +42 s Table 3 charges analytically):

serving configurationpeak memory (GB)ms per hand-offgeneration (s)Δ-load (s)total (s)
dense, no pruning56.9—3239—3239
DEP, masked routing only (no delta loading)56.9—327503275
DEP + delta loading, exposed swap (as submitted)29.92213275+36.03311
DEP + delta loading + prefetch, average headroom35.1663275+10.43285
DEP + delta loading + prefetch, largest headroom40.0613275+9.73285
DEP + delta loading + prefetch within a workflow40.01053275+16.73292
REAP, static mask (no swap)29.9—380303803
frequency, static mask (no swap)29.9—545705457

The measurement confirms the choice. Prefetching removes almost all of the Δ-load, and does not slow the generation it hides behind, but it pays for that with a large share of the memory that pruning saved — and the more aggressively the model is pruned, the less there is left to hide and the larger that price becomes. Spending the saving to recover a cost that was already a small share of the wall runs against the purpose of expert pruning, so the configuration we report keeps the swap exposed and pays for it in full, and still finishes ahead of REAP and frequency pruning, which are charged no swap at all. We have revised Section 3.4 to present prefetching as an optional extension with its memory cost stated. We will validate these measurements further and report them in an appendix of the camera-ready version.

How your review improved the paper:

Added to the revised paper:

  • A clearer account in Section 3.4 of the granularity at which a mask is produced, cached and batched.
  • Prefetching presented as an optional extension, with its memory cost stated (Section 3.4 and Appendix H).

To be added to the camera-ready version, after further validation:

  • The prefetching measurements of memory and latency, in a new appendix.

We hope these answers address the reviewer's questions and concerns. If they do, we would be grateful if the reviewer would consider raising the score towards a clear acceptance.

Thank you for the response. The memory analysis result is useful.

However, while it is helpful to clarify that one mask (and thus a batch) is produced for an (agent system prompt, task input) pair, clarity is not my main concern, as stated in the review.

As confirmed by the authors, requests have to share the same agent and task input in order to have the same mask and then be batched together. While this technically can produce batches, the batch size is likely very limited. This is further supported by the authors' report that an average of 3.24 calls share the same mask. The number of calls sharing a mask is an upper bound on the batch size, as those calls can be made at different time/step of the agent workflow and thus can not be batched together. For example, a parallel self-consistency ensemble agent structure may have the calls grouped together, but not for the sequential debate structure, as you cannot group the current response of an agent with its future calls. Furthermore, consider deploying the method in practice; finding requests with the exact same agent system prompt and task and grouping them together is almost impossible given the request diversity.

The authors also mentioned that a batch that mixes different agents of a task needs the union of their masks to be resident. While this is feasible, it somewhat contradicts the memory footprint-saving strength of the proposed method.

I would suggest the authors address the concern directly.

We appreciate the reviewer pressing on this point, and we agree that batching is a limitation of our current algorithm. Union batching can address it but, as the reviewer notes, at the cost of additional memory. However, we believe that DEP with a batch union can still achieve better performance with lower memory than the baselines at ρ=50%, and here we provide results at a more realistic batch size of 128.

Setting. We serve DEP in batches of 128 calls in which no two calls share an (agent, task input) pair, so every call in a batch has a different mask, and each batch runs with the union of its 128 masks resident. Batches are drawn from all agent calls of the six Table 1 workflows on their evaluation problems, under three batching policies:

  • by task: 128 random calls of one task, as a deployment serving one workflow receives them;
  • by mask similarity: from the pooled traffic of all six tasks, each batch is filled with the waiting calls whose masks add the fewest new experts to it;
  • random: 128 random calls from the pooled traffic of all six tasks, with no scheduling.

Memory is the resident expert weights, computed from the footprints measured in our earlier response; we report the mean over calls and the largest batch.

methodmemory (GB), mean / largest batchMATHGSM8KHumanEval+MBPP+HotpotQAMMLUMean
Unpruned56.9 / 56.968.397.392.880.071.982.782.2
ρ = 25%
REAP16.4 / 16.45.37.70.012.34.310.76.7
HC-SMoE16.4 / 16.42.70.70.00.35.026.35.8
EASY-EP16.4 / 16.47.063.00.00.047.938.026.0
DEP16.4 / 16.451.387.778.361.753.748.063.4
DEP batch union, by task22.9 / 25.366.089.085.584.065.465.075.8
DEP batch union, by mask similarity19.4 / 29.764.089.082.669.064.857.071.1
DEP batch union, random32.9 / 33.268.091.092.886.072.567.079.5
ρ = 50%
REAP29.9 / 29.940.771.378.745.747.340.354.0
HC-SMoE29.9 / 29.960.384.075.875.060.448.367.3
EASY-EP29.9 / 29.970.093.050.748.074.069.067.4
DEP29.9 / 29.967.395.384.576.760.770.775.9

ρ is the per-call budget. Baselines and DEP with one mask are from Table 1; the union rows are one run on a different host, on which DEP with one mask scores 62.2 at ρ=25% (63.4 in Table 1).

Results. A batch union requires more memory than a single mask, both on average and in the largest batch: 19.4–32.9 GB against 16.4 GB at ρ=25%. It also scores higher than one mask per call, because more experts are resident. Still, at ρ=25% the union outperforms every ρ=50% baseline with less memory. Batched by task, it uses 22.9 GB on average and 25.3 GB in its largest batch, against 29.9 GB, and scores 75.8 against 67.4 for the best baseline, even though those baselines keep more experts per layer (64) than this union does on average (47.4). Batched by mask similarity over the mixed traffic of all six tasks, every batch stays within the baselines' 29.9 GB (19.4 GB on average, 29.7 GB at most), and it scores 71.1. Only random batching needs more, 32.9 GB. Due to the limited time of the response period, we could not run random batching with the union capped at 29.9 GB, but we believe it would still outperform the baselines.

Limitations and future work. As the reviewer pointed out, our method is not designed for batching, and a substantial amount of future work remains. There are smarter ways to adapt our algorithm to union batching, for example masking by the predictor's mean scores over a batch and training the predictor for that objective, but these are beyond the scope of our original paper. As in our response to Reviewer 8uyL, multi-agent structures beyond those we evaluate, such as the debate the reviewer describes, in which an agent's calls are spread across rounds, raise several interesting research questions, and we hope future work will address them. Our contribution is limited to showing the necessity of dynamic expert pruning and the possibility of predictor-based, task- and system-prompt-aware dynamic expert pruning. We will state these limitations and the scope of our work clearly in the paper, and include the union-batching analysis as a starting point for future work.

Reviewer_Kj5ROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Previous static expert pruning methods for expert pruning fall short on tasks that require a diverse workflow. They make expert pruning dynamic by training a small predictor to predict which experts to use based on the prompt and the task. It is tested on several workflows and retention levels and shows effectiveness compared to the baselines.

Critical strengths
  1. Clear. The motivation and limitations of previous work are clearly defined. The paper is well organized and easy to read.
  2. Novel. I think the idea of decomposing a task into a multi-agent workflow and applying dynamic expert pruning is a smart way of transforming semantic understanding of the task into expert selection with minimal additional computation.
  3. Experiments are comprehensive. The method showed strong performance on unseen workflows and provided analysis on how predicted masks are distinctive from each other.
Critical weaknesses
  1. Limited baselines. The method shows stronger performance against the static baselines but no dynamic expert pruning baselines are tested. It would be better to add baselines such as (but not limited to) EASY-EP [1] or Instruction-following pruning [2].

[1] Dong et al, Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations, Neurips 2025
[2] Hou et al, Instruction-Following Pruning for Large Language Models, ICML 2025

Questions for authors

None

3 comments

Many thanks for the positive assessment and for the pointers to EASY-EP and IFPruning, which have strengthened the paper. Here, we present additional experimental results and explanations addressing the weaknesses and questions raised.

W1-a. EASY-EP. We evaluated EASY-EP (Dong et al., NeurIPS 2025) under the Table 1 protocol: the same frozen Qwen3-30B-A3B backbone, workflows, problems and retention levels. It is calibrated with the official code on the same transcripts as the other baselines. The EASY-EP paper itself evaluates this kind of mixed-domain setting, in which one mask is calibrated across several domains. Like REAP, EASY-EP selects experts offline from a saliency score and serves one fixed mask, while DEP predicts a mask for each (agent, problem).

Table R3-1. Table 1 setting (accuracy %; F1 for HotpotQA). EASY-EP is one seed; the other columns are three-seed means.

TaskDenseREAP 50%EASY-EP 50%DEP 50%REAP 25%EASY-EP 25%DEP 25%
MATH-50068.340.770.067.35.37.051.3
GSM8K97.371.393.095.37.763.087.7
HumanEval+92.878.750.784.50.00.078.3
MBPP+80.045.748.076.712.30.061.7
MMLU82.740.369.070.710.738.048.0
HotpotQA71.947.374.060.74.347.953.7
Mean82.254.067.475.96.726.063.4

EASY-EP clearly improves over REAP, by 13.4 mean points at 50% retention and 19.3 at 25%. DEP still outperforms it at both retentions, by 8.5 and 37.4 points. On the identical seed-1 problems, DEP averages 77.3 at 50% and 64.7 at 25%.

At 50%, EASY-EP matches DEP on MATH, GSM8K and MMLU and is better on HotpotQA, but its single mask loses about 30 points on both coding tasks (p < 0.01). At 25%, it collapses on MATH and both coding tasks (7.0 / 0.0 / 0.0). DEP keeps 51–78 on these tasks by allocating experts for each agent and problem, which is the heterogeneity that motivates DEP (Section 2). We will run additional seeds, validate these results, and add them to the camera-ready version.

W1-b. IFPruning. IFPruning (Hou et al., ICML 2025) prunes FFN rows and columns of a dense LLM and fine-tunes the backbone jointly with its predictor, so it cannot be applied directly as an expert-pruning baseline on a frozen MoE. Its closest expert-pruning counterpart is in our ablations. DEP adopts IFPruning's predictor design (Section 3.2), and the "LM + BCE" rows of Table 6 replace our self-distillation term with IFPruning's next-token language-modeling loss.

Objective (ρ = 50%, n = 50)MATHHumanEval+HotpotQAMean
Dense82.092.072.282.1
DEP (KD + BCE)76.090.071.779.2
LM + BCE (REAP target)76.084.068.476.1
LM + BCE (norm target)82.080.00.054.0

The language-modeling objective lowers the mean from 79.2 to 76.1. With a norm-based target, it even yields a degenerate predictor: the pruned model outputs one repeated token on 47 of 50 HotpotQA problems. We will move this comparison into the main text.

How your review improved the paper:

  • EASY-EP as an additional baseline (Table R3-1), to be added to the camera-ready version after we run additional seeds and validate the experiments.
  • The comparison with IFPruning's training objective, to be moved into the main text.
Reviewer_Kj5ROpenReview ↗

Response to authors

Thank you for conducting additional experiments and clarifications. Along with these, I hope more discussion on dynamic expert pruning are included in the final version. I will maintain the weak acceptance score.

We appreciate the positive acknowledgment and the feedback, and we will make sure that the discussions from this review, including a broader discussion of dynamic expert pruning, are reflected in the final version.

Reviewer_2v7XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Dynamic Expert Pruning (DEP) to reduce the memory bottleneck caused by keeping all experts of a Mixture-of-Experts model resident on the accelerator. Unlike conventional expert-pruning methods, which apply a single offline-calibrated mask to every request, DEP accounts for the heterogeneous tasks and roles found in multi-agent systems by predicting a specialized expert mask for each request. A lightweight predictor, trained once on workflow transcripts, uses only the agent’s system prompt and task input to select the required experts without configuration-specific recalibration. The authors report that DEP achieves higher overall accuracy than static pruning and expert-merging baselines across diverse tasks, roles, model scales, and MoE architectures, while also generalizing to unseen workflows without retraining. Its advantage is largest under aggressive pruning, suggesting that task- and role-specific expert selection can preserve model capabilities with substantially fewer resident experts.

Critical strengths

The empirical results are compelling, particularly under aggressive pruning. In Table 1, retaining only 25% of the experts yields a reported aggregate mean score of 63.4 for DEP, compared with 1.2–6.7 for the evaluated static pruning and merging baselines. The experiments on unseen workflow structures, workflow families, benchmarks, and domains provide further evidence that the predictor can transfer beyond its training configurations. These findings make request-specific expert selection a promising approach to preserving capability under stringent expert-retention constraints, even though stronger comparisons are needed to establish its advantage over other specialized methods.

Critical weaknesses

To provide constructive feedback that may help strengthen the paper, I highlight the following concerns.

  1. The methodological novelty appears incremental. The central principle of predicting a pruning configuration from an input prompt has substantial precedent. IFPruning (Hou et al., 2025) predicts instruction-conditioned masks that remain fixed during decoding, while PuDDing (Wee et al., 2025) routes prompts to suitable layer-pruning configurations. EASY-EP (Dong et al., 2025) further shows that a sparse and stable subset of MoE experts can be localized using only a few in-domain demonstrations. DEP’s exact combination of a frozen MoE backbone, per-turn prediction, and multi-agent workflows is distinct and potentially useful. However, the paper does not sufficiently identify the new technical challenge or methodological insight that specifically arises from this combination, beyond adapting established prompt-conditioned pruning principles to a different granularity and setting. This weakens the paper’s central novelty claim.

  2. The evaluation does not establish whether learned per-request prediction is necessary. In the paper’s main evaluation, the static comparators are calibrated on the pooled six-workflow transcripts and then reused as one fixed compressed configuration across heterogeneous tasks and agent roles; this includes the REAP-based pruning mask (Lasby et al., 2026). Important controls—such as separately calibrated task-specific and role-specific masks, and a simple scheme that selects among precomputed masks under the same expert-retention ratio—are missing. Consequently, the reported gains may primarily reflect the benefit of avoiding one global configuration rather than the specific benefit of DEP’s learned predictor. The failure analysis is also underdetermined: the paper attributes repetitive generation under static masks to task or role mismatch, whereas Unveiling Super Experts (Su et al., 2026) shows that removing only a few data-agnostic critical experts can cause similar repetitive collapse in the same Qwen3-30B-A3B model. The current experiments do not distinguish between these explanations.

Hou, B., Chen, Q., Wang, J., Yin, G., Wang, C., Du, N., Pang, R., Chang, S., & Lei, T. (2025). Instruction-Following Pruning for Large Language Models. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).

Wee, J., Park, M., & Lee, J. (2025). Prompt-based Depth Pruning of Large Language Models. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).

Dong, Z., Peng, H., Liu, P., Zhao, X., Wu, D., Xiao, F., & Wang, Z. (2025). Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).

Lasby, M., Lazarevich, I., Sinnadurai, N., Lie, S., Ioannou, Y., & Thangarasa, V. (2026). REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression. International Conference on Learning Representations (ICLR 2026).

Su, Z., Li, Q., Zhang, H., Ye, W., Xue, Q., Qian, Y., Wong, N., & Yuan, K. (2026). Unveiling Super Experts in Mixture-of-Experts Large Language Models. International Conference on Learning Representations (ICLR 2026).

Questions for authors

The following are assessment-critical clarification requests concerning the paper’s current claims and experimental controls, rather than suggestions for future work.

  1. What is DEP’s specific technical contribution beyond instruction-conditioned mask prediction, especially IFPruning (Hou et al., 2025), and related prompt-routed pruning methods such as PuDDing (Wee et al., 2025)? What new technical challenge introduced by frozen MoEs or multi-agent workflows requires DEP’s design?

  2. Does DEP still outperform separately calibrated task-specific and role-specific REAP-based masks (Lasby et al., 2026), treated as oracle controls, under the same expert-retention ratio? How does it compare with selecting among a small library of precomputed task- or role-specific masks under the same loading assumptions?

  3. Do the static masks that produce repetitive outputs remove any of the data-agnostic critical experts identified in Unveiling Super Experts (Su et al., 2026)? If these experts are preserved across all pruning methods, does DEP retain a similar relative advantage?

Hou, B., Chen, Q., Wang, J., Yin, G., Wang, C., Du, N., Pang, R., Chang, S., & Lei, T. (2025). Instruction-Following Pruning for Large Language Models. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).

Wee, J., Park, M., & Lee, J. (2025). Prompt-based Depth Pruning of Large Language Models. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).

Lasby, M., Lazarevich, I., Sinnadurai, N., Lie, S., Ioannou, Y., & Thangarasa, V. (2026). REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression. International Conference on Learning Representations (ICLR 2026).

Su, Z., Li, Q., Zhang, H., Ye, W., Xue, Q., Qian, Y., Wong, N., & Yuan, K. (2026). Unveiling Super Experts in Mixture-of-Experts Large Language Models. International Conference on Learning Representations (ICLR 2026).

4 comments

We are grateful for the detailed and constructive review. Here, we present additional experimental results and explanations addressing the weaknesses and questions raised.

Q1 / W1 — technical contribution beyond prompt-conditioned pruning

We appreciate the opportunity to clarify our technical novelty. It is true that predicting a pruning mask from the input has a long history, well before IFPruning, for example hypernetwork-based and input-dependent pruning [Lin et al., 2017; Gao et al., 2019; Liu et al., 2019; Li et al., 2020]. Compared with IFPruning, DEP may even look like a step down in scale. We believe our contribution lies precisely in that deliberate narrowing, and in the training loss that makes it work for the dynamic expert-pruning problem.

The challenge is also different from the one IFPruning addresses. Expert pruning of an MoE has to actually minimize the memory resident on the accelerator, whereas IFPruning speeds up each request while its full parameter set stays available to select from. The mask therefore decides which experts are resident, so it has to come from the prompts alone, before each agent's turn. And with a frozen backbone, nothing is fine-tuned to make up for what pruning removes: the mask itself has to preserve the full model's output. Our self-distillation objective trains for exactly that, and it is where saliency-based masks fail at 25% retention.

IFPruning predicts a mask over the FFN parameters of a dense LLM for every individual request and fine-tunes the backbone together with its predictor. This raises practical problems at serving scale. The output space, a subset of a whole parameter set, is enormous, which makes the predictor harder to optimize as models grow. And because every request can select a different parameter subset, a server handling many requests cannot actually release the unselected parameters to save memory, and requests with different masks cannot share one pruned set of weights in a batch.

DEP narrows the output space to a top-K choice among the 128 routed experts of each layer of a frozen MoE, which is far easier to optimize. It produces a mask only when the agent or the problem changes, not for every call. So only the selected experts are loaded and kept resident on the accelerator (Section 3.4), and calls that share an agent and a problem share a mask and are batched together, which is how our implementation runs. Our contribution is to show that prompt-conditioned prediction is particularly well suited to dynamic expert pruning of MoE models in multi-agent systems, a direction we hope will be explored further. PuDDing, by contrast, routes a prompt to one of a few precomputed depth-pruning configurations, while DEP produces a new expert mask for any agent prompt, including agents it has never seen.

Our second technical contribution is the training loss. It combines an impact-coverage target, binary cross-entropy against the experts that contribute most to each transcript's response (REAP saliency), with self-distillation, the KL divergence between the unmasked model and the same model under the predicted mask. The first term says which experts matter individually; the second measures what removing the rest does jointly to the output. Replacing self-distillation with IFPruning's language-modeling loss lowers the mean accuracy from 79.2 to 76.1 (Table 6).

Following the reviewer's pointer to EASY-EP, we evaluated it under the Table 1 protocol, calibrated with the official code on the same transcripts as the other baselines. The EASY-EP paper itself evaluates this kind of mixed-domain setting, in which one mask is calibrated across several domains.

Table R2-1. Table 1 setting (accuracy %; F1 for HotpotQA). EASY-EP is one seed; the other columns are three-seed means.

TaskDenseREAP 50%EASY-EP 50%DEP 50%REAP 25%EASY-EP 25%DEP 25%
MATH-50068.340.770.067.35.37.051.3
GSM8K97.371.393.095.37.763.087.7
HumanEval+92.878.750.784.50.00.078.3
MBPP+80.045.748.076.712.30.061.7
MMLU82.740.369.070.710.738.048.0
HotpotQA71.947.374.060.74.347.953.7
Mean82.254.067.475.96.726.063.4

EASY-EP clearly improves over REAP, by 13.4 mean points at 50% retention and 19.3 at 25%, and DEP still outperforms it at both retentions, by 8.5 and 37.4 points. At 25%, EASY-EP's single mask collapses on MATH and both coding tasks (7.0 / 0.0 / 0.0), where DEP keeps 51–78.

Q2 / W2 — task- and role-specific controls

This comparison is exactly the right control to ask for, and we report the experiments it calls for here.

Every control below is calibrated on its own slice of the same training union our predictor saw, and is handed the true task label, or the true (task, role) label, at serving time; our predictor is handed neither and reads only the system and task prompts. The per-(task, role) column is in effect the mask library the reviewer asks about, with the true labels acting as its selector. Values are accuracy in %, token F1 for HotpotQA. The per-(task, role) column is one seed, the other columns three.

ρ = 50% (keep 64 of 128)

taskdenseREAP unionREAP per-taskREAP per-(task,role)HC-SMoE per-taskDEP per-taskDEP (ours)
MATH-50068413574597167
GSM8K97717591799595
HumanEval+93798790578785
MBPP+80467676335877
MMLU83406157457571
HotpotQA72473553377161
mean82546274527676

ρ = 25% (keep 32 of 128)

taskdenseREAP unionREAP per-taskREAP per-(task,role)HC-SMoE per-taskDEP per-taskDEP (ours)
MATH-5006851013751
GSM8K9780418888
HumanEval+930586108278
MBPP+801237420462
MMLU83113837253448
HotpotQA72459116454
mean827322455263

The reviewer is right that per-task and per-role calibration brings a substantial gain. Handing a static method the true task label lifts it well above the union-calibrated mask, and handing it the true role as well lifts it further at ρ=50%. Our method nonetheless stays above these controls on average at both retentions: the gap is narrow at ρ=50% and much larger at ρ=25%. Specialising our own predictor per task instead of pooling the tasks does not help either, and at ρ=25% it costs.

Two points follow. First, the controls buy their accuracy with calibration. Each cell of those columns is its own calibration pass over that task's, or that role's, transcripts, and every new workflow, new agent, or edited prompt needs another pass, while our predictor produces a mask for an unseen configuration from the system and task prompts in one forward pass of a small encoder. Second, the gap that remains is widest where pruning is most aggressive, and we believe the objective is the reason. A saliency criterion ranks each expert in isolation and never observes what removing most of them does jointly to the output, whereas our training measures that directly, matching the masked model's output distribution to the unmasked one. At ρ=50% enough capacity survives that the distinction matters little; at ρ=25% it is what keeps the output from degenerating. Our objective and criterion ablation points the same way, as no combination we tried improved on the output-matching objective.

We will run additional seeds, validate these controls, and add them to the appendix of the camera-ready version as the per-task and per-role ceiling they represent, and state in the text that task identity and role identity are available to a static method in principle and still do not close the gap under aggressive pruning.

Q3 — Super Experts

This is a very interesting perspective, and we have looked into it in depth. Following Su et al., we identified the Super Experts on our own checkpoint and inputs: expert 68 in layer 1, expert 92 in layer 2 and expert 82 in layer 3, the same three that Su et al. report for Qwen3-30B-A3B. We then checked whether each method's masks include them. Frequency-based pruning keeps all three at 50% retention and two at 25%, while REAP drops two of the three at both retentions. Yet frequency pruning performs poorly even with every Super Expert kept: at 50% retention it collapses on both coding tasks (HumanEval+ 2.9, MBPP+ 3.0 in Table 1), where 85–87% of its programs are not valid Python and 17% of its HumanEval+ answers are repetition loops. Keeping the Super Experts is therefore not by itself enough for coherent, non-repetitive generation.

Following the reviewer's suggestion, we forced the three Super Experts into every mask at 25% retention, at the same number of kept experts (each replaces the lowest-scoring non-Super Expert in its layer), and re-ran all six tasks.

Table R2-2. Super Experts forced in at ρ = 25% (accuracy %; F1 for HotpotQA). A separate seed-1 re-run, so the DEP row differs slightly from Table 1.

MATH-500GSM8KHumanEval+MBPP+HotpotQAMMLUMean
REAP7.07.00.014.05.913.07.8
REAP + Super Experts16.043.024.653.052.438.037.8
DEP58.087.081.253.047.846.062.2
DEP + Super Experts65.085.079.773.055.550.068.0

The reviewer's hypothesis is right in part: forcing the Super Experts in lifts REAP by 30 points on average, significantly on every task (p < 0.05, paired).

The Super Experts and DEP also work well together. Because they are data-agnostic, they can be pinned in any mask at the same number of kept experts, and doing so lifts DEP as well (+5.8 on average, significant on MATH-500, MBPP+ and HotpotQA). Our impact target is computed on response tokens, while the Super Experts act on the first token of the input, so at 25% retention our predictor rarely keeps expert 68 of layer 1 by itself. We see substantial room for future research here, for example predictors that account for such data-agnostic experts directly.

Even with the Super Experts, REAP still remains far below DEP. With them preserved in both methods, DEP leads by 30.2 points (68.0 vs. 37.8), so losing the Super Experts explains part of the static collapse, but not most of it.

How your review improved the paper:

To be added to the camera-ready version, after we run additional seeds and validate the experiments:

  • EASY-EP as an additional baseline (Table R2-1).
  • Per-task and per-(task, role) oracle controls for static pruning, in the appendix.
  • The Super-Expert analysis and Table R2-2, in the appendix.

We will also sharpen the statement of the challenge DEP addresses and of how it differs from IFPruning and PuDDing.

The reviewer's thoughtful questions have made the paper stronger. We hope our answers have addressed the reviewer's concerns well; if so, we would kindly ask the reviewer to consider raising their score.

As the discussion period ends in about 12 hours, we would be grateful to hear whether our response has addressed your concerns. In brief:

  • Q1 (contribution): we clarified the challenge that expert pruning of a frozen MoE poses, and how DEP's design and self-distillation objective address it, in contrast to IFPruning and PuDDing. We also added EASY-EP as a baseline, which DEP outperforms at both retentions.
  • Q2 (oracle controls): we evaluated static masks calibrated separately per task and per (task, role), given the true labels. They narrow the gap but stay below DEP on average, most clearly under aggressive pruning.
  • Q3 (Super Experts): we identified the Super Experts on our checkpoint and forced them into the masks. This helps the static baselines substantially, confirming part of your hypothesis, but DEP keeps a large lead even with the Super Experts kept in both.

We would be happy to clarify anything further in the remaining time.

Reviewer_8uyLOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel Expert Pruning method, called Dynamic Expert Pruning (DEP), which is specially designed for Mixture-of-Experts (MoE) architectures in backbone models in multi-agent systems (MAS). DEP enables different masks for different agents, where each agent's mask is decided by its own system and task prompts. A lightweight predictor, trained once on workflow transcripts, turns those prompts into a specialized per-request mask. DEP shows excellent performance and generalization power across diverse tasks and settings.

Critical strengths
  1. The paper identifies a very important and urgent challenge in MAS, the heterogeneity of workload of each agent, or in other words, the need for specialized agents. This is a significant barrier toward larger, more complex, and yet efficient MAS, and thus, solving this issue is highly valuable.

  2. The illustration by experiments of Static Expert Pruning shows a real gap and thus demonstrates the need for Dynamic Expert Pruning. This makes the proposed research well-motivated.

  3. The method is elegant, and the usage is direct. It only needs input of system and task prompts, and does not need per-configuration (task domain, MAS structure, backbone, etc.) calibration, making the proposed DEP easy to adopt and practically significant.

  4. The DEP algorithm is well-designed with sufficient support from ablation studies.

  5. The DEP predictor itself is lightweight and efficient, so it does not add a large computational burden to MAS.

Critical weaknesses
  1. The major experiment only reports end-to-end accuracies, while it lacks some intermediate mask-quality metrics to comprehensively support the DEP method and the claim that ‘using system and task prompts as input is sufficient’. This claim itself is also somewhat overstated, as working well empirically does not necessarily mean any other potential inputs won’t be helpful.

  2. The generalization experiments tested on unseen workflows, but these workflows are still within a relatively similar regime to the training examples. It will be interesting to see how it generalizes to drastically different unseen workflows, e.g., the one with much more complicated structures or with a much larger scale of the number of agents

  3. The paper introduces MoE comprehensively, but introduces MAS too briefly, which is an important background and motivation for the proposed method. And some important references on the topic of learning the heterogeneity/specializations within MAS are missing, for example,

  • Lu, H., Fang, L., Zhong, W. and Ma, P., (2026). NeuroMAS: Multi-Agent Systems as Neural Networks with Joint Reinforcement Learning. The First Conference on Statistics and Trustworthy AI for Cross (X)-Domain Acceleration.
  • Xue, X., Zhou, Y., Zhang, G., Zhang, Z., Li, Y., Zhang, C., ... & Bai, L. (2026). CoMAS: Co-evolving multi-agent systems via interaction rewards. In International Conference on Learning Representations (Vol. 2026, pp. 32373-32394).
  • Liu, S., Chen, T., Amiri, R., & Amato, C. (2026). Learning decentralized LLM collaboration with multi-agent actor critic. arXiv preprint arXiv:2601.21972.
  • Motwani, S. R., Smith, C., Das, R. J., Rafailov, R., Laptev, I., Torr, P. H., ... & de Witt, C. S. (2024). MALT: Improving reasoning with multi-agent LLM training. arXiv preprint arXiv:2412.01928.
Questions for authors
  1. Have the authors tried other potential input information beyond system and task prompts, and if so, how do those perform?

  2. Have the authors tried how DEP generalizes on a much larger MAS, e.g., ten times the number of agents of the current settings?

4 comments

We sincerely appreciate the recognition of our work and the insightful review and suggestions. Here, we present additional experimental results and explanations addressing the weaknesses and questions raised.

W1 — intermediate mask-quality metrics

We found this suggestion very valuable. End-to-end accuracy alone does not show how good the masks themselves are, so we have added the intermediate evaluation the reviewer asks for. For 40 agent calls per task in each of the six Table 1 workflows, we take the unpruned model's own response, replay it under each method's mask, and compare the pruned model with the unpruned one on the first 256 response tokens.

Table R1-1. Mask quality against the unpruned model, mean over the six tasks.

ρMethodMissed experts ↓KL divergence ↓NLL increase ↓Same top-1 prediction ↑
50%Frequency0.2880.3540.3640.908
REAP0.3580.4480.4420.880
NAEE0.3430.9930.9980.791
HC-SMoE–0.2940.2960.908
DEP0.2570.1070.1090.950
25%Frequency0.5762.0122.0260.673
REAP0.6590.9030.9110.803
NAEE0.6261.8851.8990.670
HC-SMoE–1.4501.4590.743
DEP0.5580.3570.3570.891

Missed experts: share of the unpruned model's expert selections that fall outside the mask. KL divergence: from the unpruned model's next-token distribution to the pruned model's. NLL increase: increase in the negative log-likelihood of the unpruned model's response. Same top-1 prediction: how often the pruned model's most likely next token matches the unpruned model's. HC-SMoE merges experts instead of removing them, so it has no missed experts.

Table R1-1 shows that our method has a lower miss rate, a smaller KL divergence and loss increase, and agrees more often with the unpruned model's predictions than every baseline, at both retention levels. We will run additional seeds, validate these results, and add them to the camera-ready version.

W2 / Q2 — drastically different and larger workflows

We have experimented on two settings that differ from the AFlow workflows our predictor was trained on, using the Table 1 predictor and static masks as they are, with no retraining or recalibration:

  1. A workflow with a different agent composition and prompts: DyLAN's HumanEval team, with four code writers over three rounds, unit-test, reflection and ranking judges whose prompts are unlike any in AFlow, and a consensus-based early stop (69 HumanEval+ problems).
  2. A much longer agentic conversation: multi-agent debate (Du et al.) with 10 agents over 3 rounds, i.e. 30 agent calls per problem, against at most about 10 in our Table 1 workflows (30 MATH problems, majority vote).

Table R1-2. Unseen workflows (accuracy %).

Debate, 10 agents × 3 rounds, ρ=50%ρ=25%DyLAN, ρ=50%ρ=25%
Unpruned90.090.094.294.2
REAP20.03.388.417.4
HC-SMoE80.010.081.20.0
DEP80.066.788.478.3

We believe these runs do not directly establish our method's efficacy on very complex agentic workflows, which time and resources did not allow us to test. We find this an interesting direction and will look into it further before the camera-ready version. The runs do show promise beyond the regime of our training workflows: our method matches the best baseline at ρ=50% and outperforms all of them by a wide margin at ρ=25% on both workflows. The predictor also reads one agent's prompts at a time, so the number of agents is not an input to it; a larger system changes how many masks are served, not what the predictor sees.

Q1 — other predictor inputs

This question touches on something our submission left open, and we are glad it was raised. To answer it, we retrained the predictor with different inputs under the same recipe, at a 2048-token window so that longer inputs fit, and evaluated at ρ=25%. The first row therefore differs slightly from Table 1, whose predictor uses a 1024-token window.

Table R1-3. Predictor inputs, ρ=25% (accuracy %; F1 for HotpotQA).

Predictor inputMATH-500GSM8KHumanEval+MBPP+HotpotQAMMLUMean
System prompt + task input (ours)59.082.073.969.051.856.065.3
System prompt only4.089.073.976.038.811.048.8
Task input only52.084.069.969.057.449.063.6
Ours + previous agent's output52.086.079.778.056.611.060.5
Ours + previous call's output53.384.578.264.352.334.261.1

The previous agent's output is the output that immediately precedes the call in the workflow's sequential stream, and the previous call's output is the same agent's own preceding output.

Our input, the system prompt and the task input, shows the best performance on average. Adding the previous agent's output or the previous call's output lowered the performance. However, this part is not sufficiently studied in our work, and much room for further research remains. We have therefore toned the statement down slightly in the revised paper, to say that the system and task prompts are largely sufficient.

W3 — background on multi-agent systems and scope

We appreciate this insightful comment: multi-agent systems are the motivation of our method and deserve a fuller treatment. We mainly evaluated agentic workflows in which agents are called one after another, and showed DEP's efficacy there. There are, however, many types of multi-agent architectures, and we believe our method can fit some of them better than others. In sequential architectures, or hierarchical ones whose subagents are homogeneous, DEP can effectively reduce memory and batch well. In settings where the inputs of many heterogeneous agents must be batched together, the gain from per-agent expert masks can shrink, because the union of their expert sets must be resident on the device. We believe many interesting research questions arise from different multi-agent architectures, and we leave them to future work. We have clarified our scope and limitations accordingly in the revised paper (Appendix H). In the camera-ready version, we will also expand the background on multi-agent systems and add the suggested references on learning heterogeneity and specialization within multi-agent systems (NeuroMAS, CoMAS, multi-agent actor-critic, MALT).

How your review improved the paper:

Added to the revised paper:

  • The claim about the predictor's inputs, toned down to "largely sufficient" (Abstract, Sections 1, 3.2 and 5).
  • A clearer scope and limitations with respect to multi-agent architectures (Appendix H).

To be added to the camera-ready version, after we run additional seeds and validate the experiments:

  • Intermediate mask-quality metrics alongside end-to-end accuracy (Table R1-1).
  • Evaluation on workflows unlike those the predictor was trained on (Table R1-2).
  • The study of predictor inputs (Table R1-3).

We will also expand the background on multi-agent systems and add the suggested references.

Thank you.

Thank you. The authors have reasonably addressed my concerns, and thus I will keep my positive score for the paper.

Thank you for the positive acknowledgment of our work and for the valuable ideas and feedback.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reviews and constructive feedback, which have substantially improved our paper. To assist the Area Chair, we provide an overall summary of the reviews and of how we addressed them.

Strengths recognized by the reviewers

  • An important and well-motivated problem. Heterogeneous agent workloads in multi-agent systems are an important and urgent challenge (8uyL), and our preliminary experiments on static pruning clearly motivate dynamic expert pruning (8uyL, 852N, Kj5R).
  • A novel and practical method. Turning the semantic understanding of a task into expert selection is a novel idea (Kj5R). The method is elegant and directly usable: it needs only system and task prompts, no per-configuration calibration, and a lightweight predictor (8uyL).
  • Strong empirical results. DEP is particularly effective under aggressive pruning (2v7X, 852N), generalizes to unseen workflows, benchmarks and domains (2v7X, Kj5R), and is supported by comprehensive experiments and ablations (8uyL, Kj5R).

Concerns and questions, and how we addressed them

1. Baselines and controls (Kj5R W1, 2v7X W1, 2v7X Q2)

  • We added EASY-EP as a domain-specific expert-pruning baseline; DEP outperforms it at both retention levels.
  • We added static masks calibrated separately per task and per (task, role), given the true labels at serving time, which also serve as an oracle mask-library selector. They narrow the gap, but DEP stays above them on average, most clearly under aggressive pruning.

2. Novelty relative to IFPruning and prior prompt-conditioned pruning (2v7X W1, 2v7X Q1, Kj5R W1)

  • We acknowledged the long history of input-conditioned mask prediction and clarified the challenge specific to our setting: expert pruning must actually minimize the memory resident on the accelerator, so the mask has to come from the prompts alone before each agent's turn, and with a frozen backbone the mask itself has to preserve the full model's output.
  • Our self-distillation objective trains for exactly this; replacing it with IFPruning's language-modeling objective degrades performance.

3. Failure analysis and Super Experts (2v7X Q3)

  • We identified the Super Experts on our checkpoint and forced them into every mask. This helps the static baselines substantially, confirming part of the reviewer's hypothesis, but DEP keeps a large lead with the Super Experts kept in both. The two are complementary, which opens room for future research.

4. Mask quality and predictor inputs (8uyL W1, 8uyL Q1)

  • We added intermediate mask-quality metrics, on which DEP is the best at both retention levels.
  • We studied other predictor inputs; system and task prompts perform best on average, and adding previous outputs lowers performance. We nonetheless toned the claim down to "largely sufficient" in the revised paper.

5. Generalization to different and larger workflows (8uyL W2, 8uyL Q2)

  • We evaluated DEP, without retraining, on a large multi-agent debate and on DyLAN's coding team, whose agents and protocols are unseen in training. DEP matches the best baseline at moderate pruning and outperforms all baselines by a wide margin under aggressive pruning.

6. Serving: batching and prefetching (852N W1, 852N W2, 852N Q1, 852N Q2)

  • We clarified in the revised paper the granularity at which masks are produced, cached and batched, and presented prefetching as an optional extension that our evaluation does not use, with its measured memory cost.
  • We acknowledged batching as a limitation of the current algorithm and evaluated union batching: under aggressive pruning, DEP with batch unions still outperforms the moderately pruned baselines with less memory.

7. Scope and background (8uyL W3, Kj5R follow-up)

  • We stated the scope and limitations with respect to multi-agent architectures in the revised paper, and will expand the background on multi-agent systems and dynamic expert pruning in the final version.

All additional experiments will be run with more seeds, validated, and included in the camera-ready version.

Once again, we thank the reviewers and the Area Chair for their time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.