- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to prune the experts in the MoE models to reduce the memory footprint. Instead of finding a fixed pruning mask, the authors propose training a mask predictor that dynamically prunes the experts for different tasks and agent roles. The paper starts with preliminary experiments on how the agent's role affects the operations it uses and how those operations affect the experts activated, motivating dynamic expert pruning. The training of the mask predictor includes an impact-coverage target that estimates the individual impact of each expert, and a Self-distillation mechanism that jointly optimizes the mask prediction of all experts under the objective of minimizing the output distribution change. Experiments show that the proposed method outperforms static pruning mask baselines and generalizes to various setups and generalizes to unseen workflows.
Critical strengths
- The paper is well-motivated, with the preliminary results showing the necessity and potential of dynamic expert pruning.
- The empirical results look promising, outperforming the static pruning baselines, especially at high pruning rates.
Critical weaknesses
- My main concern is the batching efficiency in practice. Under the proposed serving setup, which maintains only k resident experts per layer, requests with different expert masks cannot be directly batched together without exceeding the fixed memory budget. Although the authors mention batching requests by agent, the expert masks also depend on the task input and may differ even for the same agent. This limits batching to requests with the same agent and task input, which makes batching very difficult, if possible at all, in practice. And this further leads to low GPU utilization rate and low throughput.
- The paper proposes prefetching experts to hide loading latency. However, overlapping expert loading with the current agent's generation may require additional GPU memory, potentially compromising the memory savings that motivate expert pruning.
Questions for authors
- Could the authors discuss how the proposed method can be deployed in practice, especially whether batch inference is possible?
- Could the authors provide the peak memory footprint of the proposed method during inference with the prefetching mechnism?
4 comments
W2 / Q2 — peak memory with prefetching
This is an important question, and the short answer is that DEP can prefetch but deliberately does not. No result in the submission uses prefetching, and no memory figure in it includes residency caused by prefetching: the footprint we report is the delta-loading runtime with the expert swap fully exposed, charged at its analytic worst case at every agent hand-off while the static baselines are charged nothing.
We left prefetching out because its trade runs against the purpose of the method. As reported in the submission, the expert swap is a small share of the generation wall even at worst case, so hiding it can buy back only a small share; and hiding it means holding room on the accelerator for the next agent's experts while the current agent is still using its own. Prefetching does minimise the Δ-load time, and that is precisely why it collides with the design goal of expert pruning: it spends the memory the method saved to remove a cost that was already small.
Since the reviewer asks for the number, we have now implemented prefetching and measured it on the three workflows of Table 3, reserving spare expert slots per layer sized either for an average hand-off or for the largest one we observe.
| configuration | peak memory, ρ=50% | peak memory, ρ=25% |
|---|---|---|
| all experts resident (no delta loading) | 56.9 GB | 56.9 GB |
| delta loading, as submitted | 29.9 GB (−47%) | 16.4 GB (−71%) |
| + prefetch, headroom for an average hand-off | 34.6–35.8 GB (−39 to −37%) | 17.7–19.8 GB (−69 to −65%) |
| + prefetch, headroom for the largest hand-off | 40.0 GB (−30%) | 19.8–23.6 GB (−65 to −59%) |
(Expert residency, measured at batch 1; a larger batch's KV cache adds equally in every row.)
The wall-time side, extending Table 3 (mean over the three workflows at ρ=50%; generation is the submission's measured three-seed wall, Δ-load the measured swap charged over a 50-problem run, in place of the +42 s Table 3 charges analytically):
| serving configuration | peak memory (GB) | ms per hand-off | generation (s) | Δ-load (s) | total (s) |
|---|---|---|---|---|---|
| dense, no pruning | 56.9 | — | 3239 | — | 3239 |
| DEP, masked routing only (no delta loading) | 56.9 | — | 3275 | 0 | 3275 |
| DEP + delta loading, exposed swap (as submitted) | 29.9 | 221 | 3275 | +36.0 | 3311 |
| DEP + delta loading + prefetch, average headroom | 35.1 | 66 | 3275 | +10.4 | 3285 |
| DEP + delta loading + prefetch, largest headroom | 40.0 | 61 | 3275 | +9.7 | 3285 |
| DEP + delta loading + prefetch within a workflow | 40.0 | 105 | 3275 | +16.7 | 3292 |
| REAP, static mask (no swap) | 29.9 | — | 3803 | 0 | 3803 |
| frequency, static mask (no swap) | 29.9 | — | 5457 | 0 | 5457 |
The measurement confirms the choice. Prefetching removes almost all of the Δ-load, and does not slow the generation it hides behind, but it pays for that with a large share of the memory that pruning saved — and the more aggressively the model is pruned, the less there is left to hide and the larger that price becomes. Spending the saving to recover a cost that was already a small share of the wall runs against the purpose of expert pruning, so the configuration we report keeps the swap exposed and pays for it in full, and still finishes ahead of REAP and frequency pruning, which are charged no swap at all. We have revised Section 3.4 to present prefetching as an optional extension with its memory cost stated. We will validate these measurements further and report them in an appendix of the camera-ready version.
How your review improved the paper:
Added to the revised paper:
- A clearer account in Section 3.4 of the granularity at which a mask is produced, cached and batched.
- Prefetching presented as an optional extension, with its memory cost stated (Section 3.4 and Appendix H).
To be added to the camera-ready version, after further validation:
- The prefetching measurements of memory and latency, in a new appendix.
We hope these answers address the reviewer's questions and concerns. If they do, we would be grateful if the reviewer would consider raising the score towards a clear acceptance.
Thank you for the response. The memory analysis result is useful.
However, while it is helpful to clarify that one mask (and thus a batch) is produced for an (agent system prompt, task input) pair, clarity is not my main concern, as stated in the review.
As confirmed by the authors, requests have to share the same agent and task input in order to have the same mask and then be batched together. While this technically can produce batches, the batch size is likely very limited. This is further supported by the authors' report that an average of 3.24 calls share the same mask. The number of calls sharing a mask is an upper bound on the batch size, as those calls can be made at different time/step of the agent workflow and thus can not be batched together. For example, a parallel self-consistency ensemble agent structure may have the calls grouped together, but not for the sequential debate structure, as you cannot group the current response of an agent with its future calls. Furthermore, consider deploying the method in practice; finding requests with the exact same agent system prompt and task and grouping them together is almost impossible given the request diversity.
The authors also mentioned that a batch that mixes different agents of a task needs the union of their masks to be resident. While this is feasible, it somewhat contradicts the memory footprint-saving strength of the proposed method.
I would suggest the authors address the concern directly.
We appreciate the reviewer pressing on this point, and we agree that batching is a limitation of our current algorithm. Union batching can address it but, as the reviewer notes, at the cost of additional memory. However, we believe that DEP with a batch union can still achieve better performance with lower memory than the baselines at ρ=50%, and here we provide results at a more realistic batch size of 128.
Setting. We serve DEP in batches of 128 calls in which no two calls share an (agent, task input) pair, so every call in a batch has a different mask, and each batch runs with the union of its 128 masks resident. Batches are drawn from all agent calls of the six Table 1 workflows on their evaluation problems, under three batching policies:
- by task: 128 random calls of one task, as a deployment serving one workflow receives them;
- by mask similarity: from the pooled traffic of all six tasks, each batch is filled with the waiting calls whose masks add the fewest new experts to it;
- random: 128 random calls from the pooled traffic of all six tasks, with no scheduling.
Memory is the resident expert weights, computed from the footprints measured in our earlier response; we report the mean over calls and the largest batch.
| method | memory (GB), mean / largest batch | MATH | GSM8K | HumanEval+ | MBPP+ | HotpotQA | MMLU | Mean |
|---|---|---|---|---|---|---|---|---|
| Unpruned | 56.9 / 56.9 | 68.3 | 97.3 | 92.8 | 80.0 | 71.9 | 82.7 | 82.2 |
| ρ = 25% | ||||||||
| REAP | 16.4 / 16.4 | 5.3 | 7.7 | 0.0 | 12.3 | 4.3 | 10.7 | 6.7 |
| HC-SMoE | 16.4 / 16.4 | 2.7 | 0.7 | 0.0 | 0.3 | 5.0 | 26.3 | 5.8 |
| EASY-EP | 16.4 / 16.4 | 7.0 | 63.0 | 0.0 | 0.0 | 47.9 | 38.0 | 26.0 |
| DEP | 16.4 / 16.4 | 51.3 | 87.7 | 78.3 | 61.7 | 53.7 | 48.0 | 63.4 |
| DEP batch union, by task | 22.9 / 25.3 | 66.0 | 89.0 | 85.5 | 84.0 | 65.4 | 65.0 | 75.8 |
| DEP batch union, by mask similarity | 19.4 / 29.7 | 64.0 | 89.0 | 82.6 | 69.0 | 64.8 | 57.0 | 71.1 |
| DEP batch union, random | 32.9 / 33.2 | 68.0 | 91.0 | 92.8 | 86.0 | 72.5 | 67.0 | 79.5 |
| ρ = 50% | ||||||||
| REAP | 29.9 / 29.9 | 40.7 | 71.3 | 78.7 | 45.7 | 47.3 | 40.3 | 54.0 |
| HC-SMoE | 29.9 / 29.9 | 60.3 | 84.0 | 75.8 | 75.0 | 60.4 | 48.3 | 67.3 |
| EASY-EP | 29.9 / 29.9 | 70.0 | 93.0 | 50.7 | 48.0 | 74.0 | 69.0 | 67.4 |
| DEP | 29.9 / 29.9 | 67.3 | 95.3 | 84.5 | 76.7 | 60.7 | 70.7 | 75.9 |
ρ is the per-call budget. Baselines and DEP with one mask are from Table 1; the union rows are one run on a different host, on which DEP with one mask scores 62.2 at ρ=25% (63.4 in Table 1).
Results. A batch union requires more memory than a single mask, both on average and in the largest batch: 19.4–32.9 GB against 16.4 GB at ρ=25%. It also scores higher than one mask per call, because more experts are resident. Still, at ρ=25% the union outperforms every ρ=50% baseline with less memory. Batched by task, it uses 22.9 GB on average and 25.3 GB in its largest batch, against 29.9 GB, and scores 75.8 against 67.4 for the best baseline, even though those baselines keep more experts per layer (64) than this union does on average (47.4). Batched by mask similarity over the mixed traffic of all six tasks, every batch stays within the baselines' 29.9 GB (19.4 GB on average, 29.7 GB at most), and it scores 71.1. Only random batching needs more, 32.9 GB. Due to the limited time of the response period, we could not run random batching with the union capped at 29.9 GB, but we believe it would still outperform the baselines.
Limitations and future work. As the reviewer pointed out, our method is not designed for batching, and a substantial amount of future work remains. There are smarter ways to adapt our algorithm to union batching, for example masking by the predictor's mean scores over a batch and training the predictor for that objective, but these are beyond the scope of our original paper. As in our response to Reviewer 8uyL, multi-agent structures beyond those we evaluate, such as the debate the reviewer describes, in which an agent's calls are spread across rounds, raise several interesting research questions, and we hope future work will address them. Our contribution is limited to showing the necessity of dynamic expert pruning and the possibility of predictor-based, task- and system-prompt-aware dynamic expert pruning. We will state these limitations and the scope of our work clearly in the paper, and include the union-batching analysis as a starting point for future work.
We appreciate the careful reading. Both of the reviewer's points go to how DEP would actually be served, and both turned out to be answerable with measurements rather than argument, so we have run them. Here, we present these additional experimental results and explanations for each point raised.
W1 — "per-input masks make batching very difficult, if possible at all"
The reviewer raises an important point. DEP can be batched by agent, our implementation already batches, and every accuracy and timing number in the submission was produced with batched generation rather than one request at a time.
The predictor is supervised per input during training, but at serving time a mask is not computed per request and does not change during one. One mask is produced for an (agent system prompt, task input) pair before the turn begins and is then held fixed: it does not change from token to token, and it does not change across the repeated calls a workflow makes on the same problem. Requests that share an agent and a problem therefore run under one mask and are batched together, with no mask switch between them. This is the common case rather than a corner case, because workflows call one agent several times on one problem — parallel samples feeding a self-consistency ensemble, a debate agent across rounds, a verify-then-retry loop. Across the six workflows of our suite, 96% of agent calls share their mask with at least one other call, and a mask carries 3.24 calls on average. A batch that mixes different agents of a task needs the union of their masks resident: in our workflows, this is 39–49 of the 128 experts per layer at ρ=25%, against 32 for a single agent's mask.
We recognise that this part of the submission can be confusing, since the task input is part of the predictor's input while Section 3.4 speaks of "batches grouped by agent". We have clarified the granularity at which a mask is produced, cached, and batched in Section 3.4 of the revised paper.