acceptodds
Under review as a conference paper at ICLR 2027

When Routing Edits Do Not Add Up: Budgeted Selection in Mixture-of-Experts Models

Abstract

Compressing a sparse mixture-of-experts model changes which experts its router selects. Candidate edits are often scored one at a time, although an upstream edit changes the hidden states on which later edits act. We find that edits that help in isolation can lose reference-answer log probability when combined. Under additive logit responses, a local quadratic expansion expresses pairwise interaction through the softmax Fisher inner product. We propose Ranked Validation, which measures the empty set and every single edit, orders larger candidate sets by summed single-edit scores, measures the leading sets, and returns the best set measured. For bounded interactions, we establish a sharp worst-case regret limit for deterministic scan-only selection; Ranked Validation satisfies the same upper bound once its record contains the additive maximizer. Across four checkpoints and thirteen candidate populations, it improves pooled exact selection over greedy and random validation under matched query budgets. On the half-pruning population, four extra evaluations recover 29.0% of the lost reference-answer log probability. A set selected once and reused across held-out questions adds little beyond a calibration-selected single-edit baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, routing interventions, interaction effects, combinatorial selection, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rUR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates non-additive interactions among routing edits in Mixture-of-Experts (MoE) models. The authors observe that edits that improve model performance individually may become ineffective or even harmful when combined, as upstream modifications alter the hidden states received by downstream layers. To address this issue, the paper proposes Ranked Validation, a budgeted selection strategy that first evaluates individual edits, ranks candidate combinations by their summed single-edit gains, and then evaluates the highest-ranked combinations under a limited query budget. The paper also provides a theoretical analysis of interaction effects using Möbius decomposition and a local Fisher approximation, along with worst-case regret bounds and conditions under which additive ranking is optimal. The method is evaluated on four MoE checkpoints across 13 candidate populations involving pruning, expert restoration, and routing repair. Experiments compare Ranked Validation against greedy selection, random validation, and more sophisticated combinatorial search methods using exact selection, regret, and harm rate. The results show improvements over greedy and random validation under matched query budgets, although some more sophisticated search methods achieve comparable or better performance. Additional experiments examine recovery from REAP pruning and generalization across questions, finding that the benefits of joint selection are largely question-dependent and provide limited improvement over a single-edit baseline when a fixed edit set is transferred to held-out questions.

Critical strengths
  1. The paper provides a quantitative investigation of how individually beneficial routing edits can interact negatively when combined. By exhaustively evaluating feasible edit combinations, the authors directly measure non-additive effects and compare selection strategies against the exact optimum. The experiments across multiple MoE checkpoints and candidate populations provide useful empirical evidence about the limitations of additive scoring.
  2. The paper formalizes routing-edit selection as a budgeted combinatorial optimization problem and uses Möbius decomposition to characterize interaction effects. The worst-case regret analysis and the local Fisher interpretation provide a structured explanation of the limitations of single-edit scoring. Although these results do not establish a general advantage of the proposed algorithm, they help clarify the underlying selection problem and its theoretical limitations.
Critical weaknesses
  1. The central observation that upstream routing modifications affect downstream hidden states, making the effects of multiple edits non-additive, is relatively intuitive and has been considered in prior interaction-aware MoE compression and combinatorial optimization methods. Despite identifying these interactions as the main challenge, the proposed Ranked Validation algorithm simply ranks combinations by summed single-edit scores and evaluates a fixed number of top-ranked candidates. It does not explicitly model interactions or adapt its search based on the joint evaluations already observed, and can therefore miss promising combinations ranked poorly by the additive approximation. The theoretical results do not establish a general advantage over alternative search strategies: the worst-case regret bound is also attained by the additive maximizer, while the optimality of additive ordering relies on independent and identically distributed interactions, an assumption that may be unrealistic for overlapping edit sets. Moreover, existing adaptive approaches such as GP-UCB and local search achieve comparable or better results in the experiments. These limitations make it difficult to identify a sufficiently novel or principled algorithmic contribution beyond a straightforward ranking-and-validation heuristic.
  2. The main experiments consider small candidate populations (at most eight edits) with a maximum of three simultaneous modifications, leaving the effectiveness of the approach on larger MoE models and more realistic search spaces unclear. Although the number of additional joint evaluations is limited, the method still requires evaluating every individual edit and ranking a combinatorial number of candidate sets. More importantly, its main per-question selection experiments rely on the reference answer's log probability to evaluate candidate combinations, which is unavailable during ordinary inference. When a fixed edit set is instead selected using calibration questions and transferred to held-out inputs, the improvement over the best single-edit baseline is statistically insignificant. Furthermore, the REAP restoration experiments do not establish an advantage over existing compression methods under matched final inference costs. Consequently, the reported selection improvements do not convincingly demonstrate that Ranked Validation offers meaningful benefits in practical MoE compression or deployment settings.
Questions for authors
  1. Existing MoE compression and combinatorial optimization methods have explored more sophisticated search strategies, including evolutionary search and adaptive combinatorial optimization. What distinguishes Ranked Validation from these existing approaches beyond its simple additive ranking and subsequent validation? Given that GP-UCB and local search already achieve comparable or better results in the paper, what evidence supports the proposed method as a meaningful algorithmic contribution?
  2. Proposition 2 establishes the optimality of additive ordering under independent and identically distributed interactions. However, routing edits across layers are inherently dependent, and different candidate combinations often share the same edits. To what extent does this assumption hold in the evaluated settings? Can the authors provide empirical evidence that additive ranking remains effective when interactions are strongly correlated?
  3. The evaluation is restricted to at most eight candidate edits and three simultaneous modifications, while realistic MoE compression may involve substantially larger search spaces. Furthermore, the main per-question selection results require reference answers, and the fixed-set transfer experiments show little improvement over the single-edit baseline. Can the authors clarify the intended practical use case and provide evidence that Ranked Validation offers meaningful benefits under realistic compression budgets and held-out evaluation settings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer rUR1 (1/3)

Thank you for an expert and meticulous review. Each of your questions led to a new measurement that now strengthens the revised paper. Here are the replies.

W1.1: That edits interact is intuitive, and interaction-aware compression already considers it.

Answer: The contribution is the shape of the failure, measured against the exhaustive optimum, and the selection rule it implies. Summed single-edit gains fail as a predictor of joint utility and succeed as a search order. This is the optimizer's curse [1] in combinatorial form: maximizing an estimate selects for its errors. Interaction-aware compression fits a richer predictor and still commits to its unmeasured maximizer, harming decisions until the search returns its best measured set. Results at t = 4 extra evaluations on the 5,828 submitted decisions:

How the score is usedLeaves the model worseFinds the best set
Additive score as a predictor (top-scored set)17.4%10.8%
Second-order score as a predictor9.5%25.5%
Second-order search, best measured set0%35.4%
Additive score as an order (Ranked Validation)0%38.3%

Applying the top-scored set is the default step in MoE compression pipelines. It leaves the model worse than no edit in 17.4% of these decisions, and one extra forward pass prevents that loss. Hence our principle: the score decides what to test, and measurement decides what to keep.

W1.2 & Q1: Ranked Validation neither models interactions nor adapts. What distinguishes it from GP-UCB, local search and evolutionary search?

Answer: Ranked Validation supplies the order that adaptive search lacks, and the best rule from four extra evaluations onward is GP-UCB started from that order. Ranked Validation is the parallel member of this family, requiring no surrogate, no hyperparameter, and one batch for all t evaluations. Each guarantee (the floor at the unedited model, the monotone budget curve, and the 2ε worst-case bound) extends to any measured search started at the additive argmax (Proposition 10). GP-UCB alone starts behind Ranked Validation and passes it at the third query, whereas starting at the additive argmax removes the slow start. Sequential greedy and evolutionary search ignore the order and trail at every budget. Any search that leaves a low-ranked set unmeasured can miss it (the second half of Theorem 1), and Theorem 2 bounds that risk: every set ahead of the optimum in the additive order lies within 2ε of it. Exact selection (%) over all 11,204 decisions (the 5,828 submitted and 5,376 recorded after the rules were frozen) at an equal budget of n + 1 + t forward passes (Tables 12 and 13 of the revised paper):

RuleUses the additive ordert = 1t = 4t = 7
Ranked Validation (one batch)Yes28.348.160.9
Additive argmax, then GP-UCB (sequential)Yes28.349.863.8
Single-swap local searchYes28.348.762.3
GP-UCBNo25.249.563.5
Sequential greedyNo22.234.445.8
Evolutionary searchNo21.131.440.8

W1.3: The additive argmax attains the same worst-case bound, so the theory shows no advantage.

Answer: We agree, and Theorem 1 states this directly: the bound limits every rule that reads only the single-edit scan, so worst-case regret alone cannot certify safety. The additive argmax attains the bound and still harms decisions (table under W1.1). Ranked Validation adds what the bound cannot provide: a hard floor at the unedited model and a non-decreasing budget curve. The theorem also shows why a richer interaction model fitted to the scan cannot certify an unmeasured set, so a safe selector must return a measured set.

Reference

[1] A. Tsiourvas and G. Perakis. Overcoming the Optimizer's Curse: Obtaining Realistic Prescriptions from Neural Networks. ICML 2024.

AuthorsOpenReview ↗

Response to Reviewer rUR1 (2/3)

W1.4 & Q2: Proposition 2 assumes independent interactions. Does the additive order stay effective when interactions are correlated?

Answer: Yes. Independence is a sufficient condition for optimality, and the measured lead holds on the most strongly correlated decisions. In the evaluated populations, interactions are not independent: disjoint sets correlate negatively within a decision, and overlapping sets correlate weakly. We split the decisions into terciles by within-decision interaction correlation and replayed the selectors. Ranked Validation leads sequential greedy and a random order in every tercile (Table 14 of the revised paper). Exact-selection lead in percentage points at t = 4 in the most correlated tercile, with 95% intervals:

Most correlated tercileLead over sequential greedyLead over a size-matched random order
Disjoint sets (median correlation −0.484)+6.7 [5.2, 8.2]+10.4 [9.2, 11.6]
Overlapping sets (median correlation 0.086)+14.3 [12.6, 15.9]+18.0 [16.6, 19.5]

Pandora-problem theory aligns with this result: the search problem that the additive order solves under independence retains a constant-factor approximation guarantee under correlated values [2].

W2.1 & Q3.1: The candidate populations are small (at most eight edits, three at a time), and the method must still scan every edit and rank a combinatorial number of sets.

Answer: Ranked Validation never builds the combinatorial list, and its lead holds in a larger search space. The seventeen populations span five checkpoints from 7B to 35B parameters. The scan is the single-edit scoring that compression pipelines already run. From the n sorted gains, a max-heap emits sets in exact additive order (Algorithm 1 of the revised paper) in 0.06 ms at n = 64, k = 8 and t = 32, where the full space holds 5,130,659,496 sets. On OLMoE-1B-7B with twelve candidates and up to four edits, we recorded all 794 feasible sets across 256 decisions (Table 17 of the revised paper):

n = 12, k = 4, t = 16Exact selectionLeaves the model worse
Random validation2.8%0%
Sequential greedy6.6%0%
Ranked Validation45.3%0%
Additive argmax, then GP-UCB56.6%0%

W2.2 & Q3.2: Per-question selection reads the reference answer, which ordinary inference does not have.

Answer: The selector reads one scalar per measured set, so a label-free utility replaces the reference answer and every guarantee carries over in that utility. The natural choice in compression is fidelity to the unpruned model, −KL(p_unpruned ‖ p_S), the teacher signal optimized when recovering pruned LLMs via distillation from the original model [3]. With it, Ranked Validation recovers a third of the REAP pruning loss on OLMoE-1B-7B without a single label (new Section 5.5 of the revised paper, see the table under W2.4 in the next comment).

Reference

[2] N. Bansal, Z. Huang and Z. Zhu. Optimal 4-Approximation for the Correlated Pandora's Problem. FOCS 2025.
[3] S. Oh, D. Lee and J. Ryu. Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning. EMNLP 2026.

AuthorsOpenReview ↗

Response to Reviewer rUR1 (3/3)

W2.3 & Q3.3: A fixed set transferred to held-out questions adds little over the best single edit. What is the practical use case?

Answer: The transfer result is a prediction of our theory, showing where joint search pays: per input. The local Fisher form predicts that routing interactions are input-specific and cancel under dataset averaging, and the measured pair-to-single ratio drops from 0.55 per question to 0.15 on the averaged objective. A shared set therefore retains the additive gain, whereas the interaction gain is collected per input, matching recent test-time per-sample routing optimization [4, 5]. Two use cases follow:

  1. A safeguard for any pipeline that already scores single edits: measure the top-scored set before committing to it. One extra forward pass removes the 17.4% harm rate.
  2. Per-input routing repair: 13 forward passes per input recover 29.0% of the REAP pruning loss, against 34.7% for the exhaustive optimum.

W2.4: The REAP restoration experiments show no advantage at matched inference cost.

Answer: Inference cost is matched by construction: each restored expert replaces the lowest-weight active expert of its layer, so the repaired model runs as many active experts per token as the pruned model. Ranked Validation is a selection layer on top of a compressor, here the published REAP pruning, so we compare the compressed model before and after selection. The gain comes from selection rather than a larger expert pool, as the control with all candidates restored shows. Recovered share of the pruning loss on OLMoE-1B-7B, with the same available and active experts in every row (Section 5.5 and Table 16 of the revised paper):

RestorationRecovered pruning loss
None (REAP-pruned model)0%
All eight candidates on every question−16.5% [−34.9, −1.7]
Ranked Validation, at most three per question, label-free33.3% [27.1, 39.8]

If these measurements resolve your concerns regarding novelty and practical value, we would be grateful if you would consider raising your score.

Thank you again for your expert and constructive review. We're happy to address any further questions you may have.

Reference

[4] Z. Li et al. C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing. COLM 2025.
[5] G. Su et al. Rewiring Experts on the Fly: Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert Models. ICML 2026.

Reviewer_agRHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies editing (pruning and restoring) in mixture of experts for use in downstream tasks. Their proposed method for editing of experts, tests scores of single individual edits use them to form an score estimates for sets of edits, equal to the sum of scores of the edits inside the set. Then it sorts the k-element sets of edits based on their score estimates, tests top R sets, and then choses the best set with the highest tested score. They provide supporting theory and bounds on the sub-optimality, and experiments that compare the proposed methods against existing baselines

Critical strengths

The reviewer could not spot a critical strength in this paper that can really support acceptance in ICLR.

Non-critical strengths include:

  • The propose method is intuitive and simple.
  • The paper provides empirical comparison of different editing schemes. In particular, I found the results reported in Figure 2 interesting.
Critical weaknesses

significance of contributions:

  • I did not find the theoretical results useful, and I think removing them would improve the paper. They are too simple, provide very little or no new intuition, and they take extra space and energy from the reader to go through all the definitions and not-rigorous statements to get a result that does not really contribute to the flow of the paper or understanding of the reader. For example, Theorem 1 is quite trivial with an elementary proof, yet it explains a simple bound with unnecessarily complex assumptions and definitions.
  • Little or no performance improvement compared to baselines: Many baselines perform similarly or outperform the proposed solution. Some baselines, like Adaptive Sparse Mobius, even have lower complexity, according to Table 7, and the appendix claims editing has negligible complexity compared to model run cost anyways.

Presentation:

  • In general, the paper is poorly written, it is difficult to read and follow. Assumptions are written in an unclear way, and the arguments are too condensed.
  • The empirical improvement reported in the abstract is misleading. The 29% gain reported in the abstract is per-question. In other words, for each question, they apply the algorithm to find the best edit for that specific question, and then measure the improvement on the log-probability of the same sample. I think this is a misleading measure of improvement and should be removed from abstract and introduction, especially given that the per-question nature of the improvement measure is not clarified in the abstract and introduction.

Experimental setting:

  • The evaluations only edits the last token, which is too simplistic. In practice, an edit/pruning decision at one layer affects all tokens.
Questions for authors

What is the cost of model run? It should be reported alongside Table 7.

5 comments
AuthorsOpenReview ↗

Response to Reviewer agRH (1/2)

Thank you for your review and for the time you spent on our paper. Here are our replies.

W1: "I did not find the theoretical results useful, and I think removing them would improve the paper ... Theorem 1 is quite trivial with an elementary proof."

Answer: Theorem 1 explains why Ranked Validation returns a measured set, so we keep it and state it in plain words. We believe the theorem does substantive work for the reader because it turns the measured failure into a design rule: a safe selector has to return a set it has measured. The theorem proves that when a selector reads only single-edit scores, hidden interactions of size ε can cost its chosen set 2ε against an unmeasured set. No scoring rule is safe alone. The experiments agree: every rule that commits to an unmeasured prediction harms decisions, and every rule that returns a measured set harms none (table under W2.1). The proof is short because the adversary is explicit, which also makes the bound tight. The statement uses one assumption, that interactions on feasible sets are bounded by ε. Following your comment, the revised paper states this assumption in words before the theorem, lists the three consequences of the theorem for the method after it, and places the quantifiers and proof in Appendix A.

W2.1: "Many baselines perform similarly or outperform the proposed solution."

Answer: The baselines that match Ranked Validation follow the same principle, as each search returns its best measured set. Selectors used in compression pipelines commit to an unmeasured prediction and either harm the model or trail in quality. Below are the results at t = 4 extra evaluations on the 5,828 submitted decisions (Table 4 of the revised paper):

SelectorReturns a measured setLeaves the model worseFinds the best set
Top-scored set (common practice)No17.4%10.8%
Second-order, interaction-aware predictionNo9.5%25.5%
Sequential greedyYes0%31.5%
Ranked ValidationYes0%38.3%
Adaptive searches (GP-UCB, local search, sparse Möbius, Pandora)Yes0%38.0–39.3%

Applying the top-scored set is the default step in MoE compression pipelines. It leaves the model worse than no edit at all in about one of every six of these decisions, and one extra forward pass prevents that loss. The strongest search builds on our additive order: GP-UCB started at the additive argmax is the best rule from four extra evaluations onward (Tables 12 and 13 of the revised paper).

W2.2 & Q1: "Some baselines, like Adaptive Sparse Mobius, even have lower complexity, according to Table 7" and "What is the cost of model run?"

Answer: One model run takes 157 ms, and the slowest selector takes 1.0 ms per decision. Every rule spends the same number of forward passes at a given budget. Those forward passes set the time per decision, and the selectors differ by less than one millisecond. Table 8 of the revised paper reports the selector times beside the measured cost of one forward pass:

Cost per decision, Qwen3.5-35B-A3B-Base on one H200Time
One forward pass157 ms
Selector overhead of sparse Möbius (CPU)0.17 ms
Selector overhead of Ranked Validation (CPU)0.27 ms
Selector overhead of the slowest rule (CPU)1.0 ms
Ranked Validation at t = 4, 13 forward passes2.05 s
Every feasible set, 93 forward passes14.6 s
AuthorsOpenReview ↗

Response to Reviewer agRH (2/2)

W3: "In general, the paper is poorly written, it is difficult to read and follow."

Answer: Thank you for this feedback, and we have revised the paper to make it easier to follow. Sections 3 to 5 each open with a statement of their purpose. Section 3 adds a notation paragraph. Each formal result is introduced by a plain-language statement of its assumption and conclusion, and Theorem 1 is followed by its three consequences for the method. Section 5 states what the experiments verify and where each result is found, and the main results state each finding in bold at the head of its paragraph. Each appendix section opens with its purpose and introduces each formal result by its role and proof idea.

W4: "The 29% gain reported in the abstract is per-question ... the per-question nature of the improvement measure is not clarified in the abstract and introduction."

Answer: The revised abstract and introduction state "per-question selection" in the same sentence as the 29.0% figure. Thank you for raising this point. In the original submission, the scope appears one sentence after the number. The introduction reads "The gain is largely per-question: a set selected on calibration questions adds little beyond the single-edit baseline on held-out questions" (lines 092–093), and the abstract states the same contrast (lines 024–025). With the scope stated beside the number, we keep the result in the abstract and introduction.

We study per-question selection and score every selector against each question's exhaustive optimum. That optimum recovers 34.7% of the pruning loss, and Ranked Validation recovers 29.0% with 13 of the 93 possible forward passes. Selection requires no reference answer. Optimizing fidelity to the unpruned model recovers 33.3% of the pruning loss on OLMoE-1B-7B without a label (Section 5.5 of the revised paper).

W5: "The evaluations only edits the last token, which is too simplistic."

Answer: We repeated the full exhaustive study with every edit applied at every prompt token, and every conclusion holds. On OLMoE-1B-7B, we recorded all 256 subsets on 512 questions under both scopes (Section 5.3 and Table 15 of the revised paper):

t = 4 extra evaluationsScored tokenEvery token
Decisions harmed by the top-scored set10.5%11.7%
Decisions harmed by Ranked Validation0%0%
Exact selection of sequential greedy7.4%9.2%
Exact selection of Ranked Validation35.2%30.7%
Lead of Ranked Validation over greedy (points)+27.7+21.5
Median pair-to-single interaction ratio0.790.82

In the original submission, the shared-set evaluation also restores each expert at every token position (Appendix B, lines 810–811). The per-question populations edit the scored token because utility is read from that token's forward pass.

If the revision and the new measurements address your concerns, we would be grateful if you would consider raising your score.

Thank you again for your review. We're happy to address any further questions you may have.

AuthorsOpenReview ↗

Follow-up before the discussion closes

As the discussion period draws to a close, we'd like to thank you again for your thorough review and the time you've given our paper.

We'd be very grateful if you could share your views on our rebuttal before the discussion ends. Your feedback would help us understand whether our responses have addressed your concerns, and it would also give the AC a more complete picture of the discussion for the meta-review. If our responses have resolved your concerns, we'd also appreciate it if you would consider raising your rating. If any concern remains, we'd be glad to address it right away.

Thank you very much for your time and your valuable comments.

Reviewer_dPo4OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue of MoE models failing during compression and route repair due to upstream layer hidden state changes causing single-edit score combinations to fail. It proposes a "Ranked Validation" method. This method obtains the summation ranking through a single-edit scan, prioritizing the testing of leading candidate combinations within a limited evaluation budget. This effectively recovers the performance lost due to pruning across multiple models and tasks with minimal additional overhead.

Critical strengths
  1. The problems studied have certain practical value: compression, pruning and restoration of MoE models are current research hotspots for the efficient deployment of large models, while "excellent in individual aspects but failure in combination" is a deep mathematical and state interaction problem in algorithm design and combinatorial optimization.
  2. This paper performs extremely well in terms of theoretical soundness. Its core theoretical framework is rigorous and progressive.
Critical weaknesses

1.The utility function heavily relies on the ground-truth labels, making it difficult to directly run and verify in real-world unsupervised/online reasoning scenarios.

Questions for authors
  1. The proposed method heavily relies on target task ground-truth labels. However, in many real-world online inference or unsupervised downstream scenarios, ground-truth is typically unavailable. Although the paper touch upon label-free panel Prefix Validation in the limitations, how might Ranked Validation be extended in future work to unsupervised, weakly supervised settings, or proxy metrics derived from the model's internal uncertainty?
5 comments
AuthorsOpenReview ↗

Response to Reviewer dPo4

Thank you for an insightful review. Your question on deployment without ground-truth labels led to a new main-text section, Section 5.5, on selection without reference answers. Here are the replies.

W1 & Q1: The utility relies on ground-truth labels, which online and unsupervised settings do not have.

Answer: Ranked Validation requires no label: it reads one scalar utility per measured set, so any label-free utility replaces the reference answer without changing the algorithm. The floor at the unedited model, the monotone budget curve, and the 2ε worst-case bound hold in the selection utility. We evaluated label-free selection after REAP half pruning of OLMoE-1B-7B. Each label-free utility selects a set with t = 4 extra evaluations, scored against the reference answer on 512 questions. Recovered share of the pruning loss, with edits at the scored token and at every prompt token (Section 5.5 and Table 16 of the revised paper):

Utility and selector, no labelsScored tokenEvery token
None (pruned model)0%0%
No selection, all eight candidates restored−16.5%−14.6%
Negative entropy, Ranked Validation13.5%9.0%
Fidelity to the unpruned model, random validation27.9%27.9%
Fidelity to the unpruned model, sequential greedy27.7%30.9%
Fidelity to the unpruned model, Ranked Validation33.3%37.5%

Q1.1: Extension to unsupervised settings.

Answer: Fidelity to the unpruned model, −KL(p_unpruned ‖ p_S), is the unsupervised utility and recovers a third of the pruning loss without a single label. It is the teacher signal optimized when recovering pruned LLMs via distillation from the original model [1], and applies directly during compression because the unpruned model is available. It is also the objective for which the paper derives the Fisher interaction form, so the theory carries over unchanged. For online deployment, the same scalar interface accepts self-supervised objectives computed from the input context by recent rerouting methods [2].

Q1.2: Extension to weakly supervised settings.

Answer: Under weak supervision, a small labeled calibration split fixes one shared set, and a label-free utility then guides the joint search on each test input. This division follows from our measurements. With calibration labels alone, the shared set improves held-out questions by 0.092 nats over the pruned model, within 0.001 nats of the best single edit, because routing interactions are input-specific and cancel under dataset averaging.

Q1.3: Proxy metrics from the model's internal uncertainty.

Answer: Negative predictive entropy requires neither labels nor the unpruned model, and recovers part of the pruning loss at the scored token (table above). Entropy minimization is an established label-free objective for LLM reasoning [3]. Fidelity to the unpruned model is stronger in our setting because it anchors the repaired model to the unpruned distribution, whereas a confidence signal can reward a confident wrong prediction.

Section 3.1 of the revised paper now states that any scalar function of a measured set can serve as the utility, with these label-free objectives as instances.

We hope the label-free results reinforce your support for the paper, and we would be grateful if you would consider raising your score.

Thank you again for your insightful question, which made the paper stronger. We're happy to address any further questions you may have.

Reference

[1] S. Oh, D. Lee and J. Ryu. Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning. EMNLP 2026.
[2] G. Su et al. Rewiring Experts on the Fly: Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert Models. ICML 2026.
[3] S. Agarwal et al. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. NeurIPS 2025.

AuthorsOpenReview ↗

Follow-up before the discussion closes

As the discussion period draws to a close, we'd like to thank you again for your thorough review and the time you've given our paper.

We'd be very grateful if you could share your views on our rebuttal before the discussion ends. Your feedback would help us understand whether our responses have addressed your concerns, and it would also give the AC a more complete picture of the discussion for the meta-review. If our responses have resolved your concerns, we'd also appreciate it if you would consider raising your rating. If any concern remains, we'd be glad to address it right away.

Thank you very much for your time and your valuable comments.

Dear Reviewer,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.