- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates non-additive interactions among routing edits in Mixture-of-Experts (MoE) models. The authors observe that edits that improve model performance individually may become ineffective or even harmful when combined, as upstream modifications alter the hidden states received by downstream layers. To address this issue, the paper proposes Ranked Validation, a budgeted selection strategy that first evaluates individual edits, ranks candidate combinations by their summed single-edit gains, and then evaluates the highest-ranked combinations under a limited query budget. The paper also provides a theoretical analysis of interaction effects using Möbius decomposition and a local Fisher approximation, along with worst-case regret bounds and conditions under which additive ranking is optimal. The method is evaluated on four MoE checkpoints across 13 candidate populations involving pruning, expert restoration, and routing repair. Experiments compare Ranked Validation against greedy selection, random validation, and more sophisticated combinatorial search methods using exact selection, regret, and harm rate. The results show improvements over greedy and random validation under matched query budgets, although some more sophisticated search methods achieve comparable or better performance. Additional experiments examine recovery from REAP pruning and generalization across questions, finding that the benefits of joint selection are largely question-dependent and provide limited improvement over a single-edit baseline when a fixed edit set is transferred to held-out questions.
Critical strengths
- The paper provides a quantitative investigation of how individually beneficial routing edits can interact negatively when combined. By exhaustively evaluating feasible edit combinations, the authors directly measure non-additive effects and compare selection strategies against the exact optimum. The experiments across multiple MoE checkpoints and candidate populations provide useful empirical evidence about the limitations of additive scoring.
- The paper formalizes routing-edit selection as a budgeted combinatorial optimization problem and uses Möbius decomposition to characterize interaction effects. The worst-case regret analysis and the local Fisher interpretation provide a structured explanation of the limitations of single-edit scoring. Although these results do not establish a general advantage of the proposed algorithm, they help clarify the underlying selection problem and its theoretical limitations.
Critical weaknesses
- The central observation that upstream routing modifications affect downstream hidden states, making the effects of multiple edits non-additive, is relatively intuitive and has been considered in prior interaction-aware MoE compression and combinatorial optimization methods. Despite identifying these interactions as the main challenge, the proposed Ranked Validation algorithm simply ranks combinations by summed single-edit scores and evaluates a fixed number of top-ranked candidates. It does not explicitly model interactions or adapt its search based on the joint evaluations already observed, and can therefore miss promising combinations ranked poorly by the additive approximation. The theoretical results do not establish a general advantage over alternative search strategies: the worst-case regret bound is also attained by the additive maximizer, while the optimality of additive ordering relies on independent and identically distributed interactions, an assumption that may be unrealistic for overlapping edit sets. Moreover, existing adaptive approaches such as GP-UCB and local search achieve comparable or better results in the experiments. These limitations make it difficult to identify a sufficiently novel or principled algorithmic contribution beyond a straightforward ranking-and-validation heuristic.
- The main experiments consider small candidate populations (at most eight edits) with a maximum of three simultaneous modifications, leaving the effectiveness of the approach on larger MoE models and more realistic search spaces unclear. Although the number of additional joint evaluations is limited, the method still requires evaluating every individual edit and ranking a combinatorial number of candidate sets. More importantly, its main per-question selection experiments rely on the reference answer's log probability to evaluate candidate combinations, which is unavailable during ordinary inference. When a fixed edit set is instead selected using calibration questions and transferred to held-out inputs, the improvement over the best single-edit baseline is statistically insignificant. Furthermore, the REAP restoration experiments do not establish an advantage over existing compression methods under matched final inference costs. Consequently, the reported selection improvements do not convincingly demonstrate that Ranked Validation offers meaningful benefits in practical MoE compression or deployment settings.
Questions for authors
- Existing MoE compression and combinatorial optimization methods have explored more sophisticated search strategies, including evolutionary search and adaptive combinatorial optimization. What distinguishes Ranked Validation from these existing approaches beyond its simple additive ranking and subsequent validation? Given that GP-UCB and local search already achieve comparable or better results in the paper, what evidence supports the proposed method as a meaningful algorithmic contribution?
- Proposition 2 establishes the optimality of additive ordering under independent and identically distributed interactions. However, routing edits across layers are inherently dependent, and different candidate combinations often share the same edits. To what extent does this assumption hold in the evaluated settings? Can the authors provide empirical evidence that additive ranking remains effective when interactions are strongly correlated?
- The evaluation is restricted to at most eight candidate edits and three simultaneous modifications, while realistic MoE compression may involve substantially larger search spaces. Furthermore, the main per-question selection results require reference answers, and the fixed-set transfer experiments show little improvement over the single-edit baseline. Can the authors clarify the intended practical use case and provide evidence that Ranked Validation offers meaningful benefits under realistic compression budgets and held-out evaluation settings?
5 comments
Response to Reviewer rUR1 (2/3)
W1.4 & Q2: Proposition 2 assumes independent interactions. Does the additive order stay effective when interactions are correlated?
Answer: Yes. Independence is a sufficient condition for optimality, and the measured lead holds on the most strongly correlated decisions. In the evaluated populations, interactions are not independent: disjoint sets correlate negatively within a decision, and overlapping sets correlate weakly. We split the decisions into terciles by within-decision interaction correlation and replayed the selectors. Ranked Validation leads sequential greedy and a random order in every tercile (Table 14 of the revised paper). Exact-selection lead in percentage points at t = 4 in the most correlated tercile, with 95% intervals:
| Most correlated tercile | Lead over sequential greedy | Lead over a size-matched random order |
|---|---|---|
| Disjoint sets (median correlation −0.484) | +6.7 [5.2, 8.2] | +10.4 [9.2, 11.6] |
| Overlapping sets (median correlation 0.086) | +14.3 [12.6, 15.9] | +18.0 [16.6, 19.5] |
Pandora-problem theory aligns with this result: the search problem that the additive order solves under independence retains a constant-factor approximation guarantee under correlated values [2].
W2.1 & Q3.1: The candidate populations are small (at most eight edits, three at a time), and the method must still scan every edit and rank a combinatorial number of sets.
Answer: Ranked Validation never builds the combinatorial list, and its lead holds in a larger search space. The seventeen populations span five checkpoints from 7B to 35B parameters. The scan is the single-edit scoring that compression pipelines already run. From the n sorted gains, a max-heap emits sets in exact additive order (Algorithm 1 of the revised paper) in 0.06 ms at n = 64, k = 8 and t = 32, where the full space holds 5,130,659,496 sets. On OLMoE-1B-7B with twelve candidates and up to four edits, we recorded all 794 feasible sets across 256 decisions (Table 17 of the revised paper):
| n = 12, k = 4, t = 16 | Exact selection | Leaves the model worse |
|---|---|---|
| Random validation | 2.8% | 0% |
| Sequential greedy | 6.6% | 0% |
| Ranked Validation | 45.3% | 0% |
| Additive argmax, then GP-UCB | 56.6% | 0% |
W2.2 & Q3.2: Per-question selection reads the reference answer, which ordinary inference does not have.
Answer: The selector reads one scalar per measured set, so a label-free utility replaces the reference answer and every guarantee carries over in that utility. The natural choice in compression is fidelity to the unpruned model, −KL(p_unpruned ‖ p_S), the teacher signal optimized when recovering pruned LLMs via distillation from the original model [3]. With it, Ranked Validation recovers a third of the REAP pruning loss on OLMoE-1B-7B without a single label (new Section 5.5 of the revised paper, see the table under W2.4 in the next comment).
Reference
[2] N. Bansal, Z. Huang and Z. Zhu. Optimal 4-Approximation for the Correlated Pandora's Problem. FOCS 2025.
[3] S. Oh, D. Lee and J. Ryu. Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning. EMNLP 2026.
Response to Reviewer rUR1 (3/3)
W2.3 & Q3.3: A fixed set transferred to held-out questions adds little over the best single edit. What is the practical use case?
Answer: The transfer result is a prediction of our theory, showing where joint search pays: per input. The local Fisher form predicts that routing interactions are input-specific and cancel under dataset averaging, and the measured pair-to-single ratio drops from 0.55 per question to 0.15 on the averaged objective. A shared set therefore retains the additive gain, whereas the interaction gain is collected per input, matching recent test-time per-sample routing optimization [4, 5]. Two use cases follow:
- A safeguard for any pipeline that already scores single edits: measure the top-scored set before committing to it. One extra forward pass removes the 17.4% harm rate.
- Per-input routing repair: 13 forward passes per input recover 29.0% of the REAP pruning loss, against 34.7% for the exhaustive optimum.
W2.4: The REAP restoration experiments show no advantage at matched inference cost.
Answer: Inference cost is matched by construction: each restored expert replaces the lowest-weight active expert of its layer, so the repaired model runs as many active experts per token as the pruned model. Ranked Validation is a selection layer on top of a compressor, here the published REAP pruning, so we compare the compressed model before and after selection. The gain comes from selection rather than a larger expert pool, as the control with all candidates restored shows. Recovered share of the pruning loss on OLMoE-1B-7B, with the same available and active experts in every row (Section 5.5 and Table 16 of the revised paper):
| Restoration | Recovered pruning loss |
|---|---|
| None (REAP-pruned model) | 0% |
| All eight candidates on every question | −16.5% [−34.9, −1.7] |
| Ranked Validation, at most three per question, label-free | 33.3% [27.1, 39.8] |
If these measurements resolve your concerns regarding novelty and practical value, we would be grateful if you would consider raising your score.
Thank you again for your expert and constructive review. We're happy to address any further questions you may have.
Reference
[4] Z. Li et al. C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing. COLM 2025.
[5] G. Su et al. Rewiring Experts on the Fly: Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert Models. ICML 2026.
Response to Reviewer rUR1 (1/3)
Thank you for an expert and meticulous review. Each of your questions led to a new measurement that now strengthens the revised paper. Here are the replies.
Answer: The contribution is the shape of the failure, measured against the exhaustive optimum, and the selection rule it implies. Summed single-edit gains fail as a predictor of joint utility and succeed as a search order. This is the optimizer's curse [1] in combinatorial form: maximizing an estimate selects for its errors. Interaction-aware compression fits a richer predictor and still commits to its unmeasured maximizer, harming decisions until the search returns its best measured set. Results at t = 4 extra evaluations on the 5,828 submitted decisions:
Applying the top-scored set is the default step in MoE compression pipelines. It leaves the model worse than no edit in 17.4% of these decisions, and one extra forward pass prevents that loss. Hence our principle: the score decides what to test, and measurement decides what to keep.
Answer: Ranked Validation supplies the order that adaptive search lacks, and the best rule from four extra evaluations onward is GP-UCB started from that order. Ranked Validation is the parallel member of this family, requiring no surrogate, no hyperparameter, and one batch for all t evaluations. Each guarantee (the floor at the unedited model, the monotone budget curve, and the 2ε worst-case bound) extends to any measured search started at the additive argmax (Proposition 10). GP-UCB alone starts behind Ranked Validation and passes it at the third query, whereas starting at the additive argmax removes the slow start. Sequential greedy and evolutionary search ignore the order and trail at every budget. Any search that leaves a low-ranked set unmeasured can miss it (the second half of Theorem 1), and Theorem 2 bounds that risk: every set ahead of the optimum in the additive order lies within 2ε of it. Exact selection (%) over all 11,204 decisions (the 5,828 submitted and 5,376 recorded after the rules were frozen) at an equal budget of n + 1 + t forward passes (Tables 12 and 13 of the revised paper):
Answer: We agree, and Theorem 1 states this directly: the bound limits every rule that reads only the single-edit scan, so worst-case regret alone cannot certify safety. The additive argmax attains the bound and still harms decisions (table under W1.1). Ranked Validation adds what the bound cannot provide: a hard floor at the unedited model and a non-decreasing budget curve. The theorem also shows why a richer interaction model fitted to the scan cannot certify an unmeasured set, so a safe selector must return a measured set.
Reference
[1] A. Tsiourvas and G. Perakis. Overcoming the Optimizer's Curse: Obtaining Realistic Prescriptions from Neural Networks. ICML 2024.