acceptodds
Under review as a conference paper at ICLR 2027

Fragility-Aware Adaptive Routing for Quantized Mixture-of-Experts Serving

Abstract

Quantizing the experts of a Mixture-of-Experts (MoE) model is standard practice for relieving memory pressure during serving, producing a pool of pre-quantized model copies: instances, whose quality cost varies sharply across requests. Existing systems have two limitations: statically committing to a single instance for the workload and relying on ad hoc heuristic request score predictors. Closing this gap, we propose an adaptive routing framework with theoretical guarantees that assigns each incoming request to an instance so as to maximize throughput within a quality-degradation budget. Underlying the framework is a structural result: a small-noise expansion factors quality loss over experts into request-independent expert fragility and request-dependent expert affinity. This factorization yields FWP (Fragility-Weighted Perplexity), a lightweight quality estimator read from a single prefill pass on the cheapest instance and transferred to every other instance by affine calibration. Building on FWP, we cast routing as a window-level linear program and prove that the induced per-request greedy policy is KKT-consistent with the LP optimum. Empirically, on complete Qwen3-30B-A3B instances, FWP- guided allocation at a fixed quality budget comes within % of an allocation that knows each request’s true quality loss, well ahead of activation-frequency and prompt-length signals. Turning this signal into live serving, the LP-derived router over 4-bit, 4/8-bit mixed, and 8-bit instances meets every per-class quality budget at up to the measured throughput of the only static deployment that satisfies all classes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, post-training quantization, large language model serving, adaptive routing, constrained optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HJBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper serves a Mixture-of-Experts (MoE) model from a fixed pool of differently quantized copies, called instances. It routes each request to one instance to maximize throughput within per-class budgets on the mean rise in token negative log-likelihood (NLL) over the 16-bit (BF16) model. From a two-expert analysis it builds FWP, an NLL that weights up tokens routed to fragile experts, read from one prefill pass. Routing is a linear program (LP) whose dual prices give a per-request score, which a theorem ties to the LP optimum. Offline, the paper reports FWP allocation within 1.5% of an oracle at a fixed budget. The live pool runs 4-bit, 8-bit, and mixed instances (W4, W8, W4/8) in vLLM on three GPUs. It reports up to 1.20 times the throughput of W8 on all three GPUs (W8×3), the only static deployment within every budget.

Critical strengths

The live results come from a real deployment measured on held-out prompts, and the paper reports its own weaknesses when serving Qwen3-30B-A3B on three A100 GPUs and tests on a held-out half of 600 prompts.

It reports that one class mix misses the medium budget by 4% and that the loss model fitted from Theorem 3.1 has negative held-out R-squared. I trust the measurements.

Critical weaknesses
  1. The result depends on a choice the LP does not make. The pool, even the 24 layers of W4/8 that keep 8-bit experts, is set in advance (App. E). For one class at budgets of 0.002 and 0.005, the pool serves 2,325 and 3,340 tok/s (Table 5). W8×3 meets both budgets at 4,145 tok/s under the same load (Table 1). Choosing the pool is an upper level above the routing LP. Cascadia (ICLR 2026; arXiv:2506.04203) and StageRoute (ICLR 2026; arXiv:2506.17254), both uncited, couple deployment and routing in two levels.

  2. The LP the paper analyzes is not the router that runs. The live router solves the LP once, prices capacity by backlog, runs a virtual-queue quality price and checks each class at dispatch (App. E). Theorem 3.2 assumes class-level losses and optimal prices, so it does not cover this router, though Sec. 3.3 says it does (l.296-297). App. C lacks the "linearization justification" (l.286-287).

  3. The routing rule is published, with stronger guarantees, in two uncited works. PORT (NeurIPS 2025; arXiv:2509.02718) routes each query by the dual prices of a routing LP, with a competitive ratio of 1−o(1)1-o(1). MESS+ (NeurIPS 2025; arXiv:2505.19947) prices quality with a virtual queue, as the live router does, and bounds the violation. Theorem 3.2 gives neither bound.

  4. Without baselines on the same pool, the comparisons cannot show where the throughput gain comes from. the central claim is that routing on fragility raises throughput within the budgets, and the evidence ties the gain neither to FWP nor to the LP

Questions for authors
  1. What throughput and class means do the class-level LP split and the router without FWP reach on the pool of Table 1?

  2. Does FWP beat plain prompt NLL from the same prefill in Table 4? In the model of Prop. 3.1, rare requests already lose more, so weighting them up keeps the order of plain NLL (Eq. 9). If it does by more than the split-to-split spread, and if Question 1 also favors FWP, this would largely resolve my concern.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HJBK (Part 1)

Thank you for a precise review, and in particular for the two concrete questions. They identify exactly the comparisons needed to separate the value of request information from the value of allocation, and we answer both directly below, including where the answer is unfavorable to FWP.

Our starting point was expert fragility, which suggested that information acquired during prefill could guide the choice of decoding instance and raise throughput under class-mean quality-loss budgets. Log-FWP was our first fragility-motivated candidate for this information. Although the deployed predictor also used NLL, the submitted routing comparisons did not use plain prompt NLL as a standalone baseline.

The new results clarify the roles of prediction and allocation in our central contribution: quality-constrained allocation of requests across a fixed pool of quantized instances, driven by calibrated predictions of quality loss. FWP is one candidate input, not a prerequisite for this allocation formulation. Below, we also separate what the calibrated optimization problem guarantees from what its online implementation achieves empirically.

Responses to Critical Weaknesses

W1. Pool configuration is an upstream decision

We agree. The submitted single-class rates of 2,325 and 3,340 tok/s at budgets of 0.002 and 0.005 nats/token fall below the 4,145 tok/s of feasible W8×3, so the fixed pool is not uniformly the best deployment. Our problem takes the resident pool as an input and does not address optimal provisioning. Cascadia and StageRoute study the complementary joint deployment-and-routing problem, and we will cite and discuss both.

The resource accounting also requires a correction. The original Table 1 used three serving GPUs plus a fourth GPU for scoring, so the 1.20× is not an equal-total-hardware result. A separate equal-hardware study, with three GPUs in total, reaches 4,644.8 versus 4,119.3 tok/s on the loose-heavy workload: a 12.75% gain over W8×3, the best tested feasible homogeneous deployment. All three budgets are met in each of its five repetitions under that study's fixed response-loss labels. This supports a workload-specific deployment benefit, not optimal provisioning, and it does not attribute the entire gain to FWP. The same-pool tests below address attribution separately.

W2. The analyzed LP and deployed router differ

We agree that this distinction is valid. Theorem 3.2 assumes exact coefficients and optimal dual prices. It does not establish optimality or finite-run feasibility for the Appendix E controller, which combines backlog costs, constant-step virtual quality queues and a dispatch guard.

For the signed score with β=0\beta=0, a valid LP characterization remains: positive optimal flows use tight reduced-cost actions. Primal feasibility, dual feasibility and complementary slackness together certify an optimum, and recovering that optimum requires feasible proportions among tied actions and admission consistent with the demand constraints. An arbitrary argmax is insufficient, and we will amend the theorem statement with these recovery conditions.

On linearization: quality is exactly linear for fixed request-action loss coefficients. Reducing these coefficients to class means is exact under homogeneous losses, or when assignment is independent of within-class loss; otherwise, selection effects must be accounted for. We will replace the claimed "linearization" justification, and the stated link from the theorem to the deployed controller, with statements at this scope.

W3. Relationship to PORT and MESS+, and the scope of our contribution

Thank you for these references. We agree that dual-price routing and virtual quality queues have prior art, and that both papers provide online performance bounds that our Theorem 3.2 lacks. Having examined their formulations, proofs and experimental protocols, we separate the shared mechanism from the information and serving problem studied here.

PORT. Its Eq. (1) maximizes cumulative response performance under per-model token-cost budgets. Historical nearest-neighbor estimates and prices learned from an initial fraction of queries drive subsequent assignments (§2). The competitive guarantee of Theorem 1 requires random arrival order, controlled estimation error and a scale condition (§3; Appendix B). Its reported throughput counts queries served within budget; Appendix D sketches a traffic-aware capacity extension but leaves its integration to future work. PORT

AuthorsOpenReview ↗

Response to Reviewer HJBK (Part 2)

Our objective instead concerns output-token throughput under instance service-capacity constraints and separate class-mean response-loss budgets. We also measure whether acquiring prompt information consumes enough shared serving capacity to offset its allocation benefit. This distinction does not make adding a capacity constraint, or solving an LP, novel by itself.

MESS+. Its Eq. (1) minimizes average operating cost subject to a time-average binary-satisfaction target. Algorithm 1 learns satisfaction predictions online, but updates the virtual queue with observed post-response satisfaction, not with the prediction (§2.2). Corollary 1 bounds the expected average violation by O(V/T+T−1/2)O(V/T+T^{-1/2}) under its eventual predictor-quality and feasibility condition; the cost theorem adds sampling and learning assumptions (§3). MESS+

In our setting, response-NLL increases relative to BF16 are measured offline and are not available as feedback during live routing, so the implemented queue uses calibrated predictions. MESS+'s observed-feedback guarantee therefore does not directly transfer to this controller. A true-loss guarantee would require a separate calibration-error analysis and a matching execution policy; our correction to Theorem 3.2 is stated in W2.

Deployment setting. Real-hardware evaluation is not itself a distinction: MESS+ reports vLLM inference on two H100s, including 4-bit checkpoints, with sequential calls for energy measurement (Appendix B). Our evaluation concerns a different operating contract: concurrent requests across differently quantized instances of one MoE base model, with the measurement prefill sharing the same three serving GPUs, and with wall-clock output tokens/s, client TTFT and per-class loss measured together. The 46-run controls in Q1 test this acquisition-versus-allocation trade-off. We do not compare their rates numerically with PORT's query counts or MESS+'s energy savings.

The contribution we retain is the link between quantization-induced quality heterogeneity, prefill-observable information, offline per-instance loss calibration and constrained online allocation. Within that interface, FWP is one evaluated candidate alongside NLL and request features. We credit the pricing mechanism to existing optimization work; our case rests on the quantization-specific analysis and the measured serving trade-offs, not on generic offline training or a stronger online guarantee. Both pricing mechanisms could be applied or adapted to differently quantized instances of a common model, but their published guarantees and evaluations do not by themselves settle this information and resource setting.

W4. Static comparisons do not attribute gains to FWP or the LP

Agreed. The added controls include fixed class-LP mapping, class means, NLL, NLL + length and both FWP conventions, and the earlier equal-hardware study adds a load-aware guard-only ablation. Together, they test request information, allocation and the quality-price term separately. The results do not show that FWP is indispensable. They preserve the allocation question while turning the choice of estimator into an empirical calibration decision.

Responses to Questions for Authors

Q1. What throughput and class means do the class-level LP split and the router without FWP reach on the pool of Table 1?

The new screening runs use one W4, one W4/8 and one W8 instance on three A100-40GB GPUs. Policies that use prompt features share the same W4 prefill inside this pool; the class-LP map and class means skip it. Each row serves the 300 test prompts for two epochs (600 requests), with concurrency 128 and 500 output tokens. Predictors share 150 fitting requests and a separate 150-request calibration set; test labels are never used for routing.

Quality is the mean request-level ΔNLL on common fresh BF16 reference responses, averaged over five scoring repetitions, against strict/medium/loose budgets of 0.005/0.020/0.049 nats/token. These rows use the nominal controller budgets. The class-LP map samples the LP fractions rather than tracking exact quotas, and "FWP+NLL" uses the submitted same-position weighting.

AuthorsOpenReview ↗

Response to Reviewer HJBK (Part 3)

Class mixPolicytok/sStrict / medium / loose ΔNLLAll budgets met?
60/20/20Fixed class LP4,206.50.004467 / 0.012007 / 0.047461Yes
60/20/20Class means4,263.00.003321 / 0.017771 / 0.051684No
60/20/20Prompt NLL4,166.10.003571 / 0.016775 / 0.048839Yes
60/20/20FWP+NLL4,153.80.003321 / 0.022454 / 0.050488No
33/33/33Fixed class LP4,502.30.004229 / 0.019023 / 0.048808Yes
33/33/33Class means4,475.30.003773 / 0.019401 / 0.050461No
33/33/33Prompt NLL4,327.80.004032 / 0.018695 / 0.047162Yes
33/33/33FWP+NLL4,466.50.003912 / 0.018100 / 0.049738No
20/20/60Fixed class LP4,774.70.004258 / 0.016666 / 0.049138No
20/20/60Class means4,748.00.004060 / 0.015429 / 0.047332Yes
20/20/60Prompt NLL4,513.10.003358 / 0.012553 / 0.049282No
20/20/60FWP+NLL4,592.50.002635 / 0.015371 / 0.048732Yes
driftFixed class LP4,342.70.004190 / 0.010420 / 0.049513No
driftClass means4,549.40.002665 / 0.017515 / 0.050594No
driftPrompt NLL4,371.50.002438 / 0.018738 / 0.050673No
driftFWP+NLL4,391.20.002221 / 0.021001 / 0.051077No

We also compared NLL + length against predictive-position FWP + NLL, which aligns each weight with the input position producing the target-token prediction. With controller budgets set to 0.7 times the nominal values, both meet all fresh-label budgets in every preselected repeated condition. In two balanced 1,200-request runs, mean rates are 4,471.8 versus 4,318.1 tok/s, with mean strict/medium/loose losses of 0.002766/0.013173/0.033122 versus 0.002375/0.012894/0.034488. The strict-heavy repeat gives 4,145.7 versus 3,994.2 tok/s, again with both feasible. FWP is thus 3.43–3.65% slower, although it has lower mean run-level P95 first-token latency in the balanced repeats (1.856 versus 2.012 s).

Across the 46 runs, switching from archived to fresh labels changes nine run-level feasibility flags without changing assignments or rates; all 18 tightened-budget runs pass under fresh labels, versus 17 under archived labels. The 0.7 factor was an empirical candidate, not a certified bound, and observed compliance is distinct from universal feasibility or live global optimality. These repeats reuse the original prompts and support point comparisons, not throughput confidence intervals.

Q2. Does FWP beat plain prompt NLL from the same prefill in Table 4, by more than the split-to-split spread?

The original Table 4 does not establish that advantage. On its 88 requests, plain prompt NLL reaches the published 1.2841 modeled multiplier. The published FWP row uses a paired full-sequence excess over BF16, whereas stored prompt FWP reaches 1.2803 and still uses BF16-route weights. Across the original 20 splits, prompt FWP's mean multiplier advantage over prompt NLL is 0.00076, with a paired split SD of 0.00419 and 14 ties; because the splits overlap, they do not provide independent confirmation. We will correct the stated information scope of that comparison.

We then ran a new prediction study that measures NLL and both FWP conventions from the same prompt-only W4 pass on 600 prompts, with five repeats. It fits on all 300 calibration requests; the live study reserves half of these for safety assessment. With fresh response-loss targets and the original 300-calibration/300-test split, W4-target prediction gives:

PredictorTest MSE, (nats/token)2(\mathrm{nats/token})^2
NLL0.00465902
NLL + same-position FWP0.00421480
NLL + length0.00258213
NLL + length + same-position FWP0.00265621
NLL + length + predictive-position FWP0.00259173

Same-position FWP + NLL improves on bare NLL by 9.53%, with a paired conditional MSE-improvement interval of approximately [0.000151, 0.000740] (nats/token)2(\mathrm{nats/token})^2. Prompt length, however, is a stronger control, and neither FWP addition improves its test point estimate; calibration CV also selects NLL + length without predictive-position FWP for all three instances. Per-ID folds are preserved, but because the prompts are reused and the intervals are conditional on the fitted model, this is not an independent confirmation study.

The estimator changes; the allocation formulation does not. Replacing FWP with NLL or other features requires re-validating calibration error and acquisition cost, and earlier numerical performance does not transfer automatically. We retain fragility as the motivation and FWP as an evaluated candidate, and we remove the attribution of the throughput gain to FWP alone. The central objective remains quality-constrained allocation over quantized instances.

Reviewer_HJBKOpenReview ↗

Response to Authors rebuttal

I thank the authors for the rebuttal, and for the additional experiments. I appreciate that the responses report the results directly, including the ones that are unfavorable to the proposed method.

Unfortunately, the new evidence confirms my concerns rather than resolving them, so I maintain my original score. FWP cannot beat plain prompt NLL in Table 4. NLL+length is still the stronger one. On the theory side, the authors agree with my review that Theorem 3.2 does not cover the deployed controller and that it offers no bound comparable to those in PORT and MESS+, to which the pricing mechanism is now credited.

I encourage the authors to resubmit the paper under its new positioning, with the claims adjusted to match what the revised results actually support.

Best,

Reviewer HJBK

Reviewer_53FCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes adaptive routing across quantized instances of the same MoE model to improve serving throughput under per-class quality budgets. It introduces Fragility-Weighted Perplexity (FWP) to estimate request-level quantization loss and derives a routing score from a window-level linear program.

Critical strengths
  1. The paper addresses a practically relevant serving problem: exploiting request-level heterogeneity across quantized copies of the same MoE model, with explicit per-class quality budgets.
  2. The evaluation progresses from expert-level perturbations to complete-model deployment and live serving, distinguishing modeled throughput from measured results and documenting how prefill resource contention affects routing benefits.
Critical weaknesses
  1. The theoretical analysis does not directly justify the implemented FWP. The analysis assumes cross-request normalization, whereas the implementation uses intra-request normalization. Under the paper’s single-expert request model, all effective weights reduce to one, so the proved amplification does not carry over.
  2. KKT stationarity alone does not guarantee globally feasible or optimal allocations. Even with optimal dual prices, per-request argmax decisions, particularly under ties, may violate capacity or quality constraints. A clearer guarantee connecting the implemented allocation rule to primal feasibility and optimality is therefore needed.
  3. The quality constraints overlook prediction uncertainty. Both the LP formulation and admission rules rely on point estimates without safety margins, while routing may amplify underestimation biases. Consequently, predicted budget compliance does not ensure actual feasibility, and utility error bounds cannot guarantee constraint satisfaction.
  4. Experiments rely on quantized-model NLL increases on BF16-generated reference responses, without reporting task-level metrics such as accuracy or pass rates. Thus, satisfying NLL budgets does not establish preservation of downstream task performance.
  5. The main-text serving comparisons focus on static replicas at identical precision, omitting fixed-class mapping, NLL-based routing, and load-aware strategies within the same heterogeneous pool. This obscures whether the gains arise from mixed-precision deployment or the proposed FWP and LP mechanisms.
Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer 53FC (Part 1)

Thank you for the precise scrutiny of our theory and comparisons. Each of your five points identifies a real gap, and we address them in order: we correct the normalization argument (W1), restate the LP result with the conditions needed to recover a feasible optimum (W2), separate predicted from realized feasibility (W3), add generated-answer evaluation (W4), and add 46 same-pool runs with the controls you requested (W5). Throughout, the allocation framework operates on offline-calibrated quality-loss estimates; FWP is one candidate source of prefill information, not a prerequisite for the allocation formulation.

Responses to Critical Weaknesses

W1. The theoretical normalization does not match implemented FWP

We agree. In the stated single-expert-per-request model, within-request normalization makes every weight equal to one, so the implemented statistic reduces to ordinary NLL. Proposition 3.1 concerns population-normalized amplification and does not establish improved request ranking.

For token losses ℓt\ell_t and weights whose within-request mean is one, the implemented quantity satisfies the exact identity

log⁡FWP(x)=NLL(x)+Cov⁡t∣x(wt,ℓt),\log\mathrm{FWP}(x)=\mathrm{NLL}(x)+\operatorname{Cov}_{t\mid x}(w_t,\ell_t),

where the covariance is taken over the valid tokens of the same request. FWP can therefore add information beyond NLL only through this within-request covariance, and whether that contrast predicts additional quality loss is an empirical question. We will rewrite the interpretation of Proposition 3.1 on this basis.

We tested the question directly by measuring NLL and two FWP weighting conventions on the same W4 prefill: the same-position convention used in the submission, and a predictive-position convention that aligns each weight with the input position producing the target-token prediction. With fresh response-loss labels and the original 300-calibration/300-test split, same-position log-FWP plus NLL reduces W4 prediction MSE by 9.53% relative to bare NLL. However, NLL plus prompt length is a stronger baseline, and adding either FWP convention to it does not reduce test MSE. We therefore make no FWP-specific superiority claim. Replacing the estimator changes the calibration, not the form of the allocation problem's capacity and quality constraints.

W2. Stationarity alone does not ensure feasible or optimal allocation

We agree, and the gap exists even with exact losses and optimal prices. For example, two identical instances with capacity 0.5 each can tie under optimal prices; sending a demand of 1 entirely to either instance violates its capacity.

For the signed LP score without the quota-deficit term (β=0\beta=0), the valid statement is one-directional: positive optimal flows satisfy tight reduced-cost conditions. An arbitrary quota perturbation does not automatically preserve this statement. Recovering an optimal allocation additionally requires primal feasibility, dual feasibility and complementary slackness, and admission must be modeled because the LP permits unserved demand. Routing according to an optimal LP solution's assignment and admission proportions recovers its expected flows under the LP's exact-coefficient and appropriate within-class assumptions; it does not guarantee every finite realization.

We will amend the statement and interpretation of Theorem 3.2 to include these conditions. Its static LP characterization does not by itself certify the Appendix E controller, which uses backlog costs, constant-step virtual quality prices and a dispatch guard. This separation keeps the valid resource-allocation result while stating precisely what is established for the deployed procedure.

W3. Prediction uncertainty and routing-induced underestimation

We agree that a utility-error bound is not a quality-feasibility bound. A sufficient condition for a conservative LP is a simultaneous one-sided bound dga≤d^ga+egad_{ga}\le\widehat d_{ga}+e_{ga} on the relevant selected-loss coefficients. Enforcing

∑a(d^ga+ega−τg)xga≤0\sum_a(\widehat d_{ga}+e_{ga}-\tau_g)x_{ga}\le0

then implies the corresponding true-loss constraint. Bounds on conditional means give expectation-level feasibility, and deterministic bounds on selected losses give the corresponding deterministic constraint. Marginal calibration MSE establishes neither bound under adaptive selection.

Empirically, we tested tightening the controller budget to 0.7τg0.7\tau_g, fixed before the live tests. This factor failed one of the 45 approximate upper-bound checks in its calibration screen, so it is not a safety certificate. All 18 tightened-budget runs nonetheless meet the nominal point budgets under fresh labels, versus 17/18 under archived labels. Across all 46 runs, the fresh BF16 reference responses and remeasurement change nine run-level feasibility judgments without changing any route. We report both evaluations and keep observed compliance distinct from a guarantee.

AuthorsOpenReview ↗

Response to Reviewer 53FC (Part 2)

W4. NLL budgets do not establish downstream task performance

We agree, and we have added direct evaluation of generated answers on the original 75 held-out GSM8K and 75 MBPP tasks. The routed rows below use actual served outputs, with greedy decoding, natural EOS and a 500-token cap.

MethodMath correct /75MBPP public-test passes /75
BF16 reference6953
Routed, NLL + length6854
Routed, same-position FWP + NLL (submitted weighting convention)7053
Routed, predictive-position FWP + NLL6856

All paired 95% intervals include zero. For example, predictive-position FWP versus NLL + length differs by +2.67 percentage points on code, with interval [−7.57, +12.40]. This subset establishes neither superiority nor tight task-level noninferiority. MBPP's primary tests were visible in the prompts, and its additional challenge tests cover only two tasks. We report these limits and truncation explicitly rather than infer task preservation from NLL.

W5. Missing comparisons within the same heterogeneous pool

We added 46 same-pool runs on three GPUs, covering fixed class-LP mapping, class means, NLL, NLL + length and both FWP conventions. Scoring uses the pool's own W4 instance, while class-level methods skip the additional measurement prefill. All learned predictors share the same 150 fitting and 150 safety-calibration requests and the same 300 test requests. The balanced screening comparison below serves 600 requests at the nominal controller budgets; quality is the fresh mean response ΔNLL (nats/token), evaluated against budgets of 0.005/0.020/0.049.

Policytok/sStrictMediumLoose
Fixed class-LP map4,502.30.0042290.0190230.048808
Class means4,475.30.0037730.0194010.050461
NLL4,327.80.0040320.0186950.047162
NLL + length4,543.40.0036790.0174440.051636
Same-position FWP + NLL (submitted weighting convention)4,466.50.0039120.0181000.049738
Predictive-position FWP + NLL4,362.10.0040720.0214850.049318

Bold values exceed the budget. In the longer preselected runs at factor 0.7, both NLL + length and predictive-position FWP meet all fresh-label budgets, but FWP is 3.43–3.65% slower while having lower P95 client latency. These repeats reuse the original prompts and support point comparisons, not throughput confidence intervals.

Our earlier five-repeat same-pool study also included the load-aware guard-only policy you asked for: it keeps the predictor and backlog term but removes the quality-price penalty and its update. On the balanced workload it reaches 4,507.6 tok/s versus 4,479.4 for the full controller, with 4/5 versus 3/5 all-budget passes under that study's archived labels. We therefore attribute the deployment gain neither wholly to FWP nor to the quality-price term. The allocation problem remains meaningful with other calibrated estimators, and these controls now evaluate its empirical components separately.

Responses to Questions for Authors

Questions field: "See Weakness." Our responses to W1–W5 above address each item, including the mathematical conditions, prediction uncertainty, direct task measurements and same-pool controls.

Reviewer_qzyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a fragility-aware adaptive routing framework for efficient quantized Mixture-of-Experts (MoE) serving. In particular, this work introduces Fragility-Weighted Perplexity (FWP) to estimate the quality degradation of different requests based on expert fragility and request-dependent expert affinity. Based on this estimation, an LP-based routing strategy is proposed to assign requests to different quantized model instances while considering throughput and quality constraints. The authors also provide theoretical analysis to support the proposed quality estimation and routing strategy. Finally, experiments on Qwen3-30B-A3B and DeepSeek-V2-Lite are conducted, with live serving results showing up to 1.20× throughput improvement over the feasible static deployment. In general, I believe this work is interesting and addresses an important problem in efficient MoE serving. But I still have some questions for the authors for clarification.

Critical strengths
  1. The motivation is intuitive and interesting. This work considers the different impacts of expert quantization on individual requests, rather than using a fixed quantization configuration for all requests. I believe this is a meaningful direction for efficient MoE serving.

  2. The proposed framework combines expert fragility estimation with adaptive request routing. The theoretical analysis provides useful insights into the relationship between quantization sensitivity and request-dependent expert activation.

  3. The experimental results seem promising. In particular, the proposed method achieves up to 1.20× throughput improvement in live serving. Besides, the authors evaluate both quality estimation and routing performance, which makes the experimental analysis relatively comprehensive.

Critical weaknesses
  1. I have some questions about the generalizability of the theoretical analysis. The proposed fragility decomposition is mainly derived from a simplified two-expert model under small-noise assumptions, while modern MoE models contain many experts with more complicated routing behaviors. Does the theoretical conclusion still hold under aggressive quantization, such as 2-bit or 3-bit? Besides, the configuration-level surrogate achieves only 0.596 pairwise accuracy on Qwen3-30B-A3B. I think a more detailed discussion of the gap between theoretical assumptions and practical MoE serving is necessary.

  2. This paper targets efficient MoE serving, but I wonder whether the additional computation required for FWP estimation is fully considered. The proposed method requires a prefill pass on a reference instance before routing each request. Although this provides useful quality information, it also introduces additional computation and potentially increases time to first token. How much does this affect the actual end-to-end efficiency? Besides, would a simpler routing strategy using existing request features achieve a comparable performance-efficiency trade-off?

  3. Based on Table 1, the live serving results are promising, but the quality constraints are not always satisfied. For example, the medium-class quality degradation reaches 0.0208 under the strict-heavy workload, exceeding the target budget of 0.020. Does this contradict the quality guarantees claimed by the proposed routing framework? Besides, the experiments mainly measure quality through NLL degradation rather than downstream task performance. Would similar conclusions hold for reasoning, coding, and other generation tasks? I think these issues are important for practical deployment.

Questions for authors
  1. How well does the theoretical fragility decomposition generalize to modern top-k MoE architectures under aggressive low-bit quantization?

  2. What is the actual end-to-end computational and latency overhead of FWP estimation, including the additional prefill pass?

  3. Why does the proposed router violate the medium-class quality budget in Table 1, and can the quality constraints be reliably satisfied on downstream generation tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qzya (Part 1)

Thank you for the careful reading and for recognizing the value of exploiting request-level heterogeneity in quantized MoE serving. Your three concerns led us to correct the hardware accounting of Table 1, measure the prefill overhead inside an equal-hardware pool, report realized rather than predicted budget compliance, and evaluate actual routed outputs on GSM8K and MBPP. Throughout, the framework calibrates instance-specific loss predictors offline from prefill observables, then allocates decoding work across a fixed pool under class-mean loss budgets. FWP (Fragility-Weighted Perplexity) is one candidate observable; the allocation formulation does not depend on this particular estimator.

Responses to Critical Weaknesses

W1. Generalization to modern MoE architectures and aggressive quantization

Our theory is local, and the revision will state its scope explicitly. The exact decomposition in Section 3.1 holds in the two-expert sparse-MoE model. Its 2−2b2^{-2b} approximation requires centered perturbations with controlled higher moments, so it does not guarantee accuracy for arbitrary 2-bit or 3-bit multilayer configurations. Routing changes and cross-expert interactions require separate treatment.

Our diagnostics measure where this scope ends. In the sampled-expert 2-bit study, top-1 routing agreement is 95.7%, but exact top-8 set agreement is only 82.6%. For six selected expert pairs at 2 bits, the median relative additivity residual is 8.5%, with a maximum of 31.6%. Qwen's configuration-ranking accuracy of 0.596 has a bootstrap interval of [0.498, 0.679], and absolute-loss prediction has negative held-out R2R^2. The selected-pair results provide limited support for local additivity, while the routing and configuration diagnostics identify its boundaries; none establishes complete-instance low-bit validity or calibrated absolute-loss prediction.

The deployed system does not rely on the approximation being exact: per-instance predictors are fitted against measured losses, and the allocation uses those calibrated coefficients. We will make this separation between the structural analysis and the calibrated predictor explicit in Section 3.1 and Appendix B.

W2. Prefill overhead and simpler routing alternatives

We first owe a correction. Table 1 used three serving GPUs plus a fourth GPU for scoring, whereas the static fleets used three GPUs. The original 1.20×1.20\times is therefore not an equal-total-hardware gain, and the revision will say so.

A separate equal-hardware study, with three GPUs in total, gives 4,644.8 versus 4,119.3 tok/s on the loose-heavy workload: a 12.75% gain over W8×3, the best tested feasible homogeneous deployment. All budgets are met in each of its five repetitions under that study's labels. This establishes a deployment benefit for that workload; it does not show that FWP causes the whole gain.

To measure the overhead directly, the new experiments score on the pool's own W4 instance, so acquisition consumes the same three GPUs that serve requests. They reuse 300 test prompts disjoint from the 150 fitting and 150 safety-calibration prompts. Across the 600-request screening runs, the median scoring delay is 0.216–0.425 s, and the client P95 time to first token (TTFT), including scoring, is 2.291–2.613 s. The 95th percentile of scoring's per-request share of end-to-end latency is 12.18–15.20%, so the overhead is not negligible. Client timing starts when a request obtains an in-flight slot and excludes the offline calibration and LP setup.

As the reviewer anticipated, simpler features are competitive. The table below compares two preselected 1,200-request runs per method on the balanced workload, with controller targets tightened to 0.7τg0.7\tau_g and evaluation budgets left at τg\tau_g. Both methods meet all budgets under both fresh and archived labels. Predictive-position FWP aligns each weight with the input position that produces the corresponding target-token prediction.

Prefill predictorThroughput (tok/s)Median client TTFT (s)P95 client TTFT (s)
NLL + prompt length4,471.80.3582.012
Predictive-position FWP + NLL4,318.10.4331.856

Values are averages of per-run rates and per-run percentiles. FWP has lower P95 latency but lower throughput, and the framework can adopt the simpler predictor without changing its objective. Same-pool class-mean and fixed-class-LP controls, which avoid scoring altogether, are reported in our responses to Reviewers 53FC and HJBK. With reused prompts and few repetitions, these runs do not establish a universal ranking of methods.

AuthorsOpenReview ↗

Response to Reviewer qzya (Part 2)

W3. Budget violations and downstream generation quality

The reviewer is right. The reported 0.0208 exceeds the medium-class budget of 0.020 nats/token by 4%, and this contradicts an unconditional reading of our quality guarantee. The LP's feasibility statement is conditional on its loss coefficients; the submitted online controller acts on predicted losses, so it does not inherit feasibility for realized losses. Relatedly, logs recovered from our later equal-hardware campaign show that no tested quality-price step size met every calibration budget there, and the controller used a fallback value; the tuning step did not certify compliance either. The violation alone does not identify a single cause.

We then tested a controller budget factor of 0.7, frozen before the live runs. All 18 runs at this factor meet the nominal budgets under fresh reference-response labels, versus 17/18 under archived labels. The factor failed one of the 45 approximate upper-bound checks in its calibration screen, so we treat it as an empirical buffer, not a guarantee. The fresh labels score a newly generated set of BF16 reference responses on every instance, averaged over five measurements. Relabeling changes nine of 46 run-level feasibility flags without changing any route or throughput. Because the reference text itself also changed, these differences cannot be attributed to a single source, and we report both label regimes.

We also evaluated actual routed outputs on 75 held-out GSM8K and 75 MBPP tasks, with greedy decoding, natural EOS and a 500-token cap:

MethodMath correct /75MBPP public-test passes /75
BF16 reference6953
Routed, NLL + length6854
Routed, same-position FWP + NLL (submitted weighting convention)7053
Routed, predictive-position FWP + NLL6856

All paired 95% intervals include zero. For predictive-position FWP versus NLL + length, the differences are 0.00 percentage points on math [−8.19, +8.19] and +2.67 points on code [−7.57, +12.40]. Samples of this size cannot establish 1% noninferiority. Two further caveats apply: MBPP's primary assertions were visible in the prompts, and truncation at the 500-token cap is material for code. These task results complement the NLL evaluation; they do not turn the NLL budget into a correctness guarantee.

Responses to Questions for Authors

Q1. How well does the theoretical fragility decomposition generalize to modern top-k MoE architectures under aggressive low-bit quantization?

The current proof establishes the stated local two-expert result, not a top-k or low-bit guarantee. W1 reports the measured routing changes, interaction residuals and weak Qwen configuration ranking that limit broader extrapolation. The deployed allocation uses empirically calibrated losses and does not assume that the approximation is exact.

Q2. What is the actual end-to-end computational and latency overhead of FWP estimation, including the additional prefill pass?

Measured inside the serving pool, the scoring prefill has a median delay of 0.216–0.425 s, and the 95th percentile of its per-request share of end-to-end latency is 12.18–15.20%. W2 reports client TTFT including scoring, together with the matched throughput and latency comparison. The fourth-GPU scorer of the submission is excluded from this equal-hardware setting.

Q3. Why does the proposed router violate the medium-class quality budget in Table 1, and can the quality constraints be reliably satisfied on downstream generation tasks?

The violation exposes the gap between predicted and realized loss constraints in the submitted controller. Tightening the controller budget improved measured compliance in the tested settings, but it does not prove future feasibility. Direct task evaluation is now available (W3); its uncertainty is too wide to support a general downstream-preservation guarantee.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.