- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper serves a Mixture-of-Experts (MoE) model from a fixed pool of differently quantized copies, called instances. It routes each request to one instance to maximize throughput within per-class budgets on the mean rise in token negative log-likelihood (NLL) over the 16-bit (BF16) model. From a two-expert analysis it builds FWP, an NLL that weights up tokens routed to fragile experts, read from one prefill pass. Routing is a linear program (LP) whose dual prices give a per-request score, which a theorem ties to the LP optimum. Offline, the paper reports FWP allocation within 1.5% of an oracle at a fixed budget. The live pool runs 4-bit, 8-bit, and mixed instances (W4, W8, W4/8) in vLLM on three GPUs. It reports up to 1.20 times the throughput of W8 on all three GPUs (W8×3), the only static deployment within every budget.
Critical strengths
The live results come from a real deployment measured on held-out prompts, and the paper reports its own weaknesses when serving Qwen3-30B-A3B on three A100 GPUs and tests on a held-out half of 600 prompts.
It reports that one class mix misses the medium budget by 4% and that the loss model fitted from Theorem 3.1 has negative held-out R-squared. I trust the measurements.
Critical weaknesses
-
The result depends on a choice the LP does not make. The pool, even the 24 layers of W4/8 that keep 8-bit experts, is set in advance (App. E). For one class at budgets of 0.002 and 0.005, the pool serves 2,325 and 3,340 tok/s (Table 5). W8×3 meets both budgets at 4,145 tok/s under the same load (Table 1). Choosing the pool is an upper level above the routing LP. Cascadia (ICLR 2026; arXiv:2506.04203) and StageRoute (ICLR 2026; arXiv:2506.17254), both uncited, couple deployment and routing in two levels.
-
The LP the paper analyzes is not the router that runs. The live router solves the LP once, prices capacity by backlog, runs a virtual-queue quality price and checks each class at dispatch (App. E). Theorem 3.2 assumes class-level losses and optimal prices, so it does not cover this router, though Sec. 3.3 says it does (l.296-297). App. C lacks the "linearization justification" (l.286-287).
-
The routing rule is published, with stronger guarantees, in two uncited works. PORT (NeurIPS 2025; arXiv:2509.02718) routes each query by the dual prices of a routing LP, with a competitive ratio of . MESS+ (NeurIPS 2025; arXiv:2505.19947) prices quality with a virtual queue, as the live router does, and bounds the violation. Theorem 3.2 gives neither bound.
-
Without baselines on the same pool, the comparisons cannot show where the throughput gain comes from. the central claim is that routing on fragility raises throughput within the budgets, and the evidence ties the gain neither to FWP nor to the LP
Questions for authors
-
What throughput and class means do the class-level LP split and the router without FWP reach on the pool of Table 1?
-
Does FWP beat plain prompt NLL from the same prefill in Table 4? In the model of Prop. 3.1, rare requests already lose more, so weighting them up keeps the order of plain NLL (Eq. 9). If it does by more than the split-to-split spread, and if Question 1 also favors FWP, this would largely resolve my concern.
4 comments
Response to Reviewer HJBK (Part 2)
Our objective instead concerns output-token throughput under instance service-capacity constraints and separate class-mean response-loss budgets. We also measure whether acquiring prompt information consumes enough shared serving capacity to offset its allocation benefit. This distinction does not make adding a capacity constraint, or solving an LP, novel by itself.
MESS+. Its Eq. (1) minimizes average operating cost subject to a time-average binary-satisfaction target. Algorithm 1 learns satisfaction predictions online, but updates the virtual queue with observed post-response satisfaction, not with the prediction (§2.2). Corollary 1 bounds the expected average violation by under its eventual predictor-quality and feasibility condition; the cost theorem adds sampling and learning assumptions (§3). MESS+
In our setting, response-NLL increases relative to BF16 are measured offline and are not available as feedback during live routing, so the implemented queue uses calibrated predictions. MESS+'s observed-feedback guarantee therefore does not directly transfer to this controller. A true-loss guarantee would require a separate calibration-error analysis and a matching execution policy; our correction to Theorem 3.2 is stated in W2.
Deployment setting. Real-hardware evaluation is not itself a distinction: MESS+ reports vLLM inference on two H100s, including 4-bit checkpoints, with sequential calls for energy measurement (Appendix B). Our evaluation concerns a different operating contract: concurrent requests across differently quantized instances of one MoE base model, with the measurement prefill sharing the same three serving GPUs, and with wall-clock output tokens/s, client TTFT and per-class loss measured together. The 46-run controls in Q1 test this acquisition-versus-allocation trade-off. We do not compare their rates numerically with PORT's query counts or MESS+'s energy savings.
The contribution we retain is the link between quantization-induced quality heterogeneity, prefill-observable information, offline per-instance loss calibration and constrained online allocation. Within that interface, FWP is one evaluated candidate alongside NLL and request features. We credit the pricing mechanism to existing optimization work; our case rests on the quantization-specific analysis and the measured serving trade-offs, not on generic offline training or a stronger online guarantee. Both pricing mechanisms could be applied or adapted to differently quantized instances of a common model, but their published guarantees and evaluations do not by themselves settle this information and resource setting.
W4. Static comparisons do not attribute gains to FWP or the LP
Agreed. The added controls include fixed class-LP mapping, class means, NLL, NLL + length and both FWP conventions, and the earlier equal-hardware study adds a load-aware guard-only ablation. Together, they test request information, allocation and the quality-price term separately. The results do not show that FWP is indispensable. They preserve the allocation question while turning the choice of estimator into an empirical calibration decision.
Responses to Questions for Authors
Q1. What throughput and class means do the class-level LP split and the router without FWP reach on the pool of Table 1?
The new screening runs use one W4, one W4/8 and one W8 instance on three A100-40GB GPUs. Policies that use prompt features share the same W4 prefill inside this pool; the class-LP map and class means skip it. Each row serves the 300 test prompts for two epochs (600 requests), with concurrency 128 and 500 output tokens. Predictors share 150 fitting requests and a separate 150-request calibration set; test labels are never used for routing.
Quality is the mean request-level ΔNLL on common fresh BF16 reference responses, averaged over five scoring repetitions, against strict/medium/loose budgets of 0.005/0.020/0.049 nats/token. These rows use the nominal controller budgets. The class-LP map samples the LP fractions rather than tracking exact quotas, and "FWP+NLL" uses the submitted same-position weighting.
Response to Reviewer HJBK (Part 3)
| Class mix | Policy | tok/s | Strict / medium / loose ΔNLL | All budgets met? |
|---|---|---|---|---|
| 60/20/20 | Fixed class LP | 4,206.5 | 0.004467 / 0.012007 / 0.047461 | Yes |
| 60/20/20 | Class means | 4,263.0 | 0.003321 / 0.017771 / 0.051684 | No |
| 60/20/20 | Prompt NLL | 4,166.1 | 0.003571 / 0.016775 / 0.048839 | Yes |
| 60/20/20 | FWP+NLL | 4,153.8 | 0.003321 / 0.022454 / 0.050488 | No |
| 33/33/33 | Fixed class LP | 4,502.3 | 0.004229 / 0.019023 / 0.048808 | Yes |
| 33/33/33 | Class means | 4,475.3 | 0.003773 / 0.019401 / 0.050461 | No |
| 33/33/33 | Prompt NLL | 4,327.8 | 0.004032 / 0.018695 / 0.047162 | Yes |
| 33/33/33 | FWP+NLL | 4,466.5 | 0.003912 / 0.018100 / 0.049738 | No |
| 20/20/60 | Fixed class LP | 4,774.7 | 0.004258 / 0.016666 / 0.049138 | No |
| 20/20/60 | Class means | 4,748.0 | 0.004060 / 0.015429 / 0.047332 | Yes |
| 20/20/60 | Prompt NLL | 4,513.1 | 0.003358 / 0.012553 / 0.049282 | No |
| 20/20/60 | FWP+NLL | 4,592.5 | 0.002635 / 0.015371 / 0.048732 | Yes |
| drift | Fixed class LP | 4,342.7 | 0.004190 / 0.010420 / 0.049513 | No |
| drift | Class means | 4,549.4 | 0.002665 / 0.017515 / 0.050594 | No |
| drift | Prompt NLL | 4,371.5 | 0.002438 / 0.018738 / 0.050673 | No |
| drift | FWP+NLL | 4,391.2 | 0.002221 / 0.021001 / 0.051077 | No |
We also compared NLL + length against predictive-position FWP + NLL, which aligns each weight with the input position producing the target-token prediction. With controller budgets set to 0.7 times the nominal values, both meet all fresh-label budgets in every preselected repeated condition. In two balanced 1,200-request runs, mean rates are 4,471.8 versus 4,318.1 tok/s, with mean strict/medium/loose losses of 0.002766/0.013173/0.033122 versus 0.002375/0.012894/0.034488. The strict-heavy repeat gives 4,145.7 versus 3,994.2 tok/s, again with both feasible. FWP is thus 3.43–3.65% slower, although it has lower mean run-level P95 first-token latency in the balanced repeats (1.856 versus 2.012 s).
Across the 46 runs, switching from archived to fresh labels changes nine run-level feasibility flags without changing assignments or rates; all 18 tightened-budget runs pass under fresh labels, versus 17 under archived labels. The 0.7 factor was an empirical candidate, not a certified bound, and observed compliance is distinct from universal feasibility or live global optimality. These repeats reuse the original prompts and support point comparisons, not throughput confidence intervals.
Q2. Does FWP beat plain prompt NLL from the same prefill in Table 4, by more than the split-to-split spread?
The original Table 4 does not establish that advantage. On its 88 requests, plain prompt NLL reaches the published 1.2841 modeled multiplier. The published FWP row uses a paired full-sequence excess over BF16, whereas stored prompt FWP reaches 1.2803 and still uses BF16-route weights. Across the original 20 splits, prompt FWP's mean multiplier advantage over prompt NLL is 0.00076, with a paired split SD of 0.00419 and 14 ties; because the splits overlap, they do not provide independent confirmation. We will correct the stated information scope of that comparison.
We then ran a new prediction study that measures NLL and both FWP conventions from the same prompt-only W4 pass on 600 prompts, with five repeats. It fits on all 300 calibration requests; the live study reserves half of these for safety assessment. With fresh response-loss targets and the original 300-calibration/300-test split, W4-target prediction gives:
| Predictor | Test MSE, |
|---|---|
| NLL | 0.00465902 |
| NLL + same-position FWP | 0.00421480 |
| NLL + length | 0.00258213 |
| NLL + length + same-position FWP | 0.00265621 |
| NLL + length + predictive-position FWP | 0.00259173 |
Same-position FWP + NLL improves on bare NLL by 9.53%, with a paired conditional MSE-improvement interval of approximately [0.000151, 0.000740] . Prompt length, however, is a stronger control, and neither FWP addition improves its test point estimate; calibration CV also selects NLL + length without predictive-position FWP for all three instances. Per-ID folds are preserved, but because the prompts are reused and the intervals are conditional on the fitted model, this is not an independent confirmation study.
The estimator changes; the allocation formulation does not. Replacing FWP with NLL or other features requires re-validating calibration error and acquisition cost, and earlier numerical performance does not transfer automatically. We retain fragility as the motivation and FWP as an evaluated candidate, and we remove the attribution of the throughput gain to FWP alone. The central objective remains quality-constrained allocation over quantized instances.
Response to Authors rebuttal
I thank the authors for the rebuttal, and for the additional experiments. I appreciate that the responses report the results directly, including the ones that are unfavorable to the proposed method.
Unfortunately, the new evidence confirms my concerns rather than resolving them, so I maintain my original score. FWP cannot beat plain prompt NLL in Table 4. NLL+length is still the stronger one. On the theory side, the authors agree with my review that Theorem 3.2 does not cover the deployed controller and that it offers no bound comparable to those in PORT and MESS+, to which the pricing mechanism is now credited.
I encourage the authors to resubmit the paper under its new positioning, with the claims adjusted to match what the revised results actually support.
Best,
Reviewer HJBK
Response to Reviewer HJBK (Part 1)
Thank you for a precise review, and in particular for the two concrete questions. They identify exactly the comparisons needed to separate the value of request information from the value of allocation, and we answer both directly below, including where the answer is unfavorable to FWP.
Our starting point was expert fragility, which suggested that information acquired during prefill could guide the choice of decoding instance and raise throughput under class-mean quality-loss budgets. Log-FWP was our first fragility-motivated candidate for this information. Although the deployed predictor also used NLL, the submitted routing comparisons did not use plain prompt NLL as a standalone baseline.
The new results clarify the roles of prediction and allocation in our central contribution: quality-constrained allocation of requests across a fixed pool of quantized instances, driven by calibrated predictions of quality loss. FWP is one candidate input, not a prerequisite for this allocation formulation. Below, we also separate what the calibrated optimization problem guarantees from what its online implementation achieves empirically.
Responses to Critical Weaknesses
W1. Pool configuration is an upstream decision
We agree. The submitted single-class rates of 2,325 and 3,340 tok/s at budgets of 0.002 and 0.005 nats/token fall below the 4,145 tok/s of feasible W8×3, so the fixed pool is not uniformly the best deployment. Our problem takes the resident pool as an input and does not address optimal provisioning. Cascadia and StageRoute study the complementary joint deployment-and-routing problem, and we will cite and discuss both.
The resource accounting also requires a correction. The original Table 1 used three serving GPUs plus a fourth GPU for scoring, so the 1.20× is not an equal-total-hardware result. A separate equal-hardware study, with three GPUs in total, reaches 4,644.8 versus 4,119.3 tok/s on the loose-heavy workload: a 12.75% gain over W8×3, the best tested feasible homogeneous deployment. All three budgets are met in each of its five repetitions under that study's fixed response-loss labels. This supports a workload-specific deployment benefit, not optimal provisioning, and it does not attribute the entire gain to FWP. The same-pool tests below address attribution separately.
W2. The analyzed LP and deployed router differ
We agree that this distinction is valid. Theorem 3.2 assumes exact coefficients and optimal dual prices. It does not establish optimality or finite-run feasibility for the Appendix E controller, which combines backlog costs, constant-step virtual quality queues and a dispatch guard.
For the signed score with β=0, a valid LP characterization remains: positive optimal flows use tight reduced-cost actions. Primal feasibility, dual feasibility and complementary slackness together certify an optimum, and recovering that optimum requires feasible proportions among tied actions and admission consistent with the demand constraints. An arbitrary argmax is insufficient, and we will amend the theorem statement with these recovery conditions.
On linearization: quality is exactly linear for fixed request-action loss coefficients. Reducing these coefficients to class means is exact under homogeneous losses, or when assignment is independent of within-class loss; otherwise, selection effects must be accounted for. We will replace the claimed "linearization" justification, and the stated link from the theorem to the deployed controller, with statements at this scope.
W3. Relationship to PORT and MESS+, and the scope of our contribution
Thank you for these references. We agree that dual-price routing and virtual quality queues have prior art, and that both papers provide online performance bounds that our Theorem 3.2 lacks. Having examined their formulations, proofs and experimental protocols, we separate the shared mechanism from the information and serving problem studied here.
PORT. Its Eq. (1) maximizes cumulative response performance under per-model token-cost budgets. Historical nearest-neighbor estimates and prices learned from an initial fraction of queries drive subsequent assignments (§2). The competitive guarantee of Theorem 1 requires random arrival order, controlled estimation error and a scale condition (§3; Appendix B). Its reported throughput counts queries served within budget; Appendix D sketches a traffic-aware capacity extension but leaves its integration to future work. PORT