- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JORA, a training-free method that adjusts expert aggregation coefficients after expert selection. Offline calibration estimates a layerwise Gram matrix of expert outputs. At inference, JORA uses the selected expert set and native routing weights to solve a small ridge system, without evaluating omitted experts or updating model parameters. The method supports per token expert retention during prefill and shared pool rerouting during decode. The paper establishes optimality for the fixed-set calibration objective and evaluates downstream quality, reconstruction fidelity, and single-GPU vLLM serving performance. Results show consistent quality recovery over direct dropping, with workload-dependent speedups and additional overhead from reweighting.
Critical strengths
- JORA jointly adjusts retained-expert coefficients to account for output correlations without updating model parameters. Table 3 supports the value of the full Gram matrix over common scaling, restricted sourcewise fitting, and diagonal approximations.
- With identical prefill expert selection, JORA improves all ten task scores over Drop, increasing the average from 74.51 to 78.41, compared with 79.41 for Native. Additional architectures and vision–language tasks provide further supporting evidence.
Critical weaknesses
- ExFold is the closest quality baseline, but it is absent from the serving performance experiments. It therefore remains unclear whether JORA’s quality gains justify its additional online computation at matched latency or throughput.
- Table 4 uses evaluation-corpus tuning, while Figure 2 reuses configuration-selection data. Independent diagnostics are provided, but they use different calibration settings, and their downstream fidelity tables lack matched Drop/ExFold comparisons. This limits how directly those diagnostics support the deployed configuration.
Questions for authors
- What varies across the three evaluation runs under temperature-zero decoding? Can you provide paired item-level uncertainty estimates to establish the reliability of the smaller JORA-ExFold performance differences?
- How many GPQA Diamond items were evaluated, and how were they selected? This clarification would help assess the representativeness and uncertainty of the reported scores.
4 comments
Response to Q2: GPQA-Diamond Sample Size and Selection
Thank you for requesting this clarification. We evaluated all 198 questions in the official GPQA-Diamond subset in each of the three runs. All methods used the same complete set, without additional subsampling or filtering. The phrase “preselected Diamond item IDs” referred to this fixed official set, rather than an additional subset selected for our experiments.
Response to Q1: Run Variation and Paired Uncertainty
Thank you for asking us to clarify this. The three runs used different seeds to shuffle the same questions. We kept the model, prompts, scoring rules, and decoding settings unchanged. All runs used temperature-zero decoding.
When decoding kept the native Top8 experts, we used vLLM AsyncLLM with continuous batching. Changing the request order changes the batches formed during inference; numerical differences between batches can affect greedy token choices. For shared-pool decoding, we used fixed groups of 32 requests without refilling. Different shuffles change which requests share an expert pool and can therefore affect the results. Within each run, methods used the same question order and, for shared-pool evaluation, the same group assignments.
We calculated paired uncertainty by resampling the observed question-level correctness differences between JORA and ExFold. For each run, we used 100,000 bootstrap samples, resampling within each dataset and then averaging the ten dataset scores equally. The three runs were analyzed separately.
Each cell below gives the JORA–ExFold gain in ten-task average accuracy, in percentage points, followed by its nominal 95% confidence interval.
| Configuration | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| P4 / native Top8 decode | +1.343 [0.419, 2.273] | +1.220 [0.297, 2.153] | +1.237 [0.318, 2.165] |
| P8 / Pool64 | +1.451 [0.524, 2.374] | +1.376 [0.453, 2.306] | +1.239 [0.320, 2.179] |
| P8 / Pool32 | +0.407 [-0.523, 1.337] | +0.737 [-0.197, 1.676] | +0.370 [-0.556, 1.293] |
| P4 / Pool64 | +1.027 [0.114, 1.947] | +1.199 [0.287, 2.124] | +1.019 [0.102, 1.942] |
| P4 / Pool32 | +1.119 [0.186, 2.061] | +0.754 [-0.171, 1.675] | +1.187 [0.267, 2.119] |
These intervals treat question pairs within each dataset as independent and do not adjust for dependence within inference batches or shared-pool groups.
JORA's average gain is positive in all three runs for every configuration.
Response to W2
W2: Validation under the deployed configuration
Thank you for pointing out the gap between the diagnostic and deployment settings. Table 4 and Figure 2 are exploratory analyses. We have added a comparison using the main Qwen3-30B-A3B configuration: the original Gram calibrated on WikiText-2 training data, 2,048 calibration states per layer, the original layer scales, and . The calibration and regularization remain fixed throughout this evaluation.
We evaluated 128 documents from each of WikiText and C4, with 512-token prefixes and 128-token continuations. Native, Drop, ExFold, and JORA used the same documents and fixed batches of 32, with each method maintaining its own KV cache. We measured continuation PPL, final hidden-state NMSE, and full-vocabulary .
Each cell below gives JORA's relative reduction (%) versus Drop / versus ExFold. P4/P8 denotes the number of prefill experts; Top8 is native decode, and Pool32 is the batch-shared decode pool.
| Corpus | Policy | PPL reduction | KL reduction | NMSE reduction |
|---|---|---|---|---|
| WikiText | P4 / Top8 | 0.36 / 0.35 | 7.89 / 2.83 | 5.25 / 1.91 |
| WikiText | P8 / Pool32 | 2.86 / 1.55 | 8.18 / 6.31 | 6.08 / 4.08 |
| WikiText | P4 / Pool32 | 3.57 / 1.88 | 7.20 / 4.12 | 4.39 / 3.64 |
| C4 | P4 / Top8 | 1.19 / 1.02 | 18.67 / 14.35 | 10.11 / 4.06 |
| C4 | P8 / Pool32 | 2.81 / 1.11 | 7.31 / 5.24 | 3.65 / 3.74 |
| C4 | P4 / Pool32 | 3.60 / 0.70 | 11.53 / 8.15 | 3.45 / 4.60 |
JORA reduces all three metrics in all six tested corpus-policy combinations. These results support the main-experiment configuration and add direct Drop and ExFold comparisons to the final-output diagnostics.
Response to W1: Matched Quality–Latency Comparison with ExFold
Thank you for pointing out this omission. Our original evaluation emphasized Native as the full-budget quality reference and Drop as a strong speed baseline, since Drop incurs no additional coefficient-estimation cost. Although ExFold was included in our quality comparisons, we did not include it in the serving benchmarks. We agree that this left the quality–runtime trade-off against the closest competing method insufficiently evaluated. We have therefore added matched ExFold–JORA timing comparisons.
The new timings use JORA’s post-submission optimized implementation. These engineering changes preserve the original algorithm: the Gram matrices, layer scales, rho = 1e-5, full correction strength, expert budgets, selection/mapping rules, and reconstruction objective remain unchanged. Corrections are still computed online for each current route, without retraining, recalibration, or reuse of stale coefficients.
The JORA-specific optimizations are: (i) residual-checked mixed-precision solving, retaining FP64 system assembly and one FP64 residual refinement, with FP32 factorization/triangular solves and fallback to the original strict solver; (ii) reuse of triangular factors; and (iii) warp-parallel support construction. Applicable common pool-ranking and mapping optimizations were also applied to ExFold, with the same shared inference backend and matched engine settings.
We verified lossless behavior on the tested Pool32 workloads. In these checks, the optimized and archived strict implementations produced identical token-level NLLs and argmax predictions on 24,576 cached continuation targets across WikiText-2, PTB, and C4. All 32 tested free-generation requests also produced identical generated outputs, verified by matching output hashes.
Each timing run used Qwen3-30B-A3B in BF16 on one H20-3e GPU (TP=1, PP=1). Both methods received the same 128 original MMLU-Pro/GSM8K requests, in identical order and four fixed groups of 32 without refilling. Input lengths ranged from 110 to 722 tokens. Every request generated exactly 256 output tokens, including the first token, with EOS-based early stopping disabled. Matched configurations were measured on the same physical GPU within each repeat block.
Accuracy is reused from the full ten-task evaluation in Table 1(b,c); it was not re-evaluated on the 128 timing requests. E2E denotes mean per-request latency, reported as mean ± run-to-run SD over five complete runs per configuration. P denotes retained prefill experts; Pool denotes the decode expert-pool cap. Quality. Ten-task average accuracy reused from Table 1(b,c).
Latency. New E2E measurements in seconds, reported as mean ± SD over five runs.
P denotes retained prefill experts; Pool denotes the decode expert-pool cap. Latency increases are calculated from the unrounded means.
Extra latency is calculated from the ratio of mean E2E latencies. Across these four matched-budget configurations, the reported average accuracy gains of 0.51–1.36 percentage points correspond to 1.37%–1.68% additional E2E latency relative to ExFold. Output throughput decreases by 1.34%–1.65%. These measurements quantify the runtime cost accompanying the reported quality gains under a common workload and fixed output length.