acceptodds
Under review as a conference paper at ICLR 2027

JORA: JOINT EXPERT-OUTPUT REWEIGHTING FOR TRAINING-FREE MOE ACCELERATION

Abstract

Skipping experts lowers the cost of mixture-of-experts (MoE) inference, but can substantially degrade model quality. The resulting approximation depends on the retained experts and the coefficients used to combine their outputs. We intro- duce JORA, a training-free method that jointly estimates these coefficients after expert selection to approximate the original routed mixture. Offline calibration records expert-output inner products. At inference, these statistics and the cur- rent routing weights define a small regularized system, avoiding evaluation of omitted experts. Model parameters remain unchanged. JORA supports per-token prefill retention and shared-pool decoding; our analysis establishes optimality for the fixed-set calibration objective and characterizes sensitivity to calibration mis- match. Across text and vision-language benchmarks, JORA outperforms the eval- uated expert-skipping and re-routing baselines in most task–budget comparisons, with improvements under prefill-only, decode-only, and joint reduction. In single- GPU vLLM serving, Qwen3-30B-A3B with native prefill and a 32-expert decode pool achieves a maximum observed end-to-end speedup of 1.31×over Native for 256 input and 1,024 output tokens at concurrency 64.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Large Language Models, Efficient Inference, Expert Skipping, Joint Reweighting

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_XgozOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JORA, a training-free method that adjusts expert aggregation coefficients after expert selection. Offline calibration estimates a layerwise Gram matrix of expert outputs. At inference, JORA uses the selected expert set and native routing weights to solve a small ridge system, without evaluating omitted experts or updating model parameters. The method supports per token expert retention during prefill and shared pool rerouting during decode. The paper establishes optimality for the fixed-set calibration objective and evaluates downstream quality, reconstruction fidelity, and single-GPU vLLM serving performance. Results show consistent quality recovery over direct dropping, with workload-dependent speedups and additional overhead from reweighting.

Critical strengths
  1. JORA jointly adjusts retained-expert coefficients to account for output correlations without updating model parameters. Table 3 supports the value of the full Gram matrix over common scaling, restricted sourcewise fitting, and diagonal approximations.
  2. With identical prefill expert selection, JORA improves all ten task scores over Drop, increasing the average from 74.51 to 78.41, compared with 79.41 for Native. Additional architectures and vision–language tasks provide further supporting evidence.
Critical weaknesses
  1. ExFold is the closest quality baseline, but it is absent from the serving performance experiments. It therefore remains unclear whether JORA’s quality gains justify its additional online computation at matched latency or throughput.
  2. Table 4 uses evaluation-corpus tuning, while Figure 2 reuses configuration-selection data. Independent diagnostics are provided, but they use different calibration settings, and their downstream fidelity tables lack matched Drop/ExFold comparisons. This limits how directly those diagnostics support the deployed configuration.
Questions for authors
  1. What varies across the three evaluation runs under temperature-zero decoding? Can you provide paired item-level uncertainty estimates to establish the reliability of the smaller JORA-ExFold performance differences?
  2. How many GPQA Diamond items were evaluated, and how were they selected? This clarification would help assess the representativeness and uncertainty of the reported scores.
4 comments
AuthorsOpenReview ↗

Response to W1: Matched Quality–Latency Comparison with ExFold

Thank you for pointing out this omission. Our original evaluation emphasized Native as the full-budget quality reference and Drop as a strong speed baseline, since Drop incurs no additional coefficient-estimation cost. Although ExFold was included in our quality comparisons, we did not include it in the serving benchmarks. We agree that this left the quality–runtime trade-off against the closest competing method insufficiently evaluated. We have therefore added matched ExFold–JORA timing comparisons.

The new timings use JORA’s post-submission optimized implementation. These engineering changes preserve the original algorithm: the Gram matrices, layer scales, rho = 1e-5, full correction strength, expert budgets, selection/mapping rules, and reconstruction objective remain unchanged. Corrections are still computed online for each current route, without retraining, recalibration, or reuse of stale coefficients.

The JORA-specific optimizations are: (i) residual-checked mixed-precision solving, retaining FP64 system assembly and one FP64 residual refinement, with FP32 factorization/triangular solves and fallback to the original strict solver; (ii) reuse of triangular factors; and (iii) warp-parallel support construction. Applicable common pool-ranking and mapping optimizations were also applied to ExFold, with the same shared inference backend and matched engine settings.

We verified lossless behavior on the tested Pool32 workloads. In these checks, the optimized and archived strict implementations produced identical token-level NLLs and argmax predictions on 24,576 cached continuation targets across WikiText-2, PTB, and C4. All 32 tested free-generation requests also produced identical generated outputs, verified by matching output hashes.

Each timing run used Qwen3-30B-A3B in BF16 on one H20-3e GPU (TP=1, PP=1). Both methods received the same 128 original MMLU-Pro/GSM8K requests, in identical order and four fixed groups of 32 without refilling. Input lengths ranged from 110 to 722 tokens. Every request generated exactly 256 output tokens, including the first token, with EOS-based early stopping disabled. Matched configurations were measured on the same physical GPU within each repeat block.

Accuracy is reused from the full ten-task evaluation in Table 1(b,c); it was not re-evaluated on the 128 timing requests. E2E denotes mean per-request latency, reported as mean ± run-to-run SD over five complete runs per configuration. P denotes retained prefill experts; Pool denotes the decode expert-pool cap. Quality. Ten-task average accuracy reused from Table 1(b,c).

P / PoolExFold (%)JORA (%)Gain (pp)
8 / 6478.1479.50+1.36
8 / 3276.8577.36+0.51
4 / 6476.7977.87+1.08
4 / 3275.7176.73+1.02

Latency. New E2E measurements in seconds, reported as mean ± SD over five runs.

P / PoolExFold (s)JORA (s)Increase
8 / 645.9818 ± 0.00736.0658 ± 0.01251.40%
8 / 324.2679 ± 0.00734.3262 ± 0.00891.37%
4 / 645.8370 ± 0.00615.9350 ± 0.02381.68%
4 / 324.1148 ± 0.01394.1797 ± 0.01781.58%

P denotes retained prefill experts; Pool denotes the decode expert-pool cap. Latency increases are calculated from the unrounded means.

Extra latency is calculated from the ratio of mean E2E latencies. Across these four matched-budget configurations, the reported average accuracy gains of 0.51–1.36 percentage points correspond to 1.37%–1.68% additional E2E latency relative to ExFold. Output throughput decreases by 1.34%–1.65%. These measurements quantify the runtime cost accompanying the reported quality gains under a common workload and fixed output length.

AuthorsOpenReview ↗

Response to Q2: GPQA-Diamond Sample Size and Selection

Thank you for requesting this clarification. We evaluated all 198 questions in the official GPQA-Diamond subset in each of the three runs. All methods used the same complete set, without additional subsampling or filtering. The phrase “preselected Diamond item IDs” referred to this fixed official set, rather than an additional subset selected for our experiments.

AuthorsOpenReview ↗

Response to Q1: Run Variation and Paired Uncertainty

Thank you for asking us to clarify this. The three runs used different seeds to shuffle the same questions. We kept the model, prompts, scoring rules, and decoding settings unchanged. All runs used temperature-zero decoding.

When decoding kept the native Top8 experts, we used vLLM AsyncLLM with continuous batching. Changing the request order changes the batches formed during inference; numerical differences between batches can affect greedy token choices. For shared-pool decoding, we used fixed groups of 32 requests without refilling. Different shuffles change which requests share an expert pool and can therefore affect the results. Within each run, methods used the same question order and, for shared-pool evaluation, the same group assignments.

We calculated paired uncertainty by resampling the observed question-level correctness differences between JORA and ExFold. For each run, we used 100,000 bootstrap samples, resampling within each dataset and then averaging the ten dataset scores equally. The three runs were analyzed separately.

Each cell below gives the JORA–ExFold gain in ten-task average accuracy, in percentage points, followed by its nominal 95% confidence interval.

ConfigurationRun 1Run 2Run 3
P4 / native Top8 decode+1.343 [0.419, 2.273]+1.220 [0.297, 2.153]+1.237 [0.318, 2.165]
P8 / Pool64+1.451 [0.524, 2.374]+1.376 [0.453, 2.306]+1.239 [0.320, 2.179]
P8 / Pool32+0.407 [-0.523, 1.337]+0.737 [-0.197, 1.676]+0.370 [-0.556, 1.293]
P4 / Pool64+1.027 [0.114, 1.947]+1.199 [0.287, 2.124]+1.019 [0.102, 1.942]
P4 / Pool32+1.119 [0.186, 2.061]+0.754 [-0.171, 1.675]+1.187 [0.267, 2.119]

These intervals treat question pairs within each dataset as independent and do not adjust for dependence within inference batches or shared-pool groups.

JORA's average gain is positive in all three runs for every configuration.

AuthorsOpenReview ↗

Response to W2

W2: Validation under the deployed configuration

Thank you for pointing out the gap between the diagnostic and deployment settings. Table 4 and Figure 2 are exploratory analyses. We have added a comparison using the main Qwen3-30B-A3B configuration: the original Gram calibrated on WikiText-2 training data, 2,048 calibration states per layer, the original layer scales, and ρ=10−5\rho=10^{-5}. The calibration and regularization remain fixed throughout this evaluation.

We evaluated 128 documents from each of WikiText and C4, with 512-token prefixes and 128-token continuations. Native, Drop, ExFold, and JORA used the same documents and fixed batches of 32, with each method maintaining its own KV cache. We measured continuation PPL, final hidden-state NMSE, and full-vocabulary KL(pNative∥pmethod)\mathrm{KL}(p_{\mathrm{Native}}\|p_{\mathrm{method}}).

Each cell below gives JORA's relative reduction (%) versus Drop / versus ExFold. P4/P8 denotes the number of prefill experts; Top8 is native decode, and Pool32 is the batch-shared decode pool.

CorpusPolicyPPL reductionKL reductionNMSE reduction
WikiTextP4 / Top80.36 / 0.357.89 / 2.835.25 / 1.91
WikiTextP8 / Pool322.86 / 1.558.18 / 6.316.08 / 4.08
WikiTextP4 / Pool323.57 / 1.887.20 / 4.124.39 / 3.64
C4P4 / Top81.19 / 1.0218.67 / 14.3510.11 / 4.06
C4P8 / Pool322.81 / 1.117.31 / 5.243.65 / 3.74
C4P4 / Pool323.60 / 0.7011.53 / 8.153.45 / 4.60

JORA reduces all three metrics in all six tested corpus-policy combinations. These results support the main-experiment configuration and add direct Drop and ExFold comparisons to the final-output diagnostics.

Reviewer_MEkfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

JORA is a post-selection coefficient estimator for MoE expert skipping/rerouting. Given a retained set RR, omitted set TT, native router weights ww, and Gram matrix GG from offline expert, it solves the small ridge system (GRR+ρI)δ=GRTwT(G_{RR}+\rho I)\delta=G_{RT}w_T. It thereafter changes retained coefficients from wRw_R to wR+δw_R+\delta. Omitted experts do not execute online, and model parameters remain unchanged. The same mechanism supports token-wise prefill reduction and batch-shared decode pools.

Across language and vision-language benchmarks and several MoE architectures, JORA consistently beats both naive dropping and the strongest reweighting baseline at matched budgets, recovering most of the quality lost to expert skipping.

Critical strengths

Excellent problem decomposition. Selection and aggregation are cleanly separated. Strong matched controls. Drop, common scaling, diagonal Gram, source-wise fits, Oracle least squares, and full-source ridge are all informative.

Critical weaknesses

Algorithmic novelty is incremental. The core is cached ridge regression over expert outputs. Downstream uncertainty is inadequate. Three fixed greedy runs are not meaningful model variance; paired per-example intervals are needed.

Questions for authors

Can you provide end-to-end ExFold timings and quality–latency curves using the same hardware, backend, request groups, and batch schedules?

3 comments
AuthorsOpenReview ↗

Response to W1 and Q1

Response: ExFold Comparison, Methodological Contribution, and Uncertainty

Thank you for requesting a direct ExFold comparison. We have now measured both methods under matched serving conditions.

ExFold quality–latency comparison. We used Qwen3-30B-A3B in BF16 on a single H20-3e with vLLM (TP=PP=1). Each paired comparison ran on the same physical GPU with the same backend settings and 128 MMLU-Pro/GSM8K requests. The schedule consisted of four fixed B32 groups without refilling. Inputs kept their original lengths (mean 373.37 tokens; range 110–722); outputs were fixed at 256 tokens, including the first token, with EOS ignored. Each configuration was timed five times.

Accuracy below comes from the full ten-task evaluation in Table 1. Latency is measured on the controlled workload above. The table gives four quality–latency points per method; E2E is mean ± run-to-run SD in seconds per request.

ConfigurationExFold accuracy (%)JORA accuracy (%)ExFold E2E (s)JORA E2E (s)
P8 / Pool6478.1479.505.982 ± 0.0076.066 ± 0.013
P8 / Pool3276.8577.364.268 ± 0.0074.326 ± 0.009
P4 / Pool6476.7977.875.837 ± 0.0065.935 ± 0.024
P4 / Pool3275.7176.734.115 ± 0.0144.180 ± 0.018

These timings use our post-submission optimized implementation. The changes reduce solver/refinement overhead and parallelize support construction, while preserving the Gram tables, regularization, selection policies, and numerical acceptance criteria. The optimizations were verified to be lossless in our Pool32 validation: against the archived strict implementation, per-token NLL and argmax predictions were identical on 24,576 cached-decoding targets from WikiText-2, PTB, and C4, and all 32 free-generation sequences had identical output hashes. Applicable common ranking/mapping optimizations were also applied to ExFold. JORA achieves higher ten-task accuracy at all four operating points with 1.37%–1.68% additional E2E latency.

Methodological contribution. Expert selection changes the available outputs, while native aggregation coefficients were computed for the original route. JORA addresses this mismatch by reconstructing the current routed mixture jointly over the selected execution set. The retained-output cross terms let each correction account for directions already represented by the other executed experts.

In the fixed-input, fixed-route, fixed-execution-set local diagnostic, full-Gram NMSE is 0.004112, versus 0.006143 with a diagonal Gram, a 33.1% reduction. The Oracle-LS analysis separates error recoverable within the retained span from the residual outside it. JORA puts this reconstruction into an inference procedure: fixed offline statistics support a small solve that adapts to the current routing weights and execution set, without evaluating omitted experts or updating model parameters.

Evaluation variability and paired uncertainty. The three quality runs changed the question-shuffling seed while keeping the questions, prompts, model, scoring, and temperature-zero decoding fixed. Prefill-only evaluation used AsyncLLM with continuous batching, so ordering changes batch composition and numerical execution, which can affect greedy outputs. Pool-based evaluation used fixed B32 groups without refilling; shuffling changes group membership and can change the expert pools. Methods shared the order and fixed group assignments within each run.

We used 100,000 paired-bootstrap resamples within each dataset, equally averaging the ten task-level differences and analyzing each run separately. Values below are JORA–ExFold gains in percentage points [nominal 95% CI].

ConfigurationRun 1Run 2Run 3
P4 / native Top8 decode+1.343 [0.419, 2.273]+1.220 [0.297, 2.153]+1.237 [0.318, 2.165]
P8 / Pool64+1.451 [0.524, 2.374]+1.376 [0.453, 2.306]+1.239 [0.320, 2.179]
P8 / Pool32+0.407 [-0.523, 1.337]+0.737 [-0.197, 1.676]+0.370 [-0.556, 1.293]
P4 / Pool64+1.027 [0.114, 1.947]+1.199 [0.287, 2.124]+1.019 [0.102, 1.942]
P4 / Pool32+1.119 [0.186, 2.061]+0.754 [-0.171, 1.675]+1.187 [0.267, 2.119]

The bootstrap assumes independent question pairs within each dataset and does not adjust for batch/cohort dependence. The three repetitions measure ordering/batching variability. JORA's ten-task average remains higher in every run across all five configurations.

Thank you. That mostly addressed my concerns. I would retain my positive recommendation with higher confidence.

Thank you for your thoughtful feedback and for taking the time to consider our responses. We’re glad they addressed your concerns.

Reviewer_kNQiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper notes that when routed experts are skipped, quality depends on the aggregation coefficients, not only on the retained set. JORA is a training-free post-selection reweighting scheme: offline it stores a layer-scaled Gram matrix of expert-output inner products on shared calibration states; at inference, given the native route and the selected set RR, it solves one k×kk\times k ridge system (GRR+ρI)δ=GRTwT(G_{RR}+\rho I)\delta=G_{RT}w_T and applies a=wR+δa=w_R+\delta, never evaluating omitted experts and never changing parameters. The paper proves fixed-set optimality of this solve, gives a projection decomposition of the residual (Eq. 7), and evaluates ten tasks on Qwen3-30B-A3B plus four further backbones, Qwen3-VL, four external selectors and 26 vLLM serving configurations. At half the prefill budget the ten-task average rises from 74.51 (Drop) to 78.41 (Native 79.41).

Critical strengths
  1. Clean decomposition. y−y^=FTwT−FRδy-\hat y=F_Tw_T-F_R\delta split into out-of-span and aggregation error states exactly what reweighting can and cannot fix, and it predicts the observed pattern (large prefill gains, little decode gain).

  2. Simple, cheap, plug-in. One k×kk\times k solve per token per layer at O(km+k3)O(km+k^3), a 64 KiB Gram, frozen parameters, arbitrary selectors. Prefill gains are uniform across all ten tasks (beating ExFold by 0.26–2.44 points) and reproduce on three other architectures and on vision (MMMU 72.15 vs Native 72.10 at half budget).

  3. Honest and well-diagnosed. The paper reports the sourcewise/full-ridge equivalence, the small decode gains, and that it is slower than Drop; the Oracle-LS decomposition and the KV-cache-only intervention are unusually careful.

Critical weaknesses
  1. Decode/joint gains are confounded with ExFold's pool mapping. JORA runs on the mapped set πS\pi_S while "Drop" keeps the native-route∩pool intersection, so §B.5 itself concedes the comparison measures complete policies, not coefficient rules. The same-mechanism comparison is vs ExFold, where margins are thin and sometimes negative (77.36 vs 76.85; GPQA-D 41.41 vs 45.45). A same-execution-set ablation on the ten-task panel is needed.

  2. The main baseline is a strawman. "Drop" drops the omitted mass without renormalization; cheap controls (weight-mass rescaling, calibrated common scaling) appear only in an appendix configuration (PTB PPL 18.22/17.57 vs 16.46 for JORA), so the reader cannot tell how much of the +3.9 average points is geometry rather than rescaling. They should be columns in Tables 1, 13, 14.

  3. The theory is standard ridge optimality plus an inactionable bound. Eqs. (8)–(9) are completing the square, and the mismatch bound depends on the unknown conditional Gram CC, so it cannot guide ρ\rho or predict when JORA helps. The actual open question — bias of the route-marginal Gram versus the route-conditional one — is unanalysed, and ρ\rho varies from 10−510^{-5} to 1010 with no sensitivity curve.

  4. No uncertainty quantification, and the cost/benefit is weak. Effect sizes of 0.26–2.44 points are reported without intervals (GPQA-D is quantized at 1/1981/198), and JORA is 3.16–7.05% slower than Drop — slower than Native in three D=64D{=}64 configurations — with all measurements single-GPU, i.e. not the expert-parallel regime where skipping pays off.

Questions for authors
  1. How much of Table 1(b,c) survives a JORA-vs-ExFold comparison on identical executed sets?

  2. What is the ρ\rho sensitivity of the ten-task average, and was ρ\rho set per layer, per phase, or globally?

7 comments
AuthorsOpenReview ↗

Response to W4

Thank you for highlighting the need to quantify both uncertainty and runtime cost. We have added paired uncertainty estimates, repeated timing measurements, and expert-parallel comparisons.

Paired task uncertainty. The three evaluations use different seeds for question ordering. Pool-based quality evaluation uses fixed B32 groups without refill; prefill-only evaluation uses continuous batching. Methods share the question order and, where applicable, batch groups within each run. Temperature remains zero; batching can affect numerical execution and greedy outputs.

Below are JORA minus ExFold differences in the ten-task mean accuracy (percentage points), with per-run pointwise 95% paired bootstrap intervals from 100,000 resamples. The same test items are paired across methods; the three runs are not treated as three independent test sets.

ConfigurationRun 1Run 2Run 3
P4 / D8+1.343 [0.419, 2.273]+1.220 [0.297, 2.153]+1.237 [0.318, 2.165]
P8 / Pool64+1.451 [0.524, 2.374]+1.376 [0.453, 2.306]+1.239 [0.320, 2.179]
P8 / Pool32+0.407 [-0.523, 1.337]+0.737 [-0.197, 1.676]+0.370 [-0.556, 1.293]
P4 / Pool64+1.027 [0.114, 1.947]+1.199 [0.287, 2.124]+1.019 [0.102, 1.942]
P4 / Pool32+1.119 [0.186, 2.061]+0.754 [-0.171, 1.675]+1.187 [0.267, 2.119]

The prefill-only and both Pool64 configurations have positive intervals in all three runs. GPQA-D uses all 198 items; its coarse accuracy increments are reflected in the item-level analysis.

Runtime cost and multi-GPU execution. The new timings use post-submission optimizations to support construction and coefficient computation. The Gram, rho, and selection/mapping rules are unchanged, and the original strict fallback is retained. The optimizations were lossless in our validation: NLL and argmax were unchanged on 24,576 scored positions across WikiText-2, PTB, and C4, and all 32 free-generation outputs matched the archived strict implementation. Applicable common optimizations were also applied to the baselines.

Selected matched configurations are shown below for Qwen3-30B-A3B, BF16, on H20-3e GPUs. These timing traces use continuous batching with concurrency C and 2C requests, separately from the quality protocol above. Entries are mean request E2E latency ± SD (seconds), over five independent runs. P is the per-token prefill budget; D8 is native decode and Pool32 is the batch-shared pool. Native uses Top8 in both phases.

GPUsInput/output; CPolicyNativeDropExFoldJORA
18192/32; 8P4/D84.937 ± 0.0754.233 ± 0.0384.241 ± 0.0244.260 ± 0.031
116384/32; 8P4/D812.490 ± 0.05010.965 ± 0.09110.940 ± 0.11911.069 ± 0.023
1256/1024; 64P8/Pool3227.020 ± 0.76821.980 ± 0.38422.349 ± 0.20722.715 ± 0.313
12048/256; 64P4/Pool3214.401 ± 0.03511.887 ± 0.02512.132 ± 0.03712.305 ± 0.113
2 (EP)8192/32; 8P4/D82.970 ± 0.0492.629 ± 0.0952.586 ± 0.0142.619 ± 0.012
2 (EP)2048/256; 64P4/Pool328.977 ± 0.0558.209 ± 0.2218.281 ± 0.2988.461 ± 0.034
4 (EP)8192/32; 8P4/D81.871 ± 0.0221.679 ± 0.0101.683 ± 0.0261.717 ± 0.014

Across these selected configurations, JORA is within 3.52% of Drop's mean latency and 0.44–2.18% above ExFold. Single-GPU prefill overhead versus Drop is only 0.62% and 0.95%. The two-/four-GPU prefill speedups over same-GPU-count Native are 1.134x/1.089x; two-GPU joint reduction reaches 1.061x.

Separate fixed-batch confirmations provide the following speedups over Native on one H20-3e, with 128 input and 257 output tokens, native prefill, and Pool32. Candidates were frozen after exploration; each value is the median of three confirmation-run ratios.

Fixed batchTiming metricMedian speedupThree-run range
64Whole-batch E2E, including prefill1.5195x1.5122–1.5196x
128Decode span after all first tokens, divided by 256 steps1.5954x1.5945–1.5988x

These last two metrics differ from mean request latency above. Together, the measurements quantify small additional cost and show useful single- and multi-GPU regimes; speedups remain workload- and budget-dependent.

AuthorsOpenReview ↗

Response to w2

Thank you for suggesting these inexpensive controls. We have extended the comparison to all ten tasks on Qwen3-30B-A3B, using native-ranked Top4 prefill and native Top8 decode.

Mass rescales the retained weights to restore the original total routing weight. Common selects the optimal shared multiplier under the same calibrated ridge objective as JORA, while preserving the relative weights of the retained experts. JORA instead jointly adjusts their individual coefficients. The retention rule, calibration, and task protocol are unchanged.

Scores below use the same task metrics as Table 1, expressed as percentages. Average is the unweighted mean across the ten tasks.

TaskDropMassCommonJORA
MMLU-Pro62.0763.2364.5667.04
GPQA-Diamond41.4138.3841.9250.51
CommonsenseQA76.7771.0177.8179.44
OpenBookQA84.7376.0084.4088.27
SocialIQA73.8766.5374.0574.31
GSM8K92.6290.0793.1093.23
MATH50082.3386.0086.0087.33
IFEval78.8780.2283.3683.92
HumanEval+79.8881.7182.9384.76
MBPP+72.5775.6675.4075.31
Average74.5172.8876.3578.41

Mass and Common report one additional evaluation; Drop and JORA reproduce the three-run means from Table 1.

Common improves the average over Drop by 1.84 percentage points, confirming that scalar correction recovers part of the lost quality. JORA achieves a further 2.06-point improvement over Common, with higher scores on nine of ten tasks. Excluding GPQA-Diamond, which contributes the largest difference, JORA still averages 1.33 points higher across the remaining nine tasks.

These results support the value of jointly adjusting the relative contributions of retained experts beyond a common amplitude correction. The improvement over Drop therefore cannot be explained by uniform rescaling alone.

AuthorsOpenReview ↗

Response to W3 and Q2

W3/Q2: Joint reconstruction and regularization sensitivity

Thank you for these questions. We have added experiments addressing regularization sensitivity and the reconstruction achieved by the offline Gram on fixed inputs and routes.

Contribution and theoretical analysis. Equations (8)–(9) establish guarantees for JORA's estimator. The methodological contribution is the post-selection joint reconstruction of the routed MoE mixture and its implementation without executing omitted experts online. Given an execution set, all retained coefficients depend jointly on retained-output correlations and the current weighted omitted mixture, through GRRG_{RR} and GRTwTG_{RT}w_T. Ridge regression is the computational tool for this estimator.

The importance of the joint formulation is tested directly. With inputs, routes, and executed sets fixed, Table 3 reports NMSE 0.004112 for JORA, versus 0.006143 for the diagonal-Gram variant and 0.009241 for Drop. Accounting for retained-output interactions therefore improves reconstruction even when expert selection is unchanged. The projection decomposition identifies the output component outside the retained span and the aggregation error that coefficient reweighting can reduce.

The mismatch bound characterizes the effect of using calibrated geometry on a conditional route. It does not provide an online estimate of the unknown conditional Gram or a data-free choice of ρ\rho. We address these practical questions through fixed-state reconstruction diagnostics and an offline sensitivity sweep.

Offline versus route-specific geometry. Using the main Qwen3 configuration, we examined 64 documents with 512-token prefixes and 128-token continuations. Inputs, routes, and executed sets were held fixed across coefficient rules. The following percentages measure how much of Oracle-LS's local error reduction over Drop is retained by JORA:

PhaseAll layersFinal layer
Prefill, Top477.22%86.33%
Decode, Pool3265.97%76.20%

Approximation gaps varied across layers and routing states, with larger discrepancies in early and final layers and in Pool32 decode states retaining little native routing weight. WikiText and C4 showed similar aggregate recovery. These diagnostics quantify achieved local reconstruction under the main calibration and identify where its approximation is less reliable.

How ρ\rho is set. The main benchmarks use one global scalar, ρ=10−5\rho=10^{-5}, shared by all MoE layers and both prefill and decode. It was fixed before inspecting the ten-task results. Each layer has its own Gram scale, sℓ=tr⁡(G~ℓ)/Nℓs_\ell=\operatorname{tr}(\widetilde G_\ell)/N_\ell; this normalization accounts for layerwise output energy without introducing layer-specific regularizers.

The Qwen3 ρ=0.01\rho=0.01 and OLMoE ρ=10\rho=10 settings in Appendix B belong to separate fidelity diagnostics. They were selected by continuation NLL on validation documents and frozen for the independent test partition. The range cited in the review therefore spans distinct models and experimental configurations.

We added the following Qwen3-30B-A3B sensitivity sweep. Accuracy is the unweighted ten-task average with Top4 prefill and native Top8 decode. The correction columns come from a separate fixed-state diagnostic on 128 documents, with 1,024 sampled states per phase. They report ∥δ∥2/∥a0∥2\|\delta\|_2/\|a^0\|_2 for Top4 prefill and mapped Pool32 decode.

ρ\rhoTen-task accuracy (%)Prefill correction (%)Decode correction (%)
10−510^{-5}78.4146.3430.79
10−310^{-3}78.5446.3030.74
10−110^{-1}77.8042.5926.94
1177.7532.4517.68
101076.7015.118.07

The default accuracy is the paper's three-run mean; each added setting uses one evaluation.

The two small-ρ\rho settings give similar average accuracy and correction magnitudes. Stronger regularization progressively reduces the correction, with lower average accuracy at the largest penalty. The coefficient trend follows the spectral expression δρ=U diag((λi+ρ)−1)U⊤q\delta_\rho=U\,\mathrm{diag}((\lambda_i+\rho)^{-1})U^\top q: ρ\rho controls attenuation relative to each Gram eigenvalue. The task sweep provides complementary evidence for the fixed small regularizer used in the main experiments.

Reviewer_mA2wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JORA compensates for omitted experts in MoE inference by jointly adjusting the aggregation coefficients of the experts that are executed. Offline calibration estimates a layerwise Gram matrix of expert outputs. At inference time, the current route, omitted contribution weights, and retained expert set define a small regularized linear system. The method keeps model parameters frozen and does not execute omitted experts online. The evaluation includes coefficient ablations, downstream tasks, multiple MoE checkpoints, and single-GPU vLLM serving. Its main contribution is set-conditioned joint output reconstruction using cached expert-output geometry.

Critical strengths
  1. The proposed joint correction is clearly specified and supported by controlled mechanism tests. In Table 3, the local diagnostic holds inputs, routes, and retained sets fixed: NMSE decreases from 0.009241 for direct dropping to 0.004112 for JORA, compared with 0.006143 when only the diagonal retained Gram is used. In separate task configurations, MMLU-Pro improves from 62.13% to 67.00%. These controls support the value of jointly accounting for retained-expert correlations rather than applying a common rescaling or independent sourcewise corrections. The theory appropriately limits its optimality claim to the calibrated regularized objective and identifies components outside the retained span as unrecoverable.

  2. The quality recovery is complemented by actual serving measurements that include the added cost. In Table 7, a 256-input/1,024-output trace at concurrency 64 and Pool32 reduces mean end-to-end latency from 27.942 s for Native to 21.291 s for JORA, a 1.312× speedup. JORA is approximately 4.4% slower than matched Drop in that configuration, making the compensation cost visible. Together with the task evaluation and additional checkpoints, this provides evidence of a useful accuracy–efficiency mechanism in identified operating regimes.

Critical weaknesses
  1. The uncertainty of small quality and latency differences is insufficiently reported. Table 1 averages three evaluations using temperature-zero decoding but does not report per-run variation or its source. Greedy decoding does not by itself establish that all runs are identical, but repeating it also does not automatically provide independent evidence. Each serving configuration uses one complete post-warmup trace, limiting confidence in small timing differences. These issues particularly affect close comparisons; they do not negate the larger matched-Drop gains or the controlled coefficient ablation.

  2. The generalization and deployment boundaries require careful interpretation. The offline Gram approximates output geometry encountered under changed routes and execution sets; a fixed-set optimum does not guarantee optimal downstream behavior under calibration shift. Table 4's perplexity results are explicitly tuned on the scored corpora and should remain exploratory rather than evidence of held-out generalization. Some serving configurations are slower than Native, so the deployment contribution is conditional on workload and budget. Prefill comparisons using different selectors also cannot isolate the coefficient rule; matched-selector controls should carry that attribution.

Questions for authors
  1. What varies across the three temperature-zero task evaluations? Please report per-run differences or paired per-item uncertainty for the close comparisons, and, if feasible, repeat a representative serving configuration to quantify run-to-run variability.

  2. Based on the existing calibration-sensitivity experiments and traces, when does the offline Gram become a poor approximation to the route-conditional geometry? Please clarify how the regularization/calibration choices are selected independently of the main task evaluation and which observed regimes are reliable deployment targets.

2 comments
AuthorsOpenReview ↗

Response to W1 and Q1

Response to W1 and Q1: Evaluation Variability

Thank you for raising these questions and helping us improve our paper.

Accuracy variability. The three runs used different question-shuffling seeds, with the model, prompts, scoring rules, and greedy decoding fixed. Prefill-only evaluation used continuous batching, where different schedules can affect numerical execution. Shared-pool evaluation used fixed B32 groups without refilling; shuffling changes group membership and potentially the selected pools. Within each run, methods shared the question order and fixed groups where applicable.

For each run separately, we used 100,000 paired bootstrap resamples within datasets and computed the equally weighted ten-task mean difference. Below are JORA–ExFold gains in percentage points [nominal 95% CI].

ConfigurationRun 1Run 2Run 3
P4 / native Top8 decode+1.343 [0.419, 2.273]+1.220 [0.297, 2.153]+1.237 [0.318, 2.165]
P8 / Pool64+1.451 [0.524, 2.374]+1.376 [0.453, 2.306]+1.239 [0.320, 2.179]
P8 / Pool32+0.407 [-0.523, 1.337]+0.737 [-0.197, 1.676]+0.370 [-0.556, 1.293]
P4 / Pool64+1.027 [0.114, 1.947]+1.199 [0.287, 2.124]+1.019 [0.102, 1.942]
P4 / Pool32+1.119 [0.186, 2.061]+0.754 [-0.171, 1.675]+1.187 [0.267, 2.119]

All point estimates are positive. Prefill-only and both Pool64 configurations have positive lower bounds in all three runs. These item-level intervals do not adjust for shared-pool group dependence.

Timing variability. For the representative P8/Pool32 configuration, five repeated measurements gave:

MethodE2E latency (s), mean ± run-to-run SD
ExFold4.26790 ± 0.00732
JORA4.32619 ± 0.00891

Paired measurements used the same physical H20-3e GPU, Qwen3-30B-A3B/BF16, and matched vLLM settings. Both methods processed the same 128 MMLU-Pro/GSM8K requests in four fixed B32 groups, with original input lengths (mean 373.37 tokens) and exactly 256 output tokens per request, ignoring EOS. JORA's latency was 1.37% higher than ExFold's; its run-to-run SD was 0.21% of the mean.

These timings use our post-submission implementation optimizations to support-set construction and coefficient computation. The original Gram tables, layer scales, regularization, expert selection/mapping, and numerical acceptance and fallback rules remain unchanged.

We verified that these optimizations were lossless relative to the archived JORA reference implementation in our Pool32 tests: per-token NLL and argmax were unchanged across 24,576 cached-decoding targets from WikiText-2, PTB, and C4, and all 32 tested free-generation sequences had identical output hashes. Thus, the measured improvement comes from implementation efficiency, with no additional quality loss observed in these checks. Applicable common optimizations were also enabled for ExFold.

AuthorsOpenReview ↗

Response to W2 and Q2

Thank you for helping us clarify the calibration and deployment evidence.

Our optimality result concerns the regularized reconstruction objective for a fixed execution set. The main Qwen3 experiments use a Gram calibrated on WikiText-2 training data, with rho = 1e-5 fixed before inspecting the ten-task scores. Table 4 is an exploratory corpus-tuned PPL study; the downstream task evaluations provide evidence of generalization beyond calibration.

We examined 64 documents with 512-token prefixes and 128-token continuations, holding inputs, routes, and retained sets fixed. Approximation discrepancies varied across layers and routing states, including early layers and Pool32 decode states retaining little native routing weight. JORA retained the following percentages of the local error reduction achieved by same-state Oracle-LS over Drop:

PhaseAll layersFinal layer
Prefill77.22%86.33%
Decode65.97%76.20%

WikiText and C4 showed similar aggregate recovery. These diagnostics show substantial local recovery and identify layer- and route-dependent variation within the tested setting.

To isolate the coefficient rule, Table 3 holds inputs, routes, and retained sets fixed: NMSE is 0.004112 for JORA, versus 0.009241 for Drop and 0.006143 for the diagonal-Gram variant. Comparisons involving different selectors assess complete policies.

We also added matched serving measurements using our post-submission implementation optimizations. These target support construction and coefficient computation, leaving the Gram, regularization, and selection/mapping rules unchanged. Correctness tests verified numerical consistency and unchanged token predictions and generated outputs in the tested cases.

The selected configurations below use Qwen3-30B-A3B, BF16, H20-3e GPUs, continuous batching, and 2C requests per trace, where C is client concurrency. Values are mean request E2E latency ± SD in seconds over five independent runs. P denotes retained prefill experts; D8 is native decode, and Pool32 is the shared decode pool. Native uses unrestricted Top8 in both phases.

GPUsInput/output; CReduced policyNativeDropExFoldJORA
18192/32; 8P4/D84.937 ± 0.0754.233 ± 0.0384.241 ± 0.0244.260 ± 0.031
1256/1024; 64P8/Pool3227.020 ± 0.76821.980 ± 0.38422.349 ± 0.20722.715 ± 0.313
12048/256; 64P4/Pool3214.401 ± 0.03511.887 ± 0.02512.132 ± 0.03712.305 ± 0.113
2 (EP)2048/256; 64P4/Pool328.977 ± 0.0558.209 ± 0.2218.281 ± 0.2988.461 ± 0.034
4 (EP)8192/32; 8P4/D81.871 ± 0.0221.679 ± 0.0101.683 ± 0.0261.717 ± 0.014

In these configurations, JORA's mean latency is 0.44–2.18% above ExFold and 0.62–3.52% above Drop, while remaining below Native. These comparisons quantify the modest additional runtime cost of JORA. Deployment gains remain workload- and budget-dependent; the table identifies measured examples for prefill, decode, and joint reduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.