acceptodds
Under review as a conference paper at ICLR 2027

Beyond Independent Top-: Pairwise Energy Routing for Frozen Mixture-of-Experts

Abstract

Sparse mixture-of-experts (MoE) models activate only of experts per token, yet conventional Top- routing ranks experts individually without explicitly accounting for interactions within the selected subset. The highest-scoring experts therefore need not form the most effective subset. We propose PairQ, a parameter-efficient method for adapting pretrained MoEs through token-conditioned pairwise routing. PairQ combines frozen native-router scores with learned interactions between expert-selection variables, assigning a quadratic energy to each subset. Only the added routing parameters are trained; the backbone, experts, native router, and expert-mixing rule remain unchanged. An energy-weighted surrogate over candidate subsets propagates language-model loss gradients while preserving hard selection in the forward pass. The selection objective admits a quadratic unconstrained binary optimization (QUBO) formulation. Evaluations on Qwen3-30B-A3B and GPT-OSS-20B show gains over native routing across five mathematical reasoning benchmarks. A complete autoregressive inference run demonstrates that a physical coherent Ising machine (CIM) can decode the routing QUBO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Mixture-of-Experts, Pairwise Expert Routing, Parameter-Efficient Fine-Tuning, Quadratic Unconstrained Binary Optimization (QUBO), Coherent Ising Machines

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o1ZBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PairQ, a method for adapting frozen mixture-of-experts language models by learning token-conditioned pairwise costs over expert-selection variables. These costs supplement the frozen router’s unary scores, producing a quadratic objective over subsets containing exactly K experts. Selection uses simulated annealing, while an energy-weighted candidate surrogate supplies gradients through hard routing. Only the added routing parameters at the final MoE layer are trained, while the pretrained model parameters and the expert-mixing rule remain fixed. Experiments evaluate two backbones on five mathematical reasoning benchmarks, with additional unary-control, surrogate, and solver analyses. A physical coherent Ising machine also supplies routing decisions throughout one autoregressive generation.

Critical strengths
  1. A coherent formulation that isolates routing adaptation. PairQ directly scores expert subsets while preserving the pretrained experts, activation cardinality, and native mixing rule. This is a useful experimental setting for studying whether richer selection objectives improve frozen MoEs. The paper also includes evaluations on both 32-expert and 128-expert backbones.
  2. Useful controlled evidence for the training surrogate. The aligned-versus-cyclic experiment matches initialization, data order, and training settings across three paired seeds. Correctly matching candidate energies with their outputs consistently improves held-out hard-routing NLL over the cyclic control. This supports the presence of meaningful training signal in the surrogate. The paper also clearly acknowledges estimator bias, limited candidate coverage, and the distinction between gradient agreement and successful discrete optimization.
Critical weaknesses
  1. The default decoder uses 64 annealing reads and 1000 sweeps per routing decision, yet no end-to-end latency, throughput, or search-budget comparison is reported. With relatively small accuracy gains, these measurements will affect the method’s practical significance.
  2. The training forward pass uses SA routes, so the learned parameters are fitted to SA’s behavior rather than to the energy’s minimizer. Without downstream results under exact decoding (feasible for GPT-OSS), it is hard to tell whether any benefit comes from the energy or from SA’s particular approximation. Fidelity on Qwen3 is not measured.
  3. The frozen-router AIME scores are below the developers’ figures even at their lowest reasoning settings. The gpt-oss model card cited in the paper lists gpt-oss-20b at 42.1 / 37.1 (AIME24 / AIME25, no tools) at low effort and 80.0 / 72.1 at medium, versus 36.67 / 36.67 here. Qwen3-30B-A3B’s AIME25 is reported at about 71 with thinking and about 22 without, versus 16.70 here. In that regime, small accuracy differences can come from changes in output length or termination rather than reasoning quality.
  4. Native-router CPT falls below the frozen router on three of five benchmarks for each backbone. This suggests that MathPile CLM training is itself poorly matched to chat-format evaluation. Beating CPT is therefore weak evidence, and an anchored, bounded router may mostly be doing less harm.
Questions for authors
  1. What fraction of generations hit the 4096-token cap for each method and benchmark, and how do mean output lengths compare between methods? Do the Table 1 conclusions hold under the developers’ recommended generation budgets?
  2. On GPT-OSS, how do accuracy and NLL change when the trained router is decoded by exact enumeration instead of SA? If accuracy drops, that would suggest the gains come from SA’s behavior rather than from minimizing the learned energy. GPT-OSS has only C(32,4) = 35960 feasible subsets. Yet SA spends 64 reads × 1000 sweeps × 32 single-spin proposals ≈ 2.05M proposals per token, about 57 times the number of feasible subsets. Exhaustive enumeration would be exact and plausibly cheaper.
  3. How often does PairQ select exactly the native route? What are the distributions of per-token overlap with it, and of the native-logit ranks of swapped-in experts? Mixing still uses the restricted native softmax, so the lower entropy and higher top-1 weight in Table 2 could simply reflect low-logit experts entering the set with negligible weight. What fraction of selected experts receive less than 1% of the mixing weight?
  4. What is the frozen-native NLL on the GPT-OSS held-out set of Table 3?
  5. What are inference latency and throughput relative to native Top-K, for SA and for the CIM (including communication)?
  6. For the 214 CIM decisions, how often does the CIM route match the exact minimizer and the SA route, and does the frozen router also answer that question correctly?
  7. Is there any plan to include results for an intermediate layer, or for several layers, which would show whether the effect is specific to the final MoE layer?
1 comment

o1ZB-W1: Search cost

At matched output lengths, simulated annealing (SA) took 1.2419–1.3011 times native generation time on GPT and 1.685–1.720 times on Qwen. Separately, increasing SA reads from 64 to 512 at a fixed Qwen checkpoint lowered NLL from 1.606364 to 1.602179. Each read is an annealing run; all use 1,000 sweeps in this comparison. See XUmm-6 for timing and XUmm-3 for search budgets.

o1ZB-W2: Decoder dependence

Exact GPT decoding also improves over frozen native on AIME24 and held-out NLL. Qwen benefits from more SA reads at a fixed checkpoint (XUmm-3).

o1ZB-W3: Reasoning settings

Methods use matched prompts, greedy decoding and output budgets. Qwen uses enable_thinking=False, closing an empty thinking block before answer generation; the GPT length control changes only the cap, retaining its prompt and decoding configuration (XUmm-2, XUmm-6).

o1ZB-A: Native-router training

Frozen native is the pretrained baseline. Native-gate continued pretraining (CPT) is a separate adaptation control with frozen backbone and experts. PairQ's gains over frozen native are reported independently of CPT (XUmm-B, XUmm-3).

o1ZB-1: Output budget

Changing only the GPT AIME24 cap from 4,096 to 8,192 raises SA correct answers from 11/30 to 18/30 and reduces cap hits from 16/30 to 8/30. At 8,192, SA/exact mean outputs are 4,620.43/5,003.10 new tokens (XUmm-6).

o1ZB-2: Exact decoding

With matched Harmony prompts, greedy decoding and an 8,192-token cap, frozen native scores 16/30 on AIME24; the same causal-language-modeling (CLM) GPT router scores 18/30 under SA and 17/30 under exact decoding. Their respective NLLs on 747,305 held-out targets are 1.874181/1.864217/1.861681. Exact gives lower NLL, while SA answers one more question correctly (XUmm-3). GPT selects four of 32 experts: exact checks all 35,960 legal subsets, while SA uses 64 reads and 1,000 sweeps.

o1ZB-3: Swapped-in experts

Three Qwen checkpoints (seeds 123, 456, 789) each use 256 documents: 65,280 token positions and 522,240 selected-expert instances (eight per position). PairQ matches the native set at 74.380%–76.898% of positions, with mean overlap 6.9113–7.0062 experts. Swapped-in experts (outside native Top-8) have native-rank medians 25–29 and 90th percentiles 62–77. Their bfloat16 (BF16) mixing weights are below 1% in 4.592%–6.937% of swapped-in instances; the fraction among all selected instances is 0.571%–0.901%. With the same logits and BF16 conversion, only 1 out of 522,240 native-reference weights per checkpoint falls below 1%.

o1ZB-4: Frozen-native validation loss

The frozen-native NLL on the original Table 3 evaluation set is 1.613915. This result was computed using the unmodified pretrained base on the same cached text: the first 256 tokens of each of 128 documents, giving 32,640 next-token targets. The corresponding NLLs for PairQ-SA and PairQ-exact are 1.621868 and 1.615968, respectively.

These results belong to the original Table 3 checkpoint and text cache and must be kept separate from the subsequent CLM experiment. The latter uses 128 complete documents and 747,305 targets, with native/SA/exact NLLs of 1.874181/1.864217/1.861681.

o1ZB-5: Throughput

Original/reversed method order, in tokens/s:

  • GPT-OSS-20B, AIME24: SA 20.939/20.590; native 26.005/26.789.
  • Qwen, GSM8K: SA 13.500/13.203; native 22.748/22.709.

Each model uses eight fixed prompts, batch size 1, greedy decoding and 128 new tokens; methods share a GPU within each comparison. After warm-up, each prompt is timed twice and method order is reversed. Timing includes prompt processing, routing, experts and decoding, excluding model loading and tokenization (XUmm-6).

We do not yet have a communication-inclusive CIM latency measurement.

o1ZB-6: Physical-solver comparison

The reported GPT-OSS-20B GSM8K CIM run completes 214 feasible four-expert selections and answers “3 bolts” correctly (Vcjk-5). Matched CIM/SA/exact agreement and the frozen-native answer remain unverified. That comparison requires each submitted QUBO with its returned mask, paired SA/exact solves, and native generation under the same prompt and decoding settings.

o1ZB-7: Adaptation depth

Controlled runs adapt GPT layer 23 or Qwen layer 47 (zero-based), freezing all pretrained parameters. This permits direct output-replacement loss measurement without later attention layers. The construction applies to earlier or multiple MoE layers; current controlled evidence concerns the final layer (XUmm-5).

Reviewer_VcjkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

PairQ adapts frozen mixture-of-experts language models by combining native router scores with learned, token-conditioned pairwise energies over expert subsets. It uses simulated annealing for fixed-cardinality selection and an energy-weighted straight-through surrogate for training. The paper evaluates final-layer routing adaptation on two backbones and five mathematical benchmarks, provides routing and surrogate diagnostics, and demonstrates coherent Ising machine routing for one complete autoregressive generation.

Critical strengths
  • The paper investigates a reasonable and potentially useful idea: scoring expert subsets jointly while preserving pretrained model weights and the number of active experts.
  • The supplementary controls and explicit discussion of limitations are useful, particularly the parameter-matched unary comparison, identifiability analysis, and distinction between surrogate gradients and differentiation through the discrete solver.
Critical weaknesses
  • My main concern for this paper is writing. I am generally in favor of using LLMs to improve the quality of papers and research. But it is impossible to understand this paper as a human without using an LLM; for instance, “This efficient rule does ..." in the introduction is plausible motivation, but “complementarity,” “redundancy,” and “most effective subset” are not discussed or referenced, so you need to check the experiments and method to understand this section, and you can find similar examples throughout the paper (“expert complementarity,” “quadratic energy over subsets,” and “energy-weighted surrogate” appear before adequate intuitive explanation); essentially, you need to have the whole paper as context to understand/read part of the introduction!
  • The authors have not shown that pairwise interactions provide additional value. The only experiment in Appendix G.1 correctly explains that, under fixed cardinality, some off-diagonal interactions are equivalent to unary corrections. Therefore, zeroing the diagonal does not establish that PairQ benefits from irreducible pairwise structure. The parameter-matched control in Appendix E is important but inconclusive: PairQ improves NLL over the unary model by approximately 0.0026, while the reported bootstrap interval for the difference includes zero. I think this is the central claim of the paper and should be in the main text.
  • The selection objective is not motivated; The only experiments are the surrogate experiments, but the aligned-versus-cyclic comparison establishes that correct energy–output correspondence is preferable to a deliberately mismatched backward signal. It does not establish an advantage over a reasonable alternative estimator. Moreover, the 88.66% ordering agreement in Table 4 concerns a gradient-based local approximation, rather than the learned energy ranking. Appendix F.5 reports that QUBO energy agrees with actual candidate-loss ordering in only 56.69% of pairs, and that the SA route has a mean excess loss of 0.1612 nats relative to the best of six candidates.
  • The main results section does not show great improvement over the simple baseline. For GPT-OSS, PairQ improves three benchmarks but degrades AIME25 and MathQA relative to frozen routing. For Qwen3, the improvements on AIME24, AIME25, AMC23, and MathQA each correspond to approximately one additional correct answer. The main results do not report variability across training or solver seeds.
  • Some stronger baselines are needed to showcase the novelty and contribution of the paper. The QUBO formulation, cardinality penalty, simulated annealing, and straight-through estimation build on established techniques, making their integration into frozen MoE adaptation the principal methodological contribution. Although the related work discusses interaction-aware routing, router-only adaptation, and alternative approaches to discrete routing, the experiments mainly compare PairQ with frozen native routing and continued training of the native router, supplemented by a limited parameter-matched unary control. Relevant methods for adapting expert selection at inference time are also not evaluated. I understand these methods have different techniques in the training, but the authors need to show comparisons to clearly identify and control those differences where possible and help to understand the contribution of the method.
Questions for authors
  • Can you provide a concrete example demonstrating expert complementarity or redundancy that cannot be addressed adequately by correcting individual expert scores?

  • What specific findings from the cited counterfactual and co-activation studies support pairwise subset scoring, rather than routing adaptation more generally?

  • Can you explain the meaning of routing energy, its relationship to language-model loss, and the rationale for Equations 5–7 before introducing the full construction?

  • Can you add a training example showing how candidate outputs affect energy updates and subsequent expert selection?

  • What role does the CIM demonstration play in the central contribution, and which practical routing limitation is it intended to address?

1 comment

Vcjk-W1: Why score expert combinations?

PairQ keeps pretrained individual scores and learns token-dependent scores for expert pairs. The backbone, experts, and native mixing rule stay fixed. Forward routing uses a hard subset; a six-candidate mixture supplies the approximate backward signal without differentiating simulated annealing (SA). XUmm-W1 gives a partner-dependent example.

Vcjk-W2: Pairwise expressiveness

At a fixed input, independent scores cannot reverse a preference between two experts when their partners change. PairQ can: XUmm-W1 gives the example, and XUmm-B the parameter-matched Unary training control.

Vcjk-W3: Alternative training rules

Qwen hard-SA NLL is 1.6851 for normal straight-through estimation (STE), 1.7197 for Soft-forward, and 2.0106 for cyclic pairing on the same 32,640 targets. Initialization, data order, and budget match. Soft-forward trains directly on the candidate mixture.

The GPT diagnostic's 88.66% measures correctly ordered candidate pairs under an output-gradient loss prediction, not energy ranking. Duplicate sets are merged by averaging scores, including the hard-route group; pairs with predicted or measured differences within 10⁻⁶ are excluded. Ratios are computed per document, then averaged over documents and three aligned checkpoints. XUmm-1 defines the separate Qwen test.

Vcjk-W4: Training seeds

Across three paired Qwen runs, PairQ's five-benchmark average exceeds Unary by 0.60 percentage points. XUmm-B gives the means and matched training setup; the SA search seed is fixed across matched evaluations.

Vcjk-W5: Contribution and baselines

The contribution is token-dependent subset scoring and its learning interface for frozen MoEs; QUBO, the cardinality penalty, SA, and STE are established ingredients. XUmm-W1 and XUmm-B address expressiveness and matched capacity; Vcjk-W3 compares training rules.

An external-router comparison remains missing. A matched test requires the same frozen backbone, active-expert budget, training data, generation settings, and loss/accuracy evaluation.

Vcjk-1: A concrete interaction

At one fixed GPT input, expert 3 gives lower loss alongside {8,16,28}, while expert 13 is better alongside {4,16,28}. PairQ's total energy follows both preferences. XUmm-W1 provides all four subsets, energies, and losses.

Vcjk-2: Findings from prior work

Yoon et al. (2026) find lower-loss, equal-compute alternative subsets for the same token in a frozen model. Tang et al. (2025) identify input-dependent expert collaboration. Nguyen-Nhat et al. (2025) model expert relationships with a graph for routing. These motivate studying combinations; our matched Unary control and partner-reversal example test PairQ's added structure.

Vcjk-3: What routing energy means

Energy scores a whole subset; lower is preferred. Next-token cross-entropy trains the router. Equations 5–7 combine the token query and expert embeddings into pair scores; normalization sets their scale and the gain controls their strength. Zero initial gain recovers the native unary objective. Training uses executed/native/four random candidates: hard output forward, energy-weighted output mixture backward.

Vcjk-4: Backward pairing and route changes

Three GPT model pairs share initialization, data order, budget, and SA settings. Aligned pairing matches each energy-derived weight to its own output; cyclic pairing rotates weights 1–6 onto outputs (6,1,2,3,4,5). Forward routing uses each model's current hard SA route. Aligned NLL is lower in all three seeds, averaging 1.6191 versus 1.7762 on 32,640 targets.

In a separate Qwen parameter-perturbation diagnostic with solver randomness fixed, a small STE descent perturbation replaces expert 65 with 60 at the first changed position in the prespecified inspection order, keeping seven others; same-target loss falls from 9.3004 to 7.1598. This is a perturbation diagnostic, not a recorded optimizer step.

Vcjk-5: Why connect a physical solver?

A physical coherent Ising machine (CIM) supplied all 214 feasible four-expert routes in a complete GPT-OSS-20B GSM8K generation, without SA fallback or cardinality repair. The answer, “3 bolts,” is correct. Each returned route drives subsequent generation.

This establishes the hardware interface for exploring larger expert pools. At the current tractable GPT scale, exact enumeration supplies an energy-minimum reference for SA.

Reviewer_XUmmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PairQ, a way to improve expert selection in pretrained mixture-of-experts (MoE) models without changing the experts. Standard Top-K routing scores each expert on its own and picks the K highest, so it never considers whether the chosen experts complement each other or are redundant. PairQ keeps the frozen native router scores and adds learned, token-dependent pairwise interaction terms between experts, which gives a quadratic energy over subsets. Picking K experts then becomes a QUBO problem, solved with GPU simulated annealing (SA) or with a physical coherent Ising machine (CIM). Because subset selection is discrete, training uses the hard SA route in the forward pass and a straight-through surrogate in the backward pass that compares six candidate subsets (SA, native, four random). Only the new routing parameters at the final MoE layer are trained, on 10M MathPile tokens. On Qwen3-30B-A3B, PairQ gives small gains on all five math benchmarks; on GPT-OSS-20B, results are mixed.

Critical strengths
  • The question is clear and the formulation is clean. Anchoring the energy to the frozen native router is a nice design choice, and the method is parameter-efficient since experts and backbone stay frozen.
  • The paper includes useful diagnostics beyond accuracy: exact enumeration against SA on GPT-OSS, an ablation of the surrogate pairing, and a parameter-matched unary control.
Critical weaknesses
  • The central claim, that pairwise interactions improve routing, is not well supported. The parameter-matched unary router nearly matches PairQ (the NLL bootstrap interval includes zero) and is better on AIME25. App. G.1 itself shows that part of the pairwise term reduces to unary corrections under fixed K. So the gains may come from a larger learned router rather than from the pairwise structure.
  • The gains are small and statistically weak. Main results are single-seed with no error bars, each AIME question is worth 3.33 points, and GSM8K/MathQA changes are within noise. On GPT-OSS the picture is mixed (AIME24 up, AIME25 and MathQA down).
Questions for authors
  • The learned energy does not clearly track loss. In App. F.5, ranking candidates by energy agrees with their measured loss ordering only 56.7% of the time. Can you explain this?
  • Baseline accuracies are far below the models’ reported numbers (e.g., GPT-OSS-20B on AIME). It is unclear whether the gains survive at full reasoning strength. Also, native-router CPT falls below the frozen router, which raises a tuning question. Any explanation, please?
  • On GPT-OSS, exact enumeration is feasible. Does routing with the exact optimum improve accuracy over SA? That would show whether better solving matters at all.
  • Can you report multiple seeds or confidence intervals for Table 1?
  • Why only the final MoE layer? What happens when more layers are adapted?
  • Do the gains hold with longer generation budgets or full reasoning modes? And what is the wall-clock cost of SA routing compared with native Top-K?
  • Why only six candidate routes (SA, native, and four random) in the surrogate? What is the intuition behind that? Since random subsets might be poor, do the four random routes ever receive meaningful softmax weight or contribute a useful gradient signal? How sensitive are the results to M?
1 comment

XUmm-W1: Expert interactions

At one fixed GPT-OSS-20B token state, replacing partner 8 by 4 reverses the preference between experts 3 and 13. For the same QUBO (quadratic unconstrained binary optimization) matrix Q, E(S)=Σ_{i,j∈S}Q[i,j]; NLL is next-token loss in nats:

  • {3,8,16,28}: E=-8.230602, NLL=6.441942; {8,13,16,28}: E=-7.715112, NLL=6.487131.
  • {3,4,16,28}: E=-3.007604, NLL=7.117835; {4,13,16,28}: E=-3.031587, NLL=7.086349.

Native/PairQ select {8,13,16,28}/{3,8,16,28}. Independent scores cannot reverse their 3-vs-13 preference.

XUmm-B: Paired seeds

Three paired Qwen seeds (123,456,789), matching initialization, data order, budget and parameter count, give PairQ/Unary five-task means of 55.99%/55.39% (+0.60 points); each AIME mean improves by 2.22 points. Unary learns independent expert scores. Pretrained weights are frozen; solver seeds match. The main Qwen checkpoint beats frozen native and native-gate continued pretraining (CPT) on all five tasks.

XUmm-1: Energy versus local loss prediction

The appendix's GPT result measures energy–loss ordering. Energy is learned via next-token loss without direct candidate-loss/ranking supervision. Our separate Qwen test checks local output-gradient predictions used by the straight-through estimator (STE).

At three fixed checkpoints (seeds 123,456,789; step 2,442), we inspect 2,000 positions each from 251 MathPile documents. Six entries contain the executed route, native Top-8 and four random eight-expert subsets; input and upstream computations stay fixed. Predicted reduction is minus the dot product of the loss gradient at the executed output and the candidate's output change. Measured reduction is executed NLL minus replacement NLL: replace only the final MoE output, retain the residual, and recompute normalization, LM head and loss for the same true next token.

After merging duplicates and excluding the hard route, positions need at least two alternatives. Mean valid per-position Spearman is 0.7507/0.7464/0.7449. Predicted and measured loss directions agree for 75.47% of eligible replacements; candidate ordering agrees for 83.96% of eligible unordered alternative pairs. Comparisons require predicted and measured magnitudes >10^-10 (for pairs, their differences). These accuracies pool comparisons per checkpoint, then average across checkpoints.

XUmm-2: Evaluation protocol

The main table uses 4,096 tokens, batch 1, greedy decoding and matched prompts/parsers. Qwen uses enable_thinking=False. GPT AIME24 follow-ups use 8,192 tokens with other decoding settings fixed. CPT adapts the native gate (XUmm-B).

XUmm-3: Exact decoding

Frozen native/SA/exact score 16/30, 18/30 and 17/30 on AIME24 (matched Harmony prompts, greedy, 8,192 tokens); SA/exact use one fixed GPT router trained by causal language modeling. Their teacher-forced NLLs on 128 complete MathPile documents (747,305 targets) are 1.874181/1.864217/1.861681. Exact checks all 35,960 four-of-32 subsets; simulated annealing (SA) uses 64 reads/1,000 sweeps. Exact slightly lowers NLL versus SA (difference -0.00253557; paired-document 95% CI [-0.002854,-0.002219]); SA answers one more question.

At a fixed Qwen checkpoint, raising SA reads 64→512 lowers NLL 1.606364→1.602179 (1,000 sweeps; lower-energy SA/native route retained).

XUmm-4: Seed variation

XUmm-B reports the three paired seeds.

XUmm-5: Final layer

Final-layer adaptation isolates expert selection while freezing pretrained weights and retaining native mixing: GPT layer 23, Qwen layer 47 (zero-based), with four/eight active experts. Replacements pass directly through residual addition, final normalization and the LM head, without later attention. The construction applies elsewhere; controlled multi-seed evidence here is for the final layer.

XUmm-6: Budget and speed

Changing only GPT AIME24's cap 4,096→8,192 raises SA correct answers 11/30→18/30 and lowers cap hits 16/30→8/30. At 8,192, SA/exact mean outputs are 4,620.43/5,003.10 tokens; cap hits are 8/30 and 11/30.

Timing uses eight prompts, batch 1 and exactly 128 new tokens, each timed twice after warm-up, then with method order reversed. Hardware is matched within each model. Timing includes prompt processing, routing, experts and decoding, excluding loading/tokenization. Throughput (original/reversed, tokens/s): GPT/AIME24 native 26.005/26.789, SA 20.939/20.590, exact 21.867/21.637; Qwen/GSM8K native 22.748/22.709, SA 13.500/13.203.

XUmm-7: Candidate pool

Six candidates combine the executed route, a native reference and four random K-expert alternatives (K=active experts). At a fixed Qwen checkpoint, random entries together receive >=1% soft weight at 152/500 positions (30.4%).

Qwen retraining with M=1/2/4/8 random entries (M+2 total) matches seed 123, initialization, data order, 10M tokens and SA settings. NLLs are 1.606067/1.608330/1.603748/1.607183; all score 7/30 on AIME24. M=4 is lowest (single seed).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.