acceptodds
Under review as a conference paper at ICLR 2027

Adapter-free Parameter-efficient Fine-tuning with Tiny Routers

Abstract

Mixture-of-experts fine-tuning (MoE-PEFT) routes tokens among experts that are newly trained or that span every singular component of the pretrained weights, so it adds computation on top of the frozen backbone. Can it instead use the pretrained model’s own spectral components as experts, selecting only some of them for each token and learning how strongly to weight them? The frozen model already holds much of what adaptation needs, but a strict per-token bud- get over its spectral components does not train when candidates are ranked by the router’s gate logit, which encodes only how far a gate departs from one, not how much its group contributes. Ranking them by spectral energy makes the budget trainable (untrained, it keeps the base accuracy where random selection scores zero), and gates that can amplify the selected groups improve its accuracy. Execution Tuning builds on these findings: it decomposes each MLP weight ma- trix into frozen spectral groups that exactly reconstruct it and trains only a tiny router, 0.055% of Qwen3-8B’s parameters, that learns token-conditioned magni- tudes over an energy-selected support. Against MoE-PEFT baselines on six LLMs and six tasks, it trains 23–100× fewer parameters than MixLoRA and MoORE (with an 11–25% larger frozen bank), counts 1.5–1.7× fewer prefill FLOPs than the cheapest baseline, fewer even than the dense model, and runs faster than all of them, partly through implementation efficiency. It stays within 2 points of the strongest baseline on most model–task pairs, although MixLoRA remains more accurate on three quarters of them, and in single runs it also applies to vision– language, vision, and speech models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
PEFT, MoE, adapter-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iMWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Execution Training to mitigate the computational overhead of the current MoE-PEFT methods. They decompose the frozen MLP matrices into spectral components and train a tiny bottleneck layer to sparsely select and amplify these groups based on a spectral-energy prior. This way, they can adapt the model without adding new experts. They empirically evaluate their method against MoE-PEFT baselines like MixLoRA and MoORE on six LLMs across six tasks and show that their method achieves competitive accuracy with 23-100x fewer parameters and lower prefil FLOPs than the dense baseline model.

Critical strengths
  1. Unlike standard PEFT methods like LoRA, Execution Tuning is the first MoE-PEFT method that executes fewer prefill FLOPs than the dense backbone model by pruning unselected spectral groups.
  2. The bottleneck router utilizes around 0.055% trainable parameters (4.48M on Qwen3-8B), making it 23-100x smaller than the baseline routers while maintaining training stability.
Critical weaknesses
  1. While the prefill inference latency is reported, the total end-to-end wall-clock training time and memory usage are not reported. Can you report them for the benchmarks used?
  2. Unlike standard LoRA, where the weights can be merged into the dense model for zero test-time overhead, Execution Tuning is unmergeable
  3. Even though ET uses lower number of parameters, it performs worse than the baselines in Table 3. Can you report the performance of ET with parameters matched to the baselines?
  4. During deployment, the full SVD factors (A,B) need to be stored, which increases the frozen parameter storage by 11-25%.
Questions for authors

See weaknesses.

3 comments

Thank you for requesting the additional training-cost and parameter-matched comparisons. They make the paper's trade-offs directly assessable.

W1 — Router-only training is practical, with a measurable memory/time choice. The following is Qwen3-8B/GSM8K on one GB300, effective batch 64, each method alone. Values are means over 54 common optimizer-step intervals, one run/configuration, excluding startup and checkpoint-writing intervals.

MethodGradient checkpointings/stepPeak allocated GiB
ETNo5.4384.0
LoRA, r=4No6.1660.9
MoORENo17.8588.7
MixLoRANo26.9983.4
ETYes11.1941.6
LoRA, r=4Yes12.2224.8

At matched checkpointing, ET's optimizer steps are 11.9%/8.5% shorter than LoRA's without/with checkpointing, with 1.38×/1.68× peak memory. The no-checkpointing run also has shorter steps than the tested MoE baselines. Checkpointing reduces ET's allocated peak from 84.0 to 41.6 GiB. These are useful measured choices, not end-to-end wall-clock estimates: SVD/preparation, loading, validation and export are excluded. The new accuracy comparison used checkpointing only for ET, whose corresponding step-time row is 11.19 s. Speed ratios are specific to these isolated-run settings, not universal hardware guarantees.

W2 — Non-mergeability is shared by the tested token-routed comparison class. ET's input-conditioned computation cannot be replaced by one static merged matrix, as with the tested token-routed MoE baselines. Merged LoRA is an important alternative when static execution is the priority; the relevant question is the quality/parameter/execution trade-off, rather than mergeability alone.

In the supplemental unfused projection-stack benchmark, ET/dense is 1.42–1.59× for prefill and 2.20–2.30× for decode, below MixLoRA's 1.80–2.09×/2.66–2.76× and MoORE's 2.24–2.61×/2.72–2.80×. These are projection-stack measurements, not end-to-end serving. The paper already reports its different fused H200 path and graph/eager qualification (§5.5; Appendix I); an unfused benchmark does not refute that path. Historical checkpoint–timing correspondence is not independently verified by the new experiments.

Fixed support supplies a separate sliced execution option. In the new raw-SVD equal-width GSM8K sweep (seed 0, LR2e-4), fixed support at 100/75/50/37.5/25% rank gives 79.76/70.51/40.49/15.54/2.05%. No-LB adaptive routing at 75/50/25% gives 73.84/50.42/8.04%. Its gain includes curriculum/capacity differences (adaptive 4.48M parameters versus fixed 3.37/2.26/1.14M). Fixed-support analytical MLP-factor/router costs are 1.334/1.001/0.667/0.500/0.334× dense, whereas the adaptive harness uses dense masked products at every k. The energy router needs the full input factor; fixed-slice FLOPs must not be attached to adaptive accuracy.

W3 — The requested parameter match preserves a useful operating point. On three-seed ViT-B test, ET with 1,735,876 trainables scores 91.00±0.10%, compared with MixLoRA's 91.94±0.19% at 1,735,780. This uses a wider ET router at 75% rank. At 50% rank, ET gives 90.54±0.15% with 434,236 trainables, about a quarter of MixLoRA's. ET used 34 validation runs versus the reported original MixLoRA configuration; these are parameter-matched, not tuning-budget-matched results. Settings were validation-selected; the 50%-rank test followed the 75% result. Separately, the new full-rank LLM comparison in our general response gives competitive means with 2.4×/26× fewer trainables than LoRA/MixLoRA. Together, these results support useful quality–parameter trade-offs across distinct operating points.

W4 — Frozen storage is shared, and compression offers another choice. Complete Qwen BF16 files are ET 18.654 GiB versus dense 15.257 GiB; only ~18 MB FP32 router state is added per task on the shared bank. On a supplemental full-budget checkpoint, FP8-factor storage reduces ET to 11.907 GiB (−36.17%), with accuracy 79.91→79.08%, one seed; paired question-bootstrap 95% CI for the difference [−2.43,0.76] points. This is a measured storage–quality option. FP8 factors decode to BF16 for computation, so it is not evidence of native-FP8 speed, resident-VRAM savings or unchanged accuracy. The storage overhead already identified in §6 can therefore be assessed and amortized explicitly, rather than conflated with per-task trainable cost.

Reviewer_iMWYOpenReview ↗

Reply to the authors

Thank you for the detailed response and the new measurements. The rebuttal partially answers my questions. The training cost is reported per step for a single model and task, excluding SVD preparation, so the end-to-end comparison is still missing. The new comparison with three seeds uses all groups with no sparsity, so it is not the strict-budget method presented in the paper. The new benchmark also shows ET is slower than the dense model in both prefill and decode, which contradicts the abstract. I think the paper would be stronger if it limited its FLOP and speed claims to what was reproduced at the strict budget.

Thank you for this precise follow-up. We would like to distinguish what the new controls establish and make the efficiency claim correspond to that evidence.

1. Training cost. The reported timings measure optimizer-step throughput, not complete first-task adaptation. SVD construction is a one-time cost for a fixed bank shared across tasks, but this does not justify excluding it from a first-task comparison. We have not established an SVD-inclusive end-to-end training advantage; the step-time result should be read only at its stated scope.

2. Full-group control. This is the dense endpoint of ET's frozen-expert/router parameterization. It tests whether router-only adaptation can retain quality with fewer trainables; it is not evidence for strict-budget compute savings. The strict equal-width results are separate. We will not use the full-group accuracy to support a 25%-rank efficiency claim.

3. Masked versus fused execution. The supplemental unfused benchmark executes dense masked factor products, so zeros do not translate into skipped matrix work. It therefore does not exercise the output-column skipping of the paper's fused prefill path. Its slower-than-dense result is specific to that execution path; it neither establishes nor rules out the speed of the different fused path. However, the distinction alone does not verify the original accuracy–timing pairing.

We therefore accept the requested restriction: the abstract and cost discussion will distinguish trainable-parameter efficiency, analytical arithmetic counts, and measured runtime, and limit strict-budget FLOP/speed claims to configurations whose actual support rank, execution path and accuracy correspondence are verified. Unverified pairings will not be presented as established speed–quality operating points. This preserves the demonstrated frozen-expert adaptation result while making the deployment claim conditional on the evidence actually available.

Reviewer_SfgaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Execution Tuning, an adapter-free MoE-style PEFT method. Each MLP weight matrix is decomposed into frozen SVD groups. A tiny router reads spectral coordinates, ranks candidate groups by a spectral-energy prior, selects a strict per-token top-k support, and learns unbounded ReLU gates over selected groups. No new expert or adapter weights are trained. The main empirical claims are: energy ranking, not gate-logit ranking, makes strict top-k budgets trainable; unbounded gates are important; finer groups tolerate sparsity; the method trains about 0.055% of Qwen3-8B parameters and reduces prefill cost. Experiments cover 36 LLM model–task pairs and several non-LLM models, with ablations and timing.

Critical strengths

1.The energy-prior ablation is unusually clean. Gate-logit ranking collapses under a strict top-k budget, while energy ranking works. This is consequential because it identifies a concrete design requirement for sparse frozen-bank adaptation. 2.Large empirical and cost evaluation. The paper reports many LLM and non-LLM results, parameter counts, FLOP counts, and timing. This makes the practical trade-off visible and supports the method’s efficiency claims better than a single benchmark would.

Critical weaknesses

1.“Adapter-free” and compute-saving claims need qualification. The router is trained, the input spectral factor is still computed densely, decode retains a dense masked output GEMM, and the spectral bank stores extra parameters. This matters because the method is not universally cheaper or adapter-free in the strongest sense. 2.Baseline comparisons are not fully matched. Baselines use their own recipes, some non-LLM results are single-run, and nominal FLOP counts differ in scope. This weakens strong “best PEFT” conclusions, even if the method is promising.

Questions for authors

1.What is the exact end-to-end parameter and memory overhead of storing the spectral bank, including decode-time costs? 2.Are baselines tuned with comparable budgets and seeds, especially for the non-LLM results? 3.How much of the strict-format gap under fixed support can be recovered by format-aware fine-tuning rather than per-token support?

2 comments

Thank you for highlighting the energy-prior result and the empirical breadth. We clarify the meaning of the claims and add the requested controls.

W1 — Adapter-free has a precise architectural meaning. Sections 4.1–4.3 freeze the expert transformations and train the router; they do not equate adapter-free with parameter-free learning. Non-LLM task heads are counted where used. The benefit is avoiding newly trained expert matrices, which is directly reflected in the trainable-parameter counts. The paper already identifies the full input factor, masked decode and bank storage in §6; these disclosures define the execution setting, while the trainable-parameter advantage is measured separately.

W2/Q2 — Additional seeds and operating points. The general response reports the new validation-selected, three-seed Qwen comparison: ET 81.40±0.75% with 4.48M trainables, LoRA 81.27±0.79%/10.91M, MixLoRA 81.70±0.72%/118.26M and MoORE 79.78±0.94%/336.33M. The scope is competitive quality with substantially fewer trainables, rather than universal best-PEFT performance. Method-specific search budgets and hardware/checkpointing differences are stated there.

New non-LLM test results use seeds 0/1/42, validation-selected checkpoints and mean ± sample SD. Rank is selected support, not measured execution saving; ViT uses dense masked products.

Model/taskConfigurationTrainablesTest result
ViT-B, CIFAR-100ET, original PI, 34–39% rank434,21286.12±0.31%
ViT-BET, strict 25%, LB=0.01434,23673.15±0.28%
ViT-BET, strict 25%, no LB434,23686.64±0.20%
ViT-BET, 50% rank434,23690.54±0.15%
ViT-BET, 75%, wider router1,735,87691.00±0.10%
ViT-BMixLoRA1,735,78091.94±0.19%
VLM-3B, ScienceQAET2,713,82493.26±0.40%
VLM-3BMixLoRA23,777,28096.11±0.17%
VLM-3BSparse Upcycling9,739,468,80095.68±0.09%
Whisper-small, FLEURSET714,624WER 8.93±1.62
Whisper-smallMixLoRA3,465,216WER 6.68±0.11

ViT reaches 90.54% with about a quarter of MixLoRA's trainables; parameter matching narrows the gap to 0.94 points. ET used 34 validation runs, while the reported MixLoRA test uses its original configuration, so tuning budgets are unequal. The 50%-rank settings were fixed on validation; its test was read after the 75% result. The LB/no-LB ViT runs also differ in runner/bank numerics, preventing an LB-only interpretation. ScienceQA here uses all 4,241 questions, distinct from the original image-question validation entry. Original validation records and new tests are kept separate, as noted in the general response.

Q1 — Storage, memory and decode are different costs. The factors replace each MLP matrix; they are not additional copies of it. A rank-r factorization stores r(d_in+d_out) parameters. Complete Qwen BF16 files are 18.654 GiB for ET versus 15.257 GiB dense. This bank is shared once across tasks, with ~18 MB FP32 router state per task.

A supplemental full-budget checkpoint also gives a practical storage option: FP8 factors reduce complete files to 11.907 GiB (−36.17% versus BF16 ET), with GSM8K 79.91→79.08%, one seed; paired 95% CI for the difference [−2.43,0.76] points. This is an observed storage–quality trade-off, not an equivalence claim. Selection stayed unchanged on 120 clean validation questions after removing eight calibration overlaps. Factors decode to BF16 for compute; native-FP8 speed and end-to-end inference memory remain unmeasured.

For decode, the paper explicitly separates the masked path and fused prefill (§6; Appendix I). The supplemental unfused projection stack measures 1.42–1.59× dense prefill/2.20–2.30× decode; it should not be substituted for the paper's different fused decoder benchmark.

Q3 — Training largely removes the fixed-support format gap. In the new equal-width GSM8K sweep (all 1,319 test questions, seed 0, raw SVD, LR2e-4), the strict/flexible gap is ≤0.31 points. The remaining issue is answer quality: at 75/50/25% rank, fixed support gives 70.51/40.49/2.05%, while no-LB adaptive routing gives 73.84/50.42/8.04% (LB=0.01: 63.68/21.61/2.12%). Fixed support is calibrated on 512 train-only prompts. The adaptive recipe includes a curriculum and 4.48M trainables versus fixed 3.37/2.26/1.14M, so this is a recipe-level gain. It directly answers the format concern without treating the unavailable historical J.2 checkpoint as a reproduced control.

Thank you for the clarifications. Most of my concerns have been resolved. Therefore, I maintain my score.

Reviewer_hCc8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In Mixture of Experts (MoEs), the increase in capability by introducing the experts comes at a cost of increased number of parameters and routing overhead. MixLoRA alleviates this by replacing full-MLP experts with LoRA experts, yet they still execute the entire pre-trained backbone. In contrast, this paper proposes Execution Tuning (ET), an adapter-free PEFT approach that reuses the computation already present in the pre-trained model. ET applies SVD to each MLP weight matrix and groups singular directions into 256 spectral experts. It then trains a lightweight router to select the top-k spectral groups (k=64 in the main experiments) and learns multiplicative gates that rescale each selected group's contribution to the layer output. Spectral-energy prior is used to initialize the routing. The authors show that just without the initialization, the routing collapses because all groups are initially equivalent. The energy prior leads to stable sparse routing by providing a meaningful ranking signal. Despite training only a lightweight router, ET' performance is comparable to adaptation methods, using larger number of parameters. It is close to MixLoRA and outperforms MoORE across several math and reasoning benchmarks.

Critical strengths
  1. Unlike MixLoRA, it does not introduce additional trainable expert weights. Owing to its small router, the total number of trainable parameters is also much smaller than MoORE.
  2. The ablation studies are thorough. Several gating choices are evaluated, including binary gates, clipped gates, amplifying ReLU gates, energy-only selection, fixed support, and token-independent gate magnitudes. It also demonstrates why the energy prior is needed to make strict top-k routing trainable. These experiments convincingly show that both the energy-based ranking and token-conditioned amplifying gates are important.
  3. Rather than using 1 singular direction as an expert like in MoORE, this paper grouped several singular directions to form 1 expert. This substantially reduces the number of routing decisions and allows the fused pre-fill implementation to skip output-factor groups that are unused by the batch.
  4. The experiments are across various LLM families and also included vision-language, vision, and speech models, suggesting that the underlying idea is not specific to one transformer architecture.
Critical weaknesses
  1. For each dataset, the router is trained and evaluated on that dataset alone. As opposed to this, MoORE jointly trains on the training splits of nine datasets and evaluates on all their respective evaluation splits. MixLoRA also reports both single-task and multi-task settings. Please evaluate ET in a comparable joint multi-task setting, using a setup similar to Table 3 of the MoORE paper across all evaluated model families. This is necessary to determine whether ET can learn task-specific routing while avoiding interference across tasks.
  2. Does this mean that the router mainly learns token-conditioned gate magnitudes rather than meaningfully different expert selections? Please show which spectral groups are selected for each task and how the selected supports vary across tasks. In particular, we want to ensure that ET is not consistently selecting the same groups for all tasks, with adaptation occurring almost entirely through the gate magnitudes.
  3. Unlike MixLoRA and related methods, ET cannot learn new expert weight directions. This may limit its performance on tasks that require capabilities that are not already present in the pre-trained weights.
Questions for authors
  1. Have you considered using differentiable routing function instead of top-k, similar to LD-MoLE [1], where for each token, the number of experts to activate are dynamically determined for each layer.
  2. Since the spectral directions are grouped consecutively in descending singular-value order, each group may contain components with similar magnitude but not necessarily similar function. Did the authors consider learned grouping or grouping based on activation patterns? Also, did you experiment with energy balanced groups? The leading groups may contain substantially more spectral energy than later groups. Equal-width grouping could therefore produce experts with very different importance, making some groups almost always selected.
  3. How much of the latency improvement is coming from grouping the singular directions rather than considering each singular direction separately?

[1] LD-MOLE: Learnable Dynamic Routing For Mixture of LoRA Experts, Zhuang et al.

1 comment

Thank you for the detailed questions. They help distinguish what is learned by ET from the choice of deployment setting.

W1 — Per-task PEFT and joint multi-task training. Section 5.1 explicitly trains model–method–task cells independently; joint CSR-MTL, as in MoORE's Table 3, is a different setting. For this per-task objective, the direct control trains every method per task. Our additional common-harness comparison gives ET 81.40±0.75%, MixLoRA 81.70±0.72% and MoORE 79.78±0.94%, with 26×/75× fewer trainables for ET, at full rank.

With a known task, selecting a task router is analogous to selecting a task adapter. One frozen bank serves all tasks; only roughly 18 MB FP32 router state is added per task. Updating one task's router leaves the others unchanged. This is a concrete use case, distinct from a jointly trained single router or automatic task identification, for which we do not claim measured results.

W2 — The learned magnitudes matter even when rankings are stable. The method explicitly learns how strongly selected components contribute (§4.3), not merely their identities. A stable diagnostic ranking is therefore insufficient to conclude that the router learns little. Exchanging only learned router state on an identical supplemental bank/backbone gives:

EvaluationOwn routerOther routerDifference, 95% CI
GSM8K, 1,319 questions79.9156.48−23.43 [−26.38,−20.47]
MATH, 1,000 questions44.7031.40−13.30 [−17.00,−9.60]

Accuracy is %, with all 256 groups; archived identities match for all 831 non-router tensors. This supplemental bank/recipe is separate from the paper's data-free main bank. Diagnostic top-64 gate-frequency Jaccard is 0.93 across tasks for one router versus 0.24 between routers; these full-budget rankings are not executed sparse supports. Task and seed/optimization differ, so the exchange supports learned-state relevance, not task-only causality or sparse-selection necessity.

W3 — Frozen experts do not imply an unadaptable function. Input-conditioned magnitudes change the effective transformation with the input. In a separate matched-bank MATH control, unit gates give 237/1,000 correct and learned gates 409/1,000: +17.2 points, paired-bootstrap 95% CI [13.7,20.6]. The cohort, backend and 1,024-token cap match, and both archives use common rescoring; generation evaluator revisions differ. This is a one-seed, full-budget adaptation control.

The new-format experiments identify both capability and efficiency. ET learns a new tool schema to 90.80±0.89% (LoRA/MixLoRA 99.77±0.06/99.57±0.12%). On invented notation, an expanded 18.2M router (~4×), changed gate scale and extended schedule allowing up to 54 epochs reach 71.95% on 2,000 fresh examples, one seed. Under the common three-epoch recipe, ET scores 0.20±0.00% versus LoRA 66.00±7.65% and MixLoRA 67.10±17.38% (three seeds). Thus adaptation is demonstrated, while this recovery is not matched-budget evidence.

The MATH coverage analysis gives similar overlap for ET and LoRA: 84.6/82.5% of their correct answers overlap a correct base sample. Both are below the ≥95% diagnostic criterion. Base coverage@64 is 63.3%, with 75.01% of sampled outputs hitting the cap; finite sampling is not a proof of absent knowledge or of no benefit from new weight directions.

Q1 — Variable budgets. The PI-controlled ReLU variant already provides top-k-free token/layer-varying gate counts (§4.4; Appendix B.3). Differentiable learned sparsity, including LD-MoLE, is complementary related work, not a baseline we claim to have run. Group count and support rank remain distinct.

Q2/Q3 — Why group directions? Contiguity is an execution design, not a semantic claim about neighboring singular vectors. On one random Qwen-shaped MLP layer (GB300, BF16, batch 64×512), router/top-k work is 1.80–2.20 ms for 256 groups versus 8.42–8.49 ms for 4,096 directions. At shared 25% support, two-factor slice/group gather/direction gather take 2.87–2.91/2.95–2.98/2.93 ms versus dense 6.89–6.91 ms. Grouping's measured benefit is lower routing work. These are separate components, not integrated token-routed serving; the energy router still requires the full input factor. Learned boundaries are a possible alternative, without a tested comparative-quality claim here.

Comments on the paper

1 comment

We thank the reviewers for recognizing the paper's empirical scope and the importance of energy-based routing. The contribution is a different way to adapt a pretrained model: keep its spectral expert transformations frozen and learn a small, input-dependent router that selects and reweights them. The paper evaluates this design across six LLMs and six tasks. This separates adaptation capacity from the number of newly trained expert parameters.

What the method contributes. Sections 4.2–4.3 distinguish energy-based candidate selection from learned magnitudes. The energy criterion has a precise reconstruction interpretation for unit gates (Propositions 2 and 4); learned magnitudes then adapt the resulting computation. Neither a stable group ranking nor the use of frozen directions makes the adapted function fixed. This is why router behavior, trainable-parameter efficiency and execution cost should be evaluated separately.

Additional controlled evidence supports this contribution. On Qwen3-8B/GSM8K, the following three-seed comparison uses the same split, three epochs, effective batch 64 and minimum validation-loss checkpoint selection. Accuracy is mean ± sample SD over seeds 0/1/42, on all 1,319 test questions.

MethodTrainablesTest accuracy (%)
Execution Tuning4,483,29681.40 ± 0.75
LoRA, r=410,911,74481.27 ± 0.79
MixLoRA118,259,71281.70 ± 0.72
MoORE336,328,70479.78 ± 0.94

ET is within 0.30 points of MixLoRA while using 26× fewer trainables, and uses 2.4×/75× fewer than LoRA/MoORE. This follow-up keeps all 256 groups, uses raw data-free SVD and no LB loss; it establishes a quality–parameter operating point, distinct from the paper's strict-budget Table 7 and its 4,483,404-parameter main configuration. Method-specific LR searches used 5/3/3/4 trials for ET/LoRA/MixLoRA/MoORE; seed-0 validation selected rates, then seeds 1/42 were fixed repeats. MixLoRA's latter repeats used GH200, others GB300. ET alone used gradient checkpointing. These are competitive means, not a statistical-equivalence claim.

Learned gating also improves a matched supplemental frozen MATH bank from 23.7% to 40.9% (+17.2 points; paired question-bootstrap 95% CI [13.7,20.6]). Exchanging learned routers substantially reduces accuracy. The reviewer replies give these behavioral controls, new three-seed non-LLM results and measured deployment trade-offs.

Keeping the evidence at the right scope. The paper already states the dense input-factor requirement, masked decode, extra frozen storage and graph/eager differences (§§5.5–6; Appendix I). These qualify deployment; they do not negate adaptation with few trainables. Our new unfused benchmarks are separate measurements, not a refutation of the paper's fused H200 path. Historical accuracy–timing correspondence remains unverified and is not independently established by these follow-ups. Likewise, nominal slot counts must not substitute for measured rank: one released-bank Qwen reconstruction selects 97.37% rank with nominal top-64 on 32 training prompts, which does not identify every original checkpoint. We distinguish verified full-rank and strict equal-width results.

For numerical consistency, original printed→recorded validation entries are ViT-B 91.90→86.90, ViT-L 93.48→91.48, VLM-3B 93.89→91.89 and VLM-7B 96.32→96.33; the causes are not established. New test results are reported separately. Historical J.2 loss 0.629 cannot be checked without its checkpoint; the fresh strict-25% control gives loss 1.20, and the new fixed-support sweep directly answers the format question.

The evidence therefore supports the paper's core contribution: useful adaptation through learned routing over frozen experts, with a small task state and explicit quality–execution operating points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.