- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Execution Training to mitigate the computational overhead of the current MoE-PEFT methods. They decompose the frozen MLP matrices into spectral components and train a tiny bottleneck layer to sparsely select and amplify these groups based on a spectral-energy prior. This way, they can adapt the model without adding new experts. They empirically evaluate their method against MoE-PEFT baselines like MixLoRA and MoORE on six LLMs across six tasks and show that their method achieves competitive accuracy with 23-100x fewer parameters and lower prefil FLOPs than the dense baseline model.
Critical strengths
- Unlike standard PEFT methods like LoRA, Execution Tuning is the first MoE-PEFT method that executes fewer prefill FLOPs than the dense backbone model by pruning unselected spectral groups.
- The bottleneck router utilizes around 0.055% trainable parameters (4.48M on Qwen3-8B), making it 23-100x smaller than the baseline routers while maintaining training stability.
Critical weaknesses
- While the prefill inference latency is reported, the total end-to-end wall-clock training time and memory usage are not reported. Can you report them for the benchmarks used?
- Unlike standard LoRA, where the weights can be merged into the dense model for zero test-time overhead, Execution Tuning is unmergeable
- Even though ET uses lower number of parameters, it performs worse than the baselines in Table 3. Can you report the performance of ET with parameters matched to the baselines?
- During deployment, the full SVD factors (A,B) need to be stored, which increases the frozen parameter storage by 11-25%.
Questions for authors
See weaknesses.
3 comments
Reply to the authors
Thank you for the detailed response and the new measurements. The rebuttal partially answers my questions. The training cost is reported per step for a single model and task, excluding SVD preparation, so the end-to-end comparison is still missing. The new comparison with three seeds uses all groups with no sparsity, so it is not the strict-budget method presented in the paper. The new benchmark also shows ET is slower than the dense model in both prefill and decode, which contradicts the abstract. I think the paper would be stronger if it limited its FLOP and speed claims to what was reproduced at the strict budget.
Thank you for this precise follow-up. We would like to distinguish what the new controls establish and make the efficiency claim correspond to that evidence.
1. Training cost. The reported timings measure optimizer-step throughput, not complete first-task adaptation. SVD construction is a one-time cost for a fixed bank shared across tasks, but this does not justify excluding it from a first-task comparison. We have not established an SVD-inclusive end-to-end training advantage; the step-time result should be read only at its stated scope.
2. Full-group control. This is the dense endpoint of ET's frozen-expert/router parameterization. It tests whether router-only adaptation can retain quality with fewer trainables; it is not evidence for strict-budget compute savings. The strict equal-width results are separate. We will not use the full-group accuracy to support a 25%-rank efficiency claim.
3. Masked versus fused execution. The supplemental unfused benchmark executes dense masked factor products, so zeros do not translate into skipped matrix work. It therefore does not exercise the output-column skipping of the paper's fused prefill path. Its slower-than-dense result is specific to that execution path; it neither establishes nor rules out the speed of the different fused path. However, the distinction alone does not verify the original accuracy–timing pairing.
We therefore accept the requested restriction: the abstract and cost discussion will distinguish trainable-parameter efficiency, analytical arithmetic counts, and measured runtime, and limit strict-budget FLOP/speed claims to configurations whose actual support rank, execution path and accuracy correspondence are verified. Unverified pairings will not be presented as established speed–quality operating points. This preserves the demonstrated frozen-expert adaptation result while making the deployment claim conditional on the evidence actually available.
Thank you for requesting the additional training-cost and parameter-matched comparisons. They make the paper's trade-offs directly assessable.
W1 — Router-only training is practical, with a measurable memory/time choice. The following is Qwen3-8B/GSM8K on one GB300, effective batch 64, each method alone. Values are means over 54 common optimizer-step intervals, one run/configuration, excluding startup and checkpoint-writing intervals.
At matched checkpointing, ET's optimizer steps are 11.9%/8.5% shorter than LoRA's without/with checkpointing, with 1.38×/1.68× peak memory. The no-checkpointing run also has shorter steps than the tested MoE baselines. Checkpointing reduces ET's allocated peak from 84.0 to 41.6 GiB. These are useful measured choices, not end-to-end wall-clock estimates: SVD/preparation, loading, validation and export are excluded. The new accuracy comparison used checkpointing only for ET, whose corresponding step-time row is 11.19 s. Speed ratios are specific to these isolated-run settings, not universal hardware guarantees.
W2 — Non-mergeability is shared by the tested token-routed comparison class. ET's input-conditioned computation cannot be replaced by one static merged matrix, as with the tested token-routed MoE baselines. Merged LoRA is an important alternative when static execution is the priority; the relevant question is the quality/parameter/execution trade-off, rather than mergeability alone.
In the supplemental unfused projection-stack benchmark, ET/dense is 1.42–1.59× for prefill and 2.20–2.30× for decode, below MixLoRA's 1.80–2.09×/2.66–2.76× and MoORE's 2.24–2.61×/2.72–2.80×. These are projection-stack measurements, not end-to-end serving. The paper already reports its different fused H200 path and graph/eager qualification (§5.5; Appendix I); an unfused benchmark does not refute that path. Historical checkpoint–timing correspondence is not independently verified by the new experiments.
Fixed support supplies a separate sliced execution option. In the new raw-SVD equal-width GSM8K sweep (seed 0, LR2e-4), fixed support at 100/75/50/37.5/25% rank gives 79.76/70.51/40.49/15.54/2.05%. No-LB adaptive routing at 75/50/25% gives 73.84/50.42/8.04%. Its gain includes curriculum/capacity differences (adaptive 4.48M parameters versus fixed 3.37/2.26/1.14M). Fixed-support analytical MLP-factor/router costs are 1.334/1.001/0.667/0.500/0.334× dense, whereas the adaptive harness uses dense masked products at every k. The energy router needs the full input factor; fixed-slice FLOPs must not be attached to adaptive accuracy.
W3 — The requested parameter match preserves a useful operating point. On three-seed ViT-B test, ET with 1,735,876 trainables scores 91.00±0.10%, compared with MixLoRA's 91.94±0.19% at 1,735,780. This uses a wider ET router at 75% rank. At 50% rank, ET gives 90.54±0.15% with 434,236 trainables, about a quarter of MixLoRA's. ET used 34 validation runs versus the reported original MixLoRA configuration; these are parameter-matched, not tuning-budget-matched results. Settings were validation-selected; the 50%-rank test followed the 75% result. Separately, the new full-rank LLM comparison in our general response gives competitive means with 2.4×/26× fewer trainables than LoRA/MixLoRA. Together, these results support useful quality–parameter trade-offs across distinct operating points.
W4 — Frozen storage is shared, and compression offers another choice. Complete Qwen BF16 files are ET 18.654 GiB versus dense 15.257 GiB; only ~18 MB FP32 router state is added per task on the shared bank. On a supplemental full-budget checkpoint, FP8-factor storage reduces ET to 11.907 GiB (−36.17%), with accuracy 79.91→79.08%, one seed; paired question-bootstrap 95% CI for the difference [−2.43,0.76] points. This is a measured storage–quality option. FP8 factors decode to BF16 for computation, so it is not evidence of native-FP8 speed, resident-VRAM savings or unchanged accuracy. The storage overhead already identified in §6 can therefore be assessed and amortized explicitly, rather than conflated with per-task trainable cost.