acceptodds
Under review as a conference paper at ICLR 2027

ISO: An RLVR-Native Optimization Stack

Abstract

Reinforcement learning with verifiable rewards (RLVR) advances the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. We study this layer and functionally validate spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior by adapting both the associated input and output singular frames. We operationalize this finding as Isospectral Optimization (ISO), an RLVR-native optimization stack with complementary offline and online instantiations. Offline, ISO-Merger composes shared-base specialists in fixed-spectrum frame coordinates without post-merge data, rollouts, or distillation, and achieves the highest aggregate scores among the data-free merging methods. Online, ISO-Optimizer applies AdamW or Muon directly to the frame variables under fixed base spectra, improving aggregate accuracy in the reported math and coding experiments. On Qwen3-4B-Base, ISO-AdamW reaches AdamW's 400-update accuracy in 280 updates with 1.5 X fewer updates and 27% less wall-clock time, and exceeds AdamW by 2.2 points at update 400. These results support a reusable RLVR design principle: inherit the spectrum, optimize the frames.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR, optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3dnfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the spectral structure of RLVR updates and finds that the base model's singular values can be reused while adapting the singular vectors. Based on this observation, the authors propose ISO, including ISO-Merger for data-free expert merging and ISO-Optimizer for online RLVR training. Experiments show improvements in merging performance, reasoning accuracy, and training efficiency.

Critical strengths
  1. Well-motivated empirical finding. The paper goes beyond observing limited spectral drift in RLVR. The spectrum restoration and fixed-spectrum training experiments (Section 2) provide useful evidence that the base spectra can be reused without losing most of the learned capabilities. This gives a reasonable motivation for the proposed methods.

  2. Consistent empirical improvements. ISO-Optimizer outperforms the tested AdamW and Muon baselines on math and coding tasks. On Qwen3-4B, it reaches AdamW's final accuracy with 34% less wall-clock time. ISO-Merger also achieves the best aggregate scores among the evaluated data-free merging methods. The comparisons with Pion and the additional ablations strengthen the empirical evaluation.

Critical weaknesses
  1. Limited novelty in the core parameterization. Spectrum-preserving optimization has already been explored in POET and Pion, while Stiefel-manifold parameterizations have also appeared in prior work. ISO differs in how it optimizes the singular frames and applies this structure to RLVR, but the underlying geometric idea is not new. The methodological advance over these approaches is therefore somewhat incremental.

  2. The source of the optimization gains remains unclear. The trainable-spectrum ablation (Appendix Fig. 18) performs similarly to the fixed-spectrum variant. This supports the claim that spectral changes are unnecessary, but does not establish whether fixing the spectrum is responsible for the improvement over conventional optimization. The gains may instead come from the factorized parameterization or retraction dynamics. Also, the main online experiments are limited to relatively small models, with limited multi-seed evidence.

Questions for authors
  1. Can the authors better isolate the contributions of fixed-spectrum constraints, singular-frame parameterization, and orthogonal retraction to the observed optimization gains?

  2. How robust are the improvements at larger model scales and across random seeds? In particular, would the advantage over Pion remain with more extensive tuning on Qwen3-4B?

5 comments
AuthorsOpenReview ↗

Rebuttal 1/3

Reviewer 3dnf

Thank you for recognizing the functional evidence for spectral inheritance and the consistent improvements of both ISO applications. We address your concerns about novelty, attribution, and robustness below.

Q1. Novelty and relationship to prior work.

Limited novelty in the core parameterization. Spectrum-preserving optimization has already been explored in POET and Pion, while Stiefel-manifold parameterizations have also appeared in prior work. … the underlying geometric idea is not new.

Response: We appreciate the opportunity to clarify the sources of novelty in our work. We agree that ISO shares geometric structure with prior methods such as POET, Pion, and Stiefel-manifold optimizers. Sharing a geometric constraint does not imply identical scientific findings or algorithms. We believe that assessing ISO's novelty primarily through its shared spectral constraint overlooks its combined scientific and methodological contribution, which extends beyond the reparameterization alone.

(1) Identifying functional spectral inheritance not established by prior methods. We establish a functional property of RLVR: adaptation can be shifted to the singular frames while inheriting the source spectrum. Successful training with a spectrum-preserving optimizer alone, despite the shared geometry, does not establish that the source spectrum remains reusable after unconstrained RLVR. This is a distinct scientific contribution that those prior methods have not established: we move beyond observations of spectral stability to functionally validate source-spectrum reuse after unconstrained RLVR.

(2) Proposing a shared adaptation space that supports an RLVR optimization framework beyond a single optimizer design. ISO establishes an RLVR optimization framework, rather than another individual optimizer design under the same geometric constraint. Grounded in a functionally validated property of RLVR, ISO provides a shared adaptation space that supports multiple methods beyond the reparameterization itself: online learning directly optimizes the frames, while offline composition combines specialists' learned frame changes. This joint construction is a contribution at the framework level that prior work on similar reparameterizations has not established.

(3) Distinct learning and composition algorithms.

  • ISO-Optimizer provides a generic transformation of base optimizers. It applies the chosen base optimizer—AdamW or Muon in our experiments—directly to the singular frames, maintains optimizer states in those coordinates, and restores orthonormality through polar retraction. As you also noted, its concrete update rule differs from those of prior methods despite the shared weight geometry. Both instantiations improve over their corresponding baselines.

  • ISO-Merger introduces a novel formulation of expert merging: it composes changes in both singular frames relative to a shared base, using retention-aware weighting and mode selection under the inherited spectrum. It requires no post-merge data, rollouts, or distillation, and the experts need not have been trained with ISO. The shared parameterization does not specify this composition algorithm; that construction is a methodological contribution of ISO-Merger.

ISO therefore contributes an RLVR optimization stack grounded in a functional finding. Online policy learning and offline expert composition are two complementary instantiations of the same empirically supported principle. We believe that the functional finding, unified RLVR optimization framework, and concrete algorithms together constitute a substantive contribution beyond the shared geometric constraint.

AuthorsOpenReview ↗

Rebuttal 2/3

Q2. Attribution of the optimization gains.

The source of the optimization gains remains unclear...This supports the claim that spectral changes are unnecessary, but does not establish whether fixing the spectrum is responsible for the improvement... The gains may instead come from the factorized parameterization or retraction dynamics.

Can the authors better isolate the contributions of fixed-spectrum constraints, singular-frame parameterization, and orthogonal retraction to the observed optimization gains?

Response: We appreciate this question and would first respectfully clarify that we do not attribute ISO's gains over AdamW to spectrum freezing. As discussed in Q1, spectral inheritance naturally motivates optimizing the singular frames while reusing the source spectrum. Frame-coordinate updates and orthonormality restoration through polar retraction are both integral parts of this finding-grounded design. Our submission explicitly states that “The factor parameterization and optimizer-state dynamics are both part of the method” (Section 4.2, Lines 320–323).

Additional attribution controls. Following your suggestion, we add five Qwen3-4B-Base/16K ablation arms to examine the roles of spectrum fixing, frame-coordinate optimization, and polar retraction. The full design, learning rates, accuracy curves, and gradient-norm table are reported in our response to Reviewer YDHU, Q3. We report accuracy to the latest 210 steps for each arm. We show that:

  • Spectrum projection alone does not reproduce ISO's gains (A/B/C). At update 210, ordinary and projected AdamW score 42.6 and 41.9, respectively.
  • Frame-based optimization adds value beyond the spectral constraint (B/C). Both B and C enforce the source spectrum, but ISO reaches 44.0, 2.1 points higher than projected AdamW. It also exceeds ordinary AdamW by 1.4 points.
  • Orthogonal retraction helps stabilize optimization (D/E). The variant without retraction (E) learns faster initially, but its actor gradient norm rises sharply after about 190 updates. In the 10-update window labeled 210, its mean / maximum norms reach 0.23 / 0.55, versus 0.04 / 0.05 for both the variant with retraction (D) and AdamW. This is accompanied by a decline in accuracy from 43.2 at update 160 to 41.9 at update 210, when D scores 44.3—a gap of 2.4 points.

Overall, spectral inheritance motivates optimizing the singular frames while maintaining their orthonormality. Both components (frame parameterization and orthogonal retraction) are integral to ISO-Optimizer, a simple instantiation of our design principle, and the added controls support their respective roles.

Our paper does not attribute ISO’s performance gains to spectrum fixing. Figure 18 instead shows that, once optimization is shifted to the singular frames, making the spectrum trainable provides little additional performance benefit. We emphasize that establishing source-spectrum reusability is itself a key contribution, even if spectrum fixing alone offers little direct performance benefit. This finding provides the empirical basis for shifting adaptation to the singular frames, grounds our principle, “reuse the spectrum, optimize the frames,” and enables the broader ISO framework for online learning and offline expert composition.

AuthorsOpenReview ↗

Rebuttal 3/3

Q3. Larger models, independent seeds, and tuned Pion.

How robust are the improvements at larger model scales and across random seeds? In particular, would the advantage over Pion remain with more extensive tuning on Qwen3-4B?

Response: Full results for the seed and Pion experiments are reported in our response to Reviewer YDHU, Q1.a and Q2; we summarize them below alongside the 8B comparison.

(1) Independent training seed in the main setting. We have added one independent training seed per method in the main Qwen3-4B/16K setting, giving two runs per method at the same 400-update endpoint:

MethodAccuracy at update 400 (original; additional)Mean ± sample SD
AdamW (2×10−62\times10^{-6})44.82; 44.5044.66 ± 0.23
ISO-AdamW (1×10−61\times10^{-6})46.98; 46.2446.61 ± 0.52

ISO-AdamW remains ahead in both repetitions, with a mean advantage of 1.95 percentage points.

(2) Additional Pion tuning on 4B. Using the official Pion implementation, we sweep its learning rate over {1,2,3}×10−6\{1,2,3\}\times10^{-6}, extending the comparison beyond the published 1×10−61\times10^{-6} setting. Higher learning rates lead to late-stage performance collapse, and Pion's highest reported score remains below ISO-AdamW's final accuracy: 44.32 versus 46.98, a gap of 2.66 percentage points.

(3) Larger-model validation: Qwen3-8B-Base. We use one training seed per method and a batch of 512 prompts, twice the batch size in the main experiments, to accelerate the experiment. The runs reach 120 steps now, while the results through the 120 steps are already informative, as this budget covers roughly two epochs over the training datasets. AdamW uses 2×10−62\times10^{-6}, its best tested learning rate on 4B.

UpdateAdamWISO-AdamWDifference (points)
0 (shared base)20.9620.96—
2036.9439.51+2.57
4041.7643.61+1.85
6045.4347.52+2.09
8047.7648.61+0.85
10047.9849.76+1.78
12048.8450.35+1.51
  • Higher accuracy. ISO-AdamW leads at every evaluated checkpoint from updates 20–120, by 0.85–2.57 points, and is 1.51 points higher at update 120.
  • Faster time to target. Using AdamW's update-120 accuracy, 48.84, as the common target on the shared evaluation schedule, ISO first reaches it at update 100, versus update 120 for AdamW, using about 17% fewer updates and 18% less training time.

We will include these results in the revised manuscript.


Thank you again for your review and patience, and apologies for the slightly delayed response while we complete the requested ablations. We hope these additional results and clarifications address your concerns, and we would be grateful if you would consider raising your overall assessment. Please let us know if you have any further questions.

Reviewer_YDHUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies weight-space changes during reinforcement learning with verifiable rewards (RLVR) and proposes spectral inheritance: in the analyzed settings, a base model’s singular-value spectra can be functionally reused while behavioral adaptation is carried by changes in the left and right singular frames. The authors quantify checkpoint distance to fixed-spectrum matrix families and calibrate spectrum-changing motion against ambient dimensionality. They then investigate functional reuse through post-training spectrum restoration, fixed-spectrum training, and spectrum-only training. Truncated checkpoint-reconstruction experiments examine the adaptability of the two singular subspaces.

ISO parameterizes each transformed weight as W=UΣ₀Vᵀ, keeping the base spectrum Σ₀ fixed while maintaining orthonormal columns in U and V. ISO-Merger combines expert frame displacements in tangent spaces anchored at a shared base, using mode selection, retention-aware coefficients, and polar retraction, without post-merge data or training. ISO-Optimizer applies AdamW or Muon to the frame variables and restores feasibility through polar retraction.

Offline merging achieves the highest reported aggregate scores in two shared-base settings: 63.80 versus 62.88 for the strongest baseline on Qwen2.5-7B, and 44.38 versus 43.52 on DS-1.5B (Tables 1 and 3). In mathematical reasoning, ISO-AdamW achieves 29.41 and 46.98 on Qwen3-1.7B and 4B, compared with 28.20 and 44.82 for tuned AdamW (Table 2). In the profiled 4B run, ISO reaches AdamW’s 400-update accuracy after 260 updates and 240 hours, compared with 364 hours for AdamW, reducing measured time to target by approximately 34%. This is a run-specific training-efficiency result rather than a 34% reduction in optimizer-step latency.

Critical strengths

1. Well-supported motivation and appropriately qualified interpretation

The spectral analysis uses calibrated statistics and controlled interventions to distinguish functional reusability from mere correlation. Spectrum restoration preserves most RLVR gains, while transferring the RL spectrum to base frames does not reproduce them. Additional SFT/RLVR comparisons further support the motivation for ISO. Importantly, the authors acknowledge the limitations of truncated reconstruction and present ISO as an evidence-informed inductive bias rather than an exact characterization of RLVR updates.

2. Practical applications with transparent resource accounting

ISO-Merger achieves competitive aggregate performance across specialist merging settings without additional rollouts or distillation. Ablations demonstrate contributions beyond spectrum restoration, particularly from mode masking. ISO-AdamW improves aggregate accuracy across mathematical reasoning and coding tasks, with learning-rate sweeps and time-to-target evaluations supporting the results. The reported memory and latency overheads provide useful transparency regarding practical efficiency trade-offs.

Critical weaknesses

1. Limited reproducibility and incomplete comparisons with related methods

The reported gains over AdamW are modest (0.78–2.16 percentage points), while the two-seed experiment uses a smaller model and different configuration rather than replicating the main 16K results or 4B efficiency measurements. Multiple evaluation samples do not address training variability. Moreover, Pion is evaluated without configuration-specific tuning, and stronger comparisons with related methods such as POET and Pion are needed to establish the incremental online contribution.

2. Incomplete attribution of performance gains and limited robustness evidence

Fixed-spectrum training performs comparably to trainable-spectrum training, supporting its feasibility but not establishing that spectrum preservation drives the gains over AdamW. Changes in parameterization, gradient scaling, optimizer states, and retraction remain entangled.

For merging, the aggregate improvements are modest (0.86–0.92 points), and mode masking contributes substantially more than retention-aware coefficients. The sensitivity of ρ_keep=0.9, baseline tuning, and generalization beyond the two evaluated settings remain insufficiently examined. Additionally, unresolved rotational ambiguity in repeated singular-value subspaces may affect merger stability and warrants evaluation.

Questions for authors

1. Statistical stability and efficiency: Can the authors report independent training seeds and uncertainty estimates for the main 16K configuration, including accuracy and the 4B time-to-target result (260 updates, 240 hours)? Paired uncertainty estimates for merging would also strengthen the reported gains.

2. Closest baseline: How does Pion compare after comparable learning-rate tuning on 4B in accuracy and wall-clock cost? If tuning is infeasible, please qualify the conclusions and clarify stopping and checkpoint-selection criteria.

3. Performance attribution: Given the similar fixed- and trainable-spectrum results, are the gains primarily attributable to frame parameterization and optimizer dynamics rather than spectrum fixing? A budget-matched spectrum-projection or alternative-parameterization baseline would help isolate these effects.

4. Merging robustness: How were ρ_keep=0.9, clipping, and ridge parameters selected, and were evaluation labels involved? Please clarify baseline tuning, parameter sensitivity, and numerical robustness to degenerate singular values and SVD implementations.

5. Implementation and reproducibility: How does SVD-based factor reconstruction maintain consistency with Adam moments under sign or subspace changes? Please distinguish this proposed memory-saving strategy from the evaluated implementation and provide complete reproduction configurations.

7 comments
AuthorsOpenReview ↗

Rebuttal 1/4

Reviewer YDHU

Thank you so much for your review. We are encouraged by your positive assessment and appreciate your recognition of the well-supported motivation and practical value of both ISO applications.

We address your remaining concerns point by point below.

Q1. Statistical stability and efficiency

Q1.a. Additional training seed in the main 4B/16K setting.

…the two-seed experiment uses a smaller model and different configuration … Can the authors report independent training seeds and uncertainty estimates for the main 16K configuration, including accuracy and the 4B time-to-target result …?

Response: The earlier 1.7B/8K study provided an initial, lower-cost assessment of training variability. Within our compute budget, we have now added one additional independent training seed for both ISO-AdamW and tuned AdamW in the main Qwen3-4B/16K setting, using the same training/evaluation protocol and 400-update endpoint.

For time-to-target, we use 44.82, the final accuracy of the highest-accuracy AdamW run, as the common target:

MethodAccuracy (%) at update 400 (original; additional)Mean ± sample SDUpdates to targetHours to target
AdamW (2×10−62\times10^{-6})44.82; 44.5044.66 ± 0.23400363.5
ISO-AdamW (1×10−61\times10^{-6})46.98; 46.2446.61 ± 0.52260; 280239.6; 270.1

The AdamW timing entry refers to this fixed reference run; ISO timing entries follow the original/additional order.

ISO-AdamW outperforms tuned AdamW in both repetitions. Both ISO runs also reach the reference accuracy faster, using about 26–34% less training time than the fixed AdamW reference. We will update the paper's seed-variation section with these main-setting results.

Q1.b. Paired uncertainty for merging.

Paired uncertainty estimates for merging would also strengthen the reported gains.

Response: We added a paired bootstrap analysis comparing ISO-Merger with the strongest baseline in each submitted setting. Within each benchmark, questions are resampled jointly across methods, keeping all responses, evaluation seeds, and metrics from the same question together. We retain the paper's aggregate weights and use category-stratified resampling for the BFCL tool-use benchmark. We use 10,000 replicates and report percentile 95% confidence intervals:

Shared baseISO-MergerCompared baselineDifference (points)Paired 95% CI
Qwen2.5-7B-Instruct63.80RAM: 62.88+0.92[+0.22, +1.62]
DeepSeek-R1-Distill-Qwen-1.5B44.38TIES: 43.52+0.86[+0.38, +1.31]

The paired bootstrap analysis supports statistically significant accuracy improvements over the strongest compared baselines in both settings. We will include this analysis in the revised manuscript.

Q2. Closest baseline.

How does Pion compare after comparable learning-rate tuning on 4B in accuracy and wall-clock cost?

Response: ISO's accuracy and time-to-target advantages therefore remain after additional Pion tuning.

In the submitted paper, we used Pion's published RLVR learning rate of 1×10−61\times10^{-6} on Qwen3-4B-Base and swept its learning rate only on Qwen3-1.7B due to compute constraints. Following your suggestion, we have now extended the sweep to {1,2,3}×10−6\{1,2,3\}\times10^{-6} on 4B, using the official Pion implementation and the same training/evaluation settings as the main comparison. We report accuracy and time-to-target below:

Method (Qwen3-4B)Learning rateCompleted updates / statusAggregate accuracyHours to accuracy 44.18 (update)
Pion1×10−61\times10^{-6}400 / complete44.18 at update 400322.3 (400)
Pion2×10−62\times10^{-6}340 / stopped after collapsePeak 44.32 at update 240; 41.18 at 320; 2.67 at 340154.4 (240)
Pion3×10−63\times10^{-6}320 / stopped after collapsePeak 37.72 at update 160; 31.8 at 280; 22.80 at 320Not reached
ISO-AdamW1×10−61\times10^{-6}400 / complete46.98 at update 400107.7 (160)

Pion exhibits late-stage performance collapse at the higher learning rates. Even its highest reported score, 44.32 at update 240, remains below ISO-AdamW's 46.98 at update 400, a difference of 2.66 percentage points.

For timing, we use 44.18, the final accuracy of the best completed 400-update Pion run, as the common target. ISO reaches it in 107.7 hours, versus 154.4 hours for the fastest Pion run that reaches it, about 30% less training time.

We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal 2/4

Q3. Performance attribution.

Given the similar fixed- and trainable-spectrum results, are the gains primarily attributable to frame parameterization and optimizer dynamics rather than spectrum fixing? A budget-matched spectrum-projection or alternative-parameterization baseline would help isolate these effects.

Response: Following your suggestion, we add five ablation arms on Qwen3-4B-Base with a 16K response limit, using the same data, rollout and evaluation settings.

Ablation experiment design.

ArmMethodLearning rateSpectrumFrame parameterizationPolar retraction
AAdamW2×10−62\times10^{-6}FreeNoNo
BProjected AdamW2×10−62\times10^{-6}Restore$\Sigma_0$ after each weight updateNoNo
CISO-AdamW1×10−61\times10^{-6}Fixed$\Sigma_0$YesYes
DISO-AdamW with trainable$\Sigma$1×10−61\times10^{-6}TrainableYesYes
ED without retraction1×10−61\times10^{-6}Trainable diagonalYesNo

We organize the controls around three questions:

  • A/B/C: does fixing the spectrum in weight space reproduce ISO's frame-based optimization gains?
  • C/D: does learning Σ\Sigma improve ISO over reusing the source spectrum? This also tests whether Fig. 18's conclusion on source-spectrum reuse extends to 4B.
  • D/E: what is the incremental contribution of polar retraction?

Experimental results. We report accuracy for all five arms through latest step (210). Accuracy is average@16 (%) over the same five math benchmarks:

ArmUpdate 40Update 80Update 120Update 160Update 180Update 200Update 210
A: AdamW30.733.835.337.541.142.242.6
B: Projected AdamW29.833.335.139.240.841.041.9
C: Fixed-spectrum ISO31.235.941.444.343.843.144.0
D: Trainable-spectrum ISO30.937.441.344.344.043.644.3
E: D without retraction32.642.142.743.243.042.841.9

Ablation results:

(1) Adding spectrum projection to weight-space AdamW does not reproduce ISO's gains (A/B/C). At update 210, ordinary and projected AdamW score 42.6 and 41.9, while fixed-spectrum ISO reaches 44.0, 2.1 points higher than projected AdamW. This supports the contribution of frame-based optimization beyond enforcing the same spectral constraint.

(2) Fixed-spectrum training remains effective at 4B scale (C/D), consistent with Fig. 18. As in Fig. 18, the comparable performance (44.0 versus 44.3 at update 210) provides further evidence for reusing the source spectrum during optimization.

(3) Retraction helps stabilize late-stage frame optimization (D/E). Without retraction, frame orthonormality is no longer enforced after each update. E learns well initially, but its actor gradient norm rises rapidly after approximately 190 updates, reaching 0.23 / 0.55 (mean / maximum) in the 10-update window labeled 210, versus 0.04 / 0.05 for both D and ordinary AdamW—a roughly 5–6-fold increase in the mean. This is accompanied by an accuracy decline from 43.2 at update 160 to 41.9 at update 210, where D reaches 44.3, a 2.4-point advantage. Together, these observations support a stabilizing role for polar retraction within ISO's frame-coordinate optimization.

Each row reports the mean and maximum over a 10-update window, identified by its update label.

Grad-norm: mean over each 10-update window
(bar: 1 block = 0.02); numbers are mean / max

Update  E: no retraction              │ D: with retraction       │ A: AdamW
        bar             mean / max    │ bar        mean / max    │ bar        mean / max
120     ██             0.04 / 0.05    │ ██        0.05 / 0.05    │ ███       0.07 / 0.07
130     ██             0.05 / 0.05    │ ██        0.04 / 0.05    │ ███       0.07 / 0.10
140     ██             0.04 / 0.05    │ ██        0.05 / 0.05    │ ███       0.07 / 0.10
150     ██             0.04 / 0.06    │ ██        0.04 / 0.05    │ ████      0.07 / 0.16
160     ██             0.05 / 0.06    │ ██        0.04 / 0.05    │ ███       0.06 / 0.07
170     ███            0.06 / 0.11    │ ██        0.04 / 0.05    │ ███       0.05 / 0.08
180     ████           0.07 / 0.14    │ ██        0.04 / 0.05    │ ██        0.05 / 0.06
190     ███            0.07 / 0.09    │ ██        0.04 / 0.05    │ ██        0.04 / 0.05
200     ██████         0.12 / 0.27    │ ██        0.05 / 0.06    │ ██        0.04 / 0.05
210     ███████████    0.23 / 0.55    │ ██        0.04 / 0.05    │ ██        0.04 / 0.05

Interpretation and relation to our claim. Our functional finding of spectral inheritance provides the empirical basis for optimizing the singular frames while inheriting the source spectrum. The above controls complement our ablation in Fig. 18 andprovide further support for the frame-coordinate update procedure and polar retraction, consistent with our in Section 4.2: “The factor parameterization and optimizer-state dynamics are both part of the method” (Lines 320–323).

AuthorsOpenReview ↗

Rebuttal 3/4

Q4. Merging robustness.

How were ρ_keep=0.9, clipping, and ridge parameters selected, and were evaluation labels involved? Please clarify baseline tuning, parameter sensitivity, and numerical robustness to degenerate singular values and SVD implementations.

Response: The submitted comparisons use official baseline implementations with the configurations specified in Appendix G.2. ISO-Merger uses FP64 SVD and jointly aligns each expert’s singular-vector pairs to the shared base (details in Appendix F.2).

We now add parameter-sensitivity results for both merging settings.

Parameter choices. We use the same ρkeep=0.9\rho_{\mathrm{keep}}=0.9, λridge=10−12\lambda_{\mathrm{ridge}}=10^{-12}, and coefficient bounds [0,1.5][0,1.5] across both model families and all benchmarks. These shared defaults were not tuned separately for individual models or datasets based on their respective evaluation feedback. Mode masking is motivated by our empirical finding that trailing frame modes can introduce interference (Appendix F.4), with 0.9 used as the shared default with no dedicated tuning. The ridge term serves as a numerical stabilizer. Clipping keeps coefficients nonnegative and limits amplification.

Parameter sensitivity. Following your suggestion, we vary ρkeep\rho_{\mathrm{keep}} and cmax⁡c_{\max} separately, while holding the numerical stabilizer λridge=10−12\lambda_{\mathrm{ridge}}=10^{-12} fixed. Each entry reports variant ISO-Merger − default ISO-Merger in aggregate accuracy points, with a paired 95% CI from matched evaluations.

(1) ρkeep\rho_{\mathrm{keep}}. Performance shows limited sensitivity to the tested values near the default: at 0.85 and 0.95, differences from 0.9 are at most 0.33 points in magnitude, and all paired intervals include zero. Removing masking has a model-dependent effect, lowering accuracy by 2.36 points on DeepSeek and 0.33 points on Qwen.

Shared baseρkeep=0.85\rho_{\mathrm{keep}}=0.85ρkeep=0.9\rho_{\mathrm{keep}}=0.9 (default)ρkeep=0.95\rho_{\mathrm{keep}}=0.95ρkeep=1.0\rho_{\mathrm{keep}}=1.0 (no masking)
DeepSeek-R1-Distill-Qwen-1.5B+0.01 [−0.36, +0.39]Reference−0.33 [−0.73, +0.06]−2.36 [−3.08, −1.65]
Qwen2.5-7B-Instruct−0.21 [−0.59, +0.18]Reference−0.31 [−0.72, +0.07]−0.33 [−0.78, +0.12]

(2) cmax⁡c_{\max}. Changing the upper clipping bound produces observed differences below 0.3 points, and all paired intervals include zero. We detect no statistically significant accuracy change over these tested bounds.

Shared basecmax⁡=1c_{\max}=1cmax⁡=1.5c_{\max}=1.5 (default)cmax⁡=2c_{\max}=2cmax⁡=∞c_{\max}=\infty (no upper clipping)
DeepSeek-R1-Distill-Qwen-1.5B−0.07 [−0.43, +0.30]Reference+0.01 [−0.34, +0.38]+0.05 [−0.33, +0.43]
Qwen2.5-7B-Instruct−0.28 [−0.70, +0.15]Reference−0.13 [−0.51, +0.24]−0.07 [−0.46, +0.30]

Overall, ISO-Merger is robust to nearby parameter changes: the tested variations in keep ratio and upper clipping bound change accuracy by at most 0.33 points, small relative to its reported gains over the strongest baselines—0.86 points on DeepSeek and 0.92 points on Qwen.

We will include these sensitivity results in the revised manuscript.

Reviewer_m2yDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper analyzes RLVR weight updates through the SVD and finds that the base model's singular values can be reused while both singular frames change ("spectral inheritance"). Based on this, it proposes ISO with two methods: ISO-Merger, a data-free method that merges RL specialists from the same base by combining their frame changes under the base spectrum, and ISO-Optimizer, which applies AdamW or Muon to the singular frames with the base spectrum fixed. ISO-Merger achieves the best aggregate score among data-free merging baselines in two settings, and ISO-Optimizer improves math and coding accuracy over tuned AdamW and Muon, reaching AdamW's final accuracy on Qwen3-4B-Base in fewer updates and less wall-clock time.

Critical strengths
  1. The paper provides a detailed analysis, both formal and empirical, of why the spectrum stays stable under RLVR and can be reused, and derives two methods from it, each of which is useful on its own.
  2. The experiments are thorough and had been done under massive of computes. For example, ISO-Optimizer is applied to different base optimizers (AdamW and Muon), and the baselines are compared under learning-rate sweeps.
Critical weaknesses
  1. For ISO-Merger, the paper does not ablate the fixed spectrum in the final method, i.e., the same frame composition with a non-base spectrum. The spectrum is ablated for TIES, but not for ISO-Merger itself, so the necessity and benefit of using the base spectrum in the merger remain unclear.
  2. For ISO-Optimizer, it adds per-step training cost. However, as the authors report, it reaches the target accuracy in fewer steps, which also lowers the total training cost, so this does not affect my assessment.
Questions for authors
  1. Could you report an ISO-Merger variant that keeps the same frame composition but uses a non-base spectrum (e.g., the average of the experts' spectra)?
  2. The main optimizer comparisons use a single run per configuration, and the seed study uses a separate 8K setup with AdamW at 1e-6, which is not its best learning rate. Could you report seed variation for the main Qwen3-4B comparison?
  3. How does the FP64 SVD retraction overhead scale with model size and shorter rollouts? Response lengths of the final models at evaluation time would also be useful.
6 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Reviewer m2yD

Thank you for your positive assessment and for recognizing the formal and empirical analysis behind spectral inheritance and the usefulness of both ISO methods. And yes, these experiments do burn a lot of compute :(

We respond to the remaining concerns below.

Q1. Spectrum choice in ISO-Merger.

…the necessity and benefit of using the base spectrum in the merger remain unclear. Could you report an ISO-Merger variant that keeps the same frame composition but uses a non-base spectrum (e.g., the average of the experts' spectra)?

Response: Thank you for suggesting this direct control! ISO-Merger follows the principle “reuse the common spectrum and compose the learned frame changes,” motivated by RLVR's small spectral changes and the functional reusability of the base spectrum (Section 2). Since the base and expert-mean spectra are close, we expected similar performance.

We test your suggested alternative by keeping the entire frame-composition pipeline fixed, including the mode masks, retention coefficients, and final polar-retracted frames. Only the reconstruction spectrum changes, to the rank-wise mean of the experts' spectra:

Shared baseAggregate accuracy difference (points): ISO-Merger w/ expert-mean spectrum − ISO-Merger w/ base spectrum (paired 95% CI)
DeepSeek-R1-Distill-Qwen-1.5B−0.15 [−0.56, +0.25]
Qwen2.5-7B-Instruct−0.07 [−0.39, +0.25]

Both variants were re-evaluated together; confidence intervals use paired item-level bootstrap.

The expert-mean spectrum provides no statistically detectable accuracy improvement over the base spectrum in either setting. This supports using the shared source spectrum as a reusable reference for expert composition. It is also consistent with the existing TIES control: rebasing TIES onto the base spectrum does not improve its score (43.09 vs. 43.52, Table 9). Together, these controls support our principle of reusing the common spectrum and composing the learned frame changes. We will include this comparison in the revised manuscript.

Q2. Training-seed variation in the main 4B setting.

The main optimizer comparisons use a single run per configuration, and the seed study uses a separate 8K setup with AdamW at 1e-6, which is not its best learning rate. Could you report seed variation for the main Qwen3-4B comparison?

Response: Within our compute budget, we added one additional independent training seed for the ISO-AdamW and the reported best AdamW in the main Qwen3-4B/16K setting, giving two training runs per method. The table reports aggregate accuracy (%) at update 400:

MethodAccuracy at update 400 (original; additional)Mean ± sample SD
AdamW (2×10−62\times10^{-6})44.82; 44.5044.66 ± 0.23
ISO-AdamW (1×10−61\times10^{-6})46.98; 46.2446.61 ± 0.52

ISO-AdamW outperforms tuned AdamW in both repetitions, with a mean advantage of 1.95 percentage points. The earlier 1.7B/8K study served as an initial, lower-cost assessment of variability. We will update the paper's seed-variation section with these results from the main setting.

AuthorsOpenReview ↗

Rebuttal 2/2

Q3. Retraction overhead and evaluation response lengths.

How does the FP64 SVD retraction overhead scale with model size and shorter rollouts? Response lengths of the final models at evaluation time would also be useful.

Response:

Rollout length. For a fixed model and hardware setup, shorter rollouts change the relative overhead, not the FP64 retraction work. Retraction runs once per optimizer update on the factor matrices and does not process rollout tokens. Shorter rollouts reduce rollout-generation time, so the same matrix-update work accounts for a larger fraction of the end-to-end step.

We additionally report the full ISO optimizer's share of end-to-end step time across the main Qwen3-4B run (Section 5.2, Fig. 6). It accounts for only 0.9–4.1% throughout the run, with its share decreasing as mean response length grows from about 1K to 6.4K tokens:

UpdatesMean response length (tokens)Mean step timeOptimizer share
1–501,0261,195 s4.1%
51–1001,9591,857 s2.7%
101–2004,9674,196 s1.2%
201–3006,0114,949 s1.0%
301–4006,4425,238 s0.9%

Thus, the optimizer remains a small part of our end-to-end RL training cost, particularly as reasoning rollouts lengthen during learning and exploration, as observed in this run.

Model size. Retraction cost depends on the factor-matrix shapes. For q=min⁡(dout,din)q=\min(d_{\mathrm{out}},d_{\mathrm{in}}), the FP64 polar retractions of UU and VV require O((dout+din)q2)O((d_{\mathrm{out}}+d_{\mathrm{in}})q^2) work. We use a DION-style distributed implementation to parallelize matrix retractions across GPUs (Section 4.2).

The table compares the total retraction-work estimate with measured full ISO optimizer times on one node of 8 H100 GPUs. Timings include factor gradients, AdamW updates, FP64 polar retraction, weight reconstruction, and communication:

ModelRelative retraction workFull ISO optimizer step on 8 H100s
Qwen3-1.7B1.0×21.3 s
Qwen3-4B3.0×42.2 s
Qwen3-8B9.9×119.6 s

Although absolute optimizer cost grows with model size, rollout generation and actor computation also scale, and our training runs distribute matrix updates across multiple nodes. For Qwen3-4B on 16 H100s, the full optimizer takes 49 s/update, or 1.2% of end-to-end step time. For Qwen3-8B on 32 H100s, the full ISO optimizer step, measured in the same way as in Fig. 6, takes 88.9 s/update, or about 1% of its average RL step time (about 8,700 s). Its share is lower than for Qwen3-4B because the 8B RL step is substantially longer.

Response lengths at evaluation. The table summarizes the final update-400 checkpoints across the five math benchmarks, with 16 samples per problem and a 16K response limit. Accuracies are those reported in the paper.

ModelMethodMean tokensMedian tokensAccuracy (%)
Qwen3-1.7BAdamW3,7952,87628.20
Qwen3-1.7BMuon5,5084,78529.01
Qwen3-1.7BISO-AdamW4,6293,56329.41
Qwen3-4BAdamW4,0642,85244.82
Qwen3-4BMuon5,1524,19445.51
Qwen3-4BISO-AdamW5,5144,23446.98
Qwen3-4BISO-Muon5,9285,10746.2 (Appendix Fig. 17)

ISO-AdamW produces longer responses than AdamW at both scales, partly reflecting its faster learning and the accompanying earlier growth in response length during training, as observed in Fig. 6.

Across these final checkpoints, accuracy does not simply track mean response length. On 1.7B, Muon produces the longest responses but scores below ISO-AdamW (29.01 vs. 29.41). On 4B, Muon and ISO-AdamW average 5.2K and 5.5K tokens but differ by 1.47 accuracy points, while ISO-Muon produces the longest responses yet scores below ISO-AdamW.

We will include these measurements in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the rebuttal period ends tomorrow, please let us know if you have any remaining concerns. We would be grateful if you could revisit your assessment in light of our rebuttal and additional experiments. Thank you again for your review and consideration!

Authors of 27235

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.