acceptodds
Under review as a conference paper at ICLR 2027

Spectral Rewiring for Exploration, Purification, and Model Merging

Abstract

Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, Reinforcement learning, Model merging, Large language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XqUPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAR, a post-hoc method that compresses RL updates and projects them onto the reference model's dominants singular subspaces. Experiments show largely preserved reasoning accuracy, improved Pass@k, and benefits for mixed-domain training and experts model merging.

Critical strengths
  1. The paper is well presented. The motivation, method, and experiments are clearly explained, making the contribution easy to follow

  2. The idea of projecting RL updates onto the reference model’s leading singular subspaces is reasonable. Table 1 supports this intuition, showing that compact updates preserve post-training performance across different model families and scales.

Critical weaknesses
  1. Mathematical evaluation is limited to AIME2024 and AIME2025. Testing on additional benchmarks would strengthen the paper.
Questions for authors
  1. Could you evaluate SAR on HMMT February 2025 [1]?

[1] Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. arXiv:2605.00674, 2026.

Reviewer_4hv8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studies the weight update in outcome-based reward RL from geometrical perspective, specifically their method SAR decomposes the weight update into reasoning effective and residual components, and show that the reasoning effective component of Full RL update can be recovered from the pre-existing spectral subspace of base model. They apply their method on attention and MLP linear layers of the model. Empirically they show that they can use this method as a post-hoc editing method retaining same reasoning performance as full RL model while improving high-k exploration, reducing domain inference in multi-domain setting and better expert model merging.

Critical strengths
  • Proposed SAR is a simple post-hoc editing method that can extract reasoning component which improves pass-k exploration at high K, offers less domain interference and good marging, while preserving full RL model performance
  • Good mechanistic analysis tested with Multiple scale of models from 1.5B to 32B
Critical weaknesses
  • This work missed some important prior works around RL Geometric and Update Nature (specifically for RLVR), prior works like [1] or [2] , already established that the RL updates can be sparse in nature and RL refines the representation and preserves the geometry of base model. From my point to view, authors does not distinguish the spectral alignment contribution from the prior work.
  • In the experimental section all main claims are supported with benchmarks like AIME/LiveCodeBench/IFEval, some of the benchmarks are inherently sensitive because they have small problem sets, so standard deviation would be useful here and as evident in table 1/2/3 the projected performance is not much different than the other methods. A broader evaluation suits with repeated evaluations with standard deviation reported will be much helpful here to understand the gains.
  • I am not fully convinced about mechanistic explanation that the retained component is reasoning effective one or the other component produce cross domain interference issue, for example the LCB improvement in section 4.1 is modest compared to no projection control given that no standard deviation was reported , so I feel the claims about reasoning component and interference requires more stronger evidence.

[1] Mukherjee, Sagnik et al. “Reinforcement Learning Finetunes Small Subnetworks in Large Language Models.” ArXiv abs/2505.11711 (2025): n. pag. [2] Runwal, Bharat et al. “PRISM: Demystifying Retention and Interaction in Mid-Training.” ArXiv abs/2603.17074 (2026): n. pag.

Questions for authors
  • Other than the Compact Reasoning direction of RL updates in outcome-based Reward RL setting, how does the SAR compare to existing sparse update and task vector approaches?
  • Can authors provide more broader suits of benchmarks for the SAR edited model compared with its base counterparts like HumanEval, GPQA, MMLU/MMLU-Pro.?

I am happy to increase the score if authors address my concerns.

Reviewer_kPSdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In my understanding, this paper does the following. 1) after an entire RL training, the authors compares the weight changes from the base model to the trained model, and compress that with SVD. 2) The authors then apply SVD to the base model’s original weights to identify their top-k singular directions. 3) The authors update the base model weights with those important directions in the weight deltas, that are also part of the original weights' important directions, with the top-k being set to roughly 1% of each matrix rank. With such a setup, the authors find that this process recovers most of the RL training performances, while retaining a small part of the update. This benefits downstream applications such as model merging, which avoids conflicts and noises in plain RL updates, as demonstrated by the recovered coding performances that were originally surpressed by mixed domain training.

Critical strengths

The proposed method seems to be a simple operation done on trained RL models that improves the performances when the training needs to consider multiple domains. The empirical findings that only updating the base model's important directions is interesting and may benefit future work.

Critical weaknesses

While the empirical finding on the base model alignment is interesting, the overall technique of merging and updating with compressed weights or gradients is widely adopted, to my limited knowledge. The gains in the paper are relatively minor and may not have general applicability in wider scenarios. I would suggest that the authors do a broader survey of other methods for mixed-domain training and merging in RL and compare with more baselines.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.