acceptodds
Under review as a conference paper at ICLR 2027

: Spectral Null-Space Swap Makes Reasoning Models Efficient

14 views

Abstract

LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model’s weight component within the null space of a projection defined by the corresponding Non-thinking model’s dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose **Spectral Null-Space Swap** (), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate on 2B–30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across **mathematical, multimodal, and audio reasoning domains**. establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Reasoning, Weight Merge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3rUsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors take paired Non-thinking weights W0W_0 and Thinking weights WtW_t, and build a projector PP from singular directions of W0W_0. For the swap, the retained fraction is ρ\rho, and the main experiment uses 0.8. The merged weight is P(W0)+(I−P)(Wt)P(W_0)+(I-P)(W_t): the retained directions stay those of Non-thinking, and the remaining directions are taken from Thinking. The paper calls the remaining directions a null space. Setting ρ=1\rho=1 removes that free parameter and gives the probe family in Section 3.3: Base is W0W_0, Sub is W0+ΔW∥W_0+\Delta W_\parallel, Null is W0+ΔW⊥W_0+\Delta W_\perp, and Full is W0+ΔWW_0+\Delta W. The paper uses hidden-state shift relative to Instruct, together with attention entropy, to describe what thinking training changes. The experiments cover Qwen3 text, vision-language, and Omni checkpoints, with Instruct, Thinking, TIES, and an Instruct–Thinking interpolation at 0.8 as comparisons.

Critical strengths
  1. The merge requires no further training. The same formula is applied to dense, MoE, vision-language, and Qwen3-Omni checkpoints, and accuracy is reported together with generation length. The coordinate replacement can therefore be checked across these checkpoints.

  2. Figure 1 separates parameter energy from hidden-state shift on the same AIME inputs. A direction with large energy need not be a direction with large shift. This separation is the most informative measurement in the paper.

Critical weaknesses
  1. The decomposition ΔW=PΔW+(I−P)ΔW\Delta W=P\Delta W+(I-P)\Delta W holds for every pair of matrices and every retained rank kk. ker⁡(P)\ker(P) is nonempty because PP is a projector, not because W0W_0 or WtW_t has a special spectrum. A complement of this form therefore exists before any claim about where reasoning is located. The identity records the split. It does not select a split.

  2. Figure 1a is the Qwen3-4B split onto the full singular subspace of W0W_0. The aligned bar is about 38% of the Frobenius energy of ΔW\Delta W, and the complement is about 25%. The swap later keeps only the leading fraction ρ\rho of those directions, with ρ=0.8\rho=0.8 in the main tables. The 38% and 25% bars are not the energy split at ρ=0.8\rho=0.8. A measurement on the full subspace does not locate reasoning in the complement that the operating point actually imports.

  3. Table 3 is the probe family defined in Section 3.3 by setting ρ=1\rho=1: Base is W0W_0, Sub is W0+ΔW∥W_0+\Delta W_\parallel, Null is W0+ΔW⊥W_0+\Delta W_\perp, and Full is W0+ΔWW_0+\Delta W. On five math benchmarks the mean accuracies are 54.1, 58.6, 70.2, and 70.3, and the mean lengths are 8,280, 8,891, 15,071, and 20,269. The caption says Sub is almost identical to Base. The two means differ by 4.5 points. Null’s accuracy matches Full, and the mean length is shorter. Full equals Null plus ΔW∥\Delta W_\parallel, so the match compares the complement with a checkpoint that already includes that complement. The probe is the full subspace, on Qwen3-4B, and it is not the complement imported at ρ=0.8\rho=0.8.

  4. Section 3.1 defines this complement, for rectangular attention projections, as the null-space component of the projector onto the singular support of W0W_0. Under that definition the complement is ker⁡(P)\ker(P). Figure 1a assigns it about 25% of the energy of ΔW\Delta W, against about 38% for the aligned component. A component that holds 25% of the energy is not an empty kernel of WW, and it is not a singular tail of negligible energy. The word “null space,” as defined in that section, names ker⁡(P)\ker(P). It does not name a kernel of WW.

  5. The operator is Equation 5, Wρ=P(W0)+(I−P)(Wt)W_\rho=P(W_0)+(I-P)(W_t). It replaces the complement coordinates of W0W_0 with those of WtW_t. The replacement is defined for any pair and any ρ\rho. Table 2 reports ρ\rho from 1.0 down to 0.5. Each value is another projector and therefore another model. The model at one table entry is the model produced by that projector. The paper selects ρ=0.8\rho=0.8 as the accuracy–length trade-off among those entries. The identity remains true at every entry in the table.

  6. Functional change in Figure 1 is hidden-state RMS relative to Instruct, read from a forward pass after the corresponding weights have been written. Attention entropy is a second functional, read after the same write. Proposition 1 gives its first-order change under Assumption 1, which states that mean attention entropy is already locally minimized inside the row space of the query projection. Under that assumption the in-subspace gradient is zero, and the first-order term is the inner product with the complement. The assumption places the first-order effect in the complement before the expansion is evaluated. Corollary 1 uses the further statement that this inner product is negative, taken from Appendix A.5. Equation 1, the decomposition identity, does not mention either functional and does not select ρ\rho. Accuracy and token counts record the model after the write at the chosen ρ\rho. They do not select the functional or the rank.

Questions for authors
  1. Which condition decides that reasoning capacity is not in this complement, without using the chosen hidden-state RMS and without treating “(I−P)ΔW(I-P)\Delta W has already been written” as satisfying the condition?

  2. In Figure 1a the complement holds about 25% of the Frobenius energy and the aligned component about 38%. Which spectral object makes that complement a null space of WW, rather than ker⁡(P)\ker(P) as defined in Section 3.1?

  3. Figure 1 and Table 3 both use the full singular subspace of W0W_0. Section 3.3 obtains Table 3 by setting ρ=1\rho=1. The main tables use ρ=0.8\rho=0.8. Which result locates reasoning in the complement at ρ=0.8\rho=0.8, rather than in the full-subspace complement shown in Figure 1a and probed in Table 3?

  4. The decomposition identity holds for every ρ\rho reported in Table 2. Which observation would reject one of those values of ρ\rho and retain another, while the identity remains true for both?

  5. Proposition 1 assumes that attention entropy is already locally minimized in the row space of the query projection, and then the first-order change lies in the complement. Corollary 1 uses the sign ⟨ΔW⊥,G⊥⟩<0\langle\Delta W_\perp, G_\perp\rangle<0 from Appendix A.5. Which step of that argument, other than Assumption 1 and other than this observed sign, places the first-order change in the complement and fixes its direction?

  6. In Table 3, Sub mean accuracy is 58.6 and Base is 54.1, while Null is 70.2 and Full is 70.3. Full already contains ΔW⊥\Delta W_\perp. Which comparison shows that the in-subspace component is redundant, and which comparison matches Null to a model that does not already contain it?

  7. Every checkpoint is from the Qwen3 family: Qwen3-4B, Qwen3-30B-A3B, Qwen3-VL-2B, Qwen3-VL-4B, and Qwen3-Omni-30B-A3B. The evaluations are text, vision-language, and audio reasoning. Can the same swap be run on a model family other than Qwen3?

5 comments
AuthorsOpenReview ↗

Revised version updated

Dear Reviewer 3rUs,

We thank the reviewer again for the thoughtful review. We have further revised the manuscript to address your comments and strengthen the supporting evidence. The updated version includes additional experiments and clarifications, with the corresponding changes highlighted in different colors in the revised PDF. For convenience, we summarize the main updates below:

  • 2: We measured energy decomposition at different ρ\rho values in Appendix B.5, Table 14 and Figure 11. The corresponding modifications are highlighted in orange.
  • 3: We extended the probe family in ablation experiment beyond ρ=1\rho=1 in Appendix B.8 and Table 15. The corresponding modifications are highlighted in orange. We also clarified the ΔW∥\Delta W_\parallel provides minor general capability gains in Section 3.1 The corresponding modifications are highlighted in blue and orange.
  • 4: We provide more experimental evidence supporting our Assumption 1 in Section 5.2 and Appendix A.5, Figure 4 and Table 6. The corresponding modifications are highlighted in blue.
  • 6: We add additional experiment on non-Qwen model in Appendix B.7 and Table 17. The corresponding modifications are highlighted in red.

We hope the additional analyses and revisions satisfactorily address your concerns and strengthen the evidence supporting our approach. We welcome any further questions or suggestions and would be glad to provide additional clarification during the remaining discussion period. Thank you again for your thoughtful feedback and consideration.

Best Regards,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 3rUs,

Thank you for your insightful comments and the time devoted to reviewing our manuscript, which have been instrumental in improving our work.

We hope our responses and additional experiments have adequately addressed your concerns, and we mostly welcome any further feedback or requests for clarification.

Thank you again for your time and consideration.

Warm regards,

The Authors

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_WcMPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The core viewpoint of the paper is: In the weight changes generated by post thinking training, the parts that fall within the dominant spectral space of the Non linking model have limited functional impact despite having higher parameter energy; The main reasoning ability comes from the complementary part of the space.

Specifically, the author refers to the weight difference between Thinking and Non thinking as:

ΔW=Wt−W0\Delta W=W_t-W_0

Using the first \ (k=\ lceil \ rho \ min (m, n) \ rceil ) left and right singular vectors with Non linking weights, construct a projection \ (P_ {S \ rho} ), and finally obtain:

Wρ=PSρ(W0)+(I−PSρ)(Wt)=W0+(I−PSρ)ΔWW_\rho=P_{S_\rho}(W_0)+(I-P_{S_\rho})(W_t) =W_0+(I-P_{S_\rho})\Delta W

Among them, \ (\ rho ) controls how many Non linking spectral structures are retained. The author also constructed four models, Base, Sub, null, and Full, to analyze the impact of these two components on accuracy, output length, and attention entropy.

Critical strengths

The research question is important and the approach is concise. Without modifying the decoding algorithm or retraining, only using checkpoint combinations to reduce the lengthy reasoning of the Thinking model has practical appeal.

The perspective of spectral space has some novelty. Compared to global linear interpolation or general model merging, S ³ uses asymmetric sources for different spectral regions: Non linking for dominant regions and Thinking for complementary regions.

The experimental coverage is relatively wide. It includes tasks from 2B to 30B, Dense and MoE, text, visual language, and audio, with a total of five Qwen3 checkpoints and 28 main evaluation environments.

Critical weaknesses
  1. The naming of 'Invalid space' is not entirely consistent with the actual operator This is the mathematical problem that needs clarification the most. When \ (\ rho<1 )\ (I-P_ {S \ rho}) \ Delta W \ is not only the strict zero space component of \ (W0 ), but also includes truncated non-zero singular directions and possible cross directions. Therefore, the S ³ -0.8 used in the main experiment is more accurately referred to as the "orthogonal complement of the dominant spectral subspace" rather than a strictly null space. In addition, the definition of the paper
PS(X)=U(U⊤XV)V⊤P_S(X)=U(U^\top XV)V^\top

It projects onto \ (\ operationame {span} (U) \ otimes \ operationame {span} (V) ), which includes the cross terms of \ (u_i vvej ^ \ top ), not just the commonly referred paired singular directions \ (\ {u_i vv_i ^ \ top } ).

  1. Lack of key implementation details The paper does not explicitly list which weights S ³ should be applied to: -Have all Q/K/V/O projections been processed; -Is the gate/up/down projection of MLP processed; -Does each expert of MoE perform SVD separately;
  • embedding、LM head、 How to handle normalization parameters; -Does the multimodal projector and audio module participate in the merge. At the same time, there was no report on the time, peak memory, numerical accuracy, and checkpoint generation cost of the 30B MoE model executing all SVDs. 'Training free' does not mean 'construction cost can be ignored'.
Questions for authors
  1. Compared with the original Thinking model at the same maximum generation length, does S ³ still maintain its advantage?
  2. Is attention entropy measured on the generated trajectories of each model itself, or on the exact same token sequence?
  3. Does the result hold true on non Qwen models?
4 comments
AuthorsOpenReview ↗

Revised version updated

Dear Reviewer WcMP,

We appreciate your detailed comments, which have helped us improve the clarity and completeness of the manuscript. We have prepared an updated version incorporating the revisions and supplementary experiments presented in our rebuttal. For ease of reference, all modifications are highlighted in red in the revised PDF. A summary of the corresponding changes is provided below:

  • 1.1: We correct the term "null-space" to "spectral complement" when ρ\rho not equal to 1 in Section 4 and Conclusion. We emphasize that we utilize the probe family (ρ=1\rho=1 case) to make a mechanistic analysis in Section 5 so that we still call it null-space in this section.
  • 1.2: We explicitly identified the tensor-product subspace and its cross terms in Section 3.2.
  • 2.1: We specified which parameters participate in the swap under the paragraph titled “Parameters included in the swap” in Appendix B.1.
  • 2.2 We report the one-time construction cost and the numerical preservation of the models under the paragraph titled “Checkpoint Construction Cost and Numerical Preservation”, Table 7 and Table 8 in Appendix B.1.
  • 3.1 and 3.2: We clarifid the matched budgets in the original evaluation in Section 4.1 and extended the comparison to smaller matched budgets in Appendix B.4 and Table 13.
  • 4: We described the measurement of attention entropy in Appendix A.2 and experiment on reasoning trajectory in Appendix B.6 and Table 16.
  • 5: Evaluation beyond the Qwen family is in Appendix B.7 and Table 17.

We hope these revisions adequately address your concerns and improve the clarity and completeness of our work. We would be happy to answer any further questions or provide additional clarification during the remaining discussion period. Thank you again for your valuable feedback and consideration.

Best Regards,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer WcMP,

We sincerely appreciate your thoughtful feedback and the time you have invested in evaluating our work. Your comments have helped us further strengthen the manuscript and clarify several aspects of our approach.

We hope that the revisions and additional experiments have satisfactorily addressed your concerns. We would be grateful for any further thoughts or suggestions you may have, and would be glad to provide additional clarification if necessary.

Thank you again for your careful consideration and constructive feedback.

Best regards,

The Authors

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

AuthorsOpenReview ↗

Sincere Appreciation for Your Thoughtful Review and Contributions

Dear Reviewer WcMP,

We sincerely appreciate your thoughtful consideration of our rebuttal and your continued engagement with our work. We are particularly grateful for your recognition of the merits of our approach and its potential contribution to the broader research community.

Your thoughtful comments have been invaluable in improving the quality and clarity of our manuscript. We are especially grateful for your dedication to maintaining high academic standards and your valuable contributions to the research community. The time, care, and expertise you devote to the review process are deeply appreciated.

Thank you again for your valuable time and consideration.

Best regards,

The Authors

Reviewer_Dj5tOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at ways to cut down on the cost of generated tokens while maintaining reasoning ability, by merging paired Non-thinking and Thinking models. The authors introduce S³. S³ establishes a protected subspace based on the dominant singular directions of the Non-thinking model weights. It keeps the Non-thinking weights within this subspace and utilizes the Thinking weights in the complementary space. Notably, it requires no extra training and doesn’t change the decoding process. The authors assess this method across text, vision-language, and audio reasoning tasks using Qwen models ranging from 2B to 30B. They also compare their approach with TIES, MI-0.8, and other strategies. Additionally, they explore how their method operates through a rho sweep, component ablation studies, and an analysis of attention entropy (§3 to §5, Tables 1 to 3, 8, and 9).

Critical strengths

The method is straightforward, and its benefits in generation cost are supported by fairly consistent empirical evidence. Since S³ doesn’t require further training or any adjustments to the decoding process, it's quite practical. When there are paired checkpoints available, the weights can be directly combined. I equally weighted the 28 model-task combinations presented in Tables 1 and 8. All combinations result in fewer tokens than those produced by the corresponding Thinking models. On average, there's about a 27.4% relative reduction in token generation compared with these Thinking models. Additionally, accuracy point estimates improve by roughly one percentage point on average. These findings span different model scales along with text, vision-language, and audio tasks. Overall, they highlight S³’s practical value as a method that enhances inference efficiency without needing training.

Critical weaknesses

W1: The scope of the claim that the aligned component is "largely redundant" still needs more clarity. In §4.3, there's already a discussion about the trade-off between accuracy and generation costs when rho is lowered. Looking at Tables 2 and 9 shows that decreasing rho from 1 to 0.8 brings back some of the aligned updates on top of the Null model. For the two text models analyzed, all eight accuracy point estimates go up. Generated tokens increase in seven of these eight scenarios too. While this doesn’t contradict the idea that the complementary space maintains key reasoning advantages, it does suggest that some aligned directions still hold importance. The paper should explain which component combinations and evaluation scenarios are referred to by "largely redundant" in §3.1 more clearly. It should separate two assertions: first, that “adding the full aligned component to Null gives limited average gain,” and second that “some of its directions are still effective.” This would provide a more precise understanding of spectral components' roles.

W2: The analysis of attention entropy isn't quite enough to thoroughly clarify why generation costs decrease. In §5.2, there's an examination of how spectral components affect attention entropy under certain assumptions like local optimality. The authors do recognize this as a conditional explanation; however, Appendix A.4 indicates that Pythia's entropy stabilizes during training, which only offers indirect support and doesn't directly establish local optimality within the active subspace for the actual Qwen models. More crucially, Equations (11) to (14) only look at changes in entropy without establishing a connection between entropy levels and generation length itself. Therefore, while current evidence mainly supports an explanatory hypothesis, it doesn't yet fully establish the mechanism that “complementary-space updates reduce generated tokens by lowering attention entropy.” Thus, claims made in the abstract regarding this mechanism should be narrowed accordingly.

Questions for authors

[Clarification] What specific component combinations and evaluation conditions relate to “largely redundant”? How does this pertain to gains from partial aligned updates during the rho sweep? Also, can you describe the model and evaluation protocol used for Table 3? This would help clarify the full comparison of components.

[Clarification] Based on your current results, what evidence is there that actual Qwen models satisfy the assumption of local optimality within active subspaces? What evidence suggests that reduced attention entropy leads to fewer generated tokens? If current results mainly support a conditional explanatory hypothesis, could you narrow your mechanism claim in the abstract accordingly?

8 comments
AuthorsOpenReview ↗

Revised version updated

Dear Reviewer Dj5t,

Thank you for your constructive feedback and for the opportunity to further improve our work. We have uploaded a revised manuscript incorporating the changes and additional experiments described in our rebuttal. All revisions addressing the concerns raised in your review are highlighted in blue in the revised PDF. The corresponding changes are located as follows:

  • 1.1 and 1.2: We clarify that “leading” specifies the top-singular-vector spans used to construct the matrix subspace and explain why partial aligned updates can improve accuracy during the ρ\rho sweep. Please see Section 3.2.
  • 1.3: We add evaluation protocol of the experiment inTable 3 under the paragraph titled “Evaluation protocol” and paragraph titled “Generation settings” and Table 10 in Appendix B.1.
  • 1.4: We revised the scope of the redundancy claim that " the in-subspace component contributes little functional change on its own" in Section 3.1.
  • 2.1: We did experiments on Qwen models and add results to Section 5.2, Appendix A.5, Figure 4 and Table 6.
  • 2.3: We narrow our claim in Abstract, Section 5 and Conclusion.

We hope the revised manuscript adequately addresses the issues you raised. We would be happy to provide any further clarification that may be helpful during the remaining discussion period. Thank you again for your time and consideration.

Best Regards,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer Dj5t,

We appreciate your insightful review comments, which have been instrumental in improving our manuscript.

Please let us know if we have adequately addressed your concerns. Your further comments are most welcome. Thank you again for your invaluable contribution to our research.

Thank you again for your time and consideration.

Warm regards,

The Authors

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.