- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors take paired Non-thinking weights and Thinking weights , and build a projector from singular directions of . For the swap, the retained fraction is , and the main experiment uses 0.8. The merged weight is : the retained directions stay those of Non-thinking, and the remaining directions are taken from Thinking. The paper calls the remaining directions a null space. Setting removes that free parameter and gives the probe family in Section 3.3: Base is , Sub is , Null is , and Full is . The paper uses hidden-state shift relative to Instruct, together with attention entropy, to describe what thinking training changes. The experiments cover Qwen3 text, vision-language, and Omni checkpoints, with Instruct, Thinking, TIES, and an Instruct–Thinking interpolation at 0.8 as comparisons.
Critical strengths
-
The merge requires no further training. The same formula is applied to dense, MoE, vision-language, and Qwen3-Omni checkpoints, and accuracy is reported together with generation length. The coordinate replacement can therefore be checked across these checkpoints.
-
Figure 1 separates parameter energy from hidden-state shift on the same AIME inputs. A direction with large energy need not be a direction with large shift. This separation is the most informative measurement in the paper.
Critical weaknesses
-
The decomposition holds for every pair of matrices and every retained rank . is nonempty because is a projector, not because or has a special spectrum. A complement of this form therefore exists before any claim about where reasoning is located. The identity records the split. It does not select a split.
-
Figure 1a is the Qwen3-4B split onto the full singular subspace of . The aligned bar is about 38% of the Frobenius energy of , and the complement is about 25%. The swap later keeps only the leading fraction of those directions, with in the main tables. The 38% and 25% bars are not the energy split at . A measurement on the full subspace does not locate reasoning in the complement that the operating point actually imports.
-
Table 3 is the probe family defined in Section 3.3 by setting : Base is , Sub is , Null is , and Full is . On five math benchmarks the mean accuracies are 54.1, 58.6, 70.2, and 70.3, and the mean lengths are 8,280, 8,891, 15,071, and 20,269. The caption says Sub is almost identical to Base. The two means differ by 4.5 points. Null’s accuracy matches Full, and the mean length is shorter. Full equals Null plus , so the match compares the complement with a checkpoint that already includes that complement. The probe is the full subspace, on Qwen3-4B, and it is not the complement imported at .
-
Section 3.1 defines this complement, for rectangular attention projections, as the null-space component of the projector onto the singular support of . Under that definition the complement is . Figure 1a assigns it about 25% of the energy of , against about 38% for the aligned component. A component that holds 25% of the energy is not an empty kernel of , and it is not a singular tail of negligible energy. The word “null space,” as defined in that section, names . It does not name a kernel of .
-
The operator is Equation 5, . It replaces the complement coordinates of with those of . The replacement is defined for any pair and any . Table 2 reports from 1.0 down to 0.5. Each value is another projector and therefore another model. The model at one table entry is the model produced by that projector. The paper selects as the accuracy–length trade-off among those entries. The identity remains true at every entry in the table.
-
Functional change in Figure 1 is hidden-state RMS relative to Instruct, read from a forward pass after the corresponding weights have been written. Attention entropy is a second functional, read after the same write. Proposition 1 gives its first-order change under Assumption 1, which states that mean attention entropy is already locally minimized inside the row space of the query projection. Under that assumption the in-subspace gradient is zero, and the first-order term is the inner product with the complement. The assumption places the first-order effect in the complement before the expansion is evaluated. Corollary 1 uses the further statement that this inner product is negative, taken from Appendix A.5. Equation 1, the decomposition identity, does not mention either functional and does not select . Accuracy and token counts record the model after the write at the chosen . They do not select the functional or the rank.
Questions for authors
-
Which condition decides that reasoning capacity is not in this complement, without using the chosen hidden-state RMS and without treating “ has already been written” as satisfying the condition?
-
In Figure 1a the complement holds about 25% of the Frobenius energy and the aligned component about 38%. Which spectral object makes that complement a null space of , rather than as defined in Section 3.1?
-
Figure 1 and Table 3 both use the full singular subspace of . Section 3.3 obtains Table 3 by setting . The main tables use . Which result locates reasoning in the complement at , rather than in the full-subspace complement shown in Figure 1a and probed in Table 3?
-
The decomposition identity holds for every reported in Table 2. Which observation would reject one of those values of and retain another, while the identity remains true for both?
-
Proposition 1 assumes that attention entropy is already locally minimized in the row space of the query projection, and then the first-order change lies in the complement. Corollary 1 uses the sign from Appendix A.5. Which step of that argument, other than Assumption 1 and other than this observed sign, places the first-order change in the complement and fixes its direction?
-
In Table 3, Sub mean accuracy is 58.6 and Base is 54.1, while Null is 70.2 and Full is 70.3. Full already contains . Which comparison shows that the in-subspace component is redundant, and which comparison matches Null to a model that does not already contain it?
-
Every checkpoint is from the Qwen3 family: Qwen3-4B, Qwen3-30B-A3B, Qwen3-VL-2B, Qwen3-VL-4B, and Qwen3-Omni-30B-A3B. The evaluations are text, vision-language, and audio reasoning. Can the same swap be run on a model family other than Qwen3?
5 comments
Gentle Reminder
Dear Reviewer 3rUs,
Thank you for your insightful comments and the time devoted to reviewing our manuscript, which have been instrumental in improving our work.
We hope our responses and additional experiments have adequately addressed your concerns, and we mostly welcome any further feedback or requests for clarification.
Thank you again for your time and consideration.
Warm regards,
The Authors
Dear Reviewer,
As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.
If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.
Thank you again for your time and valuable feedback.
Revised version updated
Dear Reviewer 3rUs,
We thank the reviewer again for the thoughtful review. We have further revised the manuscript to address your comments and strengthen the supporting evidence. The updated version includes additional experiments and clarifications, with the corresponding changes highlighted in different colors in the revised PDF. For convenience, we summarize the main updates below:
We hope the additional analyses and revisions satisfactorily address your concerns and strengthen the evidence supporting our approach. We welcome any further questions or suggestions and would be glad to provide additional clarification during the remaining discussion period. Thank you again for your thoughtful feedback and consideration.
Best Regards,
Authors