- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Selective-RL, a training-free method for transferring reasoning updates from an RL-trained LLM to a VLM. The method first separates the RL-stage update from parameter differences that existed before RL. For each attention and MLP weight matrix, it keeps the leading rank-one component of the RL update and rescales it to preserve the original Frobenius norm. The selected update is then added to the VLM either directly or together with the pre-RL difference between the two models. Experiments on three VLM backbones and five visual reasoning benchmarks show that Selective-RL usually outperforms full-update transfer and several existing merging methods. Further ablation studies show that its gains come from selecting the leading direction, rather than simply using a low-rank update or changing the update scale.
Critical strengths
- The paper identifies an important issue in previous transfer methods: the difference between the donor and recipient mixes the RL-stage update with parameter differences that already existed before RL. The proposed decomposition provides a simple and clear way to separate these two parts.
- The experiments carefully examine the effects of update direction, update scale, retained rank, composition method, and checkpoint choice. The comparisons with random, trailing, unscaled, and full-update directions make the main result convincing.
- Selective-RL shows consistent improvements across several VLM backbones and visual reasoning benchmarks. The paper also compares the method with a broad set of recent model-merging approaches.
- Selective-RL requires no recipient-side training, additional inference module, or multimodal reasoning data. It is simple to apply when the exact pre-RL checkpoint is available and adds little construction cost.
Critical weaknesses
- The methodological novelty is limited. The related methods solve different problems, but several parts of Selective-RL already appear in them. TSV and STAR use SVD-based filtering to reduce interference in multi-task model merging. STAR restores the nuclear norm after truncation, and Cai et al. (2026) rescale a rank-one RL update to match the norm of the full update. IP-Merging and FRISM already transfer reasoning from LLMs to VLMs using selected parameter subspaces. Selective-RL brings these ideas together by applying norm-matched rank-one selection to the isolated RL-stage update. This is a clean and effective combination, but its novelty mainly comes from the combination and the new setting, rather than from a new selection or scaling method.
- The method requires the exact pre-RL checkpoint, which may be unavailable for many released models, so the practical scope of the method is limited. The performance drop with an earlier reference checkpoint (Appendix Table 15) also suggests that the method is sensitive to this requirement.
- There is limited theoretical explanation of the Rank-1 selection and Frobenius-norm rescaling designs. Both are defined in parameter space, but it is unclear why they are effective for model outputs.
Questions for authors
- What is the “earlier reference checkpoint” used in Appendix Table 15?
- Why is the leading singular direction expected to transfer better than other directions? Is there any functional or output-level evidence connecting this parameter-space direction to reasoning transfer?
6 comments
Response to Reviewer nhEz — Part 2/3
We kept the recipient , donor endpoint , transferable parameter support , and initialization displacement fixed. Only the reference used to extract the transferred update changed:
All variants used . For the selected variant, we applied leading rank-one selection to each eligible matrix of and rescaled it to the Frobenius norm of that same matrix in . Norm matching is performed within each reference construction; the update norms are not matched across the two references.
On the transferable support, the Full control was therefore
=V+\lambda[(B-V)+(R-C_{10})] =W_{\mathrm{Full}}-\lambda(C_{10}-B).$$ The two Full rows represent different models: the intermediate-reference construction omits the net donor update accumulated over the first 10 RL steps. Their scores can consequently differ. If both the initialization displacement and the extracted update were recomputed using $C_{10}$, the reference would cancel and Full interpolation would remain unchanged. Parameters outside $S$ remain those of the recipient. | Update reference | Transferred update | MathVista | MathVerse | DynaMath | MMStar | |---|---|---:|---:|---:|---:| | Actual pre-RL initialization $B$ | Full | 76.70 | 45.30 | 66.07 | 55.93 | | Actual pre-RL initialization $B$ | Selected rank-one | 81.00 | 52.00 | 70.06 | 59.67 | | Intermediate RL checkpoint $C_{10}$ | Full | 75.60 | 43.80 | 65.41 | 54.93 | | Intermediate RL checkpoint $C_{10}$ | Selected rank-one | 77.80 | 46.90 | 67.11 | 56.40 | This auxiliary control measures sensitivity to the checkpoint interval used to extract the update. It changes the update's content and potentially its magnitude, so it does not isolate the effect of separating RL updates from pre-existing donor–recipient differences. We will replace the original “supporting explicit stage isolation” interpretation with this narrower conclusion. The control also does not evaluate transfer without access to $B$, because the fixed initialization displacement still uses $B$. The present method requires the actual pre-RL checkpoint; direct transfer also uses it to compute $R-B$. We agree that this limits applicability when the relevant checkpoint is unavailable. We will rename this control “intermediate-RL reference (step 10)” and specify the construction in the Table 15 caption and accompanying discussion. The benefit of selective RL-update transfer is supported separately by matched comparisons: on MathVista, selection improves interpolation from 76.70 to 81.00 with the initialization displacement fixed, and direct transfer from 76.10 to 79.20 without that displacement. These comparisons support selection under the tested conditions; they do not establish transfer without the actual initialization. ## 3. Rank-one selection, Frobenius matching, and functional evidence **Why this parameter-space design is a candidate.** Before rescaling, the leading singular component minimizes rank-one reconstruction error in Frobenius norm. This does not guarantee optimal recipient accuracy or identify a particular reasoning operation. Frobenius matching controls each RL-update matrix's magnitude, rather than the functional output change or the total displacement after adding $\Gamma$. In the H1 normalization comparison, leading rank-one transfer achieves 75.20 MathVista without rescaling, 80.00 with nuclear-norm matching, and 81.00 with Frobenius matching. These are empirical reasons for the design, not a universal optimality claim. Rank one is likewise a simple default: the independently validation-selected layer-group ranks 1/1/4 reach 81.30 MathVista, compared with 81.00 for uniform rank one. This additional selection can improve the measured score while retaining the same general transfer framework. **We provide functional evidence at both the donor and recipient levels, together with controlled interventions on the update direction.** A working hypothesis consistent with these observations is that the leading directions retain useful RL-induced changes while reducing unfavorable interactions during transfer. The evidence below establishes functional relevance, while leaving the precise mechanism open.Response to Reviewer nhEz — Part 3/3
Source-task functionality. The selected update preserves substantial donor-side performance:
| Source model or reconstruction | MATH | GSM8K |
|---|---|---|
| Actual pre-RL initialization | 62.00 | 84.99 |
| Full RL donor | 76.20 | 90.98 |
| Rank-one reconstruction | 75.80 | 90.74 |
On MATH, rank-one reconstruction retains of the donor's improvement. This measures source-task recovery; recipient transfer is evaluated separately below.
Direction-specific recipient effects. The H1 controls keep the initialization displacement and transferable parameter support fixed:
| Transferred RL update | Magnitude treatment | MathVista |
|---|---|---|
| Full | Original update norm | 76.70 |
| Leading rank one | Matrix-wise Frobenius matched | 81.00 |
| Random rank one, 10 draws | Matrix-wise Frobenius matched | 74.05 ± 0.45 |
| Trailing singular direction | Matrix-wise Frobenius matched | 73.10 |
| Leading rank one, unscaled | Validation-tuned global coefficient | 78.50 |
| Full with rank-one-derived matrix-wise gains | Matrix-wise amplification control | 77.60 |
The random row reports mean ± sample SD over ten direction draws. All rows use and , except the unscaled leading control with validation-selected . Full, leading, random, and trailing comparisons match the matrix-wise RL-update Frobenius norms. The last two rows separately test global and matrix-wise scaling. Their results support the importance of directional content beyond arbitrary low rank or the tested magnitude adjustments.
Paired answer changes and error categories. On the same 1,000 MathVista examples, the paired outcomes are:
| Answer correctness | Examples |
|---|---|
| Both Full and Selective correct | 727 |
| Selective correct, Full incorrect | 83 |
| Full correct, Selective incorrect | 40 |
| Both incorrect | 150 |
The net gain is correct answers, or 4.30 percentage points. The 40 reverse outcomes are relative to Full, not a count of regressions from the original recipient. Within the 83 Selective-only successes, the Full errors have the following mutually exclusive labels:
| Full-error label among Selective-only successes | Examples | Share of 83 successes |
|---|---|---|
| Reasoning/perception | 65 | 78.3% |
| Repetition/loop | 8 | 9.6% |
| Token-cap truncation | 6 | 7.2% |
| Extraction pathology | 4 | 4.8% |
Only 18 of the 83 corrections have repetition, truncation, or extraction labels. These 18 labeled corrections alone cannot account for the net gain of 43 correct answers: the observed improvement extends beyond those categories. This is an outcome classification, not a causal attribution of the gain to newly acquired reasoning capability.
Consistency across problem variants. DynaMath provides a complementary test across ten variants of each of 501 seed questions:
| DynaMath metric | Full | Selective-RL |
|---|---|---|
| Mean accuracy over 5,010 variants (%) | 66.07 | 70.06 |
| Seed questions with all ten variants correct (%) | 29.1 | 34.3 |
| Median number of correct variants per seed question | 7.0 | 8.0 |
The all-ten-correct metric is measured over seed questions, separately from average accuracy over individual variants. Both measures improve in this comparison.
Additional corrections and preservation of correct recipient answers. The following examples illustrate two distinct observed behaviors:
| Example and correct answer | Recipient answer | Full answer | Selective answer | Observed behavior |
|---|---|---|---|---|
| Five-vertex odd-cycle edge coloring; answer 3 | 5, incorrect | 5, incorrect | 3, correct | Corrects a shared recipient/Full error |
| Integration of a velocity graph; answer 5.5 m | 5.5 m, correct | 5 m, incorrect | 5.5 m, correct | Preserves a correct recipient answer that Full loses |
These examples were selected by their outcomes. They demonstrate both behaviors, but do not estimate their prevalence or their separate contributions to the aggregate gain.
Together, the results connect the parameter-space intervention to source-task recovery, recipient answer correctness, and consistency across variants. They support the functional relevance of the selected directions. They do not establish a unique semantic “reasoning subspace” or a causal decomposition into newly acquired knowledge and preserved capabilities; we will make this distinction explicit.
Response to Reviewer nhEz — Part 1/3
We thank the reviewer for the careful assessment and for distinguishing the value of the controlled experiments from the novelty of their individual components. We address the contribution, the reference-checkpoint ambiguity, and the functional evidence below.
1. Contribution and relationship to existing methods
Our central contribution is a framework and controlled empirical evidence for selectively transferring actual RL-stage updates across models. Selective-RL connects the structure of updates learned by a text-only reasoning donor to their effectiveness in a multimodal recipient. The key finding is that transferring the donor's complete RL update can be less effective than transferring its dominant matrix-wise directions, even when the matrix-wise RL-update Frobenius norms and the initialization displacement are controlled.
We build on established methods, including the rank-one RL-update processing and norm rescaling of Cai et al.; we do not claim a new SVD or norm-rescaling operator. Our contribution has three aspects.
1. A training-stage formulation that enables controlled transfer comparisons. We define the RL-induced update using the donor's actual pre-RL initialization, Δ=R−B, and separately account for the initialization displacement, Γ=B−V. This formulation distinguishes the effect of selecting the RL update from the effect of moving toward the donor's initialization. Full and Selective-RL share the donor, recipient, parameter support, coefficients, and initialization displacement; spectral selection acts only on the RL-stage update. The contribution here is the controlled use of this decomposition, rather than the checkpoint subtraction itself.
2. Evidence that source reconstruction and cross-model transfer must be evaluated separately. Rank-one reconstruction retains 97.18% of the donor's MATH improvement over the actual initialization. However, preserving more source-side performance does not necessarily yield better recipient performance:
The source column evaluates donor reconstruction on MATH, while the recipient column evaluates transfer on H1 MathVista. Rank four recovers a higher source MATH score than rank one, but gives a lower recipient MathVista score. This observed ordering illustrates why source reconstruction alone is insufficient to assess transferability; it is not a claim of universal rank-one optimality.
The benefit also persists without initialization displacement: direct Full and Selective achieve 76.10 and 79.20 on MathVista. Under fixed initialization displacement, norm-matched random and trailing directions reach 74.05 ± 0.45 (10 draws, mean ± sample SD) and 73.10, respectively, versus 81.00 for the matched leading direction. A validation-tuned unscaled leading update reaches 78.50, and applying rank-one-derived matrix-wise gains to the full update reaches 77.60. These controls support the value of selecting particular RL-update directions beyond arbitrary low rank or the tested magnitude adjustments.
3. An effective transfer procedure without recipient-side gradient optimization. Selective-RL uses a fixed, donor-derived selection rule, with its merging coefficient selected on validation data. It improves over matched Full interpolation in 12 of 15 backbone–benchmark comparisons across three model families; the remaining regressions are retained. On H1 MathVista, it achieves 81.00, compared with 79.10 for STAR, 78.30 for FRISM, and 77.40 for IP-Merging. These results support the practical effectiveness of the transfer procedure under the evaluated conditions.
Cai et al., STAR/TSV, and prior LLM-to-VLM merging methods provide important foundations. We locate our contribution in the explicit RL-stage provenance, the controlled cross-model transfer findings, and the analysis of direction, magnitude, and composition. We will foreground this scope in the introduction and sharpen the corresponding distinctions in related work.
2. The reference checkpoint and checkpoint availability
Thank you for pointing out this ambiguity. In Table 15, the “earlier reference” is the checkpoint saved at step 10 of the same seed-42 H1 donor RL run, denoted by C10. The selected donor endpoint is R=C120, while B is the actual initialization before any RL updates. Thus, “earlier” means intermediate within the RL trajectory, preceding the selected donor endpoint.