acceptodds
Under review as a conference paper at ICLR 2027

Selective Transfer of RL Updates for Visual Reasoning

Abstract

Model merging provides a training-free way to transfer reasoning capabilities from language models to vision-language models (VLMs), but endpoint-based transfer can conflate pre-existing model differences with changes acquired during reasoning post-training. We instead formulate capability transfer around the training-stage update, isolating the parameter changes induced by reinforcement learning (RL). Yet transferring this update in full remains suboptimal: we find that its components differ substantially in cross-model transferability, with dominant directions transferring more effectively than the complete update. Based on this finding, we introduce Selective-RL, which isolates the RL-stage update, retains its dominant matrix-wise directions with magnitude preservation, and transfers them to the language modules of a VLM. Across three model families and five visual-reasoning benchmarks, Selective-RL improves full-update interpolation in 12 of 15 comparisons, including an 8.55 percentage-point MathVision gain on the Qwen recipient. Matched controls show that update magnitude or arbitrary low rank alone does not reproduce these gains. These results highlight a distinction between what is acquired during post-training and what remains transferable across models, providing a training-stage perspective on cross-model capability transfer. Code is available at https://anonymous.4open.science/r/selective-rl.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Reasoning, Model Merging, Reinforcement Learning, Low-Rank Approximation, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nhEzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Selective-RL, a training-free method for transferring reasoning updates from an RL-trained LLM to a VLM. The method first separates the RL-stage update from parameter differences that existed before RL. For each attention and MLP weight matrix, it keeps the leading rank-one component of the RL update and rescales it to preserve the original Frobenius norm. The selected update is then added to the VLM either directly or together with the pre-RL difference between the two models. Experiments on three VLM backbones and five visual reasoning benchmarks show that Selective-RL usually outperforms full-update transfer and several existing merging methods. Further ablation studies show that its gains come from selecting the leading direction, rather than simply using a low-rank update or changing the update scale.

Critical strengths
  1. The paper identifies an important issue in previous transfer methods: the difference between the donor and recipient mixes the RL-stage update with parameter differences that already existed before RL. The proposed decomposition provides a simple and clear way to separate these two parts.
  2. The experiments carefully examine the effects of update direction, update scale, retained rank, composition method, and checkpoint choice. The comparisons with random, trailing, unscaled, and full-update directions make the main result convincing.
  3. Selective-RL shows consistent improvements across several VLM backbones and visual reasoning benchmarks. The paper also compares the method with a broad set of recent model-merging approaches.
  4. Selective-RL requires no recipient-side training, additional inference module, or multimodal reasoning data. It is simple to apply when the exact pre-RL checkpoint is available and adds little construction cost.
Critical weaknesses
  1. The methodological novelty is limited. The related methods solve different problems, but several parts of Selective-RL already appear in them. TSV and STAR use SVD-based filtering to reduce interference in multi-task model merging. STAR restores the nuclear norm after truncation, and Cai et al. (2026) rescale a rank-one RL update to match the norm of the full update. IP-Merging and FRISM already transfer reasoning from LLMs to VLMs using selected parameter subspaces. Selective-RL brings these ideas together by applying norm-matched rank-one selection to the isolated RL-stage update. This is a clean and effective combination, but its novelty mainly comes from the combination and the new setting, rather than from a new selection or scaling method.
  2. The method requires the exact pre-RL checkpoint, which may be unavailable for many released models, so the practical scope of the method is limited. The performance drop with an earlier reference checkpoint (Appendix Table 15) also suggests that the method is sensitive to this requirement.
  3. There is limited theoretical explanation of the Rank-1 selection and Frobenius-norm rescaling designs. Both are defined in parameter space, but it is unclear why they are effective for model outputs.
Questions for authors
  1. What is the “earlier reference checkpoint” used in Appendix Table 15?
  2. Why is the leading singular direction expected to transfer better than other directions? Is there any functional or output-level evidence connecting this parameter-space direction to reasoning transfer?
6 comments

Response to Reviewer nhEz — Part 1/3

We thank the reviewer for the careful assessment and for distinguishing the value of the controlled experiments from the novelty of their individual components. We address the contribution, the reference-checkpoint ambiguity, and the functional evidence below.

1. Contribution and relationship to existing methods

Our central contribution is a framework and controlled empirical evidence for selectively transferring actual RL-stage updates across models. Selective-RL connects the structure of updates learned by a text-only reasoning donor to their effectiveness in a multimodal recipient. The key finding is that transferring the donor's complete RL update can be less effective than transferring its dominant matrix-wise directions, even when the matrix-wise RL-update Frobenius norms and the initialization displacement are controlled.

We build on established methods, including the rank-one RL-update processing and norm rescaling of Cai et al.; we do not claim a new SVD or norm-rescaling operator. Our contribution has three aspects.

1. A training-stage formulation that enables controlled transfer comparisons. We define the RL-induced update using the donor's actual pre-RL initialization, Δ=R−B\Delta=R-B, and separately account for the initialization displacement, Γ=B−V\Gamma=B-V. This formulation distinguishes the effect of selecting the RL update from the effect of moving toward the donor's initialization. Full and Selective-RL share the donor, recipient, parameter support, coefficients, and initialization displacement; spectral selection acts only on the RL-stage update. The contribution here is the controlled use of this decomposition, rather than the checkpoint subtraction itself.

2. Evidence that source reconstruction and cross-model transfer must be evaluated separately. Rank-one reconstruction retains 97.18% of the donor's MATH improvement over the actual initialization. However, preserving more source-side performance does not necessarily yield better recipient performance:

Update retainedSource MATH: donor or reconstructionRecipient MathVista: interpolated transfer
Full76.2076.70
Rank one75.8081.00
Rank four76.0080.30

The source column evaluates donor reconstruction on MATH, while the recipient column evaluates transfer on H1 MathVista. Rank four recovers a higher source MATH score than rank one, but gives a lower recipient MathVista score. This observed ordering illustrates why source reconstruction alone is insufficient to assess transferability; it is not a claim of universal rank-one optimality.

The benefit also persists without initialization displacement: direct Full and Selective achieve 76.10 and 79.20 on MathVista. Under fixed initialization displacement, norm-matched random and trailing directions reach 74.05 ± 0.45 (10 draws, mean ± sample SD) and 73.10, respectively, versus 81.00 for the matched leading direction. A validation-tuned unscaled leading update reaches 78.50, and applying rank-one-derived matrix-wise gains to the full update reaches 77.60. These controls support the value of selecting particular RL-update directions beyond arbitrary low rank or the tested magnitude adjustments.

3. An effective transfer procedure without recipient-side gradient optimization. Selective-RL uses a fixed, donor-derived selection rule, with its merging coefficient selected on validation data. It improves over matched Full interpolation in 12 of 15 backbone–benchmark comparisons across three model families; the remaining regressions are retained. On H1 MathVista, it achieves 81.00, compared with 79.10 for STAR, 78.30 for FRISM, and 77.40 for IP-Merging. These results support the practical effectiveness of the transfer procedure under the evaluated conditions.

Cai et al., STAR/TSV, and prior LLM-to-VLM merging methods provide important foundations. We locate our contribution in the explicit RL-stage provenance, the controlled cross-model transfer findings, and the analysis of direction, magnitude, and composition. We will foreground this scope in the introduction and sharpen the corresponding distinctions in related work.

2. The reference checkpoint and checkpoint availability

Thank you for pointing out this ambiguity. In Table 15, the “earlier reference” is the checkpoint saved at step 10 of the same seed-42 H1 donor RL run, denoted by C10C_{10}. The selected donor endpoint is R=C120R=C_{120}, while BB is the actual initialization before any RL updates. Thus, “earlier” means intermediate within the RL trajectory, preceding the selected donor endpoint.

Response to Reviewer nhEz — Part 2/3

We kept the recipient VV, donor endpoint RR, transferable parameter support SS, and initialization displacement Γ=B−V\Gamma=B-V fixed. Only the reference used to extract the transferred update changed:

Δ=R−B⟶Δ′=R−C10.\Delta=R-B \quad\longrightarrow\quad \Delta'=R-C_{10}.

All variants used α=β=λ=0.05\alpha=\beta=\lambda=0.05. For the selected variant, we applied leading rank-one selection to each eligible matrix of Δ′\Delta' and rescaled it to the Frobenius norm of that same matrix in Δ′\Delta'. Norm matching is performed within each reference construction; the update norms are not matched across the two references.

On the transferable support, the Full control was therefore

=V+\lambda[(B-V)+(R-C_{10})] =W_{\mathrm{Full}}-\lambda(C_{10}-B).$$ The two Full rows represent different models: the intermediate-reference construction omits the net donor update accumulated over the first 10 RL steps. Their scores can consequently differ. If both the initialization displacement and the extracted update were recomputed using $C_{10}$, the reference would cancel and Full interpolation would remain unchanged. Parameters outside $S$ remain those of the recipient. | Update reference | Transferred update | MathVista | MathVerse | DynaMath | MMStar | |---|---|---:|---:|---:|---:| | Actual pre-RL initialization $B$ | Full | 76.70 | 45.30 | 66.07 | 55.93 | | Actual pre-RL initialization $B$ | Selected rank-one | 81.00 | 52.00 | 70.06 | 59.67 | | Intermediate RL checkpoint $C_{10}$ | Full | 75.60 | 43.80 | 65.41 | 54.93 | | Intermediate RL checkpoint $C_{10}$ | Selected rank-one | 77.80 | 46.90 | 67.11 | 56.40 | This auxiliary control measures sensitivity to the checkpoint interval used to extract the update. It changes the update's content and potentially its magnitude, so it does not isolate the effect of separating RL updates from pre-existing donor–recipient differences. We will replace the original “supporting explicit stage isolation” interpretation with this narrower conclusion. The control also does not evaluate transfer without access to $B$, because the fixed initialization displacement still uses $B$. The present method requires the actual pre-RL checkpoint; direct transfer also uses it to compute $R-B$. We agree that this limits applicability when the relevant checkpoint is unavailable. We will rename this control “intermediate-RL reference (step 10)” and specify the construction in the Table 15 caption and accompanying discussion. The benefit of selective RL-update transfer is supported separately by matched comparisons: on MathVista, selection improves interpolation from 76.70 to 81.00 with the initialization displacement fixed, and direct transfer from 76.10 to 79.20 without that displacement. These comparisons support selection under the tested conditions; they do not establish transfer without the actual initialization. ## 3. Rank-one selection, Frobenius matching, and functional evidence **Why this parameter-space design is a candidate.** Before rescaling, the leading singular component minimizes rank-one reconstruction error in Frobenius norm. This does not guarantee optimal recipient accuracy or identify a particular reasoning operation. Frobenius matching controls each RL-update matrix's magnitude, rather than the functional output change or the total displacement after adding $\Gamma$. In the H1 normalization comparison, leading rank-one transfer achieves 75.20 MathVista without rescaling, 80.00 with nuclear-norm matching, and 81.00 with Frobenius matching. These are empirical reasons for the design, not a universal optimality claim. Rank one is likewise a simple default: the independently validation-selected layer-group ranks 1/1/4 reach 81.30 MathVista, compared with 81.00 for uniform rank one. This additional selection can improve the measured score while retaining the same general transfer framework. **We provide functional evidence at both the donor and recipient levels, together with controlled interventions on the update direction.** A working hypothesis consistent with these observations is that the leading directions retain useful RL-induced changes while reducing unfavorable interactions during transfer. The evidence below establishes functional relevance, while leaving the precise mechanism open.

Response to Reviewer nhEz — Part 3/3

Source-task functionality. The selected update preserves substantial donor-side performance:

Source model or reconstructionMATHGSM8K
Actual pre-RL initialization62.0084.99
Full RL donor76.2090.98
Rank-one reconstruction75.8090.74

On MATH, rank-one reconstruction retains (75.80−62.00)/(76.20−62.00)=97.18%(75.80-62.00)/(76.20-62.00)=97.18\% of the donor's improvement. This measures source-task recovery; recipient transfer is evaluated separately below.

Direction-specific recipient effects. The H1 controls keep the initialization displacement and transferable parameter support fixed:

Transferred RL updateMagnitude treatmentMathVista
FullOriginal update norm76.70
Leading rank oneMatrix-wise Frobenius matched81.00
Random rank one, 10 drawsMatrix-wise Frobenius matched74.05 ± 0.45
Trailing singular directionMatrix-wise Frobenius matched73.10
Leading rank one, unscaledValidation-tuned global coefficient78.50
Full with rank-one-derived matrix-wise gainsMatrix-wise amplification control77.60

The random row reports mean ± sample SD over ten direction draws. All rows use α=0.05\alpha=0.05 and β=0.05\beta=0.05, except the unscaled leading control with validation-selected β=0.50\beta=0.50. Full, leading, random, and trailing comparisons match the matrix-wise RL-update Frobenius norms. The last two rows separately test global and matrix-wise scaling. Their results support the importance of directional content beyond arbitrary low rank or the tested magnitude adjustments.

Paired answer changes and error categories. On the same 1,000 MathVista examples, the paired outcomes are:

Answer correctnessExamples
Both Full and Selective correct727
Selective correct, Full incorrect83
Full correct, Selective incorrect40
Both incorrect150

The net gain is 83−40=4383-40=43 correct answers, or 4.30 percentage points. The 40 reverse outcomes are relative to Full, not a count of regressions from the original recipient. Within the 83 Selective-only successes, the Full errors have the following mutually exclusive labels:

Full-error label among Selective-only successesExamplesShare of 83 successes
Reasoning/perception6578.3%
Repetition/loop89.6%
Token-cap truncation67.2%
Extraction pathology44.8%

Only 18 of the 83 corrections have repetition, truncation, or extraction labels. These 18 labeled corrections alone cannot account for the net gain of 43 correct answers: the observed improvement extends beyond those categories. This is an outcome classification, not a causal attribution of the gain to newly acquired reasoning capability.

Consistency across problem variants. DynaMath provides a complementary test across ten variants of each of 501 seed questions:

DynaMath metricFullSelective-RL
Mean accuracy over 5,010 variants (%)66.0770.06
Seed questions with all ten variants correct (%)29.134.3
Median number of correct variants per seed question7.08.0

The all-ten-correct metric is measured over seed questions, separately from average accuracy over individual variants. Both measures improve in this comparison.

Additional corrections and preservation of correct recipient answers. The following examples illustrate two distinct observed behaviors:

Example and correct answerRecipient answerFull answerSelective answerObserved behavior
Five-vertex odd-cycle edge coloring; answer 35, incorrect5, incorrect3, correctCorrects a shared recipient/Full error
Integration of a velocity graph; answer 5.5 m5.5 m, correct5 m, incorrect5.5 m, correctPreserves a correct recipient answer that Full loses

These examples were selected by their outcomes. They demonstrate both behaviors, but do not estimate their prevalence or their separate contributions to the aggregate gain.

Together, the results connect the parameter-space intervention to source-task recovery, recipient answer correctness, and consistency across variants. They support the functional relevance of the selected directions. They do not establish a unique semantic “reasoning subspace” or a causal decomposition into newly acquired knowledge and preserved capabilities; we will make this distinction explicit.

Reviewer_KcNCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective-RL, a training-free method for transferring reasoning capabilities from RL-trained LLMs to VLMs. The method isolates RL-induced parameter updates and transfers their dominant singular directions with magnitude preservation. Experiments across three model families and five benchmarks demonstrate improvements over conventional model merging methods.

Critical strengths
  1. The proposed method achieves consistent improvements across multiple visual reasoning benchmarks, particularly on Qwen3-VL.

  2. The authors investigate the effects of rank selection, update magnitude, and transfer strategies, providing useful empirical evidence for the proposed approach.

Critical weaknesses
  1. The proposed method largely combines established techniques, including SVD-based low-rank approximation, norm rescaling, and model merging. The main contribution appears to be an incremental application of these techniques to RL-update transfer rather than a fundamentally new methodology.

  2. While the experiments show that dominant singular directions transfer more effectively, the paper does not sufficiently explain why these directions are more transferable or what reasoning-related information they capture.

  3. The experiments primarily focus on mathematical reasoning, and the improvements are less consistent across different model families. It remains unclear whether the findings generalize to broader reasoning capabilities and other post-training settings.

Questions for authors
  1. How sensitive is the proposed method to the strength of the RL-trained donor, particularly with substantially longer RL training?

  2. Have the authors considered using different ranks for different layers or modules instead of applying a fixed rank-1 approximation to all matrices?

  3. Since the method modifies both the pre-RL displacement and RL update, how much of the improvement comes from transferring new reasoning capabilities versus reducing interference with the recipient's existing capabilities?

5 comments

Response to Reviewer KcNC — Part 1/5

Thank you for recognizing the empirical gains and the value of the rank, magnitude, and composition controls. We address the six concerns below. Scores are reported on a 0–100 scale; changes in accuracy are percentage points (pp).

1. Contribution beyond established components

Our central contribution is a framework and controlled empirical evidence for selectively transferring actual RL-stage updates across models. Selective-RL connects the structure of updates learned by a text-only reasoning donor to their effectiveness in a multimodal recipient. The key finding is that transferring the donor's complete RL update can be less effective than transferring its dominant matrix-wise directions, even when the matrix-wise RL-update Frobenius norms and the initialization displacement are controlled.

We build on established methods, including the rank-one RL-update processing and norm rescaling of Cai et al.; we do not claim a new SVD or norm-rescaling operator. Our contribution has three aspects.

1. A training-stage formulation that enables controlled transfer comparisons. We define the RL-induced update using the donor's actual pre-RL initialization, Δ=R−B\Delta=R-B, and separately account for the initialization displacement, Γ=B−V\Gamma=B-V. This formulation distinguishes the effect of selecting the RL update from the effect of moving toward the donor's initialization. Full and Selective-RL share the donor, recipient, parameter support, coefficients, and initialization displacement; spectral selection acts only on the RL-stage update. The contribution here is the controlled use of this decomposition, rather than the checkpoint subtraction itself.

2. Evidence that source reconstruction and cross-model transfer must be evaluated separately. Rank-one reconstruction retains 97.18% of the donor's MATH improvement over the actual initialization. However, preserving more source-side performance does not necessarily yield better recipient performance:

Update retainedSource MATH: donor or reconstructionRecipient MathVista: interpolated transfer
Full76.2076.70
Rank one75.8081.00
Rank four76.0080.30

The source column evaluates donor reconstruction on MATH, while the recipient column evaluates transfer on H1 MathVista. Rank four recovers a higher source MATH score than rank one, but gives a lower recipient MathVista score. This observed ordering illustrates why source reconstruction alone is insufficient to assess transferability; it is not a claim of universal rank-one optimality.

The benefit also persists without initialization displacement: direct Full and Selective achieve 76.10 and 79.20 on MathVista. Under fixed initialization displacement, norm-matched random and trailing directions reach 74.05 ± 0.45 (10 draws, mean ± sample SD) and 73.10, respectively, versus 81.00 for the matched leading direction. A validation-tuned unscaled leading update reaches 78.50, and applying rank-one-derived matrix-wise gains to the full update reaches 77.60. These controls support the value of selecting particular RL-update directions beyond arbitrary low rank or the tested magnitude adjustments.

3. An effective transfer procedure without recipient-side gradient optimization. Selective-RL uses a fixed, donor-derived selection rule, with its merging coefficient selected on validation data. It improves over matched Full interpolation in 12 of 15 backbone–benchmark comparisons across three model families; the remaining regressions are retained. On H1 MathVista, it achieves 81.00, compared with 79.10 for STAR, 78.30 for FRISM, and 77.40 for IP-Merging. These results support the practical effectiveness of the transfer procedure under the evaluated conditions.

Cai et al., STAR/TSV, and prior LLM-to-VLM merging methods provide important foundations. We locate our contribution in the explicit RL-stage provenance, the controlled cross-model transfer findings, and the analysis of direction, magnitude, and composition. We will foreground this scope in the introduction and sharpen the corresponding distinctions in related work.

Response to Reviewer KcNC — Part 2/5

2. Why dominant directions, and what do they represent?

Before norm rescaling, the leading singular component minimizes rank-one matrix reconstruction error. This motivates a candidate direction, but does not guarantee optimal recipient behavior. Frobenius matching controls each RL-update matrix's magnitude; it does not equate output changes or the total displacement after composition with the initialization term.

We provide functional evidence at both the donor and recipient levels, together with controlled interventions on the update direction. A working hypothesis consistent with these observations is that the leading directions retain useful RL-induced changes while reducing unfavorable interactions during transfer. The evidence below establishes functional relevance, while leaving the precise mechanism open.

Source-task functionality. The selected update preserves substantial donor-side performance:

Source model or reconstructionMATHGSM8K
Actual pre-RL initialization62.0084.99
Full RL donor76.2090.98
Rank-one reconstruction75.8090.74

On MATH, rank-one reconstruction retains (75.80−62.00)/(76.20−62.00)=97.18%(75.80-62.00)/(76.20-62.00)=97.18\% of the donor's improvement. This measures source-task recovery; recipient transfer is evaluated separately below.

Direction-specific recipient effects. The H1 controls keep the initialization displacement and transferable parameter support fixed:

Transferred RL updateMagnitude treatmentMathVista
FullOriginal update norm76.70
Leading rank oneMatrix-wise Frobenius matched81.00
Random rank one, 10 drawsMatrix-wise Frobenius matched74.05 ± 0.45
Trailing singular directionMatrix-wise Frobenius matched73.10
Leading rank one, unscaledValidation-tuned global coefficient78.50
Full with rank-one-derived matrix-wise gainsMatrix-wise amplification control77.60

The random row reports mean ± sample SD over ten direction draws. All rows use α=0.05\alpha=0.05 and β=0.05\beta=0.05, except the unscaled leading control with validation-selected β=0.50\beta=0.50. Full, leading, random, and trailing comparisons match the matrix-wise RL-update Frobenius norms. The last two rows separately test global and matrix-wise scaling. Their results support the importance of directional content beyond arbitrary low rank or the tested magnitude adjustments.

Paired answer changes and error categories. On the same 1,000 MathVista examples, the paired outcomes are:

Answer correctnessExamples
Both Full and Selective correct727
Selective correct, Full incorrect83
Full correct, Selective incorrect40
Both incorrect150

The net gain is 83−40=4383-40=43 correct answers, or 4.30 percentage points. The 40 reverse outcomes are relative to Full, not a count of regressions from the original recipient. Within the 83 Selective-only successes, the Full errors have the following mutually exclusive labels:

Full-error label among Selective-only successesExamplesShare of 83 successes
Reasoning/perception6578.3%
Repetition/loop89.6%
Token-cap truncation67.2%
Extraction pathology44.8%

Only 18 of the 83 corrections have repetition, truncation, or extraction labels. These 18 labeled corrections alone cannot account for the net gain of 43 correct answers: the observed improvement extends beyond those categories. This is an outcome classification, not a causal attribution of the gain to newly acquired reasoning capability.

Consistency across problem variants. DynaMath provides a complementary test across ten variants of each of 501 seed questions:

DynaMath metricFullSelective-RL
Mean accuracy over 5,010 variants (%)66.0770.06
Seed questions with all ten variants correct (%)29.134.3
Median number of correct variants per seed question7.08.0

Response to Reviewer KcNC — Part 3/5

The all-ten-correct metric is measured over seed questions, separately from average accuracy over individual variants. Both measures improve in this comparison.

Additional corrections and preservation of correct recipient answers. The following examples illustrate two distinct observed behaviors:

Example and correct answerRecipient answerFull answerSelective answerObserved behavior
Five-vertex odd-cycle edge coloring; answer 35, incorrect5, incorrect3, correctCorrects a shared recipient/Full error
Integration of a velocity graph; answer 5.5 m5.5 m, correct5 m, incorrect5.5 m, correctPreserves a correct recipient answer that Full loses

These examples were selected by their outcomes. They demonstrate both behaviors, but do not estimate their prevalence or their separate contributions to the aggregate gain.

Together, the results connect the parameter-space intervention to source-task recovery, recipient answer correctness, and consistency across variants. They support the functional relevance of the selected directions. They do not establish a unique semantic “reasoning subspace” or a causal decomposition into newly acquired knowledge and preserved capabilities; we will make this distinction explicit.

3. Broader tasks, model families, and post-training sources

We evaluate broader capabilities, different model families, and alternative post-training sources. The results below show the extent of the evidence and the observed exceptions.

Broader capabilities beyond the primary mathematical benchmarks. On H1, the four additional evaluations give:

BenchmarkRecipientFullSelective-RLGain over Full
TextVQA82.0381.0683.64+2.58
MMMU60.0059.2062.80+3.60
OCRBench78.6078.0078.70+0.70
POPE86.9086.3086.80+0.50

Scores are on a 0–100 scale; TextVQA uses its soft score and OCRBench is scaled to this range. Selective-RL improves over Full on all four benchmarks. This extends the evaluation beyond the primary mathematical suite, but does not establish universal preservation: POPE remains 0.10 points below the original recipient.

Transfer across model families. We evaluate matched donor–recipient configurations with Qwen, LLaMA, and Mistral language backbones. Each cell below shows Full → Selective-RL:

ConfigurationMathVistaMathVerseMathVisionDynaMathMMStar
H1: Qwen / Qwen3-VL76.70 → 81.0045.30 → 52.0040.46 → 49.0166.07 → 70.0655.93 → 59.67
H2: LLaMA / LLaVA-NeXT39.10 → 42.2022.40 → 25.8015.13 → 18.0924.07 → 26.4544.33 → 44.13
H3: Mistral / Idefics252.60 → 54.0020.40 → 22.2017.76 → 17.4322.55 → 24.1549.47 → 49.40

Bold cells mark the three regressions. Selection improves 12 of the 15 backbone–benchmark comparisons. All three configurations use the same rank-one rule, and the coefficient selected on H1 validation is reused for H2/H3. The result therefore extends beyond one model family, while remaining dependent on the configuration and task. These are matched donor–recipient pairs within each family, not transfers between arbitrarily incompatible architectures.

Transfer from different post-training sources. On H1, we also compare the selected and complete updates from offline SFT, offline distillation, and on-policy distillation:

Post-training sourceMathVista, Full → SelectedDynaMath, Full → SelectedMMStar, Full → Selected
RL76.70 → 81.0066.07 → 70.0655.93 → 59.67
Offline SFT77.10 → 78.0066.67 → 67.1556.27 → 56.13
Offline distillation77.60 → 78.8067.27 → 68.1856.67 → 56.80
On-policy distillation77.20 → 79.8067.07 → 69.2656.53 → 58.40
Reviewer_aaA3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to transfer RL-induced improvements from a language model to a compatible vision-language model through parameter merging. The proposed method, Selective-RL, extracts the difference between the donor’s actual pre-RL and post-RL checkpoints, retains the leading singular component of each eligible update matrix, and rescales it to preserve the original Frobenius norm. The resulting updates are transferred to the VLM’s language modules through direct addition or reconstructed-donor interpolation. Experiments across three model families show improvements over full-update transfer, with controls examining direction selection, update magnitude, rank, and composition.

Critical strengths
  1. The method is simple and requires no gradient-based adaptation of the recipient VLM. It also preserves the recipient’s architecture and introduces no additional inference modules.
  2. The controlled comparisons are useful. In particular, the random-direction, trailing-direction, normalization, and rank ablations help assess whether the gains come from selecting particular update directions rather than merely reducing or increasing update magnitude.
  3. The reported results are encouraging across multiple model families. The paper also includes donor-seed comparisons, paired outcome statistics, and cases where the method does not improve performance, providing a more balanced picture of its effectiveness.
Critical weaknesses
  1. The practical advantage over directly training the MLLM remains unclear. The experiments establish that, given an RL-trained donor, Selective-RL can outperform alternative merging strategies. However, the donors in this study are trained specifically for the experiments. If a practitioner already has an MLLM and the text reasoning data used here, directly applying RL to the MLLM’s language component is another natural option. The paper does not establish why training a separate LLM and then transferring its updates would be preferable in terms of total training cost, final performance, or preservation of visual capabilities. Reusing an existing reasoning donor is a reasonable use case, but this should be distinguished more clearly from training a donor specifically to improve the recipient.
  2. The methodological novelty is limited. Extracting RL-stage updates, retaining their leading singular component, and rescaling the result closely follows Cai et al., On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026). Spectral truncation and rescaling for model merging have also been explored in STAR, while LLM-to-VLM reasoning transfer and subspace-based merging are established directions in Bring Reason to Vision, IP-Merging, and FRISM. The main contribution therefore appears to be the empirical finding that this existing update-processing strategy transfers effectively to VLMs. The controlled experiments add value, but the conceptual advance beyond these prior works seems modest.
Questions for authors
  1. If I already have an MLLM and the text reasoning data used here, why should I train a separate LLM with RL and then merge its updates, rather than apply RL directly to the MLLM? Is the intended use case mainly one where a trained reasoning donor is already available? My concern is that the practical advantage is less clear when the donor also needs to be trained.
  2. What exactly does “earlier reference” mean in Table 15—which checkpoint is used, and how is it used in the construction? Could you also clarify what conclusion this comparison is intended to support?
4 comments

Response to Reviewer aaA3 — Part 1/3

Thank you for recognizing the controlled comparisons, the simplicity of the construction, and the reporting of both improvements and exceptions. We address the practical setting, the scope of our contribution, and the reference-checkpoint construction below. All benchmark scores are on a 0–100 scale; gains are percentage points (pp).

1. Practical setting: transferring from a donor versus directly training the VLM

Your distinction is important. We compare direct text-only RL of the recipient VLM with transfer from an RL-trained language donor, and distinguish the cost of training a new donor from the incremental cost of reusing an available one.

The practical motivation is access to reasoning improvements when repeating recipient-side RL is difficult within the user's resources. A practitioner may have access to a compatible reasoning donor, its actual pre-RL checkpoint, and the recipient weights, yet have limited additional compute or lack an established RL training pipeline. Illustrative use cases include:

Practical settingConstraint on repeating VLM RLOpportunity for Selective-RL
A small research team with limited remaining computeAvailable compute can cover checkpoint construction and validation, but not another full RL training workflowReuse the available donor's update without recipient gradient training
A deployment team equipped for model evaluation but not RL trainingThe team can construct and evaluate checkpoints, but does not maintain an RL rollout, reward, and optimization pipelineTransfer through offline parameter processing and held-out validation
A team maintaining several VLMs with compatible language modulesRepeating recipient-specific RL for each model adds training workReuse a suitable donor across compatible recipients, with separate construction and validation for each

These are use cases for the method, not additional deployment experiments. For a concrete budget illustration, in our measured environment an additional budget of 4 GPU-hours would cover the reported 3.025 GPU-hour donor-reuse workflow, but would not cover repeating the measured 27.750 GPU-hour direct-RL workflow. This assumes the required checkpoints are already available and still includes target-side validation. Within this transfer setting, our matched Full-versus-Selective comparisons test the benefit of selecting the RL-update directions.

For direct VLM RL, the visual encoder and projector are frozen and only the language component is trained. The direct VLM and donor RL runs use the same text training data and RL step budget. Costs use the same hardware accounting, and final evaluation settings are matched.

MethodMathVistaMathVerseMathVisionTextVQAPOPE
Recipient75.0042.0037.5082.0386.90
Full interpolation76.7045.3040.4681.0686.30
Selective-RL81.0052.0049.0183.6486.80
Direct text-only VLM RL81.8053.0050.0082.1086.60

TextVQA uses its soft score. Direct RL is higher on the three reasoning benchmarks by 0.80, 1.00, and 0.99 points. Selective-RL is higher on TextVQA and POPE by 1.54 and 0.20 points, respectively. This comparison shows a trade-off rather than uniform dominance. In particular, Selective-RL's POPE score remains 0.10 points below the recipient, and these point estimates do not establish statistical equivalence or universal preservation of visual capabilities.

The cost accounting separates RL training, training-stage checkpoint selection, target-side validation and parameter selection, and transfer construction. All entries below are in GPU-hours.

Route and accounting scopeRL trainingTraining-stage checkpoint selectionTarget-side validation / parameter selectionTransfer constructionTotal
Direct VLM RL: complete measured workflow24.0000.7503.000027.750
Newly trained donor + Selective-RL: complete measured workflow24.0000.7503.0000.02527.775
Existing donor + Selective-RL: incremental workflow0 additional0 additional3.0000.0253.025

Response to Reviewer aaA3 — Part 2/3

When a donor must be trained for this recipient, the measured total costs are effectively the same (27.775 versus 27.750 GPU-hours), while direct RL scores higher on the three reasoning benchmarks. We therefore do not claim an end-to-end compute advantage for training a separate donor solely to improve one VLM.

The cost advantage applies when a suitable RL donor and its actual pre-RL checkpoint are already available: the incremental transfer workflow costs 3.025 GPU-hours, compared with 27.750 GPU-hours for the measured direct-RL workflow, an 89.1% reduction in additional compute. The zero entries mean no additional donor training or donor checkpoint selection; those upstream costs were incurred previously and are excluded from this incremental accounting. The 3.000 GPU-hours of target-side selection are still included, so this is not a comparison of the 90-second construction step alone against full RL training.

These results make the intended opportunity more precise: reuse of an available donor offers a lower-cost route without recipient gradient training, with the measured score trade-offs above. If no donor exists and the objective is the highest score on these reasoning tasks, the present comparison favors direct RL. We will distinguish these two scenarios and qualify the “training-free” description as recipient-side transfer.

2. Contribution relative to established update processing and merging methods

Our central contribution is a framework and controlled empirical evidence for selectively transferring actual RL-stage updates across models. Selective-RL connects the structure of updates learned by a text-only reasoning donor to their effectiveness in a multimodal recipient. The key finding is that transferring the donor's complete RL update can be less effective than transferring its dominant matrix-wise directions, even when the matrix-wise RL-update Frobenius norms and the initialization displacement are controlled.

We build on established methods, including the rank-one RL-update processing and norm rescaling of Cai et al.; we do not claim a new SVD or norm-rescaling operator. Our contribution has three aspects.

1. A training-stage formulation that enables controlled transfer comparisons. We define the RL-induced update using the donor's actual pre-RL initialization, Δ=R−B\Delta=R-B, and separately account for the initialization displacement, Γ=B−V\Gamma=B-V. This formulation distinguishes the effect of selecting the RL update from the effect of moving toward the donor's initialization. Full and Selective-RL share the donor, recipient, parameter support, coefficients, and initialization displacement; spectral selection acts only on the RL-stage update. The contribution here is the controlled use of this decomposition, rather than the checkpoint subtraction itself.

2. Evidence that source reconstruction and cross-model transfer must be evaluated separately. Rank-one reconstruction retains 97.18% of the donor's MATH improvement over the actual initialization. However, preserving more source-side performance does not necessarily yield better recipient performance:

Update retainedSource MATH: donor or reconstructionRecipient MathVista: interpolated transfer
Full76.2076.70
Rank one75.8081.00
Rank four76.0080.30

The source column evaluates donor reconstruction on MATH, while the recipient column evaluates transfer on H1 MathVista. Rank four recovers a higher source MATH score than rank one, but gives a lower recipient MathVista score. This observed ordering illustrates why source reconstruction alone is insufficient to assess transferability; it is not a claim of universal rank-one optimality.

The benefit also persists without initialization displacement: direct Full and Selective achieve 76.10 and 79.20 on MathVista. Under fixed initialization displacement, norm-matched random and trailing directions reach 74.05 ± 0.45 (10 draws, mean ± sample SD) and 73.10, respectively, versus 81.00 for the matched leading direction. A validation-tuned unscaled leading update reaches 78.50, and applying rank-one-derived matrix-wise gains to the full update reaches 77.60. These controls support the value of selecting particular RL-update directions beyond arbitrary low rank or the tested magnitude adjustments.

Response to Reviewer aaA3 — Part 3/3

3. An effective transfer procedure without recipient-side gradient optimization. Selective-RL uses a fixed, donor-derived selection rule, with its merging coefficient selected on validation data. It improves over matched Full interpolation in 12 of 15 backbone–benchmark comparisons across three model families; the remaining regressions are retained. On H1 MathVista, it achieves 81.00, compared with 79.10 for STAR, 78.30 for FRISM, and 77.40 for IP-Merging. These results support the practical effectiveness of the transfer procedure under the evaluated conditions.

Cai et al., STAR/TSV, and prior LLM-to-VLM merging methods provide important foundations. We locate our contribution in the explicit RL-stage provenance, the controlled cross-model transfer findings, and the analysis of direction, magnitude, and composition. We will foreground this scope in the introduction and sharpen the corresponding distinctions in related work.

3. What “earlier reference” means and what the control supports

Thank you for pointing out this ambiguity. In Table 15, the “earlier reference” is the checkpoint saved at step 10 of the same seed-42 H1 donor RL run, denoted by C10C_{10}. The selected donor endpoint is R=C120R=C_{120}, while BB is the actual initialization before any RL updates. Thus, “earlier” means intermediate within the RL trajectory, preceding the selected donor endpoint.

We kept the recipient VV, donor endpoint RR, transferable parameter support SS, and initialization displacement Γ=B−V\Gamma=B-V fixed. Only the reference used to extract the transferred update changed:

Δ=R−B⟶Δ′=R−C10.\Delta=R-B \quad\longrightarrow\quad \Delta'=R-C_{10}.

All variants used α=β=λ=0.05\alpha=\beta=\lambda=0.05. For the selected variant, we applied leading rank-one selection to each eligible matrix of Δ′\Delta' and rescaled it to the Frobenius norm of that same matrix in Δ′\Delta'. Norm matching is performed within each reference construction; the update norms are not matched across the two references.

On the transferable support, the Full control was therefore

=V+\lambda[(B-V)+(R-C_{10})] =W_{\mathrm{Full}}-\lambda(C_{10}-B).$$ The two Full rows represent different models: the intermediate-reference construction omits the net donor update accumulated over the first 10 RL steps. Their scores can consequently differ. If both the initialization displacement and the extracted update were recomputed using $C_{10}$, the reference would cancel and Full interpolation would remain unchanged. Parameters outside $S$ remain those of the recipient. | Update reference | Transferred update | MathVista | MathVerse | DynaMath | MMStar | |---|---|---:|---:|---:|---:| | Actual pre-RL initialization $B$ | Full | 76.70 | 45.30 | 66.07 | 55.93 | | Actual pre-RL initialization $B$ | Selected rank-one | 81.00 | 52.00 | 70.06 | 59.67 | | Intermediate RL checkpoint $C_{10}$ | Full | 75.60 | 43.80 | 65.41 | 54.93 | | Intermediate RL checkpoint $C_{10}$ | Selected rank-one | 77.80 | 46.90 | 67.11 | 56.40 | This auxiliary control measures sensitivity to the checkpoint interval used to extract the update. It changes the update's content and potentially its magnitude, so it does not isolate the effect of separating RL updates from pre-existing donor–recipient differences. We will replace the original “supporting explicit stage isolation” interpretation with this narrower conclusion. The control also does not evaluate transfer without access to $B$, because the fixed initialization displacement still uses $B$. The present method requires the actual pre-RL checkpoint; direct transfer also uses it to compute $R-B$. We agree that this limits applicability when the relevant checkpoint is unavailable. We will rename this control “intermediate-RL reference (step 10)” and specify the construction in the Table 15 caption and accompanying discussion. The benefit of selective RL-update transfer is supported separately by matched comparisons: on MathVista, selection improves interpolation from 76.70 to 81.00 with the initialization displacement fixed, and direct transfer from 76.10 to 79.20 without that displacement. These comparisons support selection under the tested conditions; they do not establish transfer without the actual initialization.
Reviewer_aaA3OpenReview ↗

final justification

I appreciate the direct RL comparison and the clarification of Table 15. I am maintaining my weak reject recommendation for two reasons.

First, the practical benefit depends on having a compatible reasoning donor and its exact pre-RL checkpoint. When the donor is trained specifically for one VLM, the authors report nearly identical total costs, and direct VLM RL scores slightly higher on all three reasoning benchmarks. Reusing an existing donor is a plausible use case, but the authors trained all the donors in this study. It remains unclear how broadly the reported gains carry over to donors available in practice. I would encourage the authors to test the method with independently released donors and their published starting checkpoints.

Second, the methodological novelty is limited. Selecting the leading singular component of each RL update matrix and rescaling it follows prior work. The paper’s main contribution is the empirical finding that these selected updates can transfer better than the full update to a multimodal model. The controlled comparisons support that finding, but I see it as an incremental advance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.