acceptodds
Under review as a conference paper at ICLR 2027

BiVaR:Bidirectional Variational Discrimination for Cross-Model Reasoning Transfer

Abstract

Vision-language models (VLMs) inherit their language backbone from pretrained large language models (LLMs), yet still fall behind them on tasks requiring complex multi-step reasoning. Training-free reasoning transfer via model merging avoids costly multimodal supervision. However, existing merging strategies rely largely on parameter-level relations or individual response strength, which provide only indirect evidence of cross-model transferability and may therefore introduce model- or modality-specific interference during reasoning transfer. We introduce BiVaR, a distribution-guided framework that identifies transferable reasoning components through their functional consistency across models. Specifically, transferability is determined by whether a neuron's activation behavior remains compatible with the other model's activation distribution, while bidirectional evaluation retains only reasoning components with mutually preserved functional behavior across models. Reasoning transfer is then restricted to the corresponding shared-neuron subspaces, after adaptive low-rank decomposition preserves dominant cross-model directions while suppressing residual variations. Extensive experiments across multiple VLM-LLM pairs show that BiVaR consistently improves mathematical and text-only reasoning while preserving or improving general multimodal capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
model merging, reasoning transfer, distributional consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RcekOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BiVaR, a training-free model merging method that transfers reasoning capabilities from LLMs to VLMs. The method identifies shared reasoning neurons through cross-model activation distribution consistency. It then uses adaptive low-rank merging to improve reasoning while preserving other general multimodal capabilities of VLMs.

Critical strengths
  1. Merging LLMs with VLMs to improve VLM reasoning without costly training is an interesting topic.
  2. The main experiments cover multiple model pairs.
Critical weaknesses

W1. Methodological motivation and novelty are not sufficiently clear.

  • The novelty of BiVaR mainly lies in using cross-model activation distribution consistency to identify shared reasoning neurons for model merging, compared with prior methods such as SNRF. However, the motivation for reframing shared reasoning neuron identification as a distribution consistency problem is not sufficiently clear. In particular, why should similar activation distributions indicate that neurons have shared reasoning functions? What is the conceptual advantage of this criterion over existing neuron selection methods such as detecting neuron set intersection by SNRF? The method section, especially Sections 3.3 and 3.4 provides detailed math definitions, but is not easy to follow, especially, the motivation for each design choice is unclear.

W2. Important experiments and ablations are missing.

  • The main experiments cover multimodal mathematical reasoning, general multimodal understanding, and text-only reasoning, but non-mathematical, common-knowledge multimodal reasoning is not evaluated, the authors may consider potential benchmark datasets: A-OKVQA, VCR, MMMU-Pro, ReasonVQA, GQA. For text-only reasoning, why is MMLU-Pro reported in Figure 1(a) but not included in the main experiments?
  • For the ablations, results are mainly only reported on one model Qwen2.5-VL-7B-Instruct. Are these ablation findings generalizable to other models?
  • A key ablation appears missing: to demonstrate the benefit of distribution consistency over neuron overlap, a direct comparison between BiVaR's neuron selection and SNRF's intersection-based selection is needed, using the same merging procedure and comparable numbers of selected neurons.

W3. The performance gains over prior methods and the base VLM are modest overall. Moreover, across all methods, improvements over the base VLM are generally small, and some merging methods even harm performance (Figure 1(b) and Table 1).

  • Can the authors elaborate on the significance of these gains? Are the improvements consistent across random seeds? Additionally, besides the base VLM performance, it can be very helpful to also report the LLM performances on each benchmark, so that readers can better understand how much reasoning capability is transferred and how much room for improvement remains.
Questions for authors
  1. Why should similar activation distributions indicate that neurons have shared reasoning functions? What justifies using distributional consistency rather than individual neuron responses to identify transferable reasoning neurons?

  2. The paper states that distribution consistency corresponds to a density ratio close to 1. However, Equations (5)–(7) use only an upper threshold on the discriminator score. How does this ensure that selected neurons are distributionally consistent rather than simply having low density-ratio scores?

  3. In lines 216–218, why is mutual consistency necessary when reasoning is transferred only from the LLM to the VLM? Can the authors clarify the definition of mutual consistency and why unidirectional discrimination is insufficient?

  4. Since distribution-based neuron selection is the key contribution, can the authors provide a controlled ablation comparing BiVaR's selection with SNRF's intersection-based selection, using the same merging procedure and comparable numbers of selected neurons? Table 6 partially addresses this but does not fully isolate the effect of neuron selection.

  5. GSM8K and Geo3K are used to collect neuron activations. How sensitive are the identified neurons and final performance to the choice and number of reasoning samples? Could different datasets or subsets lead to different results?

  6. Given the relatively modest performance gains in Table 1, are these improvements consistent across different random seeds? How much of the source LLM's reasoning capability is actually transferred to the merged VLM?

  7. Table 5 studies rank sensitivity but does not demonstrate the advantage of adaptive-rank selection over fixed-rank decomposition.

7 comments

We appreciate the reviewer’s recognition of the importance of training-free reasoning transfer and the comprehensive evaluation across multiple model pairs. We address your concerns below.


W1: Methodological Motivation and Novelty.

(1) Limitations of Neuron Intersection. SNRF identifies shared neurons by intersecting highly activated channels at corresponding positions, implicitly assuming index-wise functional alignment. However, fine-tuning may alter neuronal representations, such that functionally related neurons no longer occupy corresponding positions. Consequently, SNRF may overlook transferable neurons due to this strict positional constraint.

(2) Motivation for Distributional Consistency. We instead formulate neuron selection as an ID–OOD distribution discrimination problem. If a discriminator struggles to distinguish a neuron's activation distribution from the other model's reference distribution, this suggests statistically compatible cross-model response behavior, making it a promising candidate for reasoning transfer without requiring strict functional equivalence.

Revisions: The motivation for distribution-based selection is clarified in Section 1 (L46-81).


W2: Additional Experiments and Ablation Studies.

(1) Non-Mathematical Reasoning Evaluation. We extend our evaluation to A-OKVQA, VCR, MMMU-Pro, ReasonVQA, GQA, and MMLU-Pro. As shown below, BiVaR achieves better overall performance on both models, supporting its generalizability beyond mathematical reasoning.

MethodA-OKVQAVCRMMMU-ProReasonVQAGQAMMLU-Pro
Qwen2.5-VL-7B-Instruct86.8191.1232.3146.0072.4040.99
BR2V86.8489.9430.8745.0070.8040.41
DARE86.7891.4530.1746.0071.2041.50
LED-Merging86.8591.8230.6444.0072.4040.90
SNRF86.8791.2532.1447.0072.0042.09
BiVaR86.9192.2232.3748.0073.0042.89
MethodA-OKVQAVCRMMMU-ProReasonVQAGQAMMLU-Pro
Qwen3-VL-8B-Instruct87.8692.3114.7464.0071.8070.06
BR2V87.8891.7213.6464.0068.0069.80
DARE87.8992.6915.9063.0071.4069.40
LED-Merging87.9892.0812.4964.0071.4071.22
SNRF88.0392.3114.8064.0071.8072.50
BiVaR88.0593.4916.4264.0072.6074.78

(2) Generalizability of Ablation Findings. We extend the key ablations to Qwen3-VL-8B-Instruct. As shown below, the results exhibit consistent trends with Qwen2.5-VL, further validating the effectiveness of our key components across different models.

V→LL→VMathVistaMathVerseMathVisionMMEScienceQAGPQA
——72.7745.4517.761726.9494.6531.69
✓—72.9745.7718.091730.9495.0932.14
—✓73.4045.9417.761735.0294.9932.37
✓✓73.7746.4719.131750.0895.3332.58
ModelSettingMathVistaMathVerseMathVisionBiologyChemistryPhysics
Qwen3-VL-8B-InstructBase72.7745.4517.76———
Qwen3-VL-8B-InstructShared000———
Qwen3-VL-8B-InstructRandom25.6347.939.21———
DeepSeek-R1-0528-Qwen3-8BBase———46.1533.8835.83
DeepSeek-R1-0528-Qwen3-8BShared———000
DeepSeek-R1-0528-Qwen3-8BRandom———25.6418.5816.04
BenchmarkBaseSharedS-onlyB-only
MathVision17.7618.0918.4219.08
GPQA31.6931.9231.6032.14

(3) Effectiveness of Neuron Selection (W2-3 & Q4). We compare BiVaR's distribution-based selection with SNRF's activation-based selection using the same merging procedure adopted from SNRF.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
SNRF69.2342.4327.121709.1150.5665.3828.73
BiVaR70.0742.5127.961715.1650.5667.0030.26
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
SNRF72.8746.3717.981732.9453.0794.7329.86
BiVaR73.2346.4018.421743.5653.1195.1431.25

BiVaR consistently outperforms SNRF across both models. Furthermore, Table 6 of the original paper shows that BiVaR-only neurons yield greater reasoning improvements than SNRF-only neurons under the same transfer update. Together, these results support the effectiveness of our distribution-based neuron selection.

Revisions: Additional benchmarks are reported in Appendix C.1.2 (L963–971, Table 10). Qwen3 ablations and selection comparisons are added in Appendix C.2.3–C.2.4 (L1219–1241, Tables 16, 18–19) and Table 6 (L487–493).

W3 & Q6: Significance, Stability, and Extent of Reasoning Transfer.

(1) Significance of Performance Gains. Although the absolute gains are modest, BiVaR improves reasoning performance without additional training while preserving general multimodal capabilities. As shown in Table 6, SNRF-only neurons provide limited benefits and even degrade GPQA performance, whereas BiVaR-only neurons improve both MathVision and GPQA, highlighting the importance of selective transfer in mitigating negative interference.

(2) Stability Across Random Seeds. We conduct three independent cross-machine runs and evaluate randomness in data sampling, random projections, and discriminator initialization using three seeds (41–43).

Experimental SettingMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
Cross-Machine Runs70.57±0.0942.71±0.1329.39±0.191721.83±0.6350.59±0.1767.90±0.1030.89±0.06
Data Sampling70.52±0.5442.91±0.2428.95±0.321720.83±1.2950.63±0.1368.10±0.2130.67±0.13
Random Projection70.46±0.3242.72±0.4529.17±0.501721.77±0.8550.63±0.4567.87±0.1530.78±0.17
Discriminator Initialization70.49±0.1742.70±0.6629.28±0.331721.83±1.1550.59±0.2867.89±0.3231.11±0.41
Experimental SettingMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
Cross-Machine Runs73.77±0.0746.50±0.0619.08±0.001750.16±0.4353.15±0.0695.34±0.0532.68±0.18
Data Sampling73.73±0.2846.56±0.3219.19±0.191750.03±0.9853.15±0.2895.34±0.2532.46±0.19
Random Projection73.80±0.2546.46±0.3019.08±0.331750.32±0.8453.15±0.2895.31±0.2532.46±0.50
Discriminator Initialization73.73±0.2546.49±0.3018.97±0.191750.05±0.9353.00±0.2995.39±0.2332.35±0.20

BiVaR exhibits small performance variations across both models, supporting the stability of its improvements.

(3) Extent of Reasoning Transfer. We additionally report source LLM performance and introduce Reasoning Gap Closed (RGC) to quantify the proportion of the performance gap closed by BiVaR:RGC=SBiVaR−SVLMSLLM−SVLM×100%\mathrm{RGC} = \frac{S_{\mathrm{BiVaR}} - S_{\mathrm{VLM}}}{S_{\mathrm{LLM}} - S_{\mathrm{VLM}}} \times 100\%

Target VLMBenchmarkSource LLMBase VLMBiVaRImprovement (pp)Reasoning Gap (pp)RGC (%)
Qwen2.5-VL-7B-InstructGPQA34.1529.6830.82+1.144.4725.50
Qwen2.5-VL-7B-InstructMMLU-Pro Math45.1640.9942.89+1.904.1745.56
Qwen2.5-VL-7B-InstructMath 50068.0063.8065.60+1.804.2042.86
Qwen3-VL-8B-InstructGPQA36.8331.6932.58+0.895.1417.32
Qwen3-VL-8B-InstructMMLU-Pro Math78.1370.0674.78+4.728.0758.49
Qwen3-VL-8B-InstructMath 50080.675.6079.20+3.605.0072.00

BiVaR achieves higher RGC on mathematical reasoning tasks, consistent with our objective of transferring mathematical reasoning capabilities from the source LLMs. Meanwhile, improvements on GPQA demonstrate that the transferred capabilities also generalize to scientific reasoning in biology, chemistry, and physics, albeit to a lesser extent.

Revisions: Neuron-subset results are extended in Table 6 (L487–493), and repeated-run statistics in Appendix C.3.2 (L1436–1446, Tables 25–26). Reasoning-gap closure is added in Appendix C.1.5 (L1069–1079, Table 13).

Q1: Rationale for Distributional Consistency.

(1) Conceptual Motivation. Unlike individual activation responses, which primarily reflect response strength, our method evaluates cross-model compatibility through ID–OOD distribution discrimination. If a discriminator cannot reliably distinguish a neuron's activation distribution from the other model's reference distribution, this suggests shared response behavior under reasoning inputs. We therefore use distributional consistency as a proxy for identifying transferable neurons rather than requiring strict functional equivalence.

(2) Empirical Validation. We compare BiVaR with Hungarian Fusion (HF) and K-means Fusion (KF), following Model Fusion via Retrofitting [1].

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
BiVaR70.5342.6729.281721.7550.5667.8930.82
HF69.1741.1627.631711.7250.5666.7329.91
KF69.7342.4329.611720.1350.5666.9330.58
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
BiVaR73.7746.4719.131750.0853.1195.3332.58
HF73.3345.7318.091727.2052.5694.7431.70
KF73.6046.1118.421729.2253.1195.0431.92

BiVaR achieves better overall performance on both models. Furthermore, our neuron subset ablation shows that B-only neurons outperform S-only neurons selected by activation strength, providing additional evidence for distribution-based selection.

[1] Model Fusion via Retrofitting, ICML 2026.

Revisions: The distributional rationale is clarified in Section 1 (L46–81), with matching comparisons in Appendix C.2.1 (L1126–1133, Table 14) and neuron-subset evidence in Table 6 (L487–493).


Q2–Q3: One-Sided Thresholding and Mutual Consistency.

(1) One-Sided Thresholding. We acknowledge that one-sided thresholding does not strictly guarantee a density ratio close to 1, as the discriminator score differs from the log-density ratio by an additive constant. Instead, we use reference-calibrated thresholds to filter neurons with substantial distributional deviations. As shown below, our approach outperforms two-sided thresholding and its bidirectional intersection, suggesting that stricter constraints do not necessarily improve reasoning transfer.

(2) Mutual Consistency. Although reasoning transfer is unidirectional (LLM→VLM), KL divergence is inherently asymmetric, so unidirectional discrimination may introduce uncertainty in identifying distributionally compatible neurons. We therefore evaluate compatibility in both directions (V→L and L→V) and retain their intersection to improve selection robustness.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
BiVaR (Ours)70.5342.6729.281721.7550.5667.8930.82
Two-Sided V→L68.6042.5524.011703.8448.7867.6730.26
Two-Sided L→V68.9341.9628.291696.1251.3367.7729.93
Two-Sided Intersection68.1042.6125.991719.0948.4467.8229.61
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
BiVaR (Ours)73.7746.4719.131750.0853.1195.3332.58
Two-Sided V→L68.7742.8918.361730.7353.0094.4531.91
Two-Sided L→V68.9742.7218.361736.1052.8994.7031.91
Two-Sided Intersection68.6342.8918.361744.8753.3395.1932.57

As shown in the table and our additional neuron-count-controlled experiments, bidirectional selection achieves better overall performance, supporting the effectiveness of mutual consistency.

Revisions: Thresholding is clarified in Appendix C.2.5 (L1273–1281, Table 20), and bidirectional selection is evaluated in Appendix C.2.3 (L1219–1229, Tables 16–17).

Reviewer_eKG6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiVaR to transfer reasoning capabilities from a text-only large language model to a vision-language model without fine-tuning the latter’s backbone. It uses bidirectional distribution discriminators trained on neuronal activation features to select candidate shared channels, then selectively merges a masked, adaptively low-rank weight difference. Across some model pairs, the authors report improvements in mathematical and text-only reasoning, with no declines in the evaluated general multimodal metrics relative to the original vision-language models.

Critical strengths
  1. The research question is practically important. The paper explores transferring reasoning capabilities from a text-only LLM to a VLM without fine-tuning the VLM backbone, while seeking to preserve its existing multimodal performance. It addresses the trade-off between capability transfer and interference in model merging.

  2. The paper proposes a clear neuron-selection strategy. Rather than selecting channels solely by activation magnitude, BiVaR uses distributional discrimination of cross-model activation features and screens candidate shared channels in both directions. This provides an alternative criterion for selective transfer.

  3. The paper includes several informative ablation studies. It compares unidirectional and bidirectional selection, different distribution-detection methods, neuron-selection subsets, and low-rank dimensions, providing preliminary evidence about the contribution of each design choice.

Critical weaknesses
  1. The discriminator-based selection lacks independent validation. The paper trains the discriminators on the extracted neuronal features, then uses scores from the reference-model features to set thresholds and select channels. It does not report whether the selected channels remain stable when features are extracted from a separate set of questions or held-out neurons. As a result, it is unclear how much the selection changes with the sampled questions or discriminator training. I suggest extracting features from independent questions and reporting the overlap between the selected channel sets.

  2. Some reported gains lack uncertainty estimates. In Table 1, Qwen2.5-VL improves over the base model by only 0.24 percentage points on MathVerse, while Qwen3-VL improves by 0.11 percentage points on MMMU. The paper does not report variation across different question samples, random projections, or discriminator-training seeds. I suggest reporting results from repeated runs and paired, question-level confidence intervals, and avoiding descriptions of small numerical gains as robust superiority without further evidence.

  3. The comparison with closely related merging methods is incomplete. Why are methods from the same research area, such as FRISM[1]and IP-Merging, not included in the experimental comparison? Comparing BiVaR with these methods would help clarify its empirical advantages over related approaches.

  4. The claim of preserving general multimodal capabilities relies on a small number of aggregate metrics. It is encouraging that the overall MME, MMMU, and ScienceQA scores in Table 1 do not decline. However, aggregate scores cannot rule out degradation on specific visual subtasks. I suggest reporting subtask-level results, performance on questions with and without images, and at least one independent non-mathematical visual task.

  5. The ablation does not isolate the contribution of bidirectional discrimination itself. Table 2 shows that the bidirectional configuration outperforms either unidirectional configuration, but taking the intersection also changes the number of selected channels and potentially the magnitude of the final weight update. The performance difference therefore cannot be attributed solely to more accurate bidirectional selection. I suggest repeating the comparison while controlling for the number of selected channels, their layer-wise distribution, and the update norm.

Reference:

[1] Huang, Chenyu, et al. "FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models." arXiv preprint arXiv:2601.21187 (2026).

[2] Hu, Yijie, et al. "Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?." Advances in Neural Information Processing Systems 38 (2026): 130912-130940.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

9 comments

We appreciate the reviewer’s recognition of the practical importance of our research, the alternative criterion for selective transfer, and the comprehensive ablation studies. We address your concerns below.


W1: Independent Validation of Neuron Selection.

(1) Full-Pipeline Validation. We use independent GSM8K and Geo3K questions to re-extract activations, retrain discriminators, and repeat neuron selection. As shown below, the new selections retain 82.04%/80.86% of the original neurons on Qwen2.5-VL/Qwen3-VL, respectively, while maintaining stable downstream performance.

(2) Frozen-Discriminator Validation. We further use independent questions with the original discriminators and fixed thresholds. The new selections retain 89.49%/89.97% of the original neurons, respectively, with comparable or improved downstream performance. These results suggest that the learned selection criteria generalize to unseen questions.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQARetained (%)
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68—
BiVaR70.5342.6729.281721.7550.5667.8930.82100.00
Independent Samples (Full Pipeline)70.7343.0629.281719.2550.6767.8230.5882.04
Independent Samples (Frozen Discriminator)70.1342.6432.241720.3750.6767.8231.2589.49
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQARetained (%)
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69—
BiVaR73.7746.4719.131750.0853.1195.3332.58100.00
Independent Samples (Full Pipeline)73.7746.7922.701748.3753.4495.4432.1480.86
Independent Samples (Frozen Discriminator)73.5346.4023.681745.2753.0094.8932.1489.97

Revisions: Independent-sample validation with full and frozen selection pipelines is added in Appendix C.3.1 (L1387–1395, Table 24).


W2: Statistical Reliability of Performance Improvements.

We conduct three independent cross-machine runs and evaluate randomness in data sampling, random projections, and discriminator initialization using three seeds (41–43).

Experimental SettingMathVistaMathVerseMathVisionMME(ACC)MME(ACC+)MME(Total)MMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.32873.73806.961680.6950.5666.7029.68
Cross-Machine Runs70.57±0.0942.71±0.1329.39±0.19895.98±0.38825.85±0.321721.83±0.6350.59±0.1767.90±0.1030.89±0.06
Data Sampling70.52±0.5442.91±0.2428.95±0.32895.01±0.88825.82±0.451720.83±1.2950.63±0.1368.10±0.2130.67±0.13
Random Projection70.46±0.3242.72±0.4529.17±0.50895.95±0.37825.82±0.541721.77±0.8550.63±0.4567.87±0.1530.78±0.17
Discriminator Initialization70.49±0.1742.70±0.6629.28±0.33895.59±0.68826.24±0.531721.83±1.1550.59±0.2867.89±0.3231.11±0.41
Experimental SettingMathVistaMathVerseMathVisionMME(ACC)MME(ACC+)MME(Total)MMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.76894.92832.031726.9453.0094.6531.69
Cross-Machine Runs73.77±0.0746.50±0.0619.08±0.00902.28±0.30847.88±0.141750.16±0.4353.15±0.0695.34±0.0532.68±0.18
Data Sampling73.73±0.2846.56±0.3219.19±0.19901.98±0.53848.05±0.491750.03±0.9853.15±0.2895.34±0.2532.46±0.19
Random Projection73.80±0.2546.46±0.3019.08±0.33902.41±0.38847.91±0.501750.32±0.8453.15±0.2895.31±0.2532.46±0.50
Discriminator Initialization73.73±0.2546.49±0.3018.97±0.19902.37±0.53847.68±0.471750.05±0.9353.00±0.2995.39±0.2332.35±0.20

All settings exhibit small performance variations, supporting the stability of BiVaR. For MME, we additionally report ACC (question-level accuracy) and ACC+ (requiring both questions for an image to be correct). We acknowledge that the marginal gains on MathVerse and MMMU may reflect random fluctuations and do not claim statistical significance for these improvements.

Revisions: Repeated-run statistics, MME components, and the interpretation of small gains are provided in Appendix C.3.2 (L1436–1446, Tables 25–26).

W3:Additional Baselines Comparisons.

We conduct additional comparisons with IP-Merging and FRISM, following their original settings and tuning hyperparameters for previously untested model pairs. As FRISM has no available official code, we reproduce it based on the paper.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
BiVaR70.5342.6729.281721.7550.5667.8930.82
IP-Merging68.8042.6827.961684.1750.5668.1229.91
FRISM67.1734.6425.661684.1850.5667.3830.13
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
BiVaR73.7746.4719.131750.0853.1195.3332.58
IP-Merging68.7043.7818.091740.4153.5694.4031.92
FRISM68.9040.8218.421740.4051.6795.2432.14

BiVaR outperforms both methods on most benchmarks, particularly in mathematical visual reasoning.

Revisions: IP-Merging and FRISM are added to Section 4.1 (L286–290), with results in Table 1 and Appendix C.1.1 (L949–959, Table 9).


W4: Preservation of General Multimodal Capabilities.

(1) Subtask-Level Evaluation. We further report results on ten MME visual subtasks. As shown below, BiVaR matches or improves upon the base VLM across all subtasks on both models, confirming that the overall MME gains do not conceal subtask-level degradation.

MethodexistencecountpositioncolorposterscelebrityscenelandmarkartworkOCR
Qwen2.5-VL-7B-Instruct195.00155.00163.33190.00166.33148.53161.75188.50147.25165.00
BR2V195.00156.67160.00188.33162.24148.24158.50183.00142.75175.00
DARE195.00160.00160.00190.00161.90153.24158.25186.50147.75175.00
LED-Merging195.00156.67158.33190.00159.86149.71158.75187.00144.00172.50
SNRF200.00160.00163.33193.33166.33154.12161.25188.00147.75175.00
BiVaR200.00163.33165.00195.00166.67155.00162.00188.75148.50177.50
MethodexistencecountpositioncolorposterscelebrityscenelandmarkartworkOCR
Qwen3-VL-8B-Instruct190.00168.33148.33200.00178.57174.71151.50181.25156.75177.50
BR2V190.00163.33141.67190.00179.93172.65147.00181.00156.25180.00
DARE193.33170.00146.67195.00180.27174.12151.50180.50157.75182.50
LED-Merging193.33171.67146.67200.00180.27175.88151.00180.75157.75182.50
SNRF195.00170.00146.67195.00180.61174.41150.50180.75157.50182.50
BiVaR195.00171.67148.33200.00182.65176.18151.75181.25158.25185.00

(2) Evaluation With and Without Images. We further evaluate MME using original image-text inputs and GPT-5.4-generated captions in place of images.

ModelInputMME ACCMME ACC+MME
Qwen2.5-VL-7B-InstructImage + Question873.73806.961680.69
BiVaRImage + Question895.76825.991721.75
Qwen2.5-VL-7B-InstructCaption + Question878.82812.141690.96
BiVaRCaption + Question901.31830.571731.88
ModelInputMME ACCMME ACC+MME
Qwen3-VL-8B-InstructImage + Question894.92832.021726.94
BiVaRImage + Question902.17847.911750.08
Qwen3-VL-8B-InstructCaption + Question900.46837.031737.49
BiVaRCaption + Question907.26853.591760.85

BiVaR outperforms the base VLM under both settings. Together with the improvements on text-only GPQA in Table 1, these results suggest that BiVaR improves textual reasoning while preserving visual understanding.

(3) Non-Mathematical Visual Evaluation. We additionally evaluate BiVaR on four non-mathematical visual benchmarks: A-OKVQA, VCR, ReasonVQA, and GQA.

MethodA-OKVQAVCRReasonVQAGQA
Qwen2.5-VL-7B-Instruct86.8191.1246.0072.40
BR2V86.8489.9445.0070.80
DARE86.7891.4546.0071.20
LED-Merging86.8591.8244.0072.40
SNRF86.8791.2547.0072.00
BiVaR86.9192.2248.0073.00
MethodA-OKVQAVCRReasonVQAGQA
Qwen3-VL-8B-Instruct87.8692.3164.0071.80
BR2V87.8891.7264.0068.00
DARE87.8992.6963.0071.40
LED-Merging87.9892.0864.0071.40
SNRF88.0392.3164.0071.80
BiVaR88.0593.4964.0072.60

BiVaR achieves competitive or improved performance across both models, further supporting the preservation of general visual capabilities.

Revisions: Additional benchmarks are reported in Appendix C.1.2 (L963–971, Table 10); MME subtasks and image/caption comparisons are added in Appendix C.1.3–C.1.4 (L1048–1065, Tables 11–12).

W5: Effectiveness of Bidirectional Discrimination.

We fix the SVD ranks at 4/2 for Qwen2.5-VL/Qwen3-VL and match the number of selected neurons layer-wise across all three strategies.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQAUpdate Norm
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68—
Bidirectional69.1042.8126.971723.4250.8967.8230.8030.57
V→L Only68.7042.1725.331698.6550.3367.1829.6930.98
L→V Only68.9042.7225.001715.4550.6767.4329.2430.57
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQAUpdate Norm
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69—
Bidirectional69.4043.2318.091748.8753.4495.3932.375.19
V→L Only68.7042.6417.761735.2353.4494.7031.255.18
L→V Only69.1742.6818.091746.6452.8994.9430.805.20

The update norms remain comparable, while bidirectional discrimination outperforms or matches both unidirectional variants across all seven benchmarks. These results support the effectiveness of bidirectional selection beyond differences in neuron count or update magnitude.

Revisions: The neuron-count-controlled directional ablation and update norms are added in Appendix C.2.3 (L1222–1229, Table 17).

Reviewer_vLUoOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new framework to transfer the reasoning capabilities from an LLM to a VLM without finetuning. The proposed method first identifies the neurons that are subject to the activation distribution for both LLMs and VLMs, and ignores those neurons that only belong to the VLM or LLM distribution. Then, a low-rank weight difference is calculated and only the important rows and columns are selected. The authors conduct experiments for various model-pairs and the results show the improvements are significant.

Critical strengths

First, the investigated problem is important. How to transfer the strong reasoning capabilities from LLMs to VLMs is vital for further development of VLMs. Second, the idea of identifying the neurons that are subject to both the LLM and VLM distributions is also intersting. Third, the experimental results show significant improvement.

Critical weaknesses
  1. The neuron test method here can not validate that the matched neurons represent similar functions. Each neuron is compared with the other model’s overall distribution, rather than its matched neuron. In the experiments, the authors compare each neuron with the overall distribution of the model from the other modalities. I think a more reasonable way is to find the neurons that have similar functions, in other words, the matched neurons. Besides, the inputs to the different models are different. I think a better way is find common inputs for both the VLMs and the LLMs, and then test distributions to find the real matched neurons.

  2. The SVD step may keep information that is later removed. SVD is applied before masking, so it may discard useful information in the selected neurons while keeping information outside them. The authors should compare this order with masking before SVD and transfer without SVD.

  3. Important comparisons and testing details are missing. The paper does not compare with IP-Merging [1] or FRISM [2]. It also does not explain how the settings were chosen or report variation across random samples and projections. Fair comparisons, a separate validation set, and repeated runs are needed.

[1] Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?,NeurIPS 2025 [2] FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language Models,ICML 2026

Questions for authors

Please see the Weakness part.

5 comments

We appreciate the reviewer’s recognition of the importance of the investigated problem, the novelty of our neuron identification approach, and the significant experimental improvements. We address your concerns below.


W1: Neuron Matching and Input Consistency.

(1) Neuron Matching. Our method aims to identify neurons with cross-model distributional compatibility rather than establish one-to-one functional equivalence. To investigate explicit neuron matching, we follow Model Fusion via Retrofitting [1] and compare BiVaR with Hungarian Fusion (HF) and K-means Fusion (KF).

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQAIdentification Time (min)
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68—
BiVaR70.5342.6729.281721.7550.5667.8930.8212
HF69.1741.1627.631711.7250.5666.7329.9140
KF69.7342.4329.611720.1350.5666.9330.5866
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQAIdentification Time (min)
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69—
BiVaR73.7746.4719.131750.0853.1195.3332.5813
HF73.3345.7318.091727.2052.5694.7431.7045
KF73.6046.1118.421729.2253.1195.0431.9273

BiVaR achieves better overall performance on both Qwen2.5-VL and Qwen3-VL, while requiring substantially less identification time (12/13 min vs. 40/45 min for HF and 66/73 min for KF). These results suggest that explicit neuron matching is not necessary for effective reasoning transfer.

(2) Input Consistency. We further evaluate five input configurations, including identical inputs for both models.

MethodLLM InputVLM InputNeuron Increase (%)Retained (%)MathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct————69.0742.4326.321680.6950.5666.7029.68
BiVaR (Ours)GSM8KGeo3K (Multimodal)0.00100.0070.5342.6729.281721.7550.5667.8930.82
BiVaRGSM8KGSM8K+128.6398.0167.7740.8228.291700.7449.0066.9330.36
BiVaRGSM8KGeo3K (Text-only)+4.4085.2968.5342.3427.631705.7849.2266.8329.69
BiVaRGeo3K (Text-only)Geo3K (Multimodal)+22.4072.4468.3042.3425.991721.7049.5667.0329.24
BiVaRGeo3K (Text-only)Geo3K (Text-only)+34.3175.2469.0042.4729.281709.4949.2267.5329.69
MethodLLM InputVLM InputNeuron Change (%)Retained (%)MathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct————72.7745.4517.761726.9453.0094.6531.69
BiVaR (Ours)GSM8KGeo3K (Multimodal)0.00100.0073.7746.4719.131750.0853.1195.3332.58
BiVaRGSM8KGSM8K+27.5592.2369.3043.4415.461743.8752.8994.5031.92
BiVaRGSM8KGeo3K (Text-only)+10.1985.8569.1742.9813.821751.6453.2294.6032.14
BiVaRGeo3K (Text-only)Geo3K (Multimodal)-26.5453.6668.7042.8513.821734.1253.4494.6031.47
BiVaRGeo3K (Text-only)Geo3K (Text-only)+23.8564.8668.9342.8114.471741.5053.1194.6031.25

Using GSM8K for both models retains 98.01%/92.23% of the originally selected neurons on Qwen2.5-VL/Qwen3-VL, but introduces 128.63%/27.55% more selected neurons without improving performance. These results suggest that text-only inputs may overlook vision-specific activation patterns, leading to less effective neuron selection. In contrast, our original setting preserves multimodal information and achieves the best overall performance.

[1] Model Fusion via Retrofitting, ICML 2026.

Revisions: Neuron-matching comparisons are added in Appendix C.2.1 (L1126–1133, Table 14), and activation-input comparisons in Appendix C.2.2 (L1177–1187, Table 15).

W2: Effect of SVD and Masking Order.

Performing SVD before masking preserves the global dominant directions of weight differences while restricting updates to shared neurons, whereas masking first may disrupt global correlations.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
Global SVD → Mask (Ours)70.5342.6729.281721.7550.5667.8930.82
Mask → Local SVD69.2341.9626.641712.7449.2267.5829.91
Mask Only (w/o SVD)68.2740.2724.011673.4949.2264.9025.22
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
Global SVD → Mask (Ours)73.7746.4719.131750.0853.1195.3332.58
Mask → Local SVD72.8043.4018.421736.6253.0095.1431.70
Mask Only (w/o SVD)69.9342.7214.801720.8352.6793.1629.91

With fixed SVD ranks, our strategy consistently outperforms Mask-then-SVD across all seven benchmarks on both models. Moreover, transferring weight differences without SVD leads to performance degradation. These results validate the effectiveness of combining global low-rank decomposition with selective neuron updates.

Revisions: The SVD–masking order and no-SVD comparisons are added in Appendix C.2.6 (L1285–1290, Table 21).


W3: Baseline Comparisons and Experimental Reliability.

(1) Additional Baselines. We conduct additional comparisons with IP-Merging and FRISM, following their original settings and tuning hyperparameters for previously untested model pairs. As FRISM has no available official code, we reproduce it based on the paper. As shown below, BiVaR outperforms both methods on most benchmarks, particularly in mathematical visual reasoning.

MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
BiVaR70.5342.6729.281721.7550.5667.8930.82
IP-Merging68.8042.6827.961684.1750.5668.1229.91
FRISM67.1734.6425.661684.1850.5667.3830.13
MethodMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
BiVaR73.7746.4719.131750.0853.1195.3332.58
IP-Merging68.7043.7818.091740.4153.5694.4031.92
FRISM68.9040.8218.421740.4051.6795.2432.14

(2) Reproducibility and Stability. We conduct three independent cross-machine runs and evaluate randomness in data sampling, random projections, and discriminator initialization using three seeds (41–43).

Experimental SettingMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen2.5-VL-7B-Instruct69.0742.4326.321680.6950.5666.7029.68
Cross-Machine Runs70.57±0.0942.71±0.1329.39±0.191721.83±0.6350.59±0.1767.90±0.1030.89±0.06
Data Sampling70.52±0.5442.91±0.2428.95±0.321720.83±1.2950.63±0.1368.10±0.2130.67±0.13
Random Projection70.46±0.3242.72±0.4529.17±0.501721.77±0.8550.63±0.4567.87±0.1530.78±0.17
Discriminator Initialization70.49±0.1742.70±0.6629.28±0.331721.83±1.1550.59±0.2867.89±0.3231.11±0.41
Experimental SettingMathVistaMathVerseMathVisionMMEMMMUScienceQAGPQA
Qwen3-VL-8B-Instruct72.7745.4517.761726.9453.0094.6531.69
Cross-Machine Runs73.77±0.0746.50±0.0619.08±0.001750.16±0.4353.15±0.0695.34±0.0532.68±0.18
Data Sampling73.73±0.2846.56±0.3219.19±0.191750.03±0.9853.15±0.2895.34±0.2532.46±0.19
Random Projection73.80±0.2546.46±0.3019.08±0.331750.32±0.8453.15±0.2895.31±0.2532.46±0.50
Discriminator Initialization73.73±0.2546.49±0.3018.97±0.191750.05±0.9353.00±0.2995.39±0.2332.35±0.20

All settings exhibit small performance variations, confirming the stability of BiVaR. We also clarify that discriminator training data are disjoint from evaluation benchmarks, avoiding test-data leakage.

Revisions: IP-Merging and FRISM are included in Section 4.1 (L286–290) and Tables 1 and 9. Repeated runs and data separation are reported in Appendix C.3.2 (L1436–1449, Tables 25–26).

AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewer vLUo,

We sincerely appreciate your constructive feedback. We have substantially revised the manuscript and conducted additional experiments to address your concerns. We summarize our key motivation and improvements below.

Our central motivation is that transferable reasoning components can be identified through shared activation behavior across models, without requiring explicit one-to-one neuron matching. From an ID–OOD discrimination perspective, if a discriminator cannot reliably distinguish whether a neuron's activation behavior belongs to the LLM or VLM distribution, this suggests shared cross-model response patterns rather than model-specific behavior. Such distributional compatibility provides a principled basis for identifying transferable reasoning components. Based on this insight, BiVaR identifies transferable neurons through bidirectional distribution discrimination and performs adaptive low-rank merging to enhance reasoning while preserving multimodal capabilities.

Our major experimental improvements include:

  • Neuron Matching and Input Consistency: We compared BiVaR with explicit matching methods (HF/KF), achieving better overall performance with substantially less identification time (12/13 vs. 40–73 minutes). We also evaluated five input configurations, showing that our original multimodal-aware setting achieves the strongest overall results.
  • SVD and Masking Order: We compared SVD-before-masking with masking-before-SVD and merging without SVD. Our strategy consistently outperforms both alternatives across two model pairs and seven benchmarks.
  • Additional Baselines and Reliability: We added comparisons with IP-Merging and FRISM, outperforming both on most benchmarks. Repeated runs across different machines, data samples, random projections, and discriminator initializations further demonstrate stable performance.

These additional results strengthen the motivation and experimental evidence for BiVaR. All revisions have been incorporated into the updated manuscript, and we will publicly release our code to facilitate reproducibility.

We hope these revisions address your concerns. If any concerns remain unresolved, we would greatly appreciate your feedback and will promptly provide further clarifications or additional experiments.

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank the reviewers for their constructive feedback. We have addressed the discussion concerns, conducted extensive additional experiments, and strengthened the manuscript. Below, we summarize our contributions, revisions, and clarifications.

In this paper, we propose BiVaR, a training-free framework that transfers reasoning capabilities from LLMs to VLMs while largely preserving existing multimodal capabilities: Core Contributions. BiVaR is a training-free framework that transfers reasoning capabilities from LLMs to VLMs while largely preserving existing multimodal capabilities:

  • Distribution-guided selection: Identifies transferable reasoning components through cross-model distributional consistency, avoiding the limitations of point-wise activation matching.
  • Adaptive transfer: Combines bidirectional neuron selection and adaptive low-rank merging to transfer reasoning while minimizing interference with multimodal knowledge.
  • Consistent effectiveness: Improves mathematical and text-only reasoning across four VLM–LLM pairs while preserving or improving general multimodal performance.

Following the reviewers' suggestions, we conducted extensive additional experiments to strengthen our paper:

  • Broader evaluation: Six reasoning and multimodal benchmarks, fine-grained MME results, image/caption comparisons, and additional merging baselines (Appendix C.1).
  • Transfer quantification: Reasoning Gap Closed (RGC), showing that BiVaR recovers a substantial portion of the source–target reasoning gap (Appendix C.1.5).
  • Neuron-selection validation: Controlled comparisons with HF/KF matching and SNRF, supporting distribution-guided selection (Appendix C.2.1, C.2.4).
  • Method ablations: Further studies of bidirectional selection, thresholding, and adaptive low-rank merging (Appendix C.2.3, C.2.5–C.2.8).
  • Robustness: Independent-sample validation, cross-machine/multi-seed experiments, and sensitivity analyses (Appendix C.3).

We would like to clarify three important points regarding the remaining concerns raised by Reviewers eKG6 and Rcek.

  • Research scope. BiVaR aims to transfer mathematical reasoning, not simultaneously improve visual perception, factual knowledge, or every capability tested by general-purpose benchmarks. Requiring large gains across all such tasks extends beyond this objective. Our results show 42.86%–77.27% RGC while largely preserving multimodal capabilities, directly supporting the intended transfer.
  • Model scale. BiVaR improves 7B/8B VLMs without training or increasing model size, across multiple model families. The request for larger-model evaluation arose only approximately 20 hours before the rebuttal deadline, whereas a complete 32B evaluation would take more than 20 hours on eight NVIDIA A100 GPUs. Nevertheless, we evaluated the Qwen2.5-VL-32B baseline: our BiVaR-enhanced 7B model approaches its performance, trailing by only 0.50 points on MathVista and 2.96 on MathVision. This demonstrates the practical value of efficient transfer without claiming a direct 32B merging result.
  • Degradation in competing methods. Degradation in baselines such as SNRF reflects a limitation BiVaR addresses: unreliable neuron selection can introduce merging interference. Unlike SNRF's point-wise activation matching, BiVaR uses distributional consistency and bidirectional discrimination, outperforming competing methods under identical merging procedures. BiVaR-only neurons outperform SNRF-only neurons across both evaluated VLMs, further supporting our motivation.

We believe these experiments and clarifications directly address the central methodological and empirical concerns and strengthen BiVaR as a training-free approach to bridging the LLM–VLM reasoning gap while preserving multimodal capabilities.

We respectfully ask the ACs and PCs to consider this additional evidence in their final assessment.

Thank you for your time and thoughtful consideration.

Best regards,

Authors of Paper #4985

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.