acceptodds
Under review as a conference paper at ICLR 2027

Layer-Specialized Model Merging for Training-Free Cross-Modal Reasoning Transfer

Abstract

Text-only reasoning LLMs are strong multi-step problem solvers, but transferring this behavior to multimodal LLMs (MLLMs) usually requires costly multimodal post-training and scarce verifiable reasoning supervision. We study a training-free alternative: fusing a text-only reasoning decoder with a homologous MLLM decoder while keeping the MLLM visual encoder and projector unchanged. This setting is more delicate than standard model merging because the fused decoder must integrate reasoning behavior from a text-only model while continuing to interpret visual inputs. We introduce (Attention-calibrated Norm-Orthogonal merging), a layer-specialized fusion method for cross-modal reasoning transfer. represents the two decoders as task vectors relative to a shared base model and computes closed-form, label-free fusion weights from task-vector salience. To account for depth-dependent visual-token use, it calibrates these weights with the MLLM's layer-wise visual-attention decay, using a continuous prior that favors the vision branch in shallow layers and the reasoning branch in deeper layers. Across homologous MLLM–reasoner pairs, improves reasoning-heavy multimodal benchmarks without gradient updates or labeled multimodal reasoning data. We also evaluate the boundary of this transfer: MME-RW perception results and MathVista performance reveal task-dependent tradeoffs between reasoning gains and visual capabilities. Together, the results suggest that layer-specialized merging is a practical route for training-free cross-modal reasoning transfer within the studied homologous model families.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merge, LLM, MLLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ecBqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ANO, a training-free method for merging a text-only reasoning model with a compatible multimodal model. It determines layer-wise merging coefficients using task-vector norms and a depth-dependent prior estimated from visual attention, while retaining the original visual encoder and projector. Experiments across six compatible model pairs show improvements in average reasoning performance, accompanied by regressions on some visual tasks.

Critical strengths
  1. Practical motivation. Reusing existing reasoning checkpoints without additional multimodal training is a useful direction, particularly when training resources or labeled data are limited.

  2. Evaluation across multiple model pairs. The experiments cover different model families and scales, providing evidence that the approach can improve average performance beyond a single checkpoint.

  3. Acknowledgment of trade-offs. The paper reports visual-capability regressions and appropriately distinguishes the optimality of its coefficient-space objective from optimality for the original model losses.

Critical weaknesses
  1. Insufficient justification for the proposed weighting signals. Lower direct attention to visual tokens does not necessarily indicate lower dependence on visual information, which may already be integrated into other representations. Similarly, task-vector magnitude does not necessarily measure capability importance. The coefficient-space derivation formalizes the weighting rule but does not establish that these signals reliably identify which adaptations should be retained at each layer.

  2. The figures and submission presentation need clarification. Figures 1 and 2 substantially overlap without showing how measured attention and task-vector energies determine actual merging coefficients. In Figure 4, the colored curves and shaded regions are insufficiently explained. Additionally, the OpenReview abstract contains an unresolved \method{} placeholder that should be replaced with the method name.

  3. Important implementation details are insufficiently specified for assessing reproducibility and baseline fairness. Appendix A describes the general experimental protocol, but does not fully specify several consequential settings, including the calibration sample count, exact calibration prompts, generated-query selection, the numerical value of the prior endpoint parameter, and detailed baseline hyperparameters. These omissions are particularly concerning because OptMerge, WUDI, and DOGE perform substantially worse than the original model under the reported configurations.

  4. Pre-merging visual attention may not predict the visual cost of merging. ANO uses the source model’s visual-attention profile to guide layer-wise fusion. However, lower direct attention to image tokens does not necessarily imply that a layer is less important for visual processing, since visual information may already be integrated into text-token representations or the residual stream. Moreover, merging changes attention patterns and intermediate representations, so attention statistics measured before merging may not reliably predict the resulting visual degradation. The observed concentration of visual attention in shallow layers therefore provides insufficient justification for assigning greater reasoning-branch weights to deeper layers. Retaining the visual encoder and projector alone does not ensure that the fused decoder remains compatible with their outputs. Furthermore, and this is also important, I am curious whether this method is compatible with the flash-attention framework (https://github.com/dao-ailab/flash-attention).

Questions for authors

Q1. Could the authors report the number of calibration samples, the exact calibration prompts, the selection of generated queries, the numerical values ​​of the prior endpoint parameters, and the detailed baseline hyperparameters?

Q2. Can layer-wise interventions or sensitivity analyses demonstrate that visual attention and task-vector norms predict the effects of merging on reasoning and visual performance?

Q3. Could the authors conduct controlled layer-wise replacement experiments to test whether layers with lower visual attention actually tolerate greater contributions from the reasoning branch without degrading visual performance? Does this relationship hold consistently across model families?

Q4. If feasible, could you please include experiments using Qwen3.5, Qwen3.6, or Qwen3.8? Even adding just one of these would make the paper more convincing.

10 comments

We thank Reviewer ecBq for the thorough, constructive review and for appreciating our practical motivation, multi-family evaluation, and explicit reporting of visual trade-offs and mathematical boundaries. We address every weakness and question below.

Do lower-attention layers tolerate stronger reasoning contributions across families, and how do both signals contribute? (W1, W4 / Q2, Q3)

Yes. Deeper layers with lower direct visual attention tolerate substantially stronger reasoning-branch contributions than shallow high-attention layers across both InternVL3.5-4B and Qwen2.5-VL-7B, and combining both salience signals (attention prior and task-vector norm) outperforms every single-signal and step-cutoff alternative. Because deeper layers still carry visual features integrated into the residual stream and Eq. 5 serves as a coefficient-normalization rule rather than a model-loss guarantee, ANO uses a smooth, strictly positive prior wV(l)∈[ϵw,1−ϵw]w_V^{(l)} \in [\epsilon_w, 1-\epsilon_w] (ϵw=0.09\epsilon_w=0.09) rather than hard layer replacement.

  1. Controlled shallow-vs.-deep replacement across two model families (Q3). Holding the reasoning weight 1−λV1-\lambda_V identical, we compare All-Layer linear merging (θ(l)=λVθV(l)+(1−λV)θR(l)\theta^{(l)} = \lambda_V\theta_V^{(l)} + (1-\lambda_V)\theta_R^{(l)} across all LL blocks) against Shallow-Protected merging (keeping high-attention shallow blocks l≤5l \le 5 untouched at 100% θV(l)\theta_V^{(l)} and merging only deeper blocks l>5l > 5), where MME-RW Gain (Shallow-Protected − All-Layer) isolates the perceptual retention gained by protecting shallow visual blocks:
PairλV\lambda_VAll-Layer MMMUAll-Layer MME-RWShallow-Protected MMMUShallow-Protected MME-RWMME-RW Gain
4B0.167.848.867.852.4+3.6
4B0.567.152.067.553.9+1.9
4B0.963.354.263.254.1−0.1
7B0.554.252.154.455.0+2.9

Unfused source MME-RW is 56.7 on 4B and 57.4 on 7B (BRTV's step-cutoff [1] at λV=0.85\lambda_V=0.85 on 4B scores 66.1 MMMU / 53.9 MME-RW), while Full ANO reaches 69.6 MMMU / 55.0 MME-RW on 4B and 55.0 MMMU / 56.3 MME-RW on 7B, outperforming every row above on both axes.

  1. Empirical role of visual attention and task-vector norms (Q2).
    • Shallow-vs.-deep tolerance across families: Under substantial reasoning injection (λV∈{0.1,0.5}\lambda_V \in \lbrace 0.1, 0.5 \rbrace), merging all layers causes steep MME-RW drops (48.8–52.0 on 4B; 52.1 on 7B), whereas restricting the exact same reasoning weights to l>5l > 5 recovers +1.9 to +3.6 MME-RW points on 4B and +2.9 points on 7B without hurting MMMU (at minimal 10% injection λV=0.9\lambda_V=0.9, both settings stay near 54.1–54.2).
    • Residual-stream sensitivity supports smooth positive weights: Even with l≤5l \le 5 untouched, heavy deep-layer injection (λV=0.1\lambda_V=0.1) still loses 4.3 MME-RW points (52.4 vs. 56.7), consistent with deeper layers still processing integrated visual features.
    • Multiplicative interaction of both signals: On 4B, Prior-only (λt(l)=wt(l)\lambda_t^{(l)}=w_t^{(l)} without norm scaling) scores 58.5 5-task Avg. (67.2 / 62.7 / 72.6 / 49.2 / 41.0, below Uniform 61.8), Reversed / Random schedules drop to 55.4 / 56.3 (Figure 3), and Norm-only (w/o prior) scores 62.4 Avg. / 53.7 MME-RW on 4B and 45.0 Avg. (54.5 / 38.0 / 68.3 / 26.9 / 37.1) / 55.1 MME-RW on 7B. Combining both signals multiplicatively in Full ANO reaches 64.0 Avg. / 55.0 MME-RW on 4B and 45.7 Avg. / 56.3 MME-RW on 7B, outperforming both Norm-only and DARE [2] across all 12 matched evaluations.

Complete calibration parameters, endpoint ϵw\epsilon_w, and baseline hyperparameters (W3 / Q1)

We have added all exact calibration and baseline parameters to Appendix A of the revised manuscript:

  1. Calibration settings. N = 1,000 unlabeled MSCOCO val2017 images; prompt "Describe this image in detail."; generated-query selection: the first 3030 generated response tokens averaged across all attention heads (Eq. 2); log-fit stabilizer ϵ=10−6\epsilon = 10^{-6} (R2=0.90R^2=0.90, RMSE=0.019\text{RMSE}=0.019 on InternVL3.5-4B-MPO); prior endpoint ϵw=0.09\epsilon_w = 0.09 in Eq. 3 (wV(1)=0.91w_V^{(1)}=0.91, wV(L)=0.09w_V^{(L)}=0.09; resulting λR(l)\lambda_R^{(l)} rises from 0.08 in layer 1 to 0.92 in layer 36). Task vectors and norms are computed in float32 and saved in bfloat16. Source and merged models share identical decoding settings (enable_thinking=True on Qwen3 pairs, max_new_tokens=32,768, temperature=0.6, top_p=0.95, top_k=20).
  2. Baseline hyperparameters (Appendix A.4). Weight averaging uses λV=0.5\lambda_V=0.5 (61.8); Task arithmetic [3] uses official multi-vector scaling θ0+0.3τV+0.3τR\theta_0 + 0.3\tau_V + 0.3\tau_R (61.9); TIES [4] uses top-20% density and scaling 0.30.3 (62.1); DARE [2] uses drop rate p=0.9p=0.9 with 10×10\times rescaling and 0.5/0.50.5/0.5 averaging (62.2); SLERP uses depth-interpolated t∈[0,1]t \in [0,1] (61.9). OptMerge [5], WUDI [6], and DOGE [7] (48.8–49.8) were evaluated at their official data-free default configurations, which apply unnormalized unit task-vector addition (τV+τR\tau_V+\tau_R without λV+λR=1\lambda_V+\lambda_R=1); in cross-modal pairs where deep-layer reasoning task vectors have much larger norms, unnormalized unit addition over-injects τR\tau_R.

Is ANO compatible with FlashAttention? (W4)

Yes, the fused ANO model is fully compatible with FlashAttention during all inference and deployment. Only the one-time offline calibration pass over the 1,0001,000 unlabeled images uses explicit attention to read Al,h,qkA_{l,h,qk} and fit the scalar slope α^\hat{\alpha}. Once the closed-form weights are merged, the resulting checkpoint requires no runtime attention scores, preserves the exact architecture and bfloat16 tensor shapes of the source MLLM, and runs natively with FlashAttention in Transformers, vLLM, and SGLang.

Clarifying Figures 1, 2, and 4 and fixing the OpenReview \method{} placeholder (W2)

We apologize for the unexpanded \method{} macro in the OpenReview form abstract and the omitted figure notes, and have updated all items in the revised manuscript:

  1. OpenReview abstract. We replace the unexpanded \method{} placeholder in the OpenReview metadata box with ANO.
  2. Figures 1 and 2. Figure 1 presents the high-level conceptual motivation, while Figure 2 and Appendix D report the measured values on InternVL3.5-4B-MPO (L=36L=36): fitted decay α^=0.108\hat{\alpha}=0.108 (R2=0.90R^2=0.90, RMSE=0.019\text{RMSE}=0.019), prior floor ϵw=0.09\epsilon_w=0.09 (wV(1)=0.91→wV(36)=0.09w_V^{(1)}=0.91 \to w_V^{(36)}=0.09), and resulting reasoning weights λR(l)\lambda_R^{(l)} rising from 0.08 to 0.92.
  3. Figure 4 curves and shading. Each colored curve in Figure 4 plots the layer-wise visual-attention ratio for one of the initial generated response token positions, and the shaded band indicates ±1 SD\pm 1\text{ SD} across calibration images; this explanation is now stated explicitly in Figure 4's caption.

Applicability to Qwen3.5, Qwen3.6, and Qwen3.8 (Q4)

ANO targets settings where a vision-language model and a stronger text-only reasoning specialist are trained separately from a shared base, as in Qwen2.5 and Qwen3 (e.g., InternVL3.5-4B-MPO + Qwen3-4B-Thinking, or InternVL2.5-38B + QwQ-32B). By contrast, the official Qwen3.5, Qwen3.6, and Qwen3.8 releases are trained natively as unified multimodal reasoning models with built-in thinking mode on a hybrid Gated DeltaNet and full-attention backbone, so there is no separate stronger official text-only reasoning checkpoint on the same base to serve as a donor.

[1] Chen et al., Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging, ICML 2025.

[2] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.

[3] Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023.

[4] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

[5] Wei et al., Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging, 2025.

[6] Cheng et al., Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors, ICML 2025.

[7] Wei et al., Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent, ICML 2025.

Thank you for clarifying the compatibility requirements. However, this raises a concern about the practical scope of ANO. Given that your method requires a compatible pair consisting of a vision-language model and a stronger text-only reasoning model derived from the same base, what practical use cases remain as newer models increasingly integrate multimodal and reasoning capabilities? Could you clarify when a user would benefit from applying ANO rather than directly using a stronger unified multimodal model?

If the broader objective is to integrate capabilities from multiple models, could you also discuss ANO’s practical advantages and limitations relative to distillation-based approaches such as MOPD? Since these approaches transfer capabilities through model outputs rather than parameter alignment, they may offer greater flexibility for heterogeneous architectures.

Reviewer_qxVNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents ANO, a training-free layer-specialized model merging approach designed to transfer reasoning capabilities from text-only reasoning models to compatible multimodal models. It computes per-layer fusion weights by combining the parameter salience of task vectors with a visual attention prior calibrated on unlabeled images, while retaining the original visual frontend. Experiments on multiple reasoning benchmarks demonstrate that ANO yields improved average reasoning performance. Comparative evaluations and ablation studies validate the effectiveness of its merging design, albeit with visual capability degradation observed on certain tasks.

Critical strengths

1.The method is concise with low transfer cost. Leveraging existing models and unlabeled images, ANO determines layer-wise fusion weights via closed-form computation and requires no additional gradient-based training or annotated multimodal reasoning data. 2.Experiments demonstrate cross-model transfer gains. Average reasoning scores improve across six model pairs. Baseline comparisons with fixed source models and ablation studies further support the incremental value of the attention prior.

Critical weaknesses

1.The validation scope for the core design is limited. Major ablations on the attention prior are conducted only on a single model pair. Moreover, the modest performance gains lack uncertainty analysis. The cross-model effectiveness and stability of this prior remain to be verified. 2.The evaluation setup is insufficiently clear. It is not specified whether thinking is enabled for the source vision-language model, nor are the concrete generation configurations provided. This makes it hard to assess whether the observed merging gains are affected by differences in inference settings. 3.The configuration of linear merging baselines lacks transparency. The fusion coefficients and parameter selection procedure are not described. It remains unclear how much advantage ANO holds over properly tuned simple merging methods.

Questions for authors

1.Is thinking mode enabled for the source VL model? Please specify the prompts, generation budget and sampling settings adopted for both the source model and the merged model. 2.How are coefficients determined for the linear merging baseline? Was coefficient selection performed, and does ANO retain advantages over the baseline after proper coefficient tuning? 3.Can the incremental gains from the attention prior be reproduced on another model pair? Could the authors add key ablations for this prior and report uncertainty for the main performance differences?

4 comments

We thank Reviewer qxVN for the positive assessment and for highlighting ANO's concise, training-free closed-form design and cross-model reasoning gains across six pairs. We address all three questions below.

Is thinking mode enabled for the source VLM, and are decoding settings strictly matched? (W2 / Q1)

Yes. Within every model pair, the source VLM and all merged checkpoints are evaluated under strictly identical prompts, thinking settings, generation token budgets, and sampling parameters:

  1. Thinking-mode parity. On the Qwen3-based pairs (InternVL3.5-4B-MPO / Qwen3-4B-Thinking and LLaVA-OneVision1.5-4B / Qwen3-4B-Thinking), thinking mode (enable_thinking=True) is enabled identically for both the unfused source VLM and all merged checkpoints using the exact same chat template and benchmark prompts. On the remaining four pairs (Qwen2.5-VL-7B, Idefics3-8B, NVILA-15B, InternVL2.5-38B), where the source VLMs have no separate thinking toggle, both the source VLM and ANO receive the identical benchmark prompt, image preprocessing, and answer parser.
  2. Matched generation budget, sampling settings, and negative control. Within each pair, source and merged models share max_new_tokens = 32,768 and identical sampling parameters (temperature = 0.6, top_p = 0.95, top_k = 20; independent sampling seeds account for the small run-to-run dispersion in Table 7). Moreover, in Figure 3, Reversed Prior (Avg. 55.4) and Random Prior (Avg. 56.3) merge the exact same Qwen3-4B-Thinking donor under the exact same thinking and decoding configuration as ANO (64.0), yet score below the 57.0 source VLM, confirming that gains arise from ANO's layer-wise fusion weights rather than inference settings.

How were linear merging coefficients set, and does ANO outperform properly tuned LERP? (W3 / Q2)

In Table 2a, Weight averaging (61.8) used fixed symmetric weights θF=0.5θV+0.5θR\theta_F = 0.5\theta_V + 0.5\theta_R without downstream search to match ANO's label-free protocol, while Task arithmetic (61.9) [1] used official multi-vector scaling θ0+0.3τV+0.3τR\theta_0 + 0.3\tau_V + 0.3\tau_R. Even after sweeping all nine global linear coefficients λV∈{0.1,0.2,…,0.9}\lambda_V \in \lbrace 0.1, 0.2, \dots, 0.9 \rbrace for θF(λV)=λVθV+(1−λV)θR\theta_F(\lambda_V) = \lambda_V\theta_V + (1-\lambda_V)\theta_R, label-free ANO retains a clear advantage on both reasoning and visual retention:

Global VLM Weight λV\lambda_V0.10.20.30.40.50.60.70.80.9ANO
4B MMMU67.867.066.567.067.166.062.761.463.369.6
4B MME-RW48.850.151.451.952.051.353.753.754.255.0

Selecting the best λV\lambda_V from this grid grants LERP privileged downstream test labels unavailable to ANO. Even against the best test-selected grid points across all nine candidates, label-free ANO (69.6 MMMU, 55.0 MME-RW) beats the best reasoning LERP point (λV=0.1\lambda_V=0.1: 67.8 MMMU, 48.8 MME-RW) by +1.8 on MMMU and +6.2 on MME-RW, and beats the best visual LERP point (λV=0.9\lambda_V=0.9: 63.3 MMMU, 54.2 MME-RW) by +6.3 on MMMU and +0.8 on MME-RW. Likewise on 7B, equal LERP (λV=0.5\lambda_V=0.5) scores 54.2 / 52.1 vs. ANO 55.0 / 56.3. Lowering a single global λV\lambda_V to strengthen deep reasoning inevitably overwrites shallow visual layers, whereas ANO's depth-calibrated weights (λR(1)=0.08→λR(36)=0.92\lambda_R^{(1)}=0.08 \to \lambda_R^{(36)}=0.92) preserve 92% of the MLLM's early visual features in layer 1 while scaling reasoning weight up to 92% in layer 36.

Do the incremental gains from the attention prior hold on a second model pair, and how stable are they? (W1 / Q3)

Yes. Evaluating ANO w/o prior (Norm-only) (λt(l)=∥τt(l)∥22/(∥τV(l)∥22+∥τR(l)∥22)\lambda_t^{(l)}=\lVert\tau_t^{(l)}\rVert_2^2 / (\lVert\tau_V^{(l)}\rVert_2^2+\lVert\tau_R^{(l)}\rVert_2^2), removing the attention prior while keeping task-vector norm weighting) across all five reasoning benchmarks and MME-RW on Qwen2.5-VL-7B / OpenThought3-7B reproduces the positive contribution of the attention prior on every single benchmark:

PairMethodMMMUMMMU-ProMathVistaMathVisionWeMathAvg.MME-RW
7BSource VLM53.936.367.825.435.243.757.4
7BDARE [2]54.236.768.425.936.044.253.4
7BANO w/o prior54.538.068.326.937.145.055.1
7BFull ANO (Ours)55.038.969.027.538.045.756.3
7BPrior Gain+0.5+0.9+0.7+0.6+0.9+0.7+1.2
4BSource VLM65.460.171.748.039.857.056.7
4BDARE [2]68.061.276.254.950.862.252.3
4BANO w/o prior66.663.176.954.950.362.453.7
4BFull ANO (Ours)69.665.777.655.151.964.055.0
4BPrior Gain+3.0+2.6+0.7+0.2+1.6+1.6+1.3
  1. Consistent prior gains across both pairs. On Qwen2.5-VL-7B / OpenThought3-7B, adding the attention prior over ANO w/o prior improves all five reasoning benchmarks (+0.5 to +0.9, raising Avg. from 45.0 to 45.7, compared with 44.2 for DARE and 43.9 for TIES [3]: 54.0 / 36.4 / 67.9 / 25.6 / 35.5) while cutting MME-RW perceptual loss by more than half (−2.3 →\to −1.1, +1.2 points). On 4B, Prior-only (λt(l)=wt(l)\lambda_t^{(l)}=w_t^{(l)} without norm scaling) scores 58.5 Avg. (67.2 / 62.7 / 72.6 / 49.2 / 41.0), confirming that both signals are needed together.
  2. Run-to-run uncertainty. Across 3 independent runs (temperature=0.6, Table 7), ANO's mean ± sample SD is 54.9±0.21 / 38.8±0.26 / 69.0±0.25 / 27.5±0.06 / 38.0±0.10 on 7B (45.6±0.17 Avg., where the 0.17 SD is small relative to the +0.7 and +1.5 aggregate gains), 69.53±0.12 / 65.60±0.10 / 77.53±0.06 / 55.07±0.15 / 51.83±0.12 (63.91 mean) on 4B, and 34.5±0.08 Avg. on 8B. Across all 12 matched pair-benchmark evaluations on 4B and 7B, Full ANO achieves higher scores than both ANO w/o prior (12/12 wins) and DARE (12/12 wins).

[1] Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023.

[2] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.

[3] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

The authors' response has addressed my concerns, I will raise the score.

We sincerely thank Reviewer qxVN for the careful re-evaluation and for confirming that all concerns have been addressed.

Warm regards,

Submission 40319 Authors.

Reviewer_Fwo1OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ANO represents a source MLLM and a homologous text reasoner as task vectors relative to a shared base. It combines squared task-vector norms with a visual-attention-derived depth prior to determine layer-wise fusion weights while retaining the visual encoder/projector. The prior is fitted to attention decay on unlabeled images without gradient training. Five-task averages improve over the source for six compatible checkpoint pairs; matched-checkpoint merging comparisons cover two pairs. The paper explicitly reports perception, grounding, and captioning losses, as well as some MathVista regressions.

Critical strengths
  1. The construction is simple and practical, with evidence beyond a single model. Table 1 covers models from 4B to 38B. Under fixed donors/sources, Table 2 reports ANO64.0 versus DARE62.2 at 4B and ANO45.7 versus DARE44.2 at 7B. The incremental contribution over merging is 1.8/1.5 points; the full seven-point gain over the source should not all be attributed to the new fusion rule.
  2. Visual costs and mathematical boundaries are reported honestly. Table 3 and Appendix E show MME-RW, RefCOCO, and CIDEr regressions. Appendix B separates an exact coefficient-objective solution from original model losses, avoiding the claim that Euclidean orthogonality implies independent capabilities. These limitations improve credibility and are not automatic grounds for a penalty.
Critical weaknesses
  • Bring Reason to Vision, ICML 2025 already studies text-reasoning transfer into VLMs and early-layer perception versus middle/later-layer reasoning. Training-free cross-modal transfer and depth specialization are therefore not unique to this work. The paper cites that study; the concern is not a missing citation, but a clearer correspondence to its concrete layer-merging configurations.
  • Generic merging already improves the 4B source from 57.0 to 61.8–62.2. ANO64.0 is an increment, not the first demonstration that the route works. The no-prior gain of 62.4→64.0 and approximately 0.5-point advantage over a tuned exponential prior on p. 8 better identify the specific contribution.
  • Figure 3 lacks run uncertainty, while Table 7 repeats only ANO and cannot establish paired improvements. Full-scale reruns of every method are unnecessary, but item-level paired intervals against the closest schedule and strongest generic merge are needed.
  • Equation 5 is a chosen quadratic coefficient objective. Substituting lambda_R=1-lambda_V gives derivative -2s_V+2(s_V+s_R)lambda_V and solution s_V/(s_V+s_R). This correct algebra does not prove that squared norm measures importance or that the attention prior optimizes model loss. The manuscript acknowledges this, so the design should primarily be judged empirically.

W2: Implementation and calibration records do not sufficiently support reproducible small gains or the breadth of comparisons

  • Appendices A.2–A.3 mostly describe procedural principles. Actual calibration image counts/IDs, prompts, generated query positions, epsilon_w, precision, source revisions, and merged tensor lists are needed.
  • Appendix A.4 lists baseline families without concrete TIES trimming, DARE drop/rescale, interpolation/scaling, or OptMerge/WUDI/DOGE hyperparameters. The latter three perform far below the source, roughly 49 versus 57. Limiting the claim to tested configurations is reasonable, but does not establish comparison fairness.
  • Six pairs are compared with their sources; head-to-head merging comparisons cover only two. Evidence supports relative gains on those two pairs, not superiority over merging algorithms across all six.
  • Attention-to-prior mapping is empirical. Fitted slopes alone are insufficient; fit residuals and final layer-wise lambdas would help. Norm-only, prior-only, and the closest layer-merging controls should be compared under equivalent calibration access.
Questions for authors

SEE weakness

2 comments

We thank Reviewer Fwo1 for the precise technical review and for noting ANO's practical label-free construction, six-pair breadth (4B–38B), matched-checkpoint gains over DARE at 4B (+1.8) and 7B (+1.5), and our transparent reporting of visual trade-offs and mathematical boundaries. We address every point in W1 and W2 below.

Algorithmic and empirical correspondence with Bring Reason to Vision (ICML 2025) (W1.1)

While Bring Reason to Vision (BRTV; Chen et al., ICML 2025 [1]) studies training-free cross-modal transfer using downstream-supervised scalar or cutoff selection, ANO introduces a closed-form, downstream-label-free per-layer allocation rule: BRTV selects a global scalar λV∈{0.80,0.85,0.90}\lambda_V \in \lbrace 0.80, 0.85, 0.90 \rbrace using labeled MathVista scores (or uses a hard step-cutoff keeping shallow layers l≤5l \le 5 fixed at θV\theta_V), whereas ANO derives continuous per-layer weights from unlabeled visual-attention decay and per-layer task-vector norms. Evaluating BRTV's concrete global and step-cutoff configurations directly on our primary 4B pair (InternVL3.5-4B-MPO / Qwen3-4B-Thinking) shows that ANO outperforms every configuration on both reasoning and visual retention:

Method (4B Pair)LabelsMMMUMME-RWΔ\DeltaMME-RW
Source MLLM (λV=1.0\lambda_V=1.0)No65.456.70.0
Equal linear (λV=0.5\lambda_V=0.5)No67.152.0−4.7
BRTV global (λV=0.80\lambda_V=0.80) [1]Yes61.453.7−3.0
BRTV global (λV=0.90\lambda_V=0.90) [1]Yes63.354.2−2.5
Best global LERP (λV=0.10\lambda_V=0.10)Yes67.848.8−7.9
BRTV cutoff (λV=0.85\lambda_V=0.85) [1]Yes66.153.9−2.8
Tuned cutoff (λV=0.50\lambda_V=0.50)Yes67.553.9−2.8
Tuned cutoff (λV=0.10\lambda_V=0.10)Yes67.852.4−4.3
ANO (Ours)No69.655.0−1.7

Conservative λV∈{0.80,0.90}\lambda_V \in \lbrace 0.80, 0.90 \rbrace retains more vision (53.7–54.2) but under-transfers reasoning (61.4–63.3), while aggressive global LERP (λV=0.10\lambda_V=0.10) reaches 67.8 MMMU at a severe visual cost (48.8, −7.9). Keeping l≤5l \le 5 at θV\theta_V recovers +1.9 to +3.6 MME-RW points (52.0 →\to 53.9 and 48.8 →\to 52.4), yet label-free ANO (69.6 MMMU, 55.0 MME-RW) surpasses every global and step-cutoff configuration on both axes, as well as Figure 3's downstream-tuned exponential schedule (69.0 MMMU).

Incremental contribution, Eq. 5 scope, salience ablations across 4B and 7B, and uncertainty (W1.2, W1.3 & W1.4)

Yes, both salience components (attention prior and task-vector norm) deliver clear, complementary empirical gains over generic merging. Generic merging (57.0 →\to 61.8–62.2 on 4B; 43.7 →\to 43.9–44.2 on 7B) provides baseline donor transfer, while Eq. 5 serves as a closed-form quadratic normalization rule (λV=sV/(sV+sR)\lambda_V = s_V/(s_V+s_R)) rather than a guarantee of model-loss optimality; ANO's design is therefore evaluated directly by its incremental gains over matched baselines. Moreover, in Figure 3, MMMU-tuned Exponential + Norm (63.4) selects its decay parameter using labeled downstream MMMU scores, whereas ANO uses zero downstream benchmark labels. Matching a downstream-tuned schedule without labels would already validate our rule; ANO exceeds all label-free baselines (DARE 62.2 [2] by +1.8, Norm-only 62.4 by +1.6) as well as MMMU-tuned Exponential (64.0 vs. 63.4; +0.54 across per-task scores: 63.98 vs. 63.44).

Below are the full 5-task reasoning scores on 4B and on the second controlled pair (7B: Qwen2.5-VL-7B / OpenThought3-7B), isolating Prior-only (λt(l)=wt(l)\lambda_t^{(l)}=w_t^{(l)}, attention prior without norm scaling) and Norm-only (ANO w/o prior) (λt(l)=∥τt(l)∥22/(∥τV(l)∥22+∥τR(l)∥22)\lambda_t^{(l)}=\lVert\tau_t^{(l)}\rVert_2^2 / (\lVert\tau_V^{(l)}\rVert_2^2+\lVert\tau_R^{(l)}\rVert_2^2), task-vector energy without attention prior):

PairMethodMMMUMMMU-ProMathVistaMathVisionWeMathAvg.
4BSource MLLM65.460.171.748.039.857.0
4BUniform (λV=0.5\lambda_V=0.5)67.160.576.854.150.461.8
4BDARE [2]68.061.276.254.950.862.2
4BPrior-only (λt(l)=wt(l)\lambda_t^{(l)}=w_t^{(l)})67.262.772.649.241.058.5
4BNorm-only (w/o prior)66.663.176.954.950.362.4
4BMMMU-tuned Exp. + Norm69.064.777.155.051.463.4
4BFull ANO (Ours)69.665.777.655.151.964.0
7BSource MLLM53.936.367.825.435.243.7
7BTIES [3]54.036.467.925.635.543.9
7BDARE [2]54.236.768.425.936.044.2
7BNorm-only (w/o prior)54.538.068.326.937.145.0
7BFull ANO (Ours)55.038.969.027.538.045.7
  1. Interaction and complementarity of both signals. On 4B, Prior-only (58.5, below Uniform 61.8) under-weights large deep-layer reasoning updates when used without norm scaling, while Norm-only (62.4, MME-RW 53.7) over-injects shallow updates; combining them multiplicatively lifts Full ANO to 64.0 (+5.5 over Prior-only, +1.6 over Norm-only) and 55.0 on MME-RW (+1.3, Table 3). On 7B, adding the attention prior on top of Norm-only again improves all five reasoning benchmarks (45.0 →\to 45.7, +0.5 to +0.9 per task) and MME-RW (55.1 →\to 56.3, +1.2, vs. DARE 53.4).
  2. Run dispersion and cross-benchmark consistency. Across 3 independent runs under stochastic decoding (temperature=0.6, Table 7), ANO's mean ± SD is 69.53±0.12 / 65.60±0.10 / 77.53±0.06 / 55.07±0.15 / 51.83±0.12 on 4B (63.91 mean) and 54.9±0.21 / 38.8±0.26 / 69.0±0.25 / 27.5±0.06 / 38.0±0.10 on 7B (45.6±0.17 Avg., where the 0.17 SD is small relative to ANO's +0.7 and +1.5 aggregate gains). Across both controlled pairs, Full ANO outperforms both DARE (12/12 evaluations) and Norm-only (12/12 evaluations) on every reasoning and visual benchmark, as well as MMMU-tuned Exp. on all 5 tasks on 4B. In the revised manuscript, we also clarify Figure 2 and Appendix D with the measured InternVL3.5-4B-MPO fit and layer-weight trajectory.

Complete calibration, baseline, scope, and per-layer λR(l)\lambda_R^{(l)} specifications (W2)

We apologize for omitting numerical configuration values in Appendices A.2–A.4, and have added every exact specification to the revised appendix:

  1. Calibration and merged parameters (Appendices A.2–A.3). Calibration uses N = 1,000 unlabeled MSCOCO val2017 images with prompt "Describe this image in detail.", averaging visual-key attention ratios (Eq. 2) over all attention heads and the first 3030 generated response tokens, with ϵ=10−6\epsilon = 10^{-6} and endpoint ϵw=0.09\epsilon_w = 0.09 in Eq. 3 (wV(1)=0.91→wV(L)=0.09w_V^{(1)}=0.91 \to w_V^{(L)}=0.09). Task vectors and squared L2L_2 norms are computed in float32 and saved in bfloat16; only homologous decoder transformer blocks (self_attn and mlp weights/norms) are fused, retaining the MLLM's vision encoder, projector, embeddings, and LM head.
  2. Baseline hyperparameters and comparison scope (Appendix A.4). Weight averaging uses λV=0.5\lambda_V=0.5 (61.8); Task arithmetic [4] uses official multi-vector scaling θ0+0.3τV+0.3τR\theta_0 + 0.3\tau_V + 0.3\tau_R (61.9); TIES [3] uses top-20% density and scaling 0.30.3 (62.1); DARE [2] uses drop rate p=0.9p=0.9 with 10×10\times rescaling and 0.5/0.50.5/0.5 averaging (62.2); SLERP uses depth-interpolated t∈[0,1]t \in [0,1] (61.9). OptMerge [5], WUDI [6], and DOGE [7] (48.8–49.8) were evaluated at their official data-free default configurations, which apply unnormalized unit task-vector addition (τV+τR\tau_V+\tau_R without λV+λR=1\lambda_V+\lambda_R=1) and access neither unlabeled images nor target-task inputs; Table 2 reflects these data-free defaults, and baseline superiority claims apply strictly to the two controlled pairs (4B and 7B), while Table 1's six pairs demonstrate broad cross-family transfer over each unfused source.
  3. Fit residuals and per-layer λR(l)\lambda_R^{(l)} trajectory. On InternVL3.5-4B-MPO (L=36L=36, α^=0.108\hat{\alpha}=0.108), the log-linear fit achieves R2=0.90R^2 = 0.90 and RMSE=0.019\text{RMSE} = 0.019, yielding per-layer reasoning weights λR(l)\lambda_R^{(l)} rising from 0.08 in layer 1 to 0.92 in layer 36 while adapting locally to per-layer energy ratios (reported in Figure 2 and Appendix D of the revised manuscript).

[1] Chen et al., Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging, ICML 2025.

[2] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.

[3] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

[4] Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023.

[5] Wei et al., Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging, 2025.

[6] Cheng et al., Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors, ICML 2025.

[7] Wei et al., Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent, ICML 2025.

Comments on the paper

2 comments

We thank the Area Chair and Reviewers (Fwo1, qxVN, ecBq) for their constructive reviews and for highlighting ANO's concise, training-free closed-form design, cross-family transfer across six homologous pairs (4B–38B), matched-checkpoint improvements over DARE at both 4B (+1.8) and 7B (+1.5), and our transparent reporting of visual trade-offs and mathematical boundaries. Below we directly answer the four core questions shared across your reviews.

1. How does ANO differ from Bring Reason to Vision (ICML 2025) and properly tuned global/step-cutoff linear merging?

Bring Reason to Vision (BRTV; Chen et al., ICML 2025) selects a global linear scalar λV∈{0.80,0.85,0.90}\lambda_V \in \lbrace 0.80, 0.85, 0.90 \rbrace using labeled downstream MathVista scores (or applies a hard step-cutoff keeping shallow layers l≤5l \le 5 fixed at θV\theta_V), whereas ANO computes closed-form, downstream-label-free per-layer weights (λR(1)=0.08→λR(36)=0.92\lambda_R^{(1)}=0.08 \to \lambda_R^{(36)}=0.92, preserving 92% vision weight in layer 1 while scaling up to 92% reasoning weight in layer 36) from unlabeled MSCOCO visual-attention decay (α^=0.108\hat{\alpha}=0.108, R2=0.90R^2=0.90, ϵw=0.09\epsilon_w=0.09) and per-layer task-vector norms. Sweeping all nine global linear coefficients λV∈{0.1,…,0.9}\lambda_V \in \lbrace 0.1, \dots, 0.9 \rbrace alongside BRTV's step-cutoff configurations on InternVL3.5-4B-MPO / Qwen3-4B-Thinking shows that label-free ANO surpasses even the best test-selected grid points on both reasoning and visual retention:

Global VLM Weight λV\lambda_V0.10.20.30.40.50.60.70.80.9ANO (Label-Free)
4B MMMU67.867.066.567.067.166.062.761.463.369.6
4B MME-RW48.850.151.451.952.051.353.753.754.255.0

Global LERP faces a structural trade-off: lowering λV\lambda_V to 0.1 raises MMMU to 67.8 but drops MME-RW to 48.8 (−7.9 vs. source 56.7), while BRTV's λV∈{0.80,0.90}\lambda_V \in \lbrace 0.80, 0.90 \rbrace keeps MME-RW at 53.7–54.2 (66.1 / 53.9 for step-cutoff at λV=0.85\lambda_V=0.85) but caps MMMU at 61.4–63.3. Even if a baseline is granted downstream test scores to cherry-pick its best λV\lambda_V, label-free ANO (69.6 MMMU, 55.0 MME-RW) still beats the best reasoning grid point (λV=0.1\lambda_V=0.1) by +1.8 MMMU and +6.2 MME-RW and the best visual grid point (λV=0.9\lambda_V=0.9) by +6.3 MMMU and +0.8 MME-RW.

2. Do both salience signals (attention prior and task-vector norm) generalize across model pairs with stable gains?

Yes. To isolate each component of st(l)=wt(l)∥τt(l)∥22s_t^{(l)} = w_t^{(l)}\lVert\tau_t^{(l)}\rVert_2^2, we compare Prior-only (λt(l)=wt(l)\lambda_t^{(l)}=w_t^{(l)}, using only the attention-decay prior without norm scaling; 58.5 Avg. on 4B: 67.2 / 62.7 / 72.6 / 49.2 / 41.0) and ANO w/o prior (Norm-only) (λt(l)=∥τt(l)∥22/(∥τV(l)∥22+∥τR(l)∥22)\lambda_t^{(l)}=\lVert\tau_t^{(l)}\rVert_2^2 / (\lVert\tau_V^{(l)}\rVert_2^2+\lVert\tau_R^{(l)}\rVert_2^2), using only per-layer task-vector energy without the attention prior) across both 4B and 7B:

PairMethodMMMUMMMU-ProMathVistaMathVisionWeMathAvg.MME-RW
4BSource VLM65.460.171.748.039.857.056.7
4BDARE (Yu et al., ICML 2024)68.061.276.254.950.862.252.3
4BANO w/o prior (Norm-only)66.663.176.954.950.362.453.7
4BFull ANO (Ours)69.665.777.655.151.964.055.0
7BSource VLM53.936.367.825.435.243.757.4
7BDARE (Yu et al., ICML 2024)54.236.768.425.936.044.253.4
7BANO w/o prior (Norm-only)54.538.068.326.937.145.055.1
7BFull ANO (Ours)55.038.969.027.538.045.756.3

On 7B, adding the attention prior improves all five reasoning tasks (+0.5 to +0.9, lifting Avg. from 45.0 to 45.7) while cutting MME-RW perceptual loss in half (−2.3 →\to −1.1, +1.2). Across 3 runs (temperature=0.6, Table 7), ANO's stochastic-decoding SDs are small (0.06–0.15 per task on 4B; 45.6±0.17 Avg. on 7B; 34.5±0.08 Avg. on 8B), and Full ANO wins all 12/12 matched pair-benchmark evaluations over both DARE and ANO w/o prior across the two pairs (plus 5/5 over Figure 3's downstream-selected MMMU-tuned Exponential at 63.4).

3. Do lower-attention deeper layers actually tolerate stronger reasoning contributions across model families?

Yes. Holding the reasoning weight 1−λV1-\lambda_V identical, we compare All-Layer linear merging (merging all LL blocks) against Shallow-Protected merging (keeping high-visual-attention shallow blocks l≤5l \le 5 untouched at 100% θV(l)\theta_V^{(l)} and merging only deeper blocks l>5l > 5), where MME-RW Gain (Shallow-Protected − All-Layer) directly measures the perceptual retention gained by protecting shallow visual blocks:

PairλV\lambda_VAll-Layer MMMUAll-Layer MME-RWShallow-Protected MMMUShallow-Protected MME-RWMME-RW Gain
4B0.167.848.867.852.4+3.6
4B0.567.152.067.553.9+1.9
4B0.963.354.263.254.1−0.1
7B0.554.252.154.455.0+2.9

Protecting shallow high-attention layers recovers +1.9 to +3.6 MME-RW points on 4B and +2.9 points on 7B without sacrificing MMMU, while the residual −4.3 visual drop under near-complete deep replacement (λV=0.1\lambda_V=0.1, 52.4 vs. 56.7) confirms why retaining positive visual weights (ϵw=0.09\epsilon_w=0.09) in Full ANO (69.6 / 55.0 at 4B; 55.0 / 56.3 at 7B) outperforms hard replacement.

4. Are inference settings matched, and is ANO compatible with FlashAttention and newer Qwen models?

  • Strict inference parity: Within every pair, source VLMs and merged models share identical prompts, thinking settings (enable_thinking=True on both source and merged checkpoints for Qwen3 pairs), max_new_tokens=32,768, and sampling parameters (temperature=0.6, top_p=0.95, top_k=20).
  • FlashAttention & calibration specs: Offline calibration runs once on N=1,000 unlabeled MSCOCO val2017 images ("Describe this image in detail.", first 30 generated tokens, ϵ=10−6\epsilon=10^{-6}, ϵw=0.09\epsilon_w=0.09); the saved bfloat16 checkpoint requires no runtime attention scores and runs natively with FlashAttention in Transformers, vLLM, and SGLang.
  • Newer Qwen series (3.5 / 3.6 / 3.8) & presentation: ANO targets settings where an MLLM and a stronger text-only reasoning model are trained separately from a shared base (as in Qwen2.5 and Qwen3); by contrast, official Qwen3.5, Qwen3.6, and Qwen3.8 releases are trained natively as unified multimodal reasoning models with built-in thinking mode on a hybrid Gated DeltaNet and full-attention backbone, so no separate stronger official text-only donor exists on the same base. In the revised manuscript, we also replace the OpenReview \method{} macro with ANO, add the measured InternVL3.5-4B-MPO calibration fit and weight trajectory to Figure 2 and Appendix D, and clarify the token-position curves and ±1 SD\pm 1\text{ SD} shading in Figure 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.