- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ANO, a training-free method for merging a text-only reasoning model with a compatible multimodal model. It determines layer-wise merging coefficients using task-vector norms and a depth-dependent prior estimated from visual attention, while retaining the original visual encoder and projector. Experiments across six compatible model pairs show improvements in average reasoning performance, accompanied by regressions on some visual tasks.
Critical strengths
-
Practical motivation. Reusing existing reasoning checkpoints without additional multimodal training is a useful direction, particularly when training resources or labeled data are limited.
-
Evaluation across multiple model pairs. The experiments cover different model families and scales, providing evidence that the approach can improve average performance beyond a single checkpoint.
-
Acknowledgment of trade-offs. The paper reports visual-capability regressions and appropriately distinguishes the optimality of its coefficient-space objective from optimality for the original model losses.
Critical weaknesses
-
Insufficient justification for the proposed weighting signals. Lower direct attention to visual tokens does not necessarily indicate lower dependence on visual information, which may already be integrated into other representations. Similarly, task-vector magnitude does not necessarily measure capability importance. The coefficient-space derivation formalizes the weighting rule but does not establish that these signals reliably identify which adaptations should be retained at each layer.
-
The figures and submission presentation need clarification. Figures 1 and 2 substantially overlap without showing how measured attention and task-vector energies determine actual merging coefficients. In Figure 4, the colored curves and shaded regions are insufficiently explained. Additionally, the OpenReview abstract contains an unresolved \method{} placeholder that should be replaced with the method name.
-
Important implementation details are insufficiently specified for assessing reproducibility and baseline fairness. Appendix A describes the general experimental protocol, but does not fully specify several consequential settings, including the calibration sample count, exact calibration prompts, generated-query selection, the numerical value of the prior endpoint parameter, and detailed baseline hyperparameters. These omissions are particularly concerning because OptMerge, WUDI, and DOGE perform substantially worse than the original model under the reported configurations.
-
Pre-merging visual attention may not predict the visual cost of merging. ANO uses the source model’s visual-attention profile to guide layer-wise fusion. However, lower direct attention to image tokens does not necessarily imply that a layer is less important for visual processing, since visual information may already be integrated into text-token representations or the residual stream. Moreover, merging changes attention patterns and intermediate representations, so attention statistics measured before merging may not reliably predict the resulting visual degradation. The observed concentration of visual attention in shallow layers therefore provides insufficient justification for assigning greater reasoning-branch weights to deeper layers. Retaining the visual encoder and projector alone does not ensure that the fused decoder remains compatible with their outputs. Furthermore, and this is also important, I am curious whether this method is compatible with the flash-attention framework (https://github.com/dao-ailab/flash-attention).
Questions for authors
Q1. Could the authors report the number of calibration samples, the exact calibration prompts, the selection of generated queries, the numerical values of the prior endpoint parameters, and the detailed baseline hyperparameters?
Q2. Can layer-wise interventions or sensitivity analyses demonstrate that visual attention and task-vector norms predict the effects of merging on reasoning and visual performance?
Q3. Could the authors conduct controlled layer-wise replacement experiments to test whether layers with lower visual attention actually tolerate greater contributions from the reasoning branch without degrading visual performance? Does this relationship hold consistently across model families?
Q4. If feasible, could you please include experiments using Qwen3.5, Qwen3.6, or Qwen3.8? Even adding just one of these would make the paper more convincing.
10 comments
Complete calibration parameters, endpoint , and baseline hyperparameters (W3 / Q1)
We have added all exact calibration and baseline parameters to Appendix A of the revised manuscript:
- Calibration settings.
N = 1,000unlabeledMSCOCO val2017images; prompt"Describe this image in detail."; generated-query selection: the first generated response tokens averaged across all attention heads (Eq. 2); log-fit stabilizer (, onInternVL3.5-4B-MPO); prior endpoint in Eq. 3 (, ; resulting rises from0.08in layer 1 to0.92in layer 36). Task vectors and norms are computed infloat32and saved inbfloat16. Source and merged models share identical decoding settings (enable_thinking=Trueon Qwen3 pairs,max_new_tokens=32,768,temperature=0.6,top_p=0.95,top_k=20). - Baseline hyperparameters (Appendix A.4).
Weight averaginguses (61.8);Task arithmetic[3] uses official multi-vector scaling (61.9);TIES[4] uses top-20%density and scaling (62.1);DARE[2] uses drop rate with rescaling and averaging (62.2);SLERPuses depth-interpolated (61.9).OptMerge[5],WUDI[6], andDOGE[7] (48.8–49.8) were evaluated at their official data-free default configurations, which apply unnormalized unit task-vector addition ( without ); in cross-modal pairs where deep-layer reasoning task vectors have much larger norms, unnormalized unit addition over-injects .
Is ANO compatible with FlashAttention? (W4)
Yes, the fused ANO model is fully compatible with FlashAttention during all inference and deployment. Only the one-time offline calibration pass over the unlabeled images uses explicit attention to read and fit the scalar slope . Once the closed-form weights are merged, the resulting checkpoint requires no runtime attention scores, preserves the exact architecture and bfloat16 tensor shapes of the source MLLM, and runs natively with FlashAttention in Transformers, vLLM, and SGLang.
Clarifying Figures 1, 2, and 4 and fixing the OpenReview \method{} placeholder (W2)
We apologize for the unexpanded \method{} macro in the OpenReview form abstract and the omitted figure notes, and have updated all items in the revised manuscript:
- OpenReview abstract. We replace the unexpanded
\method{}placeholder in the OpenReview metadata box withANO. - Figures 1 and 2. Figure 1 presents the high-level conceptual motivation, while Figure 2 and Appendix D report the measured values on
InternVL3.5-4B-MPO(): fitted decay (, ), prior floor (), and resulting reasoning weights rising from0.08to0.92. - Figure 4 curves and shading. Each colored curve in Figure 4 plots the layer-wise visual-attention ratio for one of the initial generated response token positions, and the shaded band indicates across calibration images; this explanation is now stated explicitly in Figure 4's caption.
Applicability to Qwen3.5, Qwen3.6, and Qwen3.8 (Q4)
ANO targets settings where a vision-language model and a stronger text-only reasoning specialist are trained separately from a shared base, as in Qwen2.5 and Qwen3 (e.g., InternVL3.5-4B-MPO + Qwen3-4B-Thinking, or InternVL2.5-38B + QwQ-32B). By contrast, the official Qwen3.5, Qwen3.6, and Qwen3.8 releases are trained natively as unified multimodal reasoning models with built-in thinking mode on a hybrid Gated DeltaNet and full-attention backbone, so there is no separate stronger official text-only reasoning checkpoint on the same base to serve as a donor.
[1] Chen et al., Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging, ICML 2025.
[2] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.
[3] Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023.
[4] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.
[5] Wei et al., Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging, 2025.
[6] Cheng et al., Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors, ICML 2025.
[7] Wei et al., Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent, ICML 2025.
Thank you for clarifying the compatibility requirements. However, this raises a concern about the practical scope of ANO. Given that your method requires a compatible pair consisting of a vision-language model and a stronger text-only reasoning model derived from the same base, what practical use cases remain as newer models increasingly integrate multimodal and reasoning capabilities? Could you clarify when a user would benefit from applying ANO rather than directly using a stronger unified multimodal model?
If the broader objective is to integrate capabilities from multiple models, could you also discuss ANO’s practical advantages and limitations relative to distillation-based approaches such as MOPD? Since these approaches transfer capabilities through model outputs rather than parameter alignment, they may offer greater flexibility for heterogeneous architectures.
We thank Reviewer
ecBqfor the thorough, constructive review and for appreciating our practical motivation, multi-family evaluation, and explicit reporting of visual trade-offs and mathematical boundaries. We address every weakness and question below.Do lower-attention layers tolerate stronger reasoning contributions across families, and how do both signals contribute? (W1, W4 / Q2, Q3)
Yes. Deeper layers with lower direct visual attention tolerate substantially stronger reasoning-branch contributions than shallow high-attention layers across both
InternVL3.5-4BandQwen2.5-VL-7B, and combining both salience signals (attention priorandtask-vector norm) outperforms every single-signal and step-cutoff alternative. Because deeper layers still carry visual features integrated into the residual stream and Eq. 5 serves as a coefficient-normalization rule rather than a model-loss guarantee, ANO uses a smooth, strictly positive prior wV(l)∈[ϵw,1−ϵw] (ϵw=0.09) rather than hard layer replacement.100%θV(l) and merging only deeper blocks l>5), whereMME-RW Gain(Shallow-Protected − All-Layer) isolates the perceptual retention gained by protecting shallow visual blocks:Unfused source MME-RW is
56.7on 4B and57.4on 7B (BRTV's step-cutoff [1] at λV=0.85 on 4B scores66.1MMMU /53.9MME-RW), while Full ANO reaches69.6MMMU /55.0MME-RW on 4B and55.0MMMU /56.3MME-RW on 7B, outperforming every row above on both axes.48.8–52.0on 4B;52.1on 7B), whereas restricting the exact same reasoning weights to l>5 recovers+1.9to+3.6MME-RW points on 4B and+2.9points on 7B without hurting MMMU (at minimal 10% injection λV=0.9, both settings stay near54.1–54.2).4.3MME-RW points (52.4vs.56.7), consistent with deeper layers still processing integrated visual features.Prior-only(λt(l)=wt(l) without norm scaling) scores58.55-task Avg. (67.2 / 62.7 / 72.6 / 49.2 / 41.0, belowUniform 61.8),Reversed/Randomschedules drop to55.4/56.3(Figure 3), andNorm-only(w/o prior) scores62.4Avg. /53.7MME-RW on 4B and45.0Avg. (54.5 / 38.0 / 68.3 / 26.9 / 37.1) /55.1MME-RW on 7B. Combining both signals multiplicatively inFull ANOreaches64.0Avg. /55.0MME-RW on 4B and45.7Avg. /56.3MME-RW on 7B, outperforming bothNorm-onlyandDARE[2] across all 12 matched evaluations.