acceptodds
Under review as a conference paper at ICLR 2027

Tracing Perceptual Transfer in Vision-Language Models via Model Merging

Abstract

Despite recent progress in visual mathematical reasoning, perception remains a major bottleneck for Vision-Language Models (VLMs). However, it remains unclear where perceptual transfer occurs within the visual hierarchy of VLMs and how its representational and functional effects vary across layers. To investigate this question, we use model merging as a tool and apply it to the perception modules of VLMs, while keeping the language model frozen, allowing us to understand how perceptual transfer occurs across layers. Across multiple VLMs, this intervention yields gains on several visual mathematical reasoning benchmarks (MathVista, MathVerse, MathVision), providing evidence that perceptual capabilities can be transferred without modifying the LLM. To understand where these gains arise, we further analyze perceptual transfer across the visual hierarchy. Layer-wise probing shows that primitive geometric attributes become linearly accessible relatively early in the visual hierarchy, whereas relation-sensitive information becomes more prominent in intermediate layers. We further perform backward and forward layer replacement to examine the functional effects of perceptual transfer. Removing merged parameters from intermediate layers causes the largest performance degradation, while inserting them into the base model yields the largest performance gains in the same layers. Together, these complementary analyses reveal a non-uniform layer-wise organization of perceptual transfer, with relation-sensitive information and its strongest functional effects concentrated in middle layers. These findings provide a layer-wise account of how perceptual knowledge is represented and transferred within VLMs, offering insights for targeted perception-side adaptation.

Primary area:
interpretability and explainable AI
Keywords:
Visual Mathematical Reasoning, Vision-Language Models, Model Merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VbQbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where perceptual knowledge is transferred in VLMs using perception-side model merging. It finds that primitive cues emerge early, while relation-sensitive information and the strongest functional effects concentrate in middle visual layers.

Critical strengths
  1. Clear problem formulation. The paper isolates perception-side transfer by keeping the LLM frozen and intervening only on the visual modules.
  2. Complementary analyses. Linear probing and bidirectional layer replacement provide both representational and functional evidence.
  3. Consistent layer-wise finding. The results suggest that primitive cues emerge early, while relation-sensitive transfer is concentrated in middle layers. 28294_Tracing_Perceptual_Trans
Critical weaknesses
  1. Unclear necessity of model merging. Similar layer-wise analyses could potentially be performed by directly comparing the base and perception-tuned models, so the specific role of merging is not fully justified.
  2. Mean pooling may obscure spatial relations. Averaging all visual tokens can discard localization and pairwise structure, which is especially relevant for relation-sensitive geometric attributes.
  3. Insufficient justification for the probing design. The paper relies on mean-pooled layer representations and linear classifiers, but does not sufficiently justify why this probing setup is appropriate for capturing complex geometric relations.
  4. New models and datasets are suggested.
Questions for authors

see above

Reviewer_jS7POpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies geometry-related transfer in VLM perception modules using model merging as an intervention. It tunes the vision encoder and projector on diagram descriptions while keeping the LLM frozen, then interpolates the tuned perception parameters with the original VLM parameters, using a default weight of 0.8 on the original model. Downstream evaluation covers three VLM backbones, four geometry datasets, and five benchmarks. A separate synthetic dataset supports linear probes of primitive and relational attributes. Forward and backward single-layer replacements on LLaVA and Qwen are used to connect parameter changes with MathVista performance. The main proposed finding is that relation-sensitive transfer and its strongest downstream effects concentrate in intermediate visual layers.

Critical strengths
  1. A useful question and a potentially informative intervention design. Studying which parts of a perception module change under geometry supervision is worthwhile. Keeping LLM parameters fixed provides a clear boundary on the parameter updates, and combining representation probes with bidirectional parameter replacement is more informative than reporting downstream scores alone. The paper appropriately states in Section 3.2 that linear decodability does not by itself establish where a capability is causally formed.
  2. Nontrivial empirical coverage and several relevant controls. Table 1 covers three backbones, four transfer sources, and five benchmarks, and shows selective rather than universal improvements. The appendix includes pure fine-tuning comparisons, relation-weakened supervision, independent probe splits with train-only normalization, and learning-rate and pooling checks. These experiments provide evidence that geometry supervision can produce transferable updates. Their presence should be distinguished from the remaining issues in the layer-specific interpretation.
Critical weaknesses
  1. The central layer-wise results cannot be reconstructed from the declared architectures. Figures 2, 3, and 7 use the same 0-26 layer axis and shallow/middle/deep ranges of 0-8, 9-17, and 18-26 for LLaVA and Qwen. However, the published configurations for the named model families specify 24 visual transformer blocks for LLaVA-NeXT-Llama3-8B and 32 for Qwen2.5-VL-7B. The manuscript does not explain any layer subsampling, reindexing, normalized-depth mapping, or inclusion of additional modules. Thus it is not possible to determine which actual visual blocks the claimed middle-layer effects correspond to. This is important because localization across architectures is the main contribution. I am not concluding that the experiments used the wrong modules; I am asking for the information needed to rule that out and interpret the results. Exact checkpoint identifiers, hooked module paths, replaced parameter groups, and a mapping from plotted indices to native blocks are necessary. The experiment specification also does not clearly identify the transfer-source checkpoint used for the main layer curves or separate encoder and projector effects, although both are tuned and merged. The external architecture references are the published config.json files for llava-hf/llama3-llava-next-8b-hf and Qwen/Qwen2.5-VL-7B-Instruct. If a different checkpoint or indexing convention was used, it should be specified and the full native-depth results reported.
  2. The evidence does not yet isolate relation-specific transfer or establish robust layer localization. The synthetic probes have construction confounds. Appendix B.1 associates intersection-count labels 0 and 1 with parallel or singly intersecting lines, and 2plus with a three-line pattern. Line count and image complexity are not reported as counterbalanced, leaving a plausible shortcut for this supposedly relational label. The line-presence task also contrasts one straight line with two to four circles, coupling the label to object count without reporting matched stroke statistics. Balanced classes and independence from textual prompts do not remove these potential visual shortcuts. This does not invalidate every probe, especially the separate line-relation task, but aggregate primitive/relation curves are insufficient to show that merging primarily improves relation formation. Per-task curves and probes that match line count, stroke statistics, and rendering distributions are needed. Bidirectional layer replacement establishes effects of particular parameter substitutions in hybrid networks. Larger effects can reflect update magnitude, generic adaptation, or compatibility with surrounding layers, rather than the location of transferred relation knowledge. Appendix C.2 does include a sensitivity experiment, but replacing an entire layer with a constant 1/N matrix is not matched to the actual merging update and does not resolve those alternatives. Appendix B.4's weakened-supervision comparison is useful, but does not report the corresponding layer-wise replacement results. A matched comparison of full and weakened updates, controlling their per-layer norms and the projector, would address the intended interpretation more directly. The stability of the functional localization is also unclear. Figure 3's largest forward gains are approximately 0.5 percentage points for LLaVA and 0.3 for Qwen on MathVista. Table 1 states that its scores average three runs, but the layer experiments do not report run dispersion, paired-item uncertainty, or stability of the layer ranking. Without this evidence, small peak differences should not support a strong cross-architecture claim about where transfer is concentrated.
Questions for authors
  1. Which exact checkpoint revisions, module paths, and transfer-source models produced Figures 2, 3, and 7? How does the 0-26 axis map to the native 24- and 32-block visual encoders, and what is called CLS-token pooling for Qwen in Table 5? Please provide the full mapping and native-depth results, including any embedding or projector entries.
  2. Do the relational gains and their depth profile persist when the number of lines and visual statistics are matched across labels? Please report separate task curves, simple line-count or image-statistic baselines, and held-out geometric/rendering configurations so that the aggregate pattern can be evaluated without the identified shortcuts.
  3. Do the middle-layer effects persist for full versus relation-weakened task vectors after matching per-layer update norms and separating encoder/projector changes? Please report paired uncertainty and variation across training/probing seeds and downstream benchmarks. Also clarify whether lambda=0.8 was selected on the same MathVista test-mini results subsequently used for evaluation and localization.
Reviewer_5B6bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks where perceptual knowledge is represented and transferred inside a VLM’s visual hierarchy. As an intervention, the authors build perception task vectors by tuning only the vision encoder and projector on four geometry datasets comprising diagram-to-description texts, while the LLM module stays frozen. They merge these vectors back into the VLM with linear task arithmetic (λ\lambda = 0.8), on LLaVA-NeXT-8B, InternVL3-8B and Qwen2.5-VL-7B. The gains are modest on MathVista, MathVerse, MathVision and others. They then run two analyses: (a) Layer-wise linear probing on a small synthetic geometry dataset, and (b) Backward and forward single-layer replacement between the base and merged models, evaluated on MathVista.

They conclude that primitive attributes are decodable early, relation-sensitive information peaks in middle layers, and the strongest functional effects of merging also reside in the middle layers.

The paper introduces no new method or metric. It applies standard task arithmetic to the vision side of a VLM only, then uses standard probing and layer swapping to see which layers matter, and effect on downstream tasks. The framing is somewhat new, but the tools are existing ones, and the findings are never turned into a new way of merging based on insights obtained.

Critical strengths
  1. Experimental design: Isolating perception changes by keeping the language model frozen is a clean and highly effective approach.
  2. Layer analysis: Using both representational probing and functional replacement offers complementary insights. The finding that middle layers are the most impactful is consistent across both the LLaVA and Qwen models in both replacement directions.
  3. Experimental controls: The inclusion of controls like the scaling coefficient (λ\lambda) sweep and the pure fine-tuning comparison adds value to the study.
  4. Transparency: The authors correctly acknowledge that the probing results do not prove causality and honestly report that no single merging method dominates across all benchmarks.
  5. Data and reproducibility: Deduplicating the training data against the evaluation benchmarks prevents data leakage, and the provided reproducibility statement is suitably detailed.
Critical weaknesses
  • Limited Novelty: No new ideas are proposed, rather existing task-arithmetic and probing ideas are applied in a straightforward manner.

  • No deep insight (actionable or otherwise) is obtained from experiments: Results are presented more as a matter of fact, than giving any deeper analysis/stating any implications on the behaviour of the model on various datasets/tasks. One of the contributions is the effect of swapping which shows that task arithmetic helps in downstream tasks, but that is still not a very surprising result, given that it has already been shown for other NLP and vision tasks.

  • More detailed comments below:

  1. MathVista gains: MathVista score changes are very small (-0.4 to +1.1). Please provide statistical significance to prove these gains are reliable.
  2. MMStar drop: MMStar overall scores drop across all 12 merged models, showing a loss in general capabilities. A single non-geometry example does not prove the model improved on general visual tasks.
  3. MathVerse text subsets: Text-heavy MathVerse tasks also improved, suggesting the gains aren't just from better visual perception. Please add text-only or image-ablated tests to confirm.
  4. Data overlap: While exact duplicates were removed, similar templates might still overlap. Testing on entirely different, out-of-domain datasets would better prove the model learned general skills rather than just memorizing geometry data.
  5. Vector content: It is unclear if improvements actually come from the geometry data. Please test using non-geometry or random data vectors. Since fine-tuning Qwen directly hurts performance but merging helps, the gains might just come from regularizing the original weights.
  6. Ablation study: The ablation study changes too many variables at once (relations, grounding, and format), making it impossible to credit any single factor. The resulting score improvements are also uneven and lack error bars.
  7. Probe evaluation: The probe tests use a small sample size (200-300 examples) and lack confidence intervals, making the results unreliable. Please explain how you selected the ``best-layer test accuracy."
  8. Layer emergence: Basic features are already readable in the very first tested layer. This shows they are available early, not where they actually originate.
  9. Pooling robustness: Claims about pooling robustness need full layer-by-layer graphs, not just peak scores. Also, explain how you extracted a CLS-token for Qwen, since its vision encoder does not have one.
  10. Layer attribution: Differences observed at a specific layer include all the changes from previous layers, so you cannot credit that layer alone for the impact.
Questions for authors
  • Experiment variance: What exactly changes across the three runs (e.g., random seeds)? Please provide statistical significance or confidence intervals for the benchmark gains, probe differences, and layer replacement effects.
  • Hyperparameter tuning: Did you choose the λ=0.8\lambda=0.8 scaling factor using an independent validation set, or was it tuned directly on the MathVista evaluation set?
  • Checkpoints and interpolation: Which exact checkpoints were used to initialize the perception tuning? Additionally, since Equation 3 simplifies to direct checkpoint interpolation, how does this align with the paper's task-vector interpretation?
  • Layer mapping: The charts display layer indices ranging from 0 to 26. How do these indices map to the actual architectural visual blocks in the LLaVA-8B and Qwen-VL-7B models?
  • Projector controls: Which projector (base or merged) is used during the layer replacement tests? Do the middle-layer findings hold up if you use a constant projector across all configurations?
  • Update controls: Are the middle-layer improvements truly due to transferred geometric knowledge? Please confirm this by testing against random or shuffled updates of the same magnitude.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.