acceptodds
Under review as a conference paper at ICLR 2027

Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging

Abstract

Multimodal model merging consolidates task experts into a single model while preserving their complementary capabilities. Most existing approaches follow the layer-wise design of unimodal merging and combine the parameter changes learned by experts within each layer. However, merging one layer alters downstream representations and affects how visual and textual information interact. These dependencies make it difficult to trace the influence of individual experts across depth and jointly select merging weights. To tackle these challenges, we propose TAC-Merge, which contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM builds graphs to trace expert influence across depth and distinguish responses to image and instruction changes. The shared objective combines Ricci curvature matching on these graphs with prediction agreement to retain expert capabilities. CMC then uses the measured responses to identify a compact subspace for optimizing this objective. Within this subspace, CMC fits a local quadratic model of weight interactions and jointly selects regional weights to reduce interference. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge and its generalization to unseen tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5t4qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAC-Merge for consolidating task-specific multimodal experts into a single static model. Its multimodal influence mapping module characterizes how regional expert updates affect downstream representations and responses to image and instruction changes, and combines influence-graph curvature matching with agreement with expert predictions. Its coupled merge control module then jointly selects regional merging coefficients using a local quadratic model in a compact, response-informed subspace.

Critical strengths
  1. The topic of merging task-specific multimodal models into a single model is interesting and important.
  2. The paper is generally well organized and easy to follow.
Critical weaknesses
  1. The main experiments use Qwen3.5-4B and Qwen3-VL-8B-Instruct, which limits direct comparison with published SOTA results obtained under earlier backbone and expert configurations. Moreover, Expert Merging [1] is cited but not included in the comparisons, despite also learning merging coefficients through expert alignment on unlabeled calibration data.
  2. Since the experiments use newly trained LoRA experts on Qwen3.5/3 backbones, it is difficult to assess the reproducibility and fairness of the comparisons with insufficient baseline implementation details and unknown hyperparameter selection procedures.
  3. LOT Merging [2] seems particularly relevant to the paper’s motivation concerning changes in intermediate representations during merging, but this work is not discussed.
  4. The evaluation is restricted to two Qwen-family backbones, leaving generalization to other model families unverified.

[1] Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking. ICLR, 2026.

[2] Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration. NeurIPS, 2025.

Questions for authors
  1. Can TAC-Merge be adapted to text-only LLM merging, a setting explored by RegMean++/Chain of Merges (CoM)? If so, how would it compare with these methods under a shared evaluation protocol?
  2. Could the authors visualize and analyze the final region-by-expert coefficient matrix, preferably showing deviations from the uniform initialization? In particular, how do the selected expert weights differ across the visual and language regions? Such an analysis would help clarify the coordination patterns produced by the controller beyond the performance improvements.
6 comments

To W1

Thank you for this comment. We have added LLaVA-v1.5-7B experiments using the public experts from RobustMerge to complement the Qwen results. The comparison includes Expert Merging and LOT-Merge. All methods share the expert checkpoints and evaluation settings. Calibration-based methods use the same 32 unlabeled inputs per task.

MethodSciQAImageVQARECOCRVizWizFlickrIconQAAvg.
Task Arithmetic76.2147.8067.2050.5163.0545.6251.6239.7555.22
TIES75.5444.5566.5551.0264.4747.8152.8135.0754.73
DC-Merge78.2158.6365.0363.4663.7749.3552.3549.4160.03
CoM77.7338.1167.0154.3563.1948.0552.5547.4056.05
RegMean++78.0963.0367.3461.5363.7948.9452.4449.1360.54
LOT-Merge77.4150.0766.6857.4462.9948.0052.0047.7757.79
RobustMerge77.0547.8067.0251.2562.3446.6052.8145.9156.35
OptMerge77.9347.8065.2561.7061.6147.0052.4546.7057.55
Expert Merging75.3660.8964.4460.5563.9544.5550.0548.2658.51
TAC-Merge78.0567.2167.8564.1063.9553.1553.9051.3462.44
MethodAVQAImage-RS2WTabMWPAvg.
Task Arithmetic78.4245.947.5117.0237.22
TIES76.1245.399.0217.4336.99
DC-Merge78.6645.579.2618.3737.97
CoM78.8245.638.3717.7037.63
RegMean++79.2145.767.9318.6537.89
LOT-Merge78.5245.207.6517.6737.26
RobustMerge79.2145.719.2217.6837.95
OptMerge78.9045.478.5617.0537.49
Expert Merging79.5643.619.8416.6437.41
TAC-Merge79.8445.8610.8219.3238.96

TAC-Merge achieves the highest average score on both seen and unseen tasks. Compared with Expert Merging, it improves the average score by 3.93 points on seen tasks and 1.55 points on unseen tasks.

To W2

Thank you for requesting these details. The baseline comparisons use the same expert checkpoints and evaluation settings within each backbone. Calibration-based methods share 32 unlabeled inputs per task. Evaluation uses greedy decoding with a 64-token limit, BF16 for Qwen and FP16 for LLaVA. All hyperparameter searches use the same validation criterion, minimizing KL divergence to task-expert predictions on separate unlabeled validation inputs from seen tasks, with equal weight assigned to each task. Parameter selection is completed before test evaluation.

The shared merge-scale grid is G=[0.05,0.1,0.125,0.2,0.3,0.5,0.7,1.0,1.5]G = [0.05, 0.1, 0.125, 0.2, 0.3, 0.5, 0.7, 1.0, 1.5]. Task Arithmetic scales the summed expert updates and searches the scale over GG, selecting 0.1. TIES fixes its retained fraction at 0.2 and resolves sign conflicts before aggregation; its merge scale is selected from GG, giving 1.0. DC-Merge retains its spectral smoothing and subspace-alignment procedure with fixed settings. CoM follows its sequential merging procedure with an off-diagonal Gram coefficient of 0.95. RegMean++ searches the off-diagonal coefficient over [0.1,0.3,0.5,0.7,0.9,0.95][0.1, 0.3, 0.5, 0.7, 0.9, 0.95], selecting 0.1. LOT-Merge collects features from the original experts and solves for all 224 adapted projections. The projector follows the same fixed averaging policy as the other methods. RobustMerge fixes the retained fraction and attention ratio at 0.2 and searches the fusion weight over [0.5,0.875,1.0,1.5,2.0][0.5, 0.875, 1.0, 1.5, 2.0], selecting 0.875 for Qwen and 2.0 for LLaVA. OptMerge retains its task-vector optimization procedure and searches the final merge scale over GG, selecting 1.0. Expert Merging learns matrix-level coefficients through expert alignment on the shared unlabeled calibration inputs, incorporating the provided visual and language LoRA updates.

To W3

Thank you for highlighting LOT Merging. This work minimizes feature drift through layer-wise task-vector reconstruction, formulated as quadratic objectives with closed-form solutions. It is directly relevant to changes in intermediate representations during merging. We will explicitly discuss this connection in the revised related work.

The distinction concerns what is matched and how updates are coordinated. LOT reconstructs local expert features. TAC-Merge measures how regional expert updates affect downstream multimodal responses and uses the resulting influence structure for matching. Its controller then selects coefficients jointly across visual and language regions, accounting for interactions between regional updates.

We additionally implemented a LoRA adaptation of LOT on the public LLaVA-v1.5-7B experts released by RobustMerge, using the same 32 unlabeled calibration inputs per task as TAC-Merge. The adaptation collects features from the original experts and solves for all 224 adapted projections. The projector follows the same fixed averaging policy as the other methods. A projection-level comparison with an independent SVD solution produced a relative error of 1.1×10−81.1 × 10^{-8}, and the exported model passed forward-generation checks.

As reported in our response to W1, LOT-Merge obtains average scores of 57.79 on seen tasks and 37.26 on unseen tasks. TAC-Merge improves these scores by 4.65 and 1.70 points, respectively. This comparison supports the benefit of joint influence coordination relative to local feature reconstruction in the shared LoRA merging setting.

Reviewer_SS4MOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TAC-Merge, which uses multimodal influence mapping (MIM) to trace the effects of expert updates across layers and coupled merge control (CMC) to jointly optimize regional merging weights. Experiments on two multimodal backbones show consistent improvements over the evaluated merging baselines, and the ablations provide some support for the proposed components.

Critical strengths

This paper addresses a practical problem in multimodal model merging. The proposed framework combines cross-layer influence modeling with joint optimization of regional merging weights. Experiments on two backbones show consistent improvements over existing merging baselines, and the ablation studies provide evidence for the effectiveness of the main components.

Critical weaknesses

The specific benefit of Ricci curvature matching is not fully explained beyond its empirical gains. The additional improvement from coupled optimization is modest, particularly on unseen tasks with the 8B model, and the connection between the theoretical analysis and empirical results remains unclear. The paper also lacks measurements of actual merging time and memory usage, making the computational cost difficult to assess.

Questions for authors
  1. Can the authors provide more direct evidence that curvature matching reduces cross-layer interference, beyond the reported performance gains over edge-weight matching?
  2. Are the Hessian conditions in Theorem 1 observed in practice? Given the small gain from coupled optimization on the 8B unseen tasks, how consistent are the paired improvements across seeds?
  3. What are the actual merging runtime and peak memory costs compared with the baselines?
4 comments

To Q1

Thank you for this question. We added a perturbation diagnostic that measures whether a downstream adjustment weakens or reverses the benefit of an earlier update on expert agreement. We evaluated identical perturbations across the two matching variants on 64 inputs excluded from coefficient calibration, using 16 pairs of visual and language regions. The example below concerns one held-out input and the pair formed by the early visual region V1 and downstream language region L1.

For each method, we measure next-token KL to the task expert on the same fixed continuation. Let K0K_0 denote KL without perturbations, and let KVK_V and KLK_L denote KL after perturbing V1 or L1 alone. Let KVLK_{VL} denote KL after both perturbations. We quantify their excess interaction as:

I=(KVL−KL)−(KV−K0).I=(K_{VL}-K_L)-(K_V-K_0).

A positive value indicates that the joint KL is worse than the additive prediction from the isolated perturbations. The table reports KL changes in units of 10−410^{-4}, with negative changes indicating improved expert agreement. All differences and percentage changes are calculated before rounding.

MethodVisual adjustment aloneVisual adjustment after language adjustmentExcess interaction
Normalized-edge matching-0.99+3.154.14
Curvature matching-5.61-5.200.40

Under normalized-edge matching, the visual adjustment improves expert agreement in isolation but worsens it after the language adjustment. Under curvature matching, the visual adjustment remains beneficial in both settings. The excess interaction decreases by 90.3% in this example. This directly measures reduced antagonism between the two regional adjustments for the held-out input, complementing the task-performance comparison.

To Q2

Thank you for this question. The local IconQA diagnostic exhibits positive Hessian diagonals together with a negative eigenvalue at the reference point. Under the stated uniform regularity bound, the gradient also satisfies the condition that the coupling gain exceeds the improvement available along an individual response direction. The numerical check is provided in our response to Reviewer XqRd (W3).

We compare coupled quadratic control with linear and diagonal quadratic variants using the same response basis. The variants share the fitted coefficients for the terms they retain. The table reports mean ± standard deviation over calibration seeds for Qwen3.5-4B and Qwen3-VL-8B-Instruct.

Control variant4B seen4B unseen8B seen8B unseen
Response basis, linear77.12 ± 0.3067.19 ± 0.2276.43 ± 0.3865.78 ± 0.27
Response basis, diagonal77.79 ± 0.3067.71 ± 0.2077.02 ± 0.3666.38 ± 0.19
Response basis, coupled78.50 ± 0.2468.10 ± 0.1877.50 ± 0.3166.50 ± 0.22

Paired comparisons consistently favor coupled control on 8B unseen tasks across the evaluated calibration seeds. Coupled control also achieves higher mean performance than both alternatives across the seen and unseen evaluations on both backbones. The diagonal comparison isolates the benefit of cross-direction coupling, while the linear comparison assesses the contribution of the full quadratic model.

To Q3

Thank you for this comment. TAC-Merge performs influence estimation and coefficient selection once offline for a fixed expert set and merge configuration. The selected updates are folded into the final model weights, so subsequent inputs use ordinary backbone forward passes. The merging cost can therefore be amortized over repeated use of the model, with no additional merging operations during inference.

We report measurements on eight NVIDIA H20 GPUs, each with 96 GB memory. Each backbone merges eight task experts, and calibration-based methods use 32 unlabeled inputs per task. Results are mean ± standard deviation over three runs. Each run measures a single merge with hyperparameters fixed before timing. The timed interval starts with local checkpoint loading and integrity checks and ends after serialization, including required feature or target construction and coefficient optimization or parameter merging. Downloads and downstream evaluation are excluded. The first table reports merging time in minutes, and the second reports peak allocated GPU memory in GB.

BackboneTask ArithmeticTIESDC-MergeCoMRegMean++RobustMergeOptMergeExpert MergingTAC-Merge
Qwen3.5-4B0.45 ± 0.134.33 ± 1.361.17 ± 0.3179.33 ± 21.2048.53 ± 14.490.80 ± 0.254.00 ± 1.474.67 ± 1.2581.67 ± 7.17
Qwen3-VL-8B-Instruct0.63 ± 0.178.00 ± 2.611.63 ± 0.44143.93 ± 41.6574.13 ± 22.651.08 ± 0.344.50 ± 1.669.17 ± 2.86139.67 ± 13.28
BackboneTask ArithmeticTIESDC-MergeCoMRegMean++RobustMergeOptMergeExpert MergingTAC-Merge
Qwen3.5-4B0.47 ± 0.192.33 ± 0.621.17 ± 0.3116.75 ± 2.0517.66 ± 2.460.55 ± 0.171.17 ± 0.3131.66 ± 3.2722.33 ± 1.03
Qwen3-VL-8B-Instruct0.64 ± 0.274.69 ± 1.331.98 ± 0.5530.55 ± 3.6931.47 ± 4.100.74 ± 0.231.91 ± 0.4155.86 ± 5.2440.33 ± 1.84

Peak memory is the maximum PyTorch allocation observed on an individual GPU, including the resident backbone and working tensors when required. CUDA context and reserved cache are excluded. Methods operating solely on LoRA factors are measured without loading the inference backbone.

Our implementation caches fixed expert targets across objective queries and reuses unchanged prefix activations across directions within each query. Directional derivatives are processed in bounded batches, and readouts are accumulated incrementally. Independent calibration pairs are distributed across GPUs. These optimizations reduce repeated computation and intermediate storage while retaining all expert-region directions. The measured merging time is comparable to CoM on both backbones, while peak allocated memory is lower than that of Expert Merging.

In the main experiments, TAC-Merge improves over the strongest baseline averages by 3.43 points on seen tasks and 2.34 points on unseen tasks for Qwen3.5-4B. The corresponding gains are 2.41 and 1.93 points for Qwen3-VL-8B-Instruct. These performance gains are obtained with the standard backbone inference structure.

Thank you for the detailed responses and additional experiments. The rebuttal addresses my concerns to a reasonable extent. Some remaining concerns regarding the generality of the interference analysis and the computational cost prevent me from increasing my score. I will maintain my rating.

Reviewer_XqRdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TAC-Merge, a method for combining multimodal task experts. It works by building graphs of regional update sensitivities, aligning the curvature and predictions of the expert graphs, and then using a quadratic controller within a response-derived subspace to pick the merging coefficients.

Critical strengths

1.The method explains how regional updates impact downstream representations, using separate responses for image and instruction changes.

2.The ablation studies break down the curvature objective, graph structure, response basis, and quadratic coupling. The prediction explain why the controller helps.

3.The chosen coefficients get baked straight into a single static model. In deploying part, the original architecture stays intact, so zero influence estimation or coefficient searching is needed during inference.

Critical weaknesses

1.Every single objective query need to evaluate all expert-region directions. It need actual wall-clock times, hardware specs, and peak memory usage to explain how it stacks up against the baselines cost-wise.

2.Swapping images or instructions randomly can lead to pairs that make zero semantic sense. I think it need a control group using semantically compatible swaps.

3.Theorem 1 relies on a laundry list of conditions: local smoothness, positive Hessian diagonals, a negative eigenvalue, and a gradient that's small enough relative to the coupling gain. None of these conditions are actually tested in the experiments.

Questions for authors

See weakness.

3 comments

To W1

Thank you for this comment. TAC-Merge performs influence estimation and coefficient selection once offline for a fixed expert set and merge configuration. The selected updates are folded into the final model weights, so subsequent inputs use ordinary backbone forward passes. The merging cost can therefore be amortized over repeated use of the model, with no additional merging operations during inference.

We report measurements on eight NVIDIA H20 GPUs, each with 96 GB memory. Each backbone merges eight task experts, and calibration-based methods use 32 unlabeled inputs per task. Results are mean ± standard deviation over three runs. Each run measures a single merge with hyperparameters fixed before timing. The timed interval starts with local checkpoint loading and integrity checks and ends after serialization, including required feature or target construction and coefficient optimization or parameter merging. Downloads and downstream evaluation are excluded. The first table reports merging time in minutes, and the second reports peak allocated GPU memory in GB.

BackboneTask ArithmeticTIESDC-MergeCoMRegMean++RobustMergeOptMergeExpert MergingTAC-Merge
Qwen3.5-4B0.45 ± 0.134.33 ± 1.361.17 ± 0.3179.33 ± 21.2048.53 ± 14.490.80 ± 0.254.00 ± 1.474.67 ± 1.2581.67 ± 7.17
Qwen3-VL-8B-Instruct0.63 ± 0.178.00 ± 2.611.63 ± 0.44143.93 ± 41.6574.13 ± 22.651.08 ± 0.344.50 ± 1.669.17 ± 2.86139.67 ± 13.28
BackboneTask ArithmeticTIESDC-MergeCoMRegMean++RobustMergeOptMergeExpert MergingTAC-Merge
Qwen3.5-4B0.47 ± 0.192.33 ± 0.621.17 ± 0.3116.75 ± 2.0517.66 ± 2.460.55 ± 0.171.17 ± 0.3131.66 ± 3.2722.33 ± 1.03
Qwen3-VL-8B-Instruct0.64 ± 0.274.69 ± 1.331.98 ± 0.5530.55 ± 3.6931.47 ± 4.100.74 ± 0.231.91 ± 0.4155.86 ± 5.2440.33 ± 1.84

Peak memory is the maximum PyTorch allocation observed on an individual GPU, including the resident backbone and working tensors when required. CUDA context and reserved cache are excluded. Methods operating solely on LoRA factors are measured without loading the inference backbone.

Our implementation caches fixed expert targets across objective queries and reuses unchanged prefix activations across directions within each query. Directional derivatives are processed in bounded batches, and readouts are accumulated incrementally. Independent calibration pairs are distributed across GPUs. These optimizations reduce repeated computation and intermediate storage while retaining all expert-region directions. The measured merging time is comparable to CoM on both backbones, while peak allocated memory is lower than that of Expert Merging.

In the main experiments, TAC-Merge improves over the strongest baseline averages by 3.43 points on seen tasks and 2.34 points on unseen tasks for Qwen3.5-4B. The corresponding gains are 2.41 and 1.93 points for Qwen3-VL-8B-Instruct. These performance gains are obtained with the standard backbone inference structure.

To W2

Thank you for this suggestion. We added the requested semantically compatible control and a deliberately incompatible stress test alongside the default random pairing within each task. The controls use the same expert checkpoints and calibration inputs under a fixed merging budget. Results are mean ± standard deviation across three calibration seeds.

BackbonePairingSeenUnseen
Qwen3.5-4BDefault random pairing78.50 ± 0.2468.10 ± 0.18
Qwen3.5-4BSemantically compatible78.72 ± 0.2168.25 ± 0.16
Qwen3.5-4BIncompatible stress test78.08 ± 0.3267.82 ± 0.24
Qwen3-VL-8B-InstructDefault random pairing77.50 ± 0.3166.50 ± 0.22
Qwen3-VL-8B-InstructSemantically compatible77.68 ± 0.2866.61 ± 0.20
Qwen3-VL-8B-InstructIncompatible stress test77.13 ± 0.3966.27 ± 0.29

Compatible pairing gives higher mean scores on both backbones, with improvements of 0.11 to 0.22 points over the default pairing. The incompatible stress test reduces the corresponding means by 0.23 to 0.42 points. Both control conditions produced accepted coefficient updates, so the reported scores reflect the resulting merged models.

The calibration design separates response measurement from prediction alignment. For each pairing condition, the expert and merged models are evaluated under the same input perturbations. Swapped inputs supply internal response measurements, while output KL continues to align predictions on valid original pairs. The trust region bounds coefficient displacement, and the acceptance check retains a candidate only when it improves the combined calibration objective relative to the uniform reference. Semantic compatibility therefore changes the structural measurements while the original-task prediction criterion and update bound remain in place. The controls show that compatible pairing benefits the reported means, while the default pairing remains within 0.22 points of the compatible control across all four averages.

To W3

Thank you for this comment. We address the four conditions separately using a local IconQA diagnostic on Qwen3.5-4B. The restricted calibration objective combines teacher KL with curvature MSE.

Local smoothness. Appendix A.2 gives sufficient conditions for local regularity of the reference objective. Using the stated uniform Hessian-regularity bound M≤0.0378M≤0.0378, the Taylor remainder within the coefficient ball is bounded by MR3/6≤0.00009844MR^3/6≤0.00009844. This calculation is conditional on the regularity bound holding throughout that ball.

Positive Hessian diagonals. The diagonal entries are (0.0837,0.0916,0.1073,0.1269)(0.0837,0.0916,0.1073,0.1269), giving β=0.0837>0β=0.0837>0. The associated radius condition also holds under the stated regularity bound, since MR≤0.00945<3β=0.2511MR≤0.00945<3β=0.2511.

A negative eigenvalue. The mixed entries B12=B21=0.1934B_{12}=B_{21}=0.1934 give curvature −0.10575-0.10575 along (e1−e2)/21/2(e_1-e_2)/2^{1/2}. The minimum eigenvalue of the full restricted Hessian is approximately −0.107296-0.107296. Thus, the diagnostic exhibits negative curvature along a joint direction despite positive curvature along each individual axis.

A sufficiently small gradient relative to the coupling gain. The gradient infinity norm is 0.009370.00937, below the sufficient threshold 0.0191240.019124. The resulting single-direction improvement bound is approximately 0.0005450.000545. After accounting for the quadratic-fitting and Taylor errors, the coupling bound is ΔR≈0.002270Δ_R≈0.002270, exceeding both the single-direction bound and the acceptance threshold τ=0.0001τ=0.0001. The separation margin is approximately 0.001725 under the stated regularity bound.

DiagnosticValueRelation to Theorem 1
Regularity bound usedM≤0.0378M≤0.0378Taylor remainder MR3/6≤0.00009844MR^3/6≤0.00009844
Hessian diagonal entries(0.0837,0.0916,0.1073,0.1269)(0.0837,0.0916,0.1073,0.1269)β=0.0837>0β=0.0837>0
Coordinate regularityMR≤0.00945MR≤0.00945MR<3β=0.2511MR<3β=0.2511
Minimum Hessian eigenvalue−0.107296-0.107296Negative curvature with γ≈0.107296γ≈0.107296
Gradient infinity norm0.009370.00937Below the sufficient threshold 0.0191240.019124
Single-direction improvement bound0.0005450.000545Below the coupling bound
Coupling bound after approximation errorΔR≈0.002270Δ_R≈0.002270Exceeds both 0.0005450.000545 and τ=0.0001τ=0.0001

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.