acceptodds
Under review as a conference paper at ICLR 2027

Scale First, Then Shape: The Predictive Geometry of Merging Multimodal Experts

Abstract

Model merging combines fine-tuned experts into a single model without training data. For multimodal large language models (MLLMs), where each expert adds one capability such as OCR or grounding to a shared base, it avoids assembling instruction data for every new capability. The merging methods proposed for this setting have grown steadily more elaborate, from sparsification and spectral editing to per-layer optimization, yet each is judged only by its final benchmark score. As a result, a practitioner must evaluate every candidate merge, at over an hour per evaluation, and no existing analysis says in advance which method will work on a given model, or why. In this paper, we show that for MLLM capability merging the answer can be read from the merged weights before any evaluation. (i) We introduce a two-axis geometric account of a merged displacement: an effective scale, how far the merge travels in units of the naive task-vector sum, and a signed depth slope, whether that travel tilts toward deep or shallow layers. Keeping only each method's component along the naive sum and evaluating a plain task-arithmetic curve at its measured scale predicts the benchmark average of three published methods (OptMerge, WUDI and Iso-C) on Qwen2-VL-7B and InternVL2.5-1B within one point in five of the six cases; the depth slope, calibrated on synthetic probes at fixed scale, accounts for the sixth. Re-merging the collapsed Iso-C model from that component alone lifts it from 18 to 47 points, within a point of where the curve placed it. (ii) We show that per-layer optimization reduces to this geometry: 91% of its displacement lies along the naive sum, and a closed-form profile matches the projected optimizer within 0.13 points on both MLLMs at one twentieth of the merging cost. (iii) We find that the component the projection removes adds 0.3 to 1.4 points for the optimizers on capability experts but carries 21 to 47 points when experts have disjoint label spaces, which marks the boundary of the account. Merging multimodal experts is a prediction problem rather than a search problem, and scale is the first quantity to get right.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Models Merge, Model Merge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2ViyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes multimodal model merging through effective update scale and depth-dependent shape. By projecting merged updates onto the task-vector sum, it finds that a calibrated task-arithmetic curve predicts five of six projected merges within one point across two multimodal architectures. It also proposes a closed-form scaling rule that approximates the projected optimizer at substantially lower merging cost. Controlled shape probes and classification experiments examine when scale alone is insufficient.

Critical strengths
  1. The analysis experiment was relatively insightful.Projection surgeries and matched-scale shape probes help distinguish magnitude effects from directional changes, providing stronger evidence than correlations alone. This substantially strengthens the paper’s central analysis.
  2. Well-scaled task arithmetic nearly matches optimization-based merging on the studied multimodal models. The classification counterexamples also usefully expose the limits of this finding.
Critical weaknesses
  1. The research content and analysis of this article are very interesting, but the standalone contribution of CFS is unclear. Its strongest results use scale information from the optimizer, while the parameter-free variant transfers poorly to InternVL. Moreover, the proposed layer profile adds negligible improvement over matched-scale task arithmetic, weakening the algorithmic contribution.
  2. The presentation of the paper is relatively weak, especially the readability of Figure 1 is very poor. I even suspect that this was randomly generated by the author using LLM (just a feeling, please forgive me if there are any errors), I suggest the author pay more attention.
Questions for authors

See Critical Weaknesses

5 comments

Thank you for your review and for highlighting the projection surgeries and matched-scale probes. Below we address your questions on the standalone role of CFS and the readability of Figure 1 and the text. All new results and revisions are included in the revised PDF.

1. Standalone Value of CFS, Transfer to InternVL2.5, and Profile Gains (W1)

(a) Why Tables 1 and 3 used OptMerge's scale. In Sections 5.1 and 5.4 (Tables 1 and 3), we set cc to OptMerge's measured aˉ\bar{a} as a controlled ablation so global scale is held fixed while testing the orthogonal residual D⊥D^\perp. That comparison establishes our mechanistic result: 300300 gradient steps per layer (3.03.0 hours at 7B) learn a layer profile already given in closed form by task-vector Gram inner products rℓr_\ell (Spearman 0.7020.702, 8.4%8.4\% mean residual in Section 5.1), while D⊥D^\perp adds only +0.29+0.29 points on the benchmarks (Section 5.4).

(b) Fixing parameter-free transfer to InternVL2.5-1B without an optimizer (CFS-Prior). CFS-0 dropped to 44.7444.74 on InternVL2.5-1B not because the relative Gram profile rℓr_\ell failed (at aˉ=0.367\bar{a} = 0.367, CFS-0 still sits within 0.30.3 points of curve(aˉ)\text{curve}(\bar{a})), but because its median-norm rule (rˉ2=meanℓ [mediani∥τi,ℓ∥/∥Δℓ∥]=0.367\bar r_2 = \mathrm{mean_{\ell}}\thinspace [\mathrm{median_i} \lVert \tau_{i,\ell}\rVert / \lVert \Delta_\ell\rVert] = 0.367) ignores spectral rank. Rank-8 LoRA updates on Qwen2-VL-7B perturb at most 8 singular directions per tensor and tolerate larger scales (aˉ∈[0.30,0.60]\bar{a} \in [0.30, 0.60]), whereas full fine-tunes on InternVL2.5-1B perturb hundreds of directions, pulling the optimal scale down toward 1/n=0.201/n = 0.20 (aˉ∈[0.15,0.22]\bar{a} \in [0.15, 0.22]) and leaving 0.3670.367 on the downward slope.

In the revised PDF (Abstract, Sections 1, 3, 5.4, and the new CFS-Prior Appendix), we fix this from stored task vectors alone via a subspace scale prior (CFS-Prior). Minimizing ∑i=1n∥(λΔℓ−τi,ℓ)Pi,ℓ∥F2\sum_{i=1}^n \lVert (\lambda \Delta_\ell - \tau_{i,\ell}) P_{i,\ell}\rVert_F^2 over λ∈R\lambda \in \mathbb{R}, where Pi,ℓ=Vi,ℓVi,ℓ⊤P_{i,\ell} = V_{i,\ell}V_{i,\ell}^\top projects onto the top-90% Frobenius energy subspace of τi,ℓ\tau_{i,\ell} (spanned by thin right-singular vectors Vi,ℓV_{i,\ell}), gives aˉprior=meanℓ ∑i⟨ΔℓPi,ℓ,τi,ℓPi,ℓ⟩F∑i∥ΔℓPi,ℓ∥F2\bar a_{\mathrm{prior}} = \mathrm{mean_{\ell}}\thinspace \frac{\sum_i \langle \Delta_\ell P_{i,\ell}, \tau_{i,\ell} P_{i,\ell}\rangle_F}{\sum_i \lVert \Delta_\ell P_{i,\ell}\rVert_F^2} and sets c=aˉprior/rˉc = \bar a_{\mathrm{prior}} / \bar r (with rˉ=meanℓ rℓ\bar r = \mathrm{mean_{\ell}}\thinspace r_\ell). Computing thin products ΔℓVi,ℓ\Delta_\ell V_{i,\ell} adds under 20 seconds on GPU without forming dense projection matrices (disk I/O dominates total merge time), setting aˉprior=0.288\bar a_{\mathrm{prior}} = 0.288 on Qwen2-VL-7B and 0.2140.214 on InternVL2.5-1B with zero data or gradient steps. Below we also compare tuning cc (CFS-Val) against tuning TA's λ\lambda (TA-Val) across a 4-point grid on the same 5% held-out split (150 items per benchmark):

MethodOptimizer-Free?Data-Free?7B Time (Merge + Tune)1B Time (Merge + Tune)Qwen2-VL-7B (Avg7\mathrm{Avg_7})InternVL2.5-1B (Avg7\mathrm{Avg_7})
OptMerge (300 gradient steps)NoYes180 + 0 = 180 min9 + 0 = 9 min57.7949.04
OptMerge∥\text{OptMerge}^\parallel (projected)NoYes180 + 0 = 180 min9 + 0 = 9 min57.5047.88
CFS-0 (median-norm, initial draft)YesYes9 + 0 = 9 min1 + 0 = 1 min57.3744.74
CFS-Prior (subspace prior)YesYes9 + 0 = 9 min1 + 0 = 1 min57.4647.81
TA-Val (4-grid on 5% subset)YesNo< 1 + 17 = 18 min< 1 + 6 = 7 min57.4847.58
CFS-Val (4-grid on 5% subset)YesNo9 + 17 = 26 min1 + 6 = 7 min57.5247.92

(c) What CFS contributes as a standalone method. First, standard Task Arithmetic has no data-free rule for picking λ\lambda (λ=1.0\lambda = 1.0 collapses, and fixed defaults like λ=0.30\lambda = 0.30 drop on full fine-tunes). CFS-Prior sets both global scale and layer profile from stored task vectors alone, scoring 57.4657.46 on Qwen2-VL-7B (99 min) and improving over CFS-0 by +3.07+3.07 points on InternVL2.5-1B (44.7444.74 to 47.8147.81 in 11 min, vs. 47.8847.88 for OptMerge∥\text{OptMerge}^\parallel).

Second, as shown by our fixed-scale probes in Section 5.3, even 300300-step OptMerge∥\text{OptMerge}^\parallel sits only +0.28+0.28 to +0.54+0.54 points above constant-scale curve(aˉ)\text{curve}(\bar{a}) on shared-vocabulary models, which bounds what any parallel profile can gain. On 2828-layer rank-88 LoRA Qwen2-VL-7B, cross-layer interference is nearly flat and CFS ties TA. When cross-layer interference IℓI_\ell varies more across depth, CFS-Val consistently captures that parallel gain over tuned constant-scale TA-Val under the same 5%5\% validation budget, across full fine-tuned InternVL2.5-1B and the 4040-layer LLaVA-1.5-13B and full-FT Llama-3-8B tracks detailed in our reply to Reviewer jrE7 (Section 5.5 and the Generalization Appendix):

Model & Expert ConfigurationTuned Constant-Scale TA (TA-Val)Tuned CFS (CFS-Val)Gain (Δ\Delta)
InternVL2.5-1B (5 full-FT MLLM experts)47.5847.92+0.34
LLaVA-1.5-13B (4 LoRA MLLM experts)60.9061.38+0.48
Llama-3-8B (4 full-FT language experts)63.7164.12+0.41

2. Rewriting the Manuscript and Redrawing Figure 1 (W2)

No apology is needed at all, and we appreciate you pointing this out constructively. We designed and ran all experiments and derivations ourselves, but as noted in our AI Use Statement, right before the deadline we used an LLM to squeeze an over-length draft into 99 pages and used layout scripts to pack sub-panels into Figures 1 and 2. That compression left the prose overly dense and made Figure 1 crowded and hard to follow.

In the revised PDF, we rewrote the paper by hand from the Abstract through Section 7 so that the sentences read naturally and use consistent notation throughout. We also rebuilt Figure 1 from scratch: Panel (a) lays out the workflow in two clear stages (Part 1 tracing task vectors Δℓ\Delta_\ell to the merged displacement DℓD_\ell, orthogonal split D∥D^\parallel vs. D⊥D^\perp, projection surgery, fixed-scale probes, and (aˉ,g^)(\bar{a}, \hat{g}); Part 2 showing cross-expert interference IℓI_\ell, the Gram ratio rℓr_\ell, and CFS λℓ=c⋅rℓ\lambda_\ell = c \cdot r_\ell), paired with clean vector plots for curve prediction in Panel (b) and the 28-layer OptMerge profile in Panel (c). Figure 2 was likewise redrawn on a white background with all 28 per-layer points shown in Panel (b).

In the revised PDF, newly added methods and experiments are highlighted in dark blue, while our full-text prose rewrite is kept in black for readability. Thank you again for your helpful feedback, and please let us know if anything else can be clarified.

Reviewer_jrE7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes data-free merging of multimodal experts through the merged displacement rather than through the procedure that produced it. It summarizes the displacement with two quantities. The effective scale ā is the mean per-tensor ratio of the merged displacement’s norm to the naive task-vector sum’s. The signed depth slope ĝ is an energy-weighted regression of that ratio on layer depth. Each method’s displacement is projected onto the naive sum, and its score is compared with a task-arithmetic curve evaluated at the measured ā. On Qwen2-VL-7B (LoRA experts) and InternVL2.5-1B, using OptMerge’s five capability experts, five of the six projected merges land within ±1 point of the curve. Fixed-scale depth probes explain the sixth. Per-layer optimization is found to be about 91% parallel in energy on Qwen2-VL, and its per-layer scale profile is approximated by a closed-form Gram-matrix ratio r_ℓ. The resulting rule, CFS, matches the projected optimizer at about 1/20 of the merging cost. The orthogonal component adds 0.3–1.4 points on the multimodal tracks but carries more points on the CLIP ViT-B/32 eight-task classification suite.

Critical strengths
  1. Analyzing the displacement a method leaves behind, instead of the mechanism it claims, is a clean way to compare very different merging methods. The projection surgery makes the claims falsifiable and could be adopted as a standard diagnostic for future merging papers.

  2. Iso-C’s collapse is fully reversed by keeping only its parallel component. Task arithmetic at λ = 0.45 is statistically tied with 300-step OptMerge (57.55 vs. 57.79), and the optimizer’s margin over the best single scalar is under half a point under half-split selection. ogether, these show how much of the reported gain from elaborate methods comes from scale alone.

  3. At matched scale, deep-tilted profiles beat shallow-tilted ones in all three architectures, and the shallow branch degrades monotonically with dose. This dose–response pattern, from probes that hold ā fixed, separates shape from scale more rigorously than prior layer-scaling work such as LiNeS. It also gives a plausible explanation for why depth re-weighting helps on CLIP classification but hurts on Qwen2-VL.

Critical weaknesses
  1. A projected merge is, by construction, task arithmetic with per-layer coefficients. That it lands near the task-arithmetic curve at its mean coefficient is close to guaranteed whenever the layer profile matters little, and the paper itself shows the profile adds +0.01. Moreover, the Qwen2-VL curve is a plateau over λ ∈ [0.30, 0.60], where almost any ā predicts about 57. Only Iso-C∥ sits on an informative part of the curve. The abstract also says the curve “predicts the benchmark average of three published methods”, but what is predicted is the projected merges. The original Iso-C is not predicted at all.

  2. The motivation is that each candidate merge costs about an hour to evaluate. However, building the reference curve requires a full sweep of evaluated task-arithmetic merges, and the prediction then only says the method will score about as well as task arithmetic at that scale. In practice the account tells the user to tune λ for task arithmetic, which practitioners already do. It does not choose between methods before evaluation in any way beyond that.

  3. It is already known that the scaling coefficient dominates task-arithmetic performance and that tuned task arithmetic is a strong baseline. The main new elements are the projection diagnostic and the measured parallel fraction of optimizers. These are useful, but they make an analysis paper on one benchmark family more than the general predictive theory the paper suggest.

  4. All multimodal results use one expert set, two models, one evaluation protocol and a single averaged metric. The Qwen2-VL experts are rank-8 LoRA, whose low-rank task vectors may by themselves explain the high parallel fraction. On full fine-tunes, the parallel fraction falls to 68–72%. The Grounding expert’s own skill is not among the seven benchmarks. The finding that the orthogonal component matters greatly for disjoint label spaces also suggests the scale-only account may not transfer to other capability mixes, larger models or language-only merging.

Questions for authors
  1. If the reference curve is replaced by any monotone-then-plateau function fitted to a few points, how many of the six projected methods stay in band? What fraction of the predictions fall on the flat region of the curve, where any ā would succeed?

  2. Without access to the optimizer’s ā, how does CFS with c chosen from the task vectors alone (or on held-out items) compare with task arithmetic tuned on the same held-out items? Please report the time comparison for this practical setting.

7 comments

Thank you for the detailed critique. Your points on the Qwen2-VL-7B plateau, why projected merges fall near the task-arithmetic curve, and testing CFS without OptMerge's aˉ\bar{a} led directly to new checks and framing changes in the revised PDF. We answer Q1 and Q2 first, then summarize our updates for W1-W4.

1. Parametric Curve Fit, Flat-Region Fraction, and Why In-Band Accuracy Is Not Automatic (Q1 & W1)

Q1(a): Fitting a simple monotone-then-plateau curve to a few points. Following your suggestion, we replaced the 99-node sweep with a 33-parameter monotone-then-plateau function f(λ)=a−bexp⁡(−kλ)f(\lambda) = a - b\exp(-k\lambda) on Qwen2-VL-7B (and the 33-parameter unimodal form y0+aλexp⁡(−kλ)y_0 + a\lambda\exp(-k\lambda) on InternVL2.5-1B to reflect its post-peak decline beyond λ=0.15\lambda = 0.15) fitted to 33 points (λ∈{0.15,0.30,0.50}\lambda \in \{0.15, 0.30, 0.50\}). Five of the 66 projected merges stay within ±1.0\pm 1.0 point of the fitted curve, with residuals of +0.38+0.38, +0.69+0.69, and −1.31-1.31 on Qwen2-VL-7B and +0.34+0.34, −0.22-0.22, and +0.19+0.19 on InternVL2.5-1B (a mean shift of 0.070.07 points vs. the 99-node curve; curve MAE of 0.140.14 and 0.180.18). Evaluating those 33 points on a 5%5\% stratified subset (150150 items per benchmark; 12.612.6 minutes total at 7B) also keeps 55 of 66 projected merges within ±1.0\pm 1.0 point (Section 5.2 and the new Appendix on low-budget calibration).

Q1(b): Flat-region fraction and the near-tautology question. On Qwen2-VL-7B, your point is fair: 22 of its 33 projected merges (OptMerge∥\text{OptMerge}^\parallel at aˉ=0.295\bar{a} = 0.295, predicting 57.2157.21, and WUDI∥\text{WUDI}^\parallel at aˉ=0.68\bar{a} = 0.68, predicting 56.7156.71) sit near the edges of the [0.30,0.60][0.30, 0.60] plateau, while only Iso-C∥\text{Iso-C}^\parallel (aˉ=0.1427\bar{a} = 0.1427, predicting 46.0646.06 vs. 46.8346.83 actual) lies on the steep rising slope. On InternVL2.5-1B, however, 00 of 33 sit on a flat region because that curve has no plateau at all: it peaks sharply near λ=0.15\lambda = 0.15 and drops steadily thereafter (down to 45.1245.12 at aˉ=0.35\bar{a} = 0.35, where the 33 parameter-free rows in Table 2 also sit on a steep downward slope). Across both MLLMs, 22 of 66 primary projected merges (33%33\%, or 22 of 9=22%9 = 22\% including the parameter-free rules) sit near a plateau, while 67%67\% (78%78\% overall) lie on sloped regions. Also, Qwen2-VL-7B's flat aggregate curve comes from opposing steep slopes on individual tasks (Math and OCR drop with λ\lambda, whereas Perception rises); evaluating separate task curves curvek(aˉ)\text{curve}_k(\bar{a}) on each of the 77 sloped benchmarks across both MLLMs puts 3838 of 4242 method-benchmark pairs (90.5%90.5\%) within ±1.0\pm 1.0 point (MAE=0.62\text{MAE} = 0.62; updated Section 5.2).

We also agree with your mathematical point in W1: when a curve is locally smooth and layer-wise variation has little effect, any nearly flat per-layer scaling ∑ℓλℓ∥Δℓ\sum_\ell \lambda_\ell^\parallel \Delta_\ell will naturally land near curve(aˉ)\text{curve}(\bar{a}). What makes the test falsifiable rather than automatic is two empirical facts that were not clear beforehand:

  1. Per-layer scalings along Δℓ\Delta_\ell do miss the curve when they tilt across depth, even inside Qwen2-VL-7B's [0.30,0.60][0.30, 0.60] plateau: WUDI∥\text{WUDI}^\parallel misses by −1.39-1.39 points because of its deep tilt (g^=+0.45\hat{g} = +0.45), fixed-scale shallow probes at aˉ=0.45\bar{a} = 0.45 lose 5.655.65 to 12.7712.77 points, and LiNeS (also a pure per-layer scaling along Δℓ\Delta_\ell) drops 11.3311.33 to 12.4012.40 points below the curve. Landing in band thus rules out strong depth reshaping.
  2. Methods built around spectral isotropy or 300300-step per-layer optimization are generally assumed to get their gains from orthogonal directions or complex layer schedules. Showing that OptMerge and WUDI leave 68%68\% to 91%91\% of their update energy along Δℓ\Delta_\ell with near-zero depth tilt, and that projecting collapsed Iso-C (17.9817.98) onto Δℓ\Delta_\ell restores it onto curve(aˉ)\text{curve}(\bar{a}) (46.8346.83 vs. 46.0646.06), is an empirical finding exposed by projection surgery rather than an identity.

2. Practical CFS vs. Task Arithmetic Without OptMerge's aˉ\bar{a} (Q2)

In Tables 1 and 3, we set cc to OptMerge's aˉ\bar{a} only to hold global scale fixed when isolating D⊥D^\perp. To compare Task Arithmetic (TA) and CFS when no optimizer is available, we added two practical settings in Section 3, Section 5.4, and the new CFS-Prior Appendix. First, from task vectors alone, we derive CFS-Prior by minimizing ∑i=1n∥(λΔℓ−τi,ℓ)Pi,ℓ∥F2\sum_{i=1}^n \lVert (\lambda \Delta_\ell - \tau_{i,\ell}) P_{i,\ell}\rVert_F^2 over λ∈R\lambda \in \mathbb{R} (where Pi,ℓ=Vi,ℓVi,ℓ⊤P_{i,\ell} = V_{i,\ell} V_{i,\ell}^\top projects onto the top-90%90\% spectral energy subspace of τi,ℓ\tau_{i,\ell} spanned by thin right-singular vectors Vi,ℓV_{i,\ell}), giving aˉprior=meanℓ ∑i⟨ΔℓPi,ℓ,τi,ℓPi,ℓ⟩F∑i∥ΔℓPi,ℓ∥F2\bar a_{\mathrm{prior}} = \mathrm{mean_{\ell}}\, \frac{\sum_i \langle \Delta_\ell P_{i,\ell}, \tau_{i,\ell} P_{i,\ell}\rangle_F}{\sum_i \lVert \Delta_\ell P_{i,\ell}\rVert_F^2} and c=aˉprior/rˉc = \bar a_{\mathrm{prior}} / \bar r. Evaluating thin products ΔℓVi,ℓ\Delta_\ell V_{i,\ell} takes under 2020 seconds on GPU (checkpoint I/O dominates wall time), setting aˉprior=0.288\bar a_{\mathrm{prior}} = 0.288 on Qwen2-VL-7B and 0.2140.214 on InternVL2.5-1B without data and fixing CFS-0's over-scaling (aˉ=0.367\bar{a} = 0.367) on full fine-tunes. Second, on a 5%5\% held-out subset (150150 items per benchmark), we tune λ\lambda for TA (TA-Val) and cc for CFS (CFS-Val) over a 44-point grid on the exact same items:

Method / Practical Protocol7B Time (Merge + Tune)1B Time (Merge + Tune)Qwen2-VL-7B (Avg7\mathrm{Avg_7})InternVL2.5-1B (Avg7\mathrm{Avg_7})
OptMerge (300 gradient steps)180 + 0 = 180 min9 + 0 = 9 min57.7949.04
OptMerge∥\text{OptMerge}^\parallel (projected)180 + 0 = 180 min9 + 0 = 9 min57.5047.88
CFS-0 (median-norm, data-free)9 + 0 = 9 min1 + 0 = 1 min57.3744.74
CFS-Prior (subspace prior, data-free)9 + 0 = 9 min1 + 0 = 1 min57.4647.81
TA-Val (4-grid on 5% subset)< 1 + 17 = 17 min< 1 + 6 = 6 min57.4847.58
CFS-Val (4-grid on 5% subset)9 + 17 = 26 min1 + 6 = 7 min57.5247.92

On rank-88 LoRA Qwen2-VL-7B, interference varies little across depth, so TA-Val and CFS-Val are tied (57.4857.48 vs. 57.5257.52; and 57.3257.32 vs. 57.3857.38 across the 200200 half-splits of Section 5.2). Tuned TA already reaches the parallel ceiling there, while CFS serves to explain OptMerge analytically. Where interference IℓI_\ell varies across layers, CFS-Val consistently beats TA-Val under the same 5%5\% budget: +0.34+0.34 on full fine-tuned InternVL2.5-1B (47.9247.92 vs. 47.5847.58, 95%95\% CI [+0.09,+0.59][+0.09, +0.59]; 47.8547.85 vs. 47.5147.51 on 200200 half-splits), +0.48+0.48 on 4040-layer LLaVA-1.5-13B (61.3861.38 vs. 60.9060.90, CI [+0.16,+0.80][+0.16, +0.80]), and +0.41+0.41 on full fine-tuned Llama-3-8B (64.1264.12 vs. 63.7163.71, CI [+0.12,+0.70][+0.12, +0.70]). Data-free CFS-Prior also reaches 57.4657.46 and 47.8147.81 with zero evaluations.

3. Re-Framing Around Mechanistic Decomposition (W1, W2 & W3)

We updated the Abstract, Section 1, Section 6, and Section 7 along the lines you suggested: (1) we removed all "before any evaluation" claims and state plainly that curve(λ)\text{curve}(\lambda) is calibrated once per expert set via task arithmetic; (2) we state up front in the Abstract and Introduction that curve(aˉ)\text{curve}(\bar{a}) predicts projected merges (OptMerge∥\text{OptMerge}^\parallel, WUDI∥\text{WUDI}^\parallel, Iso-C∥\text{Iso-C}^\parallel), contrasting unprojected Iso-C (17.9817.98) with Iso-C∥\text{Iso-C}^\parallel (46.8346.83 vs. 46.0646.06 predicted); and (3) we framed the paper around the weight decomposition (aˉ\bar{a}, g^\hat{g}, D⊥D^\perp) and our closed-form Gram approximation rather than evaluation-free method selection. We also rewrote the full text by hand for clarity.

4. Grounding Evaluation, LoRA vs. Full FT, 13B MLLM, and Language-Only Merging (W4)

Grounding expert evaluation (RefCOCO / RefCOCO+ / RefCOCOg). We added explicit pointers in Sections 4 and 5.5 to our referring expression comprehension table in the Appendix. On Qwen2-VL-7B, projected merges keep the Grounding expert's skill and track the curve closely (Iso-C∥\text{Iso-C}^\parallel at aˉ=0.1427\bar{a} = 0.1427 scores 75.8475.84 vs. 75.3075.30 predicted, recovering +51.72+51.72 points over collapsed Iso-C):

BenchmarkBaseGrounding ExpertTA (0.45)OptMergeOptMerge∥\text{OptMerge}^\parallelCFSIso-CIso-C∥\text{Iso-C}^\parallel
RefCOCO/+/g Avg68.4084.2081.1881.6281.4181.3424.1275.84

Subsets, LoRA vs. full fine-tuning, 13B MLLM, and Llama-3-8B language merging. In Section 5.5 and the new Generalization Appendix, we added 33- and 44-expert subsets on Qwen2-VL-7B, a 44-expert LoRA (r=16r=16) merge on 4040-layer LLaVA-1.5-13B (liuhaotian/llava-v1.5-13b, fine-tuned on the VQA, OCR, Chart, and Geometry splits of Wei et al., 2026), and a 44-expert unconstrained full-FT language merge on meta-llama/Meta-Llama-3-8B (TIGER-Lab/MAmmoTH2-8B, ajibawa-2023/Code-Llama-3-8B, allenai/llama-3-tulu-2-8b, and meta-llama/Meta-Llama-3-8B-Instruct, all sharing the exact 128,256128,256-token base vocabulary, evaluated on GSM8K, HumanEval, ARC-Challenge, and IFEval):

Model & ModalityFine-TuningExpertsOptMerge η∥\eta_\parallelCFS SpearmanOptMerge∥\text{OptMerge}^\parallel Scorecurve(aˉ)\text{curve}(\bar{a}) Pred. (Residual)
Qwen2-VL-7B (MLLM)LoRA (r=8)3 Experts92.1%0.7559.1258.84 (+0.28)
Qwen2-VL-7B (MLLM)LoRA (r=8)4 Experts91.4%0.7258.3457.95 (+0.39)
Qwen2-VL-7B (MLLM, paper)LoRA (r=8)5 Experts90.7%0.7057.5057.21 (+0.29)
LLaVA-1.5-13B (MLLM)LoRA (r=16)4 Experts84.6%0.6961.4260.88 (+0.54)
InternVL2.5-1B (MLLM, paper)Full FT5 Experts67.8%0.6647.8847.60 (+0.28)
Llama-3-8B (Language-only)Full FT4 Experts79.4%0.7164.1563.79 (+0.36)

Your intuition about LoRA holds up: low-rank updates have higher parallel energy (84.6%84.6\% to 92.1%92.1\%) than unconstrained full fine-tunes (67.8%67.8\% and 79.4%79.4\%), since full-rank updates put more tail energy into orthogonal singular directions. Still, full fine-tunes on shared-vocabulary MLLMs and LLMs place two-thirds to four-fifths of OptMerge's energy along Δℓ\Delta_\ell (compared to 31.4%31.4\% to 42.8%42.8\% on disjoint-label ViT-B/32), and curve(aˉ)\text{curve}(\bar{a}) predicts OptMerge∥\text{OptMerge}^\parallel within +0.28+0.28 to +0.54+0.54 points across every track.

In the revised PDF, newly added methods and experiments are highlighted in dark blue, while our full-text prose rewrite is kept in black for readability. Thank you again for the constructive critique, and we hope these additions earn your support.

Reviewer_zBBUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between weight geometry and the performance of multimodal model merging. In particular, this work introduces two geometric measurements, effective scale and signed depth slope, to characterize the displacement of merged model weights. Based on these measurements, the authors analyze several existing merging methods and show that their performance can be largely explained by the merging scale. Furthermore, a closed-form scaling strategy (CFS) is proposed to approximate optimization-based merging with lower computational cost. Finally, experiments on 3 model architectures, including Qwen2-VL-7B, InternVL2.5-1B, and CLIP ViT-B/32, are conducted to verify the proposed analysis. This work seems complete; however, I have several concerns about it.

Critical strengths
  1. The motivation is intuitive and interesting. Instead of proposing another complicated merging algorithm, this work investigates why existing methods perform differently from the perspective of weight geometry.

  2. The proposed analysis is relatively comprehensive. The authors consider both merging scale and layer-wise displacement, and use projection experiments to investigate their individual contributions. These analyses provide useful insights into the behavior of existing merging methods.

Critical weaknesses
  1. This paper claims that merging performance can be predicted before evaluation. However, the proposed prediction relies on a task-arithmetic reference curve constructed through benchmark evaluations. Does this mean that multiple evaluations are still required? I think the actual computational savings should be clarified, especially compared with directly searching for a suitable merging coefficient.

  2. Based on Tables 1 and 3, the contribution of CFS needs further clarification. CFS achieves 57.50 on Qwen2-VL, while simple task arithmetic already achieves 57.55 with a suitable coefficient. Besides, the proposed depth profile only contributes 0.01 points under matched scale, and the CFS coefficient is calibrated using the optimizer's measured scale. Does this suggest that selecting the scaling coefficient is more important than the proposed closed-form strategy? How can this coefficient be determined without running the optimizer?

  3. The generalizability of the proposed geometric analysis needs further investigation. The main conclusions are based on two vision-language models with the same set of capability experts, while Table 2 shows considerably different behavior on ViT-B/32. Is there any common criterion to determine when scale alone is sufficient and when shape or orthogonal components become important? I think further validation on different expert configurations would strengthen the conclusions.

  4. I am so concerned about Figure 2. It seems like a totally AI-generated figure.

Questions for authors
  1. How many benchmark evaluations are required to construct the reference curve, and what is the actual computational saving compared with conventional coefficient searching?

  2. How can the CFS scaling coefficient be determined without running an optimization-based method, given that its depth profile provides only a marginal improvement?

  3. Is there any general criterion for determining when the proposed scale-based prediction is applicable? Would the conclusions remain consistent with larger MLLMs and different expert configurations?

  4. Why does this paper give readers the feeling that it is totally generated by AI?

6 comments

Thank you for reviewing our paper and for the direct questions on calibration cost, running CFS without an optimizer, applicability across models, and presentation. We have updated both the experiments and the manuscript (all reflected in the revised PDF) and address each point below.

1. Reference-Curve Calibration vs. Direct Coefficient Search (W1 & Q1)

You are right that building curve(λ)\text{curve}(\lambda) requires benchmark evaluations of task arithmetic across scalar values λ\lambda (99 nodes in our diagnostic plots, taking a median of 8585 minutes per full pass on Qwen2-VL-7B). Saying "before any evaluation" in our original draft went too far, and we have removed that phrasing from the Abstract, Section 1, and Section 7.

Whether calibrating curve(λ)\text{curve}(\lambda) saves compute depends on what a user needs to do. If the goal is simply to produce one merged model, calibrating the curve is the same work as tuning λ\lambda, so it saves nothing over scalar search. For that use case, one can either run our data-free CFS-Prior (99 minutes, zero evaluations; see Point 2) or tune directly on a 5%5\% held-out split (1717 to 2626 minutes).

Compute savings come up when testing several fast weight-space merging methods and their hyperparameter grids (such as TIES, DARE, Iso-C, LiNeS schedules, or MetaGPT, each taking 11 to 33 minutes to merge in weight space) to see if any candidate improves over task arithmetic via orthogonal updates (D⊥D^\perp) or suffers from collapse. Once a reference curve is calibrated for a given set of experts, computing (aˉ,g^,η∥)(\bar{a}, \hat{g}, \eta_\parallel) for a new merge takes under 1010 seconds in weight space. That quick check filters out collapsed merges (like unprojected Iso-C at η∥=13.5%\eta_\parallel = 13.5\%), harmful depth skew (like LiNeS at g^≥+0.62\hat{g} \ge +0.62), and near-parallel trials (η∥≳0.90,∣g^∣≈0\eta_\parallel \gtrsim 0.90, \lvert\hat{g}\rvert \approx 0) whose score is already bounded by curve(aˉ)\text{curve}(\bar{a}), leaving full benchmark evaluation only for well-scaled candidates with non-trivial D⊥D^\perp.

Calibrating curve(λ)\text{curve}(\lambda) in practice also does not need 99 full passes (Section 5.2 and the new Appendix on low-budget calibration). Fitting a 33-parameter saturating curve to 33 points (λ∈{0.15,0.30,0.50}\lambda \in \{0.15, 0.30, 0.50\}) matches our 99-node curve with an MAE of 0.140.14 points on Qwen2-VL-7B and 0.180.18 points on InternVL2.5-1B, and 55 of the 66 projected merges stay within ±1.0\pm 1.0 point. Running those same 33 nodes on a 5%5\% stratified subset (150150 items per benchmark; 4.24.2 minutes per node, 12.612.6 minutes total) also keeps 55 of 66 full-benchmark projected scores within ±1.0\pm 1.0 point:

Screening 6 Non-Optimizer Methods on Qwen2-VL-7BCurve Calibration18 Merges + Weight Checks1 Final Full EvalTotal GPU TimeSpeedup
Conventional full tuning (6 methods, 3 trials each)0 min1,566.0 min (18 full evals)included26.1 hrs1.0x
Dense reference curve (9 full nodes)765.0 min39.0 min85.0 min14.8 hrs1.8x
Sparse reference curve (3 full nodes)255.0 min39.0 min85.0 min6.3 hrs4.1x
Micro-subset curve (3 nodes, 5% items)12.6 min39.0 min85.0 min2.3 hrs11.5x

Counting 22 minutes per merge and 1010 seconds per weight check across 1818 trials (39.039.0 minutes total) plus one final full evaluation on the chosen model cuts total GPU time from 26.126.1 hours to 2.32.3 hours (11.5×11.5\times faster).

2. Why Global Scale Dominates, and Setting cc Without an Optimizer (W2 & Q2)

Your reading of Tables 1 and 3 is spot on: on Qwen2-VL-7B, global scale matters far more than the per-layer profile (+0.01+0.01 points at matched scale), and task arithmetic at λ=0.45\lambda = 0.45 already reaches 57.5557.55 (statistically tied with CFS at 57.5057.50 and within bootstrap noise of 300300-step OptMerge at 57.7957.79).

CFS serves two roles in the paper (clarified in updated Sections 3 and 5.4). First, on the analysis side, without a closed-form approximation we could not tell whether OptMerge's 3.03.0-hour per-layer optimization learns new task directions or mostly rescales layers. Showing that c⋅rℓc \cdot r_\ell matches OptMerge's 196196-tensor profile to an 8.4%8.4\% mean residual (and lands within 0.010.01 points of OptMerge∥\text{OptMerge}^\parallel) shows that gradient optimization mainly recovers Gram-matrix interference shrinkage. On rank-88 LoRA Qwen2-VL-7B, cross-expert interference IℓI_\ell varies very little across layers and the λ\lambda curve is flat over [0.30,0.60][0.30, 0.60], which is why a single scalar works just as well there.

Second, in Tables 1 and 3 we matched cc to OptMerge's aˉ\bar{a} only to hold global scale fixed when isolating the orthogonal residual D⊥D^\perp. When no optimizer is run, cc can be set in two ways (added to Section 3, Section 5.4, and the new Appendix on CFS-Prior):

  1. From stored task vectors alone (CFS-Prior): Median-norm CFS-0 (rˉ2=0.367\bar r_2 = 0.367) ignores spectral rank and over-scales full fine-tunes. Minimizing the layer-wise subspace discrepancy ∑i=1n∥(λΔℓ−τi,ℓ)Pi,ℓ∥F2\sum_{i=1}^n \lVert (\lambda \Delta_\ell - \tau_{i,\ell}) P_{i,\ell}\rVert_F^2 along span(Δℓ)\text{span}(\Delta_\ell), where Pi,ℓ=Vi,ℓVi,ℓ⊤P_{i,\ell} = V_{i,\ell} V_{i,\ell}^\top projects onto the top-90%90\% spectral energy subspace of τi,ℓ\tau_{i,\ell} (spanned by thin right-singular vectors Vi,ℓV_{i,\ell}), gives aˉprior=meanℓ ∑i⟨ΔℓPi,ℓ,τi,ℓPi,ℓ⟩F∑i∥ΔℓPi,ℓ∥F2\bar a_{\mathrm{prior}} = \mathrm{mean_{\ell}}\, \frac{\sum_i \langle \Delta_\ell P_{i,\ell}, \tau_{i,\ell} P_{i,\ell}\rangle_F}{\sum_i \lVert \Delta_\ell P_{i,\ell}\rVert_F^2} and sets c=aˉprior/rˉc = \bar a_{\mathrm{prior}} / \bar r (with rˉ=meanℓ rℓ\bar r = \mathrm{mean_{\ell}}\, r_\ell). As effective rank approaches full rank (Pi,ℓ→IP_{i,\ell} \to I), this ratio approaches 1/n=0.201/n = 0.20. It gives aˉprior=0.214\bar a_{\mathrm{prior}} = 0.214 on full fine-tuned InternVL2.5-1B and 0.2880.288 on rank-88 LoRA Qwen2-VL-7B with zero data or gradient steps. Because we evaluate thin products ΔℓVi,ℓ\Delta_\ell V_{i,\ell} without forming dense projection matrices, the extra SVD step takes under 2020 seconds on GPU; disk I/O for loading and saving checkpoints dominates total wall time, so both CFS-0 and CFS-Prior round to 99 minutes at 7B (11 minute at 1B).
  2. With a small validation split (CFS-Val vs. TA-Val): We tune cc or λ\lambda over a 44-point grid on the same 5%5\% subset (150150 items per benchmark; 1717 minutes of evaluation at 7B).
MethodUses Optimizer?7B Merge + Tune TimeQwen2-VL-7B (Avg7\mathrm{Avg_7})InternVL2.5-1B (Avg7\mathrm{Avg_7})
OptMerge (300 gradient steps)Yes180 min57.7949.04
OptMerge∥\text{OptMerge}^\parallel (parallel projection)Yes180 min57.5047.88
CFS-0 (median-norm, in initial submission)No9 min57.3744.74
CFS-Prior (data-free subspace prior)No9 min57.4647.81
TA-Val (tuned on 5% subset)No17 min57.4847.58
CFS-Val (tuned on 5% subset)No26 min57.5247.92

Without validation data or an optimizer, CFS-Prior reaches 57.4657.46 on Qwen2-VL-7B and improves over CFS-0 by +3.07+3.07 points on InternVL2.5-1B (47.8147.81 vs. 47.8847.88 for OptMerge∥\text{OptMerge}^\parallel). On full fine-tuned InternVL2.5-1B, where IℓI_\ell varies much more across depth, CFS-Val also beats TA-Val by +0.34+0.34 points (95%95\% CI [+0.09,+0.59][+0.09, +0.59]).

3. Criterion for Applicability and Scaling to Larger Models / Subsets (W3 & Q3)

Comparing our generative tracks with ViT-B/32 gives a simple two-step check that requires no benchmark evaluations (added to Section 6). First, check the output space: scale-based prediction applies when experts share a generative vocabulary and output head (as in MLLM or LLM skill fine-tunes). When experts instead use separate classification heads over disjoint label sets (as in 88-task CLIP ViT-B/32), cross-task routing conflicts cannot be resolved by rescaling Δℓ\Delta_\ell, and orthogonal updates carry 21.0721.07 to 47.0547.05 points.

Second, once θm\theta_m is formed on a shared-vocabulary model, compute η∥=∥D∥∥2/∥D∥2\eta_\parallel = \lVert D^\parallel\rVert^2 / \lVert D\rVert^2 and ∣g^∣\lvert\hat{g}\rvert (Eq. 2), which takes under 1010 seconds: when η∥≳0.65\eta_\parallel \gtrsim 0.65 and ∣g^∣≲0.30\lvert\hat{g}\rvert \lesssim 0.30, curve(aˉ)\text{curve}(\bar{a}) predicts the score within ±1.0\pm 1.0 point; ∣g^∣>0.35\lvert\hat{g}\rvert > 0.35 warns of depth-tilt shifts (as in WUDI∥\text{WUDI}^\parallel at +0.45+0.45 and LiNeS at ≥+0.62\ge +0.62); and η∥≪0.50\eta_\parallel \ll 0.50 warns of strong orthogonal distortion before running any benchmark (such as Iso-C at 13.5%13.5\%).

To check different expert counts, a larger MLLM, and full fine-tuned language models, we ran 33-expert (OCR, Chart, Geometry) and 44-expert (VQA, OCR, Chart, Geometry) subsets on Qwen2-VL-7B, a 44-expert LoRA (r=16r=16) merge on 4040-layer LLaVA-1.5-13B (liuhaotian/llava-v1.5-13b), and a 44-expert full fine-tuned merge on Llama-3-8B (meta-llama/Meta-Llama-3-8B; see Section 5.5 and the new Generalization Appendix):

Model & Expert ConfigurationOptMerge η∥\eta_\parallelCFS SpearmanOptMerge∥\text{OptMerge}^\parallel Scorecurve(aˉ)\text{curve}(\bar{a}) Pred. (Residual)
Qwen2-VL-7B (3 LoRA experts)92.1%0.7559.1258.84 (+0.28)
Qwen2-VL-7B (4 LoRA experts)91.4%0.7258.3457.95 (+0.39)
Qwen2-VL-7B (5 LoRA experts, Table 1)90.7%0.7057.5057.21 (+0.29)
LLaVA-1.5-13B (4 LoRA experts)84.6%0.6961.4260.88 (+0.54)
InternVL2.5-1B (5 full-FT experts)67.8%0.6647.8847.60 (+0.28)
Llama-3-8B (4 full-FT language experts)79.4%0.7164.1563.79 (+0.36)

Across subset mixes, at 13B scale, and on full fine-tunes, OptMerge places 67.8%67.8\% to 92.1%92.1\% of its energy along Δℓ\Delta_\ell, CFS tracks the learned profile (Spearman 0.660.66 to 0.750.75), and every projected merge lands within ±1.0\pm 1.0 point of curve(aˉ)\text{curve}(\bar{a}).

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.