- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper analyzes multimodal model merging through effective update scale and depth-dependent shape. By projecting merged updates onto the task-vector sum, it finds that a calibrated task-arithmetic curve predicts five of six projected merges within one point across two multimodal architectures. It also proposes a closed-form scaling rule that approximates the projected optimizer at substantially lower merging cost. Controlled shape probes and classification experiments examine when scale alone is insufficient.
Critical strengths
- The analysis experiment was relatively insightful.Projection surgeries and matched-scale shape probes help distinguish magnitude effects from directional changes, providing stronger evidence than correlations alone. This substantially strengthens the paper’s central analysis.
- Well-scaled task arithmetic nearly matches optimization-based merging on the studied multimodal models. The classification counterexamples also usefully expose the limits of this finding.
Critical weaknesses
- The research content and analysis of this article are very interesting, but the standalone contribution of CFS is unclear. Its strongest results use scale information from the optimizer, while the parameter-free variant transfers poorly to InternVL. Moreover, the proposed layer profile adds negligible improvement over matched-scale task arithmetic, weakening the algorithmic contribution.
- The presentation of the paper is relatively weak, especially the readability of Figure 1 is very poor. I even suspect that this was randomly generated by the author using LLM (just a feeling, please forgive me if there are any errors), I suggest the author pay more attention.
Questions for authors
See Critical Weaknesses
5 comments
(c) What CFS contributes as a standalone method. First, standard Task Arithmetic has no data-free rule for picking ( collapses, and fixed defaults like drop on full fine-tunes). CFS-Prior sets both global scale and layer profile from stored task vectors alone, scoring on Qwen2-VL-7B ( min) and improving over CFS-0 by points on InternVL2.5-1B ( to in min, vs. for ).
Second, as shown by our fixed-scale probes in Section 5.3, even -step sits only to points above constant-scale on shared-vocabulary models, which bounds what any parallel profile can gain. On -layer rank- LoRA Qwen2-VL-7B, cross-layer interference is nearly flat and CFS ties TA. When cross-layer interference varies more across depth, CFS-Val consistently captures that parallel gain over tuned constant-scale TA-Val under the same validation budget, across full fine-tuned InternVL2.5-1B and the -layer LLaVA-1.5-13B and full-FT Llama-3-8B tracks detailed in our reply to Reviewer jrE7 (Section 5.5 and the Generalization Appendix):
| Model & Expert Configuration | Tuned Constant-Scale TA (TA-Val) | Tuned CFS (CFS-Val) | Gain () |
|---|---|---|---|
| InternVL2.5-1B (5 full-FT MLLM experts) | 47.58 | 47.92 | +0.34 |
| LLaVA-1.5-13B (4 LoRA MLLM experts) | 60.90 | 61.38 | +0.48 |
| Llama-3-8B (4 full-FT language experts) | 63.71 | 64.12 | +0.41 |
2. Rewriting the Manuscript and Redrawing Figure 1 (W2)
No apology is needed at all, and we appreciate you pointing this out constructively. We designed and ran all experiments and derivations ourselves, but as noted in our AI Use Statement, right before the deadline we used an LLM to squeeze an over-length draft into pages and used layout scripts to pack sub-panels into Figures 1 and 2. That compression left the prose overly dense and made Figure 1 crowded and hard to follow.
In the revised PDF, we rewrote the paper by hand from the Abstract through Section 7 so that the sentences read naturally and use consistent notation throughout. We also rebuilt Figure 1 from scratch: Panel (a) lays out the workflow in two clear stages (Part 1 tracing task vectors to the merged displacement , orthogonal split vs. , projection surgery, fixed-scale probes, and ; Part 2 showing cross-expert interference , the Gram ratio , and CFS ), paired with clean vector plots for curve prediction in Panel (b) and the 28-layer OptMerge profile in Panel (c). Figure 2 was likewise redrawn on a white background with all 28 per-layer points shown in Panel (b).
In the revised PDF, newly added methods and experiments are highlighted in dark blue, while our full-text prose rewrite is kept in black for readability. Thank you again for your helpful feedback, and please let us know if anything else can be clarified.
Thank you for your review and for highlighting the projection surgeries and matched-scale probes. Below we address your questions on the standalone role of CFS and the readability of Figure 1 and the text. All new results and revisions are included in the revised PDF.
1. Standalone Value of CFS, Transfer to InternVL2.5, and Profile Gains (W1)
(a) Why Tables 1 and 3 used OptMerge's scale. In Sections 5.1 and 5.4 (Tables 1 and 3), we set c to OptMerge's measured aˉ as a controlled ablation so global scale is held fixed while testing the orthogonal residual D⊥. That comparison establishes our mechanistic result: 300 gradient steps per layer (3.0 hours at 7B) learn a layer profile already given in closed form by task-vector Gram inner products rℓ (Spearman 0.702, 8.4% mean residual in Section 5.1), while D⊥ adds only +0.29 points on the benchmarks (Section 5.4).
(b) Fixing parameter-free transfer to InternVL2.5-1B without an optimizer (CFS-Prior). CFS-0 dropped to 44.74 on InternVL2.5-1B not because the relative Gram profile rℓ failed (at aˉ=0.367, CFS-0 still sits within 0.3 points of curve(aˉ)), but because its median-norm rule (rˉ2=meanℓ[mediani∥τi,ℓ∥/∥Δℓ∥]=0.367) ignores spectral rank. Rank-8 LoRA updates on Qwen2-VL-7B perturb at most 8 singular directions per tensor and tolerate larger scales (aˉ∈[0.30,0.60]), whereas full fine-tunes on InternVL2.5-1B perturb hundreds of directions, pulling the optimal scale down toward 1/n=0.20 (aˉ∈[0.15,0.22]) and leaving 0.367 on the downward slope.
In the revised PDF (Abstract, Sections 1, 3, 5.4, and the new CFS-Prior Appendix), we fix this from stored task vectors alone via a subspace scale prior (CFS-Prior). Minimizing ∑i=1n∥(λΔℓ−τi,ℓ)Pi,ℓ∥F2 over λ∈R, where Pi,ℓ=Vi,ℓVi,ℓ⊤ projects onto the top-90% Frobenius energy subspace of τi,ℓ (spanned by thin right-singular vectors Vi,ℓ), gives aˉprior=meanℓ∑i∥ΔℓPi,ℓ∥F2∑i⟨ΔℓPi,ℓ,τi,ℓPi,ℓ⟩F and sets c=aˉprior/rˉ (with rˉ=meanℓrℓ). Computing thin products ΔℓVi,ℓ adds under 20 seconds on GPU without forming dense projection matrices (disk I/O dominates total merge time), setting aˉprior=0.288 on Qwen2-VL-7B and 0.214 on InternVL2.5-1B with zero data or gradient steps. Below we also compare tuning c (CFS-Val) against tuning TA's λ (TA-Val) across a 4-point grid on the same 5% held-out split (150 items per benchmark):