- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates which task is more severely degraded when two or more fine-tuned experts are merged, rather than how much the merged model degrades on average. The theoretical analysis applies a second-order Taylor expansion along the segment from each expert to the merged model. This yields a two-sided bound on each expert’s loss increase, expressed in terms of the interferer energy and a coherence bracket. The empirical study covers eight GLUE tasks, LoRA experts from four ~7B model families, and TA, TIES, and DARE merges, for a total of 486 directional evaluations. Finally, the authors intervene on training exposure and on coefficient allocation. These interventions show that both the task-vector norm and the partners’ degradation increase monotonically with training-set size. They also show that shrinking only the larger vector protects the smaller-norm task better than scaling both vectors uniformly, although sometimes at the expense of the larger-norm task.
Critical strengths
- Careful statistical treatment. Because experts are shared across pairs, the observations are not independent. The authors account for this dependence through shared-expert (dyadic) standard errors, expert-level bootstraps, and exact permutation tests over all norm assignments. This is an appropriate and commendable choice.
- A meaningful baseline. Comparing the task-vector norm against training-set size and optimizer step count is a valuable control. The finding that the norm carries no predictive information beyond training exposure is useful for the community.
Critical weaknesses
- Presentation and readability. The manuscript is difficult to follow:
- Several paragraphs are dense with formulas, which makes them very hard to read.
- The notation is overloaded.
- Many paragraphs read as experimental logs, with long chains of numerical results and frequent references to the appendix.
- The text contains unusual formatting choices, such as chained arrows to report numerical series (e.g., “11 → 27 → 69 → 302”). This style is often typical of AI-generated text.The use of AI tools is not a concern in itself, and the authors disclose it. My concern is whether the generated text has been sufficiently proofread and edited for a human reader.
- Limited novelty. The contribution appears incremental with respect to prior work, in particular Section 6.1 of DisTaC (Yoshida et al., 2026). That section already establishes that norm imbalance between task vectors drives the asymmetric degradation, and traces it to differences in training steps and learning rate.
- Limited experimental scope. The main evidence comes from GLUE, which is a relatively easy benchmark for 7B models. The standard vision merging benchmarks (e.g., ViT models on the 8/14/20-task suites) are absent. The generative MergeBench experts appear only in an appendix check (Appendix G), and there the proposed ordering rules do not transfer. This raises doubts about how general the conclusions are.
Questions for authors
- Could the authors carefully revise the manuscript for readability? Specifically, reducing the density of numerical results in the main text, simplifying the notation, and moving secondary statistics to tables or the appendix would make the paper substantially more accessible.
- The degradation analysis aggregates heterogeneous GLUE metrics (accuracy, F1, Matthews correlation, and Pearson correlation), which are not directly comparable in magnitude. For instance, a drop of a few points in accuracy is not equivalent to a drop of the same size in F1 or MCC. How does the analysis account for these differences, and how sensitive are the main conclusions to them?
- Beyond the role of norm imbalance, which is already documented in prior work, what do the authors consider the central novel contribution of this paper?
- What distinguishes the settings in which the norm-based ordering holds, such as full-data GLUE experts, from those in which it fails, such as exposure-matched experts and the MergeBench generative experts?