- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether merging independently finetuned models can reproduce the results of joint multi-task learning (MTL). The authors argue that there is a missing direction that models trained independently cannot recover compared with MTL, and link it with the performance gap between merging approaches and multi-task learning. The authors propose a one-parameter fit curve that predicts how this gap grows with the number of tasks (i.e., they fit it on small task groups, and use it to predict larger ones). Extracting this direction from the joint training trajectory and adding it to the merged model reduces the loss.
Critical strengths
- The paper addresses a fundamental question, whether independently trained model endpoints can reproduce joint training. It distinguishes geometric reconstruction error from functional loss, showing that directly optimized merging coefficients still leave a loss gap.
- The trajectory analysis provides insight into the missing component by separating local escape from coefficient drift.
- The paper proposes a simple, single-parameter scaling law for the missing energy fraction as task count increases.
Critical weaknesses
W1. The paper is extremely hard to read and engage with. The writing quality is not suitable for a top-tier conference.
W2. The paper is motivated by the claim that "model merging assumes that reweighting independently trained endpoints can stand in for joint training". I believe this is too strong framing. Many merging papers treat MTL as an upper bound. It is also unsurprising that joint training produces components absent from independently trained endpoints.
W3. There are some central claims in the paper that are not supported by the results provided. For example, authors repeatedly claim that the endpoint span captures most of the energy of the joint update. However, Table 1 contradicts this on Suite A where the captured energy is between 21-44%, showing that most of the joint update lies outside the span. This same table also shows that the captured energy does not predict the functional gap (SpanGap column). Concretely, some configurations show smaller captured energy while having a smaller error.
W4. The results across the paper and its own argument indicate that even if we can predict the energy at larger task counts, predicting the missing energy at larger task counts does not establish how well the merged model will perform. Therefore, the practical implications and why it matters are unclear.
W5. One of the points of model merging is that there is no need to perform costly joint training. However, calibrating y requires jointly training on small compositions, and the reinjected direction is read from the joint training.
Questions for authors
See weaknesses.
4 comments
Dear Reviewer dCrh,
We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which we tried our best to address each concern you raised in your feedback.
If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.
Thank you once again for your valuable feedback.
Thanks for the response. I went through the revised manuscript again.
W1: The reply lists five presentation changes, but the only blue edits I could find are the two short passages on p.4 and p.9. I couldn’t locate the plain-language explanation of the growth law, the drift vs. escape sentence, or the at-a-glance table for Sec. 4. If I missed them, please point me to them.
W2/W3: You agree that the gap to joint training is already known and that capture varies a lot by suite. Yet the abstract still opens with “model merging rests on a working assumption ...” and still says the span “captures about 82% of the joint update”. Meanwhile, Sec. 3.1 reports capture dropping from 36.1% at two tasks to 7.1% at twelve. The abstract should match the body.
Scaling law: Your new analysis in the reply to ZFyw finds three things. Captured energy does not grow with K, missing energy grows sub-quadratically and saturates, and a logistic curve predicts held-out K about as well as Eq. 3. That doesn’t fit the per-task / per-pair story behind Eq. 2. Still, the abstract says the energy “scales with the number of task pairs” and the gap “grows quadratically”. The data support a one-parameter saturating fit that extrapolates well, and the claims should be stated at that level.
W4/W5: I accept that missing energy is a structural quantity rather than a performance predictor, but then the “early feasibility assessment” argument is weak. The forecast also isn’t cheap. Table 15 calibrates the coefficient separately for each sequence using joint runs at K = 2 to 9. That is roughly 44 task-trainings to forecast K = 10 to 12, more than training the 12-task joint model once.
The reinjection results with reversed and norm-matched controls, and the optimizer 2x2 control, are interesting. But the headline claims are ahead of the evidence and the presentation problems remain, so I’ll keep my score.
Further Response to Reviewer dCrh
Many thanks for your time and effort to help strengthen our paper, it is our fortune to have a reviewer like you!
W1: Locations of the presentation changes
-
A roadmap of the paper: Page 2, first paragraph of Sec. 2. This explains how Sec. 2 defines the outside-span residual, Sec. 3 studies its task-count dependence and origin, and Sec. 4 tests its functional effect through reinjection.
-
A plain-language explanation of the predictive fit: Immediately after Eq. (3). This explains that a coefficient fitted at small task counts determines the curve used to predict larger counts without refitting.
-
The drift-versus-escape sentence: Page 7, first paragraph of Sec. 3.3: “Drift is the part that better-chosen mixing weights could have tracked, while escape is the part that no choice of weights can reach at all.”
-
The at-a-glance table for Sec. 4: Table 7 in Appendix E. It groups the recovery/history, direction, and dose comparisons, identifies their matched controls, and reports the corresponding sample counts.
W2/W3: Aligning the abstract with the body
We now do not present the gap itself as a new finding, and the capture statistic is clearly qualified throughout.
W2/W3: Abstract–body consistency.
- Abstract opening rewritten (p.1, lines 11–14): the "working assumption" framing is replaced by a testable statement of what linear merging can and cannot span. The Introduction likewise frames our question as what joint training creates beyond isolated endpoints, not whether one merging rule wins (p.1, lines 45–48).
- Both capture numbers now appear together (p.1, lines 16–19): the 82.2% median is explicitly qualified as strongly setting-dependent, alongside the nested result of 36.1% at two tasks falling to 7.1% at twelve.
- The body distinguishes the two populations: Sec. 2.1 (p.3, lines 154–156) defines 82.2% as a pooled median over heterogeneous suites, with a 21%–91% spread; Sec. 3.1 (p.5, lines 233–236) attributes 36.1%→7.1% to one fixed nested twelve-task sequence, not the 81-cell cross-sectional grid. Abstract and body now carry the same qualification.
Scaling law: claim narrowed to an empirical saturating fit.
-
Abstract (p.1, lines 20–25): now states a compact one-parameter saturating law and the statistical rational–logistic tie; the earlier "scales with task pairs" and "quadratic growth" wording is removed.
-
Sec. 3.1 (p.5, lines 252–257): direct measurement shows captured energy is statistically flat and missing energy grows sub-quadratically. Eq. (3) is now presented as an empirical law rather than a mechanism identification.
-
Sec. 3.2 / Table 3 (p.6, lines 309–313): six forms are fit on K=2,…,9 and tested without refitting on K=10,11,12. We retain the rational form for parsimony and report the tie transparently.
W4/W5: Calibration cost and the smaller-window evidence
We would like to clarify the actual training requirements and point out the results for a smaller calibration window.
First, the estimate counts the shared training process repeatedly. This calculation assumes that obtaining each calibration point requires starting a separate joint-training run. This is not how our nested experiment is implemented: within each sequence, the measurements at come from one training trajectory and its saved prefix checkpoints. In other words, we save multiple measurement points during one run, rather than retraining independently for each point.
Given the independently trained endpoints, calibration therefore requires one joint-training trajectory through , plus eight checkpoint projections per sequence, not 44 task-training units. These projections compute inner products and solve small least-squares problems using the existing endpoint updates and checkpoint parameters; they require no additional gradient-based training. Their cost is parameter-space computation rather than repeated model training. The calibration cost is therefore lower than the independent-run sum suggests. This is now explained explicitly in the “Calibration cost” paragraph of Sec. 3.2.
Second, accurate forecasting does not require calibrating through . In Table 6 (Appendix C, p. 16), with held out throughout, calibration on gives a median per-sequence held-out RMSE of 0.0061, comparable to the 0.0077 obtained with the full window. Calibration can therefore stop at : one joint trajectory and five checkpoint projections.
In addition, we have softened this framing in the revision: the forecast is now presented strictly as an objective-agnostic structural diagnostic, not a cost-saving deployment estimator, and the deployment-oriented claims have been removed (Sec. 3.2, p. 6).
We hope these clarifications and added results are helpful as you finalize your assessment. Thanks again for all your valuable feedbacks of our paper!
Response to Reviewer dCrh
Thank you for the detailed comments and for stating so concretely what would strengthen it.
W1 (Readability). To make the paper easier to follow, we revise the presentation in five ways: (i) an opening guide separating the geometric question from the functional-loss question; (ii) a short note on how to read Table 1 and why capture varies across suites, rather than a single headline number; (iii) a plain-language explanation of the growth law; (iv) a one-sentence distinction between the two parts of the joint update (coefficient drift vs. local escape); and (v) a brief scope statement and an at-a-glance table of the controlled comparisons in Section 4. All edits are marked in blue in the revised PDF.
W2 (Novelty). That merging can lag behind joint training is already known, and the bare existence of a gap is not our contribution; what was missing was a precise account of it, which we provide in two ways.
W3 (Clarification). We've replaced “captures most” with explicit captured-energy ranges for each suite (Edits are marked in blue in revised PDF page 4 and page 9). Captured energy measures how much joint-update energy the endpoints reconstruct, while SpanGap measures the loss difference from joint training; loss also depends on which changes are missing, so the two quantities need not track each other monotonically.
Our directional reinjection experiments show that recovery depends on the correction’s direction: adding back the identified missing direction lowers loss in 78/78 audited cases (median reduction: 0.0815) and outperforms reversed and norm-matched random corrections in 81/81 cases (Sec. 4.1, Table 11). These results support our central claim: merging misses a specific, functionally useful direction, and energy alone does not determine its effect on prediction.
W4 (Why predicting missing energy matters). It is an upstream structural quantity rather than a performance predictor, and its value lies in two points.
W5 (reliance on joint training).