acceptodds
Desk rejected at ICLR 2027

What Merging Misses: The Defining Direction of Joint Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging rests on a working assumption: that reweighting a set of independently trained models can stand in for one model trained jointly on all their tasks. We question this assumption and show that it fails in a precise, structural way. Across 81 LLM settings, even the best reweighting of independent endpoints loses to joint training in 80 cases. This failure is not because the endpoints are aimed in the wrong directions. Their span already captures about 82% of the joint update, but it misses a single direction outside that span, and no reweighting of the endpoints can recover it, no matter how it is tuned. This missing direction follows a simple law: its energy, measured by the missing squared norm, scales with the number of task pairs rather than the number of tasks. As a result, the gap grows quadratically as tasks accumulate, and a law fit on a few task counts can predict the gap at counts held out from the fit. The missing direction also has a definite origin. Almost all of it comes from a path-dependent term that joint training builds as the trajectory is pulled from one task toward the next. When this term is read from the training path and added back, the lost performance returns; when its reverse, or a random direction of the same length, is added instead, it does not. What sets joint training apart, then, is not where independently trained runs end up, but what their weights never record: a contribution built from the interaction between tasks as they are learned together. Recombining separately trained results, however carefully they are weighted, cannot reconstruct it.

Primary area:
optimization
Keywords:
model merging, joint training, endpoint sufficiency, task interaction, path-dependent residual

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dCrhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether merging independently finetuned models can reproduce the results of joint multi-task learning (MTL). The authors argue that there is a missing direction that models trained independently cannot recover compared with MTL, and link it with the performance gap between merging approaches and multi-task learning. The authors propose a one-parameter fit curve that predicts how this gap grows with the number of tasks (i.e., they fit it on small task groups, and use it to predict larger ones). Extracting this direction from the joint training trajectory and adding it to the merged model reduces the loss.

Critical strengths
  • The paper addresses a fundamental question, whether independently trained model endpoints can reproduce joint training. It distinguishes geometric reconstruction error from functional loss, showing that directly optimized merging coefficients still leave a loss gap.
  • The trajectory analysis provides insight into the missing component by separating local escape from coefficient drift.
  • The paper proposes a simple, single-parameter scaling law for the missing energy fraction as task count increases.
Critical weaknesses

W1. The paper is extremely hard to read and engage with. The writing quality is not suitable for a top-tier conference.

W2. The paper is motivated by the claim that "model merging assumes that reweighting independently trained endpoints can stand in for joint training". I believe this is too strong framing. Many merging papers treat MTL as an upper bound. It is also unsurprising that joint training produces components absent from independently trained endpoints.

W3. There are some central claims in the paper that are not supported by the results provided. For example, authors repeatedly claim that the endpoint span captures most of the energy of the joint update. However, Table 1 contradicts this on Suite A where the captured energy is between 21-44%, showing that most of the joint update lies outside the span. This same table also shows that the captured energy does not predict the functional gap (SpanGap column). Concretely, some configurations show smaller captured energy while having a smaller error.

W4. The results across the paper and its own argument indicate that even if we can predict the energy at larger task counts, predicting the missing energy at larger task counts does not establish how well the merged model will perform. Therefore, the practical implications and why it matters are unclear.

W5. One of the points of model merging is that there is no need to perform costly joint training. However, calibrating y requires jointly training on small compositions, and the reinjected direction is read from the joint training.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dCrh

Thank you for the detailed comments and for stating so concretely what would strengthen it.

W1 (Readability). To make the paper easier to follow, we revise the presentation in five ways: (i) an opening guide separating the geometric question from the functional-loss question; (ii) a short note on how to read Table 1 and why capture varies across suites, rather than a single headline number; (iii) a plain-language explanation of the growth law; (iv) a one-sentence distinction between the two parts of the joint update (coefficient drift vs. local escape); and (v) a brief scope statement and an at-a-glance table of the controlled comparisons in Section 4. All edits are marked in blue in the revised PDF.

W2 (Novelty). That merging can lag behind joint training is already known, and the bare existence of a gap is not our contribution; what was missing was a precise account of it, which we provide in two ways.

  • How large it is, and how it behaves at scale. We quantify the part of the joint update that linear merging cannot reach and show it is a ceiling of the approach, not a fixable tuning error—even the theoretically best possible coefficients cannot recover it. This unreachable part follows a simple pattern as tasks are added, so one can tell in advance, without running the large joint training, whether linear merging still has room at that scale.
  • Where it comes from. We trace it to a direction that is created during joint training itself. It never exists in any independently trained model, which is exactly why no mixing of those models can produce it.

W3 (Clarification). We've replaced “captures most” with explicit captured-energy ranges for each suite (Edits are marked in blue in revised PDF page 4 and page 9). Captured energy measures how much joint-update energy the endpoints reconstruct, while SpanGap measures the loss difference from joint training; loss also depends on which changes are missing, so the two quantities need not track each other monotonically.

Our directional reinjection experiments show that recovery depends on the correction’s direction: adding back the identified missing direction lowers loss in 78/78 audited cases (median reduction: 0.0815) and outperforms reversed and norm-matched random corrections in 81/81 cases (Sec. 4.1, Table 11). These results support our central claim: merging misses a specific, functionally useful direction, and energy alone does not determine its effect on prediction.

W4 (Why predicting missing energy matters). It is an upstream structural quantity rather than a performance predictor, and its value lies in two points.

  • Early, objective-agnostic feasibility assessment. We can predict the structural limits of linear merging at larger task scales before training those combinations or running labeled evaluation. This helps guide planning without first choosing a downstream performance objective.
  • Guidance for follow-up work: Even the strongest endpoint-derived space we tested leaves a substantial reconstruction residual (main-text Table 3, normalized residual: 0.609). This suggests a direction for future work: exploring how to recover the missing coordination learned during joint training, beyond coefficient tuning alone.

W5 (reliance on joint training).

  • Reinjection uses joint training as a diagnostic reference to establish the missing direction’s functional value, rather than as a deployment procedure. Joint training provides the benchmark for studying what low-cost endpoint merging misses. Reinjection is a controlled test of whether restoring the missing information closes the functional gap, rather than a step required to perform endpoint merging.
  • Joint training calibrates a reusable scaling law on smaller compositions, allowing us to forecast missing energy at larger task counts without training a joint model at each of those counts. In our experiment, measurements at K=2–9 calibrate the law, which then predicts the missing energy at K=10–12 (Sec. 3.2, Fig. 3). The larger joint-training runs are used to validate the forecasts, not to generate them, so the calibration cost can support multiple larger-scale predictions within the same setting.

Dear Reviewer dCrh,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which we tried our best to address each concern you raised in your feedback.

If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.

Thank you once again for your valuable feedback.

Thanks for the response. I went through the revised manuscript again.

W1: The reply lists five presentation changes, but the only blue edits I could find are the two short passages on p.4 and p.9. I couldn’t locate the plain-language explanation of the growth law, the drift vs. escape sentence, or the at-a-glance table for Sec. 4. If I missed them, please point me to them.

W2/W3: You agree that the gap to joint training is already known and that capture varies a lot by suite. Yet the abstract still opens with “model merging rests on a working assumption ...” and still says the span “captures about 82% of the joint update”. Meanwhile, Sec. 3.1 reports capture dropping from 36.1% at two tasks to 7.1% at twelve. The abstract should match the body.

Scaling law: Your new analysis in the reply to ZFyw finds three things. Captured energy does not grow with K, missing energy grows sub-quadratically and saturates, and a logistic curve predicts held-out K about as well as Eq. 3. That doesn’t fit the per-task / per-pair story behind Eq. 2. Still, the abstract says the energy “scales with the number of task pairs” and the gap “grows quadratically”. The data support a one-parameter saturating fit that extrapolates well, and the claims should be stated at that level.

W4/W5: I accept that missing energy is a structural quantity rather than a performance predictor, but then the “early feasibility assessment” argument is weak. The forecast also isn’t cheap. Table 15 calibrates the coefficient separately for each sequence using joint runs at K = 2 to 9. That is roughly 44 task-trainings to forecast K = 10 to 12, more than training the 12-task joint model once.

The reinjection results with reversed and norm-matched controls, and the optimizer 2x2 control, are interesting. But the headline claims are ahead of the evidence and the presentation problems remain, so I’ll keep my score.

AuthorsOpenReview ↗

Further Response to Reviewer dCrh

Many thanks for your time and effort to help strengthen our paper, it is our fortune to have a reviewer like you!

W1: Locations of the presentation changes

  • A roadmap of the paper: Page 2, first paragraph of Sec. 2. This explains how Sec. 2 defines the outside-span residual, Sec. 3 studies its task-count dependence and origin, and Sec. 4 tests its functional effect through reinjection.

  • A plain-language explanation of the predictive fit: Immediately after Eq. (3). This explains that a coefficient fitted at small task counts determines the curve used to predict larger counts without refitting.

  • The drift-versus-escape sentence: Page 7, first paragraph of Sec. 3.3: “Drift is the part that better-chosen mixing weights could have tracked, while escape is the part that no choice of weights can reach at all.”

  • The at-a-glance table for Sec. 4: Table 7 in Appendix E. It groups the recovery/history, direction, and dose comparisons, identifies their matched controls, and reports the corresponding sample counts.

W2/W3: Aligning the abstract with the body

We now do not present the gap itself as a new finding, and the capture statistic is clearly qualified throughout.

W2/W3: Abstract–body consistency.

  • Abstract opening rewritten (p.1, lines 11–14): the "working assumption" framing is replaced by a testable statement of what linear merging can and cannot span. The Introduction likewise frames our question as what joint training creates beyond isolated endpoints, not whether one merging rule wins (p.1, lines 45–48).
  • Both capture numbers now appear together (p.1, lines 16–19): the 82.2% median is explicitly qualified as strongly setting-dependent, alongside the nested result of 36.1% at two tasks falling to 7.1% at twelve.
  • The body distinguishes the two populations: Sec. 2.1 (p.3, lines 154–156) defines 82.2% as a pooled median over heterogeneous suites, with a 21%–91% spread; Sec. 3.1 (p.5, lines 233–236) attributes 36.1%→7.1% to one fixed nested twelve-task sequence, not the 81-cell cross-sectional grid. Abstract and body now carry the same qualification.

Scaling law: claim narrowed to an empirical saturating fit.

  • Abstract (p.1, lines 20–25): now states a compact one-parameter saturating law and the statistical rational–logistic tie; the earlier "scales with task pairs" and "quadratic growth" wording is removed.

  • Sec. 3.1 (p.5, lines 252–257): direct measurement shows captured energy is statistically flat and missing energy grows sub-quadratically. Eq. (3) is now presented as an empirical law rather than a mechanism identification.

  • Sec. 3.2 / Table 3 (p.6, lines 309–313): six forms are fit on K=2,…,9 and tested without refitting on K=10,11,12. We retain the rational form for parsimony and report the tie transparently.

W4/W5: Calibration cost and the smaller-window evidence

We would like to clarify the actual training requirements and point out the results for a smaller calibration window.

First, the estimate 2+3+...+9=442+3+...+9=44 counts the shared training process repeatedly. This calculation assumes that obtaining each calibration point requires starting a separate joint-training run. This is not how our nested experiment is implemented: within each sequence, the measurements at K=2,...,9K=2,...,9 come from one training trajectory and its saved prefix checkpoints. In other words, we save multiple measurement points during one run, rather than retraining independently for each point.

Given the independently trained endpoints, calibration therefore requires one joint-training trajectory through K=9K=9, plus eight checkpoint projections per sequence, not 44 task-training units. These projections compute inner products and solve small least-squares problems using the existing endpoint updates and checkpoint parameters; they require no additional gradient-based training. Their cost is parameter-space computation rather than repeated model training. The calibration cost is therefore lower than the independent-run sum suggests. This is now explained explicitly in the “Calibration cost” paragraph of Sec. 3.2.

Second, accurate forecasting does not require calibrating through K=9K=9. In Table 6 (Appendix C, p. 16), with K=10,11,12K=10,11,12 held out throughout, calibration on K=2,...,6K=2,...,6 gives a median per-sequence held-out RMSE of 0.0061, comparable to the 0.0077 obtained with the full K=2,...,9K=2,...,9 window. Calibration can therefore stop at K=6K=6: one joint trajectory and five checkpoint projections.

In addition, we have softened this framing in the revision: the forecast is now presented strictly as an objective-agnostic structural diagnostic, not a cost-saving deployment estimator, and the deployment-oriented claims have been removed (Sec. 3.2, p. 6).

We hope these clarifications and added results are helpful as you finalize your assessment. Thanks again for all your valuable feedbacks of our paper!

Reviewer_HxQCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work titled “WHAT MERGING MISSES: THE DEFINING DIRECTION OF JOINT TRAINING” sheds light on a fundamental assumption made in the model merging literature, which is that tuning of scaling coefficients corresponding to task-vectors can help obtain a merged model that can match a jointly trained model. The authors of this paper show that geometrically analyzing across 81 cases shows that, in most cases, the endpoints do not sufficiently span the space corresponding to the jointly trained model and highlight it as a missing direction. They also propose a task-count scaling law that shows that adding a correction extracted from the joint-training trajectory improves performance.

Critical strengths

[1] Reweighting of task-vectors is a common practice across merging methods, this work highlights its limitations.

[2] The study evaluates across a wide range of LLMs across different model backbones.

Critical weaknesses

[1] The study is limited to LoRA fine-tuned models across ranks 8,32,64 which, by design, cover a much smaller rank and hence cover a limited span. It is unclear if these findings translate to fully fine-tuned models.

[2] The evaluation is limited to simple Question Answering Letter prediction tasks and does not cover complex reasoning/instruction following tasks.

Questions for authors

[1] Is Geometric mismatch corresponding to functional mis-match? Works such as TIES [1] and DARE[2] have shown sparse task-vectors to be equally effective as the original task-vectors in terms of performance.


[2] Can the evaluation suite be expanded to include instruction following and reasoning tasks. Prior merging [2] methods have shown to outperform often even the individual experts on such tasks. 



References:


[1] Yadav, Prateek, et al. "Ties-merging: Resolving interference when merging models." Advances in neural information processing systems 36 (2023): 7093-7115.


[2]Yu, Linyan, et al. "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch." Icml. Vol. 2. No. 8. 2024.


[3] Xu, Jing, Jiazheng Li, and Jingzhao Zhang. "Scalable model merging with progressive layer-wise distillation." arXiv preprint arXiv:2502.12706 (2025).

4 comments
AuthorsOpenReview ↗

Response to Reviewer HxQC

We are truly encouraged by your positive assessment of our work and we appreciate the opportunity to further strengthen the empirical support.

Response to W1 & W2 & Q2

We carry out additional experiments under both LoRA and full fine-tuning settings, extending the evaluation beyond simple QA / letter-prediction tasks to Dolly Brain and Dolly Summarization based on databricks-dolly-15k for instruction following and summarization [1], GSM8K for mathematical reasoning [2], and SQuAD for reading comprehension [3]. For LoRA, each reported range corresponds to results over rank = 8, 32, and 64. The results are as follows:

TaskTraining SetupSpanGap ChangeAdding Back Direction R, alpha=1Adding Back Opposite Direction −RLoss-Optimized Direction RRandom Direction
Dolly BrainLoRA+0.0122~+0.0270+0.0237~+0.0391−0.1194~−0.0511+0.0097~+0.0173−0.0003~+0.0004
Dolly BrainFull Fine-Tuning+0.0074+0.0026−0.0485+0.0103−0.0181
Dolly SummarizationLoRA+0.0258~+0.0472+0.0416~+0.0713−0.1259~−0.0743+0.0239~+0.0353−0.0004~+0.0001
Dolly SummarizationFull Fine-Tuning+0.0337+0.0970−0.1230+0.0289−0.0130
GSM8KLoRA+0.0012~+0.0040+0.0908~+0.1305−0.2969~−0.1671−0.0044~−0.0004+0.0009~+0.0016
GSM8KFull Fine-Tuning+0.0071+0.1338−0.6229+0.0102−0.0110
SQuADLoRA+0.0296~+0.0362−0.0059~+0.0037−0.0147~−0.0069−0.0107~+0.0042−0.0015~+0.0002
SQuADFull Fine-Tuning+0.0298+0.0086−0.0335+0.0110−0.0050

We observe that under full fine-tuning, SpanGap remains positive, indicating that this phenomenon does not depend on the low-rank LoRA setting. Furthermore, similar directional patterns also appear in instruction following, summarization, mathematical reasoning, and reading comprehension tasks, suggesting that the paper’s conclusion is not limited to simple QA or letter-prediction evaluations.

These additional results also help clarify how our claim should be interpreted in relation to prior merging methods. The fact that prior merging methods may outperform a single expert on these tasks does not contradict our conclusion. Those comparisons focus on whether the merged model is stronger than an individual expert. Our analysis instead asks whether a strong merged model, even after achieving good task performance, still lacks recoverable beneficial directions compared with joint / full training. The further gains from adding back R show that this missing information remains.

Response to Q1

To directly test whether the identified missing direction is functionally relevant, we add the original missing direction R back to each starting model with alpha = 1 and measure the held-out NLL reduction, where higher is better. The experiments are carried out under five different LoRA configurations with rank = 4, 8, 16, 32, and 64, respectively.

Starting modelMedian NLL reduction ↑Configurations improved
Geometric projection, unpruned+0.05115/5
DARE merge, 90% dropped+0.01525/5
TIES merge, 90% dropped+0.00854/5

These results show that the geometrically identified missing direction carries useful predictive information and remains beneficial after DARE/TIES sparsification.

Reference:

[1] Databricks. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm

[2] Cobbe et al. Training Verifiers to Solve Math Word Problems. https://arxiv.org/abs/2110.14168

[3] Rajpurkar et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text. https://aclanthology.org/D16-1264/

Dear Reviewer HxQC,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which we tried our best to address each concern you raised in your feedback.

If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.

Thank you once again for your valuable feedback.

I’d like to thank the authors for the additional evidence, the results with full finetuning strengthen the evidence that the observed limitation is not specific to low-rank adaptation. 



Could the authors please clarify the metric and starting model for each column in the new table, particularly “SpanGap Change” and “Loss-Optimized Direction R”?

Also, for the expanded tasks, are the results based on held-out NLL alone, or also on generation metrics such as GSM8K exact match?

AuthorsOpenReview ↗

Further Response to Reviewer HxQC

We are delighted to have a chance to clarify the experimental setup, metrics, and starting models below.

Training setup. Starting from the same pretrained Qwen2.5-1.5B checkpoint θ0\theta_0, we train four single-task models on GSM8K, SQuAD answer generation, Dolly summarization, and Dolly brainstorming, plus one joint model on all four tasks. Isolated and joint runs use matched per-task training streams and a common 376-step schedule. We evaluate both LoRA and full finetuning, with embeddings and the output head frozen in both settings.

Metric and starting models. The functional metric is held-out completion NLL. Let DD contain the four single-task updates relative to θ0\theta_0. The intervention starting models are endpoint combinations added to θ0\theta_0, not the pretrained checkpoint or the joint model itself.

  • q: the fraction of the joint update’s squared norm outside the endpoint span; it is dimensionless.
  • SpanGap Change: Leval(θL)−Leval(θJ)L_{\mathrm{eval}}(\theta_L)-L_{\mathrm{eval}}(\theta_J), where θJ\theta_J is the joint model and θL=θ0+Da^\theta_L=\theta_0+D\hat a is the endpoint merge whose coefficients are optimized on calibration NLL, with DD fixed. Positive values favor joint training.
  • Geometric R / −R / random gains: the starting model is θG=θ0+Dβ\theta_G=\theta_0+D\beta, where β\beta minimizes weight-space distance to the joint model. We add R, −R, or a norm-matched Gaussian random direction at α=1\alpha=1. Gain is NLL(starting model) − NLL(modified model); random gains average three directions.
  • Loss-Optimized Direction R: the same R is added to θL\theta_L, with gain measured relative to θL\theta_L. R is the accumulated joint-path residual defined in Eq. 4; only the starting merge’s coefficients are loss-optimized, not R.
  • Oracle bestcal / stopping step: the best calibration NLL found during coefficient optimization and the number of coefficient-optimization steps executed, respectively—not additional model-training steps.

For the DARE/TIES rows, the starting models are actual sparse merges with a common scale selected on calibration NLL. The original R is then added at α=1\alpha=1, without further training. Median gains summarize five configurations; “configurations improved” counts positive gains. DARE gains first average three masks within each configuration.

Expanded-task evaluation. These results use held-out, teacher-forced completion NLL only, excluding prompt tokens. NLL is token-normalized within each task and averaged equally across the four tasks for aggregate results. Training, calibration, and evaluation partitions are disjoint; merging coefficients are selected on calibration data, not evaluation data. GSM8K exact match and other generation metrics are not included: the reported gains measure reference-completion likelihood, not generated-answer accuracy.

If you have any further questions or concerns, we would be happy and eager to address them promptly within the limited time remaining for discussion. We hope these clarifications and added details are helpful as you finalize your assessment.

Reviewer_ZFywOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines if linear combinations of independently trained task models can match the parametrization adn performance of a jointly trained model. This question is formulated geometrically by projecting the joint training update onto the span of independently trained task updates. It is shown that the component orthogonal to this span represents that what cannot be reconstructed through just endpoint reweighting.

Experimentally it is shown that this residual is nonzero and that even coefficients optimized directly against the joint objective generally fail in matching the joint training performance. A scaling law relating the missing energy fraction to the number of tasks is proposed. Finally, they demonstrate that reinjecting the accumulated trajectory residual improves performance.

Critical strengths

1. The geometric characterization of endpoint merging. The paper outlines a clear framework for distinguishing limitations of the endpoint representation from limitations of coefficient selection. The experiments demonstrate that the observed limitations are not simply attributable to poorly chosen merging weights.

2. Extensive empirical validation and diagnostic analysis, going beyond a straightforward comparison of joint training and model merging. Multiple model architectures, task configurations, and adaptation ranks, and include controlled additive targets, structured nulls, trajectory decompositionsn directional interventions, and added optimizer and task-count analyses are covered in the experiments.

Critical weaknesses

1. The proposed explanation in terms of task interactions is not sufficiently established.

The paper interprets the missing direction as a contribution created through task coordination along the joint optimization trajectory. However, the exact decomposition in Equation 4 is an algebraic identity, and the dominance of the local-escape term does not, by itself, establish that this residual is caused by task interactions rather than other properties of the optimization dynamics.

In particular, and suggested by Appendix tables 23-24, nonlinear optimizer transformations can generate directions outside the isolated-update span even at the first optimization step. This does not necessarily require the accumulation of interactions along a training trajectory.

While this does not invalidate the geometric findings per se, they substantially complicate their interpretation. Effects attributable to the optimizer are not convincingly distinguished from those attributable to the proposed pairwise coordination mechanism.

2. The pairwise scaling law is not sufficiently justified as a mechanism rather than an empirical approximation.

The proposed scaling law relies on the assumptions that captured energy grows approximately linearly with the number of tasks and missing energy grows proportionally to the number of task pairs. However, these assumptions are not independently established. The experiments mainly validate the resulting ratio, which does not uniquely identify the scaling behavior of its numerator and denominator.

The held-out task-count predictions are encouraging, particularly given the replicated sequences. Still, agreement with the proposed functional form does not necessarily mean that the underlying missing energy originates from pairwise interactions. Other accumulation mechanisms could potentially produce similar behavior.

Moreover, the predictive comparisons primarily involve a last-value baseline and a linear extrapolation of the missing fraction. These are relatively limited alternatives for a bounded quantity exhibiting saturation. Stronger comparisons with alternative nonlinear scaling laws, together with direct measurements of captured and missing energies, would make the proposed explanation considerably more convincing.

I believe that the experiments do support the existence and functional relevance of an endpoint-external component. However,, the evidence for the specific mechanism responsible for its emergence and scaling is less conclusive than the paper currently sells it to be.

Questions for authors
  1. Can you distinguish the proposed task-coordination mechanism from optimizer-induced nonlinearity? Can you relate this to the substantial differences between SGD and adaptive optimizers in Appendix Table 23? It would be interesting to see whether the functional merging gap persists under optimizers producing smaller outside-span residuals, also when optimization progress and final performance are comparable?

  2. Can more direct evidence for the assumed pairwise accumulation mechanism be presented? As the projection mechanism provides the captured and missing energies, how do these quantities scale individually with task count? Can, in the multi-task experiments, be distinguished between pairwise contributions from higher-order interactions or alternative growth mechanisms? I would like to see how the proposed law compares against other saturating functional forms?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZFyw

We feel truly fortunate that our paper received such an exceptionally careful and insightful reading from a reviewer who engaged so deeply with our work!

Q1

  1. To test whether the gap is merely an optimizer-induced nonlinearity or instead requires a task-coordination mechanism, we stop each optimizer at the same per-task calibration loss, then compare single-task and matched joint-training runs across optimizers that differ in whether they can generate out-of-span directions at the first step. | Optimizer | Matched cells | First-step out-span | Final q | SpanGap | +R | −R | Random | |---|---:|---:|---:|---:|---:|---:|---:| | AdamW | 33 | 0.2461 | 0.3537 | 0.0408 | +0.0255 | −0.1042 | −0.0002 | | SGD | 6 | ≤7.86×10⁻⁵ (med ~10⁻¹⁵) | 0.9991 | 0.4869 | +0.4570 | −1.0331 | −0.0080 | | Global norm | 9 | ≤7.05×10⁻⁵ (med ~10⁻¹⁵) | 0.4774 | 0.0636 | +0.0378 | −0.1848 | −0.0014 |

q = out-of-span squared-norm fraction; +R/−R/Random = NLL gains (positive = better) from reinjecting the out-of-span residual, its reversal, or a norm-matched random direction.

To make the identification logic explicit, we reorganize the result as a 2×2 comparison that switches two factors on and off: O denotes instantaneous optimizer nonlinearity, probed by the first-step out-of-span energy (zero by construction for optimizers whose first update is a linear combination of per-task gradients), and T denotes a shared multi-task trajectory.

T=0 single taskT=1 joint trajectory (matched)
O=1 AdamWK=1 control, 12 cells, q ≡ 033 cells, phenomenon present
O=0 SGD/global normK=1 control, 12 cells, q ≡ 015/15: SpanGap>0, +R beats −R and 3 randoms
Optimizer nonlinearity is therefore insufficient (nonlinear AdamW produces zero residual on a single task) and unnecessary (the full gap survives under optimizers whose first step is linear and cannot leave the span); the shared multi-task trajectory is the only factor present in every cell where the residual appears.
  1. The numerical differences from Tables 23–24 are expected because the endpoints are defined differently. Tables 23–24 stop all runs after a fixed number of optimization steps, whereas the new control stops each optimizer as soon as it reaches the same per-task calibration loss. Thus, the stopping rule changes endpoint magnitudes but not the qualitative conclusion: first-step-linear optimizers still show a joint-training residual, while single-task controls remain residual-free.

3)Single-task runs show no residual, even with nonlinear AdamW. Matched joint runs still show the residual under first-step-linear optimizers. Thus, in our tested panel, the residual requires a shared multi-task trajectory, not optimizer nonlinearity alone.

Q2 We directly measure two quantities at each task count K: how much of the joint update is recoverable from isolated endpoints, and how much is missing from that endpoint span. We then test how each quantity grows with K and whether our scaling law predicts held-out task counts better than alternative curves.

Quantity measured at each KGrowth with K95% intervalTakeaway
Energy recovered by isolated endpointsα = 0.036[−0.138, 0.082]No detectable growth
Energy missing from the endpoint spanβ = 1.240[1.215, 1.270]Clear growth with K
Free exponent for missing energyp = 0.797—Sub-quadratic, saturating growth
The endpoint-recoverable part stays flat, while the missing part is the one that grows with task count. Its fitted exponent is below quadratic, meaning that the growth is slower than K² and therefore not an undamped pairwise-count effect. The pattern is better described as saturating growth.
Model fitted on K=2…9 and tested on K=10…12Median RMSETakeaway
M1 one-parameter rational form, ours0.0112Best overall
M4 logistic saturation0.0136Statistically tied with M1
M2 free power law0.0257Worse
M6 last-value baseline0.0258Worse
M3 non-saturating exponential0.0732Much worse
M5 linear extrapolation0.5070Much worse
Our one-parameter form outperforms all simple alternatives on held-out task counts and is statistically tied only with logistic saturation, which is itself a saturating curve. Thus, the evidence supports saturating accumulation of endpoint-external energy, but does not uniquely identify pairwise interactions over other damped higher-order mechanisms. We hope these clarifications and added results are helpful as you finalize your assessment. Thank you again for the exceptionally careful reading of our paper.

Dear Reviewer ZFyw,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which we tried our best to address each concern you raised in your feedback.

If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.

Thank you once again for your valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Area Chairs

We thank all reviewers for the careful discussion. We are encouraged that the reviews recognized the importance of the question we study, the value of the geometric formulation, and the usefulness of the empirical diagnostics. The rebuttal process was also very helpful in identifying where the original submission needed stronger evidence, clearer scope, and more conservative wording. We have revised the paper accordingly and summarize the main changes here for the reviewers and area chairs.

  • Expanded empirical scope. In response to concerns that the phenomenon might be specific to low-rank LoRA or simple QA-style evaluations, we added experiments with full fine-tuning and with a broader task set: Dolly brainstorming, Dolly summarization, GSM8K, and SQuAD. The SpanGap remains positive under full fine-tuning, and the same directional pattern appears beyond the original letter-prediction setting. These results strengthen the evidence that the missing direction is not an artifact of the LoRA rank constraint or of the original evaluation suite.

  • Functional relevance of the missing direction. Several comments asked whether the geometric residual matters for prediction, especially in relation to strong sparse merging methods such as DARE and TIES. We therefore added reinjection tests from different starting merges. Adding the identified residual direction improves held-out completion NLL across the audited settings, including after DARE/TIES sparsification, while the reversed direction and norm-matched random directions do not show the same behavior. This supports the central claim that the endpoint-external component is not only a geometric mismatch, but also carries useful predictive information.

  • Optimizer and mechanism controls. Reviewer ZFyw raised an important concern that out-of-span directions might be induced by optimizer nonlinearity rather than by multi-task coordination. We added matched controls across AdamW, SGD, and global-norm updates. Single-task controls remain residual-free, including with nonlinear AdamW, while matched joint-training runs still show a residual under optimizers whose first update is effectively linear and cannot leave the isolated-update span at the first step. We have revised the discussion to make the identification logic explicit: optimizer nonlinearity alone is not sufficient in our tested panel, and the shared multi-task trajectory is the common factor in the settings where the residual appears.

  • Scaling-law claim narrowed. The discussion also made clear that our original wording around pairwise scaling was too strong. We now present Eq. 3 as a compact empirical saturating law rather than as unique evidence for a pairwise mechanism. We added direct measurements showing that the endpoint-recoverable energy is statistically flat over task count, while the missing energy grows sub-quadratically and saturates. We also compare against alternative functional forms. The rational form remains the most parsimonious and performs best overall on held-out task counts, but logistic saturation is statistically tied; the revised paper states this explicitly.

  • Presentation and claim alignment. We also revised the framing throughout the paper. The abstract no longer presents the joint-training gap itself as the novelty, and it no longer claims quadratic growth or unqualified “task-pair” scaling. The capture statistics are now reported with their setting dependence: the pooled median and the nested task-count sequence are described separately. We added a roadmap in Sec. 2, a plain-language explanation of the predictive fit after Eq. 3, a clearer drift-versus-escape sentence in Sec. 3.3, and an at-a-glance table of the controlled comparisons in Appendix E.

Overall, the revised paper makes a narrower but stronger claim: the limitation is in the endpoint span itself, not merely in coefficient selection. Even when merging coefficients are optimized directly on calibration loss, a gap remains; the required update direction is outside the span of independently trained endpoints. The added controls further show that this outside-span direction is created during joint training and is functionally useful when reinjected.

We believe the added experiments, controls, and revised framing address the main concerns raised during discussion while preserving the paper’s core contribution. We hope these clarifications and added details are helpful as you finalize your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.