acceptodds
Withdrawn from ICLR 2027

COMET: Continual Model Merging via Adaptation to Mismatch in Distributional Tails

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual model merging requires preserving past solution expertise (stability) while retaining the ability to adapt (plasticity), without access to past data. Recent approaches cast this problem as recursive posterior inference, representing accumulated expertise as a model prior and incorporating new knowledge through the target task likelihood. The resulting posterior often assumes a Gaussian structure, with task-specific curvature statistics accumulated in its precision matrix. Our analysis reveals a previously overlooked geometric interpretation of this formulation. Incorporating knowledge from a new model distribution can be viewed as updating the mode of the existing model distribution along a set of orthogonal latent directions in the inner-product space of its precision matrix. Each direction is associated with a unit scale of update that acts as a base speed controlling how fast new knowledge should be incorporated along that direction. This reveals a novel geometric characterization of stability and plasticity that is largely independent of the model architecture. Importantly, the appropriate unit scale along each direction should adapt to the relative tail behaviors of the existing and incoming model distributions as their disagreement determines how strongly newly acquired knowledge should alter the existing solution. Gaussian structures, however, ignore such tail behavior and consequently impose fixed unit scales of update that are empirically suboptimal. Replacing the Gaussian structure with an energy based distributional structure whose tail decay rate is adaptive restores this missing dependence. Extensive experiments with billion-parameter foundation models show that our approach consistently outperforms state-of-the-art methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model merging, continual adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aaFnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

COMET studies data-free continual merging of task-specific checkpoints. It replaces quadratic merging penalties with logarithmic penalties, making the influence of historical and incoming models depend on their residuals. It builds Kronecker curvature surrogates from task displacements and uses an iteratively reweighted solver. Experiments cover CLIP, LoRA-adapted Flan-T5, and a three-checkpoint 7B merging example.

Critical strengths
  1. Meaningful setting. Merging a stream of checkpoints without their training data is a useful problem. Restricting retained historical state gives the continual setting a concrete constraint.

  2. Competitive vision results. Table 2 covers three CLIP architectures and 8-, 14-, and 20-task streams, averaged over ten task orderings. COMET has the highest mean final accuracy in six of nine settings and the second-highest in the other three.

  3. Broader evaluation. Flan-T5 experiments and runtime measurements extend the evidence beyond vision. The runtime results show that computational benefits depend on model scale.

Critical weaknesses
  1. The advance over prior probabilistic merging is unclear. Bui et al. (2026), cited by the paper, already develops heavy-tailed product-of-experts merging with residual-dependent fixed-point updates. COMET needs to show what its sequential formulation adds. In particular, what fails when the earlier method is applied recursively, and how do the accumulated factors and scale schedule fix that failure? The current comparisons do not isolate this contribution.

  2. The curvature construction does not ensure a valid solver. Equation (15) uses Gram factors of task displacements. For a rectangular displacement, at least one factor is singular; low-rank displacements can make both singular. Theorem 1 and Algorithm 2 nevertheless require invertible Cholesky factors. The paper does not specify damping, a restricted support, or another way to make these operations valid.

  3. Initialization and accumulation are inconsistent. Algorithm 1 initializes the historical factor to zero, computes an apparently unused updated factor at line 6, and passes the historical factor to IRFP at line 10. Equation (16)'s recurrence, Go+Gnc\frac{G_o + G_n}{c}, is not a running mean. These details determine the merge geometry and need to match the implementation used in the experiments.

  4. The main claim lacks a matched ablation. COMET changes both the penalty and the curvature/accumulation scheme. A Gaussian or constant-responsibility variant using the same factors, retained state, regularization, and solver is needed to show that tail adaptation causes the gains. The current ablations do not separate these effects.

  5. The probability model needs additional conditions. In dimension dd, with positive-definite precision, a density proportional to (1+sν)−ν\left(1 + \frac{s}{\nu}\right)^{-\nu} is normalizable only when ν>d2\nu > \frac{d}{2}. The paper assumes only ν>0\nu > 0. Singular precision also prevents normalization over the full space. The posterior interpretation therefore needs correction.

  6. The 7B experiment measures only one capability. GSM8K results after merging conversational, mathematical, and coding checkpoints do not show whether conversational and coding performance is retained.

Questions for authors

See Weaknesses

Reviewer_P5saOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies data-free continual model merging. It casts each merge step as a MAP problem over the product of an "existing" and an "incoming" model distribution. COMET replaces the Gaussian with a heavy-tailed, Student-t-like energy, builds a data-free K-FAC-style curvature proxy from each task vector, and accumulates it across tasks.

Critical strengths

A clean, efficient solver for Kronecker-structured precision merging (Theorem 1, App. A.3-A.4). After a one-time Cholesky and eigendecomposition of the M×MM\times M and N×NN\times N factors, each majorize-minimize step is an entry-wise update in the transformed space. The iteration monotonically decreases the objective (Corollary 1, a standard MM argument via concavity of hνh_\nu). This matters because it makes curvature-aware, non-Gaussian merging feasible at 7B scale without data access (Table 5: about 4 s/layer vs. 5.3 s for OPCM and 39 s for DOP).

Critical weaknesses
  1. The tail-adaptation mechanism reduces to a bounded per-layer rescaling of a Gaussian merge. Dividing Eq. 21 by ωo\omega_o gives λij=piqj/(piqj+η)\lambda_{ij}=p_iq_j/(p_iq_j+\eta) with η=ωo/ωn\eta=\omega_o/\omega_n a single scalar per layer, i.e., a Gaussian Kronecker merge with Λn\Lambda_n scaled by 1/η1/\eta. Under the ν\nu-schedule (Eq. 23), η=1+sn(W)/(κsn(Wo))1+so(W)/(κso(Wn))\eta=\frac{1+s_n(W)/(\kappa s_n(W_o))}{1+s_o(W)/(\kappa s_o(W_n))}. Every iterate lies between WoW_o and WnW_n in both metrics, so η∈[κ/(κ+1),(κ+1)/κ]=[0.5,2]\eta\in[\kappa/(\kappa+1),(\kappa+1)/\kappa]=[0.5,2] for κ=1\kappa=1. In a small numerical check, the IRFP iterate decreases monotonically from η=2\eta=2 and converges within 15 steps. The Student-t model, ν\nu-schedule and MM solver thus pick one factor in [0.5,2][0.5,2] per layer, far narrower than the range in Fig. 1. The decisive baseline, the same proxy (Eqs. 15-16) with η=1\eta=1 or a tuned scalar, is missing, and Table 3 (81.6 vs. 76.8 without accumulation) suggests the proxy, not the tails, drives the gains.

  2. The proxy and algorithm are weakly founded, and the empirical claims are overstated. (∑jAj)⊗(1c∑kGk)(\sum_jA_j)\otimes(\frac1c\sum_kG_k) contains cross terms Aj⊗GkA_j\otimes G_k (j≠kj\ne k) that match no task. Alg. 1 never assigns GoG_o, and the first merge leaves the existing precision singular (for LoRA, AoA_o has rank at most 8r8r), so an unreported damping term decides the solution outside the task span. "Consistently outperforms" is contradicted by Table 2 (DOP wins 3 of 9 settings); gains over DOP clearly exceed the spread in only two settings and are small on Flan-T5. The 7B result uses one order and only GSM8K. The stored factors take 2-4 times the memory of each layer (about 2.7 times the model for Llama-2-7B), which is not discussed.

Questions for authors
  1. Please report Gaussian merging with your exact curvature proxy and accumulation (Eqs. 15-16), i.e. h(s)=sh(s)=s, on all Table 2 settings. Please also report it with a single tuned scalar η\eta (global and per-layer). If COMET does not clearly beat the tuned-scalar Gaussian, what remains of the tail-adaptation claim?
  2. How are singular factors handled? Ao=Go=0A_o=G_o=0 at initialization (l.303), and at the first merge (Alg. 1 line 6) they equal An,GnA_n, G_n, which are rank-deficient whenever Δn\Delta_n is low-rank (e.g. LoRA). Yet Alg. 2 requires Cholesky factors of AoA_o and GoG_o, and Alg. 1 computes G~t\tilde G_t but never assigns it to GoG_o. How is the first merge handled (is the first model taken as is?), what damping is used, and how sensitive are results to it?
  3. The accumulated precision (∑jAj)⊗(1c∑kGk)(\sum_jA_j)\otimes(\frac1c\sum_kG_k) adds curvature in input-output pairs that no task constrains. Please compare with an accumulation that avoids these cross terms (e.g., keeping per-task factors or a low-rank sum of Kronecker terms) and justify the choice of a sum for AA and a mean for GG.
  4. Justify Eq. 14 (Δn\Delta_n as a sum of gx⊤g x^\top over ∣D∣|D| samples), which assumes single-pass plain SGD and does not hold for Adam-style preconditioned updates or weight decay. Please also justify the asymmetric accumulation in Eq. 16 (sum for AA, 1/c1/c-rescaling for GG), e.g. by ablating alternatives.
Reviewer_LxgkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of continuous model merging in the absence of historical data, with a focus on the limitations of existing Gaussian posterior methods in balancing stability and plasticity. The author points out that the standard Gaussian assumption fails to fully capture the differences in the tails of the distribution between the old and new models, thereby hindering the adaptive adjustment of update amplitudes in different directions. To address this issue, this paper proposes an energy based distribution modeling method that has an adaptive tail decay rate, allowing for more flexible control of the fusion of new and existing knowledge. Empirical evaluations of various scenarios have shown the clear advantages of this method. However, there is some overlap between the core innovation of this paper and existing literature, and further analysis and comparison are necessary.

Critical strengths
  1. This paper is written very well.
  2. This paper discusses the limitations of existing Gaussian posterior methods in balancing stability and plasticity. A solution has been proposed, providing detailed theoretical analysis and derivation.
Critical weaknesses

1.The core issue of this paper is the overlap between method innovation and existing methods, which requires further comparative analysis. The problem of Gaussian hypothesis failing to capture the actual heavy tailed behavior of data has been clearly addressed and addressed in existing literature[1]. In this paper, there was no clear comparative analysis between this paper and the literature, and it is necessary to elaborate on the differences in research questions and solutions between the two. It is best to provide comparative experimental results.

2.There are significant issues with the experimental design:

Firstly, this paper lacks key ablation experiments. This paper only focuses on the ablation of Kronecker factor accumulation. However, the core of this paper is the problem of Gaussian assumption failing to capture the actual heavy tail behavior of the data, and an adaptive tail decay rate is designed. If this design is not dissolved, it will make this paper seriously lacking in persuasiveness.

Secondly, this paper only selected four comparison methods, but did not compare this method consistently in all experiments. What is the reason for this phenomenon? Suggest completing all comparison methods.

In addition, this paper should provide more comprehensive experimental results. The result display diagrams of Figures 5 and 6 can be drawn in each scenario, but this paper does not provide a complete one.

Finally, the LLM (7B) experiment only reports the experimental results on the GSM8K dataset. Evaluation results from other datasets should be provided to further demonstrate the effectiveness of the merging method.

3.There are many details in this paper: The "Research Gap" section in the introduction contains a clear placeholder for "(cite)". This is a clear negligence, indicating a lack of confidence in proofreading the paper. It is recommended to re-examine the paper.

Many mathematical formulas in the paper are not centered correctly, and some lack equation numbers. Careful review is required. The TIES-merging method is not mentioned in the baseline description, but it suddenly appears in the comparison in Table 2. In addition, it was omitted in subsequent experiments, which is very confusing. Please provide an analysis.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.