- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
COMET studies data-free continual merging of task-specific checkpoints. It replaces quadratic merging penalties with logarithmic penalties, making the influence of historical and incoming models depend on their residuals. It builds Kronecker curvature surrogates from task displacements and uses an iteratively reweighted solver. Experiments cover CLIP, LoRA-adapted Flan-T5, and a three-checkpoint 7B merging example.
Critical strengths
-
Meaningful setting. Merging a stream of checkpoints without their training data is a useful problem. Restricting retained historical state gives the continual setting a concrete constraint.
-
Competitive vision results. Table 2 covers three CLIP architectures and 8-, 14-, and 20-task streams, averaged over ten task orderings. COMET has the highest mean final accuracy in six of nine settings and the second-highest in the other three.
-
Broader evaluation. Flan-T5 experiments and runtime measurements extend the evidence beyond vision. The runtime results show that computational benefits depend on model scale.
Critical weaknesses
-
The advance over prior probabilistic merging is unclear. Bui et al. (2026), cited by the paper, already develops heavy-tailed product-of-experts merging with residual-dependent fixed-point updates. COMET needs to show what its sequential formulation adds. In particular, what fails when the earlier method is applied recursively, and how do the accumulated factors and scale schedule fix that failure? The current comparisons do not isolate this contribution.
-
The curvature construction does not ensure a valid solver. Equation (15) uses Gram factors of task displacements. For a rectangular displacement, at least one factor is singular; low-rank displacements can make both singular. Theorem 1 and Algorithm 2 nevertheless require invertible Cholesky factors. The paper does not specify damping, a restricted support, or another way to make these operations valid.
-
Initialization and accumulation are inconsistent. Algorithm 1 initializes the historical factor to zero, computes an apparently unused updated factor at line 6, and passes the historical factor to IRFP at line 10. Equation (16)'s recurrence, , is not a running mean. These details determine the merge geometry and need to match the implementation used in the experiments.
-
The main claim lacks a matched ablation. COMET changes both the penalty and the curvature/accumulation scheme. A Gaussian or constant-responsibility variant using the same factors, retained state, regularization, and solver is needed to show that tail adaptation causes the gains. The current ablations do not separate these effects.
-
The probability model needs additional conditions. In dimension , with positive-definite precision, a density proportional to is normalizable only when . The paper assumes only . Singular precision also prevents normalization over the full space. The posterior interpretation therefore needs correction.
-
The 7B experiment measures only one capability. GSM8K results after merging conversational, mathematical, and coding checkpoints do not show whether conversational and coding performance is retained.
Questions for authors
See Weaknesses