acceptodds
Under review as a conference paper at ICLR 2027

Data-Free Continual Model Merging: Arrival Order, Not Missing Data, Is the Bottleneck

Abstract

Continual model merging folds a stream of task experts into one deployed model, one at a time, with neither joint retraining nor the data any expert was trained on, and without revisiting one already absorbed. The merge step is denied what one-shot merging takes for granted: the data and the whole pool. However, the gap is read as a data problem, and the remedy is not free: calibration needs labelled samples at every arrival, which raises the merge cost, and data saturates within a few samples per task. In addition, arrival order goes unpriced, though taking the stream in order costs a data-free rule up to 18.9 points. To address those issues, we hold pool, backbone and evaluator fixed, price each constraint on its own, and propose SCOPE (Sign-Conflict, Order-scaled, Parameter-class Estimation), a merge-time-data-free operator of three scalars. Moreover, the constraints interact rather than add: the order penalty is large for a data-free rule and nearly vanishes for a calibrated one. On eight CLIP ViT-B/32 experts over thirty arrival orders, SCOPE improves on the machine-designed operator of our earlier search by a small but consistent margin. Two results bound the claim: parity on longer streams needs the damping scaled by stream length, and automated discovery is blocked by the fitness signal. These results argue for measuring the order constraint on its own terms, since a data-free number need not transfer from a pool to a stream.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Model Merging, Weight Interpolation, Data‑Free Merging, Arrival Order Robustness, Automated Algorithm Design

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JxggOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies data-free, single-pass continual model merging without revisiting historical experts and proposes SCOPE, which carries only three scalar state variables. The method updates CLIP ViT-B/32 experts using parameter-category coefficients, a sign-conflict factor relative to the current merged model, stream-length-scaled step-size decay, norm stabilization, and final-step clipping. The paper evaluates pools of 8, 14, and 20 experts under multiple arrival orders and argues that arrival order, rather than the absence of data, is the main bottleneck in continual merging.

Critical strengths
  • The problem constraints are clearly defined. Section 2 explicitly distinguishes the no-revisiting/order constraint C1 from the no-data-during-merging constraint C2, and provides the state variables and complete operator code.
  • The experimental reporting is relatively transparent. Table 2 uses paired comparisons under identical arrival orders; Table 3 provides component-wise ablations; and Section 5.5 proactively reports the negative result that the original operator falls below the unmerged baseline on long streams.
  • Reproducibility, ethics, and AI use are explicitly addressed. The appendix provides arrival orders, per-order results, implementation details, and the sources of reported values.
Critical weaknesses
  1. The causal claim that “arrival order is the bottleneck” is not rigorously isolated. In Section 3.1, the comparison among one-shot TIES and two sequential TIES implementations changes both the objects available to the algorithm and the update rule across regimes; the data-enabled condition in Section 3.2 then switches to RegMean. The sequential-with-data results in Appendix A.10 use the default eight-expert evaluation rather than orders 100–109 from Table 1. The current results therefore demonstrate an interaction between method and regime, but are insufficient to attribute the approximately 0.70-to-0.83 gap primarily to arrival order.
  2. Additional conditions exclude stronger methods that satisfy the broader setting. DOP in Section 3.3 satisfies both sequential arrival and no data during merging, but falls outside the authors' additionally defined “closed-form” subclass because it performs optimization at each step; its accuracy is 0.7798, approximately 8 percentage points above SCOPE's 0.6999. MagMax in Appendix A.14 likewise satisfies the single-pass, data-free constraints and achieves 0.7110, above SCOPE, but is excluded because its result is order-invariant and does not depend on a running model. Order insensitivity is not, by itself, a natural basis for exclusion from performance comparisons. Thus, the “best in class” and broader continual-merging claims hold only within the authors' more narrowly defined subclass.
  3. The core gain is very small, and statistical repetition does not substitute for task generalization. Table 2 shows an average improvement of only 0.0015 over the closely related MergeOp-M1 baseline. All 30 permutations come from the same set of 8 task experts. The paired test establishes consistency across orders for that fixed pool, but does not provide independent evidence across task pools, backbones, or modalities.
  4. The long-stream results weaken the methodological contribution. In Table 5, unrepaired SCOPE degrades severely with 14/20 experts. After the repair, its 20-expert accuracy is 0.6093, still below sequential TIES (0.6119) and simple averaging (0.6110). Moreover, the 14- and 20-expert settings use κ=3.0 and κ=2.5, respectively, rather than one fixed operator that requires no stream-length-specific hyperparameter selection. The paper states that the exponent is selected on orders 100–104 and then reports the mean over 100–109; the final result therefore still includes the five orders used for selection and is not a fully independent blind evaluation.
  5. There is a risk of design overfitting to the same task assets. Constants for the main eight-task result are derived from a proxy objective and design orders on TA8, while the final evaluation changes only the permutations, not the task pool. The 14/20-expert expanded pools still come from the same FusionBench assets and use the same backbone. Appendix A.9 further shows that the proxy optimum does not coincide with the optimum under the official metric.
Questions for authors
  1. Can the authors conduct a complete C1xC2 factorial experiment using the same task pool, order, evaluator, and operator family, to genuinely separate the effects of order and data?
  2. Why should DOP and MagMax not be considered valid competitors under the stated problem setting? If they are included, how should the paper's main conclusions be revised?
  3. Can the authors pre-specify a single κ and evaluate it blindly across new task pools, backbones, and stream lengths, rather than using different values for 14/20 experts?
  4. Does the 0.15-percentage-point gain persist on new task sets, rather than only across permutations of the same eight-task pool?
Reviewer_mmXLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper “Data-Free Continual Model Merging: Arrival Order, Not Missing Data, Is the Bottleneck” studies continual, sequential merging of task vectors without assuming access to data or previously merged experts. It proposes the method ’ SCOPE ’, which scales task vectors based on sign conflict, order-scaled parameter-class estimation, while retaining 3 meta-parameters at each merging state. The work also studies two important constraints relevant to the continual learning setting of [1] Arrival Order and [2] Data-Free rule, confirming arrival order to be the bigger bottleneck than missing data.

Critical strengths

[1] The work studies an important setting of continual learning via model merging, which is not widely explored.

[2] The work states two important constraints relevant to the continual learning setting: arrival order and data-free, and shares empirical evidence that arrival order is a bigger constraint than missing data.

Critical weaknesses

[1] SCOPE combines sign-conflict scaling, order-dependent damping, and parameter-class coefficients, but relies on several heuristic formulations whose generalization across architectures, modalities, and task settings remains unclear.

[2] The performance gains over existing methods are minor and inconsistent across settings. As shown in Table 5, SCOPE does not consistently outperform the baselines on longer task streams, limiting the demonstrated benefit of the proposed method.

[3] The presentation requires improvement. Results are reported with inconsistent numerical precision (e.g., lines 70 and 284 and Table 3), making accuracy values and improvements harder to interpret and compare. Figures 1 and 4 also appear to contain misaligned or clipped elements, reducing readability.

Questions for authors

[1] How are the handcrafted parameters of Sign-Conflict, Order-scaled, and Parameter-class Estimation picked?

[2] Can the authors evaluate the methods across different modalities like language with the use of LLMs.

Reviewer_mN7WOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a complex continual model merging task and proposes a method, SCOPE. SCOPE constructs a closed-form merge operator with a three-scalar state. Besides, per-tensor coefficients combine parameter-class scaling, sign-conflict attenuation against the running merge, step damping, and norm stabilization. Extensive experiments are conducted to verify the methods' perfoemance.

Critical strengths
  1. Clear motivation. This paper clearly summarizes the key constraints of the continual model merging task.

  2. Well-designed framework. The proposed SCOPE operator is simple and well matched to the stated constraints. It uses a closed-form per-tensor update with an O(1) state, no samples, and no growing per-task statistics.

Critical weaknesses
  1. Limited comparison. This paper only compares with some alternative techniques. The comparison with state-of-the-art methods should be expanded.

  2. Limited representation. This paper does not provide a figure of the framework to show the overall process of the proposed method.

  3. Limited organization. This paper mixes the experiments and technical designs. It can be hard for readers to understand the key motivation and technical contributions. The authors are advised to briefly discuss the key problems and designs in the Introduction section.

  4. Limited novelty. The proposed method mainly consists of existing algorithms, including sign-conflict attenuation, parameter-class coefficients, norm stabilization, and magnitude trimming. The authors should highlight the original designs in this paper.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.