- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies data-free, single-pass continual model merging without revisiting historical experts and proposes SCOPE, which carries only three scalar state variables. The method updates CLIP ViT-B/32 experts using parameter-category coefficients, a sign-conflict factor relative to the current merged model, stream-length-scaled step-size decay, norm stabilization, and final-step clipping. The paper evaluates pools of 8, 14, and 20 experts under multiple arrival orders and argues that arrival order, rather than the absence of data, is the main bottleneck in continual merging.
Critical strengths
- The problem constraints are clearly defined. Section 2 explicitly distinguishes the no-revisiting/order constraint C1 from the no-data-during-merging constraint C2, and provides the state variables and complete operator code.
- The experimental reporting is relatively transparent. Table 2 uses paired comparisons under identical arrival orders; Table 3 provides component-wise ablations; and Section 5.5 proactively reports the negative result that the original operator falls below the unmerged baseline on long streams.
- Reproducibility, ethics, and AI use are explicitly addressed. The appendix provides arrival orders, per-order results, implementation details, and the sources of reported values.
Critical weaknesses
- The causal claim that “arrival order is the bottleneck” is not rigorously isolated. In Section 3.1, the comparison among one-shot TIES and two sequential TIES implementations changes both the objects available to the algorithm and the update rule across regimes; the data-enabled condition in Section 3.2 then switches to RegMean. The sequential-with-data results in Appendix A.10 use the default eight-expert evaluation rather than orders 100–109 from Table 1. The current results therefore demonstrate an interaction between method and regime, but are insufficient to attribute the approximately 0.70-to-0.83 gap primarily to arrival order.
- Additional conditions exclude stronger methods that satisfy the broader setting. DOP in Section 3.3 satisfies both sequential arrival and no data during merging, but falls outside the authors' additionally defined “closed-form” subclass because it performs optimization at each step; its accuracy is 0.7798, approximately 8 percentage points above SCOPE's 0.6999. MagMax in Appendix A.14 likewise satisfies the single-pass, data-free constraints and achieves 0.7110, above SCOPE, but is excluded because its result is order-invariant and does not depend on a running model. Order insensitivity is not, by itself, a natural basis for exclusion from performance comparisons. Thus, the “best in class” and broader continual-merging claims hold only within the authors' more narrowly defined subclass.
- The core gain is very small, and statistical repetition does not substitute for task generalization. Table 2 shows an average improvement of only 0.0015 over the closely related MergeOp-M1 baseline. All 30 permutations come from the same set of 8 task experts. The paired test establishes consistency across orders for that fixed pool, but does not provide independent evidence across task pools, backbones, or modalities.
- The long-stream results weaken the methodological contribution. In Table 5, unrepaired SCOPE degrades severely with 14/20 experts. After the repair, its 20-expert accuracy is 0.6093, still below sequential TIES (0.6119) and simple averaging (0.6110). Moreover, the 14- and 20-expert settings use
κ=3.0andκ=2.5, respectively, rather than one fixed operator that requires no stream-length-specific hyperparameter selection. The paper states that the exponent is selected on orders 100–104 and then reports the mean over 100–109; the final result therefore still includes the five orders used for selection and is not a fully independent blind evaluation. - There is a risk of design overfitting to the same task assets. Constants for the main eight-task result are derived from a proxy objective and design orders on TA8, while the final evaluation changes only the permutations, not the task pool. The 14/20-expert expanded pools still come from the same FusionBench assets and use the same backbone. Appendix A.9 further shows that the proxy optimum does not coincide with the optimum under the official metric.
Questions for authors
- Can the authors conduct a complete C1xC2 factorial experiment using the same task pool, order, evaluator, and operator family, to genuinely separate the effects of order and data?
- Why should DOP and MagMax not be considered valid competitors under the stated problem setting? If they are included, how should the paper's main conclusions be revised?
- Can the authors pre-specify a single
κand evaluate it blindly across new task pools, backbones, and stream lengths, rather than using different values for 14/20 experts? - Does the 0.15-percentage-point gain persist on new task sets, rather than only across permutations of the same eight-task pool?