- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes debtmerge, a method for continual model merging that tracks accumulated violations of forgetting, plasticity, and capacity budgets, and uses them to weight each merge objective. Each merge is chosen by an evolutionary search that reuses parts of earlier merge decisions. Experiments on vision and language models show improved average performance over prior continual merging baselines.
Critical strengths
Strengths
-
The proposed method shows the best average performance among model merging baselines across vision and language models, up to a 1.5B-scale LLM. Its advantage grows as the task stream gets longer, which supports the paper's core motivation for long-horizon merging.
-
The ablations clearly separate the contribution of each component, and the extensive analysis throughout the main paper and the appendix provides controlled comparisons that isolate the effect of the proposed debt-based control from simpler control schemes.
Critical weaknesses
Weaknesses
- The formulation is not self-contained, which significantly hurts readability. The method builds on Lyapunov drift-plus-penalty optimization from network control domain, but never explains why it suits long-horizon continual merging, making the reader who is unfamiliar to the domain hard to understand the method's core motivation. It is only cited in passing in the introduction and related work. Including this, Many sentences rely on external references without explanation, and the following are only a few examples:
- "Following the standard virtual-queue construction ... (Neely, 2010)" (Line 216): "virtual queue" is never properly explained. In the current manuscript, virtual queue, debt, and merge-debt state all refer to , in my understandings.
- "Motivated by the data-free interference signals studied in existing work ..." (Line 160): the related work covers these papers only in short words, so it is unclear what those signals actually are in the paper's notation and how the paper is motivated from those.
- "DebtMerge employs evolutionary transfer optimization (Tan et al., 2021; Xue et al., 2025)": the paper does not explain what evolutionary transfer optimization is or how it differs from standard evolutionary search. Also, the terms for crossover, mutation, and repair are used but their specific operators are not specified.
- There are lots of undefined notations throughout the paper, which require substantial revision. Some examples I found are listed below:
- Undefined: (Line 226); (Line 258); (Eq. 8)
- Defined only in the appendix: (used in Eq. 7)
As a result, the paper is unnecessarily jargony, also coming with domain-specific terms that are not well explained.
Questions for authors
Questions
- Are "virtual queue", "debt", and "merge-debt state" the same quantity ?
- What is a "compatible" block, and how is compatibility determined? It is never properly defined, even in Algorithm 1. How does "compatible" differ from "feasible"? Also, what does "adapted to the current expert bank" mean?
- Comparing Table 1 in the main paper with Table 11 in the appendix, the MADE-IT results differ significantly between its original implementation and in the authors' implementation, making the gap marginal in the controlled table. What causes this difference?
The authors’ responses have further reinforced my concerns. Prior work has consistently addressed settings where merging interference and forgetting are genuine challenges. In contrast, under the authors’ experimental setup, Allocate-per-task, without using the proposed strategies, already achieves performance close to independent fine-tuning with almost no forgetting. This raises concerns that the underlying setup may have substantially weakened or even circumvented the core difficulties the paper aims to address. Consequently, subsequent experiments conducted under this setup cannot adequately establish the effectiveness of the proposed strategies when these difficulties are actually present, and the current comparisons may overestimate their relative performance advantages.
The capacity–compression trade-off results also show that different configurations can yield different performance under similar actual storage budgets. Therefore, the number of expert slots, M, cannot be directly equated with the actual resource budget. Using slot counts as the primary proxy for resource modeling and cross-method comparisons may not adequately ensure fair comparisons and may confound configuration benefits with methodological contributions.
Furthermore, Retain accounts for only 1.9% and 1.6% of executed actions when free slots are available and when the expert bank is full, respectively, which further raises questions about its necessity. Its independent contribution still needs to be verified through a dedicated ablation that removes Retain.