- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies data-free continual model merging, where independently fine-tuned experts are sequentially integrated into a single model without access to task data or previously seen experts. It proposes BiFilt, a bilateral subspace filtering framework that decomposes parameter updates into four input-output interaction blocks and selectively retains cross-subspace information while protecting historical knowledge. Theoretical analysis and experiments across multiple CLIP backbones demonstrate the effectiveness of partial cross-block retention, highlighting a stability-plasticity trade-off between new-task acquisition and historical knowledge preservation.
Critical strengths
-
The paper identifies a plausible limitation of one-sided subspace filtering: suppressing all updates along historical input directions may unnecessarily discard useful new-task information.
-
BiFilt retains a single merged backbone without adding inference-time modules, making the approach potentially useful for data-free continual model merging.
Critical weaknesses
-
Limited novelty. The core contribution is an extension of existing subspace-filtering methods, particularly NUFILT, through bilateral decomposition and cross-block retention. The paper should more clearly distinguish this advance from existing projection-based approaches and establish its broader significance.
-
The core-preservation condition is a parameter-space constraint whose functional interpretation depends on the stated subspace-alignment assumptions. Moreover, the subsequent low-rank recovery can violate exact core preservation, so the guarantee does not directly extend to the final merged model.
-
Although BiFilt consistently outperforms NUFILT in the main benchmark, the accuracy improvements are relatively small. The structural comparisons use only three to five seeds, and the long-horizon analysis relies on a single task order, limiting the strength of the conclusions about robustness and the stability–plasticity trade-off.
-
Insufficient sensitivity analysis of filtering coefficients. The method primarily uses the fixed symmetric setting (\alpha=\beta=0.5), despite the potentially different effects of the two cross-subspace directions. More systematic sensitivity analyses are needed to establish the robustness of this choice and whether task- or layer-specific coefficients could improve performance.
-
Although NUFILT and BiFilt use the same recovery configuration, the optimal recovery hyperparameters may differ between filtering strategies. The authors should clarify whether the baselines were fairly tuned and, ideally, provide matched sensitivity analyses to ensure that the reported gains are attributable to bilateral filtering rather than configuration choices.
Questions for authors
-
Can you empirically validate the subspace coverage assumption by measuring activation and gradient energy captured by the SVD-derived subspaces?
-
How sensitive are the results to (\alpha) and (\beta), and would layer-specific or task-specific settings be more effective?
-
Can you provide results across multiple task orders and longer horizons to clarify when HARD may outperform BiFilt-Sym?
6 comments
W5 — Fairness of recovery hyperparameters across filtering strategies
All recovery hyperparameters were inherited directly from NUFILT (, 50 steps, lr , ). To verify that gains do not depend on specific recovery tuning, we swept recovery steps (Table R.2a) and recovery rank (Table R.2b) across three matched seeds under .
BiFilt-Sym outperforms RIGHT across all tested step budgets and ranks:
- Disabling recovery entirely (
0 steps) yields a +2.12 pp ACC advantage (3/3 seeds), directly isolating the gain from bilateral filtering. - Across recovery ranks (), both ACC and BWT improve in 3/3 seeds at every rank.
- Historical-rank sweeps (, App. D.7) show consistent advantages across all configurations.
Table R.2a: Unified recovery-step sweep on ViT-B/32 TA8 (, 3 matched seeds, fixed recovery rank )
| Recovery Steps | RIGHT ACC (%) | BiFilt-Sym ACC (%) | ACC (pp) | BWT (pp) | ACC Wins |
|---|---|---|---|---|---|
| 0 | 78.66 | 80.78 | +2.12 | -0.42 | 3 / 3 |
| 10 | 81.62 | 82.93 | +1.31 | +0.66 | 3 / 3 |
| 50 | 83.29 | 83.77 | +0.48 | +0.37 | 3 / 3 |
| 200 | 83.37 | 83.58 | +0.21 | +0.24 | 2 / 3 |
Table R.2b: Unified recovery-rank sweep on ViT-B/32 TA8 (, 3 matched seeds, fixed recovery steps)
| Recovery Rank | RIGHT ACC (%) | BiFilt-Sym ACC (%) | ACC (pp) | BWT (pp) | ACC Wins | BWT Wins |
|---|---|---|---|---|---|---|
| 16 | 82.94 | 83.69 | +0.75 | +0.56 | 3 / 3 | 3 / 3 |
| 64 | 83.29 | 83.77 | +0.48 | +0.37 | 3 / 3 | 3 / 3 |
| 128 | 83.43 | 83.77 | +0.35 | +0.36 | 3 / 3 | 3 / 3 |
Q1 — Empirical validation of the subspace coverage assumption
Across 84 task-stage observations on ViT-B/32 TA8 (3 seeds, stages 2–8, 60 linear layers), we measured historical layer-input activation and layer-output loss-gradient energy captured by rank-128 SVD bases of .
Table R.3 shows that input bases capture 39.96% of activation energy and output bases capture 40.72% of gradient energy, exceeding same-rank random orthogonal baselines (16.66% and 14.17%) by 2.40$\times$ and 2.87$\times$. Layer-mean projection residuals positively correlate with one-step forgetting ( and ). These findings confirm an informative functional correspondence while justifying cross-block shrinkage to manage residual drift (Sec. 3, App. C.2).
Table R.3: Subspace energy capture and correlation with one-step forgetting on ViT-B/32 TA8 (84 task-stage observations)
| Subspace Side | Historical SVD () | Random Orthogonal () | Ratio (Hist. / Rand.) | Spearman (Residual vs. Forgetting) |
|---|---|---|---|---|
| Input Activation () | 39.96% | 16.66% | 2.40$\times$ | 0.316 |
| Output Gradient () | 40.72% | 14.17% | 2.87$\times$ | 0.343 |
Q2 — Sensitivity to and layer- or task-specific settings
Beyond the 25-point uniform grid (which exhibits a broad performance plateau, W4), we investigated module-specific schedules assigning different cross-block weights to attention q/k_proj, v/out_proj, and MLP fc1.
While relaxing v/out_proj slightly improved incoming-task plasticity, no layer-specific schedule consistently improved historical retention across diverse task sequences. Furthermore, under strict data-free continual merging, no historical or validation data is available at merge time to tune layer- or task-specific parameters. The fixed symmetric default avoids heuristics, eliminates tuning overhead, and demonstrates stable performance across all evaluated settings (discussed in App. D.5).
Q3 — Multiple task orders on longer horizons: when HARD outperforms BiFilt-Sym
We expanded the ViT-B/32 TALL20 structural comparison to 5 matched seeds (42, 43, 44, 46, 47) to analyze the performance trade-off between HARD and BiFilt-Sym across horizon lengths (Table R.4).
Using the exact decomposition :
BiFilt-Symattains the highestAcquireacross both suites (85.43% on TA8, 80.33% on TALL20), whereasHARDachieves the strongestBWT(-1.63 pp on TA8, -6.91 pp on TALL20).- On TA8 (8 tasks), new-task acquisition dominates, so
BiFilt-Sym's acquisition advantage (+0.80 pp) yields higher final ACC in all 5 orders (+0.57 pp overHARD). - On TALL20 (20 tasks), forgetting accumulates over long horizons, making retention decisive;
HARD's BWT advantage (+1.44 pp) outweighs its acquisition deficit, producing +0.64 pp higher final ACC in all 5 orders, withLEFTalso surpassingBiFilt-Sym(Sec. 5.4, App. C.7).
Table R.4: Structural comparison across 8-task (TA8) and 20-task (TALL20) horizons on ViT-B/32 (, 5 seeds)
| Filter Variant | TA8 ACC (%) | TA8 BWT (pp) | TA8 Acquire (%) | TALL20 ACC (%) | TALL20 BWT (pp) | TALL20 Acquire (%) |
|---|---|---|---|---|---|---|
| RIGHT | 83.33 0.27 | -2.25 0.70 | 85.30 0.73 | 70.83 0.81 | -9.08 1.91 | 79.45 |
| LEFT | 83.62 0.35 | -1.74 0.52 | 85.14 0.73 | 72.66 0.93 | -7.45 1.82 | 79.73 |
| HARD | 83.21 0.37 | -1.63 0.57 | 84.64 0.81 | 73.03 0.50 | -6.91 1.57 | 79.59 |
| BiFilt-Sym | 83.79 0.27 | -1.88 0.54 | 85.43 0.67 | 72.39 0.72 | -8.36 1.53 | 80.33 |
W1 — Distinction from existing projection-based approaches
Thank you for the insightful comment. In Sections 2 and 4, we clarify the structural distinctions between BiFilt and existing methods (
OPCM,DOP,NUFILT):OPCMorthogonalizes flattened 1D parameter vectors globally, ignoring layer-wise matrix structure.DOPoptimizes singular-value-weighted dual projection objectives iteratively.NUFILT(RIGHT) projects matrix updates via right singular vectors only (ZV=0), discarding both historical-core (HH) and cross-block (CH) updates.BiFiltprojects via both left and right singular subspaces (U,V) of accumulated updates, decomposing updates into four blocks (HH,HC,CH,CC).RIGHT,LEFT, andHARDarise as natural boundaries within this unified family.As shown in Table R.1,
Shared-SoftsharesBiFilt-Sym's cross-block weights (0.5,0.5) but retains 0.25τ∥∥;BiFilt-SymoutperformsShared-Softacross all 5 seeds, confirming that strictly zeroingHHis essential. In addition, filtering alone without recovery outperforms NUFILT by +2.12 pp ACC (W5), establishing the distinct benefit of our bilateral formulation (Sec. 5.4).Table R.1: Comparison with one-sided and two-sided soft-projection alternatives on 8 tasks (ViT-B/32, B0, 5 seeds)
W2 — Subspace-alignment assumptions and core preservation under recovery
We clarify that Theorem 1 characterizes conditional first-order safety under functional subspace coverage. As empirically established in Q1 (Table R.3), parameter SVD bases capture ~40% of measured activation and gradient energy. Because coverage is partial, Eq. 29 governs the resulting sensitivity bound via cross-block shrinkage.
Regarding recovery, our 5-seed constrained ablation on ViT-B/32 TA8 shows that enforcing exact core preservation on recovered updates (R⊥=R−UU⊤RVV⊤, residual <10−6) reduces mean ACC from 83.79% to 83.13% and BWT from -1.88 pp to -2.22 pp across all 5 seeds. Thus, while zeroing the core is essential for the initial proposal τt, unconstrained trainable recovery provides the necessary flexibility to fit the joint probe objective (Sec. 3, Apps. A.6 and C.5).
W3 — Accuracy improvements, seed coverage, and long-horizon robustness
To substantiate statistical rigor across backbones and horizons:
BiFilt-Symconsistently surpassesNUFILTin mean ACC and BWT (winning 10/10 seeds on ViT-B/32 TA8). Mean ACC gains scale with horizon length: +0.4, +1.2, and +1.6 pp on ViT-B/32 (8, 14, 20 tasks), and +0.2, +0.9, and +1.5 pp on ViT-B/16.HARD,LEFT,BiFilt-Sym) outperformRIGHTin 5/5 orders (detailed in Q3, Table R.4; Sec. 5.4).W4 — Sensitivity analysis of filtering coefficients (α,β)
We emphasize that α=β=0.5 was selected a priori from equal unit penalties (λU=λV=1) and evaluated across all 9 benchmark settings without tuning.
To assess robustness around this default, we evaluated a 25-point grid over (α,β)∈{0,0.25,0.5,0.75,1}2 on ViT-B/32 TA8 (Figure 6, Tables 23–24). Thirteen configurations lie within 0.1 pp of the peak (84.0%), and all nine interior settings outperform both the
RIGHT(83.7%) andHARD(83.5%) boundaries. Multi-seed evaluations (Table 25) confirm that (0.5,0.5) achieves the highest mean ACC and BWT, verifying a stable plateau around the default.