acceptodds
Under review as a conference paper at ICLR 2027

BiFilt: Bilateral Subspace Filtering for Data-Free Continual Model Merging

Abstract

Data-free continual model merging (DFCMM) aims to sequentially integrate independently fine-tuned experts into a single model without access to task data or previously seen experts. Existing subspace-based methods typically protect historical knowledge through input-side filtering, imposing and thereby suppressing all update responses along selected historical input directions. We show that preserving the selected historical input-output core requires only the weaker condition , revealing cross-subspace updates that are removed by one-sided filtering despite leaving the protected core unchanged. Based on this observation, we introduce BiFilt, a bilateral subspace filtering framework that decomposes each incoming update into four input-output interaction blocks, removes the jointly historical component, preserves the complementary component, and independently controls the two directional cross components. We derive BiFilt as the unique solution of a convex quadratic objective, yielding a simple closed-form filter and a fixed symmetric configuration without task-dependent tuning. Controlled task-pair analyses show that the released cross direction contains substantial new-task information while causing only limited historical degradation, and complete continual-merging experiments across multiple CLIP backbones and task horizons consistently demonstrate the benefit of partial cross-block retention over one-sided filtering. Further analyses reveal a stability-plasticity trade-off: partial retention improves task acquisition and final accuracy on standard sequences, whereas stricter filtering can provide stronger retention over longer horizons. These results show that distinguishing input-output interaction directions provides a more flexible parameter-only principle for continual model merging.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Data-Free Continual Model Merging, Model Merging, Subspace Filtering, Bilateral Subspace, Task Vectors

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pWf9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data-free continual model merging, where independently fine-tuned experts are sequentially integrated into a single model without access to task data or previously seen experts. It proposes BiFilt, a bilateral subspace filtering framework that decomposes parameter updates into four input-output interaction blocks and selectively retains cross-subspace information while protecting historical knowledge. Theoretical analysis and experiments across multiple CLIP backbones demonstrate the effectiveness of partial cross-block retention, highlighting a stability-plasticity trade-off between new-task acquisition and historical knowledge preservation.

Critical strengths
  1. The paper identifies a plausible limitation of one-sided subspace filtering: suppressing all updates along historical input directions may unnecessarily discard useful new-task information.

  2. BiFilt retains a single merged backbone without adding inference-time modules, making the approach potentially useful for data-free continual model merging.

Critical weaknesses
  1. Limited novelty. The core contribution is an extension of existing subspace-filtering methods, particularly NUFILT, through bilateral decomposition and cross-block retention. The paper should more clearly distinguish this advance from existing projection-based approaches and establish its broader significance.

  2. The core-preservation condition is a parameter-space constraint whose functional interpretation depends on the stated subspace-alignment assumptions. Moreover, the subsequent low-rank recovery can violate exact core preservation, so the guarantee does not directly extend to the final merged model.

  3. Although BiFilt consistently outperforms NUFILT in the main benchmark, the accuracy improvements are relatively small. The structural comparisons use only three to five seeds, and the long-horizon analysis relies on a single task order, limiting the strength of the conclusions about robustness and the stability–plasticity trade-off.

  4. Insufficient sensitivity analysis of filtering coefficients. The method primarily uses the fixed symmetric setting (\alpha=\beta=0.5), despite the potentially different effects of the two cross-subspace directions. More systematic sensitivity analyses are needed to establish the robustness of this choice and whether task- or layer-specific coefficients could improve performance.

  5. Although NUFILT and BiFilt use the same recovery configuration, the optimal recovery hyperparameters may differ between filtering strategies. The authors should clarify whether the baselines were fairly tuned and, ideally, provide matched sensitivity analyses to ensure that the reported gains are attributable to bilateral filtering rather than configuration choices.

Questions for authors
  1. Can you empirically validate the subspace coverage assumption by measuring activation and gradient energy captured by the SVD-derived subspaces?

  2. How sensitive are the results to (\alpha) and (\beta), and would layer-specific or task-specific settings be more effective?

  3. Can you provide results across multiple task orders and longer horizons to clarify when HARD may outperform BiFilt-Sym?

6 comments

W1 — Distinction from existing projection-based approaches

Thank you for the insightful comment. In Sections 2 and 4, we clarify the structural distinctions between BiFilt and existing methods (OPCM, DOP, NUFILT):

  • OPCM orthogonalizes flattened 1D parameter vectors globally, ignoring layer-wise matrix structure.
  • DOP optimizes singular-value-weighted dual projection objectives iteratively.
  • NUFILT (RIGHT) projects matrix updates via right singular vectors only (ZV=0ZV=0), discarding both historical-core (HH) and cross-block (CH) updates.
  • BiFilt projects via both left and right singular subspaces (U,V)(U, V) of accumulated updates, decomposing updates into four blocks (HH, HC, CH, CC). RIGHT, LEFT, and HARD arise as natural boundaries within this unified family.

As shown in Table R.1, Shared-Soft shares BiFilt-Sym's cross-block weights (0.5,0.5)(0.5, 0.5) but retains 0.25τ∥∥0.25\tau_{\parallel\parallel}; BiFilt-Sym outperforms Shared-Soft across all 5 seeds, confirming that strictly zeroing HH is essential. In addition, filtering alone without recovery outperforms NUFILT by +2.12 pp ACC (W5), establishing the distinct benefit of our bilateral formulation (Sec. 5.4).

Table R.1: Comparison with one-sided and two-sided soft-projection alternatives on 8 tasks (ViT-B/32, B0\mathcal{B}_0, 5 seeds)

MethodBlock Weights (HH, HC, CH, CC)ACC (%) ↑\uparrowBWT (pp) ↑\uparrow
RIGHT (NUFILT)(0, 1, 0, 1)83.33 ±\pm 0.27-2.25 ±\pm 0.70
RIGHT-Soft(0.5, 1, 0.5, 1)83.19 ±\pm 0.25-2.63 ±\pm 0.80
Shared-Soft(0.25, 0.5, 0.5, 1)83.65 ±\pm 0.31-1.98 ±\pm 0.51
BiFilt-Sym (Ours)(0, 0.5, 0.5, 1)83.79 ±\pm 0.27-1.88 ±\pm 0.54

W2 — Subspace-alignment assumptions and core preservation under recovery

We clarify that Theorem 1 characterizes conditional first-order safety under functional subspace coverage. As empirically established in Q1 (Table R.3), parameter SVD bases capture ~40% of measured activation and gradient energy. Because coverage is partial, Eq. 29 governs the resulting sensitivity bound via cross-block shrinkage.

Regarding recovery, our 5-seed constrained ablation on ViT-B/32 TA8 shows that enforcing exact core preservation on recovered updates (R⊥=R−UU⊤RVV⊤R_\perp = R - UU^\top R VV^\top, residual <10−6<10^{-6}) reduces mean ACC from 83.79% to 83.13% and BWT from -1.88 pp to -2.22 pp across all 5 seeds. Thus, while zeroing the core is essential for the initial proposal τ^t\widehat{\tau}_t, unconstrained trainable recovery provides the necessary flexibility to fit the joint probe objective (Sec. 3, Apps. A.6 and C.5).

W3 — Accuracy improvements, seed coverage, and long-horizon robustness

To substantiate statistical rigor across backbones and horizons:

  • Across all 9 settings in Table 1 (90 paired runs), BiFilt-Sym consistently surpasses NUFILT in mean ACC and BWT (winning 10/10 seeds on ViT-B/32 TA8). Mean ACC gains scale with horizon length: +0.4, +1.2, and +1.6 pp on ViT-B/32 (8, 14, 20 tasks), and +0.2, +0.9, and +1.5 pp on ViT-B/16.
  • The expanded 5-seed TALL20 structural comparison confirms that all bilateral variants (HARD, LEFT, BiFilt-Sym) outperform RIGHT in 5/5 orders (detailed in Q3, Table R.4; Sec. 5.4).

W4 — Sensitivity analysis of filtering coefficients (α,β)(\alpha, \beta)

We emphasize that α=β=0.5\alpha=\beta=0.5 was selected a priori from equal unit penalties (λU=λV=1\lambda_U=\lambda_V=1) and evaluated across all 9 benchmark settings without tuning.

To assess robustness around this default, we evaluated a 25-point grid over (α,β)∈{0,0.25,0.5,0.75,1}2(\alpha, \beta) \in \{0, 0.25, 0.5, 0.75, 1\}^2 on ViT-B/32 TA8 (Figure 6, Tables 23–24). Thirteen configurations lie within 0.1 pp of the peak (84.0%), and all nine interior settings outperform both the RIGHT (83.7%) and HARD (83.5%) boundaries. Multi-seed evaluations (Table 25) confirm that (0.5,0.5)(0.5, 0.5) achieves the highest mean ACC and BWT, verifying a stable plateau around the default.

W5 — Fairness of recovery hyperparameters across filtering strategies

All recovery hyperparameters were inherited directly from NUFILT (r=64r=64, 50 steps, lr 10−310^{-3}, k=128k=128). To verify that gains do not depend on specific recovery tuning, we swept recovery steps {0,10,50,200}\{0, 10, 50, 200\} (Table R.2a) and recovery rank {16,64,128}\{16, 64, 128\} (Table R.2b) across three matched seeds under B0\mathcal{B}_0.

BiFilt-Sym outperforms RIGHT across all tested step budgets and ranks:

  • Disabling recovery entirely (0 steps) yields a +2.12 pp ACC advantage (3/3 seeds), directly isolating the gain from bilateral filtering.
  • Across recovery ranks (r∈{16,64,128}r \in \{16, 64, 128\}), both ACC and BWT improve in 3/3 seeds at every rank.
  • Historical-rank sweeps (k∈{32,64,128,256}k \in \{32, 64, 128, 256\}, App. D.7) show consistent advantages across all configurations.

Table R.2a: Unified recovery-step sweep on ViT-B/32 TA8 (B0\mathcal{B}_0, 3 matched seeds, fixed recovery rank r=64r=64)

Recovery StepsRIGHT ACC (%)BiFilt-Sym ACC (%)Δ\DeltaACC (pp)Δ\DeltaBWT (pp)ACC Wins
078.6680.78+2.12-0.423 / 3
1081.6282.93+1.31+0.663 / 3
5083.2983.77+0.48+0.373 / 3
20083.3783.58+0.21+0.242 / 3

Table R.2b: Unified recovery-rank sweep on ViT-B/32 TA8 (B0\mathcal{B}_0, 3 matched seeds, fixed 5050 recovery steps)

Recovery Rank rrRIGHT ACC (%)BiFilt-Sym ACC (%)Δ\DeltaACC (pp)Δ\DeltaBWT (pp)ACC WinsBWT Wins
1682.9483.69+0.75+0.563 / 33 / 3
6483.2983.77+0.48+0.373 / 33 / 3
12883.4383.77+0.35+0.363 / 33 / 3

Q1 — Empirical validation of the subspace coverage assumption

Across 84 task-stage observations on ViT-B/32 TA8 (3 seeds, stages 2–8, 60 linear layers), we measured historical layer-input activation and layer-output loss-gradient energy captured by rank-128 SVD bases (Vℓ,Uℓ)(V_\ell, U_\ell) of St−1S_{t-1}.

Table R.3 shows that input bases VℓV_\ell capture 39.96% of activation energy and output bases UℓU_\ell capture 40.72% of gradient energy, exceeding same-rank random orthogonal baselines (16.66% and 14.17%) by 2.40$\times$ and 2.87$\times$. Layer-mean projection residuals positively correlate with one-step forgetting (ρ=0.316\rho = 0.316 and 0.3430.343). These findings confirm an informative functional correspondence while justifying cross-block shrinkage to manage residual drift (Sec. 3, App. C.2).

Table R.3: Subspace energy capture and correlation with one-step forgetting on ViT-B/32 TA8 (84 task-stage observations)

Subspace SideHistorical SVD (k=128k=128)Random Orthogonal (k=128k=128)Ratio (Hist. / Rand.)Spearman ρ\rho (Residual vs. Forgetting)
Input Activation (VℓV_\ell)39.96%16.66%2.40$\times$0.316
Output Gradient (UℓU_\ell)40.72%14.17%2.87$\times$0.343

Q2 — Sensitivity to (α,β)(\alpha, \beta) and layer- or task-specific settings

Beyond the 25-point uniform grid (which exhibits a broad performance plateau, W4), we investigated module-specific schedules assigning different cross-block weights to attention q/k_proj, v/out_proj, and MLP fc1.

While relaxing v/out_proj slightly improved incoming-task plasticity, no layer-specific schedule consistently improved historical retention across diverse task sequences. Furthermore, under strict data-free continual merging, no historical or validation data is available at merge time to tune layer- or task-specific parameters. The fixed symmetric default α=β=0.5\alpha=\beta=0.5 avoids heuristics, eliminates tuning overhead, and demonstrates stable performance across all evaluated settings (discussed in App. D.5).

Q3 — Multiple task orders on longer horizons: when HARD outperforms BiFilt-Sym

We expanded the ViT-B/32 TALL20 structural comparison to 5 matched seeds (42, 43, 44, 46, 47) to analyze the performance trade-off between HARD and BiFilt-Sym across horizon lengths (Table R.4).

Using the exact decomposition ACC=Acquire+T−1TBWT\mathrm{ACC} = \mathrm{Acquire} + \frac{T-1}{T}\mathrm{BWT}:

  • BiFilt-Sym attains the highest Acquire across both suites (85.43% on TA8, 80.33% on TALL20), whereas HARD achieves the strongest BWT (-1.63 pp on TA8, -6.91 pp on TALL20).
  • On TA8 (8 tasks), new-task acquisition dominates, so BiFilt-Sym's acquisition advantage (+0.80 pp) yields higher final ACC in all 5 orders (+0.57 pp over HARD).
  • On TALL20 (20 tasks), forgetting accumulates over long horizons, making retention decisive; HARD's BWT advantage (+1.44 pp) outweighs its acquisition deficit, producing +0.64 pp higher final ACC in all 5 orders, with LEFT also surpassing BiFilt-Sym (Sec. 5.4, App. C.7).

Table R.4: Structural comparison across 8-task (TA8) and 20-task (TALL20) horizons on ViT-B/32 (B0\mathcal{B}_0, 5 seeds)

Filter VariantTA8 ACC (%) ↑\uparrowTA8 BWT (pp) ↑\uparrowTA8 Acquire (%) ↑\uparrowTALL20 ACC (%) ↑\uparrowTALL20 BWT (pp) ↑\uparrowTALL20 Acquire (%) ↑\uparrow
RIGHT (0,1)(0, 1)83.33 ±\pm 0.27-2.25 ±\pm 0.7085.30 ±\pm 0.7370.83 ±\pm 0.81-9.08 ±\pm 1.9179.45
LEFT (1,0)(1, 0)83.62 ±\pm 0.35-1.74 ±\pm 0.5285.14 ±\pm 0.7372.66 ±\pm 0.93-7.45 ±\pm 1.8279.73
HARD (0,0)(0, 0)83.21 ±\pm 0.37-1.63 ±\pm 0.5784.64 ±\pm 0.8173.03 ±\pm 0.50-6.91 ±\pm 1.5779.59
BiFilt-Sym (0.5,0.5)(0.5, 0.5)83.79 ±\pm 0.27-1.88 ±\pm 0.5485.43 ±\pm 0.6772.39 ±\pm 0.72-8.36 ±\pm 1.5380.33
Reviewer_a5WGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data-free continual model merging, and proposes BiFilt, a bilateral subspace filtering framework that decomposes incoming updates into four input-output interaction blocks using the left and right singular subspaces of the accumulated update. Extensive experiments show the effectiveness of the proposed approach.

Critical strengths
  • The writing is generally good.
  • The motivation is clear.
  • Extensive ablation studies.
Critical weaknesses
  • Limited technical novelty. BiFilt mainly extends NUFILT's filtering, while keeping the same low-rank recovery process.
  • Historical task preservation depends on alignment assumptions, which are empirically unverified. Moreover, recovery can introduce a nonzero historical-core component and violate the preservation constraint.
  • The improvements over baselines are not consistent.
Questions for authors

Please see Weaknesses.

5 comments

W1 — Technical novelty beyond NUFILT

Thank you for the constructive comment. In the revised Section 4, we explicitly distinguish bilateral filtering from the shared recovery stage and provide controlled evaluations isolating the filtering mechanism:

  1. Four-block formulation vs. one-sided suppression: NUFILT suppresses all updates along historical input directions (ZV=0ZV=0), zeroing both HH and CH. BiFilt decomposes matrix updates into four blocks (HH, HC, CH, CC), removes only the joint historical core (HH, U⊤ZV=0U^\top ZV=0), and introduces partial shrinkage on cross blocks.
  2. Filtering isolation without recovery: In our recovery sweep on ViT-B/32 TA8 (3 seeds), disabling recovery entirely (0 steps) yields 80.78% ACC for BiFilt-Sym vs. 78.66% for RIGHT (+2.12 pp). Under light recovery (10 steps), BiFilt-Sym maintains a +1.31 pp lead, directly confirming that filtering alone drives substantial gains.
  3. Soft-projection ablation (Table R.1): Shared-Soft applies identical cross-block weights (0.5,0.5)(0.5, 0.5) but retains 0.25τ∥∥0.25\tau_{\parallel\parallel}. Strictly eliminating HH in BiFilt-Sym improves ACC and BWT across all 5 seeds, demonstrating the specific necessity of our four-block core removal (Sec. 5.4, App. C.4).

Table R.1: Comparison of BiFilt-Sym with NUFILT and soft-projection alternatives on 8 tasks (ViT-B/32, B0\mathcal{B}_0, 5 seeds)

MethodBlock Weights (HH, HC, CH, CC)ACC (%) ↑\uparrowBWT (pp) ↑\uparrow
RIGHT (NUFILT)(0, 1, 0, 1)83.33 ±\pm 0.27-2.25 ±\pm 0.70
RIGHT-Soft(0.5, 1, 0.5, 1)83.19 ±\pm 0.25-2.63 ±\pm 0.80
Shared-Soft(0.25, 0.5, 0.5, 1)83.65 ±\pm 0.31-1.98 ±\pm 0.51
BiFilt-Sym (Ours)(0, 0.5, 0.5, 1)83.79 ±\pm 0.27-1.88 ±\pm 0.54

W2 — Subspace alignment assumptions and core preservation under recovery

We have addressed both points through direct empirical validation in Section 3 and Appendices A.6, C.2, and C.5:

  1. Subspace coverage: Across 84 task-stage observations on ViT-B/32 TA8 (Table R.2), rank-128 SVD bases capture 39.96% of historical activation energy and 40.72% of gradient energy (2.40$\times$ and 2.87$\times$ over random references), with residuals correlating with forgetting (ρ=0.316,0.343\rho = 0.316, 0.343). The presence of residual energy qualifies Theorem 1 as an informative first-order proxy and motivates partial cross-block shrinkage (Eq. 29).
  2. Core preservation under recovery: To evaluate whether recovery disrupts core protection, we tested constrained recovery with R⊥=R−UU⊤RVV⊤R_\perp = R - UU^\top R VV^\top, enforcing exact selected-core preservation (U⊤τ~tV=0U^\top \widetilde{\tau}_t V = 0, residual <10−6<10^{-6}) at each merge. Constraining recovery reduces mean ACC from 83.79% to 83.13% and BWT from -1.88 pp to -2.22 pp across all 5 seeds. While strict core preservation is optimal for the initial update proposal, trainable recovery requires flexibility to optimize the joint probe objective.

Table R.2: Subspace energy capture and correlation with one-step forgetting on ViT-B/32 TA8 (84 task-stage observations)

Subspace SideHistorical SVD (k=128k=128)Random Orthogonal (k=128k=128)Ratio (Hist. / Rand.)Spearman ρ\rho (Residual vs. Forgetting)
Input Activation (VℓV_\ell)39.96%16.66%2.40$\times$0.316
Output Gradient (UℓU_\ell)40.72%14.17%2.87$\times$0.343

W3 — Consistency of improvements over baselines

To verify robustness across backbones and sequence lengths, we reference our 90 paired benchmark runs (Table 1) and expanded 5-seed TALL20 structural evaluation (Table R.3):

  • Across all 9 benchmark settings (3 backbones × 3 task suites × 10 seeds; Table 1), BiFilt-Sym achieves the highest mean ACC among all evaluated methods and improves mean BWT over NUFILT in every setting.
  • On the 20-task continual benchmark (Table R.3, 5 matched seeds), HARD, LEFT, and BiFilt-Sym each improve ACC over RIGHT in 5/5 orders (by +2.20, +1.83, and +1.56 pp). On TA8, BiFilt-Sym leads due to superior new-task acquisition; on TALL20, HARD achieves highest ACC and BWT, delineating the regime where strict retention dominates long-horizon stability (Sec. 5.4).

Table R.3: Multi-seed comparison on 20-task continual merging (ViT-B/32 TALL20, B0\mathcal{B}_0, 5 seeds)

MethodBlock Weights (HH, HC, CH, CC)ACC (%) ↑\uparrowBWT (pp) ↑\uparrowAcquire (%) ↑\uparrowWins vs. RIGHT
RIGHT (NUFILT)(0, 1, 0, 1)70.83 ±\pm 0.81-9.08 ±\pm 1.9179.45—
BiFilt-Sym (Ours)(0, 0.5, 0.5, 1)72.39 ±\pm 0.72-8.36 ±\pm 1.5380.335 / 5 (+1.56 ACC)
LEFT(0, 0, 1, 1)72.66 ±\pm 0.93-7.45 ±\pm 1.8279.735 / 5 (+1.83 ACC)
HARD(0, 0, 0, 1)73.03 ±\pm 0.50-6.91 ±\pm 1.5779.595 / 5 (+2.20 ACC)

We sincerely thank you for your thoughtful comments and constructive suggestions, which have been valuable in improving our work. We have carefully addressed your concerns in our responses and revised manuscript. We would greatly appreciate your feedback on whether our revisions have adequately resolved the issues you raised. We look forward to further discussion and welcome any additional suggestions.

Reviewer_QccbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies data-free continual model merging, where each merge can access only a shared pretrained model, the current merged model, and an incoming expert. The method extracts leading left and right singular subspaces from the accumulated parameter update and treats them as historical output and input directions. Instead of imposing the one-sided constraint ZV=0ZV=0, BiFilt imposes the weaker compressed-core constraint U⊤ZV=0U^\top ZV=0. It decomposes the incoming update into four orthogonal blocks, removes the jointly historical block, preserves the doubly complementary block, and independently shrinks the two cross blocks. The default configuration retains half of each cross block and subsequently applies low-rank recovery. Evaluation includes ordered task-pair diagnostics and continual merging on three CLIP backbones with 8-, 14-, and 20-task suites.

Critical strengths
  1. The bilateral perspective offers a useful mechanistic distinction. Separating historical input and output alignment is more expressive than deciding retention solely from input-side overlap. The task-pair diagnostic provides empirical evidence that a component removed by one-sided filtering can contain useful incoming-task information. This is the most informative observation in the paper.
  2. The comparison with the closest baseline has meaningful experimental controls. The main benchmark covers nine backbone–suite combinations with ten matched task orders per setting. NUFILT and BiFilt use the same recovery configuration, supporting the reported modest but consistent accuracy gains under the evaluated protocol. The paper also discloses task-specific degradation, competitive results from stricter filtering, and the effect of recovery on the final block geometry.
Critical weaknesses
  1. The link between parameter-subspace preservation and historical-task preservation is not empirically established. The first-order safety result in Appendix A.1 requires historical activations and output-side loss gradients to be covered by, or sufficiently aligned with, VV and UU, respectively. Truncated SVD of the accumulated parameter update does not guarantee these conditions, and the paper does not measure their validity in the evaluated models. Consequently, U⊤ZV=0U^\top ZV=0 guarantees preservation of a selected compressed parameter mapping, not preservation of historical-task behavior. The fixed-layer, first-order analysis also does not directly establish safety for finite, simultaneous multi-layer updates followed by recovery. Measuring activation and gradient projection residuals, and relating them to observed forgetting, would help validate the proposed explanation rather than relying on final accuracy alone.
  2. The significance of the unique closed-form solution and the algorithmic advance needs clearer qualification. Uniqueness holds for the particular strictly convex quadratic objective after fixing the subspaces and penalty coefficients. The derivation amounts to standard quadratic shrinkage over orthogonal blocks; it does not establish optimality with respect to task risk or justify the specific default α=β=0.5\alpha=\beta=0.5. Moreover, the symmetric filter is exactly the arithmetic mean of left- and right-sided hard filters, and the subsequent recovery mechanism follows NUFILT. The derivation is a valid formal interpretation, but its independent theoretical novelty appears limited. The paper should more clearly distinguish inherited components from the new mechanism and establish the latter's incremental value over simple alternatives.
  3. Aggregate metrics do not adequately reveal the task-acquisition and retention trade-off. The mean ACC improvements are useful, but they do not establish whether gains are broadly distributed or driven by a subset of tasks, potentially at the expense of persistent degradation on others. Table 16 already provides final per-task results for B/32 with eight tasks; however, complete stage-by-task accuracy matrices are not provided across the principal settings. SVHN fails to improve in all ten paired runs in that table, illustrating the heterogeneity hidden by the average. Furthermore, BWT is referenced to each method's own task-arrival accuracy, so a better BWT alone cannot distinguish stronger retention from weaker initial acquisition. Please provide the complete accuracy matrices at,ia_{t,i}, per-task arrival and final accuracies, forgetting measures, and paired uncertainty estimates, particularly for long sequences.
  4. Figure readability and visual organization need improvement. Text, axis labels, and annotations in some figures are too small to read comfortably. The visual distinction between the four interaction blocks, historical and complementary directions, and pre- and post-filtering states is insufficiently prominent. The layout and visual hierarchy do not communicate the central differences effectively, limiting the figures' explanatory value. Larger fonts, consistent colors and notation, fewer nonessential elements, and an explicit distinction between the filtered proposal and the recovered update would improve clarity.
Questions for authors
  1. Can you measure the projection residuals of historical activations and loss gradients onto VV and UU in representative settings, and assess whether these residuals predict forgetting? If alignment is weak, how should the functional-safety interpretation be qualified?
  2. Can you provide stage-by-task accuracy matrices and paired task-level results for the principal settings to separate acquisition gains from retention gains? Can LEFT/HARD comparisons also be extended to multiple long-sequence orders, with paired differences and uncertainty estimates?
  3. Can you provide the historical-task accuracy change for each pair to determine whether the mean decrease of 0.2 percentage points masks substantial individual degradations?
9 comments

W1 — Parameter-subspace preservation and historical-task preservation

Thank you for this foundational question. In the revised Section 3, we explicitly clarify that Theorem 1 characterizes conditional first-order safety under historical activation and output-gradient coverage, with parameter SVD bases serving as practical empirical proxies. To address the gap between parameter and functional subspaces, we have added three complementary evaluations:

  1. Empirical subspace coverage (detailed in Q1): Parameter-derived bases capture ~40% of measured activation and gradient energy across 84 observations (2.40$\times$ and 2.87$\times$ over random references), leaving non-zero residual energy (ϵh,ϵg>0\epsilon_h, \epsilon_g > 0). Equation 29 shows that cross-block shrinkage directly bounds the resulting first-order sensitivity under this partial coverage.
  2. Finite multi-layer block attribution: In a 60-layer block attribution at stage 4 (3 seeds) before recovery, retaining the cross block CH (⊥∥\perp\parallel) yields an incoming-task gain of +25.78 pp compared to only +8.07 pp for HH (∥∥\parallel\parallel), while both induce <0.5<0.5 pp historical change. This confirms that cross blocks carry substantial new-task capacity with minimal interference.
  3. Core-constrained recovery ablation: We evaluated constrained recovery via R⊥=R−UU⊤RVV⊤R_\perp = R - UU^\top R VV^\top, enforcing U⊤τ~tV=0U^\top \widetilde{\tau}_t V = 0 at each merge. As shown in Table R.1, enforcing this constraint lowers ACC (-0.66 pp) and BWT (-0.34 pp) across all 5 seeds. This demonstrates that while exact core removal is critical for the un-tuned proposal τ^t\widehat{\tau}_t, trainable recovery requires parameter flexibility to fit the joint historical-incoming probe objective (Eq. 11).

These analyses are incorporated in Section 3 and Appendices A.6, C.2, C.3, and C.5 of the revised paper.

Table R.1: Comparison of standard recovery and core-constrained recovery on 8 tasks (ViT-B/32, B0\mathcal{B}_0, 5 seeds)

MethodCore Constraint (U⊤RV=0U^\top R V = 0)ACC (%) ↑\uparrowBWT (pp) ↑\uparrow
BiFilt-Sym (Constrained Recovery)Enforced (residual <10−6< 10^{-6})83.13 ±\pm 0.25-2.22 ±\pm 0.70
BiFilt-Sym (Ours)Unconstrained83.79 ±\pm 0.27-1.88 ±\pm 0.54

W2 — Qualification of the closed-form solution and comparison to simple alternatives

Thank you for the suggestion. In the revised Sections 4.2 and 4.3, we present Eq. 7 as a transparent block-shrinkage parametrization with a unique closed-form solution for fixed subspaces and penalty coefficients, clearly separating bilateral filtering from the inherited recovery stage.

While the identity F0.5,0.5(τ)=12[(I−PU)τ+τ(I−PV)]\mathcal{F}_{0.5,0.5}(\tau)=\frac{1}{2}[(I-P_U)\tau+\tau(I-P_V)] decomposes into one-sided projections, this equivalence holds strictly because the joint historical core τ∥∥\tau_{\parallel\parallel} (HH) is completely zeroed. Table R.2 compares BiFilt-Sym against one-sided soft filtering (RIGHT-Soft) and independent two-sided soft projection (Shared-Soft), which matches BiFilt-Sym's cross-block weights (0.5,0.5)(0.5, 0.5) but retains 0.25τ∥∥0.25\tau_{\parallel\parallel}. BiFilt-Sym outperforms both baselines in ACC and BWT across all five seeds, confirming the empirical necessity of strictly eliminating the HH component (now detailed in Sec. 5.4 and App. C.4).

Table R.2: Comparison with simple soft-projection alternatives on 8 tasks (ViT-B/32, B0\mathcal{B}_0, 5 seeds)

MethodBlock Weights (HH, HC, CH, CC)ACC (%) ↑\uparrowBWT (pp) ↑\uparrow
RIGHT (NUFILT)(0, 1, 0, 1)83.33 ±\pm 0.27-2.25 ±\pm 0.70
RIGHT-Soft(0.5, 1, 0.5, 1)83.19 ±\pm 0.25-2.63 ±\pm 0.80
Shared-Soft(0.25, 0.5, 0.5, 1)83.65 ±\pm 0.31-1.98 ±\pm 0.51
BiFilt-Sym (Ours)(0, 0.5, 0.5, 1)83.79 ±\pm 0.27-1.88 ±\pm 0.54

W3 — Task-acquisition vs. retention trade-off and task-level heterogeneity

To explicitly isolate new-task acquisition from historical retention and expose task-level dynamics, we added an exact paired decomposition: ACC=Acquire+T−1TBWT\mathrm{ACC} = \mathrm{Acquire} + \frac{T-1}{T}\mathrm{BWT}, alongside the per-task final-accuracy analysis (Appendices D.2 and D.4.1).

On the 10-seed ViT-B/32 TA8 benchmark, the paired +0.43 pp ACC gain (BiFilt-Sym over NUFILT) decomposes into +0.01 pp from Acquire (85.94% vs. 85.93%) and +0.42 pp from BWT (78×0.48 pp\frac{7}{8} \times 0.48\text{ pp}), improving in all 10 paired runs. This verifies that accuracy improvements stem from enhanced retention rather than altered initial acquisition. At the task level (Table 22), DTD, EuroSAT, RESISC45, and SUN397 improve in all 10 seeds (+0.8 to +1.3 pp), while SVHN exhibits a mild decrease (-0.7 pp, 93.2% to 92.5%); across all tasks combined, overall ACC and BWT improve in every single seed.

W4 — Figure readability and visual organization

Thank you for the constructive suggestion! We redesigned the method overview figure (Figure 3) as a four-stage pipeline:

  • Panel 1 shows the historical SVD geometry, with blue for the output subspace UU and orange for the input subspace VV.
  • Panel 2 shows the four interaction blocks and their filtering weights: HH is set to zero, HC and CH are scaled by β\beta and α\alpha, and CC is retained. The filtered proposal is τ^t\widehat{\tau}_t.
  • Panel 3 shows low-rank recovery, Rt=τtBtAtR_t=\tau_tB_tA_t, and the recovered update τ~t=τ^t+Rt\widetilde{\tau}_t=\widehat{\tau}_t+R_t. Panel 4 shows fusion into the merged model. The revised caption states that exact selected-core preservation applies to the filtered proposal and clarifies that WtW_t in the figure denotes merged layer weights.

Q1 — Measuring activation and gradient projection residuals

We quantified the fraction of historical layer-input activation energy and layer-output loss-gradient energy captured by rank-128 SVD bases (Vℓ,Uℓ)(V_\ell, U_\ell) of St−1S_{t-1} across 84 task-stage observations (3 seeds, stages 2–8, 60 linear layers on ViT-B/32 TA8).

As shown in Table R.3, parameter SVD bases capture 39.96% of activation energy and 40.72% of output-gradient energy—a 2.40$\times$ and 2.87$\times$ enrichment over random orthogonal baselines. Layer-mean residuals positively correlate with one-step forgetting (Spearman ρ=0.316\rho = 0.316 and 0.3430.343). These results validate SVD subspaces as informative functional proxies while confirming that residual energy justifies cross-block shrinkage (Sec. 3, App. C.2).

Table R.3: Subspace energy capture and correlation with one-step forgetting on ViT-B/32 TA8 (84 task-stage observations)

Subspace SideHistorical SVD (k=128k=128)Random Orthogonal (k=128k=128)Ratio (Hist. / Rand.)Spearman ρ\rho (Residual vs. Forgetting)
Input Activation (VℓV_\ell)39.96%16.66%2.40$\times$0.316
Output Gradient (UℓU_\ell)40.72%14.17%2.87$\times$0.343

Q2 — Multi-seed long-sequence (TALL20) comparisons and separating acquisition from retention

We extended the ViT-B/32 TALL20 structural comparison (B0\mathcal{B}_0) to 5 matched task orders (seeds 42, 43, 44, 46, 47) and decomposed accuracy into Acquire and BWT across both TA8 and TALL20 (Table R.4).

On TALL20, HARD, LEFT, and BiFilt-Sym all improve ACC over RIGHT in all 5 orders (mean gains of +2.20, +1.83, and +1.56 pp). Across horizons, BiFilt-Sym attains the highest Acquire (85.43% on TA8, 80.33% on TALL20), whereas HARD achieves the strongest BWT (-1.63 pp on TA8, -6.91 pp on TALL20). On TA8, higher acquisition gives BiFilt-Sym the highest final ACC in all 5 orders; on TALL20, error accumulation makes retention dominant, allowing HARD to exceed BiFilt-Sym by +0.64 pp ACC (5/5 orders) despite lower acquisition (now discussed in Sec. 5.4 and App. C.7).

Table R.4: Structural comparison across 8-task (TA8) and 20-task (TALL20) horizons on ViT-B/32 (B0\mathcal{B}_0, 5 seeds)

Filter VariantTA8 ACC (%) ↑\uparrowTA8 BWT (pp) ↑\uparrowTA8 Acquire (%) ↑\uparrowTALL20 ACC (%) ↑\uparrowTALL20 BWT (pp) ↑\uparrowTALL20 Acquire (%) ↑\uparrow
RIGHT (0,1)(0, 1)83.33 ±\pm 0.27-2.25 ±\pm 0.7085.30 ±\pm 0.7370.83 ±\pm 0.81-9.08 ±\pm 1.9179.45
LEFT (1,0)(1, 0)83.62 ±\pm 0.35-1.74 ±\pm 0.5285.14 ±\pm 0.7372.66 ±\pm 0.93-7.45 ±\pm 1.8279.73
HARD (0,0)(0, 0)83.21 ±\pm 0.37-1.63 ±\pm 0.5784.64 ±\pm 0.8173.03 ±\pm 0.50-6.91 ±\pm 1.5779.59
BiFilt-Sym (0.5,0.5)(0.5, 0.5)83.79 ±\pm 0.27-1.88 ±\pm 0.5485.43 ±\pm 0.6772.39 ±\pm 0.72-8.36 ±\pm 1.5380.33

Q3 — Distribution of historical-task accuracy changes across all 56 pairs

We evaluated the pairwise distribution across all 56 ordered task pairs on ViT-B/32 to test whether the mean historical change (-0.18 pp) hides severe local drops.

As shown in Table R.5, restoring the CH (⊥∥\perp\parallel) block improves current-task accuracy in all 56/56 pairs (mean +5.24 pp, median +3.37 pp). Meanwhile, historical accuracy changes remain within ±0.10\pm 0.10 pp in 26/56 pairs (46.4%), with the single worst-case degradation being only -0.75 pp (DTD →\to SUN397). This confirms that cross-block shrinkage provides consistent current-task gains with negligible historical risk (Sec. 3, App. C.3).

Table R.5: Distribution of accuracy changes from restoring the CH (⊥∥\perp\parallel) block across all 56 ordered task pairs

MetricMean (pp)Median (pp)Best Pair (pp)Worst Pair (pp)Pairs with Δ≥−0.10\Delta\ge-0.10 pp
Current-Task Gain+5.24+3.37+17.45 (SVHN →\to GTSRB)+0.3156 / 56 (100%)
Historical-Task Change-0.18-0.12+0.11-0.75 (DTD →\to SUN397)26 / 56 (46.4%)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

We thank all reviewers for their constructive feedback. Following the comments, we made the following revisions:

Functional alignment & post-filtering recovery — Reviewers Qccb, a5WG, and pWf9. We clarified the conditional coverage assumptions behind our first-order safe-space result and added empirical measurements confirming that parameter-derived SVD subspaces capture substantially more historical activation and gradient energy than random subspaces. We also added a core-constrained recovery ablation and a white-box recovery case study showing how recovery replaces the zeroed HH proposal with a low-rank, probe-calibrated adjustment.

Cross-block roles & recovery fairness — Reviewers Qccb, a5WG, and pWf9. We separated the inherited recovery stage from our bilateral filtering mechanism and explicitly clarified that retaining CH primarily supports new-task acquisition whereas shrinking HC drives improved historical retention. We also added comparisons against soft-projection alternatives and unified recovery-step and recovery-rank sweeps confirming consistent gains across configurations.

Multi-seed long-horizon analysis — Reviewers Qccb, a5WG, and pWf9. We expanded the 20-task (TALL20) structural evaluation from a single task order to five matched seeds, confirming that all bilateral variants consistently outperform one-sided input filtering while characterizing the suite-dependent trade-off between acquisition and retention.

Task-level decomposition & visual presentation — Reviewers Qccb and a5WG. We added the complete distribution of historical-task changes across all 56 ordered task pairs, a simultaneous 60-layer block attribution, and an exact decomposition of accuracy gains into task acquisition and backward transfer. We also redesigned Figure 3 with larger typography and an explicit visual distinction between the pre-recovery filtered proposal and the final update after recovery.

We sincerely appreciate the reviewers' helpful comments and suggestions, which have significantly enhanced the clarity and rigor of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.