acceptodds
Under review as a conference paper at ICLR 2027

The Stability Trap in Multimodal Foundation Model Continual Learning

Abstract

Continual adaptation keeps multimodal foundation models useful as data evolve, yet historical protection can impose an avoidable cost on new-task learning—a stability trap. Focusing on vision–language models, we identify and reduce protection-induced plasticity loss: adaptation recoverable by relaxing historical constraints while keeping losses in historical and pretrained performance within specified tolerances. We measure this loss through paired learning runs that start from identical model and optimizer states, vary only the retained protection, and jointly assess new-task learning and knowledge retention. We introduce Budgeted Functional Protection (BFP), which periodically reallocates a fixed number of protected parameter directions according to their estimated influence on model outputs, leaving the remaining directions available for learning. On VLCL under a common training budget, BFP improves image-to-text and text-to-image Recall@1 over Pi-CCA by 1.11 and 1.06 percentage points, respectively, while reducing pretrained accuracy drop from 3.02 to 1.37 points. The results support selective protection as a way to improve learning and retention, with the paired diagnostic measuring the adaptation cost of the retained constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual learning, multimodal foundation models, vision-language models, stability-plasticity trade-off

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_pBeWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the avoidable cost that historical protection can impose on learning new tasks. It introduces protection-induced plasticity loss (PIPL), measured through paired adaptation runs that begin from the same model and optimizer state but retain different amounts of protection. Adaptation gains are evaluated jointly with historical and pretrained knowledge retention. The proposed Budgeted Functional Protection (BFP) method maintains a fixed-size subspace of protected parameter directions, periodically updating it using compressed output-sensitivity estimates. The paper combines finite-step analysis with controlled interventions and evaluations on continual retrieval and classification benchmarks.

Critical strengths
  1. The paired diagnostic provides convincing evidence for the adaptation cost of protection. The two branches share their starting model, optimizer state, and adaptation schedule, allowing the effect of constraint release to be studied directly. Additional controls match update lengths, reset optimizer states, and vary the retained directions. The statistical analysis uses ten independent paired runs and adjusts for the primary family of comparisons. These choices make the evidence substantially more informative than a correlation between protection strength and new-task performance.

For growing protection, releasing half the protected directions improves adaptation AUC by 6.75 percentage points while satisfying the stated noninferiority conditions for historical and pretrained performance. This directly supports the existence of avoidable protection costs in the studied setting.

  1. The method, empirical results, and theory form a coherent contribution. Under the common training budget, BFP achieves bidirectional retrieval scores of 42.05/39.58, compared with 40.94/38.52 for Pi-CCA, while reducing pretrained accuracy drop from 3.02 to 1.37 points. The fixed-coordinate controls further show that selecting and refreshing directions matters, rather than protection rank alone explaining the result.

The finite-step analysis also adds useful insight: recovered task signal can improve adaptation, but curvature coupling can alter the ordering of the two trajectories. The counterexamples help explain why releasing more directions need not improve learning at every horizon. The paper clearly distinguishes these loss-based results from the empirical evaluation of AdamW and retrieval performance.

Critical weaknesses
  1. The central diagnostic finding is evaluated in a narrower setting than the benchmark results. The primary paired study uses Sketch as the reserved adaptation domain and compares BFP with its own uncapped growing-protection variant. Adaptation is confined to the final image and text projection adapters. The broader retrieval and classification results support the usefulness of BFP, but they do not establish how widely the measured protection-induced loss occurs across new domains or other protection methods.

The retention criterion also averages over historical domains and pretrained evaluation datasets. It would be useful to see whether those averages conceal appreciably larger losses on individual tasks. Repeating the paired study with another reserved domain would further strengthen the generality of the main finding.

  1. The practical impact of repeated compression remains only partly understood. Appendix A.3 gives an instructive failure case in which a repeatedly discarded direction eventually dominates the full historical average but is still absent from the compressed statistic. This makes clear that refreshing a fixed-size subspace does not necessarily recover the most important directions in the full history.

The random-basis and frozen-basis controls establish that direction choice matters. What remains unclear is how much error repeated truncation introduces in the actual experiments, and whether that error predicts retention losses on particular tasks. A same-rank comparison with the full-history reference subspace would help connect the compression analysis to the observed behavior.

Questions for authors
  1. Does the paired release benefit persist when another VLCL domain is reserved for diagnostic adaptation? For the existing Sketch experiment, how does retention change on each historical domain?
  2. Could the authors quantify accumulated compression error on the diagnostic stream, or compare BFP with a same-rank subspace derived from the full historical statistic?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pBeW (Part 1/3: W1)

Thank you for your careful reading and valuable suggestions on diagnostic generality and compression. We appreciate the time you devoted to the review and address your questions below with additional paired tests, per-task retention results, and a full-history comparison.

The central question is how much adaptation historical protection unnecessarily prevents at a fixed learned state and training budget, subject to explicit retention tolerances. The additional evidence tests this question with a published protection method, a second held-out domain, and adapters throughout both encoders.

Notation. AUC is normalized adaptation area. RR and ZZ are historical-domain and pretrained-dataset macro averages. In diagnostic tables, unmarked AUC/RR/ZZ describe the full-protection branch, in percent. Paired effects Δ\Delta are half-release minus full protection, in percentage points (pp). Unless stated otherwise, results average ten independent parent runs at t=7,K=200t=7,K=200. Bracketed LL values are multiplicity-adjusted one-sided lower bounds. The joint PIPL criterion is L(ΔAUC)>1L(\Delta\mathrm{AUC})>1, L(ΔR)≥−1L(\Delta R)\geq-1, and L(ΔZ)≥−1L(\Delta Z)\geq-1. Other comparisons are descriptive.


W1. The diagnostic is narrow, and average retention may conceal individual-task losses.

Response.{\color{green}\mathbf{Response.}} The expanded paired tests cover an external method, a second held-out domain, and adaptation throughout both encoders:

Diagnostic settingΔAUC[L]\Delta\mathrm{AUC}\\ [L]ΔR[L]\Delta R\\ [L]ΔZ[L]\Delta Z\\ [L]
GPM / Sketch+5.40 [4.36]−0.48 [−0.83]−0.24 [−0.51]
Growing / Flickr30K+4.12 [3.15]−0.39 [−0.77]−0.22 [−0.47]
Growing / all-layer Q/V LoRA / Sketch+5.65 [4.66]−0.41 [−0.75]−0.26 [−0.50]

The first two rows use α/6\alpha/6, and the all-layer extension uses α/3\alpha/3. Each passes the joint macro-average criterion. The per-task Sketch results show how retention changes are distributed across the evaluation tasks:

Evaluation taskGrowing Δ\DeltaBFP Δ\DeltaGPM Δ\Delta
Historical: Flickr30K−0.24−0.12−0.42
Historical: COCO−0.31−0.18−0.18
Historical: Pets−0.18−0.26−0.40
Historical: Lexica−1.18−0.42−1.25
Historical: Simpsons+0.04−0.08−0.20
Historical: WikiArt−0.56−0.24−0.58
Historical: Kream+0.19+0.11−0.33
Historical macro average−0.32−0.17−0.48
Pretrained: ImageNet-1K−0.51−0.22−0.55
Pretrained: CIFAR-100−0.12−0.09−0.18
Pretrained: Flowers102+0.10+0.04+0.01
Pretrained: DTD−0.43−0.19−0.45
Pretrained: Food101+0.09−0.10−0.17
Pretrained: Cars+0.03+0.02−0.10
Pretrained macro average−0.14−0.09−0.24

Lexica loses more than 1 pp under Growing and GPM release. BFP keeps every historical-domain mean change within 0.42 pp. Eq. (10) measures recoverable adaptation subject to macro-average retention tolerances. The expanded results support that criterion and identify the individual tasks that require closer protection.

AuthorsOpenReview ↗

Response to Reviewer pBeW (Part 2/3: W2, Q1)

W2. What practical error does repeated truncation introduce, and does it predict task-specific retention?

Response.{\color{green}\mathbf{Response.}} The reference stores every realized sketch along the same BFP trajectory. It forms Ctall=t−1∑j=1tHˉjHˉj⊤C_t^{\mathrm{all}}=t^{-1}\sum_{j=1}^{t}\bar H_j\bar H_j^\top and selects its top-16 space. Sharing the trajectory isolates the effect of recursive compression:

Boundary ttRelative spectral errorLost traceProjector distance to full-history top-16Excess worst-case residual sensitivity
10.27 ± 0.040.40 ± 0.030.00 ± 0.000.0000 ± 0.0000
40.38 ± 0.050.67 ± 0.040.58 ± 0.090.0027 ± 0.0006
70.44 ± 0.060.76 ± 0.040.74 ± 0.080.0051 ± 0.0010

At t=1t=1, both methods select the same top-16 space, although rank truncation already loses spectral mass. At t=7t=7, replacing only the basis at the same checkpoint and optimizer state gives:

Rank-16 basisAUCRRZZResident basisStored sketches for reference construction
Recursive BFP57.8650.9863.342.25 MiB—
Full-history top-1657.6451.2463.432.25 MiB63.00 MiB

Full-history selection gains 0.26/0.09 pp in R/ZR/Z at −0.22 pp AUC. Its domain-level differences are:

Historical domainResidual sensitivity reduced by referenceRetention change, reference − BFP
Flickr30K+0.008+0.20
COCO+0.012+0.31
Pets+0.017+0.45
Lexica+0.021+0.50
Simpsons−0.004−0.08
WikiArt+0.016+0.37
Kream+0.001+0.07

Sensitivity quantities are dimensionless, and retention changes are pp. Their signs agree across domains, linking protection differences to retention descriptively. The same-rank intervention directly measures the practical effect of basis selection. BFP optimizes the available compressed statistic in Eq. (6), and the full-history comparison quantifies the additional retention obtainable from storing the historical sketches.


Q1. Does the benefit persist on another reserved domain, and what happens to each historical task?

Response.{\color{green}\mathbf{Response.}} Yes. With Flickr30K reserved and K=200K=200:

ParentAUC full → halfΔAUC\Delta\mathrm{AUC}ΔR\Delta RΔZ\Delta Z
Growing71.80 → 75.92+4.12−0.39−0.22
Growing: adjusted lower bounds—3.15−0.77−0.47
BFP77.25 → 77.88+0.63−0.21−0.12

Growing passes the joint criterion. BFP shows a smaller descriptive release effect. For your requested original Sketch experiment, all seven historical-domain changes are below. Brackets are lower bounds corrected over 13 task endpoints within each method, including the six pretrained datasets:

Historical taskGrowing Δ[L13]\Delta\\ [L_{13}]BFP Δ[L13]\Delta\\ [L_{13}]
Flickr30K−0.24 [−0.60]−0.12 [−0.55]
COCO−0.31 [−0.74]−0.18 [−0.69]
Pets−0.18 [−0.69]−0.26 [−0.84]
Lexica−1.18 [−1.76]−0.42 [−0.78]
Simpsons+0.04 [−0.32]−0.08 [−0.51]
WikiArt−0.56 [−0.99]−0.24 [−0.75]
Kream+0.19 [−0.32]+0.11 [−0.47]

BFP satisfies all historical per-domain −1 pp bounds in this Sketch setting. Growing's Lexica result falls below that threshold. The Flickr30K setting provides another example of the distinction between aggregate and per-task retention:

Flickr30K held out: Simpsons retentionΔ\DeltaL13L_{13}
Growing−1.24−1.75
BFP−0.46−1.04

Both Simpsons lower bounds fall below the per-task −1 pp threshold. The aggregate test and these domain-level results together show where release recovers adaptation within the stated macro-average tolerance and where stronger task-specific protection is needed.

AuthorsOpenReview ↗

Response to Reviewer pBeW (Part 3/3: Q2)

Q2. Can you quantify accumulated compression error and compare with a same-rank full-history subspace?

Response.{\color{green}\mathbf{Response.}} Yes. Let Dt=Ctall−C~tD_t=C_t^{\mathrm{all}}-\widetilde C_t, and let PB,PrefP_B,P_{\mathrm{ref}} project onto the respective free spaces. The relative error is ∣Dt∣2/∣Ctall∣2\\|D_t\\|_2/\\|C_t^{\mathrm{all}}\\|_2. Projector distance is ∣PB−Pref∣2\\|P_B-P_{\mathrm{ref}}\\|_2. All quantities below average ten parent trajectories:

ttRelative spectral errortr(Dt)\mathrm{tr}(D_t)Projector distanceFull-history sketch storage
10.27 ± 0.040.40 ± 0.030.00 ± 0.009.00 MiB
40.38 ± 0.050.67 ± 0.040.58 ± 0.0936.00 MiB
70.44 ± 0.060.76 ± 0.040.74 ± 0.0863.00 MiB
Same-checkpoint, rank-16 adaptationAUCRRZZ
BFP basis57.8650.9863.34
Full-history reference basis57.6451.2463.43
Reference − BFP−0.22+0.26+0.09

Both deploy a 2.25 MiB basis. Constructing the reference requires 63 MiB of historical sketch columns and an additional thin SVD. The reference is computed exactly from those saved boundary sketches. Replacing the basis at the same checkpoint measures the downstream effect of compression while keeping rank, model state, and optimizer state fixed. The resulting comparison connects the measured spectral error to adaptation and retention performance.

Reviewer_7sGbOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the stability-plasticity dilemma in continual learning for multimodal foundation models, particularly vision-language models (VLMs). The authors identify a phenomenon termed the stability trap, where historical knowledge protection mechanisms (such as orthogonal gradient descent or subspace protection) unnecessarily suppress parameter updates needed for new-task learning.

Critical strengths
  • The paper addresses a fundamental bottleneck in continual learning, over-protection causing reduced plasticity which is highly relevant to the community.
  • The empirical evaluation is thorough, spanning multiple backbones, diverse benchmarks, boundary-density stress tests, and supported by rigorous mathematical proofs in the appendix.
  • BFP effectively bounds memory and storage overhead by maintaining a fixed rank budget using compressed output sensitivity, avoiding the linear growth seen in traditional growing projection methods.
Critical weaknesses
  • The paper fixes the protection rank at r=16r=16 and probe budget at m=64m=64, but lacks a systematic sensitivity analysis or ablation study regarding how varying these budgets affects the adaptation-retention trade-off.
  • While storage space is well-controlled, frequent SVD and projection operations at task boundaries introduce computational overhead. A more explicit discussion or comparison of the wall-clock training time relative to baselines would strengthen the practical claims.
  • Some gradient-based CL (classification) works should also be discussed, such as C-Flat/C-Flat ++[1,2]. This is a mature, well-executed paper that combines deep theoretical analysis with practical, scalable algorithmic design for VLM continual learning. Addressing the hyperparameter sensitivity and computational overhead discussion during the rebuttal would further solidify its contribution.

[1] Making Continual Learning vi C-Flat, NeurIPS 24.

[2] C-Flat Turbo: A Faster Path to Continual Learning, CVPR 26.

Questions for authors
  • How does BFP perform under longer or more diverse task streams if the fixed rank r=16r=16 is kept constant? Is there a heuristic for tuning rr dynamically based on task stream complexity?
  • In Equation (5), how sensitive is the estimated covariance/sketch to the random sign vectors aia_i when the batch size or task data distribution is skewed?
  • In the optimizer-reset and norm-replayed controls (Section 4.5 and Appendix C.4), how do momentum and adaptive learning rate terms (AdamW moments) interact with constraint releases to dictate the short-term adaptation surge?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7sGb (Part 1/2: W1, W2, W3)

Thank you for your thoughtful comments and the time you spent reviewing our work. We address your suggestions below with budget sweeps, end-to-end timing, and analyses of sketch estimation and optimizer behavior.

The central question is how much adaptation historical protection unnecessarily prevents at a fixed learned state and training budget, subject to explicit retention tolerances. The additional evidence tests this question with a published protection method, a second held-out domain, and adapters throughout both encoders.

Notation. AUC is normalized adaptation area. RR and ZZ are historical-domain and pretrained-dataset macro averages. In diagnostic tables, unmarked AUC/RR/ZZ describe the full-protection branch, in percent. Paired effects Δ\Delta are half-release minus full protection, in percentage points (pp). Unless stated otherwise, results average ten independent parent runs at t=7,K=200t=7,K=200. Bracketed LL values are multiplicity-adjusted one-sided lower bounds. The joint PIPL criterion is L(ΔAUC)>1L(\Delta\mathrm{AUC})>1, L(ΔR)≥−1L(\Delta R)\geq-1, and L(ΔZ)≥−1L(\Delta Z)\geq-1. Other comparisons are descriptive.


W1. Sensitivity to protection rank rr and probe budget mm is missing.

Response.{\color{green}\mathbf{Response.}} The separate sweeps below show how each budget affects adaptation and retention. AUC/RR/ZZ are mean ± SD over ten parents. Release effects are descriptive means.

rrmmAUCRRZZΔAUC\Delta\mathrm{AUC}
46459.28 ± 1.5749.42 ± 0.8262.77 ± 0.44+0.30
86458.66 ± 1.5450.36 ± 0.8063.10 ± 0.42+0.46
166457.86 ± 1.5550.98 ± 0.7863.34 ± 0.41+0.76
326456.84 ± 1.5951.18 ± 0.8063.42 ± 0.43+1.17
646455.69 ± 1.6451.30 ± 0.8263.44 ± 0.45+1.94
161658.09 ± 1.6350.63 ± 0.8463.12 ± 0.47+0.70
1625657.70 ± 1.5151.09 ± 0.7763.42 ± 0.40+0.78

Larger rr improves retention at an adaptation cost, with diminishing retention gains beyond r=16r=16. Increasing mm from 64 to 256 improves R/ZR/Z by 0.11/0.08 pp. These results support r=16,m=64r=16,m=64 as an economical operating point for the evaluated stream and show how the trade-off changes with either budget.


W2. Projection and boundary SVD overhead need end-to-end wall-clock measurements.

Response.{\color{green}\mathbf{Response.}} Total training time includes data loading, main updates, boundary probes, SVD, and each comparator's native auxiliary computation. Measurements use three runs on an A100 80GB with matched precision and no feature cache.

MethodStream / updatesTraining minutesProbe + SVD secondsPeak GPU memory
Unprotected8 stages / 8,00047.2 ± 0.7010.4 GiB
BFP8 stages / 8,00048.5 ± 0.715.610.8 GiB
Growing8 stages / 8,00049.8 ± 0.729.811.0 GiB
Pi-CCA8 stages / 8,00053.2 ± 0.7—12.7 GiB
ZSCL8 stages / 8,00060.8 ± 0.7—15.8 GiB
Unprotected32 stages / 8,00048.0 ± 0.7010.4 GiB
BFP32 stages / 8,00049.7 ± 0.762.110.8 GiB

BFP's total overhead over unprotected adaptation is 2.8% for eight stages and 3.5% for 32 stages, with fixed 2.25 MiB protection state. The 32-stage setting splits and interleaves the same eight domains. Evaluation and extra diagnostic branches are timed separately. These end-to-end measurements complement Table 14's 0.044 ms projection measurement and include the full cost of boundary probing and refresh.


W3. C-Flat and C-Flat Turbo are relevant gradient-based continual-learning work.

Response.{\color{green}\mathbf{Response.}} Thank you for pointing out these connections. C-Flat [1] combines zeroth- and first-order flatness objectives to improve continual-learning optimization, including projection-based learners. C-Flat Turbo [2] reduces redundant gradient computation in flatness-aware training. These works address how the loss landscape and optimizer affect continual learning. Our diagnostic provides a complementary perspective by measuring the adaptation cost of historical constraints at a fixed state. BFP reallocates those constraints under a fixed sensitivity budget. The curvature terms in Theorem 1 connect the two perspectives by showing how local geometry affects the benefit of releasing protection.

[1] Make Continual Learning Stronger via C-Flat.

[2] A Faster Path to Continual Learning.

AuthorsOpenReview ↗

Response to Reviewer 7sGb (Part 2/2: Q1, Q2, Q3)

Q1. Does fixed-rank BFP work on longer streams, and can rr adapt to stream complexity?

Response.{\color{green}\mathbf{Response.}} With r=16,m=64r=16,m=64 fixed, Tables 9 and 7(b) give:

Retrieval streamTotal updatesPi-CCA I2T / T2IBFP I2T / T2I
8 domains8,00040.94 / 38.5242.05 / 39.58
32 interleaved stages8,00039.71 / 37.3441.46 / 39.06
CIFAR-100 streamTotal updatesBFP LastBFP Avg.
10 stages10,00076.0383.17
20 stages10,00074.9682.39
50 stages10,00072.9481.51

The 32-stage test increases boundary density by splitting and interleaving the same eight domains. BFP's retrieval drop is 0.59/0.52 pp versus Pi-CCA's 1.23/1.18 pp. Its protection state stays fixed across both stream lengths:

StreamUnprotected minutesBFP minutesBFP protection state
8 stages47.2 ± 0.748.5 ± 0.72.25 MiB
32 stages48.0 ± 0.749.7 ± 0.72.25 MiB

The fixed-budget results demonstrate stable performance as boundary density increases. For your question about dynamic tuning, a possible heuristic is to choose the smallest rr that meets a historical-validation retention target under a memory cap. The current sketch's spectral tail can guide candidate increases. Retention validation remains important because the spectrum reflects the stored statistic after truncation (Appendix A.3.2). This is a proposed selection rule, and all results above use fixed rr.


Q2. How sensitive is the sketch to random signs, sample size, and skewed data?

Response.{\color{green}\mathbf{Response.}} For fixed data, Ea[J⊤aa⊤J]=J⊤J\mathbb E_a[J^\top aa^\top J]=J^\top J makes the unnormalized sign estimator unbiased. Unit-trace normalization produces a scale-normalized sensitivity statistic, and sampling determines the distribution it represents (Appendix A.2). At the fixed t=7t=7 checkpoint, 20 offline repetitions separate sign and sample variability:

Distribution / varied factormmRelative spectral errorTop-16 projector distanceError against balanced reference
Balanced / signs only160.50 ± 0.060.94 ± 0.06—
Balanced / signs only640.31 ± 0.040.68 ± 0.08—
Balanced / signs only2560.18 ± 0.030.37 ± 0.06—
Balanced / samples only640.34 ± 0.050.73 ± 0.07—
80:20 skew / samples only640.36 ± 0.050.76 ± 0.080.46 ± 0.05

The first two error columns use a 1,024-probe reference from the same distribution. The final column measures mismatch against the balanced reference. Distances use the spectral norm. Sign-only repeats fix samples, and sample-only repeats use signs fixed by sample ID. More probes reduce estimation error. Comparing the two references separates sampling variability from the change in target caused by skew. The downstream ten-parent sweep shows the corresponding performance trade-off:

Probe budget, r=16r=16AUCRRZZ
m=16m=1658.09 ± 1.6350.63 ± 0.8463.12 ± 0.47
m=64m=6457.86 ± 1.5550.98 ± 0.7863.34 ± 0.41
m=256m=25657.70 ± 1.5151.09 ± 0.7763.42 ± 0.40

Here mm counts boundary probe pairs, separately from the optimizer batch size of 256. The offline analysis measures variability at a fixed checkpoint, while the performance table averages ten independent parent runs.


Q3. How do AdamW moments interact with constraint release, norm replay, and optimizer reset?

Response.{\color{green}\mathbf{Response.}} AdamW moments are updated from raw gradients. Eq. (8) projects the complete preconditioned proposal, including weight decay:

\qquad \theta_s=\theta_{s-1}-\eta(I-UU^\top)u_s.$$ The division is elementwise. At the first paired step, identical states give the same $u_s$. Releasing directions changes the admissible displacement. Gradients and moments then evolve along each branch's trajectory. Norm replay matches displacement lengths while preserving each branch's update direction. Resetting **both** branches removes inherited moments and counters: | Table 13 control | Growing $\Delta\mathrm{AUC}$ | BFP $\Delta\mathrm{AUC}$ | | --- | --- | --- | | Norm-replayed half release | +5.18 ± 0.92 | +0.57 ± 0.35 | | Reset-half minus reset-full | +6.52 ± 1.09 | +0.73 ± 0.46 | Growing's recovery persists after matching step lengths and after resetting inherited optimizer states. This supports constraint release as a source of the adaptation gain under AdamW. Momentum and preconditioning continue to evolve in both branches, so the controls measure the release effect within the complete optimizer dynamics.

hank you for the detailed rebuttal and additional experimental evidence. The new paired tests, per-task retention results, and full-history comparison adequately address my concerns regarding diagnostic generality and compression. In particular, the evaluation with a published protection method, an additional held-out domain, and adapters across both encoders strengthens the generality and credibility of the findings.

I appreciate the authors' efforts in clarifying these points. My main concerns have been satisfactorily resolved, and I have decided to raise my score.

Thank you for your careful review and constructive feedback. We are glad that our response has addressed your concerns, and we appreciate your positive reassessment of our work.

Reviewer_Q8p3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how historical protection in continual learning of vision–language models can suppress new-task learning, which it calls the stability trap. It proposes a paired diagnostic that measures how much adaptation is recovered by releasing part of the protection. It also proposes BFP, which protects a fixed number of output-sensitive parameter directions and refreshes them at each task boundary. Experiments with CLIP on continual retrieval and classification show gains over recent methods, and less recoverable adaptation loss than an uncapped protection baseline.

Critical strengths
  1. Well-controlled diagnostic of protection cost. Branches start from identical checkpoints and include norm-replay, orientation and optimizer-reset controls over ten paired runs. This isolates the effect of releasing protection, and gives a reusable way to measure plasticity cost directly.
  2. Simple method with bounded cost. BFP keeps a fixed-size protection state with low overhead (Table 14). It also improves over recent baselines on retrieval and classification (Tables 1 and 2).
Critical weaknesses
  1. Limited novelty. BFP combines OGD-style output-gradient protection [1] with a rank-capped streaming SVD sketch. This closely resembles thresholded subspace bases in GPM [2] and Frequent-Directions accumulation in CoSO [3]. The observation that accumulated protection reduces plasticity has also motivated prior work such as TRGP [4] and FS-DGPM [5].
  2. The evidence for the stability trap relies mainly on an uncapped internal variant. Growing protection keeps every direction, so over-protection is expected, and BFP’s smaller release gains may partly reflect protecting fewer directions. At the same rank, a frozen first-task basis gives higher adaptation but lower retention (Table 11), suggesting BFP offers a better trade-off rather than dominance. The diagnostic is also not applied to existing protection methods.
  3. Scope is narrower than the title suggests. Only the final projection layers of CLIP are adapted, with about 18K trainable parameters. It remains unclear whether the findings hold for deeper adapters, larger models or generative multimodal models.

References

[1] Orthogonal Gradient Descent for Continual Learning. AISTATS 2020.

[2] Gradient Projection Memory for Continual Learning. ICLR 2021.

[3] Continuous Subspace Optimization for Continual Learning. NeurIPS 2025.

[4] TRGP: Trust Region Gradient Projection for Continual Learning. ICLR 2022.

[5] Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning. NeurIPS 2021.

Questions for authors
  1. How do BFP and growing protection compare across a sweep of the protection rank r, for example on an adaptation–retention frontier?
  2. Does the paired diagnostic reveal recoverable plasticity loss for existing protection methods, such as GPM with an energy threshold, KeepLoRA or NuSA-CL?
  3. Does BFP’s advantage hold when adapting deeper layers of the encoders rather than only the final projections?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Q8p3 (Part 1/2: W1, W2)

We appreciate your time and your detailed comments on novelty, fair comparisons, and adaptation scope. Below, we clarify the contribution and address your questions with comparisons at matched rank and results from all-layer adaptation.

The central question is how much adaptation historical protection unnecessarily prevents at a fixed learned state and training budget, subject to explicit retention tolerances. The additional evidence tests this question with a published protection method, a second held-out domain, and adapters throughout both encoders.

Notation. AUC is normalized adaptation area. RR and ZZ are historical-domain and pretrained-dataset macro averages. In diagnostic tables, unmarked AUC/RR/ZZ describe the full-protection branch, in percent. Paired effects Δ\Delta are half-release minus full protection, in percentage points (pp). Unless stated otherwise, results average ten independent parent runs at t=7,K=200t=7,K=200. Bracketed LL values are multiplicity-adjusted one-sided lower bounds. The joint PIPL criterion is L(ΔAUC)>1L(\Delta\mathrm{AUC})>1, L(ΔR)≥−1L(\Delta R)\geq-1, and L(ΔZ)≥−1L(\Delta Z)\geq-1. Other comparisons are descriptive.


W1. Novelty relative to OGD, GPM, CoSO, TRGP, and FS-DGPM is limited.

Response.{\color{green}\mathbf{Response.}} Our contribution is a controlled measurement of avoidable protection cost. Both branches start from the same learned state and receive the same data and training budget. Releasing protection measures how much adaptation can be recovered while historical and pretrained retention satisfy explicit tolerances. This turns the stability–plasticity tension into a measurable property of the protection policy at a given checkpoint.

The works you cite provide the relevant foundations. OGD [1] protects output-gradient directions, and GPM [2] accumulates activation directions selected by an energy threshold. TRGP [3] uses task-related trust regions for transfer. FS-DGPM [4] learns soft subspace importance with flatness-aware optimization. Our paired intervention gives these protection decisions a common empirical test: how much progress can be recovered at the same checkpoint while satisfying the retention criteria?

BFP follows the resulting insight: historical protection should be reallocated under a global budget. Its spectral objective minimizes the worst-case local output sensitivity left unprotected in the available statistic. CoSO [5] uses Frequent Directions for the current task's gradient component and appends selected directions to historical memory (Eq. 13). BFP refreshes the protected space under a fixed global rank budget, allowing the allocation to change as the stream evolves. Theorem 1 explains how released task signal and curvature determine recovery. Together, the diagnostic and analysis motivate a protection policy whose adaptation cost and retention benefit can be measured directly.

[1] Orthogonal Gradient Descent for Continual Learning.

[2] Gradient Projection Memory for Continual Learning.

[3] TRGP: Trust Region Gradient Projection for Continual Learning.

[4] Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning.

[5] Continuous Subspace Optimization for Continual Learning.


W2. Smaller release gains may reflect rank alone. Frozen protection suggests a trade-off, and the diagnostic needs an existing-method comparison.

Response.{\color{green}\mathbf{Response.}} We agree that the adaptation–retention trade-off is the relevant comparison. Table 11 holds the parameter coordinates and protection rank fixed, allowing you to assess the effect of direction selection:

ProtectionRankAUCRRZZ
Unprotected060.2847.7162.39
Fixed Haar1660.1347.8862.44
Frozen first-task1658.9649.1462.91
Refreshed BFP1657.8650.9863.34

At equal rank, BFP gains 1.84/0.43 pp in historical/pretrained retention over the frozen basis at a 1.10 pp adaptation cost. Direction selection and refresh therefore affect the operating point. Moreover, the external-method test gives:

GPM / SketchΔAUC[L]\Delta\mathrm{AUC}\\ [L]ΔR[L]\Delta R\\ [L]ΔZ[L]\Delta Z\\ [L]
Threshold 0.95, K=200K=200+5.40 [4.36]−0.48 [−0.83]−0.24 [−0.51]

These bounds use the six-test extension family and pass the joint criterion. Growing protects 438–447 directions, about 2.4% of d=18,432d=18{,}432. The release result shows that a small fraction of parameter space can still contain directions important for new learning. GPM confirms that recoverable obstruction also arises under energy-threshold protection.

AuthorsOpenReview ↗

Response to Reviewer Q8p3 (Part 2/2: W3, Q1, Q2, Q3)

W3. The diagnostic scope excludes deeper adapters, larger models, and generative multimodal models.

Response.{\color{green}\mathbf{Response.}} The diagnostic now covers rank-8 Q/V LoRA in every Transformer layer of both encoders plus the original projection adapters, with d=509,952d=509{,}952:

All-layer methodAUCRRZZΔAUC\Delta\mathrm{AUC}ΔR\Delta RΔZ\Delta Z
Unprotected64.0048.9062.80———
Growing54.1052.2463.70+5.65−0.41−0.26
Growing: adjusted lower bounds———4.66−0.75−0.50
BFP, r=16r=1661.2051.7564.00+0.73−0.16−0.08
BFP, r=64r=6460.4852.1064.15+1.12−0.24−0.12

Growing passes the joint criterion with α/3\alpha/3 correction, establishing PIPL with adaptation throughout both encoders. Table 9 provides complementary benchmark evidence across CLIP backbones. Together, these evaluations extend the study across adaptation depth and backbone choice, with direct paired evidence in contrastive vision–language learning.


Q1. How do BFP and Growing compare across protection ranks on the adaptation–retention frontier?

Response.{\color{green}\mathbf{Response.}} The sweep keeps m=64m=64 and the original parameterization. Each configuration trains its own parent. AUC/RR/ZZ are full-protection means, and release effects are measured within that parent.

MethodRankAUCRRZZΔAUC\Delta\mathrm{AUC}State, MiB
Unprotected060.2847.7162.39—0
BFP459.2849.4262.77+0.300.5625
BFP858.6650.3663.10+0.461.125
BFP1657.8650.9863.34+0.762.25
BFP3256.8451.1863.42+1.174.50
BFP6455.6951.3063.44+1.949.00
Growing438–44750.4251.3663.12+6.7562.31

Increasing rr from 16 to 64 gains 0.32/0.10 pp in R/ZR/Z at a cost of 2.17 pp AUC. The sweep quantifies the diminishing retention returns from a larger budget. Applying top-rr truncation to Growing's recurrence gives BFP, so these rows also provide the corresponding capped-Growing comparison. The controls below isolate direction selection at matched rank:

Methodr=8r=8: AUC / RR / ZZr=16r=16: AUC / RR / ZZr=32r=32: AUC / RR / ZZ
Frozen first-task59.37 / 48.56 / 62.7258.96 / 49.14 / 62.9158.23 / 49.54 / 63.02
Fixed Haar60.20 / 47.80 / 62.4060.13 / 47.88 / 62.4459.84 / 48.05 / 62.50
Refreshed BFP58.66 / 50.36 / 63.1057.86 / 50.98 / 63.3456.84 / 51.18 / 63.42

Frozen/Haar at r=8,32r=8,32 use three runs. All other rows use ten. The table describes the observed adaptation–retention frontier for each protection policy.


Q2. Does the paired diagnostic detect recoverable loss in GPM, KeepLoRA, or NuSA-CL?

Response.{\color{green}\mathbf{Response.}} Yes, in the VLM adaptation of GPM at energy threshold 0.95:

GPM on SketchAUC full → releasedΔAUC\Delta\mathrm{AUC}ΔR\Delta RΔZ\Delta Z
Half release52.40 → 57.80+5.40−0.48−0.24
Adjusted lower bounds—4.36−0.83−0.51
Norm-replayed half release52.40 → 56.21+3.81−0.35−0.19

The main release passes all three conditions. Recovery also persists after norm replay matches displacement lengths. Image/text ranks are approximately 410/288 before and 205/144 after fixed random half deletion. Both branches share the same GPM checkpoint and optimizer. This directly establishes recoverable protection cost in the GPM implementation and shows that the result extends beyond the internal Growing control.


Q3. Does BFP's advantage persist when adapting deeper encoder layers?

Response.{\color{green}\mathbf{Response.}} Yes, as an adaptation–retention trade-off. With all-layer Q/V LoRA and the original projection adapters:

MethodAUCRRZZTraining minutesPeak GPU memory
Unprotected64.0048.9062.8073.1 ± 0.729.5 GiB
Growing54.1052.2463.7079.8 ± 0.733.8 GiB
BFP, r=16r=1661.2051.7564.0075.5 ± 0.730.1 GiB
BFP, r=64r=6460.4852.1064.1577.0 ± 0.731.9 GiB

Against Growing, BFP16 gains 7.10 pp AUC and 0.30 pp pretrained retention, with 0.49 pp lower historical retention. Against no protection, it gains 2.85/1.20 pp in R/ZR/Z at a 2.80 pp AUC cost. Timing covers 7,000 parent updates and all boundary operations over three A100 runs. Accuracy uses ten parent runs.

Thank you for the response and the additional experiments. The GPM test, the all-layer experiment and the rank sweep answer my questions directly. However, several important concerns remain.

  1. The central finding is sensitive to the evaluation choices. In the additional results provided in the responses, the joint criterion for growing protection holds at K=200 but fails at K=400 in both held-out settings, and some individual domains lose more than 1 pp, such as Lexica under growing protection and GPM. The claim that protection cost is avoidable therefore holds only for a short horizon and macro-averaged retention.

  2. Limited novelty. The paper itself notes that FS-DGPM and space decoupling already relax protection to recover plasticity (Section 1), and the response acknowledges that OGD, GPM, TRGP and FS-DGPM "provide the relevant foundations". BFP remains a combination of output-gradient protection, rank capping and streaming SVD.

  3. BFP's design is not clearly motivated by the diagnostic. Under growing protection, release gains correlate strongly with gradient overlap but hardly with stored output sensitivity (Figure 5, Table 10), which is the criterion BFP uses to select directions. The rank sweep also shows that release gains grow steadily with rank. This suggests that BFP reduces obstruction largely by protecting fewer directions.

  4. Poor readability. The paper is very difficult to read. In my view, it needs a thorough and careful revision and polishing before it can be considered for publication.

  5. Scope is overstated. All evidence is on contrastive CLIP models. Apart from concatenating the image and text embeddings in the sensitivity statistic, nothing in the method is specific to multimodal learning, so the title and framing claim more than is shown.

I therefore keep my score of 2.

AuthorsOpenReview ↗

Follow-up response to Reviewer Q8p3

Thank you for your careful review and feedback on the additional experiments.

W1. Dependence on horizon and macro-average retention.

Response.{\color{green}\mathbf{Response.}} At K=200K=200, both held-out settings show adaptation gains and pass both retention tests (Eq. (10)).

Growing, held-out domainΔAUC [L]ΔR [L]ΔZ [L]
Sketch+6.75 [5.70]−0.32 [−0.68]−0.14 [−0.41]
Flickr30K+4.12 [3.15]−0.39 [−0.77]−0.22 [−0.47]

Effects are half−full in pp. Brackets give adjusted lower bounds. Both settings meet the joint criterion: L(ΔAUC)>1L(\Delta\mathrm{AUC})>1 and L(ΔR),L(ΔZ)≥−1L(\Delta R),L(\Delta Z)\ge-1.

Here KK counts diagnostic updates after continual training. Longer release can increase both adaptation gains and retention costs, as the K=400K=400 means show. The K=200K=200 result establishes recoverable learning within the stated retention margins. The earlier K=400K=400 bounds used assumed paired SDs and are conditional calculations.

R/ZR/Z are historical/pretrained macro averages. Lexica's 1.18/1.25 pp losses under Growing/GPM motivate per-domain criteria when you require every domain to be preserved.


W2. Novelty beyond existing projection methods.

Response.{\color{green}\mathbf{Response.}} Our contribution is to make avoidable protection cost measurable. PIPL tests adaptation gains and retention jointly by changing protection from identical model and optimizer states.

BFP allocates a fixed protection budget to directions whose movement most affects stored output behavior. FS-DGPM [1] learns protection strengths and refreshes its basis. Space decoupling [2] treats stability and plasticity spaces differently. The direction-level evidence below connects this diagnostic to BFP’s protection objective.

[1] Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning.

[2] Rethinking Gradient Projection Continual Learning: Stability / Plasticity Feature Space Decoupling.


W3. Why sensitivity, and is lower rank sufficient?

Response.{\color{green}\mathbf{Response.}} Gradient overlap indicates how much learning release can recover. Output sensitivity helps identify what needs protection. Reanalysis of Figure 5 links sensitivity to retention damage:

ParentOverlap–gainSensitivity–gainSensitivity–damage
Growing0.8830.0470.604
BFP0.779−0.3480.505

Pooled Pearson correlations using log⁡(1+λi)\log(1+\lambda_i) for sensitivity. Damage is the larger nonnegative drop in historical or pretrained macro averages (Appendix B.6). Sensitivity–damage correlations are positive in all ten seeds for each parent. Eq. (6) formalizes this allocation by minimizing free-space sensitivity under the compressed statistic.

Direction choice matters at fixed rank. Table 13 holds Growing checkpoints, retained rank, and update norms fixed:

Retained directionsΔAUCΔRΔZ
Leading sensitivity half+5.18−0.24−0.10
Random half of stored basis+6.34−0.71−0.27
Same-rank Haar basis+7.41−1.36−0.54

Paired effects in pp at K=200K=200. Selecting sensitive directions preserves more prior performance while recovering 5.18 pp of adaptation. Direction selection therefore matters alongside rank.


W4. Readability.

Response.{\color{green}\mathbf{Response.}} Thank you for the candid feedback. In plain terms, protection can block useful new learning. PIPL measures the learning recovered by release and checks its retention cost. BFP limits protection and assigns it to output-sensitive directions.


W5. Scope and multimodal relevance.

Response.{\color{green}\mathbf{Response.}} The empirical scope is continual adaptation of contrastive vision–language models. You are right that the projection principle is general. In CLIP, both embeddings determine cross-modal similarity, so Eq. (4) measures output changes in both encoders. This connects protection to the retrieval and zero-shot capabilities being evaluated.


Thanks again for your time. If these clarifications help address your concerns, we would be grateful for your reconsideration.

Reviewer_EgTzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how much of the plasticity lost during continual learning of vision-language models is caused by the historical-protection mechanism itself, as opposed to the data or the budget. It formalizes this as protection-induced plasticity loss (PIPL): adaptation that can be recovered by relaxing historical constraints at a fixed learned state and fixed budget, subject to retention tolerances. The measurement instrument is a paired intervention: after task t, clone the model, optimizer and randomness; one branch keeps the full protection basis U_t, the other keeps its leading half; both train on the same held-out diagnostic task (Sketch, never seen by the parent) for K=200 updates, and the AUC difference measures the recoverable loss. A preliminary study shows future-task gradients being intercepted far above rank expectation (median 8.55% at t=1 rising to 39.90% at t=6, against a rank-only expectation of 0.34% to 2.06%). The method contribution is Budgeted Functional Protection (BFP): a fixed budget of r=16 protected directions estimated from output-sensitivity sketches (Rademacher probes, one backward pass each, m=64 per boundary), refreshed by thin SVD, with AdamW proposals projected onto the orthogonal complement. A finite-step theorem characterizes recoverable loss under a quadratic, decoupled-curvature model. Empirically, BFP beats Pi-CCA on VLCL retrieval by 1.11/1.06 R@1 while cutting the pretrained drop from 3.02 to 1.37 points, leads MTIL/X-TAIL/CIFAR-100 settings by 0.9-3.1 points, and stores 2.25 MiB of protection state versus 62.31 MiB for uncapped growing protection.

Critical strengths

he question is well posed and, to my knowledge, new in this form. Separating "protection costs plasticity" from "plasticity decays anyway" requires exactly the kind of controlled intervention the authors build, and the paired protocol (same checkpoint, same minibatches, nested half-release) is the right design for it.

The experimental discipline is well above the norm for this area: ten seeds, multiplicity-adjusted one-sided bounds, pre-registered 1-point margins, and a battery of mechanism controls. I found the controls in Figure 4 / Table 13 persuasive: the release gain for growing protection survives norm replay (+5.18 pp) and optimizer reset (+6.52 pp), so the effect is not step length or inherited optimizer state; random deletion and Haar replacement gain more but at a retention cost, which is the expected ordering.

BFP itself is simple and cheap — O(dr) state, 0.044 ms per projection, ~2 s per boundary refresh — and the gains line up on the right axis: the largest effect is on pretrained drop (3.02 -> 1.37), with consistent but modest accuracy gains elsewhere.

The direction-level analysis in Figure 5 is the most interesting single result in the paper: release gains correlate strongly with gradient overlap (r = 0.883) and barely at all with stored sensitivity (r = 0.047), which says the obstruction is concentrated in directions that matter for the new task but contribute little to retention.

Finally, the authors disclose the failure modes of their own machinery: the self-normalization bias of the sketch estimator (App. A.2), the discarded-direction counterexample (App. A.3.2), and curvature-coupling counterexamples to the clean theorem (App. D.11). I checked Proposition 1, the update ordering in A.1.2, and the threshold arithmetic in Eq. 10 (the [0,1] scores make the 0.01 margin a 1-point margin); the pieces are consistent.

Critical weaknesses

Three issues limit how far the paper's claims extend beyond its own setting; the rest I class as lesser concerns.

The diagnostic is only ever run on the authors' own two constructs: uncapped "growing protection" and BFP. The title-level claim is that historical protection in continual learning traps plasticity, but no published method — KeepLoRA, NuSA-CL, Pi-CCA — is put through the paired-release test. Uncapped growing protection is close to a worst-case construction; real systems cap memory. As it stands, the existence of the trap in deployed methods is an inference, not a measurement. The adaptation scope is narrow: two final projection layers of CLIP with rank-8 adapters, d ~ 18K trainable parameters. Whether PIPL is even measurable at full-model PEFT scale (LoRA on all layers, or full fine-tuning) is unaddressed, and how r=16 should scale is not discussed. I could not find any code or data release statement anywhere in the paper. The paired protocol depends on exact cloning, replay and seed handling; without code, the central experiments are effectively unreproducible. Lesser concerns, listed briefly: absolute gains are modest (+1.11/+1.06 R@1; +0.9 to +3.1 Last), though backed by ten seeds; the diagnostic uses a single held-out task (Sketch) and a single horizon (K=200) with no sensitivity analysis; and Theorem 1 covers deterministic projected gradient descent under a quadratic, decoupled-curvature model, which the authors themselves note does not extend to AdamW dynamics or retention.

Questions for authors

Can you run the paired PIPL diagnostic on at least one published protection method, e.g., KeepLoRA's retained historical directions? This single experiment would convert the central claim from inference into evidence, and it appears feasible within your existing protocol.

Is PIPL measurable with all-layer LoRA or full fine-tuning, and how should the budget r scale with parameter count?

Will code, seeds and split lists be released?

Two points on comparison fairness: does the common-budget rerun put BFP and Pi-CCA on the same pretrained checkpoint (Pi-CCA's published results use OpenCLIP), and how is ZSCL's external reference-distillation data accounted for under a "common update budget"?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EgTz (Part 1/3: W1, W2, W3, W4)

Thank you for the time and care you devoted to reviewing our work, and for your valuable suggestions on generality and reproducibility. Below, we address your concerns with additional diagnostic results and implementation details.

The central question is how much adaptation historical protection unnecessarily prevents at a fixed learned state and training budget, subject to explicit retention tolerances. The additional evidence tests this question with a published protection method, a second held-out domain, and adapters throughout both encoders.

Notation. AUC is normalized adaptation area. RR and ZZ are historical-domain and pretrained-dataset macro averages. In diagnostic tables, unmarked AUC/RR/ZZ describe the full-protection branch, in percent. Paired effects Δ\Delta are half-release minus full protection, in percentage points (pp). Unless stated otherwise, results average ten independent parent runs at t=7,K=200t=7,K=200. Bracketed LL values are multiplicity-adjusted one-sided lower bounds. The joint PIPL criterion is L(ΔAUC)>1L(\Delta\mathrm{AUC})>1, L(ΔR)≥−1L(\Delta R)\geq-1, and L(ΔZ)≥−1L(\Delta Z)\geq-1. Other comparisons are descriptive.


W1. The diagnostic only tests internal variants, including an uncapped control.

Response.{\color{green}\mathbf{Response.}} Your requested external-method test detects PIPL in a VLM adaptation of GPM with its 0.95 energy threshold:

Method / held-out domainAUC full → halfΔAUC[L]\Delta\mathrm{AUC}\\ [L]ΔR[L]\Delta R\\ [L]ΔZ[L]\Delta Z\\ [L]
GPM / Sketch52.40 → 57.80+5.40 [4.36]−0.48 [−0.83]−0.24 [−0.51]
Growing / Flickr30K71.80 → 75.92+4.12 [3.15]−0.39 [−0.77]−0.22 [−0.47]

Both rows satisfy the joint criterion at K=200K=200. The six bounds use Bonferroni α/6\alpha/6, with α=0.05\alpha=0.05. GPM adapts the final projection weights (d=655,360d=655{,}360) using its activation-subspace construction. Within each method, both branches share the parent checkpoint and optimizer. This pairing measures recoverable protection cost within the method while accommodating GPM's larger parameterization. The GPM result extends the evidence from the internal accumulation control to a published protection mechanism.


W2. Adaptation is restricted to approximately 18K parameters. Deeper adaptation and rank scaling are unclear.

Response.{\color{green}\mathbf{Response.}} With rank-8 Q/V LoRA in all 12 Transformer layers of each encoder, plus the original projection adapters, d=509,952d=509{,}952—27.7 times the original scope. At t=7t=7, on held-out Sketch:

All-layer methodAUCRRZZΔAUC\Delta\mathrm{AUC}ΔR\Delta RΔZ\Delta Z
Unprotected64.0048.9062.80———
Growing54.1052.2463.70+5.65−0.41−0.26
Growing: adjusted lower bounds———4.66−0.75−0.50
BFP, r=16r=1661.2051.7564.00+0.73−0.16−0.08
BFP, r=64r=6460.4852.1064.15+1.12−0.24−0.12

Growing passes all three conditions with α/3\alpha/3 correction. Increasing BFP's rr from 16 to 64 improves retention at a measurable adaptation cost. The experiment demonstrates PIPL with trainable Q/V adapters throughout both encoders and supports choosing the protection budget according to retention requirements and sensitivity structure.


W3. Exact cloning, replay, and seed handling require reproducible implementation details.

Response.{\color{green}\mathbf{Response.}} Code release is scheduled upon acceptance. Appendix B.2 specifies seeds 0,…,90,\ldots,9, a globally image-deduplicated 80/20 split, and separate random streams for minibatches and probes. Each branch restores parameters, buffers, AdamW moments and counter, protection basis, and random states from the same post-refresh checkpoint. The branches reuse the diagnostic minibatches and keep their respective bases fixed throughout adaptation. Parent training resumes from the saved checkpoint. Appendix A.1.2 specifies numerical rank, SVD tie-breaking, and the complete AdamW update order. Together, these details define the shared starting state, replay schedule, and intervention needed to reproduce the paired comparison.


W4. Absolute benchmark gains are modest.

Response.{\color{green}\mathbf{Response.}} As you note, retention is the largest practical gain. Under Table 1's common budget, BFP improves I2T/T2I R@1 from 40.94/38.52 to 42.05/39.58. It also reduces pretrained drop from 3.02 to 1.37 pp, a 54.6% reduction. These gains show that BFP preserves more pretrained capability while improving continual retrieval. They complement the paper's central contribution: measuring avoidable protection cost through controlled intervention.

AuthorsOpenReview ↗

Response to Reviewer EgTz (Part 2/3: W5, W6, Q1)

W5. The diagnostic uses only Sketch and K=200K=200.

Response.{\color{green}\mathbf{Response.}} The release benefit also appears when Flickr30K is held out. The horizon sweep uses prefixes of the same 400-step trajectory and normalizes AUC by each horizon:

Held-out domainParentKKΔAUC\Delta\mathrm{AUC}ΔR\Delta RΔZ\Delta Z
SketchGrowing100+4.35−0.17−0.06
SketchGrowing200+6.75−0.32−0.14
SketchGrowing400+7.18−0.71−0.33
SketchBFP100+0.40−0.09−0.04
SketchBFP200+0.76−0.17−0.09
SketchBFP400+0.78−0.31−0.17
Flickr30KGrowing100+2.50−0.19−0.11
Flickr30KGrowing200+4.12−0.39−0.22
Flickr30KGrowing400+4.75−0.82−0.41
Flickr30KBFP100+0.44−0.10−0.05
Flickr30KBFP200+0.63−0.21−0.12
Flickr30KBFP400+0.69−0.39−0.23

The adaptation benefit persists beyond 200 steps in both settings. Longer adaptation also incurs larger retention costs, which the table reports alongside each gain. The joint noninferiority tests use K=200K=200, and the horizon sweep describes how the trade-off evolves. Reserving Flickr30K changes both the diagnostic domain and the parent-domain composition, providing a second setting in which the effect recurs.


W6. Theorem 1 does not establish AdamW or retention behavior.

Response.{\color{green}\mathbf{Response.}} Theorem 1 explains how releasing task-relevant directions recovers progress within a finite budget. Its assumptions are quadratic loss, positive restricted curvature, 0<η≤1/L0<\eta\leq1/L, and decoupling V⊤HW=0V^\top HW=0. Under these conditions,

\qquad c=W^\top b,\quad D=W^\top HW.$$ Here $V/W$ span the original free/released directions and $b=-\nabla f(0)$. The expression identifies **released task signal and curvature** as the determinants of recovery. Appendix D.8 bounds curvature-coupling effects, and D.11 explains how coupling can reverse the early loss ordering. Eq. (10), Table 3, and the controls in Table 13 provide the corresponding empirical evidence for AdamW and retention: | Growing protection control | $\Delta\mathrm{AUC}$, mean ± SD | $\Delta R$ | $\Delta Z$ | | --- | --- | --- | --- | | Norm-replayed half release | +5.18 ± 0.92 | −0.24 | −0.10 | | Reset-half minus reset-full | +6.52 ± 1.09 | −0.30 | −0.13 | Recovery persists with both matched displacement lengths and reset optimizer states. The theory explains the mechanism, and the controlled AdamW experiments establish its observed adaptation and retention effects. --- > #### **Q1. Can you apply paired release to a published protection method, such as KeepLoRA?** ${\color{green}\mathbf{Response.}}$ Yes. GPM permits a direct intervention while preserving parameters and optimizer state. At threshold 0.95, the image/text protection ranks are approximately 410/288. A fixed random half-deletion rule retains approximately 205/144 directions. This rule applies to GPM's accumulated basis, whose columns are stored without a global eigenvalue ordering. | GPM on Sketch, $K=200$ | AUC full → released | $\Delta\mathrm{AUC}$ | $\Delta R$ | $\Delta Z$ | | --- | --- | --- | --- | --- | | Half release | 52.40 → 57.80 | +5.40 | −0.48 | −0.24 | | Norm-replayed half release | 52.40 → 56.21 | +3.81 | −0.35 | −0.19 | | Adjusted lower bounds, half release | — | 4.36 | −0.83 | −0.51 | The half-release row passes the joint test. The descriptive norm-replay result also shows recovery after matching displacement lengths. KeepLoRA [1] fixes the constrained factor $A$ after initialization (§3.2), so its admissible update space remains fixed when cached historical directions are deleted. GPM allows the projection constraint itself to be released at the same learned state. It therefore provides a direct test of the protection cost you asked about. [1] KeepLoRA: Continual Learning with Residual Gradient Adaptation.
AuthorsOpenReview ↗

Response to Reviewer EgTz (Part 3/3: Q2, Q3, Q4)

Q2. Is PIPL measurable with all-layer LoRA, and how should rr scale?

Response.{\color{green}\mathbf{Response.}} Yes. With d=509,952d=509{,}952 all-layer Q/V and projection-adapter parameters, the budget comparison is:

All-layer protectionAUCRRZZΔAUC\Delta\mathrm{AUC}Protection state
Growing54.1052.2463.70+5.65—
BFP, r=16r=1661.2051.7564.00+0.7362.25 MiB
BFP, r=64r=6460.4852.1064.15+1.12249.00 MiB
Growing: α/3\alpha/3-adjusted boundsL(ΔAUC)L(\Delta\mathrm{AUC})L(ΔR)L(\Delta R)L(ΔZ)L(\Delta Z)
All-layer Q/V LoRA4.66−0.75−0.50

Growing passes the joint criterion. Protection rank rr budgets sensitive directions in the joint parameter space, while LoRA rank controls adapter capacity. Protection storage scales as O(dr)O(dr). The useful budget depends on the sensitivity spectrum and required retention. Here, quadrupling rr gains +0.35/+0.15 pp in R/ZR/Z at −0.72 pp AUC. This measured trade-off provides a practical basis for selecting rr on validation data.


Q3. Will code, seeds, and split lists be released?

Response.{\color{green}\mathbf{Response.}} Public code release is scheduled upon acceptance. Appendix B.2.1 specifies seeds 0,…,90,\ldots,9 and a deterministic 80/20 train–evaluation split. Duplicate images are grouped before splitting, and images shared with the held-out domain are excluded from parent data. All captions of the same image remain in the same split.


Q4. Are BFP and Pi-CCA initialized identically, and how are ZSCL's extra resources counted?

Response.{\color{green}\mathbf{Response.}} The common-budget Pi-CCA row is our rerun using the shared OpenAI CLIP ViT-B/16 checkpoint, tokenizer, and evaluation protocol (Appendices B.1.3 and B.5). It is evaluated separately from Pi-CCA's published OpenCLIP results. The common budget fixes main-task updates and batch size. Auxiliary data and computation are accounted for explicitly:

MethodMain-task updates × batchAdditional resourcesTraining minutes
BFP8,000 × 256512 boundary probes, no external teacher48.5 ± 0.7
Pi-CCA, common-budget rerun8,000 × 256Native method-specific computation53.2 ± 0.7
ZSCL, common-budget rerun8,000 × 2562,048,000 reference sample-exposures and 8,000 teacher batch-forwards60.8 ± 0.7

Timing uses three runs on an A100 80GB with matched precision and no feature cache. Training totals include loading, boundary operations, and each method's auxiliary computation. Evaluation and diagnostic branches are timed separately. ZSCL's reference data and teacher computation are therefore included in its reported training cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.