- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the avoidable cost that historical protection can impose on learning new tasks. It introduces protection-induced plasticity loss (PIPL), measured through paired adaptation runs that begin from the same model and optimizer state but retain different amounts of protection. Adaptation gains are evaluated jointly with historical and pretrained knowledge retention. The proposed Budgeted Functional Protection (BFP) method maintains a fixed-size subspace of protected parameter directions, periodically updating it using compressed output-sensitivity estimates. The paper combines finite-step analysis with controlled interventions and evaluations on continual retrieval and classification benchmarks.
Critical strengths
- The paired diagnostic provides convincing evidence for the adaptation cost of protection. The two branches share their starting model, optimizer state, and adaptation schedule, allowing the effect of constraint release to be studied directly. Additional controls match update lengths, reset optimizer states, and vary the retained directions. The statistical analysis uses ten independent paired runs and adjusts for the primary family of comparisons. These choices make the evidence substantially more informative than a correlation between protection strength and new-task performance.
For growing protection, releasing half the protected directions improves adaptation AUC by 6.75 percentage points while satisfying the stated noninferiority conditions for historical and pretrained performance. This directly supports the existence of avoidable protection costs in the studied setting.
- The method, empirical results, and theory form a coherent contribution. Under the common training budget, BFP achieves bidirectional retrieval scores of 42.05/39.58, compared with 40.94/38.52 for Pi-CCA, while reducing pretrained accuracy drop from 3.02 to 1.37 points. The fixed-coordinate controls further show that selecting and refreshing directions matters, rather than protection rank alone explaining the result.
The finite-step analysis also adds useful insight: recovered task signal can improve adaptation, but curvature coupling can alter the ordering of the two trajectories. The counterexamples help explain why releasing more directions need not improve learning at every horizon. The paper clearly distinguishes these loss-based results from the empirical evaluation of AdamW and retrieval performance.
Critical weaknesses
- The central diagnostic finding is evaluated in a narrower setting than the benchmark results. The primary paired study uses Sketch as the reserved adaptation domain and compares BFP with its own uncapped growing-protection variant. Adaptation is confined to the final image and text projection adapters. The broader retrieval and classification results support the usefulness of BFP, but they do not establish how widely the measured protection-induced loss occurs across new domains or other protection methods.
The retention criterion also averages over historical domains and pretrained evaluation datasets. It would be useful to see whether those averages conceal appreciably larger losses on individual tasks. Repeating the paired study with another reserved domain would further strengthen the generality of the main finding.
- The practical impact of repeated compression remains only partly understood. Appendix A.3 gives an instructive failure case in which a repeatedly discarded direction eventually dominates the full historical average but is still absent from the compressed statistic. This makes clear that refreshing a fixed-size subspace does not necessarily recover the most important directions in the full history.
The random-basis and frozen-basis controls establish that direction choice matters. What remains unclear is how much error repeated truncation introduces in the actual experiments, and whether that error predicts retention losses on particular tasks. A same-rank comparison with the full-history reference subspace would help connect the compression analysis to the observed behavior.
Questions for authors
- Does the paired release benefit persist when another VLCL domain is reserved for diagnostic adaptation? For the existing Sketch experiment, how does retention change on each historical domain?
- Could the authors quantify accumulated compression error on the diagnostic stream, or compare BFP with a same-rank subspace derived from the full historical statistic?
5 comments
Response to Reviewer pBeW (Part 2/3: W2, Q1)
W2. What practical error does repeated truncation introduce, and does it predict task-specific retention?
The reference stores every realized sketch along the same BFP trajectory. It forms and selects its top-16 space. Sharing the trajectory isolates the effect of recursive compression:
| Boundary | Relative spectral error | Lost trace | Projector distance to full-history top-16 | Excess worst-case residual sensitivity |
|---|---|---|---|---|
| 1 | 0.27 ± 0.04 | 0.40 ± 0.03 | 0.00 ± 0.00 | 0.0000 ± 0.0000 |
| 4 | 0.38 ± 0.05 | 0.67 ± 0.04 | 0.58 ± 0.09 | 0.0027 ± 0.0006 |
| 7 | 0.44 ± 0.06 | 0.76 ± 0.04 | 0.74 ± 0.08 | 0.0051 ± 0.0010 |
At , both methods select the same top-16 space, although rank truncation already loses spectral mass. At , replacing only the basis at the same checkpoint and optimizer state gives:
| Rank-16 basis | AUC | Resident basis | Stored sketches for reference construction | ||
|---|---|---|---|---|---|
| Recursive BFP | 57.86 | 50.98 | 63.34 | 2.25 MiB | — |
| Full-history top-16 | 57.64 | 51.24 | 63.43 | 2.25 MiB | 63.00 MiB |
Full-history selection gains 0.26/0.09 pp in at −0.22 pp AUC. Its domain-level differences are:
| Historical domain | Residual sensitivity reduced by reference | Retention change, reference − BFP |
|---|---|---|
| Flickr30K | +0.008 | +0.20 |
| COCO | +0.012 | +0.31 |
| Pets | +0.017 | +0.45 |
| Lexica | +0.021 | +0.50 |
| Simpsons | −0.004 | −0.08 |
| WikiArt | +0.016 | +0.37 |
| Kream | +0.001 | +0.07 |
Sensitivity quantities are dimensionless, and retention changes are pp. Their signs agree across domains, linking protection differences to retention descriptively. The same-rank intervention directly measures the practical effect of basis selection. BFP optimizes the available compressed statistic in Eq. (6), and the full-history comparison quantifies the additional retention obtainable from storing the historical sketches.
Q1. Does the benefit persist on another reserved domain, and what happens to each historical task?
Yes. With Flickr30K reserved and :
| Parent | AUC full → half | |||
|---|---|---|---|---|
| Growing | 71.80 → 75.92 | +4.12 | −0.39 | −0.22 |
| Growing: adjusted lower bounds | — | 3.15 | −0.77 | −0.47 |
| BFP | 77.25 → 77.88 | +0.63 | −0.21 | −0.12 |
Growing passes the joint criterion. BFP shows a smaller descriptive release effect. For your requested original Sketch experiment, all seven historical-domain changes are below. Brackets are lower bounds corrected over 13 task endpoints within each method, including the six pretrained datasets:
| Historical task | Growing | BFP |
|---|---|---|
| Flickr30K | −0.24 [−0.60] | −0.12 [−0.55] |
| COCO | −0.31 [−0.74] | −0.18 [−0.69] |
| Pets | −0.18 [−0.69] | −0.26 [−0.84] |
| Lexica | −1.18 [−1.76] | −0.42 [−0.78] |
| Simpsons | +0.04 [−0.32] | −0.08 [−0.51] |
| WikiArt | −0.56 [−0.99] | −0.24 [−0.75] |
| Kream | +0.19 [−0.32] | +0.11 [−0.47] |
BFP satisfies all historical per-domain −1 pp bounds in this Sketch setting. Growing's Lexica result falls below that threshold. The Flickr30K setting provides another example of the distinction between aggregate and per-task retention:
| Flickr30K held out: Simpsons retention | ||
|---|---|---|
| Growing | −1.24 | −1.75 |
| BFP | −0.46 | −1.04 |
Both Simpsons lower bounds fall below the per-task −1 pp threshold. The aggregate test and these domain-level results together show where release recovers adaptation within the stated macro-average tolerance and where stronger task-specific protection is needed.
Response to Reviewer pBeW (Part 3/3: Q2)
Q2. Can you quantify accumulated compression error and compare with a same-rank full-history subspace?
Yes. Let , and let project onto the respective free spaces. The relative error is . Projector distance is . All quantities below average ten parent trajectories:
| Relative spectral error | Projector distance | Full-history sketch storage | ||
|---|---|---|---|---|
| 1 | 0.27 ± 0.04 | 0.40 ± 0.03 | 0.00 ± 0.00 | 9.00 MiB |
| 4 | 0.38 ± 0.05 | 0.67 ± 0.04 | 0.58 ± 0.09 | 36.00 MiB |
| 7 | 0.44 ± 0.06 | 0.76 ± 0.04 | 0.74 ± 0.08 | 63.00 MiB |
| Same-checkpoint, rank-16 adaptation | AUC | ||
|---|---|---|---|
| BFP basis | 57.86 | 50.98 | 63.34 |
| Full-history reference basis | 57.64 | 51.24 | 63.43 |
| Reference − BFP | −0.22 | +0.26 | +0.09 |
Both deploy a 2.25 MiB basis. Constructing the reference requires 63 MiB of historical sketch columns and an additional thin SVD. The reference is computed exactly from those saved boundary sketches. Replacing the basis at the same checkpoint measures the downstream effect of compression while keeping rank, model state, and optimizer state fixed. The resulting comparison connects the measured spectral error to adaptation and retention performance.
Response to Reviewer pBeW (Part 1/3: W1)
Thank you for your careful reading and valuable suggestions on diagnostic generality and compression. We appreciate the time you devoted to the review and address your questions below with additional paired tests, per-task retention results, and a full-history comparison.
The central question is how much adaptation historical protection unnecessarily prevents at a fixed learned state and training budget, subject to explicit retention tolerances. The additional evidence tests this question with a published protection method, a second held-out domain, and adapters throughout both encoders.
Notation. AUC is normalized adaptation area. R and Z are historical-domain and pretrained-dataset macro averages. In diagnostic tables, unmarked AUC/R/Z describe the full-protection branch, in percent. Paired effects Δ are half-release minus full protection, in percentage points (pp). Unless stated otherwise, results average ten independent parent runs at t=7,K=200. Bracketed L values are multiplicity-adjusted one-sided lower bounds. The joint PIPL criterion is L(ΔAUC)>1, L(ΔR)≥−1, and L(ΔZ)≥−1. Other comparisons are descriptive.
Response. The expanded paired tests cover an external method, a second held-out domain, and adaptation throughout both encoders:
The first two rows use α/6, and the all-layer extension uses α/3. Each passes the joint macro-average criterion. The per-task Sketch results show how retention changes are distributed across the evaluation tasks:
Lexica loses more than 1 pp under Growing and GPM release. BFP keeps every historical-domain mean change within 0.42 pp. Eq. (10) measures recoverable adaptation subject to macro-average retention tolerances. The expanded results support that criterion and identify the individual tasks that require closer protection.