acceptodds
Withdrawn from ICLR 2027

What Should the Backbone Learn? Analyzing Heterogeneous Continual PET via Commit-and-Rebase

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In continual learning with pre-trained models (PTMs), parameter-efficient tuning (PET) has emerged as a prevailing paradigm, typically allocating a small budget of tunable parameters to tasks. The budget bounds interference with the pre-trained representation, but it also bounds how much a stream can teach the model: whether the parameters are appended alongside the backbone or embedded in a low-rank slice of it, the knowledge of every task must fit within an allowance far smaller than the backbone itself. A stream long enough will exhaust any such allowance, so the backbone itself must eventually be enlisted as the store — which raises a fundamental question: of everything a PET module learns, what is actually worth writing into it? The question is hard to even pose, because different PET methods express what they learn in incompatible formats. In this paper, we propose CoRe, an instrument that converts what an arbitrary PET method has learned into a shared weight space through a commit-and-rebase process, so that heterogeneous methods become directly comparable. Across seven PET methods and seven benchmarks, CoRe reproduces the behavior of the original expert more faithfully than existing merging operators, with 97.5% prediction agreement. Analyzing the converted updates, we find that only a small component of each update is invariant across methods and that this component alone moves the decisions of its task, providing a first empirical account of what a backbone should learn from a PET module.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, parameter-efficient tuning, class incremental learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_SERdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks the question of which knowledge from parameter efficient tuning (PET) modules should be consolidated into a pre-trained backbone during continual learning, specifically class incremental learning. Since prompts, adapters and Low rank modules express knowledge in different ways, the authors propose CoRe, converting learned knowledge into low rank QKV weight updates. It works using a commit and rebase step in order to write updates back into the backbone and rebasing to avoid applying the same updates twice. Across 7 PET techniques and 7 benchmarks, it reproduces the original methods predictions with 97.5% agreement. Using CoRe, the authors find that update direction is organised by task whilst energy allocation is organised by method and then identify a small shared task specific component that they test through transplant, removal and label-permuation.

Critical strengths

By placing heterogeneous PET methods in a single weight space, comparisons between these methods can be made which weren’t previously possible. Additionally, in treating each method as a black box, it can be applied to any method without any extra changes. It reaches 97.5% prediction agreement with the original experts with low variance spanning 7 benchmarks and 7 methods. The analysis includes careful controls, such as leave one out estimation of the shared direction, label permutation and matched vs. mismatched transplants.

Critical weaknesses

There is an additional step in the methodology which is mentioned in a single passing sentence in section 3.3 (“a bounded scalar multiple of C_t may additionally be added to W_t when a held-out split of the current task indicates that the write has fallen short of the expert”.) This is not discussed at all any more in the main paper - how often does this happen? Can you report this in the main tables as well as with an additional ablation with this component turned off to see how much it contributes to the overall methodology?

The shared component has a cosine of only about 0.02 with the updates and recovers 16.1% of the full updates gains (+0.731%). However, the paper’s own norm-matched baseline already recovers 11.6% (+0.527%), so the direction itself contributes only about 0.2% - with no confidence interval reported. This doesn’t seem to support the claim that the component is “causally sufficient to guide task decisions”. Could the authors report confidence intervals for the comparison or soften the claim?

If the LoRA initlalisations used for distillations are shared across methods (paper states seeds are shared), the same task alignment in Figure 1a could partially reflect a common starting subspace instead of a learned task signal.

There is no overview figure of the pipeline and the writing is generally very dense - in particular the Methodology section (e.g. Section 3.2 after equation (2) where multiple things are introduced and explained in a single long sentence). Notation is abused which makes things harder to follow (e.g. H is used to mean both the classifier and the curvature, D is used to mean datasets, KL divergence and a candidate write). The writing generally would benefit from substantial revision for clarity to reduce density and make the general arguments easier to follow.

Questions for authors

How often does the repair step introduced in section 3.3 (C_t being added to W_t) happen and how large is the deviation from the original commit?

Can you report compute and wall-clock times for this method?

Are the LoRA initialisations shared across methods during distillation? If so, could you repeat some of the experiments from Fig 1a using different seeds to remove this confound?

Reviewer_MEBPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an approach for performing class-incremental continual learning with potentially heterogeneous parameter-efficient fine-tuning (PEFT) methods (the paper uses the acronym PET, but I still to the more common PEFT). The idea of the paper is to sequentially merge task-specific PEFT modules into a single backbone; it uses an analogy to version control systems to call the underlying components "commit" and "rebase". The commit approach merges a task-specific PEFT module into the backbone through a low-rank update. The challenge here is that this is done even for PEFT modules that themselves do not use low-rank weight updates (e.g., learned prompts, adapters). The commit step identifies a low-rank update to the backbone that best preserves the feature representation and the classifier output on the incremental task; a margin-based loss function is added to preserve the decision margin from the original PEFT. To avoid the learned update from disturbing previous tasks, the paper proposes to use a curvature-based loss that penalizes changes to the margins of previous tasks. The rebase step is then used to preserve the task-specific classification head despite the update to the backbone. This is done via a low-rank correction to the weight matrix, which is merged into the base model. Experiments with an ImageNet-21k trained ViT adapted to various downstream tasks show that the proposed approach allows for highly accurate merging of the PEFT modules into the base model, outperforming various previous merging approaches when tested across 7 heterogeneous PEFT methods. Moreover, the experiments analyze which "parts" of the merged weight updates are due to the task and which are due to the specific model.

Critical strengths
  • The paper proposes an interesting approach for PEFT-based class-incremental learning. What makes this approach different from my perspective is that it can merge different families of PEFT methods into the backbone in a unified way while retaining high-levels of task accuracy.
  • One benefit of the proposed approach is that it enables analyzing which parts of the weight updates are task-specific and which parts of the weight updates are model-specific. This could enable a variety of future applications and research directions in the context of PEFT even outside of continual learning.
  • The paper is very well-written and mostly easy to follow (with a few exceptions, see below).
Critical weaknesses

Despite the fact that I like the idea of the paper, a number of significant issues prevent me from rating the paper higher at this point:

  • The motivation for the approach seems lacking to some extent. I like the idea of merging heterogeneous families of PEFT modules into a backbone, but unless I missed something, the need for this is not motivated in detail. Without this motivation, the approach is a bit more of an intellectual curiosity for me.
  • Given the terminology borrowed from version control systems and the technical relatedness of the approach, I am surprised that the Git Re-Basin paper (Ainsworth et al., ICLR 2023) is not cited and discussed.
  • The experiments seem rather incomplete. It is nice that the paper shows that the proposed merging approach outperforms previous ones. But the analysis of approach remains quite superficial. The issues include:
    • The different losses and modules of the approach are not ablated. We do not know how much for example the margin loss contributes or how much the curvature-based loss helps. Also the commit and rebase steps are not analyzed in isolation.
    • The approach has many hyperparameters, which are not analyzed.
    • What is also missing is an analysis of the final class-incremental learning performance. We see how close the merging gets to the performance of the individual tasks, but this does not evaluate the final CIL performance (as far as I could tell).
  • Some parts did not become sufficiently clear:
    • I am still somewhat confused by what exactly the rebase step does. Eq. 6 would benefit from a more detailed description given the complexity of the notation.
    • What is the signed cross-layer sketch S<tS_{<t} and how is it obtained? What is the sketch space?
    • The notion of a training base is not clearly explained. What is BtB_t precisely?
  • I enjoyed the idea of analyzing the breakdown of task-specific and model-specific parts of the weight updates, but the main paper contains only rather broad descriptions of how this is done. A concise formalization would be highly beneficial, e.g. of how the shared coordinate system is identified.
Questions for authors

Please address the weaknesses discussed above. Most important would be a clarification on the following points:

  • What is the motivation for merging different families of PEFT modules into a backbone in a unified way? Why not decide on one PEFT family and stick to it for class-incremental learning?
  • Can you discuss the relation to Git Re-Basin?
  • How important are the different components of the proposed approach?
  • How are the hyperparameters chosen and how robust is the approach to changes in the hyperparameters?
  • How well does the proposed approach work on standard class-incremental learning benchmarks?
  • What is the single-layer sketch? How is it obtained? What is the training base BtB_t and how does it differ from WtW_t?
Reviewer_YenzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks what a pre-trained backbone should absorb from parameter-efficient tuning (PET) modules in class-incremental learning. Since prompts, adapters and low-rank modules represent task knowledge in incompatible forms, the authors propose CoRe (Commit-and-Rebase), which converts the knowledge of an arbitrary PET method into a common weight space. At a consolidation step, the commit stage distills the expert's classifier-input representation, posterior and decision margin into a rank-64 LoRA on the query, key and value projections (Eq. 2), and writes the result into the backbone under a quadratic penalty built from a fixed-size sketch of the Gauss-Newton curvature of protected margins, solved in closed form with the Woodbury identity (Eqs. 3 to 5). The rebase stage trains a temporary low-rank correction so that the method's retained module, attached to the updated backbone, reproduces the consolidated model without moving the base far from the previous one (Eq. 6). Across seven PET methods (L2P, DualPrompt, CODA-Prompt, APER-VPT, APER-Adapter, InfLoRA, LCA) and seven benchmarks with a ViT-B/16 backbone, the consolidated model agrees with the expert on 97.46% of held-out predictions, compared with 95.80% for the best merging operator (Table 1). The converted updates are then analyzed. Signed update direction is reported to be organized by task and energy allocation over layers and projections by method (Fig. 1). A component estimated from the other methods' updates for the same task is reported to carry learned rather than input-driven signal (label permutation, Fig. 2a), to improve its own task more than other tasks when transplanted into the pre-trained backbone (Fig. 2b), to recover 16.1% of the full update's accuracy gain, and to pass through the regularized write almost unchanged (Appendix E). Appendix F uses this component in a compressed low-rank learner that exceeds LCA on CARS.

Critical strengths

I didn't see any particular strength in this paper.

The conversion fidelity of 97.46% agreement (Table 1) is not a contribution in itself, since its only purpose is to license the parameter-space analysis, and the paper's own measurements show that this parameter space is not identifiable (re-distillations of the same expert reach a cosine of about 0.06, Appendix E) while the released implementation appears to share the conversion initialization across methods along exactly the axis the analysis studies (weakness 1). The same consolidation also lowers incremental accuracy in 39 of 49 cells (Table 5), so the instrument is not behaviorally transparent in the sense required by Eq. (1). The rebase ablation (Fig. 3) is the one result that supports a design choice, but it covers two methods on one benchmark, and the values stated in Appendix D do not match the plotted curves. The analytical findings that the title promises are either confounded or of a magnitude of about 0.2 accuracy points beyond a norm-matched reference (weakness 2), and the practical demonstration consists of a single run on a single benchmark against a single baseline. I therefore cannot identify a strength that meets the standard for acceptance at ICLR.

Critical weaknesses
  1. The shared weight space produced by CoRe is not shown to be identifiable, and the central finding that signed update direction is organized by task may be produced by the conversion procedure rather than by what the methods learned. The paper's own measurement indicates the problem. Independent re-distillations of the same expert under different LoRA initializations reach a cosine of only about 0.06 (Appendix E), so two functionally equivalent conversions of a single expert are nearly orthogonal in this coordinate system, and the alignment of an update with the shared direction is about 0.02. Most of each converted update is therefore determined by the conversion rather than by the expert's function, and parameter-space comparisons between conversions need to control for the factors that the conversion shares across runs. The released code suggests that these factors are shared precisely along the task axis. In core/distillation.py the distillation seed is the global seed plus 300007 times the task index, independent of the method, and the student's per-task LoRA slots are initialized together under that seed (backbone/vit_inflora.py, init_param), with zero-mode initialization using slot tt for task tt. Since the paper states that seeds are shared by every method, the conversions of all seven methods for a given task appear to start from identical AA matrices and to see identical minibatch order and augmentation, while conversions for different tasks use different slots. This alone would make same-task pairs more aligned than same-method pairs, which is the pattern reported in Fig. 1a (+9.10 SD against −1.26 SD). The label-permutation control does not exclude this explanation, because it measures the transplant gain of the estimated component rather than the alignment between methods (Fig. 2a), and an alignment produced by shared initialization would not be expected to transfer task accuracy under permuted labels in any case. The required control is to repeat the analysis of Fig. 1 with independent LoRA initialization and data order for each method, and to report the re-distillation reference with the same initialization as well as with a different one.

A second validation that the setup permits is also absent. InfLoRA and LCA already learn exact weight-space increments, so the converted update of these methods can be compared directly with the increment it is meant to represent. Given the re-distillation cosine of 0.06, I expect this similarity to be low, which would mean that the coordinates do not recover the weight-space knowledge even where it exists. The choice of target space raises a related concern. The student acts only on the query, key and value projections, justified by the statement that prompt effects act there (He et al., 2022; Dherin et al., 2025). However, Dherin et al. (2025) derive that the effect of context is equivalent to a rank-1 update of the MLP weights following attention, not of the attention projections, and APER-Adapter inserts its adapter at the MLP. Energy allocation is measured over these 36 projection blocks only, so the reported method effect on allocation (+0.35 SD against +0.13 SD for task, Fig. 1a) may largely reflect where each method inserts its parameters, for example the layers that receive prompts, which is known by construction. All similarity results are reported in standardized units relative to the both-different pairs, without raw cosines, and the pairs are not independent because each update enters many pairs. The magnitude of the effect behind "nine standard deviations" therefore cannot be assessed.

The behavioral transparency of the instrument is also weaker than presented. Eq. (1) requires that consolidation leaves the method's decisions unchanged, yet Table 5 shows that CoRe lowers average incremental accuracy in 39 of the 49 method and benchmark cells, by 0.75 points on average, by 2.13 points on average for LCA and by up to 4.57 points (LCA on ImageNet-A). The Pearson correlation of 0.9947 cited as evidence of preservation is dominated by differences between benchmarks (CARS near 0.5 against CIFAR-100 near 0.9) and is not informative about this systematic decrease. The comparison with merging operators in Table 1 is also difficult to interpret. With λ=1\lambda=1 and the spectral normalization of Eq. (3), the regularized write attenuates the update by at most a factor of two along at most P=8P=8 directions in a space of about 2.1×1072.1\times10^7 coordinates, and Appendix C reports that it changes the past-margin response by only 3% to 19% with the default sketch. It is therefore unclear how this write yields 4.2 points more agreement and a five-fold lower logit KL than the unconstrained write, unless the rows also differ in the rebase step or in the validation-selected repair W←Wˉ+ρCtW\leftarrow\bar W+\rho C_t (Algorithm 1, line 26). The baselines are implemented in a replay script (tools/replay_pet_merge_fidelity.py, referenced in merging/operators.py) that is not included in the supplementary material, so this cannot be checked. Since every subsequent analysis treats CoRe as a transparent and identifiable measurement instrument, this is the weakness that most strongly determines my rating.

  1. The evidence for the functional role of the shared component is small in magnitude, partly an artifact of the normalization used, and does not support the conclusion about what a backbone should learn. Transplanted alone, the shared component recovers 16.1% of the full update's gain (+0.731 points), while the held-out method's own update scaled to the same norm recovers 11.6% (+0.527 points). The difference attributable to the shared direction is therefore about 0.2 points, reported without uncertainty, which is difficult to reconcile with the conclusion that the component is "causally sufficient to guide task decisions". The statement that a matched component improves its own task "by six times the gain obtained from components of other tasks" is not meaningful under the reported procedure. Outcomes are centered on each evaluation task's mean over donors, so the mismatched values sum to the negative of the matched value and are negative by construction, as visible in Fig. 2b, and their ratio to the matched gain is not defined in a useful sense. Fig. 2b also shows that roughly a fifth of the 49 matched values are at or below zero.

The conceptual interpretation of the component also needs qualification. The shared direction is the leading eigenvector of six unit-norm donor updates (Eq. 7), which approximates their normalized average, and with a single donor it is simply that donor's update direction, as Appendix F notes. The finding is therefore close to the statement that an update trained on task τ\tau by one method improves task τ\tau more than updates trained on other tasks, which is expected from task arithmetic and does not by itself isolate a method-invariant core. Comparisons against the plain average of donor updates, and against re-distillations of the held-out expert itself, would clarify what the eigendecomposition adds. The preservation result in Appendix E follows from dimension counting alone. A sketch spanning at most eight directions in 2.1×1072.1\times10^7 dimensions will preserve almost any direction that is not aligned with past margins, so the retention of 99.98% would hold for a random direction as well, and the claim that writing the shared component "costs essentially none of the protection budget" is not specific to the component. A random-direction control would establish this. Finally, the practical demonstration in Appendix F uses a single benchmark (CARS), a single run, a single baseline (LCA), and a shared direction estimated from other PET methods trained on the same stream. This gives the proposed learner additional training runs and supervision that LCA does not have. The demonstration also lacks the ablation that would isolate the contribution of the shared component, namely the same rank-32 compression and fold without the closed-form adjustment toward ct⊤atc_t^\top a_t. The statement in the conclusion that a minimal learner "can reach the level of a recent method of the same family" is therefore not supported as stated.

To sum up, the paper's central empirical claims depend on parameter-space comparisons in a coordinate system that the paper's own measurements show to be weakly identified, with a plausible confound in the released implementation. The instrument does not preserve the behavior of the methods it is meant to measure, the causal evidence amounts to about 0.2 accuracy points beyond a norm-matched reference, and the conclusion about what a backbone should learn is not supported by any experiment that writes this component in a continual-learning stream beyond a single run on CARS.

Questions for authors

Questions 1 to 8 address the issues that would have to be resolved before the paper could be considered for acceptance. They follow the order of the weaknesses above.

  1. In the experiments reported in Fig. 1 and Section 4.4, did the conversions of different methods for the same task share the LoRA initialization and the distillation data order? The released code (core/distillation.py, the seeding around the distillation call and the construction of ProposedLoRAStudent) suggests that they did, unless proposed_lora_seed was set per method. Please report Fig. 1a and Fig. 1b with an independent initialization and data order for each method.

  2. Please report the raw cosine similarities underlying Fig. 1a for same-task, same-method and both-different pairs, together with the re-distillation reference under both the same and a different LoRA initialization. How was the standard deviation in Fig. 1a computed, and how was the dependence between pairs that share an update taken into account?

  3. For InfLoRA and LCA, whose experts are exact weight-space increments, what is the cosine between the CoRe-converted update and the method's own increment over the same projections? This would directly test whether the coordinate system recovers the weight-space knowledge where a ground truth exists.

  4. Why is the conversion target restricted to the query, key and value projections, given that Dherin et al. (2025) describe the implicit context update as acting on the MLP weights and that APER-Adapter adapts the MLP? Do the allocation results in Fig. 1 change if the student also includes the attention output and the MLP projections?

  5. Do the rows of Table 1 share the same rebase step and the same validation-selected repair ρ\rho, with only the write rule changed? Please include the replay script used for the baselines. Given that the regularizer acts on at most eight directions with attenuation of at most one half, which component of the pipeline accounts for the difference of 4.2 agreement points between CoRe and the unconstrained write?

  6. What is the uncertainty, across methods, benchmarks and seeds, of the difference between the shared-component transplant (+0.731 points) and the norm-matched own update (+0.527 points)? How does the shared component compare with the plain average of the donor updates, and with a re-distillation of the held-out expert under a different initialization, at the same norm?

  7. Please report the retention measurement of Appendix E for random directions and for the held-out method's full update direction. If these are also retained above 99.9%, the result reflects the dimension of the sketch rather than a property of the shared component.

  8. For Appendix F, please report results on additional benchmarks and seeds, the ablation that applies the same compression and fold without the shared-coordinate adjustment, and the total training cost including the donor methods. How does the rule compare with other recent PTM-based class-incremental methods on CARS?

The remaining points are requests for clarification or suggestions. They do not individually determine my rating.

  1. Table 5 shows that CoRe lowers incremental accuracy in 39 of the 49 cells, with the largest losses for the two low-rank methods (on average 1.36 points for InfLoRA and 2.13 points for LCA). The text states that individual cells "deviate in both directions", which understates this pattern, and the cross-benchmark correlation does not address it. Please discuss why consolidation is least transparent for the methods whose knowledge is already in weight space, and report the mean and spread of the differences.

  2. The sketch is updated only at consolidation steps and only with examples of the consolidating task (Algorithm 1, lines 29 to 37). With K=2K=2, the margins of tasks 2, 4, 6 and 8 never enter the protected curvature. Is this intended, and does it affect the stability of those tasks? Appendix C also shows that the full empirical curvature attenuates past-margin responses much more strongly (ratios of 0.59 to 0.76) than the default sketch (0.81 to 0.97), so the description of the write as second-order protection overstates what the eight-probe sketch delivers.

  3. The consolidated backbone in Algorithm 1 is W←Wˉ+ρtCtW\leftarrow\bar W+\rho_t C_t, whereas Eq. (4) defines Wt=Wc+unvec(dt∗)W_t=W_c+\mathrm{unvec}(d_t^*). Since CtC_t is initialized from Bc−WˉB_c-\bar W, the repair moves the consolidated backbone toward the previous base. Please reconcile the two definitions, report how often ρt>0\rho_t>0 is selected, and clarify whether the analyses use Wˉ\bar W or WW.

  4. Several reported values do not match the figures. In Appendix D, the text gives final agreement without rebase of 94.61% (APER-VPT) and 94.87% (APER-Adapter) and with rebase of 98.03% and 96.83%, whereas Fig. 3 shows approximately 93.2% and 91.8% without rebase and 98.4% and 94.5% with rebase. The caption of Fig. 1 describes the decoding accuracies as chance-adjusted, but the plot shows raw accuracies with a chance marker, and task decoding from allocation (23.1% against a chance level of about 11%) is roughly twice chance, which is not weak as stated in Section 4.3. In Fig. 4 the confidence interval at one donor reaches zero, contrary to the statement that it excludes zero, and the value at six donors is a single estimate. The transplant gain is +0.731 points in Section 4.4, about 0.88 points at zero permutation in Fig. 2a and 1.8 points in Fig. 4. Please state which subsets each figure uses. Section 4.4 reports retention above 99.98% and Appendix E reports above 99.99%.

  5. How many seeds are used for the analyses and for Table 5, and what is the variance across seeds? The ± values in Table 1 are spreads across methods and benchmarks rather than across runs.

  6. The claim to propose "the first approach that expresses the knowledge of heterogeneous PET methods in a comparable shared weight space" should be positioned against Progress and Compress (Schwarz et al., 2018), which distills an active module into a persistent knowledge base under an online EWC penalty and is the closest precedent for the commit step, and against Trans-LoRA (Wang et al., 2024b), which already distills low-rank modules across models. MagMax (Marczak et al., ECCV 2024), which merges sequentially fine-tuned models by maximum magnitude for continual learning, is closely related to the LCA writing rule compared in Table 1 and is not discussed.

  7. The reproducibility statement indicates that the complete source code, environment and instructions are provided. The supplementary material contains no README, configuration files or entry point, and the code imports modules that are not included (backbone.vit_native, core.contracts, utils.inc_net from PILOT, and the replay script mentioned in question 5). The analysis code for Sections 4.3 and 4.4 and Appendices E and F is also absent. Please provide a runnable release so that the analyses can be reproduced.

Reviewer_kvofOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoRe, a commit-and-rebase framework for consolidating heterogeneous parameter-efficient tuning modules into a common low-rank update to the backbone. Prompt-based experts, adapters, and LoRA modules are first distilled into a rank-constrained update on selected Q/K/V projections. The update is then committed to the backbone using a curvature or Jacobian-sketch penalty intended to preserve previous decision margins. A temporary rebase module is optimized to compensate for the fact that the original PET module remains active after the backbone update; the temporary module is discarded after consolidation.

The paper evaluates CoRe across seven PET methods and seven visual class-incremental learning benchmarks using a shared ViT-B/16 backbone. It reports approximately 97.5% behavioral agreement with the original PET experts and studies the decomposition of converted updates into task-driven signed directions and method-driven energy allocation. Additional experiments include label-permutation controls, cross-method transfer, update removal, repeated writes, and curvature-sketch analyses.

Critical strengths

A useful consolidation perspective. The commit/rebase separation addresses a real problem: after writing an adapter or prompt effect into the backbone, retaining the original PET module can apply the same adaptation twice. This makes the framework practically motivated.

Broad within-domain diagnostic coverage. The paper evaluates several PET families and multiple visual benchmarks, and includes ablations for writing rules, label permutation, removal, repeated writing, cross-method transfer, and curvature sketches. This is more comprehensive than a standard single-method continual-learning experiment.

Critical weaknesses

The claimed behavioral equivalence is not demonstrated for continual retention. Distillation and curvature-sketch construction appear to use current-task data and current-task margin gradients. The reported 97.5% agreement therefore shows fidelity on current-task held-out examples, but does not establish preservation of old classes or old tasks after a write. There is no systematic table of old-task accuracy, old-class logit KL, calibration, or margin preservation after each consolidation step. A low-rank sketch based on a small number of current-task probes may miss precisely the directions required for old-class retention.

The paper overclaims generality and has circularity in its shared-direction analysis. “Arbitrary PET” and “what the backbone should learn” are broad claims based on seven visual PET methods, one ViT-B/16 backbone, and fixed rank-64 Q/K/V updates. The shared task direction is estimated from several methods trained on the same task and data, and then evaluated on another method from the same task. This can recover a generic current-task gradient direction by construction, rather than demonstrate a task-invariant reusable component. The reported statistical tests also do not clearly account for dependence across method-by-benchmark pairs or multiple comparisons.

Questions for authors

After each consolidation step, please report old-task and old-class accuracy, calibration, logit KL, and margin preservation on a held-out buffer containing all previously seen tasks. Please also evaluate a distillation target formed by a balanced mixture of current-task data and held-out or synthetic old-task inputs. Without these measurements, the reported current-task fidelity does not establish continual-learning retention.

Do the claimed shared directions and consolidation benefits persist across different backbones, task orders, random seeds, and data distributions, or in a non-vision/non-QKV PET setting? If not, please narrow the claim that CoRe applies to “arbitrary” PET modules.

Please report per-method and per-benchmark confidence intervals, exact sample sizes, and correction for multiple comparisons. In particular, are the method-by-benchmark observations treated as independent in the Wilcoxon tests, and how sensitive are the conclusions to the number of curvature-sketch probes and its rank?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.