acceptodds
Under review as a conference paper at ICLR 2027

Dissecting PEFT-Based Continual Learning: From Expert Training to Inference

Abstract

Parameter-efficient fine-tuning (PEFT) has become a common strategy for class-incremental learning, with lightweight task-specific experts trained on a frozen pretrained backbone. However, existing methods often change several parts of the system simultaneously, making it difficult to determine whether reported gains arise from expert training or from choices such as the backbone, classification head, routing signal, and inference rule. We address this problem through a controlled component analysis of PEFT-based class-incremental learning. Holding expert training fixed, we vary these components across four benchmarks and two task granularities. Strong frozen-feature methods such as RanPAC provide a demanding reference point, yet a mechanism-free PEFT learner with a regularized least-squares readout and Mahalanobis-weighted expert ensembling achieves the highest mean across the eight settings. The value of an inference rule nevertheless depends on the head with which it is paired. Entropy remains a useful and simpler routing signal, particularly for probability ensembling, while Mahalanobis routing is more reliable when predictive confidence is poorly calibrated. Routing stress tests further show that retrieving the correct expert is not sufficient when unrelated experts dilute its prediction. Applying the same readout and combination choices to experts learned by established methods improves 19 of 24 settings and substantially changes their relative ranking, although the gains remain representation and dataset dependent. These findings show that the inference system surrounding expert learning is a major source of performance variation and should be evaluated separately from the learning mechanism itself.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Parameter Efficient Fine-Tuning, Artificial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XEtWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates the impact of inference strategies in parameter-efficient fine-tuning (PEFT)-based class-incremental learning (CIL). By fixing task-specific LoRA experts and varying the backbone, classification head, routing signal, and expert combination strategy, the authors show that inference choices can lead to substantial performance differences, with accuracy varying by 19–38 percentage points across benchmarks. The proposed combination of a regularized least-squares classifier and Mahalanobis-weighted expert ensemble achieves an average accuracy of 78.12% across eight settings, outperforming RanPAC (76.88%). Further experiments demonstrate that the proposed inference strategies can improve existing PEFT-based CIL methods, highlighting the importance of inference design beyond expert training.

Critical strengths
  1. The paper highlights an underexplored aspect of PEFT-based continual learning: inference design. Its controlled analysis demonstrates that inference choices can substantially affect performance, providing useful insights for the community.
  2. The authors systematically investigate multiple inference components, including classification heads, routing signals, and expert combination strategies. The analysis of expert interference and routing reliability helps explain why different inference strategies perform differently.
  3. The proposed inference configuration improves multiple existing methods across a range of experimental settings. The paper also discusses the trade-off between training efficiency and inference cost, making the findings practically relevant.
Critical weaknesses
  1. The paper demonstrates that changing the classification head and inference rule can substantially affect accuracy when the LoRA experts are fixed. However, the main controlled analysis removes continual-learning-specific mechanisms, while the transfer study covers only three existing methods. The results also show that replacing the readout does not consistently improve performance across methods and datasets. This leaves an important question unresolved: whether the identified inference configurations are generally effective or depend strongly on the feature representations and inter-expert relationships induced by a particular training method. A more systematic cross-evaluation of multiple training methods and inference configurations, ideally with matched backbones and data splits, would help quantify this interaction and clarify the extent to which the findings generalize beyond the mechanism-free setting.
  2. The paper compares entropy- and Mahalanobis-based routing and includes a candidate-expert stress test, but the analysis is limited to ImageNet-R and ImageNet-A and primarily reports final classification accuracy. Final accuracy alone does not reveal whether a routing strategy selects the most appropriate expert, nor whether an incorrect selection is harmful because of the selected expert itself or because of how its predictions are combined with those of other experts. This distinction is important given the paper's observation that including irrelevant experts can dilute predictions. The analysis would be more informative if it separately measured expert-selection quality and downstream accuracy, for example by comparing the selected experts with oracle-selected experts and reporting the performance gap under different candidate sets. Such an analysis could better establish when each routing signal is effective and explain the mechanisms behind its observed performance.
  3. The proposed lstsq + maha_e configuration achieves an average accuracy of 78.12% across eight settings, compared with 76.88% for RanPAC. Although this result supports the competitiveness of the proposed configuration, the absolute difference is relatively modest, and the paper reports results averaged over only three runs. Furthermore, the proposed configuration does not dominate across benchmarks: it trails RanPAC on CUB-200 and performs differently across the other datasets. Without paired statistical tests or confidence intervals for the aggregate comparison, it is difficult to determine whether the overall advantage is robust to random seeds and data splits. Reporting paired per-seed differences and uncertainty estimates, alongside per-dataset results, would make the comparative claims more reliable.
Questions for authors

Please refer to the Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XEtW

We thank the reviewer for the constructive suggestions; they led to the main new experiments in this rebuttal.

W1. Do inference choices transfer beyond independently trained experts? Yes; see General Response 1(GR1), where the study now covers further published methods. On EASE's own adapters we evaluated all 18 head × rule configurations from our grid, with no retraining and no exemplars:

  • The head ordering lstsq > ridge-1k > NCM holds for both expert sets.
  • Mahalanobis ensembling is the best rule on average for both (76.7% with our experts, 75.3% with EASE's).
  • The configuration rankings correlate across the two expert sets (Spearman 0.35–0.86).
  • The configuration main effect explains 59% of within-setting variance; the method × configuration interaction explains 7%.

Replacing EASE's native inference with lstsq + maha_e improves 6 of 8 settings (+1.9 on average, +8.3 on ImageNet-A, T=20). The best readout can still depend on the representation, and we will say so.

W2. Is the bottleneck expert selection or prediction combination? We now measure the two separately on all 8 settings (lstsq + maha_e, 3 seeds):

MeasureMean
Correct expert in the selected top-k94.0% of samples
Accuracy lost because the correct expert was not selected1.8 points
Accuracy lost when combining the selected experts9.8 points

Combination loss is the larger of the two in all 8 settings, accounting for 71–97% of the gap between oracle and ensemble accuracy. Even when the correct expert is selected, it receives only 41–58% of the ensemble weight.

Mahalanobis does improve selection; for example, top-1 retrieval on ImageNet-A (T=20) rises from 40.7% to 54.5%. Yet final accuracy rises by only 1.8 points.

We also ran a stress test with the head and combiner fixed. Mahalanobis top-k gives the best candidate set in all 8 settings (+3.3 to +6.8 over random-k). However, forcing the oracle expert into a set of three still leaves a gap of 3.2–9.4 points to oracle-only inference. The open problem is therefore combination, not only retrieval.

W3. How robust is the gain over RanPAC? The gain is modest and depends on the benchmark, and we will state this explicitly. Mean accuracy is 78.12% for ours and 76.88% for RanPAC.

Ours − RanPACT=10T=20
CIFAR-100−0.08+0.09
CUB-200−2.97−4.33
ImageNet-R+2.80+5.20
ImageNet-A+6.34+2.87

RanPAC draws its class orders differently, so the runs are not paired, and we use unpaired Welch t-tests:

  • RanPAC is significantly better on both CUB-200 settings (|t| ≥ 10).
  • Ours is significantly better on all four ImageNet-R/A settings (t ≥ 4.3).

The strongest statistical evidence comes from comparisons inside our grid, where the experts, class orders and seeds are identical:

  • lstsq + maha_e beats lstsq + ens_p in 24 of 24 matched runs (+1.50 points; 95% CI [+1.25, +1.75]).
  • lstsq beats ridge-1k in 24 of 24 matched runs (+1.45 points).

The higher-capacity ridge-10k head is reported in GR2.

The response has addressed my concerns, and I therefore raise my score.

Reviewer_KxENOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides an experimental analysis of parameter efficient finetuning (PEFT) based continual learning methods which progressively train a set of experts and perform inference on their mixture.

The proposed evaluation framework is based on a standard class incremental continual learning scenario with hidden task ID during inference. The authors use a simplified expert training protocol which trains one independent LoRA adapter per task and then freezes it. No continual learning specific mechanisms are used (naive training protocol).

The paper composes a family of continual learning (CL) configurations along 4 axis: 1) backbone choice, 2) classification head choice, 3) routing signal for inference and 4) inference rule, i.e. how the routed experts are combined into a single prediction.

Experiments are designed to inspect the contribution of each individual dimension, disentangling their contribution across 4 benchmarks. One result is that the inference system is at least as influential as the expert training mechanism. Further, the proposed CL method family is compared against 12 established CL methods.

Critical strengths
  • The paper is clear and well motivated. It adresses the real problem that new PEFT-based CL methods change several components at once, effectively hiding where reported gains come from.
  • It provides controlled experiments. E.g. experts are trained once with the naive protocol and then fixed. Heads, routing signals and inference rules are then evaluated on that identical set of experts, meaning that performance changes can be attributed to specific configurations.
  • The paper provides a cheap baseline and concrete reporting recommendations which are useful for practitioners and authors of new papers.
Critical weaknesses
  • The analysis assumes disjoint class sets of equal size. For more realistic scenarios like overlapping or domain-incremental streams routing based on the distance to an expert’s training distribution may become ambiguous. (see Q1)

  • Experts are independent and frozen. As explicitly stated by the authors, this avoids the problem of forgetting. However, it is not clear if the conclusions transfer to methods that share or update parameters across tasks. (see Q2)

  • A central claim of the paper is that the inference system is at least as influential as expert training. The paper does not provide a clean experiment for this claim, because experts are always trained with the same protocol. A clean test would have to vary the expert-training mechanism while holding the readout fixed. (see Q3)

  • The paper argues that published methods bundle backbone, head, routing and inference rule. However, it provides no summary which combination each of the 12 baselines uses. Can they be represented as elements of the used 4 dimensional configuration space? This mapping exists only partially, and only for the three transfer methods. (see Q4)

Questions for authors

Q1: How would the framework extend to overlapping or domain-incremental streams, i.e. if no single correct expert per test sample exists? Would entropy/Mahalanobis routing and the oracle analysis still be meaningful there? Do you expect your conclusions to hold?

Q2: Do you expect your conclusions to transfer to methods that share or continually update parameters across tasks? could you include such a method (e.g. SD-LoRA or CL-LoRA) in the transfer experiment, or explain why this isn’t feasible?

Q3: Is it possible to run an experiment with fixed readout (e.g. lstsq+ens_p) and to vary the expert training mechanism? E.g. initialize each new LoRA expert from the previous one instead of from scratch (to see if forward transfer helps).

Q4:. Could you provide a table placing each of the 12 baselines within your configuration space (head, routing signal, combiner), noting where a method falls outside it? Does any baseline coincide with one of your evaluated configurations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KxEN

We thank the reviewer for the positive assessment and the precise questions.

W1/Q1. Overlapping or domain-incremental streams? The framework extends to these streams, but our oracle-based evaluation assumes disjoint classes. In domain-incremental streams, several experts can be relevant to the same input, so a single "correct" expert is no longer defined.

The routing signals still apply; Mahalanobis distance would act as a measure of proximity to each domain. The natural extension would measure recall of the relevant set of experts and redefine the oracle as the best combination within that set. These are proposed extensions, not measurements we have made. We will state the disjoint-class scope in Sec. 6.

W2/Q2. Methods that share or update parameters? The head analysis transfers; the routing analysis requires experts that can be selected separately.

  • L2P and CODA-Prompt continually update a shared prompt pool, and InfLoRA accumulates its task branches into one evolving model. Swapping in our head improves all three in every setting, by +7.0, +8.4 and +7.6 points on average (GR1).
  • For methods with a single final representation there is no per-input choice among experts, so the selection-and-combination analysis applies only to methods that keep task-specific modules, such as EASE (R1-W1).

W3/Q3. Change expert training, keep the readout fixed. We ran three tests:

  1. Cross-method. With lstsq + maha_e fixed, switching from our experts to EASE's changes accuracy by 1.4 points. With the expert set fixed, switching between maha_e and entropy selection changes accuracy by 10.6 points on our experts and 6.7 on EASE's.
  2. Separation loss. Training experts against earlier, frozen experts improves oracle (known-task) accuracy by +1.19 points (24 of 24 runs). Under fixed task-agnostic readouts the change shrinks or flips sign: entropy +0.31, probability ensemble +0.35, logit ensemble +0.24, voting −0.83. By comparison, the inference rules themselves differ by 4.0 points on the same experts.
  3. Warm start (as suggested). Each expert is initialized from the previous one:
Fixed ruleIndependentWarm-startedΔ
Oracle expert89.6289.84+0.22
lstsq + entropy71.8367.47−4.36
lstsq + ens_p76.4372.96−3.47
lstsq + maha_e77.9274.97−2.95
Mahalanobis top-1 retrieval75.8270.28−5.54

Warm starting leaves each expert's own accuracy unchanged, but makes the experts harder to tell apart, so task-agnostic accuracy falls by 3–4 points. A training change can therefore improve individual experts while hurting the system's accuracy. In these settings, changes to the readout matter as much as, or more than, the training changes we tested. We will restate our claim in exactly these terms.

W4/Q4. Mapping baselines onto the components. We will add a table to the appendix listing, for each method, its expert type, head, routing signal, combiner and inference passes. Examples:

  • SimpleCIL is close to our frozen-backbone + NCM configuration.
  • RanPAC uses a large random-feature ridge head, which motivated GR2.
  • EASE runs a bank of task modules.
  • InfLoRA and GLAM form a single representation, so they have no routing step.

Mechanisms outside our grid will be marked as such rather than forced into it; examples are learned prompt mixing, classifier alignment and trainable merging.

Reviewer_KxENOpenReview ↗

reviewer response

We can make this short:

Thank you for all your explanations and for running the additional experiments. I especially like the new results for W3/Q3 and agree with your interpretation there. Some good addition to the paper came out of it.

Bonne chance!

Reviewer_QzVUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper runs a controlled component analysis of class-incremental learning with parameter-efficient fine-tuning (PEFT). Each task gets one independently trained LoRA expert on a frozen ViT-B/16 backbone, with no continual-learning mechanism. On top of these fixed experts, the authors vary: the backbone: ViT, CLIP, or DINOv2; the classification head: NCM, ridge, least squares (lstsq), linear, or linmask; the routing signal: entropy or Mahalanobis distance; the combiner: selection, ensembling, voting, or parameter merging.

Experiments cover four benchmarks, each with 10- and 20-task streams. The head and inference rule alone shift accuracy by 19–38 points. An lstsq head with Mahalanobis-weighted ensembling has the best mean (78.1% vs 76.9% for RanPAC). Swapping only the readout of EASE, InfLoRA, and HiDe-LoRA improves 19 of 24 settings and reorders their ranking.

Critical strengths
  1. The question matters. Separating expert training from the inference system around it addresses a real confound in recent PEFT continual-learning papers.

  2. The design is clean and controlled: identical experts, the same backbone and splits for all twelve baselines, three seeds, and a factorial grid with variance decomposition.

  3. The transfer experiment (Table 6) is convincing and actionable. For example, InfLoRA on 20-task CUB-200 goes from 14.9 points behind EASE to within 0.5 points just by changing the readout.

  4. The routing stress test gives a useful insight: retrieving the correct expert is not enough when unrelated experts dilute its prediction.

  5. The reporting recommendations are concrete.

Critical weaknesses
  1. Unequal inference compute. The mechanism-free learner runs a full ViT forward pass for every expert on every image, so inference cost grows linearly with T (Fig. 3a: 1,272 s at T=20). Most baselines use one or two passes, so the comparison is not compute-matched. A fixed-budget comparison is needed (e.g., a top-k forward from a cheap router).

  2. Hyperparameters tuned on the test set. k and τ are chosen on CIFAR-100 using final accuracy (Table 7, seed 0), and λ=100 and shrinkage 0.5 are fixed globally. A held-out validation protocol is needed.

  3. Unclear transfer setup. For InfLoRA, a single lstsq head is fitted on a backbone that keeps changing across tasks. If old-task statistics are not recomputed, there is feature drift. If they are, this needs old data, the same concern used to exclude GLAM.

  4. Variance decomposition skewed by a weak head. linmask is a deliberately weak head (−17 points); it inflates the "head explains 68.8%" figure. Please report the decomposition without it.

  5. Limited novelty and modest gains. The best configuration combines known pieces (analytic least squares as in RanPAC/ACIL, Mahalanobis as in FeCAM). Its gain over RanPAC is +1.2 points on average, and it trails RanPAC on CUB-200 by 3–4 points.

Limited scope.

  1. The baselines use only one backbone.

  2. CODA-Prompt, HiDe-Prompt, and SLCA are cited but not compared.

  3. Ridge uses only 1,000 random features, while RanPAC typically uses far more, which may weaken that head.

  4. Merging is tested only at T=10, routing only on ImageNet-R/A, and transfer only on three methods.

Minor:

Table 3 and A.6 say maha_e combines top-k logits, but the abstract says "Mahalanobis-weighted expert ensembling". Also, the Fig. 3b caption calls the weights entropy-derived, yet the "Mahalanobis top-k" row uses Mahalanobis for retrieval; it would help to say how the weights are set for that row.

Questions for authors
  1. Can you compare at matched inference FLOPs, e.g., routing to k experts and running only those forward passes?

  2. How exactly are the lstsq statistics for InfLoRA obtained on its evolving merged backbone? Is any past data revisited?

  3. Were k, τ, λ, and shrinkage chosen on held-out data? How do the results change if they are tuned per dataset on validation splits?

  4. What is the variance decomposition without linmask, and with a stronger ridge (e.g., 10k features as in RanPAC)?

1 comment
AuthorsOpenReview ↗

Response to Reviewer QzVU

We thank the reviewer for the detailed technical critique, which we addressed with new experiments.

C1. Is the comparison fair in inference compute? We agree that the full scan over all experts costs more at inference than one-pass methods. The cost is not unique to our method: on ImageNet-R (T=20, one A100 GPU), our scan takes 63.6 s, EASE 105.9 s and CL-LoRA 100.0 s; L2P (15.6 s) and GLAM (2.6 s) are faster.

We implemented the requested budgeted variant. One frozen-backbone pass computes Mahalanobis distances to per-task statistics, and then only the top-k experts are run, each needing its own forward pass. The total is 1 + k passes per image instead of T.

VariantPasses / imageMean acc. (8 settings)
Frozen backbone, no experts172.93
Router + top-1274.19 ± 0.10
Router + top-2377.08 ± 0.10
Router + top-3477.71 ± 0.12
Router + top-5677.98 ± 0.17
Full scanT77.92 ± 0.16

Four passes recover all but 0.21 points of full-scan accuracy; at T=20 this means 4 passes instead of 20. At three passes the variant is close to RanPAC's single pass (77.08 vs 76.88). We do not claim a strict compute-matched win over RanPAC.

C2. Were hyperparameters selected on held-out data? We ran a new selection study. We held out 10% of each class's training data, selected the configuration on validation accuracy averaged over all 24 runs, then retrained the experts on 100% of the data and evaluated on the test set.

ChoiceSubmittedHeld-out choiceΔ validation acc.
Head + rulelstsq + maha_esame0.00
k35+0.12
τ / shrinkage1 / 0.5same0.00
λ1001000+0.03

Choosing the hyperparameters per dataset changes test accuracy by only −0.12 to +0.03 points. Once ridge-10k is added to the candidates, validation selects ridge-10k + maha_e (GR2). We will relabel Table 7 as a sensitivity analysis.

C3. How are InfLoRA statistics obtained despite feature drift? No old data are revisited:

  1. When a class first appears, its features are extracted with InfLoRA's model at that task.
  2. These features update the least-squares statistics once; old-class statistics are never recomputed.
  3. Test images then pass through InfLoRA's final model.

The resulting drift is therefore present in the results we report. Despite it, the swap improves InfLoRA in all 8 settings, by +1.9 to +15.5 points. We now report this protocol for further methods (GR1).

GLAM already re-trains its classifier on features sampled from stored class Gaussians, which is a learned correction for drift. Disabling that step lowers GLAM's accuracy in all 8 settings (−11.4 points on average). GLAM's accuracy is thus also carried by its readout. We will separate the replay-based diagnostic in the appendix from this exemplar-free protocol.

C4. Does linmask inflate the variance decomposition? Task-masked training is a real design choice in prompt-based CL, but we agree that the claim should not depend on it. Removing linmask gives:

Share of variance*All headsWithout linmask
Head68.8%34.3%
Inference rule6.7%23.9%
Head × rule5.9%21.8%
Readout total81.4%80.0%

*After removing dataset and stream-length effects.

The head's individual share falls, as the reviewer expected, but the total share of the readout stays near 80% and the interaction becomes prominent. Across all grid variants we tested, including ridge-10k, the readout total lies between 74% and 84%. We will revise the claim to refer to the readout as a whole.

C5. What is novel beyond existing components? We do not claim a new classifier or routing rule; least squares and Mahalanobis scoring follow RanPAC/ACIL and FeCAM. Our contribution is the controlled dissection: holding the experts fixed exposes accuracy differences that would otherwise be credited to expert training. GR1 shows that this holds for published methods. We also accept the critique of classifier capacity: the ridge-10k head is now our strongest configuration (GR2), and we will correct the original ranking.

Other points.

  • Baselines: GR3.
  • Backbones: three in the main paper plus ImageNet-1K (GR4).
  • maha_e selects the top-k experts by standardized Mahalanobis distance and combines their logits with softmax weights derived from those distances.
  • In the stress test, Mahalanobis is used only to choose the candidate set. Every row of that test uses the same entropy-weighted probability combiner.
  • We will use these definitions consistently throughout the paper.
  • The incremental merge grid covers T=10; a per-sample merge-rule ablation additionally covers T=20 (CIFAR-100, CUB-200, ImageNet-R), where all merge rules except naive summation are within 1.7 points of each other, again pointing to the readout rather than the merge algorithm (see GR5).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

General Response to All Reviewers

We thank the reviewers for their careful and constructive reviews. A common thread across the reviews is the question of how far our conclusions about the classification head and the inference rule extend beyond our reference learner. We ran new experiments to answer it. Unless noted, every experiment covers all four datasets (CIFAR-100, CUB-200, ImageNet-R, ImageNet-A), both stream lengths (T = 10, 20) and 3 seeds, and none uses exemplars. This response summarizes the results shared across reviewers. Individual responses cite these sections as GR1–GR4.

GR1. The readout effect transfers to published continual learners. We keep each method's own trained model unchanged and replace only its native readout with our closed-form least-squares head. Each class is fitted once, through the model as it was when that class was learned, so no old data are revisited and the feature drift of later tasks is not corrected.

Method (own trained model kept)How its parameters are learnedSettings improvedMean change
InfLoRAtask branches accumulated into one evolving model8 / 8+7.6 (range +1.9 to +15.5)
L2Pprompt pool shared and updated across all tasks8 / 8 (24/24 runs)+7.0 (range +1.9 to +17.6)
CODA-Promptshared prompt components updated across tasks8 / 8 (24/24 runs)+8.4 (range +2.3 to +15.4)
EASEper-task adapters (readout: lstsq + maha_e)6 / 8+1.9 (+8.3 on ImageNet-A, T=20)

The adapters and prompts are not retrained, yet accuracy rises by up to 15–18 points. For example, on CUB-200 (T=20) the swap raises InfLoRA from 71.76% to 86.23%, within 0.45 points of EASE's native accuracy. Some methods already re-train their readout on features sampled from stored class Gaussians (classifier alignment: GLAM, SLCA, HiDe-LoRA). A once-fitted head improves these only on CIFAR-100, because their readout is already a learned drift correction. Removing only that alignment step from GLAM lowers its accuracy in all 8 settings (mean −11.4 points). In both groups, therefore, the readout carries a first-order share of the accuracy that is usually credited to expert training.

GR2. Classifier capacity. Our original ridge head used 1,000 random features. With 10,000 features and regularization selected on held-out validation data, ridge-10k + maha_e reaches 79.40%, against 76.50% for ridge-1k. On the same retrained experts it is 1.47 points above lstsq + maha_e (77.92%), and higher in 20 of 24 runs. The random projection is fixed and needs no gradient training. We will correct the lstsq > ridge ranking, which held only for the undersized ridge head.

GR3. Additional baselines under our protocol (mean over the 8 settings, seed-shuffled class orders):

Ours (lstsq + maha_e)RanPACSLCAHiDe-Prompt†GLAMHiDe-LoRACODA-Prompt
78.1276.8876.2775.7873.8373.1667.33

† HiDe-Prompt uses its official fixed class order. Strengths depend on the benchmark: HiDe-Prompt leads on CIFAR-100, RanPAC on CUB-200, SLCA and HiDe-Prompt on ImageNet-R, and ours on ImageNet-A.

GR4. Backbones. The main paper already evaluates three backbones with the head and inference rule held fixed: ImageNet-21K ViT-B/16, CLIP ViT-B/16 and DINOv2 ViT-B/14, all at 224 px (Sec. 5.2, Fig. 1). The best backbone depends on the benchmark. DINOv2 beats the ImageNet-21K ViT by 6.2 points on ImageNet-R and 12.3 points on ImageNet-A (T=10), but is slightly weaker on CIFAR-100 and CUB-200. This backbone effect is larger than most gaps between published methods. For this rebuttal we added an ImageNet-1K ViT-B/16. On it, Mahalanobis ensembling remains better than entropy ensembling. The cross-method comparison deliberately uses a single backbone, so that differences between methods are not confounded with differences between backbones.

** GR5 Merging Baselines **

Merge ruleCIFAR T=10CIFAR T=20IN-R T=10IN-R T=20
linear88.86 ± 0.2386.36 ± 0.1374.64 ± 0.2671.45 ± 0.62
ties89.48 ± 0.0488.03 ± 0.2874.53 ± 0.5372.36 ± 0.70
dare88.87 ± 0.1686.28 ± 0.1174.64 ± 0.2671.37 ± 0.58
slerp89.63 ± 0.1687.17 ± 0.2674.39 ± 0.1971.07 ± 0.57
concat75.47 ± 2.5579.76 ± 1.6152.97 ± 2.2455.58 ± 0.85

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.