acceptodds
Under review as a conference paper at ICLR 2027

ProtoTTA: Explainable Test-Time Adaptation via Prototype Guidance

Abstract

Deep networks that rely on prototypes—interpretable representations that can be related to the model input—have gained significant attention for balancing high accuracy with inherent interpretability, which makes them suitable for critical domains such as healthcare. However, their prototypes are limited by their reliance on training data, which hampers their robustness to distribution shifts. While test-time adaptation (TTA) improves the robustness of deep networks by updating parameters and statistics, the prototypes of interpretable models have not been explored for this purpose. We introduce ProtoTTA, a general framework that minimizes the binary entropy of prototype similarities to encourage confident, prototype-specific activations, while adaptively balancing this objective with output entropy based on a label-free estimate of prototype reliability. We stabilize these updates by geometric filtering and weighting by prototype importance and model confidence. Experiments across four prototypical backbones on four benchmarks in fine-grained vision, histopathology, and NLP demonstrate that ProtoTTA improves robustness over standard output entropy minimization while restoring the correct grounding of prototypes. We also introduce novel interpretability metrics and a vision-language model (VLM) evaluation framework to quantify and explain our test-time updates, finding positive associations between prototype alignment and VLM-rated reasoning quality. Code will be released upon publication.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, prototypical networks, prototype-based adaptation, interpretability and explainable AI, case-based reasoning, distribution shift, entropy minimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WAbkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TTA for prototype-based interpretable models under dist. shift. ProtoTTA maps proto similarities to a proper range,adds a proto-level binary entropy loss as intermediate supervision, and mixes it w/ output entropy min adaptively based on unlabeled reliability signals. plus reliable sample filtering, grad norm, restricted param updates. For ProtoLens (shared protos) they use head-derived importance weights + soft targets -> weighted BCE.

Exps on ProtoViT/CUB-200-C, ProtoPNet/SICAPv2-C, ProtoPFormer/Dogs-C, ProtoLens/Amazon-C. besides acc: activation consistency, correct-class proto alignment, pred stability, VLM scores on evidence boards.

Critical strengths

Most convincing for me is Tab 4 (ProtoViT/CUB): output ent only 55.92, proto obj only 60.33, full 60.15. This supports the core idea that prototype-level signals provide useful adaptation supervision beyond output entropy alone, not just “TTA helps”.

covers several prototype architectures (ProtoViT and ProtoPFormer for image classification, ProtoPNet for histopathology, and ProtoLens for text classification with prototypes shared across classes). adaptive ProtoTTA beats strongest non-ProtoTTA baseline on avg in all 4, though some margins small. evidence boards (the paper’s reasoning boards showing the corrupted input, predicted-class prototype matches with spatial focus, and the strongest prototype contributions across classes; Appendix E.1) + consistency metrics give a useful framework for asking if adaptation changes the evidence used.

Critical weaknesses
  1. Controller value not isolated

The full method combines two lambda estimates—the activation-margin coefficient λ^act (Eq. 4) and the relative-evidence coefficient λ^rel (Eq. 5)—with routing based on gradient agreement between two interleaved halves of the reliable samples (Eq. 6; Appendix L, Eqs. 52–54), a coverage guard (Appendix L, Eq. 56), and gradient normalization (Eqs. 2–3). The ablations vary fixed lambda, filtering, and adapted parameters (Table 4), and aggregation and weighting (Appendix I, Table 8), but do not isolate these controller components: fixing lambda bypasses the estimates and routing together, while the coverage guard and gradient normalization are not separately ablated. The fixed-lambda sweep (Table 4, right) shows that both signals matter across backbones, but does not show that routing beats a simpler label-free combination, such as using λ^act alone or λ^rel alone to mix the same two losses.

This matters because ProtoPFormer reaches 57.43 with output-only adaptation versus ~57.4 with the full method, and ProtoPNet 55.97 versus ~56.2 (Tables 4 and 1), while the full method’s throughput in Table 2 is only 14.9% / 11.1% relative to the corresponding unadapted models. So what does the extra computation buy? (I am not saying that small gains warrant rejection; the concern is that accuracy is essentially unchanged on ProtoPFormer and only slightly higher on ProtoPNet, while the current ablations do not establish the controller’s own benefit relative to its added computation.)

Also README versus paper: Appendix L says lambda can reach 0 and 1 and ProtoViT’s two loss terms share the same confidence weight, but the README ProtoViT command keeps the default [0.05,0.95] clipping range and does not set --proto-shared-confidence-weighting. This is probably fine, but please give the exact commands and configurations for the main tables.

  1. Interp metrics vs “correct grounding”

PCA-W = the fraction of weighted evidence from the most activated prototypes that supports the true class, PAC = activation consistency. neither checks localization or semantic match,e.g. 2 models could score the same while one looks at bird parts and the other at background. useful but doesnt show grounding is restored.

VLM: Tab3 overall 3.78 vs 3.75 (vs EATA), with no confidence interval for the paired score difference. The within-method correlation analysis subtracts each method’s mean from both PCA-W and VLM scores, but does not control for prediction correctness, sample difficulty, or corruption type. These factors could affect both scores, so accounting for them would help determine whether the association reflects better spatial/semantic grounding or partly reflects easier samples, correctly classified samples, or particular corruption types.

Code: vlm_eval.py call_vlm_for_part_a (lines 1463-1465) passes the predicted class, ground-truth label and CORRECT/WRONG status to the VLM; failure_detection.py has an entry point that hides the ground-truth label. Appendix E.3 says that the correlation analysis on 650 images withholds ground-truth labels and prediction correctness from the VLM, so please clarify which entry point, prompt and version each part of Table 3 used. Not claiming leakage, just two pipelines exist and it should be made explicit.

Questions for authors

Q1. full configs for main tables (ProtoViT lambda clip, conf weighting)? for Tab3, script version/prompt/board processing for the 100-img scoring and 650-img corr? is the GT-exposing entry only for other exps?

Q2. same params/filtering/optim — lambda^act only vs lambda^rel only vs full routing, acc + runtime? how were thresholds picked, any test labels used? (just checking existing contribution, no new tasks needed)

Q3. restricted to samples w/ same pred or matching correctness, does VLM advantage and PCA-W–VLM corr still hold? CIs for main VLM paired diffs, and pls separate “stronger correct-class evidence” vs “better spatial/semantic grounding”.

5 comments
AuthorsOpenReview ↗

Response to Reviewer WAbk (Part 1/3)

We thank the reviewer for recognizing that prototype-level signals provide useful adaptation supervision beyond output entropy across several prototype architectures. We especially appreciate the observation that our reasoning boards and consistency metrics provide a useful framework for asking whether adaptation changes the evidence a model uses, not only its accuracy. This is our central claim: interpretable prototype signals can both guide adaptation and reveal how it changes the evidence behind predictions.

In brief:

  • A component ablation on all four backbones shows that the controller avoids poor weighting choices (W1, Q2).
  • On ProtoViT, a calibrate-once mode runs at nearly the speed of EATA while staying more accurate (W1, Q2).
  • On ProtoViT, the gain over EATA remains in three stream settings when hyperparameters are re-selected in different ways (Q2).
  • We give the exact configuration used for the main tables and updated the supplementary code (Q1).
  • An evaluation with CUB’s human part annotations shows that ProtoTTA’s prototypes stay on the same bird parts more often than EATA’s, which we separate from correct-class evidence (W2, Q3).
  • We moved Table 3 to the appendix and replaced it with blinded comparisons by 26 participants and two VLM judges; we also validated the narrator against the model’s internals (W2, Q1, Q3).

Below, we address each point in detail. Revisions are marked in blue at the manuscript locations listed in each heading.


W1 & Q2: Controller ablation (Section 5, Table 3, Appendix N)

The extra computation buys robustness across backbones: every other variant falls about one point or more behind the best one on some backbone, whereas the full controller does not. We isolated each controller component, keeping the filter, updated parameters and optimizer fixed (3 seeds): λact\lambda^{\mathrm{act}} or λrel\lambda^{\mathrm{rel}} alone mix the same two losses without routing, and the coverage guard and gradient normalization are removed one at a time. The last column reports the largest accuracy shortfall from the best variant shown on each backbone; identifying that best variant requires labels.

VariantProtoViTProtoPFormerProtoLensProtoPNetLargest shortfall
Router (ours)60.1557.4481.3156.180.18
λact\lambda^{\mathrm{act}} only60.3256.9981.3152.633.58
λrel\lambda^{\mathrm{rel}} only56.5557.4178.5556.213.78
Mean of the two59.9957.2580.7255.240.97
No coverage guard60.1257.3478.5656.182.76
No gradient normalization59.3857.4378.4456.052.88
Fixed λ=0.5\lambda=0.559.8956.9980.2656.161.06
Fixed λ=0\lambda=0 (output only)55.9257.4378.4455.974.41
Fixed λ=1\lambda=1 (prototype only)60.3355.7781.3251.704.51

The router stays within 0.18 points of the best variant on all four backbones, and it is the only variant within 0.2 points everywhere. These results support the controller's role in avoiding poor weighting choices across backbones, while showing that some components add little on individual models.

W1 & Q2: Speed (Section 5, Tables 2 and 13, Appendix N)

We agree that the full controller is slower, but on ProtoViT a fixed λ or our calibrate-once mode removes much of the overhead when speed matters. Most of this overhead comes from routing, which needs four extra backward passes per batch to choose λ without labels (7 in total, versus 3 for λact\lambda^{\mathrm{act}} or λrel\lambda^{\mathrm{rel}} alone). In absolute terms this cost is small: on one B200 GPU, the full controller adds about 2.4 to 3.3 ms per image on the three vision backbones. When speed matters, λ can be fixed, or our new calibrate-once mode can freeze the routing decision and gradient scales after the first reliable batch, so that each later batch needs a single backward pass, as in EATA (no new hyperparameter, not re-tuned). We timed all methods on ProtoViT five times in a single job on the same GPU over three corruptions, and report accuracy over all 13:

Method (ProtoViT, CUB-200-C)Speed relative to EATAAccuracy (%)
ProtoTTA, adaptive λ66.7%60.1
ProtoTTA, fixed λ = 1 (best in the Table 10 test sweep)86.1%60.3
ProtoTTA, calibrate-once98.2%60.0
EATA100%59.0

Calibrate-once runs at almost the speed of EATA while staying about one point more accurate. EATA's extra pass costs time, and ProtoTTA doesn't need one: unlike EATA, which estimates its Fisher regularizer from an additional set of approximately 500-2,000 samples, it relies only on the incoming test batches.

Continued in Comment 2.

AuthorsOpenReview ↗

Response to Reviewer WAbk (Part 2/3)

Q2: Hyperparameter selection (Sections 4 and 5, Appendices L and M)

We selected the hyperparameters of all methods, including ProtoTTA's filter thresholds, by a grid search on a single corruption, without a separate validation set, and then evaluated all methods on the 13 corruptions; labels were used in this selection but never during adaptation. This is now stated in Section 4 and Appendix L.

To check that our gains do not depend on this choice, we re-selected ProtoTTA's filter threshold and learning rate and EATA's own hyperparameters on ProtoViT with protocols in which selection and test data differ: 4-fold cross-validation over the noise, blur, weather and digital corruption families (in both directions) and a 20% validation split of the images, in three stream settings. The table reports ProtoTTA minus EATA in mean accuracy over the 13 corruptions (three seeds).

Test streamSelect on 3 families, test on 1Select on 1 family, test on 320% validation split
Shuffled+1.05+1.22+1.78
Mixed corruptions+1.06+1.45+1.13
Batch size 16+2.39+2.92+2.79
Average+1.50+1.87+1.90

ProtoTTA outperforms EATA under every selection protocol in all three streams, by 1.5 to 1.9 points on average, which suggests that its gain is robust to how the hyperparameters are selected.

W1 & Q1: Code and configurations (Appendix L)

The main tables use the configuration of Appendix L. The originally uploaded code was an early snapshot of our development, and its README does not reflect the final configurations. For ProtoViT, λ ranges over [0, 1] (--proto-lambda-min 0 --proto-lambda-max 1) and the two loss terms share the same confidence weight (--proto-shared-confidence-weighting). We have updated the supplementary code.

W2 & Q3: Interpretability metrics and grounding (Sections 3.5 and 5, Table 4, Appendix O)

CUB’s part annotations show that ProtoTTA keeps a more stable link between prototypes and bird parts than EATA, which we separate from the correct-class evidence that PCA-W measures. PCA-W and PAC were not designed to test localization. PAC measures how close the adapted model's prototype activations on a corrupted image stay to those of the model on the clean photo, which serves as an oracle for how the model reasons without shift. PCA-W measures whether the strongest prototype evidence supports the true class, that is, whether adaptation helps the model find patterns similar to those of the class it should predict. To test grounding at the finer level of bird parts, we used CUB's human annotations of 15 parts and the part-based scores of [1] (ICCV 2023) on all 5,794 test photos under 13 corruptions. “Consistency” here is the share of prototypes that identify the same annotated part in at least 80% of their class's images. “Stability” measures how often a true-class prototype covers the same annotated parts in the corrupted image as in the clean image. For rows 3-5, the reference is the unadapted model on the clean photo.

UnadaptedEATAProtoTTAProtoTTA − EATA [95% CI]
Part consistency (%)10.711.612.3+0.7 [0.4, 1.0]
Part stability (%)27.028.930.0+1.1 [1.0, 1.2]
Top-10 prototype overlap with clean (%)46.353.956.6+2.7 [2.5, 2.8]
True-class activation error vs. clean (lower is better)0.9190.7190.642-0.077 [-0.080, -0.074]
Same error, images where all three keep the clean prediction0.5130.3710.319-0.052 [-0.054, -0.050]

This separates the two notions the reviewer asks about. Correct-class evidence is what PCA-W measures: on ProtoViT, higher PCA-W means that a true-class prototype appears among the most activated more often, and ProtoTTA is higher there (Table 2). The part annotations show that ProtoTTA's prototypes stay on the same annotated part more often, across a class's images and under corruption, and its evidence stays closer to the clean photo's, even when all methods keep the clean prediction (last row). We therefore claim a more stable prototype-part link and recovery of clean-input evidence, not necessarily better localization.

Continued in Comment 3.

AuthorsOpenReview ↗

Response to Reviewer WAbk (Part 3/3)

W2 & Q1 & Q3: VLM evaluation (Section 5, Table 5, Appendices E.1, E.3 and P)

In blinded comparisons, humans and two VLM judges show a modest preference for ProtoTTA that is largest when both methods are correct, and the PCA-W–VLM association remains after controlling for correctness. In the revised paper, we moved Table 3 to Appendix E.1 (now Table 9) and replaced it in the main text with more comprehensive blinded pairwise comparisons: judges saw the five highest-contributing prototypes of ProtoTTA and EATA, without method names, class labels or correctness, and chose the better explanation or "no clear difference". We ran these comparisons with two VLM judges on 104 images and, in addition, with 26 human participants on 16 images. The table reports the share of decisive judgments that preferred ProtoTTA:

JudgeAll images [95% CI]Images where both methods are correct [95% CI]
Humans (26 participants)53.9% [38.1, 69.8] (16 images)63.3% [43.8, 80.7] (9 images)
Gemini 3.8 Flash56.1% [47.4, 65.1] (104 images)60.0% [47.4, 72.0] (52 images)
Qwen3.6-35B-A3B56.2% [48.8, 64.1] (104 images)62.6% [53.0, 71.6] (52 images)

Across all images, all three judges show a modest preference for ProtoTTA, although each confidence interval includes 50%. The margin is largest where accuracy cannot separate the methods: when both methods are correct, ProtoTTA is preferred in 60% to 63% of decisive judgments (humans: 100 times versus 58 for EATA, with 76 ties). Human and Gemini preferences were positively correlated across images (ρ=0.63\rho=0.63).

We also validated the VLM narrator against the model's internals. For 104 corrupted CUB images, two narrators compared the reasoning boards before and after adaptation under a neutral prompt that hid the true class, correctness, method name and which board was adapted, and we scored their answers to fixed questions against the model's own records:

What the narrator reportsQwen3.6-35B-A3BGemini 3.8 Flash
whether the retrieved training examples changed (accuracy)89%100%
whether evidence for competing classes rose or fell (accuracy)99%100%
whether the model's focus moved (AUC against the measured shift)0.770.89

For Q3, we checked whether VLM judgments track PCA-W once correctness is accounted for. In the 650-image correlation analysis (label-blind prompt, Qwen3.6-35B-A3B as judge), the PCA-W-VLM association remains after controlling for correctness, corruption type and method (partial Spearman 0.29-0.33) and among correctly classified images (ρ\rho = 0.36-0.43). In an additional analysis of the blinded 104-image comparison, Gemini's preference between ProtoTTA and EATA follows their PCA-W difference (Spearman ρ\rho = 0.26 [0.06, 0.43]), also when we exclude images where the two methods differ in correctness (ρ\rho = 0.25 [0.05, 0.43]). Together, these suggest that PCA-W relates to VLM-rated explanation quality and not only to correctness; we rely on the blinded comparisons above and the part annotations (W2) to compare methods' interpretability.


Thank you again for your constructive feedback. We hope our clarifications and new results provide a clearer perspective on our work and are taken into consideration in your final assessment.

References

[1] Huang et al. "Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks." ICCV 2023.

Reviewer_whHgOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on how prototype-based explainability is impacted by test-time adaptation. Authors provide a set of tricks for updating the prototypes as the model adapts, including updating only a reliable set, encouraging decisiveness in the scores, and aggregating signal with top-k. They identify prototype and output entropy as good signals, and propose an adaptive strategy to balance them during training. The also choose to update only the normalization parameters.

They evaluate accuracy when using their methods in conjunction with several prototype-based methods, comparing with other prototype adaptation strategies. They also propose a few metrics that measure how consistent the prototypes stay and how fast it runs.

Critical strengths

None

Critical weaknesses

W1. Unclear impact. The overall impact and takeaways from this work are unclear to me, for the following reasons:

  1. The work reads more as an accumulation of training tricks, many of which have been already explored (e.g. training only with high-certainty labels [1], that encouraging decisiveness in TTA can be helpful in the short-run [2]). Still, scientific takeaways are unclear to me.

  2. The tasks solved here are quite out-of-date. The datasets are reasonably small and the adaptations rather simple (e.g. blur), so given modern resources, most would choose to just retrain the model rather than use a TTA. Because it is only tested on these simple datasets, it is not clear that the findings here would be applicable to the more challenging TTA problems studied today.

  3. When comparing speed (Table 2), it looks like this method significantly decreases speed while only marginally increasing accuracy (e.g. ProtoPNet --> EATA vs ProtoTTA). Again, this detriments the potential impact of the work.

  4. Due to a lack of user study, it is impossible to understand how the method impacts how user-friendly the interpretations are.

  5. The related works section misses situating this work among many key prototype-based methods, including [3,4,5]. This makes it challenging to fully assess how the work fits into the field of prototype-based explanations.

W2. Lack of soundness in experimental validation. I have multiple concerns about the soundness of the evaluation, including:

  1. The evaluation of how this method impacts the interpretability of the prototypes is lacking. It is not clear to me how the chosen metrics directly correlate with interpretability (see Q2). There is also no user study measuring interpretability directly, which is typically considered the best way to evaluate this.

  2. While the VLM-based explanation framework is considered a contribution and used to measure the method, it is never validated. Hence, it is not clear how accurate this framework is (also see Q1).

  3. It is unclear to me whether the settings were chosen based on test-accuracy, of if a validation set was involved (a validation set was not discussed). If only the test-accuracy were used, this would undermine the generalizability of the results shown.

  4. Across the board, the improvements are quite marginal. This is especially concerning, given that it is not clear how this method would generalize to more modern problems (see W1.2).

References:

[1] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels, Han et al., NeurIPS'18.

[2] The Entropy Enigma: Success and Failure of Entropy Minimization, Press et al., ICML'24.

[3] Icicle: Interpretable class incremental continual learning, Rymarczyk et al, ICCV'23.

[4] Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification, Rymarczyk et al, ACM SIGKDD Conference on Knowledge Discovery & Data Mining'21.

[5] Interpretable image classification with differentiable prototypes assignment, Rymarczyk et al, ECCV'22

Questions for authors

Q1. Which VLM is used?

Q2. Could you please explain why PAC, PCA-W, and prediction stability act as interpretability metrics? To my understanding, they measure that the prototypes do not change very much as the model adapts, but it is not clear to me why this is necessary or tied directly to interpretability.

6 comments
AuthorsOpenReview ↗

Response to Reviewer whHg (Part 1/3)

We thank the reviewer for the detailed feedback and for summarizing our focus on how test-time adaptation affects prototype-based explanations. Our central claim is that interpretable prototype signals can both guide adaptation and reveal how it changes the evidence behind predictions.

In brief:

  • An objective-by-filter ablation on ProtoViT shows that the gain comes from the prototype objective, not only from the filter (W1.1).
  • Our datasets are the standard, large benchmarks for part-prototype networks (W1.2).
  • On ProtoViT, a calibrate-once mode runs at nearly the speed of EATA while staying more accurate (W1.3).
  • A blinded study with 26 participants shows that, when both methods are correct, participants more often prefer ProtoTTA’s explanations (W1.4).
  • We now discuss ProtoPShare, ProtoPool and ICICLE in the related work (W1.5).
  • An evaluation with CUB’s human part annotations shows that ProtoTTA restores prototype evidence more closely than EATA (W2.1, Q2).
  • We specify the VLMs used and validated the narrator against the model’s internals (W2.2, Q1).
  • On ProtoViT, the gain over EATA remains in three stream settings when hyperparameters are re-selected in different ways (W2.3, W2.4).

Below, we address each point in detail. Revisions are marked in blue at the manuscript locations listed in each heading.


W1.1: Contribution beyond individual techniques (Section 5, Table 3)

We respectfully disagree that ProtoTTA is an accumulation of training tricks: its core is a new adaptation signal from the prototype layer, and the gain comes from this signal, not only from the filter. We agree that sample selection [1] and entropy minimization [2] are established ideas. Our contribution is to use the prototype layer of an interpretable network as the adaptation signal, which supervises parameter directions to which output entropy is locally insensitive (Proposition 1). Our ablation separates this objective from the filter: with the same filter, adding the prototype objective raises accuracy from 55.92% to 60.15%, whereas our filter with output entropy alone stays below EATA (59.0%). The takeaway is twofold: interpretable models offer a label-free adaptation signal beyond the output, and their prototypes let us measure whether adaptation restores the evidence behind a prediction, not only the prediction itself (W2.1).

W1.2: Benchmarks (Section 4, Appendix A)

We respectfully disagree that the tasks are out of date: we chose benchmarks that suit both prototype networks and test-time adaptation. Most prototype networks (even three works the reviewer cites [3, 4, 5]) evaluate on CUB-200-2011, since fine-grained classes differ in local parts, which is exactly where prototypes are meaningful, and CUB's part annotations let us test whether explanations recover (W2.1). Our corruptions follow ImageNet-C [6], the benchmark on which most TTA baselines are evaluated, at severity 5, the hardest level. The datasets are not small: CUB-200-C and Stanford Dogs-C contain 75,322 and 111,540 test images over 13 corruptions, with 200 and 120 fine-grained classes. We also include a medical benchmark (SICAPv2 prostate histopathology) and an NLP benchmark that combines a cross-dataset shift with corruption: ProtoLens is trained on Yelp reviews and adapted on corrupted Amazon reviews (Amazon-C). The task is far from simple: corruption drops ProtoViT from 86.1% to 51.9% on CUB-200, ProtoPFormer from 82.4% to 52.8% on Stanford Dogs, ProtoPNet from 63.4% to 31.4% on SICAPv2, and ProtoLens from 94.0% on Yelp to 80.8% on Amazon-C.

W1.3: Speed (Section 5, Tables 2 and 13, Appendix N)

We agree that the full controller is slower, as reported in Table 2, but on ProtoViT a fixed λ or our calibrate-once mode removes much of this overhead when speed matters. Most of this overhead comes from the extra gradient computations the controller uses to set λ without labels. When speed matters, λ can be fixed, or our new calibrate-once mode can freeze the routing decision and gradient scales after the first reliable batch, so that each later batch needs a single backward pass, as in EATA (no new hyperparameter, not re-tuned). We timed all methods on ProtoViT five times in a single job on the same GPU over three corruptions, and report accuracy over all 13:

Method (ProtoViT, CUB-200-C)Speed relative to EATAAccuracy (%)
ProtoTTA, adaptive λ66.7%60.1
ProtoTTA, fixed λ = 1 (best in the Table 10 test sweep)86.1%60.3
ProtoTTA, calibrate-once98.2%60.0
EATA100%59.0

Calibrate-once runs at almost the speed of EATA while staying about one point more accurate. These timings do not include EATA's Fisher estimation pass over about 500 test images before adaptation, which adds considerable time.

Continued in Comment 2.

AuthorsOpenReview ↗

Response to Reviewer whHg (Part 2/3)

W1.4: User study (Section 5, Table 5, Appendix P)

We added a blinded user study: when both methods are correct, participants more often prefer the explanations of ProtoTTA. In this study, 26 participants, most of them graduate students who do not work in this research area, compared the five highest-contributing prototypes of ProtoTTA and EATA on 16 corrupted CUB images, without method names or class labels. Participants chose ProtoTTA's explanation in 53.9% of decisive judgments [95% CI: 38.1, 69.8], and on 104 further images Gemini 3.8 Flash and Qwen3.6-35B-A3B did so in 56.1% [47.4, 65.1] and 56.2% [48.8, 64.1]. The margin is largest where accuracy cannot separate the methods: on the nine images where both methods were correct, participants preferred ProtoTTA 100 times versus 58 for EATA (63% of decisive judgments), with 76 ties. Human and Gemini preferences were positively correlated across images (ρ=0.63\rho=0.63). Across all images, these preferences alone do not establish a clear overall preference, so the stronger evidence comes from CUB's human part annotations (W2.1), where ProtoTTA's prototypes stay on the same annotated bird parts more often than EATA's.

W1.5: Related work (Section 2)

We thank the reviewer for these references, which we now discuss in the related work (Section 2). ProtoPShare [4] and ProtoPool [5] share prototypes across classes to make prototype networks more compact, and ICICLE [3] addresses class-incremental continual learning, which preserves prototype interpretability as new labeled classes are added. All three belong to the prototype-network family but are training-time methods that use labels; none of them addresses test-time adaptation, where a trained model adapts to unlabeled, shifted test data. They are therefore not baselines for our method, and their absence does not affect our comparisons.

W2.1 & Q2: Interpretability Metrics (Sections 3.5 and 5, Table 4, Appendix O)

Our metrics ask whether adaptation restores the evidence behind a prediction, not only the prediction itself, and CUB’s human part annotations now show that ProtoTTA restores this evidence more closely than EATA. Adaptation changes how a test image is matched to fixed prototypes in these networks. This matching is the decision process itself: the prediction is computed directly from the prototype activations, so the prototypes behind a prediction are a faithful account of the decision. Because CUB-200-C is built by corrupting test photos, we also have each clean photo, and the unadapted model's explanation on it shows how the model explains that image without shift (diagnostic oracle). Corruption breaks this explanation: on a noisy image, the model may match prototypes on background texture instead of the wing and breast patterns it used on the clean photo.

The three metrics differ in what they need. PAC and Prediction Stability compare the adapted model on the corrupted image with the unadapted model on the clean photo: PAC measures how close their prototype activations are, and Prediction Stability measures whether their predictions agree. PCA-W needs no clean photo: on the corrupted image alone, it measures whether the strongest prototype evidence belongs to the true class, that is, whether the prediction rests on the right evidence. Like accuracy, it needs only test labels, so it also applies to real shifts where no clean image exists as a diagnostic oracle.

To measure interpretability against human ground truth, we used CUB's annotations of 15 bird parts and the part-based scores of [7] (ICCV 2023) on all 5,794 test photos under 13 corruptions. Consistency is the share of prototypes that identify the same annotated part in at least 80% of their class's images. Stability measures how often a true-class prototype covers the same annotated parts in the corrupted image as in the clean image.

UnadaptedEATAProtoTTAProtoTTA − EATA [95% CI]
Part consistency (%)10.711.612.3+0.7 [0.4, 1.0]
Part stability (%)27.028.930.0+1.1 [1.0, 1.2]
True-class activation error vs. clean (lower is better)0.9190.7190.642-0.077 [-0.080, -0.074]
Same error, images where all three keep the clean prediction0.5130.3710.319-0.052 [-0.054, -0.050]

Corruption reduces consistency from 20.0% to 10.7%, and ProtoTTA restores 1.6 points compared with EATA's 0.9. The last row isolates a difference that accuracy cannot capture: even when all three methods keep the clean prediction, ProtoTTA reduces activation error by 38%, compared with 28% for EATA.

Continued in Comment 3.

AuthorsOpenReview ↗

Response to Reviewer whHg (Part 3/3)

W2.2 & Q1: Validating the narrator (Section 5, Table 5, Appendices E.1, E.3 and P)

We validated the narrator against the model’s own records: both VLM narrators reliably report how adaptation changes the retrieved prototypes, the competing evidence and the model’s focus. Table 3 of the submission (now Table 9 in Appendix E.1) was scored by Qwen3-VL-32B-Thinking, and the 650-image correlation analysis (Appendix E.3) used Qwen3.6-35B-A3B. New VLM comparative judging is reported above in W1.4. To validate the narrator, we tested whether what it says about adaptation matches what actually changed inside the model. For 104 corrupted CUB images, we showed two narrators the reasoning boards before and after adaptation, with a neutral prompt that hid the true class, correctness, method name and which board was adapted. Each narrator answered fixed questions about what changed, and we scored its answers against the model's own records:

What the narrator reportsQwen3.6-35B-A3BGemini 3.8 Flash
whether the retrieved training examples changed (accuracy)89%100%
whether evidence for competing classes rose or fell (accuracy)99%100%
whether the model's focus moved (AUC against the measured shift)0.770.89

To our knowledge, this is the first narrator that explains test-time adaptation in natural language and can be validated against the model's internals.

W2.3 & W2.4: Hyperparameter selection, robustness and size of the gains (Sections 4 and 5, Appendices L and M)

Our gain over EATA remains when the hyperparameters are re-selected in three different ways. In the paper, we selected the hyperparameters of all methods by a grid search on a single corruption, without a separate validation set, and then evaluated all methods on the 13 corruptions. This is now stated in Section 4 and Appendix L. To check that our gains do not depend on this choice, we re-selected ProtoTTA and EATA's hyperparameters on ProtoViT with protocols in which selection and test data differ: 4-fold cross-validation over the noise, blur, weather and digital corruption families (in both directions) and a 20% validation split of the images. We did so in three stream settings that also test robustness. The table reports ProtoTTA minus EATA in mean accuracy over the 13 corruptions (three seeds).

Test streamSelect on 3 families, test on 1Select on 1 family, test on 320% validation split
Shuffled+1.05+1.22+1.78
Mixed corruptions+1.06+1.45+1.13
Batch size 16+2.39+2.92+2.79
Average+1.50+1.87+1.90

ProtoTTA outperforms EATA under every selection protocol in the three streams shown, by 1.5 to 1.9 points on average and by up to 2.9 points with small batches. We agree that the margin over the strongest baseline is smaller on some backbones. However, ProtoTTA's accuracy gain is only part of our contribution. Our main message is that the internal representations of interpretable networks are useful for adaptation: the prototype layer provides a label-free signal beyond the model output, and it lets us check whether adaptation restores the evidence behind predictions (W2.1), which accuracy alone does not reveal. ProtoTTA also needs no extra data before adaptation: unlike EATA, which estimates its Fisher regularizer from an additional set of approximately 500-2,000 samples, it relies only on the incoming test batches.


Thank you again for your constructive feedback. We hope our clarifications and new results provide a clearer perspective on our work and are taken into consideration in your final assessment.

References

[1] Han et al. "Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels." NeurIPS 2018.

[2] Press et al. "The Entropy Enigma: Success and Failure of Entropy Minimization." ICML 2024.

[3] Rymarczyk et al. "ICICLE: Interpretable Class Incremental Continual Learning." ICCV 2023.

[4] Rymarczyk et al. "ProtoPShare: Prototypical Parts Sharing for Similarity Discovery in Interpretable Image Classification." KDD 2021.

[5] Rymarczyk et al. "Interpretable Image Classification with Differentiable Prototypes Assignment." ECCV 2022.

[6] Hendrycks and Dietterich. "Benchmarking Neural Network Robustness to Common Corruptions and Perturbations." ICLR 2019.

[7] Huang et al. "Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks." ICCV 2023.

Reviewer_qFPoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper adapts prototype networks, interpretable classifiers that explain predictions by matching image regions to learned part templates, to corrupted test data without labels. On top of the standard approach of making the final prediction more confident, it also pushes each predicted-class prototype's match score toward 0 or 1, mixes the two losses with a per-image weight chosen by a hand-designed controller, and updates only on images that pass a prototype-based reliability filter. It is tested on four prototype models (birds, dogs, prostate pathology, and Yelp-to-Amazon reviews) against Tent, EATA, SAR and MEMO, with new metrics and a VLM judge for whether the explanations recover.

Critical strengths
  1. It is the first test-time adaptation method aimed at part-prototype networks, as the paper argues, and it checks whether explanations recover, not only accuracy: on the bird data the share of top prototype evidence from the true class rises to 79.2 versus 77.7 for EATA.
  2. The evaluation is statistically careful for this area, with three seeds, paired tests with correction for multiple comparisons, four quite different models, and ablations that are reported honestly.
Critical weaknesses
  1. Little new beyond standard test-time adaptation, and the new loss is not what drives the gains. The method is the usual make-the-prediction-confident approach with one extra confidence term on the prototype scores, and it beats the best baseline by only 1.1, 0.3, 0.2 and 0.5 points on the four models. The paper's own sweep shows the prototype term clearly helps only on the bird model: on the dog model the best mix gives 57.47 versus 57.43 with the output loss alone, and on the pathology model using the prototype loss alone drops accuracy to 51.7. On the bird model, removing the reliability filter costs 3.7 points, more than the method's whole gain over EATA.
  2. The adaptive mixing rule is complex, weakly motivated and barely matters. One estimate measures how far prototype scores are from 0.5, which reflects saturation rather than correctness, the other is an ad hoc ratio of two margins, and a batch-level switch between them adds further hand-set thresholds. None of these parts is ablated separately, and together they add at most 0.4 points over a single fixed weight of 0.5 and never beat the best fixed weight.
  3. Much slower than existing methods for tiny gains, with per-model hand tuning. ProtoTTA runs at 11.1% and 14.9% of unadapted speed on the pathology and dog models, against 89.6% and 55.2% for EATA, for gains of 0.3 and 0.2 points, and the paper attributes the slowdown to the extra gradient computations, which the algorithm uses to set the mix between the two losses. The reliability filters and score mappings also differ for each model, with thresholds of 0.92, 0.55, 0.8 plus an entropy cut, and 0.1, and the paper does not say how they were chosen.
Questions for authors
  1. How does the plain output-confidence loss perform with your prototype-based filter, compared with EATA's entropy-based filter, which would show whether the gains come from the new loss or from the new filter?
  2. Can you report each part of the mixing controller separately (each estimate alone and with or without the switch), and the speed of ProtoTTA with a fixed weight, given that the controller adds at most 0.4 points?
  3. How were the per-model filter thresholds and score mappings chosen, and were test labels or test accuracy used at any point?
  4. Would the VLM-based quality scores, currently from one judge on 100 bird images with small differences such as 3.78 versus 3.75, agree with human ratings and hold on the other models?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qFPo (Part 1/3)

We thank the reviewer for recognizing the novelty of (i) test-time adapting part-prototype networks as representative of interpretable networks, and (ii) evaluating explanation recovery alongside accuracy, and for appreciating our statistical evaluation across four architectures. Our central claim is that interpretable prototype signals can both guide adaptation and reveal how it changes the evidence behind predictions.

In brief:

  • An objective-by-filter ablation on ProtoViT shows that both the prototype objective and the reliability filter contribute to the gain (W1).
  • An evaluation with CUB’s human part annotations shows that ProtoTTA restores prototype evidence more closely than EATA (W1).
  • A component ablation on all four backbones shows that the controller avoids poor weighting choices (W2).
  • On ProtoViT, a calibrate-once mode runs at nearly the speed of EATA while staying more accurate (W3).
  • On ProtoViT, the gain over EATA remains in three stream settings when hyperparameters are re-selected in different ways (Q3).
  • A blinded study with 26 participants and two VLM judges shows that Gemini’s preferences correlate with human ratings (Q4).

Below, we address each point in detail. Revisions are marked in blue at the manuscript locations listed in each heading.

W1 & Q1: Separating the prototype objective from the reliability filter (Section 5, Tables 3 and 4, Appendix O)

Both the prototype objective and the reliability filter contribute to the gain, and the filter alone does not explain our advantage. To separate their effect, we crossed two objectives with three filtering settings on ProtoViT/CUB-200-C, with other settings fixed (3 seeds, 13 corruptions). “EATA rule” denotes EATA’s sample-selection rule applied within our adaptation setup.

Objective \ FilterNoneEATA ruleOurs
Output entropy only52.8350.3155.92
ProtoTTA objective (adaptive mix)56.4557.4260.15

Our filter with output entropy alone reaches 55.92, about three points below EATA. Adding the prototype objective improves accuracy under every filter, including without filtering (52.83 → 56.45). With our filter, the adaptive mix reaches 60.15, outperforming EATA (59.0). The filter also matters: it raises the adaptive objective’s accuracy by 3.7 points. With our filter, the prototype objective adds 4.2 points, more than the filter itself. Unlike EATA, ProtoTTA also needs no Fisher estimation on approximately 500-2,000 additional samples before adaptation.

Beyond accuracy, ProtoTTA better restores the model’s prototype evidence after corruption. We evaluated this on all 5,794 CUB test photos under 13 corruptions, using CUB’s human annotations of 15 bird parts and the two part-based interpretability scores from [1] (ICCV 2023). Consistency measures the share of prototypes that identify the same annotated part in at least 80% of their class’s images. Stability measures how often a true-class prototype covers the same annotated parts in the corrupted image as in the clean image. For stability and rows 3–5, the reference is the unadapted model on the same clean photo.

UnadaptedEATAProtoTTAProtoTTA − EATA [95% CI]
Part consistency (%)10.711.612.3+0.7 [0.4, 1.0]
Part stability (%)27.028.930.0+1.1 [1.0, 1.2]
Top-10 prototype overlap with clean (%)46.353.956.6+2.7 [2.5, 2.8]
True-class activation error vs. clean (lower is better)0.9190.7190.642-0.077 [-0.080, -0.074]
Same error, images where all three keep clean prediction0.5130.3710.319-0.052 [-0.054, -0.050]

ProtoTTA outperforms EATA on every measure shown. Across the full test set, it also leads on all 13 corruptions in part consistency and all eight pre-specified measures of closeness to the clean explanation (bootstrap over photos; over classes for consistency). For context, corruption reduces consistency from 20.0% to 10.7%; ProtoTTA restores 1.6 points, compared with EATA’s 0.9. The last row isolates a difference that accuracy cannot capture: even when all three methods retain the clean prediction, ProtoTTA reduces activation error by 38%, compared with 28% for EATA, so even when the prediction is unchanged, ProtoTTA's explanation stays closer to the oracle.

We agree that the prototype signal is not equally useful across backbones, as discussed in Section 5 (Role of the Interpolation Coefficient). The prototype representation affects how much adaptation should rely on it relative to the output. As shown in Table 3 and the W2 table, the benefit is not limited to the bird model: on ProtoLens, the prototype objective with our filter reaches 81.32, compared with 78.44 using output entropy alone. On ProtoPFormer, the two objectives perform similarly, and on ProtoPNet the gain is small. This is why ProtoTTA sets the weight from label-free signals instead of fixing it per backbone.

Continued in Comment 2.

AuthorsOpenReview ↗

Response to Reviewer qFPo (Part 2/3)

W2 & Q2: What each part of the controller measures and adds (Section 5, Table 3, Appendix N)

The controller aims to choose a reliable weight without labels. It is not designed to outperform the best fixed weight. This weight varies across backbones (λ=1\lambda=1 for birds, 0.2 for dogs, and 0.4 for pathology in Table 10), and selecting it through an accuracy-based sweep requires validation with a grid search during deployment. Proposition 2 motivates our design: the optimal mix of normalized gradients depends on their errors and covariance. Since these quantities cannot be measured without labels, the controller uses observable proxies informed by this analysis.

First, λact\lambda^{\mathrm{act}} measures how far prototype scores are from the uncertain midpoint of 0.50.5. Binary entropy pushes scores below 0.50.5 down and scores above 0.50.5 up (Eq. 7). A small change can therefore reverse the update direction for a score of 0.520.52, while a score of 0.900.90 is less sensitive. This measures decisiveness, not correctness: a confident prototype may still fail to distinguish classes. Second, λrel\lambda^{\mathrm{rel}} asks whether prototypes distinguish the top two classes better than the output does. If prototypes give similar evidence for two dog breeds but the output is confident, it gives more weight to output entropy. The switch also checks how consistent each loss's gradients are across two half-batches. It selects λact\lambda^{\mathrm{act}} when prototype gradients are sufficiently stable relative to output gradients, and λrel\lambda^{\mathrm{rel}} otherwise. The coverage guard also selects λact\lambda^{\mathrm{act}} when output probabilities are near 00 or 11 but few samples pass the filter.

We isolate each component below, keeping the filter, updated parameters, and optimizer fixed (3 seeds). The last column reports the largest accuracy shortfall from the best variant shown across models. Identifying that best variant requires labels.

VariantProtoViTProtoPFormerProtoLensProtoPNetLargest shortfall
Router (ours)60.1557.4481.3156.180.18
λact\lambda^{\mathrm{act}} only60.3256.9981.3152.633.58
λrel\lambda^{\mathrm{rel}} only56.5557.4178.5556.213.78
Mean of the two59.9957.2580.7255.240.97
No coverage guard60.1257.3478.5656.182.76
No gradient normalization59.3857.4378.4456.052.88
Fixed λ=0.5\lambda=0.559.8956.9980.2656.161.06
Fixed λ=0\lambda=0 (output only)55.9257.4378.4455.974.41
Fixed λ=1\lambda=1 (prototype only)60.3355.7781.3251.704.51

The router stays within 0.18 points of the best variant on all four backbones, and it is the only variant within 0.2 points everywhere. These results support the controller's role in avoiding poor weighting choices across backbones, while showing that some components add little on individual models.

W3: Speed (Section 5, Tables 2 and 13, Appendix N)

We agree that the full controller is slower, but on ProtoViT a fixed λ or our calibrate-once mode removes much of the overhead when speed matters. Most of this overhead comes from the extra gradient computations the controller uses to set λ without labels, both to balance the scales of the two losses and to choose between its two estimates. In absolute terms, this overhead is small: on one B200 GPU, the full controller adds about 2.7 ms per image on ProtoPFormer and 3.3 ms on ProtoPNet. We introduced the adaptive λ for generalization, since how much adaptation should rely on prototypes differs across backbones. When a deployment needs more speed, λ can instead be fixed, either from what is known about the model's prototype representation or by checking a few values on a small validation set. We also added a calibrate-once mode, which keeps the adaptive λ but freezes the routing decision and gradient scales after the first reliable batch, so each later batch needs a single backward pass, as in EATA. It adds no new hyperparameter and was not re-tuned. For a fair comparison, we timed all methods on ProtoViT five times in a single job on the same GPU over three corruptions, and report mean speed relative to EATA alongside mean accuracy over all 13 corruptions.

Method (ProtoViT, CUB-200-C)Speed relative to EATAAccuracy (%)
ProtoTTA, adaptive λ66.7%60.1
ProtoTTA, fixed λ = 1 (best in the Table 10 test sweep)86.1%60.3
ProtoTTA, calibrate-once98.2%60.0
EATA100%59.0

Both options remove much of the overhead while keeping nearly the accuracy of the full controller, and calibrate-once runs at almost the speed of EATA while staying about one point more accurate than EATA in its published configuration. These timings do not include EATA's Fisher estimation pass over about 500 test images before adaptation, which adds considerable time.

Continued in Comment 3.

AuthorsOpenReview ↗

Response to Reviewer qFPo (Part 3/3)

Q3: How the per-model thresholds were chosen (Sections 4 and 5, Appendices L and M)

We selected the hyperparameters of methods by a grid search on a single corruption, and then evaluated all methods on the 13 corruptions. This specification is now in Section 4. How hyperparameters should be selected in TTA is still an open and debated question [2]. To check that our gains do not come from this choice, we repeated the selection on ProtoViT with three protocols in which selection and test data differ: 4-fold cross-validation over the noise, blur, weather and digital corruption families, run in both directions (select on three families and test on the fourth, or select on one and test on the other three), and a validation split (select on 20% of the images, test on the remaining 80%). In both, we re-selected ProtoTTA's filter threshold and learning rate and EATA's own hyperparameters from a search grid, in three different stream settings to also test robustness. The table reports ProtoTTA minus EATA in mean accuracy over the 13 corruptions, averaged over three seeds.

Test streamSelect on 3 families, test on 1Select on 1 family, test on 320% validation split
Shuffled+1.05+1.22+1.78
Mixed corruptions+1.06+1.45+1.13
Batch size 16+2.39+2.92+2.79
Average+1.50+1.87+1.90

The average gain over EATA is similar under all protocols and at least as large as the gap in Table 1, which suggests that the gain is not an artifact of how the hyperparameters were selected.

Q4: Agreement with human ratings, and other VLMs (Section 5, Table 5, Appendix P)

On the human-rated images, Gemini’s preferences correlate with those of the participants, and a second VLM judge gives a similar preference rate, although no judge establishes a clear overall preference between ProtoTTA and EATA. We conducted a blinded human study on 16 CUB images sampled randomly across corruption types and prediction-correctness groups. Participants compared each method’s five highest-contributing prototypes, without method names or class labels, choosing either explanation or “no clear difference.” Gemini 3.8 Flash evaluated the same cards in both orders.

Across 26 participants, ProtoTTA received 164 preferences, EATA 140, and 112 responses indicated no clear difference. ProtoTTA’s share of decisive judgments was 53.9% (95% CI: 38.1–69.8%). Human and Gemini image-level preferences were positively correlated (ρ=0.63\rho=0.63, 95% CI: 0.20–0.85), agreeing on 12 of 13 images with non-tied aggregate preferences. The breakdown reveals differences that accuracy alone cannot capture. On the nine images where both methods were correct, humans preferred ProtoTTA 100 times versus 58 for EATA, with 76 ties. The cards show only the top five prototypes, and for some participants distinguishing them might be difficult. We then evaluated 104 additional images, eight per corruption on VLMs:

VLM judgeProtoTTA preference among decisive judgments [95% CI]
Gemini 3.8 Flash56.1% [47.4, 65.1]
Qwen3.6-35B-A3B56.2% [48.8, 64.1]

These judges likewise do not establish a clear overall preference. The stronger evidence for explanation recovery comes from W1/Q1 we discussed above: ProtoTTA improves annotation-based part consistency and stability and restores prototype activations closer to their clean-image values, even when predictions remain unchanged. These findings support assessing the evidence behind predictions alongside accuracy, while distinguishing explanation recovery from human-perceived quality.


We thank the reviewer again for the constructive feedback. We hope our clarifications and new results provide a clearer perspective on our work and are taken into consideration in your final assessment.

References

[1] Huang et al. "Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks." ICCV 2023.

[2] Zhao et al. "On Pitfalls of Test-Time Adaptation." ICML 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.