- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TTA for prototype-based interpretable models under dist. shift. ProtoTTA maps proto similarities to a proper range,adds a proto-level binary entropy loss as intermediate supervision, and mixes it w/ output entropy min adaptively based on unlabeled reliability signals. plus reliable sample filtering, grad norm, restricted param updates. For ProtoLens (shared protos) they use head-derived importance weights + soft targets -> weighted BCE.
Exps on ProtoViT/CUB-200-C, ProtoPNet/SICAPv2-C, ProtoPFormer/Dogs-C, ProtoLens/Amazon-C. besides acc: activation consistency, correct-class proto alignment, pred stability, VLM scores on evidence boards.
Critical strengths
Most convincing for me is Tab 4 (ProtoViT/CUB): output ent only 55.92, proto obj only 60.33, full 60.15. This supports the core idea that prototype-level signals provide useful adaptation supervision beyond output entropy alone, not just “TTA helps”.
covers several prototype architectures (ProtoViT and ProtoPFormer for image classification, ProtoPNet for histopathology, and ProtoLens for text classification with prototypes shared across classes). adaptive ProtoTTA beats strongest non-ProtoTTA baseline on avg in all 4, though some margins small. evidence boards (the paper’s reasoning boards showing the corrupted input, predicted-class prototype matches with spatial focus, and the strongest prototype contributions across classes; Appendix E.1) + consistency metrics give a useful framework for asking if adaptation changes the evidence used.
Critical weaknesses
- Controller value not isolated
The full method combines two lambda estimates—the activation-margin coefficient λ^act (Eq. 4) and the relative-evidence coefficient λ^rel (Eq. 5)—with routing based on gradient agreement between two interleaved halves of the reliable samples (Eq. 6; Appendix L, Eqs. 52–54), a coverage guard (Appendix L, Eq. 56), and gradient normalization (Eqs. 2–3). The ablations vary fixed lambda, filtering, and adapted parameters (Table 4), and aggregation and weighting (Appendix I, Table 8), but do not isolate these controller components: fixing lambda bypasses the estimates and routing together, while the coverage guard and gradient normalization are not separately ablated. The fixed-lambda sweep (Table 4, right) shows that both signals matter across backbones, but does not show that routing beats a simpler label-free combination, such as using λ^act alone or λ^rel alone to mix the same two losses.
This matters because ProtoPFormer reaches 57.43 with output-only adaptation versus ~57.4 with the full method, and ProtoPNet 55.97 versus ~56.2 (Tables 4 and 1), while the full method’s throughput in Table 2 is only 14.9% / 11.1% relative to the corresponding unadapted models. So what does the extra computation buy? (I am not saying that small gains warrant rejection; the concern is that accuracy is essentially unchanged on ProtoPFormer and only slightly higher on ProtoPNet, while the current ablations do not establish the controller’s own benefit relative to its added computation.)
Also README versus paper: Appendix L says lambda can reach 0 and 1 and ProtoViT’s two loss terms share the same confidence weight, but the README ProtoViT command keeps the default [0.05,0.95] clipping range and does not set --proto-shared-confidence-weighting. This is probably fine, but please give the exact commands and configurations for the main tables.
- Interp metrics vs “correct grounding”
PCA-W = the fraction of weighted evidence from the most activated prototypes that supports the true class, PAC = activation consistency. neither checks localization or semantic match,e.g. 2 models could score the same while one looks at bird parts and the other at background. useful but doesnt show grounding is restored.
VLM: Tab3 overall 3.78 vs 3.75 (vs EATA), with no confidence interval for the paired score difference. The within-method correlation analysis subtracts each method’s mean from both PCA-W and VLM scores, but does not control for prediction correctness, sample difficulty, or corruption type. These factors could affect both scores, so accounting for them would help determine whether the association reflects better spatial/semantic grounding or partly reflects easier samples, correctly classified samples, or particular corruption types.
Code: vlm_eval.py call_vlm_for_part_a (lines 1463-1465) passes the predicted class, ground-truth label and CORRECT/WRONG status to the VLM; failure_detection.py has an entry point that hides the ground-truth label. Appendix E.3 says that the correlation analysis on 650 images withholds ground-truth labels and prediction correctness from the VLM, so please clarify which entry point, prompt and version each part of Table 3 used. Not claiming leakage, just two pipelines exist and it should be made explicit.
Questions for authors
Q1. full configs for main tables (ProtoViT lambda clip, conf weighting)? for Tab3, script version/prompt/board processing for the 100-img scoring and 650-img corr? is the GT-exposing entry only for other exps?
Q2. same params/filtering/optim — lambda^act only vs lambda^rel only vs full routing, acc + runtime? how were thresholds picked, any test labels used? (just checking existing contribution, no new tasks needed)
Q3. restricted to samples w/ same pred or matching correctness, does VLM advantage and PCA-W–VLM corr still hold? CIs for main VLM paired diffs, and pls separate “stronger correct-class evidence” vs “better spatial/semantic grounding”.
5 comments
Response to Reviewer WAbk (Part 2/3)
Q2: Hyperparameter selection (Sections 4 and 5, Appendices L and M)
We selected the hyperparameters of all methods, including ProtoTTA's filter thresholds, by a grid search on a single corruption, without a separate validation set, and then evaluated all methods on the 13 corruptions; labels were used in this selection but never during adaptation. This is now stated in Section 4 and Appendix L.
To check that our gains do not depend on this choice, we re-selected ProtoTTA's filter threshold and learning rate and EATA's own hyperparameters on ProtoViT with protocols in which selection and test data differ: 4-fold cross-validation over the noise, blur, weather and digital corruption families (in both directions) and a 20% validation split of the images, in three stream settings. The table reports ProtoTTA minus EATA in mean accuracy over the 13 corruptions (three seeds).
| Test stream | Select on 3 families, test on 1 | Select on 1 family, test on 3 | 20% validation split |
|---|---|---|---|
| Shuffled | +1.05 | +1.22 | +1.78 |
| Mixed corruptions | +1.06 | +1.45 | +1.13 |
| Batch size 16 | +2.39 | +2.92 | +2.79 |
| Average | +1.50 | +1.87 | +1.90 |
ProtoTTA outperforms EATA under every selection protocol in all three streams, by 1.5 to 1.9 points on average, which suggests that its gain is robust to how the hyperparameters are selected.
W1 & Q1: Code and configurations (Appendix L)
The main tables use the configuration of Appendix L. The originally uploaded code was an early snapshot of our development, and its README does not reflect the final configurations. For ProtoViT, λ ranges over [0, 1] (--proto-lambda-min 0 --proto-lambda-max 1) and the two loss terms share the same confidence weight (--proto-shared-confidence-weighting). We have updated the supplementary code.
W2 & Q3: Interpretability metrics and grounding (Sections 3.5 and 5, Table 4, Appendix O)
CUB’s part annotations show that ProtoTTA keeps a more stable link between prototypes and bird parts than EATA, which we separate from the correct-class evidence that PCA-W measures. PCA-W and PAC were not designed to test localization. PAC measures how close the adapted model's prototype activations on a corrupted image stay to those of the model on the clean photo, which serves as an oracle for how the model reasons without shift. PCA-W measures whether the strongest prototype evidence supports the true class, that is, whether adaptation helps the model find patterns similar to those of the class it should predict. To test grounding at the finer level of bird parts, we used CUB's human annotations of 15 parts and the part-based scores of [1] (ICCV 2023) on all 5,794 test photos under 13 corruptions. “Consistency” here is the share of prototypes that identify the same annotated part in at least 80% of their class's images. “Stability” measures how often a true-class prototype covers the same annotated parts in the corrupted image as in the clean image. For rows 3-5, the reference is the unadapted model on the clean photo.
| Unadapted | EATA | ProtoTTA | ProtoTTA − EATA [95% CI] | |
|---|---|---|---|---|
| Part consistency (%) | 10.7 | 11.6 | 12.3 | +0.7 [0.4, 1.0] |
| Part stability (%) | 27.0 | 28.9 | 30.0 | +1.1 [1.0, 1.2] |
| Top-10 prototype overlap with clean (%) | 46.3 | 53.9 | 56.6 | +2.7 [2.5, 2.8] |
| True-class activation error vs. clean (lower is better) | 0.919 | 0.719 | 0.642 | -0.077 [-0.080, -0.074] |
| Same error, images where all three keep the clean prediction | 0.513 | 0.371 | 0.319 | -0.052 [-0.054, -0.050] |
This separates the two notions the reviewer asks about. Correct-class evidence is what PCA-W measures: on ProtoViT, higher PCA-W means that a true-class prototype appears among the most activated more often, and ProtoTTA is higher there (Table 2). The part annotations show that ProtoTTA's prototypes stay on the same annotated part more often, across a class's images and under corruption, and its evidence stays closer to the clean photo's, even when all methods keep the clean prediction (last row). We therefore claim a more stable prototype-part link and recovery of clean-input evidence, not necessarily better localization.
Continued in Comment 3.
Response to Reviewer WAbk (Part 3/3)
W2 & Q1 & Q3: VLM evaluation (Section 5, Table 5, Appendices E.1, E.3 and P)
In blinded comparisons, humans and two VLM judges show a modest preference for ProtoTTA that is largest when both methods are correct, and the PCA-W–VLM association remains after controlling for correctness. In the revised paper, we moved Table 3 to Appendix E.1 (now Table 9) and replaced it in the main text with more comprehensive blinded pairwise comparisons: judges saw the five highest-contributing prototypes of ProtoTTA and EATA, without method names, class labels or correctness, and chose the better explanation or "no clear difference". We ran these comparisons with two VLM judges on 104 images and, in addition, with 26 human participants on 16 images. The table reports the share of decisive judgments that preferred ProtoTTA:
| Judge | All images [95% CI] | Images where both methods are correct [95% CI] |
|---|---|---|
| Humans (26 participants) | 53.9% [38.1, 69.8] (16 images) | 63.3% [43.8, 80.7] (9 images) |
| Gemini 3.8 Flash | 56.1% [47.4, 65.1] (104 images) | 60.0% [47.4, 72.0] (52 images) |
| Qwen3.6-35B-A3B | 56.2% [48.8, 64.1] (104 images) | 62.6% [53.0, 71.6] (52 images) |
Across all images, all three judges show a modest preference for ProtoTTA, although each confidence interval includes 50%. The margin is largest where accuracy cannot separate the methods: when both methods are correct, ProtoTTA is preferred in 60% to 63% of decisive judgments (humans: 100 times versus 58 for EATA, with 76 ties). Human and Gemini preferences were positively correlated across images ().
We also validated the VLM narrator against the model's internals. For 104 corrupted CUB images, two narrators compared the reasoning boards before and after adaptation under a neutral prompt that hid the true class, correctness, method name and which board was adapted, and we scored their answers to fixed questions against the model's own records:
| What the narrator reports | Qwen3.6-35B-A3B | Gemini 3.8 Flash |
|---|---|---|
| whether the retrieved training examples changed (accuracy) | 89% | 100% |
| whether evidence for competing classes rose or fell (accuracy) | 99% | 100% |
| whether the model's focus moved (AUC against the measured shift) | 0.77 | 0.89 |
For Q3, we checked whether VLM judgments track PCA-W once correctness is accounted for. In the 650-image correlation analysis (label-blind prompt, Qwen3.6-35B-A3B as judge), the PCA-W-VLM association remains after controlling for correctness, corruption type and method (partial Spearman 0.29-0.33) and among correctly classified images ( = 0.36-0.43). In an additional analysis of the blinded 104-image comparison, Gemini's preference between ProtoTTA and EATA follows their PCA-W difference (Spearman = 0.26 [0.06, 0.43]), also when we exclude images where the two methods differ in correctness ( = 0.25 [0.05, 0.43]). Together, these suggest that PCA-W relates to VLM-rated explanation quality and not only to correctness; we rely on the blinded comparisons above and the part annotations (W2) to compare methods' interpretability.
Thank you again for your constructive feedback. We hope our clarifications and new results provide a clearer perspective on our work and are taken into consideration in your final assessment.
References
[1] Huang et al. "Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks." ICCV 2023.
Response to Reviewer WAbk (Part 1/3)
We thank the reviewer for recognizing that prototype-level signals provide useful adaptation supervision beyond output entropy across several prototype architectures. We especially appreciate the observation that our reasoning boards and consistency metrics provide a useful framework for asking whether adaptation changes the evidence a model uses, not only its accuracy. This is our central claim: interpretable prototype signals can both guide adaptation and reveal how it changes the evidence behind predictions.
In brief:
Below, we address each point in detail. Revisions are marked in blue at the manuscript locations listed in each heading.
W1 & Q2: Controller ablation (Section 5, Table 3, Appendix N)
The extra computation buys robustness across backbones: every other variant falls about one point or more behind the best one on some backbone, whereas the full controller does not. We isolated each controller component, keeping the filter, updated parameters and optimizer fixed (3 seeds): λact or λrel alone mix the same two losses without routing, and the coverage guard and gradient normalization are removed one at a time. The last column reports the largest accuracy shortfall from the best variant shown on each backbone; identifying that best variant requires labels.
The router stays within 0.18 points of the best variant on all four backbones, and it is the only variant within 0.2 points everywhere. These results support the controller's role in avoiding poor weighting choices across backbones, while showing that some components add little on individual models.
W1 & Q2: Speed (Section 5, Tables 2 and 13, Appendix N)
We agree that the full controller is slower, but on ProtoViT a fixed λ or our calibrate-once mode removes much of the overhead when speed matters. Most of this overhead comes from routing, which needs four extra backward passes per batch to choose λ without labels (7 in total, versus 3 for λact or λrel alone). In absolute terms this cost is small: on one B200 GPU, the full controller adds about 2.4 to 3.3 ms per image on the three vision backbones. When speed matters, λ can be fixed, or our new calibrate-once mode can freeze the routing decision and gradient scales after the first reliable batch, so that each later batch needs a single backward pass, as in EATA (no new hyperparameter, not re-tuned). We timed all methods on ProtoViT five times in a single job on the same GPU over three corruptions, and report accuracy over all 13:
Calibrate-once runs at almost the speed of EATA while staying about one point more accurate. EATA's extra pass costs time, and ProtoTTA doesn't need one: unlike EATA, which estimates its Fisher regularizer from an additional set of approximately 500-2,000 samples, it relies only on the incoming test batches.
Continued in Comment 2.