- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate prompt-selection changes in domain-incremental learning when the classifier used at deployment is retained. The authors propose an intervention-matched evaluation that separates prompt-only, classifier-only, and joint changes through a four-cell decomposition, and further analyze prediction repairs, harms, and the effect of classifier training exposure. Experiments cover S-iPrompts, CP-Prompt, CPrompt, and an adapted DualPrompt configuration with per-domain heads on Office-Home and DomainNet. The results show that prompt correction can have opposite effects under different retained classifiers. In particular, a Gaussian router improves both domain identification and joint accuracy on DomainNet, while its prompt assignments slightly reduce accuracy when combined with the original classifier assignments. The paper argues that evaluation should match the component being changed to the actual deployment intervention.
Critical strengths
-
The paper identifies a meaningful evaluation issue in prompt-based continual learning. Improving routing accuracy or joint prompt–classifier accuracy does not necessarily imply that a prompt-only update is beneficial when the original classifier assignments are retained. The four-cell decomposition makes this distinction explicit and helps avoid conflating component-level effects with joint improvements.
-
The experimental analysis is carefully structured. The authors use paired predictions, report repair and harm events, compare multiple configurations, and conduct additional seed and domain-order checks for S-iPrompts on DomainNet. The matched-refit experiments also distinguish changes in component interactions from improvements in predictive accuracy. The paper generally acknowledges the limitations of its evidence rather than claiming that the observed behavior is universal.
Critical weaknesses
-
The novelty and broader significance of the contribution remain somewhat limited. The proposed evaluation protocol is conceptually useful, but the four-cell decomposition is a straightforward way to analyze independently changeable components. The paper would benefit from a clearer explanation of what new insight this framework provides beyond existing work on prompt–classifier compatibility, task inference, and component-level evaluation. The empirical evidence is also concentrated on DomainNet: the same sign reversal is not consistently observed on Office-Home, and some evaluated configurations involve implementation adaptations rather than unmodified native methods. Consequently, the results support an important diagnostic case study, but the generality of the conclusions remains uncertain.
-
The statistical evidence and reproducibility could be stronger for the main fixed-bank comparisons. The primary results report means and standard deviations over three training seeds, and the appendix explicitly notes that these standard deviations are descriptive rather than formal confidence intervals. This is relevant because several headline prompt-only effects are small in absolute magnitude. Although the exposure experiments include paired image-bootstrap intervals, these condition on the fitted banks and do not capture training-seed or model-selection uncertainty. In addition, the released artifact contains summary data and aggregation code but lacks some of the image-level prediction archives and checkpoints needed to reproduce every reconstruction end to end.
Questions for authors
-
Could the authors provide paired uncertainty estimates for the main fixed-bank prompt-only and interaction effects in Table 2, while clearly distinguishing uncertainty over test images from variability across training seeds? This would help clarify the reliability of the relatively small prompt-only effects.
-
How broadly should the central sign-reversal finding be interpreted? Since the effect is consistent on DomainNet for S-iPrompts and the adapted DualPrompt+PDH configuration but is not consistently reproduced on Office-Home, additional clarification about the scope of the claim would be helpful. Can the authors provide evidence from an unmodified native configuration or another backbone/pretraining setup?
-
Could the authors provide sufficient per-image prediction exports or reconstruction artifacts to independently verify the primary Table 2 and Gaussian-router results? The current artifact supports regeneration of reported summaries and figures, but does not contain all the data needed to rerun every reconstruction.
4 comments
Reviewer gcgY (2/2)
Q2. How broadly should the reversal be read? As a per-domain-head result on DomainNet in two configurations, S-iPrompts and DualPrompt+PDH, both ViT-B/16 with different pretrained weights.
- Native implementation. S-iPrompts is the released S-Prompts image-only implementation with its native router and per-domain heads, with DomainNet optimisation values and training split adapted (App. A). The reversal holds in all five primary-order and all three reversed-order banks (Table 25: prompt only negative, prompt only + I_PH positive).
- Adapted configuration. DualPrompt+PDH is an evaluation adaptation (Table 1): an independent DualPrompt code base with different pretrained weights and preprocessing, with per-domain heads and four further code changes that we made (App. H).
- Other backbone and pretraining. CP-Prompt differs in both backbone and classifier: CLIP image and text encoders and a text-generated classifier (Table 1). Its prompt correction is positive under both heads on both datasets (Table 2), so it is the positive boundary case of Table 1 rather than a replication.
- Office-Home. With 1 pp equal to about 32 images, S-iPrompts' prompt-only change is negative in every seed (§4.1), but its correction under the domain head (+0.041 ± 0.171 pp) is not separated from zero (§5).
- Shared classifier. In three banks that are the DualPrompt+PDH code with the original shared classifier (App. H.1; Table 30, quoted in our reply to Wwbo), true-domain correction is not negative, as under the shared head of Table 18. There, higher routing accuracy does not reliably raise accuracy: of the three selectors fixed before training, the primary one (true domain, about +23 pp routing) changes accuracy by at most +0.13 pp (significant in one seed), the Gaussian router gives no significant change, and a nearest-mean selector with higher routing accuracy lowers accuracy in every seed.
Q3. Per-image predictions? Added: prediction_bundle/ in the anonymous repository (https://anonymous.4open.science/r/prompt-head-decomposition-F1C1) holds one NPZ file per Table 2 bank with the full prompt × head prediction grid, labels, domains, image ids and routes, including every S-iPrompts router of App. D. python prediction_bundle/verify.py needs only NumPy; it checks file hashes and recomputes Table 2 and Tables 11–13, and its last line should start with PASS: 15 banks. REBUTTAL_ADDITIONS.md maps each new table to its files. Checkpoints will be released with the camera-ready version.
We hope these additions address your questions.
Dear Reviewer gcgY,
The discussion period ends on November 18. We would be grateful to hear whether our response addresses your concerns. For the statistical evidence, Table 6 adds paired image-level 95% intervals and exact McNemar tests for every DomainNet bank and seed. For reproducibility, prediction_bundle/ in the anonymous repository holds the per-image predictions of every Table 2 bank, and verify.py recomputes Table 2 and Tables 11–13. If any point remains unclear, we would be glad to answer it before the deadline.
Thank you for your time.
Dear Reviewer gcgY,
Thank you again for your thoughtful and constructive review. Your suggestions, especially on the paired intervals (now Table 6) and the per-image predictions with the verification script, have strengthened the paper.
As the discussion period closes within a day, we would be happy to answer any remaining questions about our response or the revised version.
Reviewer gcgY (1/2)
Thank you for the specific suggestions. Each of your three questions led to an addition: paired image-level intervals and tests (Table 6), a stated scope with its reasons, and per-image predictions with a verification script.
W1. What is new beyond prompt–classifier compatibility, task inference and component-level evaluation? Compatibility is known, and task-inference work such as HiDe-Prompt (Wang et al., 2023) and Kim et al. (2022) explains why identification accuracy need not describe final accuracy (App. J). Those analyses factor a prediction into task identification and within-task prediction; we hold a trained bank fixed and change its prompt and classifier assignments separately on the same images, so each deployment action (prompt only, head only, both) has its own cell (App. J). With this design we find a case in which the evidence used to adopt a selector rises while accuracy falls. Routing accuracy does not involve the classifier, and with per-domain heads, joint accuracy and oracle-routing gain are measured after the classifier is replaced together with the prompt. In S-iPrompts on DomainNet, a Gaussian router raises routing accuracy from 80.43% to 83.30% and joint accuracy by 1.28 pp, yet its prompts alone lower accuracy by 0.23 pp, with the same sign in all five training seeds, and a router that stores fewer numbers than the native keys does the same (Tables 12–13, quoted in our reply to Wwbo). Changing only the head reaches 89% (S-iPrompts) and 95% (DualPrompt+PDH) of the oracle-routing gain. On the same banks, replacing the per-domain heads by one shared head turns true-domain prompt correction from negative to positive in all three seeds of both systems (Table 18, quoted in our reply to Wwbo), so the classifier alone can set the sign.
W2/Q1. Paired uncertainty for Table 2? Added. Table 6 gives paired image-level 95% intervals and exact McNemar tests for each DomainNet bank and seed. Every prompt-only interval lies below zero and every interaction interval above zero, in S-iPrompts (per-seed effects −0.238 to −0.366 pp, McNemar p between 1.3×10⁻³¹ and 1.3×10⁻¹⁴) and in DualPrompt+PDH (−0.055 to −0.144 pp, p between 5.2×10⁻⁷ and 0.047). Because these intervals condition on each trained bank, we report seed-level intervals separately (App. B). The S-iPrompts effect is negative in all eight banks: −0.306 ± 0.076 pp over five seeds in the primary domain order (t-interval [−0.400, −0.211]) and −0.277 ± 0.007 pp over three reversed-order banks ([−0.295, −0.260]; Table 25). For DualPrompt+PDH it is negative in all three seeds; with three seeds the seed-level interval is [−0.240, +0.012], while each seed's image-level interval lies below zero (Table 6). The analyses behind Tables 13, 18 and 30 were each fixed before their test scoring, with any choice made on validation data (§3; App. D, E.1, H.1).
Table 6, S-iPrompts and DualPrompt+PDH rows. DomainNet, pp for each training seed. Prompt only = the change from switching to the true-domain prompt under the selected head; paired 95% image-bootstrap intervals (5,000 resamples within domain); exact McNemar p of the prompt-only change.