acceptodds
Under review as a conference paper at ICLR 2027

When Better Routing Hides a Worse Prompt Update: Classifier-Conditioned Evaluation in Domain-Incremental Learning

Abstract

Prompt-based continual-learning models store small sets of prompt parameters learned for different domains. At test time, a router selects a prompt for each input and, in some systems, a classifier as well. A router change can therefore be deployed in two ways: by changing only the prompt assignment while retaining the original classifier assignment, or by changing both together. Accuracy measured with the routed prompt–classifier pair correctly evaluates the joint action but does not reveal whether changing the prompt alone helps, and this distinction can reverse a deployment decision. On DomainNet with S-iPrompts, a Gaussian router improves joint accuracy by percentage points (pp), whereas applying only its prompt assignments under the original classifier assignments lowers accuracy by pp. Fixed-bank analyses show the same classifier dependence: in S-iPrompts and a DualPrompt bank from an independent code base with added per-domain heads, the same domain-prompt correction changes sign depending on which classifier is retained, whereas in CP-Prompt it helps under both. Changing classifier exposure through matched refits further changes this dependence in all three per-domain-head configurations on DomainNet, without improving routed classification accuracy over matched baselines. We therefore propose intervention-matched evaluation: a prompt-only action is evaluated with the classifier that will use it, a joint action by joint accuracy, and a training change against a baseline fitted under matched conditions, while reporting paired repairs and harms.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, Domain-incremental learning, Prompt-based continual learning, Prompt routing, Model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gcgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate prompt-selection changes in domain-incremental learning when the classifier used at deployment is retained. The authors propose an intervention-matched evaluation that separates prompt-only, classifier-only, and joint changes through a four-cell decomposition, and further analyze prediction repairs, harms, and the effect of classifier training exposure. Experiments cover S-iPrompts, CP-Prompt, CPrompt, and an adapted DualPrompt configuration with per-domain heads on Office-Home and DomainNet. The results show that prompt correction can have opposite effects under different retained classifiers. In particular, a Gaussian router improves both domain identification and joint accuracy on DomainNet, while its prompt assignments slightly reduce accuracy when combined with the original classifier assignments. The paper argues that evaluation should match the component being changed to the actual deployment intervention.

Critical strengths
  1. The paper identifies a meaningful evaluation issue in prompt-based continual learning. Improving routing accuracy or joint prompt–classifier accuracy does not necessarily imply that a prompt-only update is beneficial when the original classifier assignments are retained. The four-cell decomposition makes this distinction explicit and helps avoid conflating component-level effects with joint improvements.

  2. The experimental analysis is carefully structured. The authors use paired predictions, report repair and harm events, compare multiple configurations, and conduct additional seed and domain-order checks for S-iPrompts on DomainNet. The matched-refit experiments also distinguish changes in component interactions from improvements in predictive accuracy. The paper generally acknowledges the limitations of its evidence rather than claiming that the observed behavior is universal.

Critical weaknesses
  1. The novelty and broader significance of the contribution remain somewhat limited. The proposed evaluation protocol is conceptually useful, but the four-cell decomposition is a straightforward way to analyze independently changeable components. The paper would benefit from a clearer explanation of what new insight this framework provides beyond existing work on prompt–classifier compatibility, task inference, and component-level evaluation. The empirical evidence is also concentrated on DomainNet: the same sign reversal is not consistently observed on Office-Home, and some evaluated configurations involve implementation adaptations rather than unmodified native methods. Consequently, the results support an important diagnostic case study, but the generality of the conclusions remains uncertain.

  2. The statistical evidence and reproducibility could be stronger for the main fixed-bank comparisons. The primary results report means and standard deviations over three training seeds, and the appendix explicitly notes that these standard deviations are descriptive rather than formal confidence intervals. This is relevant because several headline prompt-only effects are small in absolute magnitude. Although the exposure experiments include paired image-bootstrap intervals, these condition on the fitted banks and do not capture training-seed or model-selection uncertainty. In addition, the released artifact contains summary data and aggregation code but lacks some of the image-level prediction archives and checkpoints needed to reproduce every reconstruction end to end.

Questions for authors
  1. Could the authors provide paired uncertainty estimates for the main fixed-bank prompt-only and interaction effects in Table 2, while clearly distinguishing uncertainty over test images from variability across training seeds? This would help clarify the reliability of the relatively small prompt-only effects.

  2. How broadly should the central sign-reversal finding be interpreted? Since the effect is consistent on DomainNet for S-iPrompts and the adapted DualPrompt+PDH configuration but is not consistently reproduced on Office-Home, additional clarification about the scope of the claim would be helpful. Can the authors provide evidence from an unmodified native configuration or another backbone/pretraining setup?

  3. Could the authors provide sufficient per-image prediction exports or reconstruction artifacts to independently verify the primary Table 2 and Gaussian-router results? The current artifact supports regeneration of reported summaries and figures, but does not contain all the data needed to rerun every reconstruction.

4 comments

Reviewer gcgY (1/2)

Thank you for the specific suggestions. Each of your three questions led to an addition: paired image-level intervals and tests (Table 6), a stated scope with its reasons, and per-image predictions with a verification script.

W1. What is new beyond prompt–classifier compatibility, task inference and component-level evaluation? Compatibility is known, and task-inference work such as HiDe-Prompt (Wang et al., 2023) and Kim et al. (2022) explains why identification accuracy need not describe final accuracy (App. J). Those analyses factor a prediction into task identification and within-task prediction; we hold a trained bank fixed and change its prompt and classifier assignments separately on the same images, so each deployment action (prompt only, head only, both) has its own cell (App. J). With this design we find a case in which the evidence used to adopt a selector rises while accuracy falls. Routing accuracy does not involve the classifier, and with per-domain heads, joint accuracy and oracle-routing gain are measured after the classifier is replaced together with the prompt. In S-iPrompts on DomainNet, a Gaussian router raises routing accuracy from 80.43% to 83.30% and joint accuracy by 1.28 pp, yet its prompts alone lower accuracy by 0.23 pp, with the same sign in all five training seeds, and a router that stores fewer numbers than the native keys does the same (Tables 12–13, quoted in our reply to Wwbo). Changing only the head reaches 89% (S-iPrompts) and 95% (DualPrompt+PDH) of the oracle-routing gain. On the same banks, replacing the per-domain heads by one shared head turns true-domain prompt correction from negative to positive in all three seeds of both systems (Table 18, quoted in our reply to Wwbo), so the classifier alone can set the sign.

W2/Q1. Paired uncertainty for Table 2? Added. Table 6 gives paired image-level 95% intervals and exact McNemar tests for each DomainNet bank and seed. Every prompt-only interval lies below zero and every interaction interval above zero, in S-iPrompts (per-seed effects −0.238 to −0.366 pp, McNemar p between 1.3×10⁻³¹ and 1.3×10⁻¹⁴) and in DualPrompt+PDH (−0.055 to −0.144 pp, p between 5.2×10⁻⁷ and 0.047). Because these intervals condition on each trained bank, we report seed-level intervals separately (App. B). The S-iPrompts effect is negative in all eight banks: −0.306 ± 0.076 pp over five seeds in the primary domain order (t-interval [−0.400, −0.211]) and −0.277 ± 0.007 pp over three reversed-order banks ([−0.295, −0.260]; Table 25). For DualPrompt+PDH it is negative in all three seeds; with three seeds the seed-level interval is [−0.240, +0.012], while each seed's image-level interval lies below zero (Table 6). The analyses behind Tables 13, 18 and 30 were each fixed before their test scoring, with any choice made on validation data (§3; App. D, E.1, H.1).

Table 6, S-iPrompts and DualPrompt+PDH rows. DomainNet, pp for each training seed. Prompt only = the change from switching to the true-domain prompt under the selected head; paired 95% image-bootstrap intervals (5,000 resamples within domain); exact McNemar p of the prompt-only change.

SystemSeedPrompt only, selected head95% intervalMcNemar pInteraction I_PH95% interval
S-iPrompts87−0.238[−0.298, −0.177]1.3×10⁻¹⁴+0.634[0.549, 0.717]
S-iPrompts1013−0.287[−0.347, −0.227]1.5×10⁻²⁰+0.835[0.750, 0.919]
S-iPrompts1993−0.366[−0.430, −0.303]1.3×10⁻³¹+0.837[0.752, 0.922]
DualPrompt+PDH87−0.055[−0.110, −0.002]0.047+0.347[0.271, 0.420]
DualPrompt+PDH1013−0.142[−0.199, −0.087]5.9×10⁻⁷+0.297[0.221, 0.373]
DualPrompt+PDH1993−0.144[−0.201, −0.088]5.2×10⁻⁷+0.345[0.270, 0.419]

Reviewer gcgY (2/2)

Q2. How broadly should the reversal be read? As a per-domain-head result on DomainNet in two configurations, S-iPrompts and DualPrompt+PDH, both ViT-B/16 with different pretrained weights.

  • Native implementation. S-iPrompts is the released S-Prompts image-only implementation with its native router and per-domain heads, with DomainNet optimisation values and training split adapted (App. A). The reversal holds in all five primary-order and all three reversed-order banks (Table 25: prompt only negative, prompt only + I_PH positive).
  • Adapted configuration. DualPrompt+PDH is an evaluation adaptation (Table 1): an independent DualPrompt code base with different pretrained weights and preprocessing, with per-domain heads and four further code changes that we made (App. H).
  • Other backbone and pretraining. CP-Prompt differs in both backbone and classifier: CLIP image and text encoders and a text-generated classifier (Table 1). Its prompt correction is positive under both heads on both datasets (Table 2), so it is the positive boundary case of Table 1 rather than a replication.
  • Office-Home. With 1 pp equal to about 32 images, S-iPrompts' prompt-only change is negative in every seed (§4.1), but its correction under the domain head (+0.041 ± 0.171 pp) is not separated from zero (§5).
  • Shared classifier. In three banks that are the DualPrompt+PDH code with the original shared classifier (App. H.1; Table 30, quoted in our reply to Wwbo), true-domain correction is not negative, as under the shared head of Table 18. There, higher routing accuracy does not reliably raise accuracy: of the three selectors fixed before training, the primary one (true domain, about +23 pp routing) changes accuracy by at most +0.13 pp (significant in one seed), the Gaussian router gives no significant change, and a nearest-mean selector with higher routing accuracy lowers accuracy in every seed.

Q3. Per-image predictions? Added: prediction_bundle/ in the anonymous repository (https://anonymous.4open.science/r/prompt-head-decomposition-F1C1) holds one NPZ file per Table 2 bank with the full prompt × head prediction grid, labels, domains, image ids and routes, including every S-iPrompts router of App. D. python prediction_bundle/verify.py needs only NumPy; it checks file hashes and recomputes Table 2 and Tables 11–13, and its last line should start with PASS: 15 banks. REBUTTAL_ADDITIONS.md maps each new table to its files. Checkpoints will be released with the camera-ready version.

We hope these additions address your questions.

Dear Reviewer gcgY,

The discussion period ends on November 18. We would be grateful to hear whether our response addresses your concerns. For the statistical evidence, Table 6 adds paired image-level 95% intervals and exact McNemar tests for every DomainNet bank and seed. For reproducibility, prediction_bundle/ in the anonymous repository holds the per-image predictions of every Table 2 bank, and verify.py recomputes Table 2 and Tables 11–13. If any point remains unclear, we would be glad to answer it before the deadline.

Thank you for your time.

Dear Reviewer gcgY,

Thank you again for your thoughtful and constructive review. Your suggestions, especially on the paired intervals (now Table 6) and the per-image predictions with the verification script, have strengthened the paper.

As the discussion period closes within a day, we would be happy to answer any remaining questions about our response or the revised version.

Reviewer_WwboOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper notes that in prompt-based domain-incremental learning, systems that route each input to a domain-specific prompt and classifier are usually evaluated by domain identification accuracy, joint accuracy, or oracle routing, yet these metrics change the prompt and classifier together and cannot capture the effect of changing only the prompt while keeping the classifier. To address this, the authors propose intervention-matched evaluation. On frozen trained models, it swaps the routed and true-domain prompts and classifier heads in a 2×2 crossover, decomposes the joint gain into prompt-only, head-only, and interaction terms, and reports sample-level fixes and breaks. The effect of classifier training is tested by matched retraining under a pre-specified protocol (Own vs. Mixed, which mixes in prompts from other domains). Experiments on Office-Home and DomainNet evaluate 21 checkpoints spanning S-iPrompts, CP-Prompt, DualPrompt with added per-domain heads, and CPrompt. On DomainNet, the main results are as follows. In S-iPrompts, a Gaussian router requiring no test-domain labels raises domain identification accuracy from 80.43% to 83.30% and joint accuracy by 1.28 %, yet adopting only its prompt assignments while keeping the original heads lowers accuracy by 0.23 pp. The effect of true-domain prompt correction flips sign depending on the retained head in S-iPrompts and DualPrompt (−0.30 % vs. +0.47 % in S-iPrompts) but is positive under both heads in CP-Prompt. Removing the prompt costs 15 to 24 %, far exceeding the correction effects, and Mixed exposure lowers the interaction term without improving accuracy. On this basis, the authors argue that joint accuracy applies only when the entire prompt-classifier branch is deployed, that prompt changes should be evaluated under the classifier that actually reads them, and that training changes should be compared against matched retraining. These conclusions hold mainly on DomainNet and are limited to the evaluated ViT configurations, a single primary domain order, and three random seeds.

Critical strengths

1.This paper clearly defines an evaluation problem of practical importance. In domain-incremental learning systems where prompts and classifiers can be selected separately, the commonly used domain identification accuracy, joint accuracy, and oracle routing gain all change both components at once, so they cannot show whether changing only the prompt helps. The authors use a concrete decision reversal to show that this distinction affects real decisions. The Gaussian router raises joint accuracy by 1.28 %, yet adopting only its prompt assignments lowers accuracy by 0.23 %. Building on this, the paper gives concrete, actionable guidance for choosing the evaluation according to the actual deployment action.

2.The diagnostic method is simple and general. The 2×2 analysis requires only reassigning prompts and classifier heads on frozen trained models, without retraining, to decompose the joint gain exactly into prompt-only, head-only, and interaction terms. Combined with the removal cost and sample-level fix and break counts, it separates three quantities that are often conflated: reliance on the prompt, the value of correcting the prompt, and the whole-branch gain. For example, removing the prompt costs 15 to 24 %, whereas the correction effect is only about −0.3 to +0.9 %. The approach could also transfer to other modular router-plus-expert systems.

3.The empirical coverage is fairly broad, and the authors acknowledge its limitations. The experiments span four structurally different configurations (including DualPrompt+PDH, built on an independent codebase) and two datasets, totaling 21 checkpoints. The paper reports where the phenomenon holds, namely the sign flips for S-iPrompts and DualPrompt+PDH on DomainNet. It also reports where the phenomenon is absent or the evidence is insufficient, namely CP-Prompt, where correction helps under both heads, and Office-Home, where the results are not significant. The conclusions are scoped accordingly and not overgeneralized.

4.The experimental design is rigorous. The Gaussian router specification, the main comparison of the exposure experiment, and the 0.5 % accuracy threshold were all established before test results were examined, and the decision criteria for DualPrompt+PDH were fixed in writing before training. The paper includes targeted controls (Own, Own-half, retraining without prompts, and others) and reports paired McNemar tests and image-level bootstrap intervals in addition to seed standard deviations. It also replicates the findings with additional seeds and a reversed domain order, and releases the code that generates the figures and tables together with per-seed summary data.

5.The experiment provides controlled evidence that the features a classifier sees during training change the dependence between prompts and classifier heads. Mixed exposure lowers the interaction term in all three per-domain-head configurations on DomainNet. In the two configurations that include a data-quantity control, the control barely changes the interaction term. The experiment also directly supports the paper’s methodological claim that better diagnostic metrics do not imply higher accuracy. Even when the joint gain is nearly unchanged, the individual components can still change markedly.

Critical weaknesses

1.The practical relevance of the motivating scenario is questionable. The headline decision reversal appears only in branch-bank systems such as S-iPrompts. In these systems, keeping the old head assignments while swapping only the prompt is itself a constructed counterfactual deployment. The natural practice is to swap the whole branch, in which case the paper itself acknowledges that joint accuracy is the correct evaluation. The paper presents shared-readout methods as the most natural setting for prompt-only changes, yet never tests a change in prompt selection on any native L2P, DualPrompt, or CODA-Prompt model. In the only fixed-readout case (the merged readout of CPrompt), the prompt change from the same Gaussian router actually improves accuracy (+0.19 %), with no reversal. Moreover, the prompt-related effects are mostly within 1 pp (the reversal is −0.23 % vs. +1.28 %). They are statistically significant, but the practical cost of not following the proposed guidance may be quite limited.

2.The methodological novelty is limited, and the insights lack depth. The paper neither explains where the interaction term comes from or why S-iPrompts flips while CP-Prompt does not, nor proposes a method that improves the model. The only lever it identifies, Mixed exposure, lowers the interaction term without raising accuracy. Without a new method or a mechanistic explanation, the contribution may easily be viewed as incremental.

3.Evidence for the generality of the core phenomenon is weak. The sign flip holds only on DomainNet and only for S-iPrompts and DualPrompt+PDH. The latter is modified by the authors, whose per-domain expert prompts are highly similar (mean cosine similarity 0.91), and its effect is only −0.11 %. CP-Prompt shows no flip, and the Office-Home results are not significant. The headline example relies on a single probe-style router on a single dataset. This router stores about 1.78 million numbers, more than the combined parameters of the six classifier heads, and the sign of its effect is unstable on Office-Home. The cross-method comparison used to suggest that the readout determines the direction (CPrompt vs. S-iPrompts) is further confounded by differences in backbone, preprocessing, and training. It therefore remains hard to tell whether this is a general phenomenon or an artifact of particular implementations.

4.The experimental scope is narrow. All results come from four ViT-based image classification configurations on two datasets, with three seeds per setting and a single primary domain order. Additional seeds and a reversed order were run only for S-iPrompts on DomainNet, and only one reversal was tried among the 720 possible orders of the six domains. The paper also does not address class-incremental settings (where heads cover different classes and cannot be cross-swapped), other backbones and pretraining, larger models, or other router-expert systems, so it is unclear how far the conclusions generalize.

5.The experiments offered limited value. It is a post-training diagnostic that requires revisiting data from all domains, which is inconsistent with the replay-free continual learning setting. Accuracy does not improve in any of the three configurations. CP-Prompt drops by about 1 pp, exceeding the preset threshold, and its interaction term crosses zero and actually grows in magnitude. DualPrompt+PDH lacks a data-quantity control, and the Office-Home results are not significant either. The experiment shows that exposure is a contributing factor but offers little help in training better models.

6.The writing is hard to follow. The main text is extremely dense and heavy with notation (A_ab, J, I_PH, D_h, C_h, R_u, H_u, M, and others), and different quantities use different reference baselines that readers must carefully keep apart. For example, the main text reports the removal cost for S-iPrompts as 23.76 % relative to the selected prompt with the deployed head, whereas Figure 5 reports it as 24.66 % relative to the true-domain prompt with the true-domain head. Many key arguments rely on an appendix of more than twenty pages, making it hard for readers to quickly grasp the main thread.

Questions for authors

1.Could the authors compare the native selector with a selector of higher domain identification accuracy on native L2P, DualPrompt, or CODA-Prompt, to see whether better identification again comes with lower classification accuracy? If this can be shown, my concern about practical relevance would be substantially reduced.

2.Could the authors give an example from a real system or a published method in which the prompt selector or prompt pool was updated on its own while the classifier assignment was kept fixed?

3.In Table 11, the only router that improves domain identification accuracy is the Gaussian router, which stores more numbers than the six classifier heads combined. Is there any other router that improves domain identification (for example, one with storage comparable to the native keys) and also shows a negative prompt-only effect on DomainNet?

4.Why do S-iPrompts and DualPrompt+PDH flip while CP-Prompt does not? For example, how large are the differences between the features produced by different prompts, and do these differences correlate with the size of the interaction term?

5.Can this framework be extended to class-incremental settings or to other router-expert systems such as mixture-of-experts models? Is it possible to design a training scheme that reduces the dependence between prompts and classifier heads without sacrificing accuracy?

4 comments

Reviewer Wwbo (1/2)

Thank you for the detailed review; your Q1 and Q3 led to our two main new results, Tables 30 and 13.

W1/Q1. With a shared classifier, does better identification again lower accuracy? For one of the three selectors we fixed before training, yes; for the other two, accuracy barely changes. Our three DomainNet banks are the DualPrompt+PDH code with the original shared classifier: they keep its four other code changes (App. H) and differ from DualPrompt+PDH only in the classifier (App. H.1). All three selectors raise routing accuracy in every seed (Table 30). The pre-specified primary selector, the true domain, adds about 23 pp routing accuracy but at most 0.133 pp accuracy, significant in one seed. The Gaussian router gives no significant change in any seed. A nearest-mean selector that stores as many numbers as the keys (6 × 768) lowers accuracy in every seed and harms more images than it repairs. In these banks, where every selector update is prompt-only, higher routing accuracy therefore does not reliably raise accuracy (§4.1). True-domain correction is not negative here, as under the shared head of Table 18; the sign reversal we claim is a per-domain-head result.

Table 30. DualPrompt+PDH code with the original shared classifier, DomainNet. Change from the native key match (RA 76.90–77.40%, accuracy 49.23–49.39%); paired 95% image-bootstrap interval; exact McNemar p, Holm-corrected over the three selectors of each seed.

SeedSelectorΔRA (pp)ΔAcc (pp)95% intervalHolm p
87True domain+22.60+0.133[+0.077, +0.188]1.5×10⁻⁵
87Gaussian+1.41+0.025[−0.023, +0.075]0.32
87Nearest mean+0.38−0.071[−0.105, −0.037]1.0×10⁻⁴
1013True domain+23.10+0.018[−0.038, +0.075]0.56
1013Gaussian+1.90−0.028[−0.079, +0.022]0.56
1013Nearest mean+0.87−0.143[−0.184, −0.104]1.1×10⁻¹¹
1993True domain+22.95+0.050[−0.007, +0.109]0.18
1993Gaussian+1.76−0.016[−0.067, +0.032]0.53
1993Nearest mean+0.73−0.136[−0.176, −0.097]3.0×10⁻¹¹

W1 (rest). Is a prompt-only change a real deployment, and does a small loss matter? With a shared classifier, every selector change is prompt-only, as in Table 30. With per-domain heads, a prompt-only change arises when the selector or prompts are updated while the head assignments are kept (App. J); when the whole branch is replaced, joint accuracy is the right test (Table 4). The losses are small, but they occur for selectors that raise routing accuracy and, with per-domain heads, joint accuracy (App. D, H.1), and the check that measures them needs no retraining: the test images are scored again under the classifier that will be kept (§6). On the CPrompt result you cite: under CPrompt's combined readout, the Gaussian router's prompts raise accuracy (+0.189 pp). On the S-iPrompts banks, changing only the classifier gives both signs for the same prompts: +0.192 pp under one shared head, in every seed, and −0.225 pp under the per-domain heads (§5, App. E.1). A selector validated under one classifier therefore needs to be checked again under another.

Q2. A real example? With a shared classifier, every change to prompt selection keeps the classifier by construction (L2P, DualPrompt, CODA-Prompt; Wang et al., 2022c;b; Smith et al., 2023), and DualPrompt reports key-matching accuracy as the measure of its selector (App. J). Table 30 tests a selector change judged by that measure in our shared-classifier banks (App. H.1). With per-domain heads, we cite no published system that updated only its selector while keeping the head assignments. The evaluations we cite either replace the head together with the prompt, as SOYO's oracle routing does (Wang et al., 2025b), or, like routing accuracy, do not involve the classifier. Neither isolates the accuracy effect of the prompt choice: changing only the head reaches 89% of the oracle-routing gain in S-iPrompts and 95% in DualPrompt+PDH (§4.1).

W2/Q4. What is new, and why do some systems flip? Compatibility is known, and we do not propose a new learner. What is new is that a selector can raise routing and joint accuracy yet lower accuracy when only its prompts are adopted (App. D), and that the classifier alone can set the sign of the same prompt correction (Table 18, part 2/2). Feature differences do not order the interaction: CP-Prompt changes the features most (mean cosine distance 0.188 vs. 0.106 in S-iPrompts and 0.102 in DualPrompt+PDH) and has the most specialised heads (20.1 vs. 15.0 and 6.7 pp), yet has the smallest interaction (+0.154 vs. +0.769 and +0.330 pp; App. E.1).

Reviewer Wwbo (2/2)

W2/Q4 (continued). A shared head on the same banks. CP-Prompt differs from the other two in backbone and classifier (CLIP encoders, text-generated classifier; Table 1), so cross-system comparison cannot isolate why it does not flip. Replacing only the classifier avoids these confounds—including those between CPrompt (not CP-Prompt) and S-iPrompts (W3): with banks, prompts and routes fixed, replacing the per-domain heads by one shared head turns prompt correction from negative to positive in every seed (Table 18). The sign does not follow head accuracy (Acc. columns; App. E.1).

Table 18. DomainNet, with banks, prompts and routes fixed. Accuracy (%) and the change from true-domain prompt correction (pp) under one shared head or the selected per-domain head (PDH); paired 95% image-bootstrap interval.

SystemSeedAcc., sharedAcc., PDHCorrection, shared95% intervalCorrection, PDH
S-iPrompts8757.7362.56+1.03[0.96, 1.10]−0.24
S-iPrompts101355.4260.54+1.03[0.96, 1.10]−0.29
S-iPrompts199357.3762.38+0.96[0.90, 1.03]−0.37
DualPrompt+PDH8757.0453.24+0.98[0.90, 1.05]−0.06
DualPrompt+PDH101357.1053.11+0.79[0.72, 0.86]−0.14
DualPrompt+PDH199357.0252.76+0.86[0.79, 0.93]−0.14

W3/Q3. Generality; a router that stores no more numbers than the keys? Yes: a diagonal Gaussian router that stores 9,216 numbers, against 23,040 for the keys, was chosen on validation before test scoring. On S-iPrompts DomainNet it raises routing accuracy, yet its prompts lower accuracy under the original heads and raise it under its own, in all five seeds (App. D).

Tables 12–13. S-iPrompts, DomainNet. Change from the native router in pp, mean ± SD over seeds 87/1013/1993; routing accuracy (RA) in %. The fitted routers have the same RA in every seed; seeds 7 and 42 give the same prompt-only signs for both Gaussian routers (§4.1, App. D).

RouterStored numbersRAPrompt only, original headsPrompt only, own headsHead onlyJoint (ΔJ)
Native keys23,04080.426 ± 0.096
Gaussian1.78M83.296−0.225 ± 0.029+0.371 ± 0.074+0.907 ± 0.044+1.277 ± 0.045
Diagonal Gaussian9,21681.336−0.456 ± 0.071+0.159 ± 0.050+0.346 ± 0.024+0.505 ± 0.042
Domain oracle100.000−0.297 ± 0.064+0.472 ± 0.076+3.669 ± 0.039+4.141 ± 0.038

DualPrompt+PDH is an independent DualPrompt code base with different pretrained weights, to which we added the per-domain heads and four further code changes (App. H). Its expert prompts are similar (mean cosine 0.91; Table 29), yet true-domain correction changes sign between the two heads in every seed, with image-level intervals that exclude zero (Table 6). The Gaussian router's prompts also lower accuracy under the original heads in every seed (−0.131 pp; App. D). We state these results for DomainNet: on Office-Home (1 pp is about 32 images), in S-iPrompts the Gaussian router's prompt-only change under the original heads varies in sign across seeds (App. D), and the correction under the domain head is not separated from zero (§5).

W4. Scope. All three reversed-order S-iPrompts banks also show the reversal (+0.41 to +0.62 pp under the domain head; prompt only + I_PH, Table 25). This reversed order is one of the 720 possible orders; the pattern holds under both orders tested (App. G).

W5. What the exposure experiment is for. It tests whether the features a head is trained on change the interaction (W2). Refitting each head partly on other domains' prompt features lowers the signed interaction in every DomainNet seed of all three systems (in CP-Prompt it crosses zero; §4.2), while the DomainNet data-quantity control (two systems) changes it by at most 0.012 pp on average (Table 20). Because the refit uses stored training data, it is a diagnostic, not a rehearsal-free training method (App. F).

Q5. Other systems and training schemes. Crossing prompts and heads needs a shared label space: in class-incremental learning, another task's head excludes the correct class by construction (§2), but a prompt-only check under a shared classifier needs no crossing (App. J). Our experiments cover prompt-based systems. On training schemes, mixed exposure lowers the signed interaction at a cost of 0.11–1.02 pp of joint accuracy (Table 3), and the shared head of Table 18 is fitted on all domains at once, which needs past data.

W6. Readability. Notation is cut to the four cells, J, I_PH and the mismatch set, and Figures 2–3 are redrawn. The main text reports the removal cost against one reference (23.76 pp); Fig. 5's 24.66 pp uses the domain prompt and head (App. B).

Tables 30 and 13 are the new evidence for your Q1 and Q3.

Dear Reviewer Wwbo,

The discussion period ends on November 18. We would be grateful to hear whether our response addresses your questions. The two additions most relevant to them are Table 30, which tests three selectors under a shared classifier (W1/Q1, Q2), and Table 18, which replaces only the classifier on the same banks (W2/Q4). If any point remains unclear, we would be glad to answer it before the deadline.

Thank you for your time.

Dear Reviewer Wwbo,

Thank you again for your detailed and constructive review. Your questions, especially Q1 on selectors under a shared classifier (now Table 30) and Q3 on a router that stores no more numbers than the keys (now Table 13), have strengthened the paper.

As the discussion period closes within a day, we would be happy to answer any remaining questions about our response or the revised version.

Reviewer_ra2POpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper studies how to evaluate prompt-selection changes in domain-incremental prompt-based continual learning when prompts and classifier heads can be selected independently. It proposes a four-cell, intervention-matched evaluation that distinguishes prompt-only, head-only, and joint changes under fixed trained banks. Experiments on Office-Home and DomainNet demonstrate that a prompt correction can improve joint accuracy while harming accuracy when only its prompt assignment is deployed under retained heads.

Critical strengths

The experimental design is rigorous and the findings are trustworthy, e.g., exact paired substitutions at fixed checkpoints, matched controls isolating the proposed mechanism, statistically significant replication.

Critical weaknesses
  1. The paper offers limited conceptual novelty and lacks methodological contribution. It is widely accepted that a classifier trained on features from one prompt performs differently when given features from another prompt. This is the compatibility problem that has already been studied, and the paper itself cites these works. The proposed four-cell decomposition is a standard factorial analysis rather than a new evaluation protocol. Moreover, the practicality of its application scenarios is questionable. Deploying new prompt assignments while deliberately retaining the old head assignments is an unusual choice that the paper motivates only abstractly.

  2. Although an AI Use Statement is provided, the manuscript reads more like the accumulated output of an iterative, automated research-and-writing process than a focused, hypothesis-driven investigation. Many passages document the development history instead of the science. The writing is also verbose and defensive, with nearly every result accompanied by qualifications about what it does not show. Finally, several analyses are only loosely connected to the central claim, and numerous post-hoc explanations make the paper difficult to follow and fall short of what is expected of an ICLR paper.

Questions for authors

At this stage, I have no specific questions whose answers could materially affect my assessment.

4 comments

Reviewer ra2P

Thank you for the review. We start with W2, because it changed the paper most.

W2. Presentation. We agree that the paper was hard to read, and we rewrote it without changing any submitted result:

  • The introduction now states the two questions we test (p. 2, lines 100–103): can the same prompt correction change sign only because the retained classifier changes, and does changing the features used to train a classifier alter this dependence, and at what cost in accuracy?
  • Qualifications sit in one Scope paragraph (§5, p. 10); the sentences after individual results that listed what they do not show are removed.
  • Development history is removed, three side analyses are moved to the appendices, notation is reduced, and Figures 2–3 are redrawn.

New results appear only in App. B, D, E.1 and H.1 (Tables 6, 13, 18 and 30 and their text).

W1. Novelty and methodological contribution. Compatibility itself is known (§1), and we do not propose a new learner. The contribution is a finding about evaluation and the protocol that follows from it (§5, Table 4): a selector can score higher on the usual evidence (routing accuracy, joint accuracy) and still lower accuracy when only its prompts are adopted under the classifier that is kept. In S-iPrompts on DomainNet, a Gaussian router raises routing accuracy from 80.43% to 83.30% and joint accuracy by 1.28 pp, yet its prompts alone lower accuracy by 0.23 pp, with the same sign in all five training seeds (App. D). On the same banks, replacing the per-domain heads by one shared head turns true-domain prompt correction from negative to positive in all three seeds (Table 18). We kept the 2×2 design a standard factorial so that each deployment action (prompt only, head only, both) has its own cell; the prompt-only check needs no retraining (§6).

Practicality. In rehearsal-free learners with one shared classifier, such as L2P, DualPrompt and CODA-Prompt, keeping the classifier is not a choice: it cannot be refit to new prompt choices without past data, so every selector update is prompt-only (p. 2, lines 082–090). App. H.1 tests this setting in three DualPrompt banks that keep the shared classifier and carry four of our code changes (Table 30, quoted in our reply to Wwbo). All three selectors fixed before training raise routing accuracy. The primary one, the true domain, adds about 23 pp routing accuracy but at most 0.13 pp accuracy (significant in one seed), the Gaussian router gives no significant change, and a nearest-mean selector that stores as many numbers as the keys lowers accuracy in every seed. With per-domain heads, a prompt-only change arises when the selector or prompts are updated while the head assignments are kept (App. J); when the whole prompt–head branch is replaced, joint accuracy is the right test (Table 4).

The shared-classifier paragraph of the introduction (p. 2) and Table 30 contain the evidence on practicality.

Dear Reviewer ra2P,

The discussion period ends on November 18. We would be grateful to hear whether our response addresses your concerns about presentation and novelty. The revised PDF is rewritten without changing any submitted result, and the new results appear only in App. B, D, E.1 and H.1. If any point remains unclear, we would be glad to answer it before the deadline.

Thank you for your time.

Dear authors,

Thanks for your reply and for the substantial effort to improve the readability of the paper. I will raise the presentation score and the overall rating accordingly. The clarification on novelty and practicality makes it clearer that the contribution is an evaluation finding with an accompanying protocol. However, I still feel the current contribution below the ICLR threshold.

The paper would be much stronger if it went one step further from observation to method. For instance, you could train or calibrate a selector for accuracy under the retained classifier, rather than for domain recognition. Alternatively, borrowing ideas from backward-compatible training to design a safe update rule for prompt-only changes would also make the work more complete. With such extensions, I believe this work would be a more interesting submission for a future venue.

Best regards,

Reviewer ra2P

Dear Reviewer ra2P,

Thank you for reading the revised paper and for raising the presentation score and the rating. Your comments made the paper clearer and stronger, and we are grateful for them. We also appreciate the two concrete directions: training or calibrating a selector for accuracy under the retained classifier, and a backward-compatible update rule for prompt-only changes. Both follow directly from the finding, and we will pursue them as the next step of this work. If you have any further questions or suggestions, we would be glad to hear them at any time.

Best regards, the authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the reviewers and the AC

Summary for the reviewers and the AC

We thank the Area Chair for handling our manuscript and the three reviewers for their insightful comments. Table numbers match the revised PDF.

In short

Prompt-based continual learners are usually improved by making them pick the input's domain prompt more often. We show that, when the classifier is kept, this can lower final accuracy on DomainNet, consistently across seeds, routers and statistical tests. This gives a simple check that needs no retraining: a new prompt-picking rule should be judged by final accuracy with the classifier it will be used with, so that changes which look better but hurt accuracy are filtered out. We believe this finding will improve how such rules are evaluated and help research on domain-incremental learning.

The finding

Prompt–classifier compatibility is known; we ask which evidence should be used to decide whether a new prompt selector is adopted. Routing accuracy, reported by S-Prompts, HiDe-Prompt, CPrompt and SOYO, does not involve the classifier. With per-domain heads, joint accuracy and oracle-routing gain are measured after the classifier is replaced together with the prompt. Neither kind of evidence measures a prompt-only update, which keeps the classifier. In rehearsal-free learners with one shared classifier, such as L2P, DualPrompt and CODA-Prompt, every selector update is prompt-only, because the classifier cannot be refit without past data. The paper supports two claims:

  • A selector that scores higher on the usual evidence can lower accuracy when only its prompts are adopted. In S-iPrompts (the image-only variant of S-Prompts) on DomainNet, a Gaussian router raises routing accuracy from 80.43% to 83.30% and joint accuracy by 1.28 pp, yet its prompts alone lower accuracy by 0.23 pp, with the same sign in all five training seeds (App. D).
  • The classifier can set the sign of the same prompt correction. In S-iPrompts and DualPrompt+PDH on DomainNet, switching to the true-domain prompt lowers accuracy under the selected per-domain head and raises it under the true-domain head and under one shared head fitted to the same training data, in every seed (Tables 6 and 18).

Main concerns and what changed

  1. Novelty beyond compatibility; why only some systems flip (ra2P, Wwbo, gcgY). Table 18 (new) replaces only the classifier, with banks, prompts and routes fixed, so backbone, preprocessing and training are held constant; prompt correction turns positive in every seed. Feature change and head specialisation do not order the interaction across systems (App. E.1).
  2. One router, one implementation (Wwbo, gcgY). A diagonal Gaussian router that stores fewer numbers than the native keys (9,216 vs 23,040; Table 13, new) and the Gaussian router on DualPrompt+PDH (App. D) both raise routing and joint accuracy, yet their prompts lower accuracy under the original heads in every seed. We claim the sign reversal for S-iPrompts and DualPrompt+PDH on DomainNet.
  3. Practical relevance of prompt-only updates (ra2P, Wwbo). App. H.1 and Table 30 (new) compare selectors in three DualPrompt banks that keep the original shared classifier and carry four of the code changes we made for DualPrompt+PDH (App. H). All three selectors fixed before training raise routing accuracy. The primary one, the true domain, adds about 23 pp routing accuracy but at most 0.13 pp accuracy (significant in one seed). The Gaussian router gives no significant change, and a nearest-mean selector that stores as many numbers as the keys lowers accuracy in every seed. In these banks, higher routing accuracy does not reliably raise accuracy (§4.1).
  4. Statistics and reproducibility (gcgY). We added per-seed paired image-level intervals and exact McNemar tests (Table 6, new), plus per-image predictions with a NumPy verification script.
  5. Readability (ra2P, Wwbo). We rewrote the introduction around the two questions we test, cut notation, and consolidated caveats into one Scope paragraph (§5). The method is renamed classifier-conditioned evaluation, as in the title (intervention-matched evaluation in the submission).

Update (18 Nov): Reviewer ra2P found the revision clearer and raised the presentation score and the overall rating.

REBUTTAL_ADDITIONS.md in the anonymous repository (https://anonymous.4open.science/r/prompt-head-decomposition-F1C1) lists the files behind each new result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.