acceptodds
Withdrawn from ICLR 2027

A Dataset Is Worth Decomposing into Multiple Latent Domains: Reframing and Mitigating Base–New Trade-Off in Prompt Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompt learning provides an efficient way to adapt vision-language models, yet optimizing prompts on seen base classes often compromises generalization to unseen new classes, giving rise to the Base–New Trade-off. We identify the prevalent monolithic assumption as a key source of this tension, with a single global prompt expected to model the complex and non-uniform semantic manifold spanning all classes in a dataset. To mitigate this tension, we reframe the class semantic space of a dataset as multiple latent domains and propose **L**atent **D**omain **P**rompt (LDP). LDP constructs semantic anchors from the frozen textual geometry of the full predefined class vocabulary and learns a global prompt together with a set of latent domain bases using only base-class visual supervision. For each class, LDP synthesizes a class-adaptive prompt by weighting these bases according to the class's semantic affinities to the anchors. Through this affinity-weighted synthesis, latent domains serve as structural bridges between base and new classes, allowing new classes to inherit domain-level knowledge learned from base classes without requiring new-class images. Theoretically, monolithic prompting contains a first-order global-displacement term, whereas LDP is governed by local fitting and reconstruction errors, with a second-order geometric term under approximate barycentric reconstruction. Extensive experiments spanning 15 datasets show that LDP consistently improves four representative prompt learning baselines across base-to-new, cross-dataset, and cross-domain settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
prompt learning, vision-language models, base-to-new generalization, transfer learning, domain generalization, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XxRxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the Base–New Trade-off in vision-language prompt learning and attributes it to the limitations of using a single global prompt to represent heterogeneous class semantics. The authors propose Latent Domain Prompt (LDP), which clusters frozen class-text representations into semantic anchors and synthesizes class-adaptive prompts through affinity-weighted combinations of learnable domain bases. A validation-based procedure selects the number of latent domains. The paper also presents theoretical approximation-error analyses and evaluates LDP as a plug-in extension to CoOp, MaPLe, PromptSRC, and MMRL++ across base-to-new, cross-dataset, and cross-domain generalization benchmarks.

Critical strengths

S1. Simple and modular design. LDP introduces class-adaptive prompting through semantic anchors and affinity-weighted domain bases without requiring novel-class images. Its modularity facilitates integration with existing prompt-learning frameworks.

S2. Broad empirical evaluation. The paper evaluates four prompt-learning baselines across 15 datasets and three generalization settings. Positive aggregate improvements are observed, particularly for CoOp.

Critical weaknesses

W1. Limited gains on stronger baselines and insufficient experimental validation. As a plug-in method, LDP achieves only +0.54, +0.75, and +0.34 H improvements over MaPLe, PromptSRC, and MMRL++, respectively. The integrated baselines do not include methods newer than 2025, despite relevant 2026 advances such as Promise (ICLR), ManiPT, and NeRP (ICML). Although several 2026 methods are included as reference results, the paper lacks matched comparisons with recent strong approaches, paired uncertainty estimates, and computational overhead analysis. Moreover, Table 4 reports LDP results that differ from Table 1 even under the stated default temperature, without explaining the experimental differences.

W2. Insufficient evidence for the proposed semantic mechanism and theoretical interpretation. The cross-dataset anchor construction and validation-based selection of (K) are not sufficiently specified. Existing ablations do not isolate semantic decomposition from additional prompt capacity or generic routing. The claimed second-order geometric advantage depends on Assumption 4, which is not implied by the proposed affinities or empirically verified. Furthermore, the approximation bounds are not connected to classification performance, and Figures 1 and 5 do not provide sufficiently controlled evidence of semantic preservation.

Questions for authors

Q1. Transfer protocol. How are anchors and (K) determined in cross-dataset evaluation? Are target class vocabularies or validation samples involved? How is the validation loss in Eq. (13) computed for ImageNet, whose validation split is marked N/A in Table 5?

Q2. Recent methods. Why are the plug-in experiments restricted to baselines no newer than 2025? Can the authors provide matched comparisons with recent approaches such as Promise, ManiPT, or NeRP, where applicable? How does LDP differ substantively from TAC (CVPR 2025)?

Q3. Practical significance. Given the +0.34 H gain over MMRL++, can the authors report paired confidence intervals, trainable parameter counts, training/search costs, and inference overhead to establish whether the improvement justifies the additional complexity?

Q4. Contribution attribution. How does LDP compare with parameter-matched global prompts, random/uniform routing, and simple learned gating? Why are the routing and (K) ablations not extended to MMRL++, the strongest integrated baseline?

Q5. Theoretical relevance. Can the authors empirically validate Assumption 4 across different anchor resolutions and establish a connection between the approximation bounds and classification performance?

Q6. Geometric evidence. Can Figures 1 and 5 be evaluated using consistent class sets, centroid definitions, ordering, and distance scales? Do quantitative neighborhood-preservation metrics support the claimed reduction in semantic distortion?

Q7. Result consistency. Table 1 reports LDP H scores of 77.73, 78.89, and 80.01 for CoOp, MaPLe, and PromptSRC, whereas Table 4 reports 76.26, 78.41, and 79.91 at the stated default (\lambda=0.1). Since the original baseline scores match exactly across the two tables, what experimental differences explain these discrepancies? Was (K), the training protocol, or another setting changed?

Reviewer_YN4BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LDP to improve generalization from seen to unseen classes. It groups class names by semantic similarity and combines group-specific prompts for each class, using only seen-class images for training. It provides an approximation analysis and reports average improvements across four baselines on 15 datasets.

Critical strengths
  1. LDP improves average performance across four different baselines and three evaluation settings. This supports its usefulness as a general addition to prompt learning.

  2. Correct class–group assignments outperform permuted assignments in Table 4, providing some support for knowledge sharing through semantic relationships.

Critical weaknesses
  1. The theory analyzes distance to hypothetical optimal prompts, but does not connect this distance to classification accuracy. Its second-order result also depends on Assumption 4, which is neither guaranteed by the method nor verified experimentally.

  2. The permutation experiment is useful, but a shared-prompt baseline with the same parameter count and tuning budget is missing. Without this control, it is difficult to separate the benefit of semantic grouping from added capacity and tuning.

  3. CoOp with the default temperature achieves H = 76.26 in Table 4 versus 77.73 in Table 1. The paper should explain the setting differences behind these results.

  4. It is unclear which classes provide validation images for selecting K and whether source anchors are retained during cross-dataset transfer. These details are needed to assess whether the comparisons are fair and reproducible.

Questions for authors

Please see weaknesses.

Reviewer_HEPNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the base-to-novel trade-off in vision-language model prompt learning by reframing class semantics as multiple latent domains. It first constructs semantic anchors from all classes, including base and novel classes, using K-means clustering of CLIP textual representations. These anchors are then used to construct class-adaptive text prompts by combining a global prompt, latent domain prompts, and class token embeddings. Specifically, each class receives a latent domain prompt constructed as a weighted combination of learnable latent domain prompt bases, where the weights are determined by the semantic affinity between the class textual representation and the semantic anchors. The number of clusters (K) is selected by learning multiple candidate sets of latent domain prompts and alternating between optimizing prompt parameters and search scores using cross-entropy losses on training and validation sets, respectively. The method is evaluated on base-to-new, cross-dataset, and cross-domain generalization, showing improvements when applied on top of existing baseline methods.

Critical strengths
  1. The paper attempts to address the base-to-novel trade-off by leveraging semantic information from the text embedding space, which is a valuable direction for improving generalization in vision-language models.

  2. When applied to different existing prompt learning methods, the proposed approach improves their performance across base-to-new, cross-dataset, and cross-domain generalization settings. This demonstrates the applicability of the proposed approach across different methods and generalization settings.

Critical weaknesses
  1. The paper violates the standard base-to-new setting commonly followed in prompt learning papers. It uses all classes, including novel classes, to construct semantic anchors during training. The CLIP textual embeddings of novel classes are used for semantic anchor clustering, which introduces information about novel classes during training, whereas only base-class information should be used. This raises concerns about the validity of the base-to-new generalization setting and the fairness of comparisons with existing methods.

  2. The novelty of the proposed methodological components is limited. The paper constructs multiple candidate latent domain prompts, which are closely related to existing multi-prompt approaches [1, 2, 3]. ATPrompt [1], in particular, has substantial methodological similarities, as it constructs class prompts using a combination of attribute-related prompts for vision-language prompting instead of semantic clusters. Furthermore, the proposed search-score optimization closely follows ATPrompt [1], which also employs DARTS-based alternating optimization to learn representative attribute prompts, rather than the semantic-anchor-based latent domain prompts used in this paper.

  3. The paper is poorly written and motivated, with many methodological details hidden behind mathematical equations, making the approach difficult to understand. In particular, the local second moment shown in Figure 3 is not part of the explicit optimization objective and does not directly contribute to the proposed method. Its inclusion in the methodological figure is confusing, as it appears to be presented as a methodological contribution despite being part of the theoretical analysis.

  4. Finally, the method constructs unique text prompts for each of the NN classes, with multiple sets of such prompts corresponding to different candidate numbers of latent domains. In contrast, previous methods generally use shared learnable prompt contexts across classes. This increases the computational overhead during training, while the additional inference cost will also be increased.

[1] Advancing Textual Prompt Learning with Anchored Attributes (ATPrompt), ICCV 2025.

[2] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models, ICLR 2023.

[3] Mixture of Prompts Learning for Vision-Language Models, 2025.

Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.