A Dataset Is Worth Decomposing into Multiple Latent Domains: Reframing and Mitigating Base–New Trade-Off in Prompt Learning

Prompt learning provides an efficient way to adapt vision-language models, yet optimizing prompts on seen base classes often compromises generalization to unseen new classes, giving rise to the Base–New Trade-off. We identify the prevalent monolithic assumption as a key source of this tension, with a single global prompt expected to model the complex and non-uniform semantic manifold spanning all classes in a dataset. To mitigate this tension, we reframe the class semantic space of a dataset as multiple latent domains and propose **L**atent **D**omain **P**rompt (LDP). LDP constructs semantic anchors from the frozen textual geometry of the full predefined class vocabulary and learns a global prompt together with a set of latent domain bases using only base-class visual supervision. For each class, LDP synthesizes a class-adaptive prompt by weighting these bases according to the class's semantic affinities to the anchors. Through this affinity-weighted synthesis, latent domains serve as structural bridges between base and new classes, allowing new classes to inherit domain-level knowledge learned from base classes without requiring new-class images. Theoretically, monolithic prompting contains a first-order global-displacement term, whereas LDP is governed by local fitting and reconstruction errors, with a second-order geometric term under approximate barycentric reconstruction. Extensive experiments spanning 15 datasets show that LDP consistently improves four representative prompt learning baselines across base-to-new, cross-dataset, and cross-domain settings.