acceptodds
Under review as a conference paper at ICLR 2027

Understanding Prompt Learning as Low-Rank Manifold Alignment

Abstract

Under a linear shared-latent model with frozen linear encoders and conditionally zero-mean text noise outside the shared subspace, we show that the population contrastive objective admits an optimal prompt operator with the rank smaller than the shared latent dimension, for any positive temperature. For nonlinear representations, we establish a latent factorization of the centered cross-covariance operator, with a rank bound under low-dimensional conditional-mean structure and a spectral-tail bound when that structure is approximate. HSIC provides a one-way dependence guarantee for centered trace alignment, rather than an equivalent objective. We also derive a generalization bound for explicitly rank-constrained prompt classes. Synthetic diagnostics distinguish exact trace-objective recovery from the nonzero spectral tails of normalized InfoNCE, while real-CLIP measurements provide operator-level evidence of spectral concentration. Inspired by the shared-coordinate structure, LR-MaPLe couples visual and textual prompts through shared factorization codes. Across 10 benchmarks and three seeds, it achieves accuracy comparable to MaPLe with approximately fewer trainable parameters. The factorization width is an architectural choice, not an estimate of the latent dimension.

Primary area:
interpretability and explainable AI
Keywords:
Prompt learning, Low-Rank Manifold Alignment, Representation Learning Theory, Multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GPYQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to explain why prompt learning methods such as CoOp can work effectively when optimizing only a small number of continuous vectors in a high-dimensional embedding space, even with only 1–16 labeled examples per class. The authors propose low-rank cross-modal alignment as the underlying mechanism. Under a linear shared-latent model with frozen linear encoders and a conditional-centering assumption, they show that an optimal population prompt operator exists with rank at most the shared latent dimension k (Proposition 3.1). For nonlinear representations, they analyze the centered cross-covariance operator in an RKHS, establish a latent factorization under conditional independence, and prove a rank bound under a low-dimensional conditional-mean assumption (CM); when CM is only approximate, they provide a spectral-tail control (Theorem 3.4). They also derive a generalization bound for explicitly rank-constrained prompt classes (Proposition 3.8). Inspired by the shared-coordinate structure, they propose LR-MaPLe, which couples visual and textual prompts through shared factorization codes.

Critical strengths
  1. The difficulty and effort of the theoretical work deserve recognition. Formulating prompt learning as a cross-modal alignment problem under a shared-latent model, and deriving rank bounds, spectral-tail controls, and generalization bounds in both linear and nonlinear settings, is a technically demanding endeavor. The authors handle conditional centering, cross-covariance operators in RKHS, trace-class operator decompositions, and Rademacher complexity. The mathematical derivation is complete and self-contained. Regardless of the final explanatory power, this effort to elevate an empirical phenomenon into formal analysis is commendable.

  2. The paper provides a discussable geometric framework for prompt learning. It connects prompt learning with cross-modal covariance, CCA, and HSIC, offering a relatively complete geometric perspective. Although whether this perspective truly explains actual training remains debatable, it gives subsequent research a starting point that can be tested, refuted, or extended. For the multimodal PEFT community, such a theoretical framework is itself of reference value.

Critical weaknesses
  1. The motivation is not fully pursued, and the low-rank finding actually exposes a tension with high-dimensional parameterization. The core finding is that the optimal prompt operator has rank at most k. If this conclusion holds, what is the point of CoOp optimizing tens or hundreds of thousands of parameters in a high-dimensional embedding space? A large portion of the parameter dimensions would be redundant. The paper does not directly address this question, nor does it ask further: why has direct low-rank parameterization of prompts not become mainstream? Would direct low-rank parameterization lead to performance degradation? If so, why? If not, is the high-dimensional parameterization of CoOp merely a need of training dynamics? These questions directly determine whether the paper’s motivation holds, but the paper provides no substantive analysis.

  2. The paper identifies an effective factor but does not exploit it for methodological innovation, resulting in weak practical impact. The paper identifies low-rank cross-modal alignment as a potentially effective factor, but apart from LR-MaPLe—a simple shared factorization that algebraically overlaps with MMLoP—it does not further use this finding to improve methods. For example: How can the low-rank structure be extended to the entire network? Can low-rank codes be shared at every layer, every attention head, or every modality interaction so that the low-rank space covers the whole model and improves performance? Can the input or prompt injection be modified so that the low-rank structure better matches pretrained representations? These directions are not systematically explored. As a result, the paper stops at “discovering a factor that makes learning effective” without converting that factor into new design principles or performance gains. The novelty is therefore clearly insufficient.

Questions for authors
  1. Do the conditional-centering assumption (5) and the conditional-mean assumption (CM) approximately hold on real CLIP features? The theory is entirely built on (5) and (CM), but the paper provides no empirical validation on real data. Can the authors give quantitative evidence, for example by estimating the deviation of E[η∣x,Πy]\mathbb{E}[\eta \mid x, \Pi y] or testing whether the centered conditional means lie in low-dimensional subspaces? Even a negative result would help readers understand the scope of the theory.

  2. Can the paper provide a falsifiable prediction: under what training conditions will actual optimization exhibit the low-rank structure characterized by the theory? Proposition 3.1 is an existence result for the population optimum, but actual CoOp training is finite-step, non-convex, and typically uses normalized InfoNCE. Can the authors specify under what temperature, normalization scheme, initialization, optimizer, or number of training steps the learned prompt would exhibit the spectral concentration predicted by the theory? Without such testable predictions, how does the theory explain the success of actual training?

  3. When CM is only approximately satisfied, can the spectral-tail control be converted into actionable performance or generalization conclusions? Theorem 3.4 provides a spectral-tail control for approximate structure, but the paper does not relate the size of the spectral tail to downstream accuracy, generalization error, or sample complexity. How large can the spectral tail be before the theoretical guarantee becomes meaningless? How should this control guide practical prompt design or rank-constraint selection?

  4. In what sense does the rank bound in the linear surrogate carry over to real transformer prompt learning? The paper abstracts the prompt as a linear operator PP acting on text representations, whereas real CoOp injects continuous tokens into multiple transformer layers, involving attention, layer normalization, and nonlinear interactions. Can the authors provide evidence or arguments that the rank conclusion from the linear surrogate still holds, or partially holds, in the real architecture? If not, does the theoretical conclusion apply only to the surrogate model?

7 comments
AuthorsOpenReview ↗

Response to Reviewer GPYQ (1/5)

We thank the reviewer for the constructive feedback and the detailed questions. Below, we respond to each comment. All changes are marked in red in the revised paper.


W1: If the optimal operator has rank at most kk, what is the point of optimizing in a high-dimensional space? Would direct low-rank parameterization degrade performance, or is the high-dimensional parameterization merely a need of the training dynamics?

R1: We answer these questions in turn.

Low rank is not the same as few parameters. The theory says that range⁡(Ay)\operatorname{range}(A_y) is low-dimensional, but not where it lies. Locating a kk-dimensional subspace of Rdy\mathbb R^{d_y} costs about kdyk d_y parameters, so each prompt direction must live in the full embedding space. In this sense mainstream prompt learning is already low rank: CoOp trains an nctx×512n_{\text{ctx}} \times 512 token matrix (2K–8K parameters, rank ≤nctx≪512\le n_{\text{ctx}} \ll 512), and MaPLe uses nctx=2n_{\text{ctx}} = 2. Most of MaPLe's 3.55M parameters sit in its coupling maps, which LR-MaPLe removes.

In the surrogate, gradient training does not use the extra dimensions. Gradient descent from P0=0P_0 = 0 stays in the rank-kk set in population, without an explicit rank constraint, and within tεNt\varepsilon_N of it with NN samples, where εN=O~(dyC/N)\varepsilon_N = \tilde O(d_y \sqrt{C/N}) (new Props. 3.1A–3.1B, §3.2). The finite-sample deviation grows with dyd_y and with training time, so the extra dimensions are not free. The result also depends on the optimizer: it holds for gradient descent, the surrogate counterpart of the SGD used by CoOp, MaPLe and LR-MaPLe, but not for Adam, whose coordinate-wise scaling is not rotation-equivariant. In the new Fig. 2(c) (τ=0.07\tau = 0.07), the off-subspace share stays at 0.46–0.50 with Adam, while it falls from 0.19 to 0.012 with gradient descent.

Explicit low rank gives comparable accuracy on average, with dataset-dependent effects. At the default nctx=2n_{\text{ctx}} = 2, LR-MaPLe matches MaPLe (77.69 ± 0.44 vs 77.27 ± 0.54 Macro HM on 11 datasets), no width is uniformly best (Table 4, now with seed SDs), and a single 4-dimensional subspace for the whole network gives a comparable Macro HM with 5,192 parameters (see R2). With an active rank cap (nctx=16n_{\text{ctx}} = 16, Table 5; HM in %, 3 seeds), the effect depends on the dataset.

SettingDTDEuroSAT
LR-MaPLe, r=1r = 165.46 ± 2.5074.32 ± 2.30
LR-MaPLe, r=2r = 263.08 ± 1.3976.48 ± 1.35
LR-MaPLe, r=4r = 465.36 ± 0.6869.68 ± 13.94
LR-MaPLe, r=8r = 860.05 ± 4.1172.14 ± 8.29
LR-MaPLe, r=16r = 1663.88 ± 0.5572.32 ± 19.22
MaPLe, same nctxn_{\text{ctx}}, no rank cap60.76 ± 1.7281.54 ± 4.40

LR-MaPLe is above the matched MaPLe on DTD at most ranks and below it on EuroSAT at all ranks, with large seed variation. This is consistent with MMLoP's ablation, where rank-one prompts alone are slightly below IVLP (77.06 vs. 77.51 HM) and the gap reverses once the prompts are anchored to zero-shot CLIP.

AuthorsOpenReview ↗

Response to Reviewer GPYQ (2/5)

W2: The paper identifies an effective factor but does not exploit it for method design, e.g. by extending the low-rank structure to the entire network.

R2: Thank you for these suggestions. The paper is primarily theoretical (primary area: interpretability), with LR-MaPLe as a minimal testbed. In the revision we added two experiments motivated by the theory and one theoretical result.

A network-wide shared subspace (new Exp. 4, Appendix G.3, Table 13). All 9 prompted layers share one pair of bases (Utext,Uvis)(U_{\text{text}}, U_{\text{vis}}) and keep per-layer codes, so each modality's 18 prompt vectors lie in one rr-dimensional subspace. Setting of Table 2 (11 datasets, 3 seeds).

ModelTrainable parametersMacro BaseMacro NewMacro HM
MaPLe3.55M81.91 ± 0.2173.78 ± 0.7877.27 ± 0.54
LR-MaPLe184,60881.84 ± 0.3774.52 ± 0.6477.69 ± 0.44
Bases tied across depth, r=16r = 1620,76882.13 ± 0.1674.32 ± 0.1977.68 ± 0.06
Bases tied across depth, r=4r = 45,19281.22 ± 0.3575.01 ± 0.2877.76 ± 0.24

Macro HM is comparable with 5,192 parameters (1/36 of LR-MaPLe and about 1/680 of MaPLe), and no dataset differs beyond seed variation (paired t-tests, p≥0.15p \ge 0.15). In this setting, the prompts of the whole network thus fit in one low-dimensional subspace per modality.

Sharing versus rank (new Exp. 3, Appendix G.2, Table 12). At nctx=16n_{\text{ctx}} = 16, where the rank cap is active, we give each modality its own codes at the same width and rank ceiling. HM in %, 3 seeds, the shared-code runs being those of Table 5 in the HM convention of Appendix D.1; the difference is independent minus shared, paired by seed.

rrDatasetIndependent codesShared codesDifferencepp
1DTD64.39 ± 2.9865.47 ± 2.50−1.080.43
1EuroSAT72.06 ± 4.2874.42 ± 2.30−2.340.53
2DTD65.24 ± 0.7363.10 ± 1.39+2.150.07
2EuroSAT76.30 ± 2.5576.50 ± 1.35−0.190.90
4DTD63.40 ± 1.9365.37 ± 0.68−1.980.12
4EuroSAT77.31 ± 1.8169.81 ± 13.94+7.470.49

The differences have both signs and stay within seed variation. A shared low-dimensional coordinate system therefore suffices in this setting, while sharing by itself gives no detectable accuracy gain; with three seeds, small effects cannot be excluded.

An approximation–estimation trade-off in the surrogate (new Cor. B.4, see A3). The loss of rank-rr prompt operators is bounded by a term linear in the centering error εr\varepsilon_r plus the rank-rr estimation term, and εr\varepsilon_r can be estimated from frozen CLIP features. Here rr is the operator rank in the surrogate, not the factorization width of LR-MaPLe; a rank-selection rule for token prompts is left for future work.

AuthorsOpenReview ↗

Response to Reviewer GPYQ (3/5)

Q1: Do the conditional-centering assumption (5) and the conditional-mean assumption (CM) approximately hold on real CLIP features?

A1: We added diagnostics on frozen CLIP ViT-B/16 features, without training, for the test splits of all 11 datasets and COCO val2017 captions (new Exp. 2, Appendix G.1, Tables 10–11). The class serves as a proxy for the latent, so ranks are bounded by C−1C-1 and are not estimates of kk.

Conditional-mean assumption (CM): the cross-covariance is concentrated, the conditional means much less so. (CM) requires the centered conditional means of each modality, given the latent, to lie in a subspace of dimension at most kk. In the table, ranks are 95%-energy ranks. The tail is the share of the nuclear norm of CxyC_{xy} beyond 16 directions, and the last column divides the measured tail by the bound of the new Cor. B.3(iii) at q=16q = 16 (see A3).

DatasetC−1C-1Rank of CxyC_{xy}Rank, image meansRank, text meansTail beyond 16Measured / bound
ImageNet9992015822149.7%0.57
SUN3973962611415747.4%0.59
StanfordCars19510416622.3%0.44
OxfordFlowers10119416633.6%0.56
Food10110023477137.8%0.65
UCF10110025556841.0%0.61
Caltech1019920546737.4%0.63
FGVCAircraft996254215.2%0.41
DTD4620293729.0%0.59
OxfordPets369182610.9%0.39
EuroSAT9578––

The cross-covariance, which Thm. 3.4 bounds, needs 5–26 directions, while the class means themselves need far more. Low cross-covariance rank follows from (CM) but does not imply it, so these measurements support concentrated cross-modal dependence, not (CM) at the same dimension.

Conditional centering (5): there is residual predictability, largest at small qq. The condition requires E[y∣x,Πy]=Πy\mathbb E[y \mid x, \Pi y] = \Pi y, i.e. the part of the text outside the shared subspace cannot be predicted from the image and the part inside it. For an estimated projector Πq\Pi_q, we regress the off-subspace text component (I−Πq)g~(I-\Pi_q)\tilde g on [f(x),Πqg~][f(x), \Pi_q \tilde g] and on Πqg~\Pi_q \tilde g alone, on held-out pairs; under conditional centering, neither explains it. ΔR2\Delta R^2 is the part that needs the image, and εq2\varepsilon_q^2 is the centering error of the new Cor. B.4 relative to the text variance (linear ridge).

DatasetΔR2\Delta R^2, q=4q = 4ΔR2\Delta R^2, q=16q = 16ΔR2\Delta R^2, q=C−1q = C-1εq2\varepsilon_q^2, q=1q = 1εq2\varepsilon_q^2, q=16q = 16
COCO captions0.290.19–0.320.14
ImageNet0.230.18–0.240.16
SUN3970.230.180.000.240.17
StanfordCars0.260.130.000.290.08
Food1010.290.210.050.310.16
UCF1010.300.250.060.310.20
OxfordFlowers0.260.180.060.290.15
Caltech1010.310.290.060.330.22
FGVCAircraft0.090.050.000.130.08
DTD0.140.100.030.170.12
OxfordPets0.320.140.050.390.08
EuroSAT0.130.010.030.260.03

The residual shrinks with qq, and the image contribution is 0.00–0.06 once Πq\Pi_q spans the class information (on COCO it is 0 at q=256q = 256). Three caveats apply: a vanishing image contribution is necessary but not sufficient for conditional centering; linear ridge gives a lower bound on the error; and Π\Pi is unobserved, so the test concerns the estimate Πq\Pi_q. Conditional centering is thus an idealization on real features, and the new Cor. B.4 bounds the loss incurred by the residual (see A3).

Reviewer_g7CeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper theoretically investigates why vision-language models, such as CLIP, generalize to few-shot tasks through context optimization (CoOp), which optimizes continuous-valued prompts. The paper first assumes that images and text are generated from a shared latent space and proves the existence of a low-rank optimal solution by translating the optimization target of CoOp as a simple linear operator on the text input. Next, as an extension to nonlinear encoders, the paper shows that when feature mappings on an RKHS are introduced, the cross-covariance operator between image and text modalities becomes low-rank. Furthermore, the paper analyzes how constraining the hypothesis class to be low-rank reduces the sample complexity bound. Based on these theoretical observations, the paper introduces LR-MaPLe, which parameterizes trainable prompts using modality-shared factors. Experiments confirm that LR-MaPLe can achieve competitive performance with 1/20 the number of parameters compared to the original MaPLe.

Critical strengths
  • The paper theoretically proves that low-rank optimal solutions exist in a setting where a linear operator corresponding to prompt optimization is the target.
  • The paper demonstrates that, in nonlinear cross-modal representations, if the conditional-mean structure is low-dimensional, the rank of the cross-covariance operator is also constrained.
  • The paper shows that low-rank constraints can tighten the generalization bounds of hypothesis classes.
  • The paper demonstrates that the proposed method, LR-MaPLe, can achieve competitive performance at nearly 20 times the efficiency of MaPLe.
Critical weaknesses
  • There is a significant discrepancy between the theoretical and practical problem formulations for CoOp. Although the problem formulation in Section 3 focuses on optimizing the linear operator PP, actual CoOp and MaPLe insert trainable tokens into Transformers and perform nonlinear feature transformations via attention. While the paper’s motivation is "why does prompt learning work so well? (L041)," the theoretical results it provides do not fully address this objective.
  • The result in Theorem 3.4 demonstrates the low rank of cross-covariance, but does not demonstrate the low rank of prompts. The fact that nonlinear shared representations are low-rank does not directly prove why prompts should be low-rank.
  • It is questionable whether the improvement in LR-MaPLe is due to low-rank properties. For both MaPLe and LR-MaPLe, the number of prompt tokens being optimized is 2, and the rank of the prompt matrix is at most 2. In other words, the improvement observed in LR-MaPLe may not stem from low-rank properties, but rather from a regularization effect resulting from the cross-modality parameter sharing via the shared factors introduced in this study. The current paper does not sufficiently isolate this factor. In this sense, the connection between the theoretical analysis and LR-MaPLe is weak.
  • LR-MaPLe has limited technical novelty. As the paper acknowledges, low-rank multimodal prompting based on modality-common factors has already been introduced in MMLoP. As mentioned above, because the relationship between the theoretical analysis and LR-MaPLe is unclear, the impact of LR-MaPLe is limited.
Questions for authors

See the critical weaknesses and address the concerns.

5 comments
AuthorsOpenReview ↗

Response to Reviewer g7Ce (1/3)

We thank the reviewer for the constructive feedback. Below, we respond to each comment. All changes are marked in red in the revised paper.


W1: The theory optimizes a linear operator PP, whereas CoOp and MaPLe insert trainable tokens into transformers, so the results do not fully address "why does prompt learning work so well?".

R1: Studying a controlled abstraction is common practice in the theory of prompt tuning and multi-modal contrastive learning [1–4], and it allows us to isolate the mechanism. In the revision we establish the connection to practical prompt learning in two directions, the objective and the parameterization (new paragraph at the end of §3.3; new Appendix B.1).

  • Objective (new Cor. B.2): in the linear surrogate, the low-rank optimum extends to the supervised cross-entropy over class texts used by CoOp. The proof of Prop. 3.1 uses the loss only through its convexity in logits that are linear in PyPy. With linear encoders and unnormalized scores, the cross-entropy over class texts has this form, so it also admits a minimizer of rank at most kk at every temperature. The new gradient-dynamics result (Prop. 3.1A: from zero initialization, gradient descent keeps rank at most kk and converges to the global optimal value) holds for it as well. With a single fixed text per class, the centering condition requires the prototypes to lie in the shared subspace.
  • Parameterization (new Cor. B.3): the cross-covariance results extend to transformer token prompts. Thm. 3.4 bounds the rank of the cross-covariance by kk under its conditional-mean assumption (CM): the centered conditional means of the image and text features given the latent zz each lie in a subspace of dimension at most kk. Its proof only needs each prompted encoder to act on its own modality, so the latent factorization and the rank bound hold for continuous tokens processed by CLIP's attention layers in one or both branches (CoOp, IVLP, MaPLe, LR-MaPLe). A new explicit spectral-tail bound covers the case where (CM) holds only approximately.

Measurements on real CLIP ViT-B/16 features are consistent with the second result. With learned MaPLe and LR-MaPLe prompts on OxfordPets, the cross-covariance has 95%-energy rank 7–8 (Table 9). On frozen features of the 11 datasets (new Exp. 2, Appendix G.1, Table 10), it is 5–26, below the class-count ceiling C−1C-1 on every dataset, and the measured tail beyond 16 directions is 0.39–0.65 of the new tail bound.

DatasetC−1C-1Rank of CxyC_{xy}Tail / bound
ImageNet999200.57
SUN397396260.59
StanfordCars195100.44
OxfordFlowers101190.56
Food101100230.65
UCF101100250.61
Caltech10199200.63
FGVCAircraft9960.41
DTD46200.59
OxfordPets3690.39
EuroSAT95–

We have also made the scope precise: the abstract now reads "We study a sufficient mechanism behind it" instead of "We answer this question". The paper identifies which structure makes low-dimensional prompting lossless in the surrogate, shows that gradient training reaches it there, and characterizes the cross-modal signal that a prompt can act on. A rank bound for the transformer token matrix is left for future work.


W2: Thm. 3.4 shows that the cross-covariance is low rank, not that prompts are low rank.

R2: We would like to clarify the role of Thm. 3.4. As the paper notes right after the theorem, it concerns the cross-covariance: it characterizes the cross-modal signal that any prompt can act on. The revision makes the link to prompts explicit (new Cor. B.3(iv)). If (CM) holds on the image side, tr⁡Cxy(θ)=Ez⟨a(z),ΠVxb(z;θ)⟩\operatorname{tr} C_{xy}(\theta) = \mathbb E_z \langle a(z), \Pi_{V_x} b(z;\theta) \rangle, where aa and bb are the centered conditional means of the image and text features and VxV_x is the subspace of dimension at most kk that contains aa. For text prompts, the centered alignment therefore depends on the prompt only through a projection of the text conditional means onto at most kk dimensions. This is what motivates a low-dimensional prompt parameterization; it is a statement in the output feature space and does not bound the rank of the tokens. The rank of the prompt itself is addressed by Prop. 3.1 and Cor. B.2 (optimum) and by the new Props. 3.1A–3.1B (training), in the linear surrogate.


References

  • [1] Wei et al. "Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning." NeurIPS 2021.
  • [2] Oymak et al. "On the role of attention in prompt-tuning." ICML 2023.
  • [3] Malladi et al. "A kernel-based view of language model fine-tuning." ICML 2023.
  • [4] Nakada et al. "Understanding multimodal contrastive learning and incorporating unpaired data." AISTATS 2023.
AuthorsOpenReview ↗

Response to Reviewer g7Ce (2/3)

W3: Both MaPLe and LR-MaPLe use 2 prompt tokens, so the prompt rank is at most 2. The improvement of LR-MaPLe may come from cross-modal parameter sharing rather than low rank, and the paper does not isolate this factor.

R3: The parameter saving comes from the low-dimensional factorization, not from cross-modal sharing, and neither factor yields an accuracy gain. At the default nctx=2n_{\text{ctx}} = 2 the rank cap is indeed inactive: both methods have prompt rank at most 2 (eq. 14), and LR-MaPLe matches MaPLe (Macro HM 77.69 ± 0.44 vs 77.27 ± 0.54; the paired 95% CI of the difference, [−0.35, 1.21], includes 0) with about 19× fewer parameters. We do not claim an accuracy improvement. To isolate the two factors, we added two experiments in which the constraints are active.

Sharing (new Exp. 3, Appendix G.2, Table 12). At nctx=16n_{\text{ctx}} = 16 and r∈{1,2,4}r \in \lbrace 1, 2, 4 \rbrace, we give the vision branch its own codes, with the same width, rank ceiling and initialization. HM in %, 3 seeds; the difference is independent minus shared, paired by seed.

rrDatasetIndependent codesShared codesDifferencepp
1DTD64.39 ± 2.9865.47 ± 2.50−1.080.43
1EuroSAT72.06 ± 4.2874.42 ± 2.30−2.340.53
2DTD65.24 ± 0.7363.10 ± 1.39+2.150.07
2EuroSAT76.30 ± 2.5576.50 ± 1.35−0.190.90
4DTD63.40 ± 1.9365.37 ± 0.68−1.980.12
4EuroSAT77.31 ± 1.8169.81 ± 13.94+7.470.49

Removing the sharing changes HM only within seed variation: the differences have both signs and none is significant (the +7.47 comes from one shared-code seed that collapses); with three seeds, small effects cannot be excluded. Sharing also saves only about 1% of the parameters (11,664 against 11,808 at r=1r = 1). Against the matched MaPLe without a rank cap (60.77 ± 1.72 on DTD, 81.59 ± 4.40 on EuroSAT), both low-rank variants are higher on DTD and lower on EuroSAT, so the effect of an active rank cap depends on the dataset.

Low-dimensional subspace (new Exp. 4, Appendix G.3, Table 13). All 9 prompted layers share one pair of bases and keep per-layer codes, so each modality's 18 prompt vectors lie in one rr-dimensional subspace. Setting of Table 2 (11 datasets, 3 seeds).

ModelTrainable parametersMacro BaseMacro NewMacro HM
MaPLe3.55M81.91 ± 0.2173.78 ± 0.7877.27 ± 0.54
LR-MaPLe184,60881.84 ± 0.3774.52 ± 0.6477.69 ± 0.44
Bases tied across depth, r=16r = 1620,76882.13 ± 0.1674.32 ± 0.1977.68 ± 0.06
Bases tied across depth, r=4r = 45,19281.22 ± 0.3575.01 ± 0.2877.76 ± 0.24

Confining all prompts of a modality to a 4-dimensional subspace keeps Macro HM with 5,192 parameters, 1/36 of LR-MaPLe and about 1/680 of MaPLe, and no dataset differs beyond seed variation (paired t-tests, p≥0.15p \ge 0.15). These are empirical findings in the evaluated setting.

AuthorsOpenReview ↗

Response to Reviewer g7Ce (3/3)

W4: LR-MaPLe has limited novelty, since low-rank multimodal prompting with modality-common factors was introduced in MMLoP.

R4: We would like to clarify how LR-MaPLe differs from MMLoP and where its parameter saving comes from.

MethodTrainable parametersTraining lossMacro HM
MaPLe3.55Mcross-entropy77.27 ± 0.54
IVLP (independent deep prompts)about 23Kcross-entropy76.97 ± 1.00
MMLoP control (shared factor)–cross-entropy76.94 ± 0.80
LR-MaPLe184,608cross-entropy77.69 ± 0.44
LR-MaPLe, bases tied across depth (r=4r = 4)5,192cross-entropy77.76 ± 0.24
MMLoP, full system (as reported)11.5Kcross-entropy, consistency loss, drift correction, 60-template ensemble79.70

The first five rows are our runs (11 datasets, 3 seeds; Tables 2 and 13, Appendix D.4; no significance claim). The last row is the number reported by MMLoP, which takes the HM of the averaged Base and New accuracies; in that convention LR-MaPLe gives 78.0.

  • What is shared. MMLoP learns its factors separately at each of the nine prompted layers. In our design, motivated by the shared-latent view, the low-dimensional subspace belongs to the modality, not to the layer: one basis pair (Utext,Uvis)(U_{\text{text}}, U_{\text{vis}}) serves all prompted layers, and a layer keeps only a small code Zℓ∈Rnctx×rZ_\ell \in \mathbb R^{n_{\text{ctx}} \times r} that both modalities read, Ptext(ℓ)=ZℓUtext⊤P_{\text{text}}^{(\ell)} = Z_\ell U_{\text{text}}^\top and Pvis(ℓ)=ZℓUvis⊤P_{\text{vis}}^{(\ell)} = Z_\ell U_{\text{vis}}^\top (new Exp. 4, Appendix G.3).
  • Where the saving comes from. MaPLe couples the modalities through a dense 512 → 768 map at every layer, which accounts for most of its 3.55M parameters. LR-MaPLe replaces these maps with the shared code and thin bases (184K, 19× fewer). Tying the bases across depth with r=4r = 4 then leaves 4 × (512 + 768) + 9 × 2 × 4 = 5,192 parameters, less than half of MMLoP's 11.5K, at a comparable Macro HM.
  • How it is trained. LR-MaPLe uses cross-entropy alone, with two tokens per layer and five epochs. In MMLoP's own ablation, low-rank prompting alone is below IVLP (77.06 vs. 77.51 HM); the full system reaches a higher HM with its additional components and 30 epochs. Under our protocol, the cross-entropy-only MMLoP control is at 76.94 ± 0.80, against 77.69 ± 0.44 for LR-MaPLe.

Within a layer, both methods couple the two modalities through a shared factor, and the paper cites MMLoP for this (§1, §2). Beyond the method, the main contribution is the theory: when shared low-dimensional prompting is lossless and that gradient training finds it, both in the linear surrogate (Prop. 3.1, new Cor. B.2, new Props. 3.1A–3.1B), and what cross-modal signal any prompt can act on (Thm. 3.4, new Cor. B.3). These results do not depend on our parameterization and also cover shared-factor prompting as in MMLoP.

If our response has addressed your concerns, we would appreciate it if you could consider reevaluating our submission.

Reviewer_VM1oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to explain the effectiveness of few-shot prompt learning in vision-language models through low-rank cross-modal alignment. The authors introduce a linear shared-latent model and formulate prompt learning as optimizing a linear transformation of text representations. Under a conditional centering assumption, they prove the existence of a population-optimal prompt operator whose rank is bounded by the shared latent dimension. The analysis is further extended to nonlinear representations, where a rank bound on the centered cross-covariance operator is established under an additional low-dimensional conditional-mean assumption. The paper also derives a Rademacher-based generalization bound for explicitly rank-constrained prompt classes. Motivated by the theoretical framework, the authors propose LR-MaPLe, which factorizes visual and textual prompts using shared latent codes and modality-specific projection matrices. Experiments on 11 vision-language benchmarks demonstrate performance comparable to MaPLe while reducing trainable parameters from 3.55M to 184K.

Critical strengths

S1. The paper attempts to explain the effectiveness of prompt learning through the geometry of shared cross-modal representations. The connection between shared latent variables, low-rank structure, and cross-modal alignment provides an interesting perspective on parameter-efficient adaptation.

S2. LR-MaPLe achieves a Macro HM of 77.69 ± 0.44 compared with 77.27 ± 0.54 for MaPLe, while using approximately 19 times fewer trainable parameters. The evaluation covers 11 benchmarks and three random seeds, providing useful evidence that shared-factor prompt parameterization can preserve downstream performance with substantially reduced parameter costs.

Critical weaknesses

W1. The title frames prompt learning as Low-Rank Manifold Alignment. However, the theoretical results primarily characterize the ranks of linear prompt operators, cross-covariance operators, and factorized prompt matrices. The paper does not explicitly define the underlying modality-specific manifolds, formulate a manifold alignment objective, or establish geometric properties involving manifold structure, such as local neighborhoods, tangent spaces, or mappings between data manifolds.

A low-dimensional shared latent variable or a low-rank cross-covariance operator does not, by itself, establish manifold alignment in the usual geometric sense. Consequently, the use of "manifold alignment" appears stronger than what is formally supported by the theoretical analysis.

W2. The central theoretical result, Proposition 3.1, models prompt learning as a linear operator PP applied directly to the text input, followed by a frozen linear encoder: g(y;P)=WyPyg(y;P)=W_yPy. However, practical prompt learning methods such as CoOp and MaPLe optimize continuous prompt tokens inserted into pretrained transformers. These tokens interact with other tokens through nonlinear self-attention layers, rather than acting as a linear transformation of the input representation. Furthermore, Equation (4) considers a population contrastive objective over independently sampled image-text pairs, whereas CoOp uses a supervised classification objective with class-specific text prototypes.

These differences concern both the model parameterization and the optimization objective. Consequently, the existence of a low-rank optimizer under the proposed linear surrogate does not directly characterize the optimization or generalization behavior of actual prompt learning methods. Although the authors acknowledge that the formulation is a surrogate, the paper does not establish a mathematical connection between this surrogate and transformer-based prompt tuning. This substantially limits the applicability of the central theoretical result to the phenomenon the paper aims to explain.

W3. Proposition 3.1 establishes that there exists a population-optimal prompt operator satisfying $rank(P) \leq k $. However, this result does not imply that unconstrained empirical optimization will converge to such a solution, nor that the learned operator will remain approximately low rank during training. This distinction is central to the paper's motivating question: why can conventional prompt learning methods such as CoOp generalize from very few labeled examples despite optimizing high-dimensional continuous parameters?

The subsequent generalization analysis in Proposition 3.8 applies to an explicitly rank-constrained hypothesis class. However, conventional CoOp does not impose this rank constraint, and Proposition 3.1 does not establish that unconstrained empirical optimization effectively operates within this hypothesis class.

Therefore, the paper establishes that low-rank solutions can exist under certain assumptions, but does not demonstrate that this property explains the empirical sample efficiency of existing prompt learning methods. Addressing this gap would require an analysis of optimization dynamics, implicit regularization, or the effective complexity of the hypothesis class induced by actual prompt parameterizations.

Questions for authors

Please see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer VM1o (1/2)

We thank the reviewer for the constructive feedback. Below, we respond to each comment. All changes are marked in red in the revised paper.


W1: The title frames prompt learning as "Low-Rank Manifold Alignment", but the paper does not define modality manifolds or a manifold alignment objective.

R1: Thank you for the suggestion. Our results concern ranks and spectra of linear and cross-covariance operators: the shared structure is a kk-dimensional latent coordinate that indexes the conditional means of both modalities. "Low-rank cross-modal alignment" names this more precisely, so we have changed the title to "Understanding Prompt Learning as Low-Rank Cross-Modal Alignment" (subject to venue rules) and replaced "manifold" throughout the text.


W2: Prop. 3.1 models the prompt as a linear operator with a contrastive objective, whereas CoOp and MaPLe optimize tokens inside transformers with a supervised classification objective.

R2: The linear-operator formulation follows the common practice of theoretical work on prompt tuning and multi-modal contrastive learning, which studies controlled abstractions to isolate the mechanism [1–4]. In the revision we establish the mathematical connection to practical prompt learning in two directions, the objective and the parameterization (new paragraph at the end of §3.3; new Appendix B.1).

  • Objective (new Cor. B.2): in the linear surrogate, the low-rank optimum extends to the supervised cross-entropy over class texts used by CoOp. Let an image xx have label cc, and let the class texts y1,…,yCclsy_1, \dots, y_{C_{\text{cls}}} satisfy E[yc′∣x,c,Πy1,…,ΠyCcls]=Πyc′\mathbb E[y_{c'} \mid x, c, \Pi y_1, \dots, \Pi y_{C_{\text{cls}}}] = \Pi y_{c'}. Then the cross-entropy over class-text prototypes satisfies LCE(PΠ)≤LCE(P)L_{\text{CE}}(P\Pi) \le L_{\text{CE}}(P) for every PP and every τ>0\tau \gt 0, so a minimizer of rank at most kk exists. The proof of Prop. 3.1 applies verbatim, because the cross-entropy is convex in the logits and the logits are linear in PyPy; i.i.d. candidates are not needed. The new dynamics result (see R3) also holds for LCEL_{\text{CE}}. With a single fixed text per class, the condition requires the prototypes to lie in the shared subspace. This removes the difference between the two loss formulations; the encoders stay linear and the scores unnormalized.
  • Parameterization (new Cor. B.3): the cross-covariance results extend to transformer token prompts. Let θ\theta be any prompt parameter, and let the image feature fθ(x)f_\theta(x) depend on the image only and the text feature gθ(y)g_\theta(y) on the text only. This holds for continuous tokens inserted into CLIP's transformers in one or both branches (CoOp, IVLP, MaPLe, LR-MaPLe). Then Cxy(θ)C_{xy}(\theta) factorizes through the latent as in Thm. 3.4. Under its conditional-mean assumption (CM), i.e. the centered conditional means a(z),b(z)a(z), b(z) of the two features given the latent zz each lie in a subspace of dimension at most kk, rank⁡Cxy(θ)≤k\operatorname{rank} C_{xy}(\theta) \le k. When (CM) holds only approximately, ∑j>kσj(Cxy)≤min⁡{εx∥b∥L2,εy∥a∥L2}\sum_{j \gt k} \sigma_j(C_{xy}) \le \min\lbrace \varepsilon_x \Vert b \Vert_{L^2}, \varepsilon_y \Vert a \Vert_{L^2} \rbrace, where εx,εy\varepsilon_x, \varepsilon_y are the L2L^2 distances of a,ba, b from such subspaces. The proof of Thm. 3.4 only uses x⊥y∣zx \perp y \mid z, which implies fθ(x)⊥gθ(y)∣zf_\theta(x) \perp g_\theta(y) \mid z.

The second result is a statement about actual token prompts, at the level of the cross-covariance, and measurements on real CLIP ViT-B/16 features are consistent with it. With learned MaPLe and LR-MaPLe prompts on OxfordPets, the cross-covariance has 95%-energy rank 7–8 (Table 9). On frozen features of the 11 datasets (new Exp. 2, Appendix G.1, Table 10), it is 5–26, below the class-count ceiling C−1C-1 on every dataset, and the measured tail beyond 16 directions is 0.39–0.65 of the bound above.

DatasetC−1C-1Rank of CxyC_{xy}Tail / bound
ImageNet999200.57
SUN397396260.59
StanfordCars195100.44
OxfordFlowers101190.56
Food101100230.65
UCF101100250.61
Caltech10199200.63
FGVCAircraft9960.41
DTD46200.59
OxfordPets3690.39
EuroSAT95–

A rank bound on the transformer token matrix, and low-rank optimality for the cosine-normalized CoOp model, are beyond the present analysis; §3.3 and the Limitations state this.


References

  • [1] Wei et al. "Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning." NeurIPS 2021.
  • [2] Oymak et al. "On the role of attention in prompt-tuning." ICML 2023.
  • [3] Malladi et al. "A kernel-based view of language model fine-tuning." ICML 2023.
  • [4] Nakada et al. "Understanding multimodal contrastive learning and incorporating unpaired data." AISTATS 2023.
AuthorsOpenReview ↗

Response to Reviewer VM1o (2/2)

W3: Prop. 3.1 is an existence result and does not show that unconstrained training converges to a low-rank solution; Prop. 3.8 applies only to a rank-constrained class.

R3: Thank you for this suggestion. We have added the optimization analysis for the surrogate (§3.2; proofs in Appendix B).

  • Dynamics (new Prop. 3.1A). Assume, as in Prop. 3.1, conditional centering (5), i.e. E[y∣x,Πy]=Πy\mathbb E[y \mid x, \Pi y] = \Pi y with Π\Pi the projector onto the shared text subspace, and bounded support. Then: (i) LCL_C is convex in PP; (ii) along projected gradient flow on {∥P∥F≤B}\lbrace \Vert P \Vert_F \le B \rbrace, the nuisance component ∥Pt(I−Π)∥F\Vert P_t(I-\Pi) \Vert_F is non-increasing from any initialization; (iii) from P0=0P_0 = 0, projected gradient descent keeps Pt=PtΠP_t = P_t\Pi, so rank⁡(Pt)≤k\operatorname{rank}(P_t) \le k for all tt, and LC(Pt)L_C(P_t) converges to the global optimal value at rate O(1/t)O(1/t). Gradient training without an explicit rank constraint therefore stays at rank at most kk and reaches the optimal value. The reason is short: if P=PΠP = P\Pi, the softmax weights depend only on (xi,Πyi)(x_i, \Pi y_i), so conditional centering gives ∇LC(P)=∇LC(P)Π\nabla L_C(P) = \nabla L_C(P)\Pi; and convexity together with Prop. 3.1 gives ⟨∇LC(P),P(I−Π)⟩≥0\langle \nabla L_C(P), P(I-\Pi) \rangle \ge 0 for every PP.
  • Finite samples (new Prop. 3.1B). Along gradient flow on the empirical loss with NN samples, ∥Pt(I−Π)∥F≤∥P0(I−Π)∥F+tεN\Vert P_t(I-\Pi) \Vert_F \le \Vert P_0(I-\Pi) \Vert_F + t\varepsilon_N, with εN=O~(GdyC/N)\varepsilon_N = \tilde O(G d_y \sqrt{C/N}) and GG a bound on the per-batch gradient. From P0=0P_0 = 0, the empirical trajectory thus stays within Frobenius distance tεNt\varepsilon_N of the rank-constrained class HB,k\mathcal H_{B,k}. The bound grows with dyd_y and with training time, so it controls horizons t≪1/εNt \ll 1/\varepsilon_N.
  • Generalization (new Cor. 3.8A). Since Prop. 3.1 places a global minimizer in HB,k\mathcal H_{B,k}, rank-kk ERM has excess risk of at most twice the Prop. 3.8 bound relative to the unconstrained population optimum. Remark 3.9 now presents Prop. 3.8 as a guarantee for explicitly rank-constrained classes; we do not claim it for the trajectory of training without a rank constraint. Its comparison with the unconstrained class has been removed, since a norm-based bound for the Frobenius ball has no explicit dyd_y dependence.

Experiment (new Exp. 1, Appendix C.3, Fig. 2). It illustrates the mechanism and the predicted scaling (Gaussian inputs and discrete steps, slightly outside the stated assumptions): linear shared-latent model with k=4k = 4, d=16d = 16, 300 projected steps from P0=0P_0 = 0, 5 seeds. The table reports the off-subspace share ν=∥PT(I−Π)∥F/∥PT∥F\nu = \Vert P_T(I-\Pi) \Vert_F / \Vert P_T \Vert_F at τ=0.07\tau = 0.07.

NNGradient descent, centering holdsGradient descent, centering violatedAdam, centering holdsCosine scores
292^{9}0.1860.240.490.76
2112^{11}0.0940.190.480.68
2132^{13}0.0510.170.470.64
2152^{15}0.0260.160.460.63
2172^{17}0.0120.160.460.63

With gradient descent, ν\nu decays as N−1/2N^{-1/2} when conditional centering holds (fitted slopes −0.50, −0.51 and −0.56 for τ=0.2,0.07,0.03\tau = 0.2, 0.07, 0.03 at N≥214N \ge 2^{14}) and plateaus when it is violated. The last two columns mark the scope of the result: with Adam or with cosine-normalized scores (there, ν\nu of the update PT−IP_T - I), both outside the theory, ν\nu does not vanish.

The implicit regularization of attention-based token prompts is an interesting direction for future work.

If our response has addressed your concerns, we would appreciate it if you could consider reevaluating our submission.

Dear Reviewer VM1o,

Thank you again for your review. We would like to ask whether our responses have addressed your concerns. In brief:

  • W1 (title). The title is now "Understanding Prompt Learning as Low-Rank Cross-Modal Alignment".
  • W2 (linear operator and contrastive objective). In the surrogate, the low-rank optimum also holds for CoOp's cross-entropy over class texts (Cor. B.2). The cross-covariance results extend to transformer token prompts (Cor. B.3), with measurements on CLIP features of 11 datasets (Tables 10 and 11).
  • W3 (existence only, no convergence). We added the optimization analysis: from zero initialization, gradient descent without a rank constraint keeps rank at most kk and reaches the optimal value (Prop. 3.1A), with a finite-sample bound (Prop. 3.1B), a generalization corollary relative to the unconstrained optimum (Cor. 3.8A) and simulations (Figure 2).

The latest revision also adds a roadmap of the analysis in §3 and an overview figure of how the theory and the experiments relate (Figure 4, Appendix H); these changes are marked in blue.

If any point remains unclear, we are glad to clarify it. If our responses have addressed your concerns, we would appreciate it if you could consider updating your rating.

We appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and ACs,

Thank you all for your time and constructive feedback. We are encouraged by the positive comments on our work, such as the interesting perspective that connects shared latent structure, low rank and cross-modal alignment (Reviewer VM1o, Reviewer GPYQ), the theoretical analysis covering both linear and nonlinear settings (Reviewer g7Ce, Reviewer GPYQ), and MaPLe-level accuracy with about 19× fewer trainable parameters (Reviewer VM1o, Reviewer g7Ce).

We have uploaded a revised paper with all changes marked in red. Every comment is answered in full in the response to the corresponding reviewer, so each response can be read on its own, and the responses reproduce the key tables of the new experiments. For reference, the main changes are:

  1. Gradient dynamics (new Props. 3.1A–3.1B, §3.2). In the linear surrogate, gradient training from zero initialization finds the low-rank solution without any rank constraint, and finite-sample trajectories stay close to it.
  2. Objective and parameterization (new Cors. B.2–B.3, Appendix B.1). Within the linear surrogate, the low-rank optimum and the dynamics extend to the cross-entropy over class texts used by CoOp. At the level of the cross-covariance, the results extend to continuous token prompts in CLIP's transformers, with an explicit spectral-tail bound.
  3. Approximate structure (new Cors. 3.8A and B.4). In the surrogate, rank-constrained ERM has no approximation loss, and the loss of rank-rr prompts is bounded when conditional centering holds only approximately. Prop. 3.8 is now stated per independent batch (m=N/Cm = N/C, as in Prop. 3.1B), with its loss-range constant made explicit.
  4. New experiments (Appendix C.3 and new Appendix G): a synthetic study of the dynamics (Exp. 1), assumption diagnostics on real CLIP features of 11 datasets and COCO (Exp. 2), shared versus independent codes (Exp. 3), and one shared subspace for the whole network, which reaches 77.76 ± 0.24 Macro HM with 5,192 parameters (Exp. 4).
  5. Title and scope. The title is now "Understanding Prompt Learning as Low-Rank Cross-Modal Alignment" (subject to venue rules), and the abstract now reads "We study a sufficient mechanism behind it".

The numbering of existing results is unchanged. In the responses we keep three levels apart: what is proved in the linear surrogate, what holds for nonlinear prompted encoders at the level of the cross-covariance, and what the experiments show. We would be glad to discuss any point in more detail.

Best regards,

Authors of Submission 40082

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.