- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper attempts to explain why prompt learning methods such as CoOp can work effectively when optimizing only a small number of continuous vectors in a high-dimensional embedding space, even with only 1–16 labeled examples per class. The authors propose low-rank cross-modal alignment as the underlying mechanism. Under a linear shared-latent model with frozen linear encoders and a conditional-centering assumption, they show that an optimal population prompt operator exists with rank at most the shared latent dimension k (Proposition 3.1). For nonlinear representations, they analyze the centered cross-covariance operator in an RKHS, establish a latent factorization under conditional independence, and prove a rank bound under a low-dimensional conditional-mean assumption (CM); when CM is only approximate, they provide a spectral-tail control (Theorem 3.4). They also derive a generalization bound for explicitly rank-constrained prompt classes (Proposition 3.8). Inspired by the shared-coordinate structure, they propose LR-MaPLe, which couples visual and textual prompts through shared factorization codes.
Critical strengths
-
The difficulty and effort of the theoretical work deserve recognition. Formulating prompt learning as a cross-modal alignment problem under a shared-latent model, and deriving rank bounds, spectral-tail controls, and generalization bounds in both linear and nonlinear settings, is a technically demanding endeavor. The authors handle conditional centering, cross-covariance operators in RKHS, trace-class operator decompositions, and Rademacher complexity. The mathematical derivation is complete and self-contained. Regardless of the final explanatory power, this effort to elevate an empirical phenomenon into formal analysis is commendable.
-
The paper provides a discussable geometric framework for prompt learning. It connects prompt learning with cross-modal covariance, CCA, and HSIC, offering a relatively complete geometric perspective. Although whether this perspective truly explains actual training remains debatable, it gives subsequent research a starting point that can be tested, refuted, or extended. For the multimodal PEFT community, such a theoretical framework is itself of reference value.
Critical weaknesses
-
The motivation is not fully pursued, and the low-rank finding actually exposes a tension with high-dimensional parameterization. The core finding is that the optimal prompt operator has rank at most k. If this conclusion holds, what is the point of CoOp optimizing tens or hundreds of thousands of parameters in a high-dimensional embedding space? A large portion of the parameter dimensions would be redundant. The paper does not directly address this question, nor does it ask further: why has direct low-rank parameterization of prompts not become mainstream? Would direct low-rank parameterization lead to performance degradation? If so, why? If not, is the high-dimensional parameterization of CoOp merely a need of training dynamics? These questions directly determine whether the paper’s motivation holds, but the paper provides no substantive analysis.
-
The paper identifies an effective factor but does not exploit it for methodological innovation, resulting in weak practical impact. The paper identifies low-rank cross-modal alignment as a potentially effective factor, but apart from LR-MaPLe—a simple shared factorization that algebraically overlaps with MMLoP—it does not further use this finding to improve methods. For example: How can the low-rank structure be extended to the entire network? Can low-rank codes be shared at every layer, every attention head, or every modality interaction so that the low-rank space covers the whole model and improves performance? Can the input or prompt injection be modified so that the low-rank structure better matches pretrained representations? These directions are not systematically explored. As a result, the paper stops at “discovering a factor that makes learning effective” without converting that factor into new design principles or performance gains. The novelty is therefore clearly insufficient.
Questions for authors
-
Do the conditional-centering assumption (5) and the conditional-mean assumption (CM) approximately hold on real CLIP features? The theory is entirely built on (5) and (CM), but the paper provides no empirical validation on real data. Can the authors give quantitative evidence, for example by estimating the deviation of or testing whether the centered conditional means lie in low-dimensional subspaces? Even a negative result would help readers understand the scope of the theory.
-
Can the paper provide a falsifiable prediction: under what training conditions will actual optimization exhibit the low-rank structure characterized by the theory? Proposition 3.1 is an existence result for the population optimum, but actual CoOp training is finite-step, non-convex, and typically uses normalized InfoNCE. Can the authors specify under what temperature, normalization scheme, initialization, optimizer, or number of training steps the learned prompt would exhibit the spectral concentration predicted by the theory? Without such testable predictions, how does the theory explain the success of actual training?
-
When CM is only approximately satisfied, can the spectral-tail control be converted into actionable performance or generalization conclusions? Theorem 3.4 provides a spectral-tail control for approximate structure, but the paper does not relate the size of the spectral tail to downstream accuracy, generalization error, or sample complexity. How large can the spectral tail be before the theoretical guarantee becomes meaningless? How should this control guide practical prompt design or rank-constraint selection?
-
In what sense does the rank bound in the linear surrogate carry over to real transformer prompt learning? The paper abstracts the prompt as a linear operator acting on text representations, whereas real CoOp injects continuous tokens into multiple transformer layers, involving attention, layer normalization, and nonlinear interactions. Can the authors provide evidence or arguments that the rank conclusion from the linear surrogate still holds, or partially holds, in the real architecture? If not, does the theoretical conclusion apply only to the surrogate model?
7 comments
Response to Reviewer GPYQ (2/5)
W2: The paper identifies an effective factor but does not exploit it for method design, e.g. by extending the low-rank structure to the entire network.
R2: Thank you for these suggestions. The paper is primarily theoretical (primary area: interpretability), with LR-MaPLe as a minimal testbed. In the revision we added two experiments motivated by the theory and one theoretical result.
A network-wide shared subspace (new Exp. 4, Appendix G.3, Table 13). All 9 prompted layers share one pair of bases and keep per-layer codes, so each modality's 18 prompt vectors lie in one -dimensional subspace. Setting of Table 2 (11 datasets, 3 seeds).
| Model | Trainable parameters | Macro Base | Macro New | Macro HM |
|---|---|---|---|---|
| MaPLe | 3.55M | 81.91 ± 0.21 | 73.78 ± 0.78 | 77.27 ± 0.54 |
| LR-MaPLe | 184,608 | 81.84 ± 0.37 | 74.52 ± 0.64 | 77.69 ± 0.44 |
| Bases tied across depth, | 20,768 | 82.13 ± 0.16 | 74.32 ± 0.19 | 77.68 ± 0.06 |
| Bases tied across depth, | 5,192 | 81.22 ± 0.35 | 75.01 ± 0.28 | 77.76 ± 0.24 |
Macro HM is comparable with 5,192 parameters (1/36 of LR-MaPLe and about 1/680 of MaPLe), and no dataset differs beyond seed variation (paired t-tests, ). In this setting, the prompts of the whole network thus fit in one low-dimensional subspace per modality.
Sharing versus rank (new Exp. 3, Appendix G.2, Table 12). At , where the rank cap is active, we give each modality its own codes at the same width and rank ceiling. HM in %, 3 seeds, the shared-code runs being those of Table 5 in the HM convention of Appendix D.1; the difference is independent minus shared, paired by seed.
| Dataset | Independent codes | Shared codes | Difference | ||
|---|---|---|---|---|---|
| 1 | DTD | 64.39 ± 2.98 | 65.47 ± 2.50 | −1.08 | 0.43 |
| 1 | EuroSAT | 72.06 ± 4.28 | 74.42 ± 2.30 | −2.34 | 0.53 |
| 2 | DTD | 65.24 ± 0.73 | 63.10 ± 1.39 | +2.15 | 0.07 |
| 2 | EuroSAT | 76.30 ± 2.55 | 76.50 ± 1.35 | −0.19 | 0.90 |
| 4 | DTD | 63.40 ± 1.93 | 65.37 ± 0.68 | −1.98 | 0.12 |
| 4 | EuroSAT | 77.31 ± 1.81 | 69.81 ± 13.94 | +7.47 | 0.49 |
The differences have both signs and stay within seed variation. A shared low-dimensional coordinate system therefore suffices in this setting, while sharing by itself gives no detectable accuracy gain; with three seeds, small effects cannot be excluded.
An approximation–estimation trade-off in the surrogate (new Cor. B.4, see A3). The loss of rank- prompt operators is bounded by a term linear in the centering error plus the rank- estimation term, and can be estimated from frozen CLIP features. Here is the operator rank in the surrogate, not the factorization width of LR-MaPLe; a rank-selection rule for token prompts is left for future work.
Response to Reviewer GPYQ (3/5)
Q1: Do the conditional-centering assumption (5) and the conditional-mean assumption (CM) approximately hold on real CLIP features?
A1: We added diagnostics on frozen CLIP ViT-B/16 features, without training, for the test splits of all 11 datasets and COCO val2017 captions (new Exp. 2, Appendix G.1, Tables 10–11). The class serves as a proxy for the latent, so ranks are bounded by and are not estimates of .
Conditional-mean assumption (CM): the cross-covariance is concentrated, the conditional means much less so. (CM) requires the centered conditional means of each modality, given the latent, to lie in a subspace of dimension at most . In the table, ranks are 95%-energy ranks. The tail is the share of the nuclear norm of beyond 16 directions, and the last column divides the measured tail by the bound of the new Cor. B.3(iii) at (see A3).
| Dataset | Rank of | Rank, image means | Rank, text means | Tail beyond 16 | Measured / bound | |
|---|---|---|---|---|---|---|
| ImageNet | 999 | 20 | 158 | 221 | 49.7% | 0.57 |
| SUN397 | 396 | 26 | 114 | 157 | 47.4% | 0.59 |
| StanfordCars | 195 | 10 | 41 | 66 | 22.3% | 0.44 |
| OxfordFlowers | 101 | 19 | 41 | 66 | 33.6% | 0.56 |
| Food101 | 100 | 23 | 47 | 71 | 37.8% | 0.65 |
| UCF101 | 100 | 25 | 55 | 68 | 41.0% | 0.61 |
| Caltech101 | 99 | 20 | 54 | 67 | 37.4% | 0.63 |
| FGVCAircraft | 99 | 6 | 25 | 42 | 15.2% | 0.41 |
| DTD | 46 | 20 | 29 | 37 | 29.0% | 0.59 |
| OxfordPets | 36 | 9 | 18 | 26 | 10.9% | 0.39 |
| EuroSAT | 9 | 5 | 7 | 8 | – | – |
The cross-covariance, which Thm. 3.4 bounds, needs 5–26 directions, while the class means themselves need far more. Low cross-covariance rank follows from (CM) but does not imply it, so these measurements support concentrated cross-modal dependence, not (CM) at the same dimension.
Conditional centering (5): there is residual predictability, largest at small . The condition requires , i.e. the part of the text outside the shared subspace cannot be predicted from the image and the part inside it. For an estimated projector , we regress the off-subspace text component on and on alone, on held-out pairs; under conditional centering, neither explains it. is the part that needs the image, and is the centering error of the new Cor. B.4 relative to the text variance (linear ridge).
| Dataset | , | , | , | , | , |
|---|---|---|---|---|---|
| COCO captions | 0.29 | 0.19 | – | 0.32 | 0.14 |
| ImageNet | 0.23 | 0.18 | – | 0.24 | 0.16 |
| SUN397 | 0.23 | 0.18 | 0.00 | 0.24 | 0.17 |
| StanfordCars | 0.26 | 0.13 | 0.00 | 0.29 | 0.08 |
| Food101 | 0.29 | 0.21 | 0.05 | 0.31 | 0.16 |
| UCF101 | 0.30 | 0.25 | 0.06 | 0.31 | 0.20 |
| OxfordFlowers | 0.26 | 0.18 | 0.06 | 0.29 | 0.15 |
| Caltech101 | 0.31 | 0.29 | 0.06 | 0.33 | 0.22 |
| FGVCAircraft | 0.09 | 0.05 | 0.00 | 0.13 | 0.08 |
| DTD | 0.14 | 0.10 | 0.03 | 0.17 | 0.12 |
| OxfordPets | 0.32 | 0.14 | 0.05 | 0.39 | 0.08 |
| EuroSAT | 0.13 | 0.01 | 0.03 | 0.26 | 0.03 |
The residual shrinks with , and the image contribution is 0.00–0.06 once spans the class information (on COCO it is 0 at ). Three caveats apply: a vanishing image contribution is necessary but not sufficient for conditional centering; linear ridge gives a lower bound on the error; and is unobserved, so the test concerns the estimate . Conditional centering is thus an idealization on real features, and the new Cor. B.4 bounds the loss incurred by the residual (see A3).
Response to Reviewer GPYQ (1/5)
We thank the reviewer for the constructive feedback and the detailed questions. Below, we respond to each comment. All changes are marked in red in the revised paper.
W1: If the optimal operator has rank at most k, what is the point of optimizing in a high-dimensional space? Would direct low-rank parameterization degrade performance, or is the high-dimensional parameterization merely a need of the training dynamics?
R1: We answer these questions in turn.
Low rank is not the same as few parameters. The theory says that range(Ay) is low-dimensional, but not where it lies. Locating a k-dimensional subspace of Rdy costs about kdy parameters, so each prompt direction must live in the full embedding space. In this sense mainstream prompt learning is already low rank: CoOp trains an nctx×512 token matrix (2K–8K parameters, rank ≤nctx≪512), and MaPLe uses nctx=2. Most of MaPLe's 3.55M parameters sit in its coupling maps, which LR-MaPLe removes.
In the surrogate, gradient training does not use the extra dimensions. Gradient descent from P0=0 stays in the rank-k set in population, without an explicit rank constraint, and within tεN of it with N samples, where εN=O~(dyC/N) (new Props. 3.1A–3.1B, §3.2). The finite-sample deviation grows with dy and with training time, so the extra dimensions are not free. The result also depends on the optimizer: it holds for gradient descent, the surrogate counterpart of the SGD used by CoOp, MaPLe and LR-MaPLe, but not for Adam, whose coordinate-wise scaling is not rotation-equivariant. In the new Fig. 2(c) (τ=0.07), the off-subspace share stays at 0.46–0.50 with Adam, while it falls from 0.19 to 0.012 with gradient descent.
Explicit low rank gives comparable accuracy on average, with dataset-dependent effects. At the default nctx=2, LR-MaPLe matches MaPLe (77.69 ± 0.44 vs 77.27 ± 0.54 Macro HM on 11 datasets), no width is uniformly best (Table 4, now with seed SDs), and a single 4-dimensional subspace for the whole network gives a comparable Macro HM with 5,192 parameters (see R2). With an active rank cap (nctx=16, Table 5; HM in %, 3 seeds), the effect depends on the dataset.
LR-MaPLe is above the matched MaPLe on DTD at most ranks and below it on EuroSAT at all ranks, with large seed variation. This is consistent with MMLoP's ablation, where rank-one prompts alone are slightly below IVLP (77.06 vs. 77.51 HM) and the gap reverses once the prompts are anchored to zero-shot CLIP.