acceptodds
Under review as a conference paper at ICLR 2027

Beyond Endpoints: Bézier Paths for Visual Prompt Tuning

Abstract

Visual prompt tuning (VPT) adapts a frozen vision foundation model by learning a few tokens shared across all images. When prompts are updated as data shift or tasks change, the distribution of image features moves from one endpoint to another. Prior theory asks which endpoints prompts can reach; we ask whether they can travel between them along the shortest path. Because shared prompts compete with each image's own content for attention, they often cannot. We quantify this restriction as the extra transport cost that shared attention imposes over the shortest path. In an exactly solvable attention model, the reachable features form a cone; unrolling it into a plane gives the exact minimum action for every pair of endpoints and explicit optimal prompts on short arcs. The analysis reveals a sharp threshold in content normalization, which measures how much attention each image's own tokens absorb: with at most two distinct levels, prompts can follow the shortest path, whereas three or more force a strictly positive extra cost unless the endpoints differ only by rescaling. This gap persists with more prompts, small content perturbations, and aligned multihead outputs. On ViT-B/16 with VTAB-1k, a quadratic Bézier prompt path that minimizes feature action reduces held-out transport action relative to linear and cross-entropy-trained paths between the same endpoints, while cross-entropy-trained paths retain higher minimum accuracy. Reachable endpoints do not guarantee a reachable shortest path: how prompts move matters, not only where they end.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Foundation Model, PEFT, Prompt Tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_WaWDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether, in frozen-attention models, as a shared visual prompt varies continuously between two endpoints, the feature distribution moves along the shortest W₂ path. The authors construct a single-head attention model with a common query and scalar output. The reachable family of distributions forms, on its positive branch, an intrinsically flat cone. Unfolding this cone yields a closed-form solution for the minimum action, as well as an intrinsic gap. In the experiments, the scalar model decomposes intrinsic, routing, and temporal costs, while ViT-B/16 + VTAB-1k compares the feature action and accuracy of different prompt interpolation paths.

Critical strengths
  1. The theoretical derivation is reasonably rigorous, and the action decomposition is clean and practical. This decomposition separates the intrinsic cost from avoidable routing and temporal costs, and can serve as a general diagnostic tool.

  2. Beyond endpoint reachability, studying whether the path itself is reachable under a shared prompt is a reasonable conceptual extension of the theory of prompt expressivity.

  3. The scalar experiments agree with the theory. The authors also candidly report unfavorable results, e.g., the CE path achieving higher minimum accuracy, and the cases in Appendix F where the gap can vanish.

Critical weaknesses
  1. The practical importance of the problem lacks justification and empirical support. The paper is motivated by prompts being updated as the distribution drifts or the task changes, but no concrete use case for continuous prompt interpolation is given.

  2. The core model assumptions are too strong. Thm 4.3 and Cor. 4.4 assume all images share a single query and that A = 0. Since adding a constant to the value merely shifts the output family, A = 0 amounts to all images having identical outputs on that head before prompting.

  3. The ViT experiments do not test the theoretical predictions, and the reporting is incomplete.

Questions for authors
  1. In the prompt-insertion layers of a pretrained ViT-B/16, what are the empirical distributions of Z_i and q_i? With per-image queries, what is the shape of the reachable distribution family? Is the estimated intrinsic gap non-negligible?

  2. Is there a downstream scenario that demonstrates a measurable improvement from prompt trajectories with lower action

Reviewer_2KcsOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the feature-distribution paths that shared visual prompts can realize between reachable endpoints. In an explicitly constructed frozen attention head, the authors characterize the attainable family through cone geometry, derive the exact minimum action, and construct optimal prompt paths. A central result identifies a threshold: at most two distinct content normalizers permit zero excess action for positive endpoints, whereas three or more produce a strict gap for non-rescaling endpoints. The paper also examines content perturbations, LayerNorm, and multihead extensions. Experiments distinguish route and timing costs and compare feature-action and cross-entropy paths in a frozen ViT. The theoretical analysis is the main contribution, while its implications for practical prompt adaptation require stronger validation.

Critical strengths

1, The paper extends prompt-tuning theory beyond endpoint expressivity to the geometry and cost of feasible transitions. Distinguishing the unconstrained transport cost, the attainable minimum action, and the additional cost introduced by route.

2, The theoretical results are exact and constructive. The cone coordinates yield an explicit minimum-action formula and a smooth prompt realization for short positive arcs.

3, The two-versus-three-normalizer threshold identifies a concrete structural source of the transport obstruction in the constructed family. The analysis also establishes persistence under small content perturbations.

4, The paper clearly reports that feature-action optimization reduces transport action while cross-entropy optimization preserves higher minimum accuracy.

Critical weaknesses

1, The ViT experiments measure final LayerNorm features followed by unit normalization, lower action in this setting does not directly validate the three-normalizer mechanism.

2, The practical benefit of minimizing feature action remains unclear. In Table 5, the shallow-prompt Bézier path reduces action, but minimum accuracy decreases from 68.8% to 67.4%. Deep prompts exhibit a similar tradeoff.

3, The numerical comparisons require clearer uncertainty and discretization analysis. The results do not report the actual refinement errors or variability across endpoint pairs.

Questions for authors

see weaknesses

Reviewer_8AuUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether shared visual prompts can trace minimum-action feature paths between reachable endpoints in frozen vision transformers. The authors argue that endpoint reachability does not guarantee that a shared-prompt trajectory can realize the shortest path in representation space. They construct an exactly realizable scalar-attention model whose attainable feature distributions form a geometric cone, derive the minimum-action path within this family, and provide a Bézier-based prompt construction. A central result identifies a sharp threshold: up to two distinct content normalizers permit zero intrinsic transport excess, whereas three or more can produce a strictly positive gap relative to unconstrained Wasserstein transport.

The paper combines 24 controlled analytical cases with frozen ViT-B/16 experiments on VTAB-1k. The numerical studies separate path geometry from traversal speed and investigate prompt counts, normalization, and architecture variants. In the practical experiments, quadratic Bézier prompt paths optimized for feature action reduce measured representation motion relative to linear and cross-entropy-oriented alternatives. For shallow and deep visual prompting, the feature-action objective lowers the action relative to cross-entropy-trained Bézier paths, although the latter retain better worst-point classification accuracy. Qualitative attention visualizations and supplementary architectural analyses further illustrate the proposed perspective.

Critical strengths

The paper's principal strength is its precise distinction between endpoint expressivity and the cost of moving between endpoints. The cone geometry, explicit minimum-action formula, sharp normalizer-count threshold, and constructive prompt realization provide an intellectually coherent account of an obstruction that conventional endpoint-based prompt analyses would miss. The derivations are complemented by stability results, making the contribution more substantial than a single pathological example.

A second strength is the separation of structural and optimization-related effects. The exact-model comparisons distinguish unavoidable intrinsic excess from additional route and timing costs, while the practical experiments hold prompt endpoints fixed when comparing interpolation strategies. The discussion of LayerNorm, multiple heads, and vector-valued outputs is also useful because it makes clear that the theoretical obstruction depends on architectural conditions, instead of automatically applying to every transformer.

Critical weaknesses

The main concern is the gap between the exact theoretical model and realistic ViT architectures. The central theorem relies on a specially constructed scalar-attention setting with a common query and controlled prompt aggregates, whereas pretrained transformers involve input-dependent attention, LayerNorm, residual connections, and multiple interacting heads. Appendix F also shows that certain architectural configurations can eliminate the obstruction. Although the ViT-B/16 experiments demonstrate reduced feature action, they do not establish whether the predicted intrinsic gap exists in practical models. This limits the generality of the theoretical conclusions.

A second concern is the unclear practical benefit of minimizing feature action. Compared with cross-entropy-trained paths, feature-action optimization reduces representation motion but yields worse minimum intermediate accuracy and loss barriers, while requiring additional computation. Moreover, the experiments mainly connect endpoints within the same task, instead of demonstrating benefits under domain shifts or continual adaptation. Therefore, the practical advantages of the proposed optimization criterion remain insufficiently established.

Questions for authors

Could the authors clarify how the three-normalizer threshold applies to realistic pretrained ViTs? Given the architectural exceptions discussed in Appendix F, it would be valuable to examine whether the predicted intrinsic gap occurs in ViT-B/16 and whether normalizer diversity explains the observed feature-action differences. This would strengthen the connection between the theoretical results and practical architectures.

Could the authors clarify when minimizing feature action would be preferable to cross-entropy-based path optimization, particularly given the latter's better intermediate accuracy? Additional per-task variability analysis, uncertainty estimates, and comparisons between the differentiable feature-action surrogate and empirical Wasserstein measurements would help establish the practical significance and robustness of the reported improvements.

Reviewer_aL6rOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies visual prompt tuning, where a frozen vision transformer is adapted by learning a few prompt tokens shared across all images. When prompts are updated, the distribution of image features moves from one endpoint to another. Prior theory asks which endpoints prompts can reach. This paper asks whether shared prompts can move the features between two reachable endpoints along the shortest path, and how much extra transport cost they pay when they cannot.

The authors use one frozen scalar attention head with a common query. Each image has a content normalizer, which measures how much attention its own tokens absorb. Only the total prompt mass and the weighted value matter, so the prompt count does not change the attainable family. With zero content value, the positive branch of this family is a flat cone. Unrolling the cone into a plane gives the exact minimum action for every endpoint sign and angle, and it gives explicit optimal prompts on short arcs. The extra cost of this optimum over the ambient shortest path is the intrinsic gap.

The main theoretical result is a sharp threshold in content normalization. For positive endpoints that are not rescalings of each other, two distinct normalizers give no gap, while three or more give a strictly positive gap at every angle. Adding prompt tokens cannot close this gap, and the gap survives small content-value perturbations. The result extends to wider heads and to aligned, unmixed multihead outputs, with a projection test for more general mixing, though independent head sums can remove the gap. A corollary uses excess action to bound the fraction of images whose prediction can change along a transition under a fixed classifier.

Beyond the intrinsic gap, the paper splits the action of a path into route excess, which comes from a longer route, and timing excess, which comes from uneven speed. Retiming removes the timing excess, while the developed chord removes both.

The experiments have two parts. A scalar study compares ten paths on synthetic cases and tests how normalizer diversity and perturbations affect the gap. A ViT-B/16 study on VTAB-1k compares endpoint trainers and then compares several paths between fixed VPT-Shallow and VPT-Deep endpoints under a fixed linear probe. A quadratic Bézier path trained for feature action lowers held-out transport action compared with a cross-entropy-trained Bézier path, while the cross-entropy path keeps higher minimum accuracy.

Critical strengths
  1. The scalar head analysis is complete for the model it studies. It covers every endpoint sign and angle, including long arcs and paths through the apex, and says exactly when smooth prompts reach the optimum. I recomputed the main numbers in Tables 1, 3 and 4, Fig. 7 and Appendix C.3 from the stated setups, and they all match.
  2. Splitting the extra action into the intrinsic gap, route excess and timing excess is simple and useful. It shows that most of the extra cost of direct prompt paths is avoidable, and that retiming alone only removes the timing part.
Critical weaknesses
  1. The experiments don't test the setting that motivates the paper. The paper motivates the problem with prompts that get updated as data shift or tasks change. But in the ViT study the two endpoints are prompts trained with different seeds on the same task. No experiment uses intermediate prompts under a shift, and the drift budget for test-time adaptation in Appendix G is never tried. So the paper never shows a case where the path between two fixed prompts matters.
  2. The theory and the ViT study barely connect. The theory covers one scalar head with a common query. Theorem 4.5 only guarantees a positive gap for very small content values, and even in the scalar model the gap is small compared with the squared distance between the endpoints. Real images have many distinct normalizers, so the threshold at three normalizers never separates real cases. The paper also shows that summing independently controlled heads can remove the gap, and every ViT layer sums many heads. So it isn't clear the gap survives in a real ViT. The abstract also leaves out the same-sign condition, though the appendix shows that opposite-sign endpoints have a gap even with two normalizers.
  3. The ViT results don't support feature-action training. The only ViT table in the main text compares endpoint trainers that do not come from this paper, and no claim depends on it. The actual path comparison is in Table 5 in the appendix, where each Bézier path wins on what it was trained for. The feature-action path has lower transport action, which is close to its own training objective measured on held-out images. The cross-entropy path is better on minimum accuracy, mean accuracy and loss barrier for both VPT-Shallow and VPT-Deep. The conclusion still says these results motivate feature-action training for frozen ViTs.
  4. Many results promised in Appendix D are missing. It says means and standard deviations across seeds and variation across endpoint pairs are reported, but Tables 2 and 5 to 8 give single numbers. The stress tests and the architecture tests with input-dependent queries, additional heads, LayerNorm and post-attention mixing have no results anywhere. Those tests are what would show whether the gap holds beyond the scalar head.
  5. Prior work is positioned loosely. The attainable family of the model is exactly the result of Petrov et al. [1] that a prefix only scales down attention on the content and shifts the output toward a fixed vector. The paper cites Petrov et al. [1] only in passing and should say its model starts from there. Pulling the W2 metric back to a parametric family goes back to Li and Montúfar [2] and Li and Zhao [3]. Minimizing the action of a distribution path through spline control points in parameter space is close to PDPO at NeurIPS 2025 [4]. None of these three works is cited.

References [1] When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations.

[2] Natural Gradient via Optimal Transport.

[3] Wasserstein Information Matrix.

[4] PDPO: Parametric Density Path Optimization.

Questions for authors
  1. For the Table 5 endpoints, what is the plain squared W2 distance between the two endpoint feature laws? If the 8-interval feature-action path is already close to it, there's no sign of an intrinsic gap in a real ViT. If it isn't, can Proposition F.5 certify any of the remaining excess, even for one layer?
  2. In Table 5 the minimum accuracy along the cross-entropy Bézier path is 68.8 for VPT-Shallow and 71.8 for VPT-Deep, so the endpoints must average at least that much. That is higher than VPT-Shallow and VPT-Deep in Table 2, even though the Table 5 endpoints are trained against a fixed linear probe. What are the endpoint accuracies, and are path accuracies measured on the 128 held-out images or on the full test split? With 128 images per task, 1-point gaps are hard to read.
  3. Did you compute the bound on prediction changes in Corollary G.1 for the ViT paths? The introduction says excess action bounds how many predictions can change along a transition. If the bound is vacuous at real margins, this claim adds little.
  4. Several table entries look like tolerances rather than measurements. All formula errors in Table 4A are exactly 1.00×10^-9, all discrepancies in Table 4B are exactly 1.00×10^-12, and the measured gap with one or two normalizers is exactly 0, though Appendix D.2 says measured gaps include optimization error. End errors in Tables 1 and 5 are bare powers of ten. Are these measured values or upper bounds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.