acceptodds
Under review as a conference paper at ICLR 2027

Prompt Search Is Not Prompt Expressivity: Continuous Prompts Beat GEPA, MIPROv2 and SIMBA, and an Exact Vocabulary Frontier

Abstract

When a prompt optimizer fails to find a good text prompt, it is tempting to conclude that none exists. We separate the two questions and give each its own evidence. Empirically, under a protocol frozen before any test row was scored, a 32-vector continuous prompt on three frozen 3-4B models achieves higher test accuracy than GEPA, MIPROv2 and SIMBA in all 18 model-dataset-regime cells (margins 7.8-34.3 points; all 18 H1 hypotheses rejected by the pre-registered procedure, with Holm adjustment over the 36 H1/H2 hypotheses at nominal level 0.05), and its lowest-accuracy run exceeds the highest-accuracy individual optimizer run in 17 of 18 cells. The ordering holds with every arm scored by one fixed decoder, and when GEPA and MIPROv2 receive the selected continuous configuration's example budget (still 5.9-18.4 points behind in the three LOW cells tested). Audits on frozen records document the scoring differences and show that removing normalised exact train-test duplicates changes the gaps by at most 0.22 points; the comparison is one of resource-unmatched pipelines with different search objectives, so it measures what the pipelines achieve, not what text can express. Theoretically, a representation limit can nonetheless be exact. In a fully specified coupled Gaussian attention model, the risk of the best legal text prompt for a tanh target converges to q - theta^2 + (theta - s)_+^2 in the dictionary's reach s, uniformly over all query-independent masses, so for s < theta no sequence of prompt lengths removes the reach penalty; the model and proof are given in full. A controlled block checks a finite paired-response certificate and shows which premise carries the gap. In a pretrained model the vocabulary constraint is exact at the first layer, where the median ratio of token to soft directional reach over sampled readouts is 11.7-12.5%; over continuation classes satisfying our stated label-consistency and readout assumptions, such task-agnostic first-layer quantities alone cannot yield a positive task-loss lower bound.

Primary area:
foundation or frontier models, including LLMs
Keywords:
prompt tuning, prompt optimization, soft prompts, hard prompts, expressivity, attention theory, vocabulary constraints, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EEQ8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies failures of prompt search. It derives a risk frontier for the best legal text prompt in a coupled Gaussian attention model. It further characterizes first-layer vocabulary constraints in pretrained Transformers and explains why these constraints alone are insufficient to establish downstream task-loss lower bounds. Experiments on frozen language models and classification benchmarks show that a 32-vector continuous prompt outperforms several baselines.

Critical strengths

The problem studied is somehow interesting.

Critical weaknesses

The paper's presentation is unclear. It reads like an experimental report rather than a research paper. Here are a few examples, and I strongly recommend that the authors polish the writing to make the paper easier to read.

  • The ''the two questions'' in the second sentence of the abstract is an ambiguous reference. Also, the meanings of q,θ,sq,\theta,s in the abstract are unclear.
  • The introduction is too short and does not provide enough motivation for this work.
  • The definition of notations is unclear. For example, what does rr mean in Line 108?
  • Every proposition and theorem statement is very long and difficult to understand.
Questions for authors
  • Line253 says '' It is not a test of Theorem 1, Proposition 4 or Theorem 5, and no number here is a vocabulary limit.'' So what is the relationship between the results in Sec. 3 and the following experiments?
  • Could the authors clarify the logical connection between the main sections? Section 3.1 studies an asymptotic risk frontier, Sections 3.2–3.4 analyze first-layer geometry, and Section 5.1 compares optimization pipelines. These sections consider different models and different quantities, but their roles in supporting the central contribution remain unclear.
  • Could the authors explicitly define the objectives and procedures of the compilers OO, MM, and JJ in the paper? These methods are central to Section 5.4, but their differences are not explained.
Reviewer_EL6xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the distinction between prompt optimization and prompt expressivity through three lines of work: (1) an empirical comparison showing that continuous prompt tuning substantially outperforms discrete prompt optimizers such as GEPA, MIPROv2, and SIMBA on several classification benchmarks; (2) a theoretical analysis establishing an expressivity limitation for discrete prompts in a specifically constructed Gaussian attention model; and (3) an analysis of vocabulary-induced geometric constraints in pretrained models. However, the paper lacks a clear overarching research question or central claim that meaningfully connects these three components.

Critical strengths
  1. The empirical evaluation is relatively extensive, covering multiple models, datasets, and training regimes.
  2. The paper makes an effort to distinguish empirical optimization gaps from representational limitations and explicitly acknowledges the limitations of its theoretical and empirical findings.
Critical weaknesses
  1. Lack of a clear central research question and coherent narrative. The title raises an important question about the distinction between prompt search and prompt expressivity, but the paper does not convincingly organize its investigation around this question. Instead, it presents three largely disconnected lines of work without a clear central claim or a coherent evidence chain connecting them. It is unclear what overarching scientific insight the paper aims to establish beyond these individual findings.
  2. Limited relevance and generality of the main empirical study. The main experiments primarily compare different optimization pipelines on supervised classification tasks, rather than directly investigating the distinction between search and expressivity. The observed performance gaps may simply reflect the limitations of the selected text prompt optimizers in these specific settings, making their broader significance unclear.
  3. Limited connection between theory and empirical findings. The theoretical expressivity result is established in a highly specialized Gaussian attention model and does not explain the observed performance gaps in pretrained LLMs. The subsequent first-layer geometric analysis also fails to establish a meaningful connection between representational constraints and downstream task performance.
Questions for authors
  1. What is the central scientific claim of this paper, and how do the three lines of work collectively address the distinction between prompt search and prompt expressivity raised by the title?
  2. How do the authors justify the broader significance of the main empirical findings beyond the specific classification tasks and optimization pipelines evaluated?
  3. How do the theoretical expressivity results and first-layer geometric analysis help explain the empirical performance gaps observed in pretrained LLMs?
Reviewer_Vp3NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about the limits of prompt optimization. It considers the relative performance of strong "hard" (textual) prompt optimizers and an optimized soft prompt, and find that the soft prompt often outperforms strong prompt optimizers. It also characterizes a limit on the task loss for hard prompts when passed through a single-layer model, based on the vocabulary size of the model.

Critical strengths

None

Critical weaknesses

W1. The paper's stated contributions --- a comparison of prompt optimizers with a 32-vector soft prompt and a new mathematical result lower-bounding loss for single-layer fixed-vocabulary models --- are, in my opinion, not substantial enough to be a conference paper. I think a very detailed and comprehensive analysis of different prompt optimizers / soft prompting across tasks, with concrete recommendations for practitioners, could be an ICLR paper, and I think a set of mathematical results bounding loss in prompting regimes could be an ICLR paper; but unfortunately I don't think this paper goes deep enough in either direction to provide fundamentally new or exciting insights.

W2. The paper is extremely hard to follow. It uses dense, non-standard jargon and in parts reads like an experiment log rather than a conference publication. I strongly suspect that the text was largely or wholly generated by an AI model. While I generally would not consider presentation issues a critical weakness of a paper, in this case I think the presentation significantly impacts the potential value of the work.

Questions for authors

Q1. Can you explain what you see as the primary implication of the mathematical results of the paper? Specifically, what is gained from characterizing the theoretical limits on first-layer performance for hard prompts (over the existing work characterizing the expressivity of soft prompts)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.