- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Paper Summary
The paper attributes two prompt-engineering failure modes, "Instruction Neglect" and "Paraphrase Sensitivity," to a mismatch between distances in prompt-text space and distances in the model's output ("functional") space.
Theory.
- It models a prompt as a point in continuous prompt-embedding space and defines as a map into . The prompt space is equipped with the pullback metric (Eq. 2).
- It states three theorems:
- attention saturation partitions the prompt manifold into and (Thm. 1);
- -null foliations cause Instruction Neglect (Thm. 2);
- unbounded transversal sectional curvature causes Paraphrase Sensitivity (Thm. 3).
Algorithm. PGDE computes an "empirical pullback distance" : the RMS output distance, over calibration inputs, between greedy outputs of the base prompt and of a candidate prompt. It discards candidates with below a percentile threshold and picks the best remaining candidate by task loss.
Validation. On Mistral-7B and Llama-3-8B/70B, the paper reports:
- gains over APE/OPRO/EvoPrompt on IFEval and GSM8K (e.g., "+8.6%" on Llama-3-8B IFEval);
- 80–90% lower output variance under paraphrasing (Table 3);
- roughly 50% lower search wall-clock time.
Critical strengths
None. Measuring prompt candidates by output-space divergence rather than text similarity is a sensible practical heuristic. However, it is not substantiated as the source of the gains (see Weakness 2), and it does not by itself support acceptance.
Critical weaknesses
Critical Weaknesses
-
The central theory does not hold, and the algorithm does not use it. Theorem 1's "partition" is the trichotomy of real numbers applied to and a threshold. With the paper's own setup (, ), has rank at most 76,800, so for prompts longer than about 18 tokens identically and Theorem 3 (stated on ) is vacuous. Theorem 3 also contradicts Definition 5, which requires where is injective. PGDE itself uses greedy decoding and none of the theoretical quantities; is simply an output-divergence heuristic.
-
The empirical claims are unreliable. The headline "+8.6" is computed against EvoPrompt rather than the strongest baseline in Table 2 (Random Proposal Search), against which the gain is +5.6. The claimed compute savings conflict with the algorithm, since already requires generating all calibration outputs. Llama-3-70B is reportedly served in FP16 on 4× L4 GPUs (96 GB), which cannot hold its ~141 GB of weights. The key ablation, PGDE's proposal generator without the filter, is missing.
Questions for authors
All questions are clarifications that could change my assessment.
- Are Figs. 12 and 13 plotted from measured data? App. F.3 says Fig. 12b "includes realistic transition noise points … [t]o reflect authentic experimental behavior."
- How was Llama-3-70B served on 4× L4 (96 GB) in FP16, and where do the IFEval and GSM8K "validation" calibration sets come from?
- What is the performance of PGDE's proposal generator without the filter, under the same budget?
Since I have not found the authors' responses to my concerns, I will keep my rating.