acceptodds
Under review as a conference paper at ICLR 2027

DiPPO: Learning Prompt Proposers with Diffusion Language Model

Abstract

Automatic prompt optimization improves the downstream performance of a frozen target language model without updating its parameters. However, many existing approaches search over prompt strings with a fixed proposer, often relying on strong external language models as proposers or critics, or on human-written initial instructions. We introduce Diffusion Prompt Proposal Optimization (DiPPO), which directly optimizes a masked diffusion language model as a prompt proposer using downstream task rewards. Bidirectional attention in the proposer enables prompt-slot infilling conditioned on the subsequent query. DiPPO samples candidate prompts and updates the proposer using only scalar rewards indicating whether the frozen target model answers correctly. It can discover prompts without a human-written seed instruction or, when one is available, use it as an anchor for preference-based refinement. Across ten benchmarks, the seed-free variant outperforms all methods that do not use a human instruction and exceeds human-written reference prompts on six tasks. The human-anchored variant improves on the human-written instruction on eight tasks, achieves the best performance among all compared methods on four tasks, and records the highest average score, including against methods that use stronger critic models. These results show that a 7B open-weight diffusion language model can not only discover effective prompts from scratch without human-written seeds, but also further improve human-written instructions when they are available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
prompt optimization, automatic prompt optimization, diffusion language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fHWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

DiPPO trains the prompt proposer itself rather than keeping it fixed. A masked diffusion LM fills an instruction slot placed immediately before the query, so bidirectional attention conditions the proposal on the query that follows; only its LoRA parameters are updated, and the sole learning signal is the scalar task score of a frozen target LM on the resulting response. No external proposer LLM and no natural-language critic are required. DiPPO-HF starts from a fully masked slot and optimizes clipped GRPO over the denoising trajectory likelihood following d2, with group-relative advantages replacing a value model. DiPPO-HS places the human-written instruction in the candidate set as a competing item and applies DPO over reward-derived preference pairs, scoring a candidate by remasking-and-reconstruction. One instruction is selected from an archive on a disjoint development set after leakage and duplication audits, then applied unchanged at test time. With Dream-7B as proposer and Qwen2.5-7B as target over ten benchmarks, plus a sweep over seven target models, the human-free variant beats every human-free baseline and exceeds the human instruction on six tasks; the human-seeded variant averages 76.75, above GEPA with a 72B reflection model.

Critical strengths
  • The claim that actually matters here is demonstrated rather than asserted: a 7B open-weight diffusion model, updated only through LoRA and only through the task metric, can replace both the strong external proposer and the natural-language critic that current prompt-optimization pipelines depend on. Table 1 covers seven target models from Phi-3.5-mini to Qwen2.5-72B, and DiPPO-HF beats the no-instruction baseline on every target/task cell reported. Most papers in this line give one target model and call it general.

  • The failure analysis is honest and useful. Section 4.1 admits that two of the five seeds on HaluEval never move beyond the human instruction, and Section 4.3 goes further: in the human-free setting every candidate generated by LLaDA scored zero, so training never began. The backbone-variance discussion explains why the infilling baselines do poorly on two tasks. The archive plus development-set selection plus audits in Section 3.3 is also the right design — it prevents the deployed prompt from being the one that happened to win on its own training batch, which is the standard failure mode of reward-driven prompt search. Reporting failures at this level of specificity is why I trust the rest of the paper more than the tables alone would warrant; it is also exactly why the best-seed policy below matters.

Critical weaknesses
  • Best-seed reporting. Section 4.1 says "Each setting is run with five seeds fixed in advance, and we report the result of the best seed in all figures and tables", while the Figure 1 caption says "Error bars show the seed mean and standard deviation." These cannot both describe the same numbers. The paper's load-bearing claims are narrow-margin and count-based: "improves on the human-written instruction on eight tasks", "highest score on five", "within 1pp of the human instruction on GSM8K and MBPP", "averaged over all tasks, DiPPO-HS scores 76.75". Taking the maximum over five seeds inflates exactly these statements, and the HaluEval discussion tells us the spread is large enough to flip conclusions. The seeds already exist, so re-reporting mean±SD with paired comparisons is a fix, not new research. This is the main reason my score is where it is.

  • No matched target-model evaluation budget, so the headline wins may be compute rather than method. The baselines are given explicit budgets — GEPA keeps its official 6,871 rollouts on HotPotQA and 2,000 on the other single-module tasks — but nowhere in the main text can I find how many target-model evaluations, denoising steps, or updates a DiPPO run consumes. DiPPO's central selling point is that it needs no critic; the honest comparison at matched target-model calls could look quite different, especially against TextGrad and GEPA with Qwen2.5-7B as critic, where DiPPO's margins are small.

  • The "human-free" framing is softer than the abstract, and the largest reported gains depend on it. SQuAD2.0 and HaluEval require a proposer-side meta-instruction describing the task before training can start, because otherwise the proposer never finds the required output convention — with LLaDA on HaluEval, every candidate scored zero. Table 3 is explicitly labelled "results with the proposer-side meta-instruction", and the numbers there are the ones quoted in Section 4.1 as gains of 78.0pp and 80.5pp over the no-instruction baseline; on HaluEval, DiPPO-HF reaches 82.85 against 54.05 for the human instruction. So the two most spectacular improvements in the paper's headline rest on a human-written task description, which is arguably the very knowledge a prompt optimizer is supposed to rediscover. I accept the paper's defense that the meta-instruction states no answer format and no labeling rule. Still, the abstract's claim that the model can "discover prompts without a human-written seed instruction" should carry that qualification where the claim is made, not in Section 4.3, and Table 2/3 should report both conditions side by side.

  • Methodologically the contribution is a composition, which raises the bar on the evidence. Section 2.2 states directly that DiPPO-HF "adopt[s] the trajectory-likelihood-based GRPO of d2 without introducing a new diffusion reinforcement-learning algorithm"; DiPPO-HS is standard DPO with a reconstruction-score log-probability surrogate, and Section 3.2's surrogate needs several masks per candidate to estimate a prompt's probability, which is not the same object the DPO derivation assumes. That is a defensible engineering contribution, but then the empirical case has to be airtight, and with best-seed reporting and no budget control it is not yet.

Questions for authors
  • How many frozen-target-model evaluations does one DiPPO-HF and one DiPPO-HS run consume per task? Can you add score-versus-number-of-target-evaluations curves for DiPPO, GEPA and TextGrad on at least HotPotQA and FEVER? (Rebuttal-feasible analysis.)

  • In Section 3.2, the DPO score for a candidate prompt is a mean log-probability of reconstructing remasked tokens, averaged over several masks. How many masks, and how stable is the resulting preference ordering? A candidate pair whose scores swap under a different mask set would make the preference label noise rather than signal.

  • Please report DiPPO-HF on all ten tasks both with and without the proposer-side meta-instruction, so the human-free claim can be read off directly.

  • When the deployed instruction differs between the seven target models in Table 1, do the winners share structure? If the instruction discovered for Qwen2.5-7B transfers to Qwen3-30B-A3B, that would be a strong practical argument for the method; if not, the per-target retraining cost needs to be stated.

5 comments

W1. Best seed reporting, its relation to Figure 1, and the results as mean and standard deviation.

A-W1. We thank the reviewer for putting this so precisely, and we apologize for not having described this clearly. We reported the maximum because deployment uses a single instruction rather than an average over runs, namely the one with the highest development set score. The same procedure was applied to the methods we compare.

We agree, however, that the best seed value is only a summary of the performance a method can reach, and that it is not an adequate statistic for claiming a small difference between methods, a count of tasks that beat the human instruction, or an advantage on the overall average. On a task such as HaluEval, where the spread across seeds is wide, no conclusion can rest on a single best seed. The revision therefore reports every main comparison as the mean and the standard deviation of the five test scores, and we rewrite claims in the text such as highest and leads on so many tasks to match that statistic.

A separate update: the revised proposer input. The DiPPO of the original submission was an answer blind condition, in which the proposer learned a prompt from the query xx and a scalar reward alone. Under that condition the proposer could infer the output convention and the correctness criteria only indirectly, from the score, and on some tasks we needed a separate experiment with a proposer side meta instruction to compensate. GEPA and TextGrad differ in implementation, but both give the optimizer textual feedback produced by comparing a prediction with the answer.

In the revision we therefore give the proposer the reference answer yy that is already in the training data as a conditioning input, that is qϕ(p∣x,y)q_\phi(p \mid x, y). This does not reproduce the feedback mechanism of GEPA or TextGrad. It is a supplement to the input, so that the proposer can reach the reference answer signal during training.

yy is given to the proposer only, and only during training. The frozen target still receives [p;x][p; x] only, and candidate rewards are computed by the same scorer as before. Development set selection is also unchanged: it compares the target's predictions with the development answers, and the proposer does not read an answer as input at development or at test time. At test time the target has no access to the answer. Apart from the proposer conditioning input, the reward, the GRPO and DPO objectives, the archive and candidate selection, the target side inference and the evaluation procedure are all the same.

In our experiments this supplement reduced the run to run spread, most clearly on the tasks that require discovering an output convention. We will update Figures 1 and 2, the definition of the method, the formulation and the DiPPO result tables accordingly. We would emphasize that this update changes only the proposer’s conditioning input; the core GRPO/DPO optimization objectives and the target-side pipeline remain unchanged.

W1, continued.

Table R3-W1.1. Human free prompt optimization. Figures are task scores, given as means with standard deviations. The DiPPO-HF row is the revised proposer input without the proposer side meta instruction. GEPA and TextGrad start from the DSPy default string with a Qwen2.5-72B-Instruct reflection model, and their HotPotQA cell does not yet have enough seeds, which we will update.

MethodGSM8KMMLUMBPPBFCLSQuAD2CLINCFEVERHaluEvalSpiderHotPotQA
None84.7660.1960.4036.140.000.1361.940.0065.7024.67
RLPrompt 82M82.21±0.5057.07±0.5861.92±1.0336.60±5.760.00±0.000.01±0.0276.58±1.970.00±0.0064.16±0.7328.47±0.99
RLPrompt 7B83.64±0.5955.66±1.8460.88±0.9440.26±2.720.00±0.0020.01±2.3576.95±2.440.64±1.1862.36±1.1928.47±0.99
infill-Dream81.60±0.5262.19±0.4261.85±0.2528.12±0.7927.60±6.4855.28±2.9279.74±1.1316.92±2.3068.53±2.7736.40±8.52
infill-LLaDA83.17±0.7256.67±7.6062.50±1.5933.06±4.1614.71±17.4013.68±23.3419.69±34.110.00±0.0065.51±0.5732.80±11.26
GEPA86.35±1.6868.07±1.0363.36±1.2930.37±0.6574.02±1.5266.75±1.1778.34±1.3631.55±7.7670.79±2.39—
TextGrad88.38±1.8662.44±2.2762.16±2.1731.34±1.0645.39±26.2239.83±29.3666.89±21.642.64±3.7469.73±1.68—
DiPPO-HF91.28±0.2870.31±0.1864.40±2.5957.46±2.5176.24±1.4073.86±0.6581.29±0.2074.78±4.5673.73±1.0260.53±1.98

Table R3-W1.2. Human seeded prompt optimization. Same convention as Table R3-W1.1. The DiPPO-HS row is the revised proposer input without the proposer side meta instruction.

MethodGSM8KMMLUMBPPBFCLSQuAD2CLINCFEVERHaluEvalSpiderHotPotQA
Human91.8169.8267.2072.9867.6372.3873.6854.0569.0534.00
TextGrad 7B91.75±0.3569.00±0.8367.76±0.3071.62±1.2567.54±0.1872.55±0.0176.32±3.5272.59±10.6369.07±0.6247.40±4.22
TextGrad 72B89.77±1.3769.13±0.6767.56±0.6273.09±2.1771.83±2.6872.28±0.0780.02±0.2960.87±10.5972.86±1.6741.73±5.62
TextGrad Opus92.10±0.5569.79±0.0267.08±1.0871.62±1.9369.17±4.1668.03±2.8480.00±1.5984.32±0.9569.46±2.2758.50±2.83
DLM-PO91.89±0.2869.80±0.0367.35±0.4474.02±1.2374.76±5.0172.17±0.2580.70±0.2757.27±4.7568.20±1.1534.00±2.66
GEPA 7B90.96±0.8669.75±0.2467.28±0.5070.67±4.5068.99±2.6774.03±0.7080.58±0.6982.76±0.6968.07±1.2957.27±2.32
GEPA 72B90.64±1.1068.81±0.9066.28±1.7474.04±3.2576.35±2.1773.91±0.8281.18±0.6081.00±1.2270.64±2.7058.00±4.23
GEPA Opus91.11±1.2869.80±0.0267.60±1.3772.82±1.6974.14±1.2973.42±2.5079.78±1.4282.81±1.7868.92±0.3761.07±1.75
DiPPO-HS91.55±0.2869.80±0.3667.80±1.4676.79±1.5578.79±0.6873.74±0.8581.04±0.5463.88±15.2373.08±1.2261.44±2.01

Q1. Target-model evaluations per run, and score against budget for DiPPO, GEPA and TextGrad.

A. Thank you for raising the budget question. We report the target-model calls per run and performance at matched budget checkpoints.

Calls per run. On a single-module task, DiPPO-HF uses 360,000 target calls in training plus 21,600 for development selection (381,600 total); DiPPO-HS uses 405,000 plus 2,400 (407,400 total). Test evaluation is excluded for every method.

Performance against evaluation budget on FEVER.

GEPA and TextGrad both start from the human instruction here, which makes DiPPO-HS the direct comparison. We also report DiPPO-HF, which starts without one. At each checkpoint we select an instruction on development performance, then evaluate on test.

FEVER evaluation budgetTextGrad 7BGEPA 7BDiPPO-HSDiPPO-HF
1,000 calls80.68 ± 0.5675.80 ± 2.06initial instruction 73.71initial instruction 60.37
10,000 calls80.33 ± 0.8780.44 ± 1.3773.71 ± 0.0060.37 ± 0.00
100,000 calls80.27 ± 0.9580.04 ± 1.0880.49 ± 0.3380.66 ± 0.31
final evaluated point80.27 ± 0.9580.24 ± 0.8880.95 ± 0.5781.09 ± 0.35

Test 16,039 examples. The baselines were extended to 405,000 calls, while the DiPPO entries are each run's final selected instruction, so the last row reports endpoints rather than an exactly matched count. DiPPO's first measured checkpoints are at 2,700 (HS) and 2,400 (HF) calls; its first update does not complete until then, which is why the 10,000-call entries still deploy the initial instruction.

GEPA reaches strong performance quickly, above DiPPO-HS at 10,000 calls, and TextGrad is also strong early. DiPPO uses more target evaluations to reach a comparable final mean. The final points lie within one standard deviation of each other, so the three are at a comparable level on FEVER.

Change from 10,000 calls to the final pointGEPA 7BTextGrad 7BDiPPO-HS
FEVER−0.20−0.06+7.23
SQuAD 2.0+2.49not extended+11.87

Extending GEPA and TextGrad to 405,000 calls produces almost no change on FEVER, while DiPPO-HS improves by 7.23 points after 10,000 calls; on SQuAD 2.0 the figures are +2.49 and +11.87. GEPA makes rapid early gains, while DiPPO improves through additional training. Time constraints prevented extending TextGrad and the HotPotQA baselines to the same budget.

SQuAD 2.0.

SQuAD 2.0 evaluation budgetGEPA 7BTextGrad 7BDiPPO-HSDiPPO-HF
1,000 calls67.49 ± 0.2266.21 ± 3.77initial instruction 67.52initial instruction 0.00
10,000 calls71.08 ± 1.2768.16 ± 6.5467.52 ± 0.000.00 ± 0.00
100,000 calls73.47 ± 2.61—73.99 ± 5.9176.90 ± 1.13
final evaluated point73.57 ± 1.3268.16 ± 6.5479.39 ± 0.9976.87 ± 1.08

Test 11,873 examples. DiPPO-HF starts from an empty instruction, hence the initial 0.00. TextGrad was not extended to 405,000 calls; its final entry is the endpoint of its own configuration, and its call counts are upper bounds computed before candidate deduplication. Additional training raises DiPPO-HS from 67.52 at 10,000 calls to 79.39, and GEPA from 71.08 to 73.57. DiPPO-HS finishes 5.82 points higher.

HotPotQA.

HotPotQA evaluation budgetGEPA 7BTextGrad 7BDiPPO-HSDiPPO-HF
1,000 calls54.56 ± 1.6847.13 ± 10.50initial instruction 33.67initial instruction 25.67
10,000 calls55.33 ± 1.4552.67 ± 2.9539.67 ± 10.3932.56 ± 11.93
final evaluated point55.33 ± 1.4553.27 ± 3.6953.44 ± 3.3659.00 ± 0.67
final call count7,816–9,25618,273–20,433204,90084,000

Test 300 examples; DiPPO's first measured checkpoints are at 3,750 (HS) and 7,712 (HF) calls. The DiPPO entries use the submission configuration, which does not give the reference answer to the proposer.

DiPPO improves here too, but GEPA finishes above DiPPO-HS far earlier. We acknowledge that GEPA gains more quickly at small budgets, and improving DiPPO's efficiency there is future work.

On cost. All comparisons here are open-weight 7B, and DiPPO uses more target rollouts. Its final mean is higher on FEVER and SQuAD 2.0, and for DiPPO-HF on HotPotQA, where DiPPO-HS falls below GEPA. Separately, GEPA and TextGrad can do better with a commercial model as critic or optimizer, which adds API cost for textual feedback on top of target evaluation; DiPPO trains an open-weight proposer from a scalar reward and makes no commercial critic calls. Proposer training and generation also cost, so this does not establish a lower total cost. The right choice depends on rollout budget, API cost and time; we will work on reducing DiPPO's rollout requirement. We thank the reviewer for prompting this analysis.

Reviewer_W3icOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiPPO trains a masked diffusion language model to propose instructions using scalar rewards from a frozen target language model. DiPPO-HF starts without a human instruction and applies trajectory-based GRPO adapted from d2. DiPPO-HS includes a human instruction among competing candidates and learns from reward-derived preferences using a reconstruction-based DPO objective. Generated prompts are archived and selected using development performance for subsequent reuse. The evaluation covers ten benchmarks, with additional experiments involving seven target models and two diffusion proposers.

Critical strengths
  1. DiPPO learns from scalar task scores without requiring natural-language critiques or updates to the target model. For single-module tasks, the selected instruction is reused without running the proposer at test time.

  2. The experiments include established prompt optimizers, stronger reflection models, seven target backbones, and two diffusion proposers.

Critical weaknesses
  1. Section 4 reports the best of five seeds but does not specify how that seed is selected. Figure 1 additionally shows seed means and standard deviations, leaving the relationship between the plotted statistics and headline comparisons insufficiently explained.

  2. The benefit of proposer training is not isolated. A frozen Dream proposer with matched sampling, evaluation, and archive-selection budgets is missing. Thus, gains could partly reflect additional search rather than parameter updates. DiPPO’s budgets and key settings, including training steps, LL, GG, KK, and preference masking, are unspecified.

Questions for authors
  1. How is the best seed selected, and why do results differ across Figure 1 and Tables 1–2? Please provide comparable five-seed means and uncertainty.

  2. Does DiPPO outperform a frozen Dream proposer under matched generation, evaluation, and selection budgets?

  3. What are the exact training settings, data splits, reconstruction-score estimator?

4 comments

Q1. How was the best seed chosen, why do Figure 1 and Tables 1 and 2 disagree, and can you report five seed means with uncertainty?

A1. We apologize for not describing our evaluation setup more clearly. The manuscript did not make clear the rule by which a prompt is selected within a seed, nor the rule by which the seeds are then aggregated, and what Figure 1 and the tables carry is the variation that arises when the same experiment is reproduced. The revision will make the reporting uniform across every main result.

Selection and aggregation. Every method is run five times with different random seeds. Within a run we select the single prompt with the highest development set score and evaluate that prompt on the test set. The figure in the submitted tables is the maximum of those five test scores. We report the maximum because the best seed is how a prompt is actually used: a deployment takes one instruction, the one that scored best on held out data. The procedure is identical for every method we compare. As the reviewer asks, we attach the mean and the standard deviation in Tables R2.1 and R2.2.

The revised proposer input. The DiPPO of the original submission is an answer blind condition, in which the proposer learns a prompt from the query xx and a scalar reward alone, so that only the generation structure is controlled. On review we found that this condition leaves the proposer to infer the output convention and the correctness criteria indirectly from the score, and that a separate meta instruction is needed to compensate. GEPA and TextGrad differ in implementation, but both give the optimizer textual feedback that can carry reference answer information, since it comes from comparing a prediction with the answer.

In the revision, therefore, we supply the reference answer yy that is already in the training data as a conditioning input to the proposer, that is qϕ(p∣x,y)q_\phi(p \mid x, y). Only the proposer reads yy, and only during training. The frozen target still receives [p;x][p; x] only, candidate rewards are computed by the same scorer, and yy enters neither development set selection nor test deployment. Everything except the proposer input is therefore unchanged: the reward, GRPO and DPO, candidate selection, target side inference and the evaluation procedure. The yiy_i input mainly reduces the seed variance on the tasks that require discovering an output convention. We will update Figures 1 and 2 and the formulation accordingly. We would emphasize that only the proposer input changes, not the algorithm itself, and the figures for this setting are the ones in Tables R2.1 and R2.2.

Q1, continued.

Table R2.1. Human free prompt optimization. Figures are task scores, given as means with standard deviations. The DiPPO-HF row is the revised proposer input without the proposer side meta instruction. GEPA and TextGrad start from the DSPy default string with a Qwen2.5-72B-Instruct reflection model, and their HotPotQA cell does not yet have enough seeds, which we will update.

MethodGSM8KMMLUMBPPBFCLSQuAD2CLINCFEVERHaluEvalSpiderHotPotQA
None84.7660.1960.4036.140.000.1361.940.0065.7024.67
RLPrompt 82M82.21±0.5057.07±0.5861.92±1.0336.60±5.760.00±0.000.01±0.0276.58±1.970.00±0.0064.16±0.7328.47±0.99
RLPrompt 7B83.64±0.5955.66±1.8460.88±0.9440.26±2.720.00±0.0020.01±2.3576.95±2.440.64±1.1862.36±1.1928.47±0.99
infill-Dream81.60±0.5262.19±0.4261.85±0.2528.12±0.7927.60±6.4855.28±2.9279.74±1.1316.92±2.3068.53±2.7736.40±8.52
infill-LLaDA83.17±0.7256.67±7.6062.50±1.5933.06±4.1614.71±17.4013.68±23.3419.69±34.110.00±0.0065.51±0.5732.80±11.26
GEPA86.35±1.6868.07±1.0363.36±1.2930.37±0.6574.02±1.5266.75±1.1778.34±1.3631.55±7.7670.79±2.39—
TextGrad88.38±1.8662.44±2.2762.16±2.1731.34±1.0645.39±26.2239.83±29.3666.89±21.642.64±3.7469.73±1.68—
DiPPO-HF91.28±0.2870.31±0.1864.40±2.5957.46±2.5176.24±1.4073.86±0.6581.29±0.2074.78±4.5673.73±1.0260.53±1.98

Table R2.2. Human seeded prompt optimization. Same convention as Table R2.1.

MethodGSM8KMMLUMBPPBFCLSQuAD2CLINCFEVERHaluEvalSpiderHotPotQA
Human91.8169.8267.2072.9867.6372.3873.6854.0569.0534.00
TextGrad 7B91.75±0.3569.00±0.8367.76±0.3071.62±1.2567.54±0.1872.55±0.0176.32±3.5272.59±10.6369.07±0.6247.40±4.22
TextGrad 72B89.77±1.3769.13±0.6767.56±0.6273.09±2.1771.83±2.6872.28±0.0780.02±0.2960.87±10.5972.86±1.6741.73±5.62
TextGrad Opus92.10±0.5569.79±0.0267.08±1.0871.62±1.9369.17±4.1668.03±2.8480.00±1.5984.32±0.9569.46±2.2758.50±2.83
DLM-PO91.89±0.2869.80±0.0367.35±0.4474.02±1.2374.76±5.0172.17±0.2580.70±0.2757.27±4.7568.20±1.1534.00±2.66
GEPA 7B90.96±0.8669.75±0.2467.28±0.5070.67±4.5068.99±2.6774.03±0.7080.58±0.6982.76±0.6968.07±1.2957.27±2.32
GEPA 72B90.64±1.1068.81±0.9066.28±1.7474.04±3.2576.35±2.1773.91±0.8281.18±0.6081.00±1.2270.64±2.7058.00±4.23
GEPA Opus91.11±1.2869.80±0.0267.60±1.3772.82±1.6974.14±1.2973.42±2.5079.78±1.4282.81±1.7868.92±0.3761.07±1.75
DiPPO-HS91.55±0.2869.80±0.3667.80±1.4676.79±1.5578.79±0.6873.74±0.8581.04±0.5463.88±15.2373.08±1.2261.44±2.01

Q2. Does DiPPO outperform a frozen Dream proposer under matched generation, evaluation and selection budgets, and what is the training budget and the main settings?

A. We thank the reviewer for naming the control we were missing. We give the budget first, then the comparison against the same Dream-7B proposer left untrained. The frozen proposer has the LoRA learning rate set to zero and otherwise runs the same pipeline as DiPPO, including generation, reward, archive and development-set selection.

ItemFrozen DreamDiPPO-HF
Training steps150150
Candidates per step, G88
Reward evaluation300 development queries per candidatesame
Denoising steps1616
Archive Ktop 64 keptsame
Final selectiontop 24 rescored on 300 development queriessame
LoRA learning rate05e-5

Slot length L is matched between the two conditions and set per task. In the human free setting it is 20 for GSM8K, MMLU and MBPP, 64 for SQuAD 2.0 and CLINC150, and 128 for BFCL, FEVER, HaluEval and Spider. In the human seeded setting it is larger so that the human instruction fits: 64 for SQuAD 2.0 and CLINC150, 128 for FEVER, 144 for BFCL and 544 for HaluEval.

  • We then ran the untrained Dream-7B at the same budget. The trained proposer is higher on five of the six tasks, and two of them show a much larger difference than the rest.
TaskFrozen proposer (lr=0)DiPPO-HF (lr>0)Difference
BFCL65.4064.46-0.94
FEVER80.7581.18+0.43
CLINC15060.2461.27+1.03
Spider68.2871.35+3.07
SQuAD 2.050.1768.24+18.07
HaluEval62.7982.49+19.70

On SQuAD 2.0 and HaluEval a gap of +18.07 and +19.70 remains after the same selection procedure. We separate what training contributes into two parts.

  • Training makes good candidates frequent, which removes collapsed seeds.
ConditionCandidatesAt or above the human instructionMedian candidate score
HaluEval, frozen1,2001 to 30.00
HaluEval, trained1,2008180.74
FEVER, frozen1,2002580.62
FEVER, trained1,2007320.75

We counted the 1,200 candidates each condition generated, scored on development queries only. On FEVER good candidates are already common without training, so the deployed score moves only +0.43 while the median rises. On HaluEval the frozen proposer produces one to three per seed with a median of 0.00, and after training there are 818 with a median of 0.74. The seed standard deviation of the deployed score falls from 19.17 to 0.45. Training therefore raises how often a good candidate appears, and that is what removes the score collapses.

  • Training also fits the output format better, or raises accuracy once the format is found.

Each cell is the percentage of gradable answers / accuracy among those answers / overall accuracy.

TaskFrozen proposerDiPPO-HFWhat training adds
FEVER99.20 / 81.21 / 80.5599.99 / 81.34 / 81.34-
CLINC15089.96 / 67.74 / 60.9593.20 / 65.80 / 61.33format
HaluEval99.97 / 72.57 / 72.55100.00 / 82.70 / 82.70accuracy on top of format
SQuAD 2.084.27 / 48.36 / 41.1094.44 / 77.36 / 73.59both format and accuracy

On CLINC150 training fits the format better than the frozen proposer. On HaluEval the frozen proposer already makes almost every answer gradable, so the share is near 100 percent and what training adds is accuracy, +10.13. On SQuAD 2.0 both the format and the accuracy rise. Training therefore has a format part and an accuracy part, and which one operates differs by task.

As a reference point, this table reproduces the median seed of each condition.

Instructions the frozen proposer selected, and how they fail.

TaskTestKey phrase of the selected instructionFailure
SQuAD 2.049.29"If the answer is unanswer, then output correct answer theable from the given passage."the unanswerable convention is broken
SQuAD 2.040.81"If the answer ... is not explicitly mentioned in the passage, return the correct answer."demands an answer even when there is none
HaluEval31.92"First, you need to find the question and the candidate answer ... Then, you need to find the candidate answer in the question passage."no decision criterion, and no meaning for Yes or No
HaluEval72.38"Output 'Yes' if ... hallucinated ... Output 'No' if ... hallucinated."the same condition is attached to both Yes and No

Q3. What are the exact training settings, data splits, and reconstruction-score estimator?

A. We thank the reviewer for pointing out the missing reproduction details. The target LM is frozen throughout. The only parameters updated are a rank-16 LoRA on the attention projections of the Dream-v0-Instruct-7B proposer. In the revision the proposer additionally conditions on the reference answer of each training example, while the frozen target still receives only the prompt and the query.

Training settings.

Human free (GRPO)Human seeded (DPO)
Steps150150
Candidates8 per step (1 group, 2 inner updates)8 per step plus the human instruction
Reward batch300 shared queries300 development queries
RegularisationKL 0.4DPO beta 0.1
Learning rate5e-5, cosine, warmup 20, min ratio 0.11e-4
Denoising steps1616
Archive and final selectionretain top 64; re-evaluate top 24 on 300 development queries (60 for MBPP)retain top 48; re-evaluate top 12 on 200 clean development queries

The prompt length L is fixed before training. For GRPO it is 20 on GSM8K, MMLU and MBPP, 64 on SQuAD 2.0 and CLINC150, and 128 on BFCL, FEVER, HaluEval and Spider. For DPO it is 48 on GSM8K, MMLU and MBPP, 64 on SQuAD 2.0 and CLINC150, 128 on FEVER and Spider, 144 on BFCL, and 544 on HaluEval.

Data splits. We use the official splits where they exist and construct the missing level ourselves otherwise. The development set never overlaps the test set. Where the official test labels are held out, we use the official validation set as test.

TaskTrain poolDevTestSplit
GSM8K6,4733001,319official test; dev from train tail
MMLU1,20030014,042official test; train and dev from official validation
MBPP37460500all official
BFCL1,428300866no official split, 50 / 25 / 25 per category
SQuAD 2.02,00030011,873test = official validation; dev from train tail
CLINC15018,3333005,500all official; train and dev resampled to the test oos rate
FEVER228,27730016,039all official
HaluEval5,0003004,000no official split, 10,000 rows split 50 / 10 / 40
Spider5,9673001,034test = official validation; dev from train, by database

Reconstruction-score estimator. The two arms differ.

DiPPO-HF (GRPO) follows d1 for the log-probability estimate: we replay the generation trajectory and read each token at the step it was committed, with no remasking.

DiPPO-HS (DPO) follows the variance-reduction recipe of LLaDA 1.5. For each candidate we draw a mask rate from U(0.3, 0.8), mask that many positions, and take the mean log-probability the proposer assigns to the original tokens at those positions, conditioned on the query. The training quantity is that score minus the same score with the LoRA adapter disabled. The policy and the reference share the mask so that mask sampling noise cancels in the difference, the chosen and the rejected candidate receive independent masks, and two draws are averaged. It is a reconstruction surrogate for relative preference, not an exact sequence likelihood.

Reviewer_wh63OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DiPPO, which trains a masked diffusion language model as a prompt proposer from scalar task rewards. DiPPO-HF starts from a fully masked prompt slot and updates LoRA parameters with GRPO and diffusion trajectory likelihood. DiPPO-HS treats a human-written instruction as a competing candidate and applies DPO to reward-derived preference pairs. Experiments cover ten tasks, seven target models and two proposer models, and report better results than baselines in both the human-free and instruction-anchored settings.

Critical strengths

Applying diffusion infilling to prompt generation is a clean idea.

Critical weaknesses

The human baseline is too weak to support the main claim.

The paper does not show what the learned prompts look like.

The source of the gain is not isolated from archive selection and format discovery.

Only 2 pictures and no appendix.

The writing can't reach the standard.

Questions for authors

Who wrote the human instructions for SQuAD 2.0, FEVER and CLINC? Can stronger public instructions be used for comparison?

Can the paper show two or three learned prompts and explain why they are not dataset-specific format templates?

How much of the gain comes from the learned proposer and how much from the archive and development-set selection?

Does an autoregressive proposer with the same training objective perform worse than the diffusion proposer?

5 comments

Q1. The human baseline appears weak. Where do the human written instructions come from, and have stronger public instructions been considered?

A1. We thank the reviewer for raising this. The human written instructions for all three tasks in question were written by us. Where a task has no official zero-shot instruction, we wrote one by restating the task's own evaluation protocol in plain language: the extractive answer format together with the abstention rule for SQuAD 2.0, the three-way label decision made only from the given evidence for FEVER, and closed-set intent classification together with the out-of-scope rule for CLINC150. The revision lists all of them, with the criteria we followed, verbatim in the appendix.

Following the reviewer's suggestion, we additionally scored every public SQuAD-v2 template we were able to find, drawn from PromptSource, FLAN v2 and Super-NaturalInstructions, and we report the result in Table R1.1.

Table R1.1. Public instructions on SQuAD 2.0. All instructions are scored on the same target (Qwen2.5-7B-Instruct) and the same full test split (11,873 examples) as the human baseline reported in the paper. SNI denotes Super-NaturalInstructions.

Ours (author written)SNI task469SNI task075FLAN v2 t8PromptSourceFLAN v2 t2FLAN v2 t9FLAN v2 t7FLAN v2 t1FLAN v2 t5
Score67.6335.3732.5836.2334.6825.6127.4728.0722.8127.86

The two strongest of them read as follows.

PromptSource (34.68; BigScience, 2022). Answer the question depending on the context. If you can't find the answer, please respond "unanswerable".

FLAN v2 t8 (36.23; Google, 2022). What is the answer? (If it cannot be answered, return "unanswerable")

As Table R1.1 shows, each public template satisfies only one of the two requirements. PromptSource and the FLAN v2 templates ask the model to abstain with unanswerable but do not ask for the exact span when an answer exists, so they score on the no-answer side and earn almost nothing under exact match on the answerable side. The two SNI templates ask for the shortest span but carry no abstention rule, so they fail on the other half. Our baseline instruction is the only one that states both parts.

For FEVER and CLINC150 we found no public template that meets the protocol we use. For FEVER this means gold evidence supplied, the three official labels as literal strings, and exactly one label returned with no explanation. For CLINC150 it means the same list of 150 intents together with the oos condition.

The three instructions, with the scores reported in the paper, are as follows.

SQuAD 2.0 (67.63). Answer the question using only the passage. Reply with the exact span from the passage and nothing else. If the passage does not contain the answer, reply with exactly: unanswerable.

FEVER (73.68). Decide whether the evidence SUPPORTS the claim, REFUTES the claim, or is NOT ENOUGH INFO to do either. Judge only from the evidence given, not from your own knowledge. Answer SUPPORTS or REFUTES only if the evidence itself verifies or contradicts the claim; if it is about something else, or leaves any part of the claim unverified, answer NOT ENOUGH INFO. Respond with exactly one of: SUPPORTS, REFUTES, NOT ENOUGH INFO and nothing else.

CLINC150 (72.38). Classify the user's utterance into exactly one of the listed intents. Respond with only the intent name, nothing else. If none of the listed intents applies, respond with 'oos'.

The human written instructions could certainly be elaborated further, but to the best of our knowledge no further public templates exist and ours follow every published guideline for each task, so we adopted these three as our baselines.

Q2. Are the learned prompts simply output format templates, so that the reported gains reflect formatting rather than task content?

A2. Thank you for this insightful question. The learned prompts are indeed specialized to each task, and we agree that on some tasks they are close to format templates. To separate the share that the output convention explains from the share that it does not, we evaluated a one line instruction carrying the output convention and nothing else, for example "Answer with Yes or No only.", under otherwise identical conditions. The results are given in Table R1.2.

Table R1.2. Format only control across nine tasks. All figures are task scores, reported as five seed means measured without any meta instruction. The last column is DiPPO-HS without meta minus the format only control.

TaskNo instructionFormat onlyHumanDiPPO-HFDiPPO-HS- format only
HaluEval0.0020.3054.0571.0370.74+50.44
BFCL36.1463.7472.9858.2975.77+12.03
GSM8K84.7679.2391.8190.7090.51+11.28
CLINC1500.1369.2072.3862.0074.73+5.53
SQuAD 2.00.0073.7067.6354.8677.79+4.09
FEVER61.9477.3073.6880.3381.25+3.95
Spider65.7071.7669.0569.7971.20-0.56
MMLU60.1970.6669.8270.1370.07-0.59
MBPP60.4067.4067.2064.3265.64-1.76

Table R1.2 separates the nine tasks into two groups.

Tasks where format alone is enough. On Spider, MMLU and MBPP the one line format instruction already matches or exceeds the human instruction, and our method adds nothing on top of it. The reviewer's reading is correct for these three tasks. The target already knows how to carry out the task, and what the prompt mainly has to settle is the shape of the output.

Tasks that format alone does not explain. On the remaining tasks a residual remains above the format only control, reaching 50.44 points on HaluEval, 12.03 on BFCL and 11.28 on GSM8K. It includes the decision criteria, the reasoning steps, and the rule for when to abstain, among others.

To look at that residual directly we scored the pieces of one run's deployed FEVER instruction. It splits into a guide, which describes the task and names none of the three labels, and a convention, which states which answer goes with which case. Table R1.3 reports them.

Table R1.3. Decomposing a FEVER instruction generated by DiPPO-HS. We divided the generated instruction into a guide part, a convention part and the whole, removing the other part in turn and rescoring. The figures come from a new scoring run, in which the human instruction reproduces the 73.68 reported in the paper.

PieceScore
The guide alone75.67
The convention alone79.36
The guide and the convention81.10
The deployed instruction in full81.63

The deployed instruction reads as follows. The guide is the first sentence pair, the convention the three that follow, and the remainder is an ungrammatical tail.

In: In this task, you will be given a evidence and and a claim. You evidence to determine if evidence is enough to support the claim, or if it can't be determined with the given evidence. If you evidence enough information to enough to support the claim, the output is SUPPORTS. If the evidence enough information to contradict the claim, the answer is REFUTES. If there don't enough information provided to make the decision, the answer is NOT ENOUGH INFO. The evidence is a in the of the evidence and will evidence space in brackets you. groups. embedded'sB). The output is be one of the output (

The two parts have different jobs and each one carries score. The guide contains none of the three labels, so nothing in it can be read as an output format, yet on its own it reaches 75.67, which is 13.73 above the 61.94 of the Table R1.2 no instruction row. The convention alone reaches 79.36. Each is worth something on top of the other: adding the guide to the convention is worth 1.74 points and adding the convention to the guide is worth 5.43, for 81.10 together, and the deployed instruction, which also carries the tail, reaches 81.63. Rerunning the same instruction moves the score by at most 0.67 points. The other prompts are given in Appendix A.

Our conclusion is therefore as follows. The learned prompts are certainly task specific, and on some tasks they are close to output format templates. On HaluEval and BFCL, among others, a gain remains that the format control does not explain, so the whole effect cannot be read as a format template effect alone. We also accept the point the reviewer raises. There are tasks for which format is not enough, and we read DiPPO as optimizing exactly that.

Q3 How much of the gain comes from the selection procedure and how much from training the proposer?

A3. Thank you, this gets at the core of the method. We measured the two shares separately, and explain each through an ablation.

Table R1.4. Ablation of the candidate archive. The variant drops pool preservation and keeps only the candidates from the final step. Figures are task scores, given as means and standard deviations over seeds.

TaskDiPPO-HF, no archivingDiPPO-HFDiPPO-HS, no archivingDiPPO-HS
FEVER80.32 ± 0.6380.33 ± 1.1377.60 ± 3.3880.87 ± 0.47
SQuAD 2.040.35 ± 0.3362.36 ± 19.7473.56 ± 5.4477.94 ± 1.31

The archive is the mechanism that keeps the good candidates discovered at any point during training available to the final selection, rather than deploying only whatever the last step happened to produce. On FEVER under HF, turning the archive off costs 0.01 points, which is to say nothing at all. On SQuAD 2.0 under HF the same variant stays at a collapsed score of about 40 points, because there the good candidates are rare and the final step frequently holds none of them. The archive carries a good candidate through to the deployed instruction.

The first effect of training is that it raises the deployed score where good candidates are rare.

Table R1.5. Frozen against trained proposer, from an empty instruction. Figures are task scores, given as seed means. Only the LoRA learning rate is set to zero, so the difference is the effect of updating the proposer parameters under an identical selection procedure.

BFCLFEVERCLINC150SQuAD 2.0HaluEval
Frozen proposer (lr = 0)65.4080.7560.2450.1762.79
DiPPO-HF (lr > 0)64.4681.1861.2768.2482.49
Effect of training-0.94+0.43+1.03+18.07+19.70

The effect of training also varies by task. On BFCL, FEVER and CLINC150 the frozen proposer already supplies enough deployable candidates for the selection stage to find a good one, so the difference is small. On SQuAD 2.0 and HaluEval, by contrast, differences of 18.07 and 19.70 points survive a selection procedure that is identical in both conditions, which suggests that the trained proposer is finding higher scoring instructions and putting them into the pool.

The second effect of training is that it removes collapsed seeds and lowers the chance of failure.

Table R1.6. Candidate prevalence. For each condition we generated 1,200 candidates.

ConditionCandidates above the human instructionMedian candidate score
HaluEval, frozen1 to 30.00
HaluEval, trained8180.74
FEVER, frozen2580.62
FEVER, trained7320.75

On HaluEval the frozen proposer yields only one to three candidates per seed above the human instruction while the trained one yields 818, the median rises from 0.00 to 0.74, and the seed standard deviation of the deployed score falls from 19.17 to 0.45. On FEVER good candidates become far more common while the deployed score barely moves, because the frozen proposer already gives the selection stage something to find.

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.