acceptodds
Under review as a conference paper at ICLR 2027

Differential Prompt Optimization: Pinpointing Student Errors via Teacher Audits

Abstract

Automatic prompt optimization typically guides instruction search using holistic evaluations or end-to-end task success. For complex multi-step reasoning, however, attributing a failure across a long trajectory remains difficult, as an argument may depart from a viable path long before its final line. Without fine-grained diagnostics, the optimizer must infer the cause of failure from sparse signals alone. We introduce Differential Prompt Optimization (DiffPO), a framework that supplies step-level diagnostic credit in natural language. During training, an auxiliary teacher model conducts a differential audit on failed student traces: it pinpoints the earliest step of divergence where the student's reasoning becomes unviable and abstracts that error into a generalizable rule for the prompt optimizer. To support the student at test time, the teacher also builds a library of general problem-solving strategies. The student selectively retrieves these strategies to guide its reasoning, providing high-level structure. Across different reasoning and instruction-following benchmarks, DiffPO consistently outperforms relevant prompt optimization baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt Optimization, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MTY6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Differential Prompt Optimization (DiffPO), a framework for APE. The authors argue that existing prompt optimization methods largely rely on final outcome feedback, although the optimizer may observe the model's reasoning trace, it must itself infer which intermediate reasoning step caused the final failure. DiffPO instead uses a stronger teacher model to inspect the student's reasoning trace against a reference solution, identify where the reasoning breaks down, and abstract the failure into a generalizable rule. These rules are aggregated across failures and provided to the prompt optimizer to generate improved prompts.

Critical strengths

(1) The paper addresses automatic prompt optimization, a timely and relevant problem.

(2) While the paper could compare against more baselines, the evaluation protocol is solid. The authors evaluate two model families, average results over five seeds, and include sufficient ablations to clarify the contributions of the main components.

(3) While several recent prompt-optimization methods are missing, the paper covers the most relevant prior work that is conceptually similar to the proposed approach.

Critical weaknesses

Major Weaknesses

(1) Most of the concepts introduced in this paper are already well established in the literature, including learning generalizable concepts for prompt optimization and leveraging intermediate reasoning steps rather than only the final answer (e.g. [1-3]). The authors do compare experimentally against some of the most closely related methods, therefore, I view the paper primarily as a well-executed combination of existing ideas rather than a fundamentally novel approach.

(2) I would like to see a broader evaluation against existing baselines. The authors cite several APE methods but do not compare against them experimentally (e.g., [4-5]). Several important recent methods are also neither cited nor evaluated [6-8]. Without these comparisons, it is difficult to assess the true benefits of DiffPO and whether its additional components are necessary.

(3) DiffPO relies on an offline knowledge-base construction stage, which introduces additional requirements. In particular, the method requires access to a stronger teacher model to construct the knowledge base for each new task or domain.

(4) The paper also lacks a fully fair compute comparison. The authors discuss compute/query budget, but exclude the offline cost of constructing the knowledge base, giving DiffPO additional compute that is not accounted for in the main comparison.

Minor Weaknesses

(1) The related-work section would benefit from discussing approaches that use reinforcement learning for automated prompt optimization, such as [9-10].

(2) Table captions should be above the tables.

References

[1] Boateng, Emmanuel Aboah, et al. "Concept distillation from strong to weak models via hypotheses-to-theories prompting." NAACL'2025

[2] Zhang, Tianjun, et al. "In-context principle learning from mistakes." arXiv'2024.

[3] Agrawal, Lakshya A., et al. "Gepa: Reflective prompt evolution can outperform reinforcement learning." ICLR'2026

[4] Wang, Xinyuan, et al. "Promptagent: Strategic planning with language models enables expert-level prompt optimization." ICLR'2024

[5] Guo, Qingyan, et al. "Connecting large language models with evolutionary algorithms yields powerful prompt optimizers." ICLR'2024

[6] Agarwal, Eshaan, et al. "PromptWizard: Optimizing prompts via task-aware, feedback-driven self-evolution." ACL'2025

[7] Shi, Wenhang, et al. "No loss, no gain: Gated refinement and adaptive compression for prompt optimization." NeurIPS'2025

[8] Batorski, Paweł, and Paul Swoboda. "PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data." ACL'2026

[9] Deng, Mingkai, et al. "Rlprompt: Optimizing discrete text prompts with reinforcement learning." EMNLP'2022

[10] Kwon, Minchan, et al. "Stableprompt: Automatic prompt tuning using reinforcement learning for large language model." EMNLP'2024

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Response to reviewer MTY6

We sincerely thank Reviewer MTY6 for their constructive feedback and questions; please find our response below.


Paper's novelty

While several ideas have existed before, our work addresses a distinct core problem: how to utilize a larger model to reliably improve the performance of a smaller model. A central finding of our paper is that naively using a stronger model to assist prompt optimization does not reliably improve a smaller student.

Why naive teacher-to-student transfer fails:

  • Optimizing prompts on the teacher (Teacher Only): As we show in the paper, optimizing prompts directly on the teacher improves the teacher itself on IFBench. Yet deploying those exact prompts onto the student drops student accuracy below optimizing directly on the student. The teacher evolves concise instructions suited to its own reasoning depth, omitting the explicit requirements of the smaller model.

  • Static teacher rules or raw teacher traces/exemplars: Distilling teacher feedback offline into static rules (CD [1]), or supplying raw teacher reference solutions sis_i directly into reflection (+ Raw Reference Solutions), overwhelms the student or the optimizer with conflicting information. Similarly, attaching retrieved Φinsight\Phi_{\text{insight}} exemplars post-hoc to prompts (+ Post-Hoc Φinsight\Phi_{\text{insight}}) often degrades the performance as students copy surface patterns.

Our novelty therefore lies in identifying why naive teacher-guided prompting fails under teacher-student capability gaps and introducing a closed-loop differential audit and co-evolution mechanism that transfers gains reliably from the teacher to the student.


Comparison with more baselines (extending Table 1 in the paper)

We thank the reviewer for sharing the additional baselines. We have implemented [2, 4, 5, 6] under identical models (gemini-3.1-flash-lite student, gemini-3-flash optimizer/teacher); the results are shown below:

MethodMATH-HardCodeIO-PuzzleIFBench
EvoPrompt-GA [5]84.0858.8950.79
LEAP [2]83.3358.6158.62
PromptWizard [6]83.8359.3972.68
CD[1]86.3258.3955.70
Base Prompt + Φinsight\Phi_{\text{insight}}83.5859.5059.18
PLD84.5757.8960.89
EvoPrompt-DE [5]82.8458.2862.59
PromptAgent [4]84.8358.1770.86
Vanilla GEPA85.0858.1661.56
Vanilla APEX85.7758.1167.92
Diff-GEPA w/o Exemplars (Ours)85.8259.1172.22
Diff-APEX w/o Exemplars (Ours)84.0859.2873.81
DIFF-GEPA (Ours)86.0760.9473.92
DIFF-APEX (Ours)86.5760.1174.94

We will add the missing citations to Related Work, including a discussion positioning the very recent GRACE [7] (NeurIPS 2025) and PIAST [8] relative to our teacher-to-student differential auditing framework.


Require stronger teacher

We agree that leveraging a stronger teacher is part of our primary problem formulation; distilling reasoning guidance from a capable teacher into a compact student. However, in terms of utilizing the teacher for the external knowledge base, as shown in the table above, DiffPO w/o Exemplars retains strong gains with zero Phase-1 knowledge base.


Compute budget

We set the optimization budget up to 20 iterations for the GEPA/APEX baselines and 10 for DiffPO to match online query cost. However, most 20-iteration baseline runs plateaued without finding a better prompt in later rounds, which means increasing the cost does not usually translate to a better performance.

We agree that our paper requires the extra cost of constructing the knowledge base, however, we want to reemphasize that our main goal is not to find a most training efficient method; we look into ways to improve the performance of the student during the test time. The knowledge base construction is a one-time offline cost that requires zero teacher calls during the inference helping us to maximize test-time student accuracy.

We will add this discussion to the paper.


RL methods

As briefly discussed in the paper, prompt optimization spans distinct paradigms: RL-based methods such as RLPrompt [9] and StablePrompt [10] train parametric policy models to guide the optimization, whereas feedback-based methods edit prompts in natural language using a LLM as an optimizer. Since our method is does not train any extra model, our empirical comparison focuses on feedback-based optimizers. We will add an explicit discussion of RLPrompt [9] and StablePrompt [10] to Related Work.


Table Caption

We thank the reviewer for catching this and will move all table captions above their respective tables in the revised manuscript.

Reviewer_Wx95OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework to improve prompt optimization through teacher-guided error diagnosis. For each failure student traj, a strong teacher identifies the earliest reasoning error and extracts a generalizable rule. These rules are aggregated and used by a prompt optimizer to refine the global instruction. A retrieval strategy is also applied by the method, which provides relevant problem-solving examples at inference time. Experiments on three benchmarks and two model families demonstrate improvements over existing prompt optimization baselines.

Critical strengths
  • The framework is intuitive and can be integrated into different prompt optimization framework.
  • The author provide ablation experiments for its main components.
Critical weaknesses
  • The novelty and effectiveness of the proposed Differential Audit mechanism are insufficiently established. Previous works like Concept Distillation have already explored extracting generalizable principles from model failures.
  • While DiffPO additionally emphasizes identifying the earliest reasoning error, the experiments do not establish whether this step-level localization provides any benefit over conventional teacher-generated error critiques.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Wx95

We sincerely thank Reviewer Wx95 for their constructive feedback and questions; please find our response below.

1. Comparison to Concept Distillation (Boateng et al., 2025)

Our paper already includes Concept Distillation (CD) as a primary baseline across all three benchmarks and both model families. For further investigation, we also include the results for our algorithm (DIFFPO) without exemplars. Under identical student-teacher pairs, our method outperforms CD across the majority of benchmarks even with exemplar retrieval completely disabled (DIFFPO w/o exemplars, matching CD's zero-shot inference setup): CD leads only on MATH-Hard against the zero-exemplar ablation (86.32 vs. 85.82 / 84.08), whereas DIFF-APEX Full (86.57) outperforms CD on all three benchmarks, and even without exemplars DiffPO beats CD on IFBench and CodeIO-Puzzle:

Table 1:

MethodMATH-HardCodeIO-PuzzleIFBench
Concept Distillation (Boateng et al., 2025)86.3258.3955.70
Vanilla GEPA85.0858.1661.56
Vanilla APEX85.7758.1167.92
GEPA + Ref Solutions (sis_i)83.0859.0660.09
APEX + Ref Solutions (sis_i)83.5858.7259.07
DIFF-GEPA w/o Exemplars (0 KB)85.8259.1172.22
DIFF-APEX w/o Exemplars (0 KB)84.0859.2873.81
DIFF-GEPA Full86.0760.9473.92
DIFF-APEX Full86.5760.1174.94

Why CD may not always work:
We identify two limitations of CD from our experiments. First, while failed cases come from the student, the teacher cannot diagnose the main failure reasons, leaving extracted rules poorly calibrated to the student's capacity. Second, CD collects mistakes only once on the initial prompt p0p_0 and greedily appends rules without re-auditing how rules interact as the student's behavior shifts.


2. Benefit of Earliest-Step Localization over Conventional Teacher Critiques

In our Vanilla GEPA, Vanilla APEX, and Concept Distillation (CD) baselines, the same stronger model (teacher) already acts as the optimizer or critic, inspecting failed student rollouts and outcomes to propose prompt edits or rules. Even more directly, simply supplying the teacher's reference solutions sis_i to the optimizer without DiffPO's step-localized differential audit (GEPA / APEX + Ref Solutions in the above table) fails to improve over Vanilla in Table 1, confirming that structured step-level failure diagnosis comparing student and reference reasoning (not just using a stronger model or exposing reference answers) is what drives the gains. We will make this comparison and mechanism clearer in the revised manuscript.

Where is the revised manuscript?

AuthorsOpenReview ↗

Respond to reviewer's question.

We thank the reviewers for checking our rebuttal, and apologize for any confusion from our phrasing! When we referred to "the revised manuscript" in our earlier comments, we meant our planned revision incorporating the discussion-phase updates rather than a newly uploaded PDF build.

To make every update concrete alongside the experimental tables posted in our responses above, here is the exact list of changes we will incorporate:

  1. Formatting (Tables 1–5): Move all table captions above their respective tables.
  2. Related Work (Section 2): Add citations and discussion for additional baselines (EvoPrompt, LEAP, PromptAgent, PromptWizard, GRACE, PIAST, and RL-based prompt optimizers).
  3. Main Results (Section 4.2, Table 1): Include the new Gemini baselines (EvoPrompt, LEAP, PromptAgent, PromptWizard) and our failure analysis of Concept Distillation (CD).
  4. Expanded Ablations (Section 4.3.2, Table 4): Extend Table 4 across MATH-Hard and CodeIO-Puzzle, and add the controlled ablations for DIFFPO w/o Exemplars, Retrieval-Only, and GEPA/APEX + Reference Solutions in reflection to isolate step-localized auditing.
  5. Setup & Appendix (Section 4.1 & Appendices C–D):
    • Explicitly state the exact feedback signals provided to Vanilla GEPA and APEX in the main paper.
    • Report KB pool sizes, test-time token overhead, and offline Phase-1 vs. online Phase-3 optimization budgets in Appendix C.
    • Add annotated step-level teacher audit examples across all three benchmarks (extending Figure 1) in Appendix D.

Thank you again for your time and feedback; please let us know if anything remains unclear or if there are any other questions!

I mean, why don't you upload it directly, instead of simply provide a plan? I think ICLR provides entrance for direct uploading.

Reviewer_ssWaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiffPO, a prompt optimization method for small student models. When the student fails a training problem, a stronger teacher compares the student's reasoning with its own reference solution, finds where the reasoning breaks, and turns the error into a general rule for the prompt optimizer. At test time, the student also retrieves summarized solutions of similar problems from a teacher-built index. The method is added on top of GEPA and APEX and evaluated on math, logic puzzles, and instruction following with two model families.

Critical strengths
  1. The idea of locating where a student's reasoning breaks and turning it into a rule is intuitive, and it can be added on top of existing reflective optimizers without changing their search.

  2. The experiments cover two model families and five random seeds, and DiffPO improves both GEPA and APEX on every task when added on top of them.

Critical weaknesses
  1. I am not fully convinced that the audit is very different from what GEPA already does. GEPA does not rely on a pass or fail signal alone. It reflects on the model's trace together with text feedback from the evaluator, and the GEPA paper also lets the user attach a short explanation to each training example. The differential audit looks close to having the teacher write those explanations automatically. On the other hand, this paper describes GEPA and APEX as seeing the problem, the model's output, and a correctness label, without the reference answer or any text feedback from the evaluator. I could not find how the feedback of the baselines was actually set up in the experiments. If the baselines were run this way, they were weaker than usual, while DiffPO also gets the teacher's reference solution on top of the audit. In that case it is hard to tell whether the gain comes from the audit or simply from more information. It would help to state exactly what feedback the baselines received, and to add a run where GEPA gets the teacher's reference solution as feedback but no step-level audit. The gap between that run and Diff-GEPA would show what the audit itself adds.

  2. The main comparison is not on equal terms. At test time, DiffPO adds retrieved solutions of similar problems from an extra labeled pool, while the baselines only use their optimized prompts. Building this index costs teacher calls that are not counted in the budget, and the paper does not say how large the pool is. The only experiment that gives the baselines the same retrieved examples is on IFBench, so for the other two tasks it is unclear how much of the gain comes from retrieval rather than from the audit. Retrieval also brings several extra design choices on top of prompt optimization. I would like to see the baselines with the same retrieval on every task, and a simple baseline that only retrieves similar solved problems without any prompt optimization.

Questions for authors
  1. How accurate are the audits? The paper checks that teacher solutions reach the correct answer, but not whether the teacher finds the right step or writes a correct rule. A small human check would help.

  2. The ablations are run only on IFBench, where the gains are largest. On MATH, the differences are within the reported variance. Do the same trends hold on the other tasks?

  3. How much longer is the student's input at test time with retrieved exemplars, and how does this affect cost?

1 comment
AuthorsOpenReview ↗

Response to reviewer ssWa

We sincerely thank Reviewer ssWa for their constructive feedback and questions; please find our response below.


Baseline Feedback Setup and Reference-Solution Control

Feedback signal construction: The feedback for GEPA and APEX includes the problem, the student's full output, and the correctness score alongside text feedback from the evaluator: on IFBench, optimizers receive per-constraint diagnostics; on MATH-Hard and CodeIO-Puzzle, the optimizers receive correctness status with the ground-truth answer. Neither baseline receives full reference solutions sis_i. We will state this exact feedback setup in the main paper.

Adding reference solutions sis_i to the feedback. Following the reviewer’s suggestion, we ran GEPA and APEX with the full reference solution sis_i directly in reflection (+ Ref Solutions sis_i), compared against DIFFPO w/o Exemplars (which sees the exact same sis_i only via the teacher's step-localized audit and zero exemplars). As the table suggests providing raw sis_i without step localization degrades the performance on MATH-Hard and IFBench as batch reflection copies problem-specific phrasing from sis_i:

MethodMATH-HardCodeIO-PuzzleIFBench
GEPA85.0858.1661.56
GEPA + Ref Solutions (sis_i)83.0859.0660.09
Diff-GEPA w/o Exemplars85.8259.1172.22
APEX85.7758.1167.92
APEX + Ref Solutions (sis_i)83.5858.7259.07
Diff-APEX w/o Exemplars84.0859.2873.81

Retrieval only and Extended Ablations

To isolate the effect of retrieval from the differential audit, we first evaluate unoptimized Retrieval-Only baselines over raw similar solved problems (p0+Φfullp_0 + \Phi_{\text{full}}) and abstracted blueprints (p0+Φinsightp_0 + \Phi_{\text{insight}}):

Unoptimized BaselineMATH-HardCodeIO-PuzzleIFBench
Initial Prompt (p0p_0)83.5857.3344.00
Retrieval-Only (p0+Φfullp_0 + \Phi_{\text{full}})82.0957.8358.50
Retrieval-Only (p0+Φinsightp_0 + \Phi_{\text{insight}})83.6159.5059.18

We also extend Table 4 (of the paper) to MATH-Hard and CodeIO-Puzzle:

DatasetBackboneFullw/o Exemplarsw/o Teacher+ Φinsight\Phi_{\text{insight}} (Post-Hoc)
MATH-HardGEPA86.0785.8284.0884.58
MATH-HardAPEX86.5784.0885.0784.33
CodeIO-PuzzleGEPA60.9459.1160.0059.67
CodeIO-PuzzleAPEX60.1159.2859.6157.22
IFBench (Table 4)GEPA73.9272.2262.3660.43
IFBench (Table 4)APEX74.9473.8169.8470.07
  • Retrieval vs. audit: The new results confirm that prompts must be co-evolved with differential auditing. However, contributions vary by domain; the step audit drives most gains on IFBench, whereas retrieval helps more on CodeIO-Puzzle.

  • Design choices: We agree with the reviewer that retrieval introduces extra hyperparameters. We did not perform extensive hyperparameter search and kept identical defaults across benchmarks (γ=0.5\gamma=0.5, K=3K=3), changing only K=1K=1 on CodeIO-Puzzle because its inputs are longer.


Knowledge pool sizes

Exact pool sizes (to be added to the Appendix) are summarized below:

Benchmark∥KB∥\|\mathcal{KB}\|
MATH-Hard12K
CodeIO-Puzzle9K
IFBench8K

We want to note that Phase 1 runs once offline per benchmark (0 teacher calls at inference). Online, DiffPO runs fewer iterations (10 vs. 20 for baselines). Furthermore, DiffPO w/o Exemplars uses 0 external pool problems and still outperforms the majority of the baselines.


Questions:

Audit accuracy: We inspected audit outputs across multiple optimization rounds to confirm the system isolates real step errors, though we did not run a formal human study. More directly, our controlled ablations show that the differential audits drive the gains: replacing them with raw reference solutions or removing them degrades accuracy across tasks. We will add more annotated audit examples across all three benchmarks (following Figure 1) to the Appendix.

MATH-Hard variance: On MATH-Hard, gemini-3.1-flash-lite starts near ceiling so margins are tighter (while DIFF-GEPA still cuts seed SD 3x), whereas Gemma-12B (Table 2) has larger headroom and shows clear gains.

Test-time input length and cost: Retrieving exemplars adds ~842–1,173 input tokens per query (with 0 test-time teacher calls); however, these are prefill input tokens, which are substantially cheaper than generated output tokens, and DiffPO w/o Exemplars avoids this overhead completely.

Comments on the paper

1 comment

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.