acceptodds
Under review as a conference paper at ICLR 2027

Learning from Evolving Prompts: Adaptive Prompt Teaching for Reinforcement Learning

Abstract

Prompt optimization and reinforcement learning (RL) improve language models through different channels: one edits the text in the model's context, while the other updates its weights. Recent methods combine the two by alternately updating the prompt and model weights to improve performance. However, the benefit of an optimized prompt may disappear as RL updates the weights, and the prompt may eventually become redundant or even harmful. We propose Adaptive Prompt Teaching (APT), an RL framework that uses evolving prompts as temporary teachers during training and tests whether any prompt is worth keeping at deployment. APT first samples each training problem from its original, non-optimized input and resamples it under teacher prompts only when every sample fails and thus provides no learning signal. It then trains the model on the guided responses with the teacher part removed from the input, allowing the model without the optimized prompt to learn the same behavior. Teacher prompts are periodically rewritten based on the model's recent failures and retired once the model performs as well without them. Since some guidance may remain unabsorbed in the weights, a separate step searches for a prompt that still improves the model with its updated weights and uses it as the final deployment prompt for evaluation. On HoVer, WebShop, and SATBench with Qwen3-8B and Gemma3-12B, APT achieves the highest mean accuracy in all six settings, 1.3 to 4.1 points above the strongest baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt Optimization, Reinforcement Learning, Reasoning, Context Distillation, Deployment Prompt

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_L5p1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new method called Adaptive Prompt Teaching (or APT) for training large language models which is designed to address the lack of a gradient signal in GRPO in special case when all responses in a group are incorrect. The proposed method proposes a 2-step procedure: (1) add teacher prompts for tasks that the base model cannot handle, (2) distil the resulted correct solutions into the model under the bare context. They also propose a mechanism to update the teacher prompts ensuring they remain useful throughout training. Finally, an evaluation method for this prompts is provided.

Critical strengths

The paper is well motivated and provides a well grouded and well designed novel method. The experiments provide a robust analysis of the methods components proving its efficiency.

Critical weaknesses

The main concern is about the validity risks: their is no separate val sprit, the prompts are selected on the training tasks which are further used in model training. Deployment-prompt is not revalidated after the last 20% and its evaluation could be noisy.

There is a risk of catastrophic forgetting (at this point each run is trained only on one benchmark), various prompt formulation experiment shows only the prompt robustness within the domain.

Questions for authors

Is APT applicable for the cross-domain scenarios? Have You evaluated it in this scenarios?

5 comments
AuthorsOpenReview ↗

Response to Reviewer L5p1 (1/2)

Thank you for the thoughtful review and for recognizing that APT is well motivated and well designed. We have simplified your comments for easier reference and address your concerns one by one. All new tables are also in Appendix G of the revised paper.

W1: On Validation of the Deployment Prompt

We want to clarify that the deployment prompt is selected on training tasks but does not overfit to them. We address your concern in three parts.

1. A prompt that overfits to a few training tasks is unlikely to pass the test. The reflection model writes new candidates from training tasks, but each one must then beat the bare model on 256 tasks at the frozen 0.8N0.8N weights, significantly after Bonferroni correction. Breaking a task that the bare model solves counts against it. The new texts are at most 128 tokens and may not contain task answers or training identifiers (Appendix F.4), and they also compete with the latest teacher texts used as is. When no candidate passes, APT deploys no prompt, as on every HoVer run. The prompt kept for the last 20% of updates therefore has significantly more rescues than harms on 256 tasks. As for a retest after these updates, Table 11 of the original paper already shows that the deployed prompt still beats the bare model on the test set at the final weights in every cell that uses one (right column of Table R1 below).

2. The choice is stable across task sets. To address your concern on noise, we additionally repeated the selection at the frozen 0.8N0.8N weights on a second set of 256 tasks. For HoVer and WebShop this is a new draw of training tasks. For SATBench, whose 302 training puzzles are too few for a second draw, it is held-out puzzles from the original pool (Table R1 below):

Table R1{\color{blue}\text{Table R1}}: Repeated selection on a second task set, and test-set gain at the final weights (6 runs per benchmark).

BenchmarkSame z^\hat z on the second setDeployed - bare on test at final weights (Qwen3-8B / Gemma3-12B)
HoVer6/6 (no prompt)0.0 / 0.0
WebShop6/6+0.7 / +0.8
SATBench (held-out)4/6+2.9 / +2.3

As shown in Table R1, the second set picks the same prompt in 16 of 18 runs. In the two SATBench runs where it differs, another short candidate written for deployment ranks first. The deployed prompt still passes the test on the held-out puzzles, so all six deployed SATBench prompts pass on unseen puzzles. The decision is also consistent across seeds, which differ in data order: no prompt in all six HoVer runs, and a prompt in all six SATBench runs and five of six WebShop runs (Table R3 in our response to Reviewer mWLe).

3. There is no contamination. The test split is never used to propose, compare or select any prompt (Appendix C.1). The revised §3.4 reports both checks in the main text.

AuthorsOpenReview ↗

Response to Reviewer L5p1 (2/2)

W2: On Forgetting

We want to clarify that APT, like every method we compare, is a task-adaptation method, and forgetting on other domains comes with this setting rather than with APT. Every method in Table 1 trains one policy per benchmark, and RL on one task can lower accuracy on unrelated tasks for any of them, especially when starting from post-trained models. APT does not add a source of forgetting. Every teacher-stage gradient is taken in the bare context, and a teacher only decides which responses the model learns from. Only 3% to 28% of each cycle's responses are guided in the run of Figure 8c, and the last 20% of updates are GRPO under B0+z^B_0 + \hat z.

Our robustness test stays within the domain because it targets the risk that APT's design addresses. A method that trains under a prompt can tie its gains to that prompt (P2 in §2). Figure 2 tests this directly, and APT's accuracy barely moves across five contexts (spread 1.2 vs. 30.1 for Seq, Table R2 in our response to Reviewer mWLe). When several domains matter, the direct remedy for forgetting is to train on them together, and APT still beats GRPO in that setting (Table R11 below).

Q1: On Cross-Domain Use

Yes. APT works when one policy is trained on several domains, and it also improves on new benchmarks. To address your question, we trained one policy on a HoVer + SATBench mixture, with one teacher set and one deployment test per domain (Table R11 below):

Table R11{\color{blue}\text{Table R11}}: One policy trained on HoVer + SATBench (Qwen3-8B, accuracy %, 300 updates, 3 seeds).

MethodHoVerSATBenchAvg.
GRPO, mixed47.969.058.5
APT, mixed52.170.961.5

As shown in Table R11, APT improves over GRPO by 3.0 points on average (+4.2 on HoVer and +1.9 on SATBench). As noted above, zero-shot transfer of a checkpoint to an unrelated domain is outside the scope of task adaptation. Joint training is how APT covers several domains.

To further show that APT is not tied to our three benchmarks, we also ran it on MiniWoB++ and ScienceWorld. It reuses the WebShop settings of Table 4 with 100 updates and needs only a new task contract for the reflection model (Table R8 below):

Table R8{\color{blue}\text{Table R8}}: New environments (Qwen3-8B, success rate, 100 updates, 3 seeds).

MethodMiniWoB++ScienceWorld
GRPO68.450.0
FST65.352.3
APT, bare context73.161.4
APT75.263.3

As shown in Table R8, the bare policy of APT already beats GRPO and FST on both environments, and the deployment prompt adds 2.1 and 1.9 points on top.

We hope these responses address your concerns. If anything remains unclear, we are happy to engage in further discussion.

AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer L5p1,

Thank you again for taking the time to review our work and for recognizing APT as well motivated and well designed. Since the discussion period is ending soon, we briefly summarize the key points of our rebuttal:

  • Validation of the deployment prompt (W1): Repeating the selection on a second task set picks the same prompt in 16 of 18 runs. At the final weights, the deployed prompt still beats the bare model on the test set in every cell that uses one (Table R1{\color{blue}\text{Table R1}}). The test split is never used to select any prompt.
  • Forgetting and cross-domain use (W2, Q1): One policy trained on HoVer + SATBench improves over GRPO by 3.0 points (Table R11{\color{blue}\text{Table R11}}). APT also improves over GRPO and FST on MiniWoB++ and ScienceWorld (Table R8{\color{blue}\text{Table R8}}).

Full details are in our responses above and in Appendix G{\color{blue}\text{Appendix G}} of the revised paper. We hope these results resolve your concerns. If anything remains unclear, we would be glad to follow up before the discussion closes.

Best regards,
The Authors

Reviewer_EUheOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Prompt Teaching (APT), a reinforcement learning framework that separates a prompt's role as a teacher during training from its role at deployment. During training, problems on which four bare samples all fail are routed to three teacher prompts. The guided responses are then trained under the bare context, with a pooled advantage and capped per-token transfer weights. The teachers are rewritten by a reflection model and retired when they no longer beat a fresh bare sample. After 80% of the updates, a paired bootstrap test at frozen weights decides whether a deployment prompt is kept. On HoVer, WebShop, and SATBench with Qwen3-8B and Gemma3-12B, APT reports the best mean accuracy in all six settings, 1.3 to 4.1 points above the strongest baseline (Table 1).

Critical strengths
  1. APT uses teacher guidance where standard Group Relative Policy Optimization (GRPO) is likely to lack a learning signal. Teachers are used only when all four bare probes fail, targeting problems where ordinary sampling often produces zero advantages (Lemma 3). Teacher responses are generated within the same eight-response budget, so the teacher stage adds no extra rollouts. Teacher performance is also measured against a fresh bare control rather than the failed probes, avoiding an unfair comparison (Lemma 2). The results support this design. APT is less sensitive to changes across five system contexts than methods trained under optimized prompts (Figures 2 and 6). Its bare accuracy also exceeds the strongest baseline's deployed average on both models (70.7 versus 68.3 and 64.3 versus 62.0, Table 11).

  2. The pooled advantage centers bare responses on their own mean, so a successful teacher response does not penalize bare failures. The capped per-token transfer weights reduce the influence of tokens that depend heavily on teacher instructions. Lemma 1 explains this choice through the overlap between the guided and bare distributions. Figure 7 provides further support: removing the cap reduces accuracy by 4.2, 1.3, and 3.5 points across the three benchmarks. The deployment stage is also carefully designed. A prompt is kept only if it passes a paired bootstrap test at frozen weights. APT selects no prompt on HoVer but keeps one on WebShop and SATBench, where Table 11 shows that it remains useful at the final weights.

Critical weaknesses
  1. The experiments do not fully explain where the gains come from, and the smaller improvements lack strong statistical support. First, removing routing causes the largest drop in Figure 5a (71.9 to 64.6). The variant without teacher evolution (68.7) also performs slightly better than FST and Seq (both 68.3) on Qwen3-8B. This suggests that much of the gain may come from guided sampling on all-fail problems. However, closely related methods are missing from the comparison, including GUIDE (Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models, arXiv:2506.13923), HiLL (Learning to Hint for Reinforcement Learning, arXiv:2604.00698), and No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping (arXiv:2509.21880). P2O (P²O: Joint Policy and Prompt Optimization, arXiv:2603.21877) is the only guidance-based baseline. Second, all prompt-optimization baselines replace the system prompt, while APT keeps B0 and appends instructions (Appendix C.3). It is unclear which method benefits from this difference. A comparison where the baselines also keep B0 would help control for it. Third, SATBench has only 70 test tasks, with each task accounting for about 1.4 percentage points of total accuracy if its correctness changes fully. The 1.3-point margin over Seq on Qwen3-8B (73.0 ± 1.6 versus 71.7 ± 1.5), based on three training seeds, needs a paired significance test or a confidence interval on the difference.

  2. The novelty mainly comes from combining and adapting existing techniques, while the added value of individual components remains unclear. Learning from guided responses under the bare context (GUIDE, HiLL, P2O), reflective prompt rewriting as in GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (arXiv:2507.19457), and truncated importance weights similar to V-trace from IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (arXiv:1802.01561) are already known. Figure 5a shows that removing teacher evolution or transfer weights reduces accuracy by 3.2 and 2.4 points, while removing the pooled advantage, retirement, or deployment selection costs 0.9–1.1 points. However, these ablations use only Qwen3-8B, report no standard deviations, and do not explain exactly how each variant is implemented. It is therefore difficult to judge which components are needed compared with a simpler guided RL method. The theoretical results are also limited. As acknowledged in Appendix A, Proposition 1 applies to an idealized estimator with exact sequence-level weights and raw rewards, not the practical update. Proposition 2 assumes identical teacher and bare policies, leaving partly absorbed teachers unaddressed. Finally, several choices lack sensitivity studies, including four probes, the promotion threshold of 0.5, and length limits of 1.5 and 2 times. No separate validation split is used (Section 4.1), and Figure 7 reports the cap sweep using test accuracy. The paper should clarify which data were used to choose these settings.

Questions for authors
  1. Table 11 shows that the selected deployment prompt remains helpful at the final weights. How sensitive are the results to the 80/20 split between the teacher and deployment stages? For example, what happens with a 70/30 or 90/10 split?

  2. Appendix E.4 acknowledges that reduced routing does not necessarily mean the model has absorbed teacher guidance. Could it divide the test set by initial difficulty (problems where the untrained model fails all four samples versus the rest) and report APT's improvement over GRPO in each group? Larger gains on the harder problems would be consistent with the claimed transfer of teacher guidance to the bare model.

7 comments
AuthorsOpenReview ↗

Response to Reviewer EUhe (1/4)

Thank you for the thorough review and detailed suggestions, and for recognizing that APT targets guidance where GRPO lacks a learning signal at no extra rollout cost. We have simplified your comments for easier reference and address your concerns one by one. All new tables are also in Appendix G of the revised paper.

W1 (first part) & W2 (first part): On the Source of the Gains and Novelty

We want to clarify that APT's margin does not come from guided sampling on all-fail problems, which it shares with prior work, but from how APT teaches, transfers and deploys its prompts. Routing matters most in Figure 5a because guiding every problem drops accuracy below GRPO (64.6 vs. 66.8), so guidance must be targeted. Targeted guidance alone, however, adds little and thus is not the primary source of gains. The w/o-evolution variant (68.7) still keeps the transfer weights and deployment selection, while fixed guidance on routed problems reaches only 67.5 (Table R4).

To address your concern, we additionally ran NPLB and GUIDE in our setup, used per-problem answer-free hints in place of HiLL, and built a cumulative path from GRPO to APT (Table R4 below). The P2OP^{2}O row is from Table 1, and each row of the path keeps the row above and adds the change it names:

Table R4{\color{blue}\text{Table R4}}: Closest methods and the cumulative path from GRPO to APT (Qwen3-8B, accuracy %, 3 seeds). The per-problem hint row is a variant of the row above it, not a step of the path.

ConfigurationHoVerWebShopSATBenchAvg.
GRPO48.981.370.266.8
NPLB (advantage shaping on zero-variance groups, no guidance)49.483.068.667.0
P2OP^{2}O (per-problem templates)49.483.369.067.2
GUIDE (per-problem hints from reference solutions, uncapped ratio)49.885.268.167.7
Path: guided responses on all-fail problems, fixed teachers49.385.667.667.5
Same with per-problem answer-free hints (in place of HiLL)49.085.467.667.3
Transfer: pooled advantage, capped weights49.487.068.168.2
Teacher rewriting, kept if better with the teacher in context (GEPA-style)51.387.068.669.0
Teaching: rewrites kept only if better than a fresh bare sample52.687.669.569.9
Teaching: retirement once a teacher no longer beats it53.688.070.770.8
Deployment selection (= APT)53.689.173.071.9

As shown in Table R4, the closest methods and the shared core all perform near 67.5. NPLB, P2OP^{2}O and GUIDE reach 67.0, 67.2 and 67.7, even though GUIDE's hints use reference solutions, and per-problem hints written by our editor are not better than shared teachers. Adding the transfer weights increases the score of fixed teachers to 68.2, and they matter more once teachers evolve, since removing them from full APT costs 2.4 points (Figure 5a). GEPA-style rewriting adds only 0.8, indicating that comparing a rewrite with the previous teacher is not enough. The largest gain comes from judging teachers against the bare model. Accepting rewrites only if they beat a fresh bare sample adds 0.9, and retiring teachers that no longer beat it adds another 0.9 (1.8 together, 95% paired bootstrap CI [0.5, 2.9], same protocol as Table R5 in part 2/4). Deployment selection adds the last 1.1, all of it on WebShop and SATBench where the test keeps a prompt, and most of it on SATBench.

These results reflect APT's novelty, which lies in how it uses prompts during RL training rather than in any single component. Prior methods give a prompt one fixed role. GUIDE, HiLL and P2OP^{2}O use guidance only in training and always drop it at deployment, while GEPA and FST keep the prompt at both stages and optimize it as part of the deployed system. In APT a prompt first acts as a teacher that evolves with the policy while every teacher-stage gradient is taken in the bare context, and it becomes a deployment prompt only for what the weights have not absorbed. The known techniques you list are adapted to this role. A GEPA-style rewrite is kept only if its responses beat a fresh bare sample, not merely because it beats the previous teacher, and the V-trace-style cap (Appendix D) is applied to a different ratio, bare-to-guided, where a cap of 1 keeps any guided token from outweighing a bare sample (Lemma 1). In Table R4 the shared core alone (67.5) stays below the strongest baseline (68.3), so the margin over the baselines comes from the steps APT adds to it. Table R13 in the revised §5 compares these methods, and Appendix C.9 defines each configuration.

AuthorsOpenReview ↗

Response to Reviewer EUhe (2/4)

W1 (second part): On the System Prompt B0B_0

APT's robustness advantage holds when the baselines also keep B0B_0. To address your concern, we ran Seq with B0B_0 kept and its prompt appended in both search and training (Seq-append, Table R2 below). Seq matches FST on average (65.1 vs. 65.2 over six cells), so we use it as the representative joint method.

Table R2{\color{blue}\text{Table R2}}: System-prompt control (Qwen3-8B, 3 seeds). Robustness uses the five contexts of Figure 2: the mean, and the max - min across contexts of the benchmark average.

MethodPromptDeployed avg.Robust. meanRobust. spread
Seqreplaces B0B_068.349.530.1
Seq-appendappends to B0B_067.957.918.6
APTappends to B0B_071.971.41.2

As shown in Table R2, keeping B0B_0 raises Seq's robustness mean from 49.5 to 57.9 and lowers its spread from 30.1 to 18.6, but both stay far from APT (71.4 and 1.2). Its deployed accuracy does not rise (67.9 vs. 68.3). The remaining gap reflects (P2) in §2: Seq-append still trains under its prompt, so its gains stay tied to that prompt. Only Seq-append is a new run, and the Seq and APT rows summarize Figure 2. The revised §4.1 points to this control.

W1 (third part): On Significance

We want to clarify that APT's margin over the strongest baseline is significant on the macro average and in four of six cells. On SATBench the 70-task test set is too small for a margin of about one task to be significant, and both margins are significant on 582 unseen puzzles. To address your concern, we computed paired bootstrap 95% CIs on each margin (Table R5 below). Seeds are paired by index (same data order), test tasks are resampled within each seed, and margins are computed before rounding:

Table R5{\color{blue}\text{Table R5}}: Margin of APT over the strongest baseline in each cell (accuracy points, 95% CI).

CellStrongest baselineMargin95% CI
Qwen3-8B, HoVerAlternate4.1[3.2, 5.1]
Qwen3-8B, WebShopFST3.9[2.4, 5.4]
Qwen3-8B, SATBenchSeq1.3[-1.1, 3.7]
Gemma3-12B, HoVerSeq1.5[1.0, 2.1]
Gemma3-12B, WebShopSeq3.5[1.6, 5.5]
Gemma3-12B, SATBenchP2OP^{2}O1.7[-0.2, 3.5]
Macro averageFST3.4[2.9, 4.1]
Qwen3-8B, SATBench (582 tasks)Seq1.6[0.1, 3.1]
Gemma3-12B, SATBench (582 tasks)Seq1.8[1.0, 2.5]

As shown in Table R5, the intervals exclude zero for the macro average and for every HoVer and WebShop cell. On SATBench each task is worth 1.4 points, so both margins (1.3 and 1.7) amount to about one task. However, this is a limit of the small test set rather than of APT's margin. When we evaluate the same checkpoints on 582 unseen puzzles (the 70 test tasks plus 512 unused puzzles that played no part in training or selection), both margins become significant. On these puzzles Seq is the strongest baseline for both models. The last two rows of Table R5 therefore compare against Seq, even for Gemma3-12B, whose strongest baseline on the 70 test tasks is P2OP^{2}O. With twice the training, the margin over Seq on Qwen3-8B also grows to 2.3 points (Table R12 in our response to Reviewer mWLe). Overall, APT's margin over the strongest baseline is significant on average and in all six settings when SATBench is evaluated on 582 puzzles. The revised §4.2 and §6 now state which margins are significant.

AuthorsOpenReview ↗

Response to Reviewer EUhe (3/4)

W2 (second part): On the Ablations

We want to clarify that every component ablated in Figure 5a lowers APT's average by more than the seed noise, and the largest effects repeat on Gemma3-12B. Table R4 in part 1/4 already compares APT with simpler guided RL methods. To address your concern, we also report the seed standard deviations of Figure 5a, state what each variant changes and repeat the ablations on Gemma3-12B (Table R9 below):

Table R9{\color{blue}\text{Table R9}}: Component ablations (average accuracy over the three benchmarks, %, mean ± std over 3 seeds). Qwen3-8B means are those of Figure 5a.

Variant (what changes)Qwen3-8BGemma3-12B
APT71.9 ± 0.365.3 ± 0.6
w/o routing (guidance on every problem, not only all-fail ones)64.6 ± 0.861.9 ± 0.4
w/o evolution (initial teachers kept, never rewritten or retired)68.7 ± 0.662.6 ± 0.6
w/o transfer weights (no per-token transfer weights)69.5 ± 0.763.4 ± 0.1
w/o deployment selection (last 20% and test under the bare prompt)70.8 ± 0.564.6 ± 0.1
w/o pooled advantage (one group mean over bare and guided responses)70.9 ± 0.364.0 ± 0.1
w/o retirement (teachers rewritten but never retired)71.0 ± 0.463.9 ± 0.5

As shown in Table R9, on Qwen3-8B the three smallest drops, from deployment selection (1.1), the pooled advantage (1.0) and retirement (0.9), are each at least twice the standard deviation of APT (0.3) and of the variant (at most 0.5). Retirement barely changes SATBench (72.7 vs. 73.0) because its teachers are rarely retired (Figure 8b), so its drop comes from HoVer and WebShop. On Gemma3-12B every variant also falls below APT, and routing, evolution and the transfer weights again cause the three largest drops (3.4, 2.7 and 1.9). Deployment selection costs least there (0.7) because it only changes WebShop and SATBench, where APT deploys a prompt. Appendix C.9 gives the full definition of each variant, and Figure 7 and Appendix E.3 now give the corrected no-cap values.

W2 (third part): On the Role of the Theory

We want to clarify that our theory motivates APT's design and is not a contribution of its own: none of the three contributions in the Introduction is theoretical. Proposition 1 isolates the idealized contrast with joint methods under stated assumptions (exact sequence-level weights, raw rewards and support coverage), and Figure 2 tests this contrast on the practical update. Proposition 2 shows that a fully absorbed teacher becomes eligible for retirement. For partly absorbed teachers, APT relies on measurement instead. In every cycle it compares each teacher's responses with a fresh bare sample, keeps a rewrite only if it beats that sample, and retires a teacher after two cycles without lift (Figure 3a). Table 14 traces one role through this process. Even if this measurement is noisy, a single wrong keep-or-retire decision does little harm. A teacher only changes which training responses are sampled, the cap limits how much a guided token counts in the gradient (Lemma 1), and each decision is revisited in later cycles, since a retired role still receives new candidates.

Reviewer_mWLeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a prompt plays two roles in an RL pipeline, a teacher during training and an instruction at deployment, and that joint prompt-and-weight methods conflate them. The proposed method, APT, samples responses under teacher prompts only on problems where every bare sample fails, then takes the gradient under the bare context so the behavior lands in the weights rather than the prompt. Teachers are rewritten by a reflection model each cycle, scored against a fresh bare sample, and retired when they stop helping. After the teacher stage, a paired test at frozen weights decides whether any prompt is worth deploying. Experiments cover three benchmarks and two model families, and APT wins every cell against GRPO and four prompt-optimization baselines.

Critical strengths

The negative result on prompt search is useful. If prompts are treated as another component of the deployed system, extensive search adds almost nothing. This is the most interesting finding in the paper in my opinion.

Guidance is made budget-neutral. Teacher samples fill slots in groups that would have produced no gradient anyway, so the rollout count never changes, and reflection calls run alongside training. The compute and API comparison against the joint baseline is a large and clean gap.

The claims are well tested.

Critical weaknesses
  1. The components are all borrowed, and the closest prior method is not distinguished carefully enough. Context distillation, reflective prompt rewriting, routing guidance to groups with no learning signal, and capped importance weights are each established. The paper cites all of them accurately, which is to its credit, but the method is a straightforward integration of these components rather than a new mechanism.

  2. The theory is standard, does not apply to the implemented method, and is given more weight than it carries. The first proposition has two halves. One is textbook importance sampling. The other observes that two objectives can have gradients pointing in different directions, shown with a toy example where the prompt inverts the model’s preference. Neither needed a proposition.

  3. The teacher is a frontier model and the student is much smaller. The numbers come from a strong external model writing curricula for far smaller policies. That is close to distillation through an unusual channel. The self-teaching ablation is the right experiment, and it partially fails: the self-edited variant matches the frontier editor on one benchmark and loses on the other two, while still beating plain RL. The conclusion is that editor strength matters on most benchmarks. The paper’s framing leans harder toward self-improvement than the table supports.

  4. The baseline comparison has an unaddressed asymmetry. APT appends to the benchmark’s system prompt; every baseline replaces it. This is stated once in an appendix and never examined. Staying anchored to the original interface is itself a stabilizing choice, so it may account for part of the robustness gap, which is one of the paper’s main results. APT’s deployment candidates are also capped at a much shorter length than the baselines’ prompts, which is a different search space.

  5. Thin statistical support in places. Deployment selection uses a small set of tasks drawn from the training stream, which the authors acknowledge may be noisy, and the selected prompt is never retested at the final weights.

  6. One benchmark contradicts the main claim and is not examined. On the smallest benchmark, the bare score falls below the strongest baseline and the deployment prompt supplies the margin. That is a counterexample to the thesis that the gain lives in the weights. It is reported in an appendix without discussion.

Questions for authors
  1. Can you run an ablation path that turns APT into the closest prior method one change at a time, so the reported margin can be attributed to shared teachers, teacher evolution, or deployment selection?
  2. What happens to the robustness comparison if baselines also append to the benchmark system prompt instead of replacing it?
  3. Does the self-teaching variant close the gap with a larger policy, or with longer training?
  4. Why does the smallest benchmark depend on the deployment prompt when the others do not?
  5. Would deployment selection on a held-out set, rather than tasks from the training stream, change which prompt is chosen?
6 comments
AuthorsOpenReview ↗

Response to Reviewer mWLe (1/3)

Thank you for the detailed and thoughtful review, and for highlighting the prompt-search finding. We have simplified your comments for easier reference and address your concerns one by one. All new tables are also in Appendix G of the revised paper.

W1 & Q1: On Novelty

We want to clarify that APT's novelty lies in redefining how a prompt is used in RL training. In prior methods a prompt has one fixed role. GUIDE, HiLL and P2OP^{2}O use guidance only in training and always drop it at deployment, while GEPA and FST keep the prompt at both stages and optimize it as part of the deployed system. In APT every gradient is taken without the teacher in context, so a teacher decides only which responses the model learns from. Three decisions, each anchored to the bare model, put this into practice:

  1. Teaching. A teacher evolves with the policy, is kept only while its responses beat a fresh bare sample, and is retired once they no longer do.
  2. Transfer. Guided responses are distilled into the bare context with a pooled advantage, which keeps bare responses on their own baseline, and capped per-token weights, under which no guided token outweighs a bare sample (Lemma 1).
  3. Deployment. A prompt is kept only if a paired test at frozen weights still favors it over the bare model.

The cumulative path below shows that the margin comes from these decisions rather than from the components APT shares with prior work. Each row keeps the row above and adds the change it names:

Table R4{\color{blue}\text{Table R4}}: Cumulative path from GRPO to APT (Qwen3-8B, accuracy %, 3 seeds).

ConfigurationHoVerWebShopSATBenchAvg.
GRPO48.981.370.266.8
Guided responses on all-fail problems, fixed teachers49.385.667.667.5
Transfer: pooled advantage, capped weights49.487.068.168.2
Teacher rewriting, kept if better with the teacher in context (GEPA-style)51.387.068.669.0
Teaching: rewrites kept only if better than a fresh bare sample, with retirement53.688.070.770.8
Deployment selection (= APT)53.689.173.071.9

The first change is the core APT shares with GUIDE, HiLL and P2OP^{2}O: guided responses on all-fail problems, a gradient without the teacher in context, and no prompt at deployment. On its own it adds 0.7 points and costs 2.6 on SATBench. Transfer adds 0.7 with fixed teachers and matters more once teachers evolve, since removing the capped weights from full APT costs 2.4 points (Figure 5a). Rewriting teachers adds only 0.8 when a rewrite is kept for scoring higher with itself in context, as in GEPA. Judging the same rewrites against a fresh bare sample, and retiring teachers that no longer beat it, adds 1.8 more (95% paired CI [0.5, 2.9]). Deployment selection adds 1.1, all of it on WebShop and SATBench where the test keeps a prompt (+1.1 and +2.3), and nothing on HoVer where it keeps none.

The first change uses unit token weights, while GUIDE reweights guided tokens by the uncapped bare-to-guided ratio, the no-cap setting of Figure 7. Run in our setup, GUIDE reaches 67.7 on average (49.8 / 85.2 / 68.1 on HoVer / WebShop / SATBench), close to the first change (67.5), so the rest of APT's margin comes from Transfer, Teaching and Deployment. We start from GRPO rather than P2OP^{2}O because P2OP^{2}O also changes the training objective and searches templates, so one step from it would change several things at once. Appendix G.4 gives the full path with NPLB, per-problem hints, and acceptance and retirement as separate steps. Appendix C.9 defines each configuration, and Table R13 in the revised §5 compares these methods.

APT combines its components for clear reasons, and Table R4 shows that combining them without these reasons helps little. It starts from the two bottlenecks of §2. Problems the policy never solves give GRPO no signal (P1), so routing places guidance only on those problems. Training under a prompt improves the prompted system rather than the model (P2), so every gradient is taken without the teacher. On top of these, APT adds the pooled advantage and the capped transfer weights, which let guided responses teach the bare model without changing how bare responses are credited or copying teacher-specific tokens. The two tests against the bare model then keep a teacher only while the model still needs it and a prompt only for what the weights have not absorbed. Together these tests add 2.9 of APT's 5.1 points.

AuthorsOpenReview ↗

Response to Reviewer mWLe (2/3)

W2: On the Role of the Theory

We want to clarify that our theory motivates the method and is not a contribution: none of the three contributions in the Introduction is theoretical. We include it because it explains APT's design choices.

Proposition 1 shows that in the idealized update a teacher changes only the variance of the bare gradient estimate. This holds even though routing picks the teacher after the probes fail (Appendix D.1). An update under a prompt instead follows the prompted objective, so its gains can stay tied to the prompt, as §4.2 observes for Seq, Alternate and FST (Figure 2).

Lemma 1 and Lemma 2 then justify two components of the implemented method: the capped transfer weights and the fresh bare control. The cap keeps a teacher token from counting for more than it would in a bare sample, with promotion as the stated exception (Appendix C.5). The bare control, unlike the probes, gives an unbiased lift under binary rewards.

W3 & Q3: On the Frontier Editor

We want to clarify that our original results already show that most of APT's advantage over the baselines does not depend on the frontier editor. What matters for editor dependence is whether APT's margin over the baselines holds without Opus, because the joint baselines themselves use Claude-Opus5 to write the prompts they train under and deploy. FST also calls Opus 6 to 36 times as often as APT (Table 13). In the original paper, self-taught APT averages 70.4 over the three benchmarks (Table 12), above every joint baseline that uses Opus (at most 68.3, Table 1). To further address your concern, we additionally ran self-taught APT with Gemma3-12B and summarize both policies in Table R7 below:

Table R7{\color{blue}\text{Table R7}}: Editor strength (accuracy %). Qwen3-8B rows are from Tables 1 and 12, and Gemma3-12B rows other than APT (Gemma3-12B itself) are from Table 1.

Method (editor)HoVerWebShopSATBenchAvg.
Qwen3-8B policy
GRPO (none)48.981.370.266.8
Seq (Opus)49.184.271.768.3
Alternate (Opus)49.583.570.867.9
FST (Opus)48.485.371.268.3
APT (Qwen3-8B itself)50.589.471.270.4
APT (Opus)53.689.173.071.9
Gemma3-12B policy
GRPO (none)39.079.653.957.5
Seq (Opus)52.380.353.061.8
Alternate (Opus)50.972.554.359.2
FST (Opus)50.379.856.062.0
APT (Gemma3-12B itself)52.381.156.863.4
APT (Opus)53.883.858.465.3

As shown in Table R7, self-taught APT on Qwen3-8B keeps 2.1 of its 3.6-point margin without Opus, indicating that most of the margin comes from the mechanism rather than the editor. A stronger editor adds further gains, as expected for any method that writes guidance, including every baseline in our paper. The revised Contribution 3 and the §4.3 paragraph "Editor strength and self-improvement" (previously "Potential for self-improvement") now state this comparison in the main text.

On Q3, self-taught APT also stays above every joint baseline with a larger policy and with longer training. With Gemma3-12B as its own editor, APT reaches 63.4 on average, above every joint baseline (at most 62.0, Table R7). With 100 SATBench updates, self-taught Qwen3-8B reaches 71.4, above GRPO (70.0) and Seq (71.0) at the same length (Table R12 in part 3/3). The gap to Opus stays between 1.5 and 1.9 points in every setting, so it tracks the editor rather than the policy's size or training length. A stronger editor is therefore a benefit for APT, not a requirement.

W4 (first part) & Q2: On the System Prompt B0B_0

APT's robustness advantage holds when the baselines also keep B0B_0. To address your concern, we ran the Seq baseline with B0B_0 kept and its prompt appended in both search and training (Seq-append, Table R2 below). Seq matches FST on average (65.1 vs. 65.2 over six cells), so we use it as the representative joint method.

Table R2{\color{blue}\text{Table R2}}: System-prompt control (Qwen3-8B, 3 seeds). Robustness uses the five contexts of Figure 2: the mean, and the max - min across contexts of the benchmark average.

MethodPromptDeployed avg.Robust. meanRobust. spread
Seqreplaces B0B_068.349.530.1
Seq-appendappends to B0B_067.957.918.6
APTappends to B0B_071.971.41.2

Keeping B0B_0 raises Seq's robustness mean from 49.5 to 57.9 and lowers its spread from 30.1 to 18.6, but both stay far from APT (71.4 and 1.2). Its deployed accuracy does not rise (67.9 vs. 68.3). The remaining gap reflects (P2) in §2: Seq-append still trains under its prompt, so its gains stay tied to that prompt. Only Seq-append is a new run, and the Seq and APT rows summarize Figure 2. The revised §4.1 points to this control.

AuthorsOpenReview ↗

Response to Reviewer mWLe (3/3)

W4 (second part): On the Length Cap

The length cap does not decide APT's deployment prompt. The deployment test chooses among no prompt, the latest teacher texts used as is (up to 1,024 tokens), and four short texts written for deployment. Only the short texts are capped at 128 tokens, since a model that has absorbed most of the guidance needs only a short reminder (Appendix F.4). Each option wins in some runs (Table R3 below):

Table R3{\color{blue}\text{Table R3}}: Winning deployment candidate over 18 runs (3 benchmarks × 2 models × 3 seeds).

BenchmarkNo promptTeacher text (≤ 1,024 tokens)Newly written (≤ 128 tokens)
HoVer600
WebShop132
SATBench024

Among the 11 runs that deploy a prompt, a full teacher text of up to 1,024 tokens wins 5. The baselines keep their published search without a cap.

W5 & Q5: On Deployment Selection

Deployment selection is a quick check of whether any guidance still helps the trained model, and neither concern (the small task set or the missing retest) changes our results. The test split is never used to propose, compare or select any prompt (Appendix C.1). On Q5, we repeated the selection at the frozen 0.8N0.8N weights on a second draw of 256 training tasks. For SATBench, whose 302 training tasks would make two draws overlap, the second set is held-out puzzles from the original SATBench pool, outside our 302/70 split (Table R1 below):

Table R1{\color{blue}\text{Table R1}}: Repeated selection on a second task set, and test-set gain at the final weights (6 runs per benchmark).

BenchmarkSame z^\hat z on the second setDeployed - bare on test at final weights (Qwen3-8B / Gemma3-12B)
HoVer6/6 (no prompt)0.0 / 0.0
WebShop6/6+0.7 / +0.8
SATBench (held-out)4/6+2.9 / +2.3

The second set picks the same prompt in 16 of 18 runs, so the choice does not hinge on which tasks are used. In the two SATBench runs where it differs, another short candidate written for deployment ranks first. The deployed prompt still passes the test on the held-out puzzles, so all six deployed SATBench prompts pass on unseen puzzles. The decision to deploy a prompt is also consistent across seeds, each with its own data order: never on HoVer, always on SATBench and in five of six WebShop runs (Table R3). Noise alone rarely passes the test, since a candidate qualifies only if its net rescues over the bare context stay significant after Bonferroni correction. At the final weights, Table 11 of the paper already shows the deployed prompt beating the bare model on the test set in every cell that uses one (+0.7 to +2.9). The revised §3.4 reports this final-weight check in the main text.

W6 & Q4: On SATBench

We want to clarify that SATBench is not a counterexample but the clearest case for why APT keeps a deployment prompt. Our thesis is not that every gain ends in the weights. After training, APT tests whether any guidance remains unabsorbed and keeps a deployment prompt only in that case. HoVer and SATBench are the two outcomes of this test: on HoVer it keeps no prompt, and on SATBench it keeps one. On average over the three benchmarks, most of the gain over GRPO still remains without the prompt (3.9 of 5.1 points on Qwen3-8B and 6.8 of 7.8 on Gemma3-12B, Appendix E.2). The revised §4.2 discusses SATBench in the main text, and its last sentence now states this decision rule.

On Q4, what sets SATBench apart is the kind of reasoning it requires. Its puzzles require long chains of deduction over many constraints, and guidance for such procedures is harder to absorb into the weights. APT's own measurements show this. In the run of Figure 8b, SATBench never retires a teacher, while WebShop retires and reactivates its roles six times and HoVer ends with one active role. Training longer does not close the gap either (Table R12 below):

Table R12{\color{blue}\text{Table R12}}: SATBench with 100 updates (Qwen3-8B, accuracy %, 3 seeds). APT keeps the 80/20 split.

Method50 updates100 updates
GRPO70.270.0
Seq71.771.0
APT, Qwen3-8B as editor71.271.4
APT, bare70.170.9
APT, deployed73.073.3

With twice the updates, every puzzle is seen twice as often. Bare APT gains 0.8 points and overtakes GRPO (70.9 vs. 70.0), while GRPO and Seq do not improve. The deployment prompt still adds 2.4 points, and deployed APT remains the best method (73.3 vs. at most 71.0). This is exactly the guidance the deployment test is designed to keep.

We hope these responses address your concerns. If anything remains unclear, we are happy to engage in further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revision

Thank you for the careful reviews and constructive feedback. We are glad that reviewers found the prompt-search finding useful (mWLe), the guidance budget-neutral (mWLe, EUhe), the method well motivated (L5p1) and the claims well tested (mWLe, L5p1). We have addressed each comment individually and uploaded a revised paper, in which all changes are marked in blue{\color{blue}\text{blue}}. We summarize the main points below.

1. APT's novelty lies in how it uses a prompt in RL training (mWLe, EUhe). A prompt first acts as a teacher whose responses the model learns from in the bare context, and it remains at deployment only if it still works better than the bare context. The guided sampling that APT shares with prior work adds less than one point, and most of the gain comes from judging teachers against a fresh bare sample and from deployment selection (Table R4). Simply combining existing techniques therefore does not reach APT's performance.

2. Deployment selection is stable, and its prompt holds at the final weights (all reviewers). A second task set picks the same prompt in 16 of 18 runs, and at the final weights the deployed prompt still beats the bare model on the test set in every cell that uses one (Table R1). Selecting the deployment prompt on training tasks is therefore reliable.

3. APT extends beyond the three benchmarks (all reviewers). APT improves over GRPO and FST on MiniWoB++ and ScienceWorld (Table R8) and also when one policy is trained on several domains together (Table R11). These results show that APT is not tied to specific tasks.

4. Our theory motivates APT's design and is not a contribution of its own (mWLe, EUhe). Proposition 1 isolates the idealized contrast with joint methods, and Lemma 1 and Lemma 2 characterize the transfer-weight cap and the fresh bare control.

Paper Revisions (in blue).

  • Significance (§4.2 and §6; pages 7, 8 and 10): which margins are significant, and why SATBench keeps a deployment prompt.
  • Method (§3.2 and §3.4; pages 5 and 6): the cap of 1 set from Lemma 1, and the stability of deployment selection with its check at the final weights.
  • Experiments (§1, §4.1, §4.3 and Figure 5a; pages 2, 7 and 9): the system-prompt control, editor strength and self-improvement, and seed standard deviations for the ablations.
  • Related work (§5; page 10): Table R13 comparing the closest methods.
  • Appendix: definitions of the new baselines and variants (new C.9, page 21), corrected no-cap values (E.3 and Figure 7, page 25), and all new results with Table Rk in Appendix G.k (new G, pages 44 to 49).

Thank you again for your valuable feedback. If you have any further questions or concerns, please feel free to share them. We are happy to engage in further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.