- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a new method called Adaptive Prompt Teaching (or APT) for training large language models which is designed to address the lack of a gradient signal in GRPO in special case when all responses in a group are incorrect. The proposed method proposes a 2-step procedure: (1) add teacher prompts for tasks that the base model cannot handle, (2) distil the resulted correct solutions into the model under the bare context. They also propose a mechanism to update the teacher prompts ensuring they remain useful throughout training. Finally, an evaluation method for this prompts is provided.
Critical strengths
The paper is well motivated and provides a well grouded and well designed novel method. The experiments provide a robust analysis of the methods components proving its efficiency.
Critical weaknesses
The main concern is about the validity risks: their is no separate val sprit, the prompts are selected on the training tasks which are further used in model training. Deployment-prompt is not revalidated after the last 20% and its evaluation could be noisy.
There is a risk of catastrophic forgetting (at this point each run is trained only on one benchmark), various prompt formulation experiment shows only the prompt robustness within the domain.
Questions for authors
Is APT applicable for the cross-domain scenarios? Have You evaluated it in this scenarios?
5 comments
Response to Reviewer L5p1 (2/2)
W2: On Forgetting
We want to clarify that APT, like every method we compare, is a task-adaptation method, and forgetting on other domains comes with this setting rather than with APT. Every method in Table 1 trains one policy per benchmark, and RL on one task can lower accuracy on unrelated tasks for any of them, especially when starting from post-trained models. APT does not add a source of forgetting. Every teacher-stage gradient is taken in the bare context, and a teacher only decides which responses the model learns from. Only 3% to 28% of each cycle's responses are guided in the run of Figure 8c, and the last 20% of updates are GRPO under .
Our robustness test stays within the domain because it targets the risk that APT's design addresses. A method that trains under a prompt can tie its gains to that prompt (P2 in §2). Figure 2 tests this directly, and APT's accuracy barely moves across five contexts (spread 1.2 vs. 30.1 for Seq, Table R2 in our response to Reviewer mWLe). When several domains matter, the direct remedy for forgetting is to train on them together, and APT still beats GRPO in that setting (Table R11 below).
Q1: On Cross-Domain Use
Yes. APT works when one policy is trained on several domains, and it also improves on new benchmarks. To address your question, we trained one policy on a HoVer + SATBench mixture, with one teacher set and one deployment test per domain (Table R11 below):
: One policy trained on HoVer + SATBench (Qwen3-8B, accuracy %, 300 updates, 3 seeds).
| Method | HoVer | SATBench | Avg. |
|---|---|---|---|
| GRPO, mixed | 47.9 | 69.0 | 58.5 |
| APT, mixed | 52.1 | 70.9 | 61.5 |
As shown in Table R11, APT improves over GRPO by 3.0 points on average (+4.2 on HoVer and +1.9 on SATBench). As noted above, zero-shot transfer of a checkpoint to an unrelated domain is outside the scope of task adaptation. Joint training is how APT covers several domains.
To further show that APT is not tied to our three benchmarks, we also ran it on MiniWoB++ and ScienceWorld. It reuses the WebShop settings of Table 4 with 100 updates and needs only a new task contract for the reflection model (Table R8 below):
: New environments (Qwen3-8B, success rate, 100 updates, 3 seeds).
| Method | MiniWoB++ | ScienceWorld |
|---|---|---|
| GRPO | 68.4 | 50.0 |
| FST | 65.3 | 52.3 |
| APT, bare context | 73.1 | 61.4 |
| APT | 75.2 | 63.3 |
As shown in Table R8, the bare policy of APT already beats GRPO and FST on both environments, and the deployment prompt adds 2.1 and 1.9 points on top.
We hope these responses address your concerns. If anything remains unclear, we are happy to engage in further discussion.
Kind Reminder and Summary of Our Rebuttal
Dear Reviewer L5p1,
Thank you again for taking the time to review our work and for recognizing APT as well motivated and well designed. Since the discussion period is ending soon, we briefly summarize the key points of our rebuttal:
- Validation of the deployment prompt (W1): Repeating the selection on a second task set picks the same prompt in 16 of 18 runs. At the final weights, the deployed prompt still beats the bare model on the test set in every cell that uses one (). The test split is never used to select any prompt.
- Forgetting and cross-domain use (W2, Q1): One policy trained on HoVer + SATBench improves over GRPO by 3.0 points (). APT also improves over GRPO and FST on MiniWoB++ and ScienceWorld ().
Full details are in our responses above and in of the revised paper. We hope these results resolve your concerns. If anything remains unclear, we would be glad to follow up before the discussion closes.
Best regards,
The Authors
Response to Reviewer L5p1 (1/2)
Thank you for the thoughtful review and for recognizing that APT is well motivated and well designed. We have simplified your comments for easier reference and address your concerns one by one. All new tables are also in Appendix G of the revised paper.
W1: On Validation of the Deployment Prompt
We want to clarify that the deployment prompt is selected on training tasks but does not overfit to them. We address your concern in three parts.
1. A prompt that overfits to a few training tasks is unlikely to pass the test. The reflection model writes new candidates from training tasks, but each one must then beat the bare model on 256 tasks at the frozen 0.8N weights, significantly after Bonferroni correction. Breaking a task that the bare model solves counts against it. The new texts are at most 128 tokens and may not contain task answers or training identifiers (Appendix F.4), and they also compete with the latest teacher texts used as is. When no candidate passes, APT deploys no prompt, as on every HoVer run. The prompt kept for the last 20% of updates therefore has significantly more rescues than harms on 256 tasks. As for a retest after these updates, Table 11 of the original paper already shows that the deployed prompt still beats the bare model on the test set at the final weights in every cell that uses one (right column of Table R1 below).
2. The choice is stable across task sets. To address your concern on noise, we additionally repeated the selection at the frozen 0.8N weights on a second set of 256 tasks. For HoVer and WebShop this is a new draw of training tasks. For SATBench, whose 302 training puzzles are too few for a second draw, it is held-out puzzles from the original pool (Table R1 below):
Table R1: Repeated selection on a second task set, and test-set gain at the final weights (6 runs per benchmark).
As shown in Table R1, the second set picks the same prompt in 16 of 18 runs. In the two SATBench runs where it differs, another short candidate written for deployment ranks first. The deployed prompt still passes the test on the held-out puzzles, so all six deployed SATBench prompts pass on unseen puzzles. The decision is also consistent across seeds, which differ in data order: no prompt in all six HoVer runs, and a prompt in all six SATBench runs and five of six WebShop runs (Table R3 in our response to Reviewer mWLe).
3. There is no contamination. The test split is never used to propose, compare or select any prompt (Appendix C.1). The revised §3.4 reports both checks in the main text.