acceptodds
Withdrawn from ICLR 2027

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher hurts generalization on benchmark families beyond the training corpus. Reinforcement learning (RL) avoids logit imitation by training on the student's own rollouts. However, on questions where every rollout fails, yielding zero advantage and being silently discarded, injecting a stronger teacher's response into the policy gradient breaks the on-policy assumption and induces drift. We introduce Zone of Proximal Policy Optimization (ZPPO), inspired by Vygotsky's zone of proximal development. ZPPO keeps the teacher inside the prompt rather than the policy gradient. On hard questions, where the student's mean rollout accuracy is below half, ZPPO constructs two reformulated prompts. A Binary Candidate-included Question (BCQ) pairs one correct teacher response with one incorrect student response as anonymized candidates the student uses as references. A Negative Candidate-included Question (NCQ) aggregates the student's wrong rollouts into a single prompt to surface their shared failure modes. A prompt replay buffer keeps each hard question eligible for re-sampling until it either graduates, the student's mean rollout accuracy on it reaches half or more, or is FIFO-evicted under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development. We post-train Qwen3.5 students at four scales (0.8B-9B) as vision-language models with a 27B teacher and evaluate them on a 31-benchmark suite (16 VLM, 10 LLM, 5 Video); ZPPO outperforms off/on-policy distillation and GRPO, with the largest gains at the smallest scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gAhzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses a teacher model to generate hints, thereby constructing hint prompts to train the student model, enabling the student model to still achieve a certain performance improvement when all rollouts are incorrect. The experiment results show its performance outperforms RL methods such as GRPO, as well as knowledge distillation methods such as OPD.

Critical strengths
  1. The paper conducts experiments across multiple model scales and provides extensive ablation studies.
  2. The paper starts from a meaningful research problem and investigates training optimization in the all wrong setting. The core idea is somewhat analogous to self-distillation, where additional privileged information is incorporated into the prompt to improve model optimization.
Critical weaknesses
  1. The paper does not sufficiently explain why capabilities learned with privileged prompts can transfer to prompts without privileged information. The underlying mechanism is unclear.
  2. The novelty appears limited and is closer to an engineering improvement. The method is similar to prompt-space curriculum learning methods [1][2][3], which improve training efficiency by incorporating additional hints into the prompt. However, no direct experimental comparisons with these methods are provided.

[1] Nudging the Boundaries of LLM Reasoning, ICLR 2026

[2] Self-Hinting Language Models Enhance Reinforcement Learning, arxiv 2026

[3] Learning to Hint for Reinforcement Learning, arxiv 2026

Questions for authors

Q1. Why can capabilities learned with privileged prompts transfer to standard prompts without privileged information? What drives this transfer, and how efficient is it?

Q2. Qwen3.5 base models of different sizes belong to the same model family and have similar policy distributions. Therefore, directly performing OPD within the same base model family may provide only limited incremental signals from the teacher, which can constrain the gains of OPD [1]. A more comprehensive comparison would be to first construct a teacher with a clear capability gap from the student through RL post-training, and then compare OPD and ZPPO under the same teacher–student setting. This would provide a more meaningful comparison in knowledge transfer.

[1] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, arxiv 2026

Reviewer_mDW5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to transfer the capabilities of a strong teacher model to a smaller student model without imitating the teacher's logits or directly incorporating teacher-generated responses into the student's policy gradient. In particular, the authors address a key limitation of standard Group Relative Policy Optimization (GRPO): when all student-generated responses within a group are incorrect, the relative advantages become zero, leaving the model with no effective learning signal for that problem during the current training iteration.To address these limitations, the authors propose Zone of Proximal Policy Optimization (ZPPO), which incorporates teacher-generated answers as reference material in the prompt. After accessing this material, the student independently generates a new response using its own policy, and the reward is determined by the correctness of the generated answer.And the paper evaluates ZPPO through vision-language model (VLM) post-training experiments on Qwen3.5 student models of four different sizes, ranging from 0.8B to 9B parameters. The main baselines include offline and online policy distillation, as well as GRPO. Experimental results show that ZPPO outperforms these baseline methods, with particularly pronounced improvements for smaller student models.

Critical strengths

The paper introduce the Binary Candidate-included Question (BCQ), which targets problems that are currently challenging for the student but can be solved correctly by the teacher. For each such problem, a compressed version of the teacher's correct solution and an incorrect solution generated by the student are included as two candidate responses. These candidates are anonymized and randomly ordered, after which the student is asked to generate a new answer.These innovations could better strenghen student and distallation. The paper also employs a Prompt Resampling Buffer to store challenging original questions for subsequent resampling. Each time a question is revisited, its BCQ/NCQ prompt can be reconstructed. A question is removed from the buffer once the student's independent accuracy on the original question reaches 50%.

Critical weaknesses

Applicability and Additional Computational Costs. ZPPO requires the teacher model to correctly solve at least some problems that are challenging for the student, as well as verifiable correctness-based rewards. When both the teacher and the student fail to solve a problem, BCQ becomes inapplicable, leaving NCQ as the only available option. this shows that ZPPO still need large enough teacher model so that comes to a larger cost , which leaves a limitation. Furthermore, a single training run requires approximately 7–17 additional hours compared with GRPO† on 64 H100 GPUs. These limitations raise concerns about computational cost-efficiency and restrict the range of tasks to which ZPPO can be effectively applied.

Questions for authors

Could the authors quantify how often ZPPO uses BCQ versus NCQ during training, and report performance as a function of the teacher’s accuracy or size? In particular, how does ZPPO behave when the teacher rarely solves student-challenging problems, or when both teacher and student fail? This would help clarify the method’s dependence on a sufficiently capable teacher and the settings in which NCQ alone is effective. Given the reported additional 7–17 hours over GRPO on 64 H100 GPUs, could the authors break down the extra cost among teacher inference, prompt construction/resampling, and student training? Could they also compare methods at matched GPU-hour budgets, or report performance per GPU-hour, to clarify whether ZPPO’s gains offset its additional compute?

Reviewer_p4isOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ZPPO, an RL post-training method in which a stronger teacher helps a small student through the prompt rather than through the loss. It adds three components to a DAPO-style GRPO pipeline, applied to hard training questions (those the student solves in fewer than half of its rollouts):

  1. Binary Candidate-included Questions (BCQ): one correct teacher response and one wrong student response are placed in the prompt, without saying which is which, and the student must produce the right answer.
  2. Negative Candidate-included Questions (NCQ): the student's own wrong responses are placed in the prompt, labeled as wrong.
  3. Prompt replay buffer: hard questions are revisited, with fresh candidates on each visit, until the student solves them at least half the time or they are evicted.

Experiments use Qwen3.5 students (0.8B to 9B) with a 27B teacher on 31 VLM, LLM, and video benchmarks; the largest gains are at the smallest scales.

Critical strengths
  1. Simple, interesting idea. Teacher guidance enters only the prompt, so updates stay on-policy, with no student–teacher importance ratio.
  2. Broad evaluation. Four student scales and 31 benchmarks across image, text, and video.
  3. Thorough ablations. Each component is ablated (Tab. 3), update counts are matched across methods, training cost is reported, and paired-bootstrap intervals show the gains over the main baselines hold across benchmarks (Tab. 20, 21).
Critical weaknesses
  1. Missing baselines and compute fairness.

    • Curriculum learning for GRPO is a popular line of work aimed at the same problem (hard questions with no learning signal), yet the paper dismisses it in one sentence: curricula "re-expose a hard prompt without saying why the last attempt failed." If the authors believe curricula fall short, they should show it empirically rather than dismiss them. This matters because ZPPO's replay buffer is itself a difficulty-based curriculum, and the paper's own ablation shows BCQ/NCQ help only when combined with re-exposure (Tab. 3). The only re-exposure baseline is plain prompt replay (GRPO†). Comparisons with adaptive curricula are needed to isolate the value of BCQ/NCQ, for example:
      • CLPO (Zhang et al., 2025), cited but not compared.
      • Goldilocks RL (Mahrooghi et al., 2026), cited but not compared.
      • ADCL/EGSR (Zhang et al., 2025, arXiv:2505.08364), neither cited or compared: combines an adaptive curriculum with reformulating expert solutions, close in spirit to BCQ.
      • AdaCuRL (Li et al., 2025, arXiv:2511.09478), neither cited or compared: targets gradient starvation and revisits past data, close to the replay buffer.
    • Compute. Matched update counts do not mean matched compute: ZPPO takes 7–17 more hours than GRPO† (L339), due to teacher calls and longer prompts. ZPPO and all baselines, including those in this list, should be compared at matched compute.
    • Dynamic sampling, DAPO's standard remedy for zero-advantage groups, is disabled in every run (App. D), and full DAPO is not compared. Dense or shaped rewards are another remedy for the same problem.
    • The standard pipeline of distillation (SFT on teacher outputs) followed by RL is missing.
  2. Presentation.

    • The paper is far more complex than the idea requires, and at 71 pages it relies heavily on the appendix. The main paper should stand on its own.
    • Statistical significance is buried in the appendix (Tab. 20, 21, 38). The main figures (Fig. 1, Fig. 6) and tables (Tab. 1–3) report means only. Please add confidence intervals to them, or at least discuss the key differences and their significance in the main paper.
    • Figures appear before the abstract, before any context is given, and are generally placed far from where they are referenced.
    • BCQ is discussed at length before it is defined (L264).
    • The baseline labeled "GRPO" shares ZPPO's DAPO-style pipeline (clip-higher, token-level loss, no KL, REINFORCE++ normalization, I = 4; App. E.2), but this is stated only in the appendix. Please state it in Sec. 4 or rename the baseline.
Questions for authors

At matched compute (e.g., equal wall-clock or GPU hours), how does ZPPO compare with:

  • an adaptive curriculum (e.g., CLPO, Goldilocks RL, ADCL/EGSR, or AdaCuRL)?
  • full DAPO, including dynamic sampling?
  • distillation (SFT on teacher outputs) followed by RL?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.