acceptodds
Withdrawn from ICLR 2027

Stabilizing Targeted On-Policy Guidance in Agentic Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Naive on-policy RL is often bottlenecked by the capabilities of the base model, as insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from highly semantic sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we develop a framework we refer to as targeted guidance (TGRL) that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so online without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by only constructing feedback on turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, on policy distillation, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_UVC4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors want RL to learn behaviors the base model never produces, using the textual feedback a judge already gives. In practice they add a self-distillation KL term to GRPO, where the teacher is the same model with a hint placed before the failing turn. They test this on one internal coding setup, and the behaviors it learns are output formats such as markdown tables and status updates written as poems.

Critical strengths

Using the judge's textual feedback instead of only a scalar reward is a reasonable idea. The finding that distilling only on failing turns is more stable than always distilling is a useful practical note.

Critical weaknesses
  1. I don't think the experiments show what the paper claims. The main claim is that TGRL learns the new behaviors without hurting coding performance. But nothing in the paper indicates more than one run per method, and the "error bars" in Table 1 are the standard deviation over the last 32 steps of training. The numbers are training metrics on a task set the paper never names, with no held-out evaluation. So when TGRL gets 0.33±.05 and GRPO gets 0.33±.07, that tells us nothing about whether coding was hurt. Figure 5 is worse. The TGRL run stops around step 420 while every other run goes to 500, and for most of that range its pass rate is slightly below the baseline. Why was it cut short?

  2. I can't see what is new here, and I can't check the experiments either. The method is GRPO plus on-policy self-distillation (Zhao et al., 2026), and the paper credits gating distillation to failing rollouts to Li et al., Ding, and Lu et al. (2026). What is left is moving the hint to the failing turn and applying it to several criteria. If the method isn't new, the paper has to stand on its experiments. Yet the dataset and the judge model are never named, only one feedback string is shown (Figure 3), and the rubric items appear only as short column names in Table 1. There is no appendix and no code. Nobody outside the authors' group could rerun any of this.

  3. I'm not convinced the baselines were run properly. LaTRO receives extra style signal on top of the coding reward, yet it does no better than the Baseline that ignores style on six of the seven criteria, and is clearly worse on Bolded (0.43 to 0.21) and Sol. Desc (0.70 to 0.57). Its pass rate also falls from 0.33 to 0.14, and Online SFT and Verifree lose a lot of pass rate too (0.26 and 0.20). A method that makes things worse on the very thing it is supposed to improve looks more like a broken implementation or poor tuning than a fair comparison. Beyond a one-sentence description of each, the paper gives no implementation or tuning details. Without that, Table 1 can't tell us whether TGRL is better or the alternatives were just set up badly.

Questions for authors

Did the GRPO baseline see the style requirements in its prompt? Markdown stays at 0.00 for both Baseline and GRPO, which looks like the model was simply never asked.

How many seeds were run, and is there any held-out evaluation?

Why does the TGRL run in Figure 5 end early, and what does pass rate do after that?

Which judge model was used, and what are the feedback strings for the style and poem criteria?

1 comment

The authors did not provide any response during the rebuttal phase, so I am maintaining my score.

Reviewer_RV2AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes targeted guidance (TGRL), a way to optimize auxiliary reward criteria in agentic RL that naive on-policy RL fails to improve because the desired behavior is outside the base policy’s support. Examples include using a new tool-call format or a particular writing style. Each binary reward criterion comes with a textual feedback string. When an agent turn fails a criterion, the feedback is inserted just before that turn to define an expert: the same model with the feedback in context. A per-token reverse-KL self-distillation loss toward this expert is then added to GRPO for the final optimization objective.

Critical strengths

S1. Interesting, well-defined experimental settings. The settings are interesting and clearly defined. In Sec. 5.1, a new tool format is introduced after SFT, and the base model fails to use it even when examples are in the prompt. In Sec. 5.3, a poem-style status update is required at every turn. Both are realistic cases where the needed behavior is outside the base policy’s support, and they isolate the auxiliary criterion from the primary coding reward, so it’s easy to see what the method contributes.

S2. Candid discussion of failure cases with proposed fixes. It’s good to see the paper analyze where the method fails and give a logical account of how to address it. Entropy spikes from too high a distillation weight are linked to output-length growth and then pass-rate collapse. This kind of analysis helps other researchers in the community trying to reproduce the method.

Critical weaknesses

W1. The problem statement, core intuition and method don’t connect clearly. The paper begins with hard-to-optimize rewards in agentic RL post-training and proposes self-distillation combined with per-criterion RL rewards as the solution. Why this works is not clearly argued, and following points remain unclear to me.

A. The policy-improvement argument conflicts with the motivation. Sec. 4 justifies distillation as policy improvement under two conditions: the expert has higher expected return, and the KL between student and expert stays small. But the motivation is behavior outside the student’s support, which implies a large divergence from the expert. It’s unclear when both conditions can hold, and the paper itself says neither is guaranteed (lines 168–169). Sec. 5.1 adds to the confusion. It says the base model fails to use the new format “even when example usage is placed in the prompt,” yet the expert is the same model with feedback in context. Why does feedback placed right before the turn work when a prompt example doesn’t?

B. Too many goals compete in one narrative. The paper aims at coverage and exploration, no regression on the primary reward, stable entropy, and fewer oscillations between criteria. The entropy behavior is presented as a further problem to solve. It’s not clear whether it’s central to the contribution or a side effect of the chosen objective. A clearer progression from problem to intuition to method would make the contribution much easier to read and judge.

W2. Limited novelty of the core idea. Using the policy being trained as its own teacher during RL, by conditioning it on privileged information or feedback, is not new. The paper acknowledges that it builds on on-policy self-distillation (Zhao et al., 2026), and it cites prior work that already restricts distillation to failing rollouts. It doesn’t cite Self-Distillation Policy Optimization (SDPO; Hübotter et al., 2026). SDPO already conditions the model on rich textual feedback (runtime errors, LLM-judge evaluations) as a self-teacher and distills the resulting token-level predictions into the policy. What seems to be novel appears to be per-turn feedback placement and a per-criterion RL reward for multiple auxiliary criteria. These are reasonable engineering choices, but the paper should state them explicitly as its contributions and compare at least against SDPO and SDAR (Lu et al., 2026).

Questions for authors
  1. (W1) How do you reconcile the small-KL condition (Eq. 2) with targeting behaviors outside the student’s support? Can you report the student–expert KL on failing turns over the course of training?

  2. (W1) In Sec. 5.1, why does the feedback-conditioned expert succeed when example usage in the prompt doesn’t?

  3. (W1) Is entropy stability a goal of the method or a side effect to manage?

  4. (W2) How does TGRL differ from, and compare empirically with, SDPO (Hübotter et al., 2026) and SDAR (Lu et al., 2026)?

  5. Can Table 1 be rerun with multiple seeds and confidence intervals?

Reviewer_86j9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is investigating a particular way of credit assignment when training LLMs with reinforcement learning for agentic multi-turn tasks such as software engineering. The paper's claim is that existing task-based rewards given at the end of rollouts may fail to teach the policy new behaviours if the base model does not have sufficient coverage. To get around this problem, the work proposes using a KL-divergence distillation loss from a teacher model only on turns where the learner policy fails. Notably, the teacher model in this case is simply the learner model again but with some steering prompts, that the paper calls textual feedback, added as additional context on the failing turn.

The paper's novel contributions are that it shows the role additional feedback could have on policy learning dynamics, and how combining a limited amount of distillation losses with existing training rewards can solve certain tasks. The experiments are carried out in agentic coding settings and many of those are motivated by wanting to maintain the agent's coding performance while teaching it to optimize other criteria like adapting writing styles or using new tool calls.

Critical strengths

The writing style of the paper is clear and easy to follow. It does not make any strong claims and instead seeks to investigate how the method performs, which I liked about the paper. The paper also acknowledges its limitations in the conclusion, and I agree with it that it is an interesting experiment to see how much an agentic policy can be improved if we leveraged more of the given data.

Critical weaknesses

The introduction of the paper is poorly presented and the reader has to spend a lot of effort to figure out what the objective of the work is. Neither the abstract nor the introduction make it explicit that the object of discussion is large language model that is being post-trained with RL. For example, the introduction begins with "The strength of large-scale reinforcement learning (RL) has been well established in recent years. However, there is often a limit to how much a policy can improve from naive on-policy RL". These sort of vacuous statements tell the reader nothing about what the problem is or what "naive on-policy RL" means. At times, I was wondering if the paper was conflating the problem of exploration in RL with the problem of not being able to learn from feedback: "Fundamentally, while RL incentivizes existing good behaviors, such methods do not communicate how to perform new behaviors." It is not at all clear what is being to referred to by "such methods". If the paragraph wants to make a remark specifically about the baseline GRPO algorithm, it should say so. It is only until much later that I realised that this is a remark about how rewards are assigned in RL and how they can influence exploration. The non-technical language makes the sentence sound ambiguous.

The ideas building up to the method are presented in a hand-wavy way, "The first main idea is that we would like to minimize the amount of unnecessary pressure from the distillation objective, which can be heuristically understood as...". "The second core idea is that while self-distillation can provide a strong directional signal of improvement, it is less clear that such objectives result in stable convergence." with no citations or explanations for the convergence claim.

The empirical results are not strong either. The experiments were not clearly described, but they appear to be quite limited and I cannot infer if they were statistically significant. The paper frequently discusses results without explaining what the experiment was: "As shown in Figure 3, a policy trained with targeted guidance learns the new tool calling format, without suffering a regression on the primary coding performance." The reader is given no information about the base model used for the experiment and the baseline in the plot is plain GRPO without distillation. In the introduction, the paper states this, "Importantly, our aim is that such signal can directly improve targeted policy behaviors without degrading baseline RL performance. Naive on-policy distillation methods have previously been shown to cause phenomena such as reasoning shortening or diversity collapse" but I feel like these statements were not adequately verified in the experiments against the baseline OPSD.

I am not entirely convinced if the idea of combining distillation losses with GRPO's training rewards adds any new information to the field. The paper does cite some related works, but it is not exactly clear what the new challenges in this particular work were that made it incomparable to prior works using OPSD since those were not discussed in the experiments section.

Questions for authors

line 188: "The distillation loss in our setting is calculated via reverse KL distance between the student and the teacher. In our experiments, we found that forward-KL and Jensen-Shannon distance performed similarly." Could the authors elaborate on what they found when using forward KL? This sentence leaves the reader wondering if forward KL did worse than reverse KL.

line 190: "To reduce undesirable effects from inference-trainer mismatch, we utilize a full-vocabulary KL loss computed entirely on the trainer side." I don't understand what a full-vocabulary KL loss means. I couldn't find it defined anywhere else in the paper.

line 178: "Thus, in contrast to prior work which places privileged information into a global context to construct the self-distillation target, we place the feedback string right before the affected agent turn, and mask out the KL loss on all other turns." it would be helpful to cite the prior works that this sentence is referring to.

Could the authors describe the empirical setup in more detail and say whether the results are statistically significant enough to be compared against other works in this space?

Reviewer_Vy14OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TGRL combines GRPO with self-distillation from the same policy conditioned on textual feedback. Guidance is restricted to turns that fail auxiliary criteria, with corresponding rewards intended to retain the learned behaviors. Experiments on Qwen3-Coder-30B address tool formatting and communication style, aiming to preserve coding performance while improving compliance.

Critical strengths
  1. The paper addresses a practical post-training problem with a simple intervention. Table 1 shows markdown compliance increasing from 0% under GRPO to 85% under TGRL, with the same reported mean coding pass rate.

  2. The ablations expose useful failure modes.

Critical weaknesses
  1. The evaluation is not sufficiently specified to establish generalization. The coding dataset, train/test separation and judge configuration are not reported, and no clearly identified held-out evaluation is presented.

  2. The incremental benefit over existing gated RL plus OPSD is not cleanly isolated. Section 2 acknowledges closely related hybrids, but the OPSD comparator in Table 1 omits auxiliary rewards. Failure gating also reduces distillation exposure, and adding thinking tokens changes its coverage without a reported control for effective loss strength. Matched rewards, comparable tuning and total-compute accounting are needed.

Questions for authors
  1. Which coding tasks and judges produced the reported scores, and what separates training, feedback-prompt selection and final evaluation? Please provide results on explicitly disjoint held-out tasks.

  2. Across independent training seeds, what is the uncertainty on TGRL’s coding pass-rate difference from baseline? Do the reported stability improvements and failure modes recur?

  3. Do turn-local feedback and thinking-token distillation still help against a reward-matched gated RL plus OPSD baseline when effective distillation strength is controlled? Please report total compute, including teacher passes, rather than training steps alone.

  4. How does the benefit change with model scale, given that the feedback-conditioned teacher shares the student’s weights? Could weak feedback comprehension in smaller models, or stronger baseline exploration in larger models, reverse TGRL’s ranking against GRPO or OPSD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.