acceptodds
Under review as a conference paper at ICLR 2027

VALVE: Critic-Guided On-Policy Distillation for Agentic Reinforcement Learning

Abstract

On-policy distillation (OPD) provides dense supervision for training large language model agents, but enforcing teacher preferences can suppress valid alternatives in tasks with multiple solution paths. The challenge is to distinguish disagreements that warrant correction from those that reflect successful alternative behaviors. Our analysis shows that student entropy and teacher–student divergence provide limited separation between these cases, while the relative performance of fixed distillation strengths reverses over training. We propose VALVE, a critic-guided, return-aware method that adapts teacher supervision when combining OPD with reinforcement learning. VALVE uses action-level advantages from the proximal policy optimization (PPO) critic as return credit for task completion, with their signs indicating whether interaction actions are estimated to yield higher or lower return than expected from the current state. It combines this return credit with token-level teacher support, measured by teacher–student log-probability gaps, assigning larger OPD weights when the two signals agree in sign and smaller weights when they conflict. The weights adapt across tokens, interaction steps, and training stages, while retaining the standard PPO actor loss and requiring no additional rollouts or model evaluations beyond PPO and OPD. Across ALFWorld and WebShop with three Qwen students ranging from 1.7B to 7B parameters, VALVE achieves the best or tied-best task success among the evaluated methods. On ALFWorld, it outperforms the strongest competing baseline by 5.3–11.7 percentage points across students. With Qwen2.5-3B, VALVE also yields greater solution diversity than OPD and GRPO on ALFWorld. It reaches the GRPO teacher’s success rate on both benchmarks using 66–68% fewer GPU-hours than GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, agentic reinforcement learning, large language model agents, critic-guided learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HqNBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript propose a novel LLM RL algorithm of VALVE, which combines PPO with on-policy distillation for interactive agents. VALVE uses the product of a raw action advantage and a teacher–student token log-probability difference to gate the distillation loss. This is intended to retain useful teacher guidance without suppressing successful student exploration. Experiments on ALFWorld and WebShop cover three policy models, including Qwen2.5-3B/7B and Qwen3-1.7B, with additional analyses of successful-solution diversity, gate components, and compute required to reach target performance.

Critical strengths
  1. A novel LLM RL algorithm of VALVE is proposed and integrate PPO with OPD to balance exploration and exploitation.

  2. The gains are substantial across both environments rather than confined to one configuration. For example, ALFWorld success reaches 91.491.4, 95.395.3, and 92.292.2 for the three students. The gate ablations also favor the complete method over unweighted distillation and either single-signal variant, supporting the usefulness of the combined design.

  3. The diversity analysis goes beyond average success. On 111 multi-solution tasks, 42% yield at least two distinct successful solutions, versus 17% for OPD and 13% for GRPO. Measuring different successful behaviors gives useful evidence for the stated exploration objective.

Critical weaknesses
  1. One of the major concerns for me is the rationale behind VALVE, which is an RL method but requires a teacher model. In modern LLM post-training workflows, we typically use RL to push specialized performance and then use knowledge distillation, such as OPD, to combine these capabilities into one model. VALVE cannot readily replace the RL stage because it requires an existing teacher, but it also seems less suitable for the distillation stage due to its additional compute burden. Its practical role in this workflow is therefore unclear.

  2. RG-OPD is already cited, but I could not find a direct comparison. Since it also gates distillation using reward and teacher–student agreement, this comparison matters to the novelty of VALVE. The strong gains over ordinary RL establish utility, but do not isolate this distinction. How much does the action-level gate add under the same teacher and rollout budget?

  3. The models used, such as Qwen2.5-3B/7B and Qwen3-1.7B, are small and relatively dated, with limited agentic pretraining or mid-training. Using the Qwen3.5 series, which already incorporates agentic knowledge, would be more convincing. Moreover, ALFWorld (2021) and WebShop (2022) are also dated benchmarks. Evaluation on benchmarks introduced in the past two years would be more meaningful.

Questions for authors

Please see Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer HqNB [1/4]

We thank you for the detailed comments and for recognizing the substantial gains and the diversity analysis. We respond to each point below.

[W1 a] On the teacher requirement

VALVE is an OPD+RL method, and using a teacher is a shared premise of this paradigm, not an additional requirement introduced by VALVE. Combining OPD with RL is an actively studied paradigm [1][2][3][4][5], and all of these methods use a teacher for OPD and environment rewards for RL. VALVE follows the same setup: in a single training stage, the student is optimized with the PPO objective on environment rewards and with an OPD loss toward the teacher (Eq. 9).

VALVE focuses on the following question: given a teacher, where should teacher supervision be applied? Our analysis (Section 2) identifies a problem that is pronounced in multi-turn agentic tasks: because these tasks admit multiple valid solution paths, a student's deviation from the teacher is often a valid alternative and not an error, and applying teacher supervision equally at every step suppresses such alternatives. This problem is central to distilling agents: suppressing valid routes both narrows the student's behavior and lowers its success (Section 4.3). VALVE addresses it by using the critic's return estimates to decide where teacher supervision is applied. We have made this positioning explicit in the revised introduction.

[W1 b] On the RL-then-OPD workflow

Merging specialized capabilities into one model is an important use of OPD, but it is not the problem studied in this paper. VALVE targets another use of OPD in the same workflow: transferring a teacher's capability to a student in multi-turn agentic tasks, as in strong-to-weak distillation (e.g., Qwen3 [6]).

The problem in this setting is multi-solution collapse. OPD supervises the student only through the teacher's distribution, so it cannot tell whether a deviation from the teacher is an error or a valid alternative route, and it pulls both back to the teacher's route. An RL-trained teacher does not remove this problem, because the problem arises during distillation: RL makes the teacher stronger, but its preferred route is still only one of several valid routes. This is consistent with prior observations that OPD reduces policy entropy and behavioral diversity [4][7][8].

References

[1] KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, Fei Mi. 2025.

[2] Self-Distilled Agentic Reinforcement Learning. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen. 2026.

[3] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning. Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang. 2026.

[4] Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL. Qizhen Lan, Xi Xiao, Xiangchen Guan, Mengchen Fan, Moule Lin, Jung Im Choi, Lijing Zhu. 2026.

[5] RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning. Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen. 2026.

[6] Qwen3 Technical Report. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. (Qwen Team). 2025.

[7] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, Ning Ding. 2026.

[8] Entropy-Aware On-Policy Distillation of Language Models. Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, Kimin Lee. 2026.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer HqNB [2/4]

[W1 b] On the RL-then-OPD workflow (continued)

Our OPD baseline in Figures 4 and 5 directly instantiates the RL-then-OPD workflow: the teacher is Qwen2.5-3B trained with GRPO for 150 steps, and it is distilled into a student of the same size with standard OPD (Appendix A.6). With the same teacher, on 111 multi-solution ALFWorld problems with eight samples each (Figure 4):

Method (same GRPO teacher)Problems with ≥2 distinct verified solutionspass@8 (%)
OPD (RL → OPD)17%64.0
VALVE42%94.6

VALVE outperforms OPD at every evaluated kk (Figure 4(a)).

This collapse is common and costly in agentic tasks: 25.42% of all actions are valid multi-solution disagreements, and replacing the student action with the teacher action at these steps lowers the expected return by 7.65 points, whereas it raises the return by 7.01 points at conflict disagreements (Figure 2(c)).

Strengthening the teacher with RL therefore does not by itself prevent this collapse. VALVE addresses it within the transfer step: it allocates teacher supervision according to return information, so that valid alternatives are preserved while errors are still corrected.

[W1 c] On the practical role and compute cost

In this workflow, VALVE is used at the distillation stage, where it is better suited than standard OPD: with the same teacher, it preserves the student's valid alternatives and reaches higher success. It reuses the task reward from the RL stage and requires no additional annotation. Its advantage over OPD grows when a task admits several valid solutions, which is common in multi-turn agentic tasks (Section 2).

On the RL stage. We agree that VALVE does not replace the RL stage that produces the teacher. VALVE is also robust to the teacher's quality (Section 4.3, Figure 4(c)):

Teacher (GRPO steps)Teacher success (%)VALVE student (%)
503468
1006488
1507391

Even a teacher with only 34% success yields a 68% student, and a 64% teacher yields 88%, close to the 91% obtained with the mature teacher.

On the distillation stage. With the same teacher, VALVE improves on standard OPD in two ways (Qwen2.5-3B):

  • More valid solutions. VALVE keeps two or more verified solutions on 42% of multi-solution problems, compared with 17% for OPD, and raises pass@8 from 64.0% to 94.6% (Figure 4, W1 b).
  • Continued improvement. OPD stops improving once it reaches the teacher's level, whereas VALVE keeps improving (Figure 5(a)).

Adding the RL objective alone does not account for this gain: with the same PPO objective, Uniform-weight OPD reaches 74.2% / 57.8% on ALFWorld / WebShop, compared with 91.4% / 71.1% for VALVE (Table 1).

On compute. The additional cost is moderate. Per step, VALVE adds a PPO update with its critic to OPD; the gate reuses the critic's values and the teacher's log-probabilities and adds no rollouts or model evaluations (Section 3). GPU-hours to reach the teacher's success rate (Table 2):

MethodWebShopALFWorld
OPD4.8410.07
VALVE7.5414.60
GRPO22.0545.31

VALVE needs about 1.5× the GPU-hours of OPD, which then plateaus at the teacher's level, and 66–68% fewer than GRPO.

VALVE thus takes the place of standard OPD in the distillation stage at a moderate extra cost, and among existing OPD+RL methods it achieves the best or tied-best success in all settings (Table 1).

[W2 a] On the direct comparison with RG-OPD

We have implemented RG-OPD following its paper [9]. For a controlled comparison, its gate runs in the same training framework as VALVE, with the same student, teacher, rollout budget, and RL objective. VALVE outperforms RG-OPD by 18.8 points on ALFWorld and 11.8 points on WebShop.

Method (student: Qwen3-1.7B; teacher: Qwen3-4B GRPO)ALFWorld success (%)WebShop Acc (%)
GRPO50.054.7
Uniform-weight OPD75.850.0
KL-weighted OPD78.150.8
ATOD80.542.1
SDAR53.958.6
RG-OPD (new)73.452.3
VALVE92.264.1

In this multi-turn setting, RG-OPD performs close to Uniform-weight OPD (73.4% / 52.3% vs. 75.8% / 50.0%). The original submission did not include it because RG-OPD was proposed and evaluated for single-turn math, coding, and instruction-following tasks [9]. These results have been added to Appendix A.13 of the revision.

References

[9] Reward-Gated On-Policy Distillation. Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski. 2026.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer HqNB [3/4]

[W2 b] On the difference between VALVE and RG-OPD

RG-OPD cannot preserve valid alternative solutions in multi-turn agentic tasks, because it makes a single keep-or-drop decision for an entire trajectory. It compares the trajectory-level advantage with the teacher–student likelihood gap summed over the whole trajectory, and then keeps or discards the distillation loss of all tokens in that trajectory together [9].

An agentic trajectory, however, spans up to 30 interaction steps and mixes two kinds of steps: steps where the student should still follow the teacher, and valid branches that the teacher does not prefer (e.g., selecting the size before the color in Figure 1). To quantify this, we additionally applied the action-replacement protocol of Appendix A.3 to trajectories with teacher–student disagreements: 58.6% of them contain both a step where the teacher's action yields a higher expected return and a step where the student's action does. We have added this analysis to Appendix A.3 of the revision. For such a trajectory, keeping the loss pulls the valid branch back to the teacher, and dropping it discards the teacher's guidance at every other step.

VALVE decides for each token: the PPO critic assigns return credit to each interaction action, so within one trajectory, teacher guidance is strengthened at some steps and weakened at others (Sections 3.2–3.3). The consequence is visible on the 111 multi-solution ALFWorld problems (Qwen2.5-3B, eight samples each):

MethodProblems with ≥2 distinct verified solutionspass@8 (%)
GRPO13%63.1
OPD17%64.0
RG-OPD (new)18%65.8
VALVE42%94.6

RG-OPD does not resolve the multi-solution collapse: its results are on par with OPD, whereas VALVE more than doubles the share of problems with multiple valid solutions and raises pass@8 by about 30 points. We have added RG-OPD to the method comparison in Table 4.

[W2 c] On how much the action-level gate adds

Under the same teacher and rollout budget, VALVE improves over RG-OPD by 18.8 points on ALFWorld and 11.8 points on WebShop, and most of this gain comes from VALVE's action-level decision. The critic estimates the advantage of each action, so teacher guidance is strengthened or weakened step by step. RG-OPD decides once per trajectory, so it cannot separate the steps that should follow the teacher from the valid branches in the same trajectory.

RG-OPD and VALVE differ in two respects:

  1. What the decision is based on. RG-OPD uses one advantage for the whole trajectory; VALVE uses the advantage of each action, estimated by the critic.
  2. How the decision is applied. RG-OPD keeps or drops the distillation loss; VALVE scales it by a weight between 0 and 1 (Eq. 8).

To measure each part, we swap them one at a time, which gives two intermediate variants. All four runs share the student (Qwen3-1.7B), teacher (Qwen3-4B GRPO), rollout budget, RL objective, and evaluation protocol; only these two choices change.

VariantDecision based onDecision applied asALFWorld success (%)WebShop Acc (%)
RG-OPDtrajectory advantagekeep or drop73.452.3
RG-OPD with weighttrajectory advantageweight in (0, 1)77.355.1
VALVE with keep/dropaction advantagekeep or drop89.862.9
VALVEaction advantageweight in (0, 1)92.264.1
  • VALVE's action-level decision is the main source of the gain. Replacing the trajectory advantage with the action advantage adds 16.4 / 10.6 points under keep-or-drop and 14.9 / 9.0 points under weighting.
  • Weighting adds a smaller gain. Replacing keep-or-drop with a weight adds 2.4–3.9 / 1.2–2.8 points.

Adding a weight to RG-OPD does not close the gap (77.3% vs. 92.2% on ALFWorld).

On novelty. This comparison isolates what distinguishes VALVE from RG-OPD: VALVE uses the critic's action-level return credit to decide, step by step, where teacher supervision is applied. This component accounts for most of the gain over RG-OPD, and it is what allows VALVE to preserve valid alternatives in multi-turn tasks (W2 b). To our knowledge, no prior OPD method allocates teacher supervision with action-level return credit (Table 4).

References

[9] Reward-Gated On-Policy Distillation. Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski. 2026.

Reviewer_9PjiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts the weight on on-policy distillation per token, using two signals: the teacher's relative support for the sampled token, and a critic-based estimate of whether the action that token belongs to is expected to help or hurt the return. The two signals are multiplied, passed through a sigmoid, and used to scale the distillation term, while the underlying actor loss is unchanged. The argument is that entropy and teacher-student divergence cannot tell a correctable mistake from a valid alternative behaviour, whereas a return estimate can. Experiments cover ALFWorld and WebShop with three student sizes and a large set of baselines.

Critical strengths
  1. The motivating example is convincing. Two actions can deviate from the teacher in the same way by local measures while only one of them loses return, and the paper makes that distinction concrete rather than asserting it.

  2. The results are broad. Three student sizes, two benchmarks, and comparisons against both fixed and adaptive supervision schemes make it harder to read the gains as one lucky configuration, and the diversity analysis addresses a concern that would otherwise be obvious.

Critical weaknesses
  1. The paper states that the comparison uses one fixed seed without a sweep.

  2. A large number of competing methods are re-run inside one shared framework. That is the fair thing to do in principle, but it also means the baselines may not be at the strength reported in their own papers, so "best or tied-best" is being claimed against implementations whose tuning I cannot check.

Questions for authors
  1. How much do the per-category results move across seeds, particularly for the categories with the fewest tasks?

  2. Were the competing adaptive methods retuned within your framework, or run with their published settings?

  3. The method needs a critic. How much of the speed advantage over GRPO comes from the critic and how much from the weighting, once the two are compared at equal cost?

2 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer 9Pji [1/2]

We sincerely thank you for the encouraging review and for the careful questions on rigor. We have added the requested analyses to the revision, and we respond to each point below.

[W1 / Q1] On seed variance and per-category results

VALVE's advantage is stable across seeds: with three seeds, its overall ALFWorld success varies by only ±1.3 points, it ranks first under every seed, and it also leads in the categories with the fewest tasks. Our original single-seed protocol follows recent agentic RL work built on the same training stack [1][2], which fixes the environment and dataloader seeds (Appendix A.6). We have now repeated the comparison with three seeds on Qwen2.5-3B:

Method (Qwen2.5-3B, 3 seeds)PickLookCleanHeatCoolPick2ALFWorld AvgWebShop Acc
GRPO92.6 ± 2.280.6 ± 6.964.8 ± 4.482.1 ± 5.362.9 ± 3.551.7 ± 6.472.9 ± 2.054.2 ± 2.4
AgentOPSD98.1 ± 1.970.2 ± 7.385.3 ± 3.668.9 ± 5.882.7 ± 3.272.4 ± 6.183.6 ± 1.868.8 ± 2.1
VALVE97.3 ± 1.484.9 ± 6.594.6 ± 2.990.4 ± 5.184.1 ± 3.885.2 ± 5.690.9 ± 1.370.6 ± 1.8
  • Overall. VALVE leads the strongest baseline by 7.3 points on ALFWorld and 1.8 points on WebShop, consistent with the single-seed results in Table 1.
  • Smallest categories. On Look, Heat, and Pick2, VALVE leads the best competing method by 4.3, 8.3, and 12.8 points.

We have added these results to Appendix A.14 (Table 13) of the revision.

[W2 / Q2] On the baseline configurations

The competing adaptive methods were run with the method-specific settings published by their authors and were not retuned. All other settings follow the standard setup of recent work on multi-turn agentic RL and distillation [1][2][3][4]: the same students, benchmarks, splits, interaction budgets, and evaluation protocol, shared across all methods (Tables 5, 8, and 9).

Where the original papers use the same student and benchmark, our runs are close to their reported results:

MethodSettingReported in original paperOur run
SDAR [3]Qwen2.5-7B, ALFWorld86.785.9
AgentOPSD [4]Qwen2.5-7B, ALFWorld88.389.1
ATOD [5]Qwen3-1.7B, ALFWorld79.780.5

All baseline configurations will be released with our code.

VALVE receives no additional tuning either. Its only method-specific hyperparameter, the gate sharpness β\beta, is fixed to 10 for all students and both benchmarks. Across β∈{6,8,10,12,14}\beta \in \{6, 8, 10, 12, 14\}, ALFWorld success stays within 88.2–91.4%, above the strongest baseline (84.4%) at every value (Table 7), so the comparison does not depend on tuning β\beta.

References

[1] Group-in-Group Policy Optimization for LLM Agent Training. Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An. 2025.

[2] RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents. Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, Xiaolong Li. 2025.

[3] Self-Distilled Agentic Reinforcement Learning. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen. 2026.

[4] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning. Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang. 2026.

[5] ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks. Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, Peng Chen. 2026.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer 9Pji [2/2]

[Q3] On the source of the speed advantage over GRPO

Most of the speed advantage over GRPO comes from the teacher's dense supervision. The critic alone does not provide it, and the weighting contributes final performance, not speed. The existing results separate the three components:

Method (Qwen2.5-3B)ComponentsGPU-hours to teacher level (WebShop / ALFWorld)Final success (ALFWorld / WebShop)
GRPORL, no critic22.05 / 45.3173.4 / 53.9
PPORL with criticnot reached66.4 / 42.2
OPDteacher4.84 / 10.07plateaus at the teacher level (Fig. 5(a))
VALVEcritic + teacher + return-aware weight7.54 / 14.6091.4 / 71.1
  • Critic: PPO uses a critic without a teacher and ends below the teacher level (66.4% / 42.2% vs. 73.4% / 53.9%).
  • Teacher: OPD reaches the teacher level fastest.
  • Weighting: with the same critic and teacher, the return-aware weight raises final success from 74.2% / 57.8% (Uniform-weight OPD) to 91.4% / 71.1% (Table 1). VALVE reaches the teacher level later than OPD but keeps improving after OPD plateaus (Figure 5(a)).

At equal cost. The GPU-hours in Table 2 already include critic updates and frozen-teacher inference (Appendix A.9). With the budget at which VALVE reaches the teacher level (14.60 / 7.54 GPU-hours on ALFWorld / WebShop), GRPO has not yet reached half of that level on either benchmark (Table 2).

The critic is needed for a different reason: it provides a state-dependent baseline at every interaction step, which is what makes action-level return credit available. A group baseline such as GRPO's exists only at the initial state, because intermediate states have distinct histories. The gate itself reuses the critic's values and the teacher's log-probabilities and adds no rollouts or model evaluations (Section 3). This decomposition is included in Appendix A.9.


Your questions led directly to the multi-seed results and the cost decomposition, and the revised paper is clearer for them. Thank you once again, and please let us know if any further clarification would help.

Reviewer_LyFGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VALVE, which combines PPO with on-policy distillation and uses a critic to adjust the distillation weight at each token position. The weight depends on the agreement between the action-level RL advantage and the teacher’s relative support for the sampled token. The goal is to use teacher guidance while allowing the student to explore other valid solutions.

Critical strengths
  1. The motivation is clear. Disagreement with the teacher does not always mean the student is wrong, especially in tasks with multiple valid solutions.
  2. The method is simple to add to PPO with distillation, since it reuses the existing critic and teacher outputs.
  3. The results show improvements, especially on ALFWorld. The ablations also support using both RL advantage and teacher support to determine the weight.
Critical weaknesses
  1. My first concern is the potential bias introduced by the weighting mechanism, which changes the OPD objective. The gate depends on the estimated RL advantage, so critic errors may cause it to suppress useful teacher guidance or give more weight to less useful guidance. Even with an accurate advantage estimate, a positive advantage does not mean the student’s action is better than the teacher’s alternative. It is unclear whether agreement between these two signals reliably indicates when distillation is helpful. More discussion and experiments on this issue would be helpful.

  2. My second concern is whether aggressive weight adjustment affects training stability. When the sigmoid gate is sharp, changes in the estimated advantage may cause large changes in the distillation weight, especially early in training when the critic is less reliable. Could the authors discuss this issue and provide evidence of stability during training?

  3. The experimental scope is relatively limited, covering only ALFWorld and WebShop, with larger gains on ALFWorld. It is unclear whether the method generalizes to other tasks, such as math reasoning, coding, or more complex tool use. Experiments on a broader range of tasks would help support the method’s generality.

Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer LyFG [1/4]

We thank you for the careful reading and for recognizing the clear motivation, the simplicity of the method, and the ablations. We respond to each point below.

[W1 a] On the effect of the weighting on the OPD objective

The weighting only adjusts how strongly each token is supervised; it does not change the form of the OPD loss or the direction in which the loss moves the student.

Every token is still distilled toward the teacher. Each token keeps its full reverse-KL term toward the teacher (Eq. 3), which is minimized only when the student matches the teacher. The weight lies in (0, 1) and is computed under stop-gradient (Eq. 8), so it scales this term without changing its gradient direction; the PPO loss is unchanged (Eq. 9). The adaptive OPD baselines we compare (Annealed, KL-weighted, and entropy-weighted OPD; Section 4.1) adjust the same strength, and differ from VALVE only in the signal that sets it.

The adjustment follows the task return. Standard OPD weights every token equally, so it imposes the teacher's preference even where that preference lowers the return. At valid multi-solution disagreements, replacing the student action with the teacher action lowers the expected return by 7.65 points, whereas it raises the return by 7.01 points at conflict disagreements (Figure 2(c)). VALVE lowers the strength where the teacher's preference conflicts with the return estimate.

The adjustment improves both success and solution diversity. The direct control is Uniform-weight OPD with the same PPO objective (Table 1; Qwen2.5-3B, same teacher):

MethodALFWorld success (%)WebShop Acc (%)
Uniform-weight OPD74.257.8
VALVE91.471.1

The adjustment also preserves more valid solutions: VALVE keeps two or more verified solutions on 42% of multi-solution problems, compared with 17% for OPD (Figure 4(b)).

[W1 b] On critic errors

Critic errors mainly occur early in training; their effect on the gate is limited, and the critic becomes reliable within about 30 steps.

The effect of critic errors is limited. The gate combines the critic's advantage with the teacher's support, so it does not rely on the critic alone. A wrong weight only changes how strongly the student follows the teacher at that token; it never pushes the student away from the teacher.

The critic becomes reliable quickly, without additional signals. The critic is trained on the real outcome rewards from the environment, collected in the same rollouts used for policy training, so each batch moves it toward the true values. The teacher's dense supervision also raises the student's success early in training, so the critic soon observes both successful and failed rollouts to learn from. A more accurate critic in turn improves the allocation of guidance. This requires no additional rollouts, annotations, or model evaluations.

Figure 5 shows that this happens within about 30 steps.

  • Training curve (Figure 5(a)). VALVE trails OPD only in the first steps, catches up with OPD by step 30, and then keeps improving after OPD plateaus, reaching a higher final success rate.
  • Weight distribution (Figure 5(b)). The early weights are polarized near 0 and 1. From about step 30, the distribution stabilizes into a broad range centered around 0.5, while the 10th–90th percentiles still keep token-level differences.

Both changes occur around step 30. This reflects a key strength of VALVE: the critic and the gate improve each other during training, so teacher guidance is allocated more accurately as training proceeds. This is why VALVE keeps improving after OPD plateaus.

Overall, VALVE does not suppress useful teacher guidance or give more weight to less useful guidance. Two results support this:

  • Case study (Figure 6). On three successful WebShop routes, VALVE keeps high probability on the decisive actions of all of them. On the teacher-preferred route, it keeps the final purchase action at 0.95, whereas GRPO and OPD drop to 0.10 and below 0.01, so the teacher's guidance on this route is retained. On the alternative and self-correcting routes, it also keeps the routes that GRPO and OPD nearly abandon.
  • Step-level analysis. Using the action-replacement protocol of Appendix A.3, we compared the returns of the teacher's and the student's actions at teacher–student disagreement steps of VALVE. Among steps that receive a high weight, the teacher's action has the higher return in 78.4% of cases by step 150, compared with 47.6% among all disagreement steps. The gate thus gives more weight where the teacher's guidance is useful.
AuthorsOpenReview ↗

Official Rebuttal to Reviewer LyFG [2/4]

[W1 c] On using positive advantage to weaken teacher guidance

VALVE does not assume that a positive advantage means the student's action is better than the teacher's. In agentic tasks, a student action that differs from the teacher's can still lead to task success. For example, in Figure 1 the teacher selects the color first, while the student selects the size first. The student's action still completes the task and receives a positive advantage: it is another valid solution, and in some tasks such a route can even take fewer steps than the teacher's. Standard OPD, however, treats the teacher's route as the only correct one and pulls the student back to it, which is exactly what causes multi-solution collapse.

VALVE therefore judges each step from two angles: whether the action leads to a good outcome (return credit) and whether it agrees with the teacher (teacher support). For an action with a positive advantage:

  • If it agrees with the teacher, both signals point the same way, and VALVE raises the OPD weight. Following the teacher here speeds up training.
  • If it differs from the teacher, the action reaches a good outcome by another route, which typically indicates a valid alternative solution. VALVE lowers the OPD weight, so the update at this step is driven mainly by the RL objective. Such steps are common: valid multi-solution disagreements account for 25.42% of all actions, and at these steps the student's action has a 7.65-point higher expected return than the teacher's (Figure 2(c)).

Even if the teacher's action is better at such a step, the student's action has still led to a good outcome, and the teacher's signal is kept with a smaller weight, so the student continues to learn from the teacher. The step-level analysis in W1 b further shows that the gate gives higher weights where the teacher's action is better (78.4% of high-weight disagreement steps vs. 47.6% of all disagreement steps). VALVE thus does not need the student to be better than the teacher; it only needs to tell when the student has found another valid route.

[W1 d] On whether agreement between the two signals is reliable

Yes. Agreement between the two signals reliably indicates where distillation helps; we verify this directly against step-level returns and through two controlled experiments. The RL signal tells whether the sampled action led to a higher or lower return than expected, and the teacher signal tells whether the teacher supports the sampled token more or less than the student does:

Return creditTeacher supportOPD weightReason
++higherthe teacher and the return both support this token
+−lowerthe student's action may be a valid alternative
−+lowerthe teacher supports a token that led to a low return
−−higherthe teacher corrects a token that led to a low return
  1. The weight matches step-level returns. At VALVE checkpoints from training steps 10, 50, and 150, we took the disagreement steps with high weights (g>0.5g>0.5) and compared the Monte Carlo returns of the teacher's and the student's actions (Appendix A.3). The teacher's action has the higher return in 56.3% / 71.9% / 78.4% of these steps at steps 10 / 50 / 150, compared with 47.6% among all disagreement steps. The gate thus gives more weight where following the teacher improves the return, increasingly so as training proceeds. We have added this analysis to Appendix A.3.

  2. Reversing the rule removes the benefit. We replaced the gate with σ(−βM)\sigma(-\beta M), which gives higher weights where the two signals disagree:

Gate (Qwen2.5-3B)ALFWorld success (%)WebShop Acc (%)
Uniform-weight OPD74.257.8
Reversed gate σ(−βM)\sigma(-\beta M)74.255.4
VALVE91.471.1

The reversed gate loses 17.2 / 15.7 points and falls to or below the level of Uniform-weight OPD.

  1. Both signals are needed. Removing the return credit or the teacher support from the gate lowers ALFWorld success from 91.4% to 81.2% and 83.5%, respectively (Table 3).
AuthorsOpenReview ↗

Official Rebuttal to Reviewer LyFG [3/4]

[W2] On training stability under a sharp gate

VALVE trains stably throughout, including the early phase when the critic is least reliable. Even if the gate is made fully sharp, so that every weight becomes either 0 or 1, training does not collapse.

Errors in the advantage estimate cause only small changes in the weight. Teacher support is clipped at the magnitude of the advantage (Eq. 14), so the gate input stays small whenever the advantage is small. With β=10\beta=10, if an estimated advantage of 0.1 flips to −0.1, the weight only moves between 0.475 and 0.525. A large change in the weight requires a large advantage, which the critic is unlikely to produce by error alone.

Even when the weight changes, its effect is bounded. The weight only scales the OPD term within (0, 1) under stop-gradient, so the OPD gradient at a token never exceeds that of Uniform-weight OPD. PPO clipping and gradient clipping are unchanged (Appendix A.10).

The training dynamics confirm this stability (Figure 5).

  • Early phase. VALVE trails OPD in the first steps, but it is already clearly above GRPO, and its success rises steadily without drops (Figure 5(a)).
  • Around step 30. The weights leave the 0/1 extremes and settle into a stable distribution centered around 0.5 (Figure 5(b)). At the same time, VALVE catches up with OPD and keeps improving after OPD plateaus (Figure 5(a)).
  • Across seeds. Final ALFWorld success varies by ±1.3 points over three seeds.

Even if the gate becomes a 0/1 switch, we observe no collapse. The parameter β\beta controls how fast the weight moves from 0 to 1. As β→∞\beta \to \infty, the gate becomes a 0/1 switch: a token's weight is 1 when the two signals agree and 0 when they conflict. Each token then falls back to one of two standard objectives, Uniform-weight OPD (weight 1) or PPO (weight 0), and both train stably on their own. The sharpest gate therefore only switches each token between two stable objectives. Results under different sharpness:

Weight setting (Qwen2.5-3B)ALFWorld success (%)WebShop Acc (%)
PPO (weight 0 everywhere)66.442.2
Uniform-weight OPD (weight 1 everywhere)74.257.8
0/1 gate (β→∞\beta \to \infty)83.562.1
β=14\beta=1489.170.3
β=10\beta=10 (default)91.471.1
β=6\beta=690.671.8

The 0/1 gate trains without collapse and still outperforms both of its endpoints. For β\beta from 6 to 14, results vary by at most 3.2 / 3.0 points (Table 7). A sharper gate thus lowers the gain but does not destabilize training.

[W3] On the experimental scope

Qwen2.5/Qwen3 students on ALFWorld and WebShop form the standard setting in which OPD and RL are studied for multi-turn agents, and we adopt it for a direct and fair comparison with prior methods.

Most recent work in this line reports its main results in this setting, including OPD methods for agents (SDAR [1], AgentOPSD [2], RetireOPD [3], ATOD [4]) and agentic RL methods (GiGPO [5], RLVMR [6]). Sharing this setting lets us compare 12 methods with their authors' settings under one protocol (Table 1).

References

[1] Self-Distilled Agentic Reinforcement Learning. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen. 2026.

[2] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning. Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang. 2026.

[3] RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning. Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen. 2026.

[4] ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks. Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, Peng Chen. 2026.

[5] Group-in-Group Policy Optimization for LLM Agent Training. Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An. 2025.

[6] RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents. Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, Xiaolong Li. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.