- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript propose a novel LLM RL algorithm of VALVE, which combines PPO with on-policy distillation for interactive agents. VALVE uses the product of a raw action advantage and a teacher–student token log-probability difference to gate the distillation loss. This is intended to retain useful teacher guidance without suppressing successful student exploration. Experiments on ALFWorld and WebShop cover three policy models, including Qwen2.5-3B/7B and Qwen3-1.7B, with additional analyses of successful-solution diversity, gate components, and compute required to reach target performance.
Critical strengths
-
A novel LLM RL algorithm of VALVE is proposed and integrate PPO with OPD to balance exploration and exploitation.
-
The gains are substantial across both environments rather than confined to one configuration. For example, ALFWorld success reaches , , and for the three students. The gate ablations also favor the complete method over unweighted distillation and either single-signal variant, supporting the usefulness of the combined design.
-
The diversity analysis goes beyond average success. On 111 multi-solution tasks, 42% yield at least two distinct successful solutions, versus 17% for OPD and 13% for GRPO. Measuring different successful behaviors gives useful evidence for the stated exploration objective.
Critical weaknesses
-
One of the major concerns for me is the rationale behind VALVE, which is an RL method but requires a teacher model. In modern LLM post-training workflows, we typically use RL to push specialized performance and then use knowledge distillation, such as OPD, to combine these capabilities into one model. VALVE cannot readily replace the RL stage because it requires an existing teacher, but it also seems less suitable for the distillation stage due to its additional compute burden. Its practical role in this workflow is therefore unclear.
-
RG-OPD is already cited, but I could not find a direct comparison. Since it also gates distillation using reward and teacher–student agreement, this comparison matters to the novelty of VALVE. The strong gains over ordinary RL establish utility, but do not isolate this distinction. How much does the action-level gate add under the same teacher and rollout budget?
-
The models used, such as Qwen2.5-3B/7B and Qwen3-1.7B, are small and relatively dated, with limited agentic pretraining or mid-training. Using the Qwen3.5 series, which already incorporates agentic knowledge, would be more convincing. Moreover, ALFWorld (2021) and WebShop (2022) are also dated benchmarks. Evaluation on benchmarks introduced in the past two years would be more meaningful.
Questions for authors
Please see Critical Weaknesses.
6 comments
Official Rebuttal to Reviewer HqNB [2/4]
[W1 b] On the RL-then-OPD workflow (continued)
Our OPD baseline in Figures 4 and 5 directly instantiates the RL-then-OPD workflow: the teacher is Qwen2.5-3B trained with GRPO for 150 steps, and it is distilled into a student of the same size with standard OPD (Appendix A.6). With the same teacher, on 111 multi-solution ALFWorld problems with eight samples each (Figure 4):
| Method (same GRPO teacher) | Problems with ≥2 distinct verified solutions | pass@8 (%) |
|---|---|---|
| OPD (RL → OPD) | 17% | 64.0 |
| VALVE | 42% | 94.6 |
VALVE outperforms OPD at every evaluated (Figure 4(a)).
This collapse is common and costly in agentic tasks: 25.42% of all actions are valid multi-solution disagreements, and replacing the student action with the teacher action at these steps lowers the expected return by 7.65 points, whereas it raises the return by 7.01 points at conflict disagreements (Figure 2(c)).
Strengthening the teacher with RL therefore does not by itself prevent this collapse. VALVE addresses it within the transfer step: it allocates teacher supervision according to return information, so that valid alternatives are preserved while errors are still corrected.
[W1 c] On the practical role and compute cost
In this workflow, VALVE is used at the distillation stage, where it is better suited than standard OPD: with the same teacher, it preserves the student's valid alternatives and reaches higher success. It reuses the task reward from the RL stage and requires no additional annotation. Its advantage over OPD grows when a task admits several valid solutions, which is common in multi-turn agentic tasks (Section 2).
On the RL stage. We agree that VALVE does not replace the RL stage that produces the teacher. VALVE is also robust to the teacher's quality (Section 4.3, Figure 4(c)):
| Teacher (GRPO steps) | Teacher success (%) | VALVE student (%) |
|---|---|---|
| 50 | 34 | 68 |
| 100 | 64 | 88 |
| 150 | 73 | 91 |
Even a teacher with only 34% success yields a 68% student, and a 64% teacher yields 88%, close to the 91% obtained with the mature teacher.
On the distillation stage. With the same teacher, VALVE improves on standard OPD in two ways (Qwen2.5-3B):
- More valid solutions. VALVE keeps two or more verified solutions on 42% of multi-solution problems, compared with 17% for OPD, and raises pass@8 from 64.0% to 94.6% (Figure 4, W1 b).
- Continued improvement. OPD stops improving once it reaches the teacher's level, whereas VALVE keeps improving (Figure 5(a)).
Adding the RL objective alone does not account for this gain: with the same PPO objective, Uniform-weight OPD reaches 74.2% / 57.8% on ALFWorld / WebShop, compared with 91.4% / 71.1% for VALVE (Table 1).
On compute. The additional cost is moderate. Per step, VALVE adds a PPO update with its critic to OPD; the gate reuses the critic's values and the teacher's log-probabilities and adds no rollouts or model evaluations (Section 3). GPU-hours to reach the teacher's success rate (Table 2):
| Method | WebShop | ALFWorld |
|---|---|---|
| OPD | 4.84 | 10.07 |
| VALVE | 7.54 | 14.60 |
| GRPO | 22.05 | 45.31 |
VALVE needs about 1.5× the GPU-hours of OPD, which then plateaus at the teacher's level, and 66–68% fewer than GRPO.
VALVE thus takes the place of standard OPD in the distillation stage at a moderate extra cost, and among existing OPD+RL methods it achieves the best or tied-best success in all settings (Table 1).
[W2 a] On the direct comparison with RG-OPD
We have implemented RG-OPD following its paper [9]. For a controlled comparison, its gate runs in the same training framework as VALVE, with the same student, teacher, rollout budget, and RL objective. VALVE outperforms RG-OPD by 18.8 points on ALFWorld and 11.8 points on WebShop.
| Method (student: Qwen3-1.7B; teacher: Qwen3-4B GRPO) | ALFWorld success (%) | WebShop Acc (%) |
|---|---|---|
| GRPO | 50.0 | 54.7 |
| Uniform-weight OPD | 75.8 | 50.0 |
| KL-weighted OPD | 78.1 | 50.8 |
| ATOD | 80.5 | 42.1 |
| SDAR | 53.9 | 58.6 |
| RG-OPD (new) | 73.4 | 52.3 |
| VALVE | 92.2 | 64.1 |
In this multi-turn setting, RG-OPD performs close to Uniform-weight OPD (73.4% / 52.3% vs. 75.8% / 50.0%). The original submission did not include it because RG-OPD was proposed and evaluated for single-turn math, coding, and instruction-following tasks [9]. These results have been added to Appendix A.13 of the revision.
References
[9] Reward-Gated On-Policy Distillation. Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski. 2026.
Official Rebuttal to Reviewer HqNB [3/4]
[W2 b] On the difference between VALVE and RG-OPD
RG-OPD cannot preserve valid alternative solutions in multi-turn agentic tasks, because it makes a single keep-or-drop decision for an entire trajectory. It compares the trajectory-level advantage with the teacher–student likelihood gap summed over the whole trajectory, and then keeps or discards the distillation loss of all tokens in that trajectory together [9].
An agentic trajectory, however, spans up to 30 interaction steps and mixes two kinds of steps: steps where the student should still follow the teacher, and valid branches that the teacher does not prefer (e.g., selecting the size before the color in Figure 1). To quantify this, we additionally applied the action-replacement protocol of Appendix A.3 to trajectories with teacher–student disagreements: 58.6% of them contain both a step where the teacher's action yields a higher expected return and a step where the student's action does. We have added this analysis to Appendix A.3 of the revision. For such a trajectory, keeping the loss pulls the valid branch back to the teacher, and dropping it discards the teacher's guidance at every other step.
VALVE decides for each token: the PPO critic assigns return credit to each interaction action, so within one trajectory, teacher guidance is strengthened at some steps and weakened at others (Sections 3.2–3.3). The consequence is visible on the 111 multi-solution ALFWorld problems (Qwen2.5-3B, eight samples each):
| Method | Problems with ≥2 distinct verified solutions | pass@8 (%) |
|---|---|---|
| GRPO | 13% | 63.1 |
| OPD | 17% | 64.0 |
| RG-OPD (new) | 18% | 65.8 |
| VALVE | 42% | 94.6 |
RG-OPD does not resolve the multi-solution collapse: its results are on par with OPD, whereas VALVE more than doubles the share of problems with multiple valid solutions and raises pass@8 by about 30 points. We have added RG-OPD to the method comparison in Table 4.
[W2 c] On how much the action-level gate adds
Under the same teacher and rollout budget, VALVE improves over RG-OPD by 18.8 points on ALFWorld and 11.8 points on WebShop, and most of this gain comes from VALVE's action-level decision. The critic estimates the advantage of each action, so teacher guidance is strengthened or weakened step by step. RG-OPD decides once per trajectory, so it cannot separate the steps that should follow the teacher from the valid branches in the same trajectory.
RG-OPD and VALVE differ in two respects:
- What the decision is based on. RG-OPD uses one advantage for the whole trajectory; VALVE uses the advantage of each action, estimated by the critic.
- How the decision is applied. RG-OPD keeps or drops the distillation loss; VALVE scales it by a weight between 0 and 1 (Eq. 8).
To measure each part, we swap them one at a time, which gives two intermediate variants. All four runs share the student (Qwen3-1.7B), teacher (Qwen3-4B GRPO), rollout budget, RL objective, and evaluation protocol; only these two choices change.
| Variant | Decision based on | Decision applied as | ALFWorld success (%) | WebShop Acc (%) |
|---|---|---|---|---|
| RG-OPD | trajectory advantage | keep or drop | 73.4 | 52.3 |
| RG-OPD with weight | trajectory advantage | weight in (0, 1) | 77.3 | 55.1 |
| VALVE with keep/drop | action advantage | keep or drop | 89.8 | 62.9 |
| VALVE | action advantage | weight in (0, 1) | 92.2 | 64.1 |
- VALVE's action-level decision is the main source of the gain. Replacing the trajectory advantage with the action advantage adds 16.4 / 10.6 points under keep-or-drop and 14.9 / 9.0 points under weighting.
- Weighting adds a smaller gain. Replacing keep-or-drop with a weight adds 2.4–3.9 / 1.2–2.8 points.
Adding a weight to RG-OPD does not close the gap (77.3% vs. 92.2% on ALFWorld).
On novelty. This comparison isolates what distinguishes VALVE from RG-OPD: VALVE uses the critic's action-level return credit to decide, step by step, where teacher supervision is applied. This component accounts for most of the gain over RG-OPD, and it is what allows VALVE to preserve valid alternatives in multi-turn tasks (W2 b). To our knowledge, no prior OPD method allocates teacher supervision with action-level return credit (Table 4).
References
[9] Reward-Gated On-Policy Distillation. Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski. 2026.
Official Rebuttal to Reviewer HqNB [1/4]
We thank you for the detailed comments and for recognizing the substantial gains and the diversity analysis. We respond to each point below.
VALVE is an OPD+RL method, and using a teacher is a shared premise of this paradigm, not an additional requirement introduced by VALVE. Combining OPD with RL is an actively studied paradigm [1][2][3][4][5], and all of these methods use a teacher for OPD and environment rewards for RL. VALVE follows the same setup: in a single training stage, the student is optimized with the PPO objective on environment rewards and with an OPD loss toward the teacher (Eq. 9).
VALVE focuses on the following question: given a teacher, where should teacher supervision be applied? Our analysis (Section 2) identifies a problem that is pronounced in multi-turn agentic tasks: because these tasks admit multiple valid solution paths, a student's deviation from the teacher is often a valid alternative and not an error, and applying teacher supervision equally at every step suppresses such alternatives. This problem is central to distilling agents: suppressing valid routes both narrows the student's behavior and lowers its success (Section 4.3). VALVE addresses it by using the critic's return estimates to decide where teacher supervision is applied. We have made this positioning explicit in the revised introduction.
Merging specialized capabilities into one model is an important use of OPD, but it is not the problem studied in this paper. VALVE targets another use of OPD in the same workflow: transferring a teacher's capability to a student in multi-turn agentic tasks, as in strong-to-weak distillation (e.g., Qwen3 [6]).
The problem in this setting is multi-solution collapse. OPD supervises the student only through the teacher's distribution, so it cannot tell whether a deviation from the teacher is an error or a valid alternative route, and it pulls both back to the teacher's route. An RL-trained teacher does not remove this problem, because the problem arises during distillation: RL makes the teacher stronger, but its preferred route is still only one of several valid routes. This is consistent with prior observations that OPD reduces policy entropy and behavioral diversity [4][7][8].
References
[1] KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, Fei Mi. 2025.
[2] Self-Distilled Agentic Reinforcement Learning. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen. 2026.
[3] AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning. Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang. 2026.
[4] Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL. Qizhen Lan, Xi Xiao, Xiangchen Guan, Mengchen Fan, Moule Lin, Jung Im Choi, Lijing Zhu. 2026.
[5] RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning. Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen. 2026.
[6] Qwen3 Technical Report. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. (Qwen Team). 2025.
[7] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, Ning Ding. 2026.
[8] Entropy-Aware On-Policy Distillation of Language Models. Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, Kimin Lee. 2026.