- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies OPD for multi-turn agents and proposes State-OPD. The student retains its full interaction history, while the teacher receives the task, executed actions, and observations without the student’s previous reasoning. The method also samples additional complete reasoning and action responses at high-entropy states. Experiments on ALFWorld, ScienceWorld, and WebShop report higher average success rates, with the clearest gains for the 4B student on ALFWorld.
Critical strengths
-
The paper studies a practical problem: accumulated student reasoning can influence the teacher’s supervision during multi-turn interaction. Separating the interaction record from the student’s interpretation provides a clear way to investigate this issue.
-
The method is simple, and the two-component ablation helps identify where the gains come from. Changing the teacher’s context accounts for most of the ALFWorld improvement, with additional response sampling providing a smaller further gain.
-
The 4B results on ALFWorld are strong, with substantial improvements on both seen and unseen environments. The reduction in persistent failed-action loops also provides useful behavioral evidence beyond the final success rates.
Critical weaknesses
-
The central design retains previous-turn thinking for the student while removing it from the teacher’s context. The key comparison with removing previous-turn thinking from both models is missing. Historical thinking may interfere with both models, so the current results do not establish the value of this asymmetric setup over simply dropping it throughout the pipeline.
-
On ALFWorld, State-OPD produces substantially longer thinking traces, reaching 638 tokens per turn versus 336 for Vanilla OPD. Matching the number of supervised responses leaves the contribution and cost of this additional reasoning unclear.
-
The evidence for broader applicability is limited. The strongest gains occur with the 4B student on ALFWorld, which also hosts the detailed diagnostics and ablations. ScienceWorld shows little consistent improvement, and the 1.7B model’s average lead over the strongest baseline is only 0.33 percentage points.
-
The reported confidence intervals capture evaluation uncertainty for fixed checkpoints, leaving training variability unmeasured. This makes the smaller gains, including those attributed to additional response sampling, difficult to assess reliably.
Questions for authors
-
Do you have results for OPD with previous-turn thinking removed from both the student and teacher during training and evaluation? How does this compare with the asymmetric setup used in State-OPD?
-
Do you have comparisons under matched thinking-token budgets? Please also summarize the total generated tokens, teacher-scored tokens, and training time to clarify the contribution and cost of the longer reasoning traces.
-
What explains the limited gains on ScienceWorld and with the 1.7B student?
-
Do you have additional training seeds for the main ALFWorld comparison and the component ablations? How much do the overall improvement and the additional sampling gain vary across runs?
5 comments
Rebuttal (2/2)
W3 & Q3. Gains with the 1.7B student and on ScienceWorld
We thank the reviewer for pointing this out. Off-policy demonstrations can be especially valuable to the weaker 1.7B student because they supply behaviors that its own rollouts rarely produce. A stronger 4B student can generate more useful behaviors and benefit more from improved feedback on its own samples. Guided-OPD combines reverse KL on student turns with forward KL on teacher turns, gradually reducing teacher intervention to zero, and reports larger gains for smaller students [1].
ScienceWorld presents the same sampling difficulty: progress requires prerequisite action sequences that student rollouts may rarely contain. Teacher-generated turns can supply these actions and reach subsequent states. This is limitation of the on-policy setting relying on student-generated experience, rather than a weakness introduced by State-OPD.
Our contribution is to improve supervision and response coverage within this on-policy setting. However, State-OPD also achieves a slightly higher macro-average than Guided-OPD at 1.7B without teacher-generated turns. At 4B, its advantage grows and extends to WebShop as well as ALFWorld.
W4 & Q4: Training variability
We thank the reviewer for requesting additional training seeds. Table 8 quantifies task-level evaluation uncertainty for the fixed seed-42 checkpoints. To assess variability across training runs, we now report the Qwen3-4B ALFWorld comparison across seeds 42, 43, and 44, using step-200 checkpoints and the same evaluation protocol. Entries are success rates as mean ± sample standard deviation across seeds 42, 43, and 44; Avg. is the mean of Seen and Unseen.
| Method | Seen | Unseen | Avg. |
|---|---|---|---|
| State-OPD | 43.10 ± 0.68 | 46.46 ± 1.13 | 44.78 ± 0.25 |
| w/o State-Centered Response Allocation | 42.14 ± 1.64 | 43.66 ± 1.13 | 42.90 ± 1.28 |
| w/o State-Grounded Supervision | 34.46 ± 1.42 | 37.69 ± 1.35 | 36.08 ± 0.64 |
| Vanilla OPD | 30.83 ± 0.63 | 34.27 ± 1.24 | 32.55 ± 0.31 |
State-OPD outperforms Vanilla OPD in every seed on both splits. The per-seed gains average 12.26 ± 0.21 percentage points on Seen and 12.19 ± 0.47 on Unseen. The approximately 12-point improvement is therefore consistent across the three training runs, complementing the task-level uncertainty analysis in Table 8.
The component ablations show that removing either component lowers the Seen/Unseen average in every training seed. Removing state-grounded supervision reduces Avg. by 8.70 ± 0.51 percentage points, while removing state-centered response allocation reduces it by 1.88 ± 1.03 points. For response allocation, the gains are 0.95 ± 2.15 points on Seen and 2.80 ± 0.19 on Unseen. These additional runs support the contribution of both components to average success rate, consistent with the ablation results in Table 2.
We are happy to address any further concerns. If all your issues are addressed, we would highly appreciate it if you consider increasing the score.
Best!
References
[1] Li, Gengsheng, et al. On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents. arXiv:2606.15912, 2026.
Request to review our responses
Dear Reviewer HZ9u we request you to please review our responses and would love to further engage with you in a constructive discussion if you have any clarifying questions. If all your concerns are addressed we would highly appreciate if you would consider increasing your score.
Rebuttal (1/2)
We thank the reviewer for the thoughtful feedback. We respond point-by-point below.
W1 & Q1. Removing previous-turn thinking from both the student and teacher
We thank the reviewer for suggesting this comparison. We have evaluated the proposed symmetric-removal variant across all three benchmarks. Previous-turn thinking is removed from both the student and teacher contexts during training, and from the student context during evaluation. Action–observation history is retained, and current-turn thinking remains enabled and supervised during training.
State-OPD achieves higher success than symmetric removal in all five settings at both student sizes. These results show that simply removing previous-turn thinking from both models does not recover State-OPD’s performance.
W2 & Q2. Thinking budgets and token cost
We thank the reviewer for suggesting matched thinking-token comparisons. Both methods use the same 32,768-token cumulative transcript budget for responses and observations, without a separate per-turn cap. The reported 638/336 thinking tokens are observed response lengths.
At step 200, OPD uses 109.60M thinking and 114.55M generated tokens, while State-OPD uses 164.57M and 169.12M, respectively. We count tokens in the student responses used for training. Each of these tokens receives a teacher score for distillation, so the generated and teacher-scored response-token totals are equal.
The Figure 8 compares the same two ALFWorld runs as Figure 4 against cumulative thinking and total generated tokens. The complete curves show higher Seen and Unseen success over most of both shared budget ranges. The table provides one directly evaluated example at approximately 99M thinking tokens, with nearly equal generated-token consumption. Please see Appendix G.4 for the full curves and counting details.