acceptodds
Under review as a conference paper at ICLR 2027

Actions Speak Louder than Thoughts: State-Grounded On-Policy Distillation for Multi-Turn Agents

14 views

Abstract

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. In this work, we investigate agentic OPD through interaction states, distinguishing the objective task and executed action–observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning–action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. Building on these findings, we introduce State-Grounded On-Policy Distillation (State-OPD), a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning–action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HZ9uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies OPD for multi-turn agents and proposes State-OPD. The student retains its full interaction history, while the teacher receives the task, executed actions, and observations without the student’s previous reasoning. The method also samples additional complete reasoning and action responses at high-entropy states. Experiments on ALFWorld, ScienceWorld, and WebShop report higher average success rates, with the clearest gains for the 4B student on ALFWorld.

Critical strengths
  1. The paper studies a practical problem: accumulated student reasoning can influence the teacher’s supervision during multi-turn interaction. Separating the interaction record from the student’s interpretation provides a clear way to investigate this issue.

  2. The method is simple, and the two-component ablation helps identify where the gains come from. Changing the teacher’s context accounts for most of the ALFWorld improvement, with additional response sampling providing a smaller further gain.

  3. The 4B results on ALFWorld are strong, with substantial improvements on both seen and unseen environments. The reduction in persistent failed-action loops also provides useful behavioral evidence beyond the final success rates.

Critical weaknesses
  1. The central design retains previous-turn thinking for the student while removing it from the teacher’s context. The key comparison with removing previous-turn thinking from both models is missing. Historical thinking may interfere with both models, so the current results do not establish the value of this asymmetric setup over simply dropping it throughout the pipeline.

  2. On ALFWorld, State-OPD produces substantially longer thinking traces, reaching 638 tokens per turn versus 336 for Vanilla OPD. Matching the number of supervised responses leaves the contribution and cost of this additional reasoning unclear.

  3. The evidence for broader applicability is limited. The strongest gains occur with the 4B student on ALFWorld, which also hosts the detailed diagnostics and ablations. ScienceWorld shows little consistent improvement, and the 1.7B model’s average lead over the strongest baseline is only 0.33 percentage points.

  4. The reported confidence intervals capture evaluation uncertainty for fixed checkpoints, leaving training variability unmeasured. This makes the smaller gains, including those attributed to additional response sampling, difficult to assess reliably.

Questions for authors
  1. Do you have results for OPD with previous-turn thinking removed from both the student and teacher during training and evaluation? How does this compare with the asymmetric setup used in State-OPD?

  2. Do you have comparisons under matched thinking-token budgets? Please also summarize the total generated tokens, teacher-scored tokens, and training time to clarify the contribution and cost of the longer reasoning traces.

  3. What explains the limited gains on ScienceWorld and with the 1.7B student?

  4. Do you have additional training seeds for the main ALFWorld comparison and the component ablations? How much do the overall improvement and the additional sampling gain vary across runs?

5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the thoughtful feedback. We respond point-by-point below.

W1 & Q1. Removing previous-turn thinking from both the student and teacher

We thank the reviewer for suggesting this comparison. We have evaluated the proposed symmetric-removal variant across all three benchmarks. Previous-turn thinking is removed from both the student and teacher contexts during training, and from the student context during evaluation. Action–observation history is retained, and current-turn thinking remains enabled and supervised during training.

StudentMethodALFWorld SeenALFWorld UnseenScienceWorld SeenScienceWorld UnseenWebShop
Qwen3-1.7BSymmetric removal30.8932.658.1111.5111.95
Qwen3-1.7BState-OPD31.2536.0111.1813.1712.20
Qwen3-4BSymmetric removal42.1439.5521.7121.8716.80
Qwen3-4BState-OPD43.3945.7124.7825.1919.65

State-OPD achieves higher success than symmetric removal in all five settings at both student sizes. These results show that simply removing previous-turn thinking from both models does not recover State-OPD’s performance.


W2 & Q2. Thinking budgets and token cost

We thank the reviewer for suggesting matched thinking-token comparisons. Both methods use the same 32,768-token cumulative transcript budget for responses and observations, without a separate per-turn cap. The reported 638/336 thinking tokens are observed response lengths.

At step 200, OPD uses 109.60M thinking and 114.55M generated tokens, while State-OPD uses 164.57M and 169.12M, respectively. We count tokens in the student responses used for training. Each of these tokens receives a teacher score for distillation, so the generated and teacher-scored response-token totals are equal.

The Figure 8 compares the same two ALFWorld runs as Figure 4 against cumulative thinking and total generated tokens. The complete curves show higher Seen and Unseen success over most of both shared budget ranges. The table provides one directly evaluated example at approximately 99M thinking tokens, with nearly equal generated-token consumption. Please see Appendix G.4 for the full curves and counting details.

Method (step)Thinking tokens (M)Generated tokens (M)Seen SRUnseen SR
OPD (180)98.31102.7833.3933.96
State-OPD (120)99.23102.0240.0042.35
AuthorsOpenReview ↗

Rebuttal (2/2)

W3 & Q3. Gains with the 1.7B student and on ScienceWorld

We thank the reviewer for pointing this out. Off-policy demonstrations can be especially valuable to the weaker 1.7B student because they supply behaviors that its own rollouts rarely produce. A stronger 4B student can generate more useful behaviors and benefit more from improved feedback on its own samples. Guided-OPD combines reverse KL on student turns with forward KL on teacher turns, gradually reducing teacher intervention to zero, and reports larger gains for smaller students [1].

ScienceWorld presents the same sampling difficulty: progress requires prerequisite action sequences that student rollouts may rarely contain. Teacher-generated turns can supply these actions and reach subsequent states. This is limitation of the on-policy setting relying on student-generated experience, rather than a weakness introduced by State-OPD.

Our contribution is to improve supervision and response coverage within this on-policy setting. However, State-OPD also achieves a slightly higher macro-average than Guided-OPD at 1.7B without teacher-generated turns. At 4B, its advantage grows and extends to WebShop as well as ALFWorld.


W4 & Q4: Training variability

We thank the reviewer for requesting additional training seeds. Table 8 quantifies task-level evaluation uncertainty for the fixed seed-42 checkpoints. To assess variability across training runs, we now report the Qwen3-4B ALFWorld comparison across seeds 42, 43, and 44, using step-200 checkpoints and the same evaluation protocol. Entries are success rates as mean ± sample standard deviation across seeds 42, 43, and 44; Avg. is the mean of Seen and Unseen.

MethodSeenUnseenAvg.
State-OPD43.10 ± 0.6846.46 ± 1.1344.78 ± 0.25
w/o State-Centered Response Allocation42.14 ± 1.6443.66 ± 1.1342.90 ± 1.28
w/o State-Grounded Supervision34.46 ± 1.4237.69 ± 1.3536.08 ± 0.64
Vanilla OPD30.83 ± 0.6334.27 ± 1.2432.55 ± 0.31

State-OPD outperforms Vanilla OPD in every seed on both splits. The per-seed gains average 12.26 ± 0.21 percentage points on Seen and 12.19 ± 0.47 on Unseen. The approximately 12-point improvement is therefore consistent across the three training runs, complementing the task-level uncertainty analysis in Table 8.

The component ablations show that removing either component lowers the Seen/Unseen average in every training seed. Removing state-grounded supervision reduces Avg. by 8.70 ± 0.51 percentage points, while removing state-centered response allocation reduces it by 1.88 ± 1.03 points. For response allocation, the gains are 0.95 ± 2.15 points on Seen and 2.80 ± 0.19 on Unseen. These additional runs support the contribution of both components to average success rate, consistent with the ablation results in Table 2.


We are happy to address any further concerns. If all your issues are addressed, we would highly appreciate it if you consider increasing the score.

Best!

References

[1] Li, Gengsheng, et al. On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents. arXiv:2606.15912, 2026.

AuthorsOpenReview ↗

Request to review our responses

Dear Reviewer HZ9u we request you to please review our responses and would love to further engage with you in a constructive discussion if you have any clarifying questions. If all your concerns are addressed we would highly appreciate if you would consider increasing your score.

Reviewer_3WLROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates on-policy distillation in multi-turn agent tasks, analyzing the impact of prior reasoning on teacher supervision. The authors propose State-OPD: a framework that grounds teacher supervision in interaction and reallocates the sampling budget to generate multiple complete reasoning-actlon responses at states with high response entropy, while retaining full-history student generation. The authors evaluating their method on three datasets against multiple baselines.

Critical strengths
  1. The research perspective is clear, and the State-OPD method is designed by focusing on how the student model's prior reasoning affect the teacher model.
  2. The paper conducts sufficient experiments, evaluating on multiple baselines and three datasets, and includes ablation studies.
Critical weaknesses
  1. The authors find that during agent inference, the student model's prior reasoning affect the teacher model's reasoning, and based on this they design a distillation method that ignores historical thoughts. However, the negative effect may mainly stem from erroneous prior reasoning rather than historical thoughts per se[1]. The paper does not adequately distinguish among correct historical plans, erroneous reasoning, and interference from redundant text.

  2. Methodologically, the teacher model cannot see prior reasoning and only observes historical tool calls; the teacher's target does not depend on old thinking, and fitting this target may weaken the student's reliance on old thinking. However, at inference time, the model's reasoning is expected to be coherent and interconnected[2-3]. It is unclear whether this design affects the student's utilization of correct and useful historical reasoning.

  3. Why does the method perform very well on ALFWorld but underperform Guided-OPD on ScienceWorld? The experimental results are insufficient to establish the cross-task applicability of the proposed mechanism. If this is a limitation of the method, it should be stated in the limitations.

  4. The visualization of the KL divergence between the student and teacher models (Figure 2) is insufficient to demonstrate the mechanism of supervision "distorts supcrvision". The magnitude of the KL divergence between student and teacher is not a measure of supervision quality. a closer distribution does not necessarily mean the teacher has been erroneously influenced. Therefore, the observed KL differences cannot yet distinguish whether the teacher is accommodating the student's erroneous reasoning or reasonably utilizing prior reasoning shared by both.

[1] Zhuang, Y., Chen, K., Duan, Y., Zheng, S., Li, J., & Zhang, X.-Y. (2026). AgentRewind: Recoverable Execution for Long-Horizon LLM Agents. https://arxiv.org/abs/2608.14380

[2] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903

[3] Qian, H., Cao, Z., & Liu, Z. (2026). MemoBrain: Executive Memory as an Agentic Brain for Reasoning. https://arxiv.org/abs/2601.08079

Questions for authors
  1. Should the authors discuss in depth whether the student model's prior reasoning is correct historical plans, erroneous reasoning, or interference from redundant text, and distinguish among them?

  2. Should the authors provide a mechanistic explanation for why the method underperforms Guided-OPD on ScienceWorld. Whether due to limitations or other reasons?

  3. Can the authors provide stronger evidence for the mechanism of "distorts supcrvision"?

8 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the thoughtful feedback. We respond point-by-point below.

W1, W2 & Q1. Historical reasoning in teacher supervision and student decisions

W1 & Q1: Removing only detected errors versus all past reasoning. We directly test whether removing only erroneous historical reasoning, while preserving the remaining thoughts, provides better supervision than state grounding. We adapt AgentHallu’s step-by-step hallucination-checking prompt [1] and use the same frozen teacher to label each reasoning block as supported or erroneous. Each judgment uses only the task, environment rules, and history available when that block was generated. The selective variant removes only blocks judged erroneous, preserving supported reasoning.

Both variants use four student rollouts per task without state-centered response allocation. The student retains its full history, and its complete current reasoning and action remain supervised. Only the treatment of past reasoning in the teacher’s context changes. The following table reports success rates.

StudentPast reasoning removed from teacher contextALFWorld SeenALFWorld UnseenScienceWorld SeenScienceWorld UnseenWebShop
1.7BOnly blocks judged erroneous30.3634.148.999.4611.80
1.7BAll past reasoning30.7135.639.6511.2512.15
4BOnly blocks judged erroneous33.0438.6222.8124.9415.80
4BAll past reasoning40.3643.1023.4625.8318.70

With the student’s access to history, current-response supervision, and rollout configuration unchanged, state-grounded supervision yields higher student success in all ten settings. The alternative motivated by the reviewer’s concern, removing detected errors while preserving the remaining thoughts, does not match this performance. Constructing teacher targets directly from the interaction record therefore provides more effective supervision than selectively retaining historical reasoning, without requiring a correctness judgment for each past thought.

W2: Utilization of useful historical reasoning. The comparison above evaluates the distilled student’s task performance with its full history available. Preserving historical reasoning not flagged as erroneous for the teacher does not produce a stronger student.

State-OPD continues to supervise the student’s use of earlier reasoning in its current response. When the student draws on an earlier plan, every token of the resulting reasoning and action remains supervised. The teacher sees the current response prefix together with the task and the complete action–observation record. Thus, earlier plans can inform the student’s decisions and receive supervision through the current response, even when those plans are omitted from the teacher’s inherited context.

Meanwhile, the full State-OPD model further improves task behavior. Below, we report mean actually executed environment actions on ALFWorld and ScienceWorld. These counts include successful and failed episodes; failures are not assigned the maximum horizon as in the main table.

StudentMethodALFWorld SeenALFWorld UnseenScienceWorld SeenScienceWorld Unseen
1.7BVanilla OPD19.6119.1917.5615.38
1.7BState-OPD17.8416.3617.4215.45
4BVanilla OPD21.7321.4520.2117.21
4BState-OPD18.3917.4913.2110.94

On ALFWorld, State-OPD achieves higher success with fewer executed actions at both student sizes. For the 4B student, average success increases from 32.42% to 44.55%, alongside the reduction in actual interactions shown above. On ScienceWorld, the 4B student uses substantially fewer actions with similar success, while the 1.7B interaction counts change little. The ALFWorld training analysis also shows longer current reasoning for the 4B student.

Together, these results support more effective integration of useful historical information into current reasoning and action selection. The judge comparison establishes the advantage of grounding teacher supervision in interaction evidence while preserving the student’s access to history. The behavioral results show that the complete method translates this learning setup into more successful and efficient task decisions. Useful earlier plans remain available to the student, and their contribution to the current response remains part of what is supervised.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3 & Q2. ScienceWorld performance and cross-task applicability

We thank the reviewer for pointing this out. In difficult environments such as ScienceWorld, effective behaviors can be rare in student rollouts. Off-policy demonstrations supply them directly. Guided-OPD combines reverse KL on student turns with forward KL on teacher turns, gradually reducing teacher intervention to zero [2]. Executing teacher actions also exposes the student to states it might otherwise rarely reach.

Dependence on student-generated experience is a limitation of the on-policy setting, not one introduced by State-OPD. Demonstrations can therefore be more valuable to the weaker 1.7B student, whereas 4B can generate more useful behaviors and benefit more from improved feedback. Guided-OPD also reports larger gains for smaller students.

This boundary does not negate our contribution: better supervision and local response coverage within the on-policy setting. State-OPD remains competitive with Guided-OPD on ScienceWorld without teacher-generated turns and achieves the highest macro-average success at both sizes. At 4B, it also outperforms all baselines on WebShop.


W4 & Q3. Evidence for interference with teacher supervision

We thank the reviewer for highlighting the distinction between distributional agreement and supervision quality. We would like to clarify that the evidence for an adverse effect on teacher task performance comes from the controlled continuation experiment in Figure 2(a). The fixed-response analysis in Figure 2(b) examines how the same context choice affect the targets used for distillation.

Controlled teacher continuation. At matched ALFWorld states, we compare the same Qwen3-32B teacher conditioned on either the full student history (ThT_h) or the interaction record with prior student reasoning removed (TsT_s). Each pair shares the restored environment, sampling seed, and continuation horizon. The intervention removes only inherited student reasoning; subsequent teacher-generated reasoning and observations are retained in full in both conditions, and the results are shown in Table 3:

Source studentAnchor depthFull history (ThT_h)Interaction state (TsT_s)Paired gain [95% CI]
Qwen3-0.6BEarly5.4717.97+12.50 [4.41, 21.67]
Qwen3-0.6BMiddle2.3415.63+13.28 [6.45, 20.51]
Qwen3-0.6BLate0.0014.06+14.06 [4.61, 25.78]
Qwen3-4BEarly14.8415.63+0.78 [-4.84, 6.90]
Qwen3-4BMiddle17.9721.09+3.13 [-9.48, 17.74]
Qwen3-4BLate10.1630.47+20.31 [5.47, 37.07]

Each row contains 128 paired continuations. Gains and confidence intervals are in percentage points; intervals are clustered by source episode. The point estimates favor removing inherited reasoning in all six groups. The clearest effect for the 4B source student occurs at late anchors, where teacher success increases from 10.16% to 30.47%. These results provide direct behavioral evidence that inheriting student reasoning can impair teacher task performance.

Fixed-response KL analysis. In the separate analysis in Figure 2(b), conducted over 200 vanilla OPD training steps, we hold the student context, response tokens, and student log probabilities fixed and change only the teacher's inherited context. Sampled reverse KL decreases with turn depth under full-history conditioning, whereas the trend reverses when prior student reasoning is removed. This difference comes from the teacher's scores for unchanged student responses. Those scores enter the teacher–student log-probability ratio used to construct OPD supervision, showing that the context choice affects the targets from which the student learns.

The observed loss in teacher continuation success, together with the change in teacher scores for fixed student responses, supports the interpretation stated in the paper: these results “suggest that inherited reasoning pulls the full-history teacher target toward the student's response distribution at later turns.”


We are happy to address any further concerns. If all your issues are addressed, we would highly appreciate it if you consider increasing the score.

Best!

References

[1] Liu, Xuannan, et al. AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents. arXiv:2601.06818, 2026.

[2] Li, Gengsheng, et al. On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents. arXiv:2606.15912, 2026.

AuthorsOpenReview ↗

Request to review our responses

Dear Reviewer 3WLR we request you to please review our responses and would love to further engage with you in a constructive discussion if you have any clarifying questions. If all your concerns are addressed we would highly appreciate if you would consider increasing your score.

Reviewer_2ah5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes State-OPD for multi-turn agents, which removes prior student reasoning from the teacher’s context while retaining action–observation histories and the current response prefix. It also resamples complete reasoning–action responses at high-entropy states. Across ALFWorld, ScienceWorld, and WebShop, State-OPD achieves the highest macro-average success rates among the evaluated OPD methods for both student sizes.

Critical strengths
  1. The paper offers an insightful perspective on extending OPD to agentic tasks by revisiting the teacher’s conditioning prefix, extending supervision to every turn but with better control over prefix drift, through a simple fix that removes prior student reasoning.
  2. The empirical analysis provides convincing support for both the motivation and effectiveness of the method. Evaluations across three agentic benchmarks show improved macro-average success rates for both student sizes, while controlled analyses and ablation studies further support the proposed design.
Critical weaknesses

I did not identify any major weaknesses in this paper.

Questions for authors
  1. Figure 7a shows that entropy-based state selection captures more action diversity than random selection, and Table 2 supports the effectiveness of the overall response allocation scheme. Have the authors also explored whether entropy-based selection improves final task success over random selection when the rest of the sampling procedure and budget are held fixed?
3 comments

Thank you for the positive assessment of our motivation and empirical analysis, and for suggesting this additional comparison.

Q1. Entropy-based state selection and final task success

We have evaluated random state selection as a direct ablation of entropy ranking in state-centered response allocation. Within each comparison, we keep the sampling procedure and budget parameters (M=4M=4, β=2\beta=2) unchanged: both selectors allocate six additional complete responses to approximately half of the states in each root trajectory. Only the state-selection rule changes.

Following the component ablations in Table 2, we conduct this comparison both with and without state-grounded supervision. The table reports final success rates on ALFWorld for a Qwen3-4B student distilled from Qwen3-32B.

MethodState selectionSeen SRUnseen SR
State-OPDEntropy43.3945.71
State-OPDRandom39.2941.98
w/o State-Grounded SupervisionEntropy33.3937.31
w/o State-Grounded SupervisionRandom28.5734.51

Entropy ranking achieves higher final success on both splits, with and without state-grounded supervision. With the remaining sampling procedure and budget settings held fixed, these results show that selecting states by entropy improves final task success over random allocation. This provides direct task-performance evidence for the selection criterion, complementing the action-diversity results in Figure 7a.


We are happy to address any further concerns. If all your issues are addressed, we would highly appreciate it if you consider increasing the score.

Best!

AuthorsOpenReview ↗

Request to review our responses

Dear Reviewer 2ah5 we request you to please review our responses and would love to further engage with you in a constructive discussion if you have any clarifying questions. If all your concerns are addressed we would highly appreciate if you would consider increasing your score.

I thank the authors for their reply. I have raised my soundness score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Summary of Our Responses

We sincerely thank all reviewers for their constructive feedback and thoughtful suggestions. We summarize the conducted additional experiments and analyses to address the major concerns raised.

  1. Retaining student history while grounding teacher supervision. State-OPD retains historical reasoning for the student while removing it from the teacher’s context. We tested two alternatives. State-OPD outperforms removing historical reasoning from both models. Separately, with the student’s history and rollout configuration unchanged, removing all past reasoning from the teacher outperforms removing only reasoning flagged as erroneous. Both findings hold across all five evaluation settings at both student sizes. Controlled teacher continuations provide behavioral evidence that inherited reasoning can impair teacher performance, while the fixed-response KL analysis shows how it changes the teacher scores used for distillation.

  2. ScienceWorld and student size: on-policy versus off-policy experience. Guided-OPD combines reverse KL on student turns with forward KL on teacher-generated turns during training. These off-policy demonstrations can supply effective behaviors and subsequent states that student rollouts rarely reach, particularly in difficult environments and for weaker students. Dependence on student-generated experience is a shared constraint of on-policy learning, not a limitation introduced by State-OPD. The gap to a method that introduces teacher-generated experience therefore does not establish a weakness specific to our supervision mechanism. State-OPD remains competitive on ScienceWorld without teacher-generated turns and achieves the highest macro-average success among the compared methods at both student sizes.

  3. Entropy-based allocation improves final task success. Holding the remaining sampling procedure and budget parameters fixed, entropy ranking improves ALFWorld success over random selection by 2.80–4.82 percentage points, across Seen and Unseen and with and without state-grounded supervision. This provides direct task-performance evidence for the selection criterion, complementing the action-diversity analysis and component ablations.

  4. The performance advantage persists at comparable token budgets. At approximately 99M thinking tokens and 102M generated tokens, the 4B State-OPD student achieves 40.00% / 42.35% Seen/Unseen success, compared with 33.39% / 33.96% for Vanilla OPD. Appendix G.4 provides the full token-budget curves and counting details. These comparisons show that the improvement cannot be explained solely by greater thinking-token consumption.

Additional runs across three training seeds reproduce the main 4B ALFWorld improvement and the positive Seen/Unseen average contribution of both components.


We hope these additional results and clarifications address the reviewers' concerns and strengthen the empirical evidence supporting State-OPD. We sincerely appreciate the reviewers' time and consideration and would highly appreciate a response from the reviewers, and a consideration in increasing score if their concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.