acceptodds
Under review as a conference paper at ICLR 2027

DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agents

Abstract

On-policy distillation (OPD) trains student agents through teacher supervision on their own interactions with an environment. However, in asynchronous multi-turn training, arrival-order batching can allow a few early or long rollouts to dominate learner updates while other valid rollouts become stale before being used, wasting already-generated experience. To address this problem, we introduce DivOPD, a simple learner-side batch-selection method that spreads a fixed turn budget across more rollouts and, within each rollout, prioritizes turns with larger cumulative teacher–student disagreement. Turns without usable teacher feedback are excluded. The per-turn loss and optimizer remain fixed; selection only changes which student-visited turns receive training weight. For no-progress rollouts, an optional extension briefly hands control to the teacher before returning it to the student. Across six teacher–student settings on the simulated ALFWorld, ScienceWorld, and WebShop benchmarks, with 1.5B–7B students, DivOPD raises cross-setting mean peak success rate from to and mean success over the last five evaluations from to . It reaches all reported setting-specific targets with geometric-mean speedups of in training tokens and in learner GPU time relative to vanilla OPD. Teacher intervention further raises this last-five mean to while retaining about learner-GPU speedup over vanilla OPD. Anonymized code and configuration files are included in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, LLM agents, multi-turn interaction, batch composition, trajectory diversity, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_GbztOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The look at on-policy distillation and carefully determine what goes into the training batch so that it is not dominated by a few long rollouts, and they prioritize samples with high student-teacher disagreement. They also experiment with allowing the teacher to take over for several turns to avoid stagnation on hard tasks - As I understand it this is a neat way to mix in SFT on teacher trajectories.

Critical strengths
  • Very relevant work - the detail of setting up batch composition well for these methods.
  • Very sensible method with many aspects that combine together to make it work well - illustrated clearly with several ablations.
  • Presented and explained clearly.
Critical weaknesses
  • I will read the reviews from the other reviewers but unusually I can't see any issues other than the common ask of more model families etc.
Questions for authors

All my questions were answered in the appendix. I will ask during the discussion period if I have any more.

2 comments
AuthorsOpenReview ↗

Response: Additional Results and Clarification of Teacher Recovery

Part 1 — Additional Evidence from Independent Runs

Thank you for your positive assessment and careful reading of the appendix. We appreciate your recognition of our batch-composition approach and ablations.

During rebuttal, we added three-run comparisons of DivOPD and vanilla OPD on ALFWorld 1.7B and WebShop 3B, along with three-seed cap-sensitivity experiments across all six settings. We also added length-based selection controls and system-level cost measurements to examine how the choice of turns affects performance and when learner-time savings translate into lower overall cost.

Table Gbzt-1. Full-method Final-5 success rate (%), three independent runs per method. For each run, Final-5 is the mean success rate over the last five evaluations. We report the mean ± sample SD across runs. The learner-update budgets are 250 for ALFWorld and 150 for WebShop, with evaluation every five updates.

SettingVanilla OPDDivOPDMean gain (percentage points)
ALFWorld 1.7B66.13 ± 4.8870.48 ± 6.75+4.35
WebShop 3B58.28 ± 12.5764.11 ± 7.73+5.83

DivOPD has higher mean Final-5 in both settings. The revised PDF reports these independent runs in Section 4.6, Table 3 (p. 9), the cap sweep in Appendix G, Table 20 (p. 24), and the additional controls and cost measurements in Appendix H, Tables 21–26 (pp. 26–29).


Part 2 — Teacher Recovery and Broader Evaluation

One detail about the optional recovery mechanism may help clarify how the teacher contributes. When an interaction stalls, the teacher briefly takes control to help reach a state from which the student can continue. Teacher-generated actions are excluded from the distillation loss. Once the student resumes control, its subsequent turns are eligible for training. Recovery is disabled during evaluation, so the student acts independently. Section 3.4 of the revised PDF (p. 5) makes this explicit.


We appreciate your suggestion to broaden model-family coverage and see this as an important direction for further evaluation. Thank you again for your thoughtful review. We welcome any further questions during the discussion.

AuthorsOpenReview ↗

Thanks again for Your Time and Thoughtful Review in ICLR 2027

Thank you for your thoughtful reading of the paper and appendix, and for your encouraging assessment of the batch-composition approach. We clarified the role of teacher recovery and added independent runs on ALFWorld and WebShop. We hope the revision addresses your concern by making the scope of the current work clearer!

Reviewer_bnk1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies batch composition in asynchronous on-policy distillation for multi-turn agents. The authors identify several problems with arrival-order batching: a few early or long rollouts can dominate updates, other valid rollouts may expire before being used, and turns without usable teacher feedback can occupy training slots.

DivOPD addresses these problems through three stages: filtering invalid turns, allocating slots across rollouts using an initial per-rollout cap, and prioritizing turns with larger cumulative teacher–student disagreement within each rollout. The method preserves the per-turn loss and optimizer configuration while deliberately changing the training distribution. An optional teacher-recovery mechanism helps students escape states in which they repeatedly fail to make progress.

Experiments across three environments and six teacher–student settings show improvements in average success rate and reductions in the learner training tokens and GPU time required to reach specified success thresholds.

Critical strengths

Strength 1: The problem is clearly motivated and supported by direct diagnostic evidence. The paper documents rollout concentration, invalid training slots, and valid rollouts that expire without contributing to an update. In shared-buffer replays, the fraction of valid rollouts never selected decreases from approximately 49–59% to 7–14%. These measurements provide concrete support for the problem formulation and demonstrate that asynchronous queue-consumption rules can materially affect learning.

Strength 2: The controlled experiments support an independent contribution from rollout coverage. Table 18 holds the candidate-pool size, validity gate, ordering, pending mechanism, and within-rollout sampling strategy fixed, changing only the initial per-rollout cap. The finite-cap variant improves peak success rate in all six settings and final-five success rate in five, with average improvements of approximately 2.1 and 3.4 percentage points, respectively.

Consequently, the gains cannot be explained entirely by filtering invalid data or accessing a larger candidate pool. This experiment is an important reason for my positive recommendation.

Strength 3: The method is simple, and the empirical improvements span several settings. The selection procedure reuses cached probabilities without requiring an additional model forward pass. Relative to vanilla OPD, DivOPD improves peak success rate in all six settings and final-five success rate in five, increasing the cross-setting final-five average from 71.5% to 78.6%. The evaluation also includes relevant alternatives, including TCOD-F2B and TurnOPD.

Critical weaknesses

Weaknesses 1: The reporting of independent repetitions and uncertainty is insufficient. This is my primary concern. The error bars in Table 1 represent variation across the final five checkpoints, rather than across independent training runs. The number of independent runs underlying the main results is unclear, making it difficult to assess sensitivity to training randomness and asynchronous scheduling.

This concern also affects the efficiency results: peak success rate and the first evaluation reaching a target can be sensitive to evaluation noise. The reported nAUC over a common update budget partly addresses dependence on particular thresholds, but it does not replace uncertainty estimates across independent runs.

Additionally, Table 19 shows that changing the initial cap can alter peak success rate by approximately 7.9 percentage points within a setting. The paper should clarify how the main-experiment caps were selected, whether selection used a separate validation set, and whether tuning budgets were comparable across methods. My concern is robustness and reproducibility, rather than an assumption of inappropriate selection.

Weaknesses 2: The incremental value of the specific selection rules over simpler alternatives remains incompletely established. Table 18 provides useful evidence for limiting individual rollout contributions. However, it does not fully establish how much the complete method improves over straightforward alternatives such as uniform sampling after validity filtering or simple rollout round-robin selection. At least one such baseline under the same candidate-pool and staleness constraints would strengthen the practical contribution.

For focus, the cumulative disagreement score depends on both valid response length and average token-level disagreement, and it may also correlate with turn depth. The current experiments therefore do not fully separate the effects of disagreement information, response length, and trajectory position.

DivOPD-base already provides an important uniform-within-rollout control. Building on this comparison, length-normalized disagreement and length-only selection would more directly test the score design. The gradient-projection audit provides supporting evidence, but its reference direction is the candidate-pool mean gradient and its coverage is limited. It does not independently establish that selected turns have greater long-term learning value.

Questions for authors

Question 1: How many independent training seeds underlie the main experiments and Table 18? Can the authors provide uncertainty across runs for success rate and time-to-target in representative settings, and clarify the validation procedure used to select the per-rollout caps?

Question 2: How does validity filtering followed by uniform sampling perform under the same candidate-pool and staleness constraints? With rollout allocation fixed, how do length-normalized disagreement and length-only selection compare with the proposed focus score?

Question 3: Under the same hardware allocation, how do total GPU consumption and elapsed completion time change when student generation, teacher scoring, and teacher recovery are included? If a complete accounting is unavailable, identifying the system conditions under which overall gains are expected would still be useful.

6 comments
AuthorsOpenReview ↗

Response 1: Independent repetitions, Uncertainty, Cap selection and sensitivity

Part 1 — Response to Weakness 1 and Question 1

Thank you for raising the questions about robustness and parameter selection! We clarify our choices and provide the additional independent-run results below.

Q1.1: How many independent runs were used, and how variable are performance and time-to-target across runs?

Our response

For each original setting, we used one training seed shared across methods. The same protocol applied to the matched-cap comparison in original Table 18 (now Table 19).

For the matched-cap comparison, we use ALFWorld 1.7B seeds 42, 100, and 123. We hold candidate access, validity checks, rollout order, pending rules, uniform within-rollout sampling, and loss fixed; only the cap changes.

Matched-cap results (revised PDF, Table 21, Appendix H, p. 26).

MethodSeedsPeak SR (%) ↑Final-5 (%) ↑nAUC ↑
DivOPD-base w/o cap370.48 ± 6.7563.90 ± 4.900.4518 ± 0.0174
DivOPD-base (k=4)378.81 ± 3.9374.86 ± 6.560.4889 ± 0.0092

We gain 8.33 points in Peak SR, 10.96 in Final-5, and 0.0371 in nAUC (updates 20–200, evaluated every five updates; trapezoidal area divided by 180).

Full-method repetitions (revised PDF, Table 3, Section 4.6, p. 9). Final-5 uses all runs; GPU-hours summarize only target-reaching runs.

SettingMethodRunsFinal-5 (%) ↑GPU-h to target ↓Reached / total
ALFWorld 1.7BVanilla OPD366.13 ± 4.886.80 (n=1)1/3
ALFWorld 1.7BDivOPD (cumulative arm)370.48 ± 6.753.92 ± 0.42 (n=3)3/3
WebShop 3BVanilla OPD358.28 ± 12.577.71 ± 0.09 (n=2)2/3
WebShop 3BDivOPD (cumulative arm)364.11 ± 7.734.40 ± 1.16 (n=2)2/3

We improve mean Final-5 by 4.35 points on ALFWorld and 5.83 points on WebShop. On ALFWorld, we reach the target in 3/3 runs versus 1/3 for vanilla.

Full-method evaluation protocol. We retain the original targets. nAUC uses the shared window, normalized by its length.

SettingLearner-update budgetEvaluation frequencyTest setTarget SRnAUC window
ALFWorld 1.7B250Every 5 learner updatesalfworld_eval_seen, 140 tasks0.7020–200
WebShop 3B150Every 5 learner updateswebshop_eval, 128 tasks0.6020–150

Q1.2: How were the rollout caps selected, and how sensitive are the results to this choice?

Our response

We chose k=4 heuristically from early ALFWorld observations of successful trajectories taking 4–5 steps, without a separate validation set. Main settings use k=4 or k=5 (revised PDF, Table 4, Appendix A). With 64 turns per batch, k=4 encourages at least 16 contributing rollouts on the first pass when enough valid candidates exist; later passes relax the cap.

Our subsequent turns-per-task statistics support this rationale; they were not used for validation-based selection. We report mean ± sample SD across seeds 42, 100, and 123, using updates 20–200 for ALFWorld and 20–130 for WebShop (revised PDF, Table 25, Appendix H.2, p. 28).

Selection ruleALFWorld 1.7BWebShop 3B
Uniform within rollout4.58 ± 0.043.86 ± 0.05
Length-only4.46 ± 0.043.86 ± 0.04
Cumulative disagreement4.43 ± 0.043.85 ± 0.01

Cap sweep: Peak SR (%). We test full DivOPD with cumulative disagreement across seeds 42, 100, and 123; the matched-cap control above uses uniform selection. Bold marks the highest mean among caps. TCOD-F2B is the updated mean ± SD reference.

MethodALFWorld 1.7BALFWorld 4BWebShop 3BWebShop 7BScienceWorld 1.5BScienceWorld 3B
TCOD-F2B74.13 ± 3.7186.96 ± 1.3067.42 ± 0.6388.72 ± 0.7172.64 ± 1.1286.13 ± 0.93
DivOPD (k=3)87.38 ± 1.8091.19 ± 1.0971.09 ± 1.5691.93 ± 0.4578.39 ± 0.8184.51 ± 1.19
DivOPD (k=4)80.24 ± 2.9791.90 ± 0.4173.70 ± 0.4590.36 ± 1.1978.78 ± 0.2386.98 ± 0.60
DivOPD (k=5)80.24 ± 1.0991.90 ± 0.8279.17 ± 1.1988.02 ± 1.1976.30 ± 0.4587.63 ± 1.13
DivOPD (k=6)80.95 ± 2.5189.52 ± 1.0973.96 ± 0.4588.28 ± 1.3575.65 ± 0.9885.68 ± 0.98

At each fixed cap, we observe run-level SD of 0.23–2.97 points. The highest-mean cap agrees with the original study in all six settings, including the ALFWorld 4B tie. Sensitivity remains greater on ALFWorld 1.7B and WebShop 3B.

In the revised PDF: Table 20, Appendix G (p. 24), replaces original Table 19. Run definitions are clarified in Section 4.1.

AuthorsOpenReview ↗

Response 2: A simpler sampling baseline and Length-only and length-normalized controls

Part 2 — Response to Weakness 2 and Question 2

Q2.1: How does DivOPD compare with validity-filtered uniform sampling under matched constraints?

We compare three rules. Global uniform samples from valid turns without rollout quotas. DivOPD-base applies rollout caps and samples uniformly within each rollout. Full DivOPD uses the same allocation rule and selects turns by cumulative teacher–student disagreement.

All three methods use seeds 42, 100, and 123. The common endpoint is update 200 on ALFWorld 1.7B and update 115 on WebShop 3B. Final-5 averages the five evaluations at updates 180–200 and 95–115, respectively. We compute nAUC over updates 20–200 and 20–115, respectively, and normalize by the window length.

Table bnk1-5. Three-seed global-uniform comparison. Entries are mean ± sample SD across runs. This extends the single-seed WebShop comparison reported in Appendix H.2, Table 24 of the revised PDF.

SettingMethodFinal-5 (%) ↑nAUC ↑
ALFWorld 1.7BValidity + global uniform73.43 ± 2.710.4449 ± 0.0069
DivOPD-base (k=4, uniform)73.24 ± 6.000.4518 ± 0.0174
DivOPD (k=4, cumulative)74.05 ± 2.720.4837 ± 0.0542
WebShop 3BValidity + global uniform39.11 ± 5.360.1979 ± 0.0402
DivOPD-base (uniform)48.28 ± 1.100.2234 ± 0.0007
DivOPD (cumulative)48.85 ± 9.560.2477 ± 0.0598

Compared with global uniform, full DivOPD improves mean nAUC by 0.0388 on ALFWorld and 0.0498 on WebShop. The mean Final-5 gain is small on ALFWorld (0.62 percentage points over updates 180–200) and larger on WebShop (9.74 percentage points over updates 95–115).


Q2.2: How does cumulative disagreement compare with length-only and length-normalized selection?

We hold candidate access, rollout allocation, validity checks, staleness, pending rules, and training loss fixed. Only the within-rollout selection rule changes. Length-only ranks turns by valid response-token count. Per-token disagreement divides cumulative disagreement by that count.

Table bnk1-6. WebShop 3B Final-5 over the last five recorded evaluations. Mean ± sample SD across matched seeds 42, 100, and 123.

Within-rollout selection ruleFinal-5 (%) ↑
Uniform63.31 ± 6.14
Length-only50.57 ± 9.40
Per-token disagreement63.59 ± 5.99
Cumulative disagreement64.11 ± 7.73

Cumulative disagreement has the highest mean Final-5. Its 13.54-point advantage over length-only supports the value of disagreement information beyond response length. Its margins over uniform (0.80 points) and per-token disagreement (0.52 points) are small relative to run-level variability.

Learning efficiency. We compare all four rules on paired seeds 42 and 123 under a budget of 150 learner updates. We compute nAUC over updates 20–150 for every rule because the restarted cumulative-disagreement run for seed 123 lacks evaluations at updates 0–15. The target is SR ≥ 0.50 on 128 evaluation tasks.

Learner GPU-hours equal two trainer GPUs times cumulative trainer time through the update corresponding to the first target-reaching evaluation. The timer covers training, experience reads, and weight synchronization. It excludes student generation, teacher scoring, evaluation, and restart downtime.

Table bnk1-7. WebShop 3B learning efficiency. Mean ± sample SD across paired seeds 42 and 123.

Within-rollout selection rulenAUC ↑Learner GPU-h to SR 0.50 ↓
Uniform0.3251 ± 0.02784.85 ± 0.44
Length-only0.2791 ± 0.00524.63 ± 0.71
Per-token disagreement0.3290 ± 0.07964.07 ± 0.96
Cumulative disagreement0.3416 ± 0.02723.94 ± 0.21

All four rules reach SR 0.50 in both runs. On these paired seeds, cumulative disagreement improves mean nAUC by 0.0165 over uniform within-rollout sampling and reduces mean learner GPU-hours to target by 18.8%, from 4.85 to 3.94 GPU-hours.

The revised PDF reports the score controls in Section 4.6 (p. 9) and Appendix H.1, Tables 22–23 (p. 27).

AuthorsOpenReview ↗

Response 3: Selected-turn statistics and system-level cost

Part 3 — Response to Weakness 2 (Turn Depth) and Question 3 (Cost)

Thank you for asking about interaction statistics and overall resource use. We provide the selected-turn statistics and distinguish learner GPU time from full-system cost below.

W2: How does the selection rule relate to response length and turn depth?

We additionally summarize response length and turns per task using turns selected into training batches across seeds 42, 100, and 123. Response length is measured in tokens per turn. We report the mean and sample SD of the three seed-level means. The learner-update windows are 20–200 for ALFWorld and 20–130 for WebShop. These windows apply to the response-length and turns-per-task statistics. The separate WebShop learning-efficiency comparison uses updates 20–150 for nAUC.

Table bnk1-8. Selected-turn statistics. Three-seed mean ± sample SD. ALFWorld updates 20–200, WebShop updates 20–130.

SettingSelection ruleSeedsResponse length (tokens/turn)Turns per task
ALFWorld 1.7BUniform within rollout42, 100, 12361.9 ± 0.94.58 ± 0.04
ALFWorld 1.7BLength-only42, 100, 12368.2 ± 1.44.46 ± 0.04
ALFWorld 1.7BCumulative disagreement42, 100, 12365.1 ± 1.04.43 ± 0.04
WebShop 3BUniform within rollout42, 100, 123214.6 ± 2.03.86 ± 0.05
WebShop 3BLength-only42, 100, 123235.1 ± 13.83.86 ± 0.04
WebShop 3BCumulative disagreement42, 100, 123230.9 ± 17.63.85 ± 0.01

On WebShop, the reported mean response length under cumulative disagreement is 7.6% higher than under uniform and lower than under length-only. Together with the controlled selection experiments, this is consistent with disagreement contributing information beyond a preference for longer responses. Turns per task describes task-level interaction length; it does not directly measure the position of selected turns within a trajectory.

Location in the revised PDF: Section 4.6 (p. 9), Appendix H.1, Tables 22–23 (p. 27), and Appendix H.2, Table 25 (p. 28).


Q3: How does DivOPD affect total GPU consumption and elapsed time under the same hardware allocation?

Our response

The reported 1.87× speedup is the geometric mean across the six original settings for learner GPU time to target. To assess overall cost, we compare allocated GPU-hours and elapsed time at the same success target and hardware allocation within each setting. Core selection reuses cached scores without an additional model pass. The full-node accounting below includes the resources allocated to student generation, teacher scoring, and recovery when enabled.

Table bnk1-9. System cost to the original success targets. Original uninterrupted runs on identical eight-GPU nodes.

Setting / targetMethodLearner GPU-hAllocated GPU-h ↓Elapsed (h) ↓
ALFWorld 4B / 0.8Vanilla OPD13.5179.889.98
ALFWorld 4B / 0.8DivOPD6.9859.427.43
ALFWorld 4B / 0.8DivOPD+R7.0664.708.09
ALFWorld 1.7B / 0.7Vanilla OPD8.2176.559.57
ALFWorld 1.7B / 0.7DivOPD3.9760.577.57
WebShop 3B / 0.6Vanilla OPD10.0275.009.37
WebShop 3B / 0.6DivOPD4.5275.289.41

Hardware and accounting

Each run occupies one 8× A800-SXM4-80GB node with two trainer GPUs, four student-rollout GPUs (two engines with tensor parallelism 2), and two teacher-scoring GPUs. Learner GPU-hours are the recorded trainer_gpu_hours. Allocated GPU-hours measure full-node occupancy through target attainment, including reserved generation and teacher resources and their idle time. Elapsed time is measured from job start to the target evaluation using timestamps.

The new multi-seed results in Q1 are reported separately.

DivOPD reduces allocated GPU-hours by 25.6% on ALFWorld 4B and 20.9% on ALFWorld 1.7B, corresponding to vanilla-to-DivOPD allocated-cost ratios of 1.34× and 1.26×. Elapsed time decreases from 9.98 to 7.43 hours and from 9.57 to 7.57 hours, respectively. On ALFWorld 4B, DivOPD+R also remains below vanilla in allocated cost (64.70 versus 79.88 GPU-hours) and elapsed time (8.09 versus 9.98 hours). On WebShop 3B, learner GPU-hours decrease from 10.02 to 4.52, while allocated GPU-hours and elapsed time remain approximately unchanged. We therefore observe end-to-end savings on both ALFWorld settings and learner-time savings on WebShop.

Location in the revised PDF: Section 4.6 (p. 9) and Appendix H.3, Table 26 (p. 29).


Thank you again for the detailed feedback. Your questions helped us make the sampling comparisons and resource accounting more complete. We are happy to discuss any remaining questions!

Reviewer_9HckOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DivOPD, a batch-selection method for asynchronous on-policy distillation of multi-turn agents. It addresses uneven rollout coverage and the expiration of queued experience under bounded policy staleness. DivOPD filters unusable turns, limits each rollout’s first-pass contribution, and selects turns within each rollout using cumulative teacher-student disagreement. An optional teacher-recovery step helps stalled rollouts continue. Experiments cover six settings on ALFWorld, ScienceWorld, and WebShop. Compared with vanilla OPD, DivOPD increases mean peak success rate from 77.4% to 84.4% and mean success rate over the final five checkpoints from 71.5% to 78.6%. It also achieves a geometric-mean speedup of 1.87x in learner GPU time to target.

Critical strengths
  1. A useful diagnosis of wasted training experience. Under bounded staleness, batch selection determines which rollouts contribute before they expire. This makes allocation a meaningful part of asynchronous training. In the shared-buffer replays in Table 11, rollout-first allocation reduces the fraction of valid rollouts never selected from about 49-59% to 7-14%. Holding the experience fixed helps isolate the allocation effect. This analysis gives a concrete explanation for wasted experience and motivates broader rollout coverage.

  2. Controlled evidence for a simple allocation rule. DivOPD separates allocation across rollouts from selection within each rollout. It reuses cached scores and requires no extra model pass for selection. Table 18 tests the rollout cap with the same candidate access, validity checks, rollout order, pending rules, and uniform within-rollout sampling. The cap improves mean peak success rate by about 2.1 percentage points and mean Final-5 by about 3.4 points. Gains on ScienceWorld, where all turns pass the validity checks, further support an allocation benefit beyond filtering. These controls make the core empirical contribution convincing.

Critical weaknesses
  1. The robustness of the component gains remains uncertain. The reported standard deviations measure variation over the last five checkpoints. They do not establish consistency across independent training runs. The matched-cap experiment uses one seed, while Table 19 shows that peak success rate can vary by up to 7.9 points across cap choices. The caps and recovery parameters also differ across settings, but their selection procedure is unclear. These gaps limit confidence in the size and stability of the gains. A clear tuning protocol and repeated runs for a representative cap comparison would strengthen the evidence.

  2. The evidence for the specific focus score is less complete. The cumulative score reflects both token-level disagreement and response length, while the per-turn loss averages over valid tokens. Comparisons with uniform sampling do not separate these effects. Focus improves mean peak performance, but raises Final-5 in only three of six settings. On WebShop 3B, Final-5 falls from 73.12% with DivOPD-base to 67.81% with focus. The gradient audit supports alignment with the candidate-pool gradient, but cannot resolve this task-performance tradeoff. Length-only and length-normalized controls would clarify what the cumulative score adds and when it helps.

Questions for authors
  1. How were the caps and recovery parameters selected? Were they chosen on a separate validation set? Could you also repeat the matched-cap comparison on a representative setting, such as ALFWorld 1.7B, using independent seeds? Final-5 and nAUC over a common update budget would help assess whether the allocation gain is stable.

  2. On a representative setting such as WebShop 3B, how does cumulative disagreement compare with length-only and length-normalized disagreement selection? Keeping candidate access and rollout allocation fixed would isolate the score's effect. Reporting both time-to-target and Final-5 would help explain the observed tradeoff between learning speed and final performance.

4 comments
AuthorsOpenReview ↗

Response 1: Cap and Recovery Parameter Selection

We sincerely thank you for recognizing the batch-allocation problem and the matched-access controls! We added independent runs, cap-sensitivity experiments, and direct score controls to address your questions on robustness and cumulative disagreement.

Q1 — Part 1: Cap and Recovery Parameter Selection

Q1.1: How were the rollout cap and recovery parameters selected?

Initial rollout cap

We chose k=4 heuristically based on early ALFWorld observations that successful trajectories typically required approximately 4–5 steps; we did not use a separate validation set. The main experiments use k=4 or k=5, as specified in Table 4 (Appendix A). With a 64-turn batch, k=4 requires contributions from at least 16 rollouts to fill the batch in the first pass, provided sufficient valid candidates are available. Later passes relax the cap if needed.

Our subsequent three-seed analysis is consistent with this initial rationale: mean turns per task range from 4.43 to 4.58 on ALFWorld 1.7B and from 3.85 to 3.86 on WebShop 3B across selection rules (Table 25). These are subsequent descriptive statistics, rather than a validation-based selection of k.

Supporting statistics from Table 25 in the revised PDF (Appendix H.2, p. 28). Mean turns per task ± sample SD across seeds 42, 100, and 123. Learner-update windows: ALFWorld 20–200; WebShop 20–130.

Selection ruleALFWorld 1.7BWebShop 3B
Uniform within rollout4.58 ± 0.043.86 ± 0.05
Length-only4.46 ± 0.043.86 ± 0.04
Cumulative disagreement4.43 ± 0.043.85 ± 0.01

To assess sensitivity directly, we evaluated k ∈ {3, 4, 5, 6} across three seeds in all six settings (Table 20). We discuss these results in Response 2.

Optional recovery

We set recovery patience and the teacher-turn budget relative to each setting’s maximum interaction horizon. Patience allows the student to explore before intervention, while the teacher-turn budget limits the extent of recovery. Across settings, recovery begins after 2–6 consecutive no-progress turns and permits at most 4–6 teacher turns per rollout, corresponding to approximately 7–27% and 20–33% of the interaction horizon, respectively. The student resumes control after recovery. Teacher-generated actions are excluded from the training loss, and recovery is disabled during evaluation.

Table 9Hck-1. Recovery parameters. Counts are turns; ratios are relative to the interaction horizon.

SettingHorizonPatiencePatience / HorizonMax teacher turnsTeacher budget / Horizon
ALFWorld 1.7B30413.3%620.0%
ALFWorld 4B30620.0%620.0%
WebShop 3B15426.7%533.3%
WebShop 7B15426.7%426.7%
ScienceWorld 1.5B3026.7%620.0%
ScienceWorld 3B3026.7%620.0%

Recovery warmup

On some benchmarks, the initial student struggles to make progress even after teacher intervention. We therefore delay recovery to allow initial student learning before enabling teacher intervention, aiming to avoid additional generation cost when early recovery offers limited benefit. Warmup is measured in learner updates; the values for each setting are provided in Appendix G.

Location in the revised PDF: Table 4 (Appendix A); recovery details in Section 3.4 and Appendix G; supporting interaction statistics in Table 25 (Appendix H.2, p. 28).

AuthorsOpenReview ↗

Response 2: Multi-Seed Allocation Gains and Cap Sensitivity

Q1 — Part 2: Multi-Seed Allocation Gains and Cap Sensitivity

Q1.2: Are the allocation gains consistent across independent training seeds?

Response

The original main-table experiments used one training seed per method and setting, shared across methods under the same evaluation conditions. We have now added three-seed experiments on ALFWorld 1.7B and report mean ± sample SD across runs.

Setting

We use seeds 42, 100, and 123. Both arms use the same candidate access, validity checks, rollout order, pending rules, within-rollout uniform sampling, and loss; only the initial cap differs.

Table 9Hck-2. ALFWorld 1.7B matched-cap comparison. Three seeds; mean ± sample SD.

MethodSeedsPeak SR (%) ↑Final-5 (%) ↑nAUC ↑
DivOPD-base w/o cap370.48 ± 6.7563.90 ± 4.900.4518 ± 0.0174
DivOPD-base (k=4)378.81 ± 3.9374.86 ± 6.560.4889 ± 0.0092

SD is the sample standard deviation across runs.

Common evaluation window: learner updates 20–200, evaluated every 5 learner updates; nAUC is the trapezoidal area divided by 180.

With matched candidate access and uniform within-rollout sampling, cap=4 improves mean Peak SR by 8.33 percentage points, Final-5 by 10.96 points, and nAUC by 0.0371 over the uncapped control. These three-seed averages support the allocation benefit in this setting.


Q1.3: How sensitive is DivOPD to the rollout cap across settings?

We extend the cap comparison in the original Table 19 (now Table 20) to seeds 42, 100, and 123, evaluating k ∈ {3, 4, 5, 6} for full DivOPD with cumulative disagreement selection. Each DivOPD cell reports Peak SR (%) as mean ± sample SD across three runs. This tests cap sensitivity with focus enabled, whereas the matched-cap comparison above uses uniform within-rollout sampling.

Table 9Hck-3. Cap sensitivity across six settings (paper Table 20). Peak SR (%); DivOPD uses three-seed mean ± sample SD, with the updated TCOD-F2B reference.

MethodALFWorld 1.7BALFWorld 4BWebShop 3BWebShop 7BScienceWorld 1.5BScienceWorld 3B
TCOD-F2B74.13 ± 3.7186.96 ± 1.3067.42 ± 0.6388.72 ± 0.7172.64 ± 1.1286.13 ± 0.93
DivOPD (k=3)87.38 ± 1.8091.19 ± 1.0971.09 ± 1.5691.93 ± 0.4578.39 ± 0.8184.51 ± 1.19
DivOPD (k=4)80.24 ± 2.9791.90 ± 0.4173.70 ± 0.4590.36 ± 1.1978.78 ± 0.2386.98 ± 0.60
DivOPD (k=5)80.24 ± 1.0991.90 ± 0.8279.17 ± 1.1988.02 ± 1.1976.30 ± 0.4587.63 ± 1.13
DivOPD (k=6)80.95 ± 2.5189.52 ± 1.0973.96 ± 0.4588.28 ± 1.3575.65 ± 0.9885.68 ± 0.98

In the revised PDF, this updated table appears as Table 20 in Appendix G (p. 24), replacing Table 19 of the original submission.

The cap with the highest mean Peak SR agrees with the original cap study (updated in Table 20) in all six settings, including the tie at k=4 and k=5 on ALFWorld 4B. Across cap choices, the range of mean Peak SR is smaller for the larger student on ALFWorld (7.14 versus 2.38 points) and WebShop (8.07 versus 3.91 points), and approximately equal on ScienceWorld (3.13 points, computed before rounding). The preferred cap is consistent with the original table, while sensitivity remains greater for the smaller students on ALFWorld and WebShop.

Revision: Sections 3.4 and 4.1; Appendix G, Table 20 (p. 24); Appendix H, Table 21 (p. 26).

AuthorsOpenReview ↗

Response 3: Disagreement, Length Controls, Learning Efficiency and Final Performance

Q2.1: How does cumulative disagreement compare with length-only and length-normalized selection?

Response

We added the requested length-only and length-normalized controls using three matched seeds and report the results in Table 22 of the revised PDF.

Setting

We keep candidate access, rollout allocation, validity checks, staleness, pending rules, and loss fixed, changing only the within-rollout selection rule. Length-only ranks by valid response-token count; normalized disagreement divides the cumulative score by that same count.

Table 9Hck-4. WebShop 3B selection controls. Mean ± sample SD across matched seeds 42, 100, and 123.

WebShop 3B selectionFinal-5 (%) ↑
Uniform63.31 ± 6.14
Length-only50.57 ± 9.40
Per-token disagreement63.59 ± 5.99
Cumulative disagreement64.11 ± 7.73

Analysis

We find that cumulative disagreement achieves the highest mean Final-5 across the three matched seeds: 0.80 points above uniform, 0.52 above per-token disagreement, and 13.54 above length-only. We report run-level SD alongside these means. The larger gap over length-only supports our use of disagreement beyond response length.

Location in the revised PDF: Appendix H.1, Table 22 (p. 27).


Q2.2: How does the selection rule affect learning efficiency and final performance?

Setting

We compare seeds 42 and 123 under a 150-learner-update budget on WebShop 3B. nAUC is the trapezoidal mean SR over updates 20–150. We exclude updates 0–15 from all methods because the restarted cumulative-s123 run has no evaluations in that interval. The target is SR ≥0.50 on the same 128 evaluation tasks. Learner GPU-hours equal two trainer GPUs multiplied by cumulative perf/time_per_step from update 0 through the update corresponding to the first evaluation reaching the target. This includes trainer updates and the experience-read/weight-sync time captured by that timer, but excludes student generation, teacher scoring, evaluation, and crash/restart downtime.

Table 9Hck-5. WebShop 3B learning efficiency. Paired seeds 42/123; mean ± sample SD; nAUC over updates 20–150.

WebShop selectionnAUC ↑GPU-h to SR 0.50 ↓
Uniform within rollout0.3251 ± 0.02784.85 ± 0.44
Length-only0.2791 ± 0.00524.63 ± 0.71
Per-token disagreement0.3290 ± 0.07964.07 ± 0.96
Cumulative disagreement0.3416 ± 0.02723.94 ± 0.21

All four rules reach SR 0.50 in both runs.

Cumulative disagreement has the highest mean nAUC and the lowest mean learner GPU time to SR 0.50 on these paired seeds. Relative to uniform, it raises nAUC by 0.0165 and reduces learner GPU time by 18.8%. Relative to per-token disagreement, the gains are 0.0126 in nAUC and 3.2% in learner GPU time.

Higher-target check

At SR 0.60, cumulative disagreement reaches the target in 1/2 runs and length-only in 0/2. We distinguish early learning efficiency from final performance: cumulative disagreement improves the mean time to SR 0.50, while the original WebShop 3B Final-5 comparison remains 73.12% for DivOPD-base and 67.81% with focus.

Location in the revised PDF: Section 4.6 (p. 9); Appendix H.1, Tables 22–23 (p. 27).


Thank you again for the careful review. We hope the added experiments and clarifications address your concerns, and we are happy to discuss any remaining questions!

AuthorsOpenReview ↗

Thanks Again for Your Time and Thoughtful Review in ICLR2027

Thank you for suggesting experiments that distinguish the effect of rollout allocation from the effect of turn selection. We clarified how we chose the cap and recovery parameters, repeated the cap comparison across seeds, and added length-based controls to examine the tradeoff between learning speed and final performance. We hope these additions address your concerns about these part. Thanks again for your time and valuable suggestions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciation to the PC, AC and Reviewers: Summary of Our Rebuttal and Key Improvements

We thank the PC, AC, and all reviewers for your careful reading and constructive feedback. In response, we added independent training runs, controlled sampling comparisons, cap-sensitivity experiments, and system-level cost measurements. We summarize the main updates below. The revised manuscript and our individual responses provide the detailed protocols and results.

Main concernAdditional experiments and clarificationsKey findings
Robustness across independent runsAdded three-run comparisons of DivOPD and vanilla OPD on ALFWorld 1.7B and WebShop 3B, reporting mean ± sample SD across runs and target-attainment counts (Table 3).DivOPD improves mean Final-5 success rate by 4.35 percentage points on ALFWorld and 5.83 points on WebShop. On ALFWorld, DivOPD reaches the original success target in 3/3 runs, compared with 1/3 for vanilla OPD.
Contribution of rollout allocationRepeated the matched-cap comparison on ALFWorld 1.7B across three seeds, holding candidate access, validity checks, rollout order, pending rules, within-rollout sampling, and loss fixed (Table 21).In the comparison reported in Table 21, adding the rollout cap improves mean Peak SR by 8.33 percentage points, Final-5 by 10.96 points, and nAUC by 0.0371.
Cap selection and sensitivityClarified how we chose the cap heuristically, without a separate validation set. Evaluated k ∈ {3, 4, 5, 6} across three seeds in all six settings (Tables 4 and 20).The cap yielding the highest mean Peak SR agrees with the original cap study in all six settings, including the tie on ALFWorld 4B. Sensitivity remains greater for the smaller students on ALFWorld and WebShop.
Value of the selection score and simpler baselinesAdded length-only and length-normalized disagreement controls on WebShop 3B (Tables 22–23). Our updated response to Reviewer bnk1 (Q2.1) also compares DivOPD with validity-filtered global-uniform sampling across three matched seeds on ALFWorld 1.7B and WebShop 3B.At the common endpoint within each setting, DivOPD improves mean Final-5 over global uniform by 0.62 percentage points on ALFWorld and 9.74 points on WebShop, with higher mean nAUC in both. Among the four within-rollout rules on WebShop, cumulative disagreement has the highest mean Final-5, although its margins over uniform and normalized disagreement are small relative to run-level variability. On two paired seeds, it reduces mean learner GPU time to SR 0.50 by 18.8% relative to uniform sampling within each rollout.
Overall computational costAdded full-node allocated GPU-hours and elapsed time to the original success targets under identical hardware allocations, including student-generation and teacher resources (Table 26).DivOPD reduces allocated GPU-hours by 25.6% on ALFWorld 4B and 20.9% on ALFWorld 1.7B. On WebShop 3B, learner GPU time decreases, while allocated GPU-hours and elapsed time remain approximately unchanged.
Role of teacher recoveryClarified the recovery mechanism, parameter choices, and training and evaluation behavior (Section 3.4 and Appendix G).Recovery helps stalled interactions reach states from which the student can continue. Teacher-generated actions are excluded from the training loss, and recovery is disabled during evaluation.

Our guiding question is how a simple, low-overhead training rule can make better use of the experience the student already generates. In multi-turn interaction, useful learning opportunities are unevenly distributed across rollouts and turns. DivOPD spreads a fixed training budget across more rollouts and uses cached teacher–student disagreement to prioritize turns within each rollout. This keeps the per-turn loss and optimizer unchanged and requires no additional model forward pass for selection.

The additional experiments clarify where the gains arise. The three-seed global-uniform comparison shows higher mean learning-curve performance for DivOPD in both settings, with a larger Final-5 gain on WebShop. With rollout allocation held fixed on WebShop, the clearest advantage of cumulative disagreement over uniform sampling is in learner time to SR 0.50.

Independent full-method repetitions and the updated three-seed global-uniform comparison both cover ALFWorld 1.7B and WebShop 3B. End-to-end cost savings vary by environment. We report across-run variability separately from variation across checkpoints and accompany time-to-target summaries with target-attainment counts.

We appreciate the reviewers’ feedback, which helped us strengthen the empirical evidence and clarify the method and its evaluation. We welcome further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.