acceptodds
Under review as a conference paper at ICLR 2027

Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics

Abstract

On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose **ActFirst-OPD**, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of on ALFWorld, on WebShop, and on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine benchmark-model settings. These results demonstrate that reasoning need not block acting during multi-turn agent distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-turn language agents, inverse dynamics, asynchronous rollout, efficient agent training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hndDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActFirst-OPD, a framework for accelerating multi-turn agent distillation by decoupling environment interaction from full-response generation. During rollout collection, the student predicts actions using reference-conditioned inverse dynamics, with autonomous action prediction after deviations from the reference trajectory. Full reasoning-and-action responses are generated asynchronously from the collected contexts and supervised by a frozen teacher using token-level distillation. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate substantial training speedups while generally maintaining or improving task performance.

Critical strengths
  1. A practical and well-motivated efficiency problem. The paper identifies reasoning generation as a significant bottleneck in multi-turn OPD and proposes a reasonable approach to removing it from the critical path of environment interaction.

  2. Substantial empirical training acceleration. ActFirst-OPD achieves considerable wall-clock speedups across three environments and multiple student model scales, while generally maintaining or improving task success rates.

  3. Useful ablations and efficiency analysis. The paper investigates reference-conditioned actions, autonomous fallback, reference-prefix replay, and serving concurrency. The profiling experiments also help explain the sources and limitations of the observed acceleration.

Critical weaknesses
  1. Distribution mismatch between training and deployment policies. Although full responses are sampled from the student policy, the interaction contexts are generated through reference-conditioned inverse dynamics or action-only prediction. Moreover, the actions in the asynchronously generated full responses are not executed. Consequently, the training context distribution may differ substantially from that induced by the student's actual think-then-act policy during deployment, potentially undermining the on-policy benefits of distillation.

  2. Incomplete accounting of offline reference preparation costs. ActFirst-OPD requires high-quality offline reference trajectories covering every training task, constructed from demonstrations, teacher rollouts, and oracle solutions. However, the reported training speedups exclude these preparation costs. Appendix Table 5 reports substantial additional computation for reference collection, making direct efficiency comparisons with methods that do not require such references potentially misleading. Although these costs can be amortized across multiple runs, their impact on end-to-end training efficiency remains unclear.

  3. Limited applicability to complex, open-ended agent tasks. The method relies on aligning actual environment transitions with a fixed reference trajectory and permanently switches to autonomous action prediction after a mismatch. In modern agent tasks such as coding, where states, actions, and valid solution paths are highly diverse, maintaining such alignment may be substantially more difficult. Frequent early divergence could diminish the benefit of reference-conditioned inverse dynamics and limit the method's generalizability beyond the evaluated environments.

Questions for authors

How significant is the distribution mismatch between fast-action rollouts and actual think-then-act interaction? Could the authors measure how frequently the executed fast actions disagree with the independently generated full-response actions, and investigate whether the resulting context distribution shift affects distillation performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer hndD (1/1)

Thank you for your thoughtful review and for recognizing the importance of the efficiency problem we address, the substantial training speedups we demonstrate, and the value of our ablations and efficiency analysis. Below, we address questions (Q) and weaknesses (W).

Q1 and W1. Distribution mismatch between training and deployment.

Frequency of action agreement. Across the four settings shown below (1.7B and 4B on ALFWorld and WebShop), the executed fast action disagrees with the action parsed from the independently generated full response on 40.59%–63.83% of paired turns over 250 updates. We compare the two actions at the same pre-action context. Each 50-update window pools turns from rollout batches completed in that window. The table reports exact action agreement (%). Agreement is higher in the final 50-update window than in the first in all four settings, with increases of 17.29–34.32 percentage points.

BenchmarkStudent1–5051–100101–150151–200201–2501–250
ALFWorld1.7B21.5229.6438.6750.0555.8436.17
ALFWorld4B45.8262.1562.7363.5363.1159.41
WebShop1.7B27.1242.3956.8459.7060.7446.84
WebShop4B37.2057.9460.5562.7161.7755.84

Context distribution. To clarify, we have indeed discussed the context distribution induced by fast rollouts and its implications for on-policy distillation (see §4.3 and Appendix E). Our multi-turn OPD objective treats each full response as a turn-level training example. ActFirst-OPD preserves on-policy token sampling within each turn. The student samples the response from its own think-then-act policy conditional on the collected context, without access to the reference next observation. The executed fast actions determine which contexts are visited across turns. Early action disagreements can expose the student to alternative contexts when they lead to different transitions. The paired comparison measures agreement at contexts visited by ActFirst-OPD.

Effect on distillation. On ALFWorld at 1.7B, Vanilla OPD (think-then-act), w/o ID (unguided fast actions), and ActFirst-OPD (reference-guided fast actions with fallback) achieve mean success rates of 67.64%, 56.93%, and 77.49%, respectively, under the same reference-free think-then-act evaluation (Table 1). Both fast-action variants retain asynchronous full-response generation. In the fixed-checkpoint rollout comparison (§5.4 and Fig. 4), ActFirst-OPD yields more successful trajectories per 100 transitions and fewer tasks with unproductive repetition than unguided fast actions. Together, these results suggest that the rollout context source matters for distillation and that reference guidance helps collect more useful training contexts.

W2. Accounting for offline reference preparation costs.

We agree that offline reference preparation costs matter for end-to-end training efficiency when comparing against methods that operate without references. Tables 1–2 report student-training time. Starting from the same prepared teacher for both methods, we add the measured reference candidate collection time from Appendix Table 5 to ActFirst-OPD below. For Qwen3-1.7B, including the one-time reference candidate collection cost makes a single 250-update run slower than Vanilla OPD on all three benchmarks. When the reference cache is reused on the same task set, the collection cost is amortized across runs. The final column shows three or four runs are needed for the cumulative training-time savings to offset that one-time cost.

BenchmarkVanilla OPD training (h)ActFirst-OPD training + reference collection (h)Runs to break even
ALFWorld4.4610.454
WebShop3.106.534
ScienceWorld10.3722.043

W3. Applicability to complex, open-ended agent tasks.

We agree that reference alignment is harder in open-ended tasks with many valid solution paths and that early divergence can limit the benefit of reference guidance. In ALFWorld with Qwen3-1.7B, 49.75% of reference-guided rollouts stayed aligned throughout, and 33.59% of all executed turns aligned with the reference (Table 10). With this partial alignment, ActFirst-OPD still achieved a 2.28× student-training speedup over Vanilla OPD and 77.49% versus 67.64% mean success under evaluation (Table 1). Extending reference guidance to open-ended coding would require usable references and validated transition checks that accommodate alternative solution paths, potentially using observable unit test or tool outcomes.

We hope these responses fully address your concerns and lead to a more positive evaluation.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer hndD,

Thank you for your careful review. We would like to gently follow up before the discussion deadline. We would appreciate your feedback on the action-agreement analysis, our accounting of reference-preparation costs, and the discussion of transition checks for coding.

Thank you again for your time and consideration.

Reviewer_7rsQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ActFirst-OPD, a framework for accelerating on-policy distillation (OPD) in multi-turn language agents by decoupling environment interaction from reasoning generation. It uses reference-conditioned inverse dynamics to generate and execute actions directly, switching to autonomous action prediction when the interaction deviates from the reference trajectory. Experiments on ALFWorld, WebShop, and ScienceWorld report substantial reductions in training time while maintaining or improving task success rates compared with other OPD baselines.

Critical strengths
  • The paper addresses the computational bottleneck of multi-turn OPD, a relevant challenge for efficient agent training.
  • Experiments are fairly comprehensive with multiple model size, benchmarks and ablation and efficiency analysis
  • The result is promising. The proposed method achieves substantial training speedups while maintaining or improving task performance.
Critical weaknesses
  • Since environment interactions are guided by reference observations during training, this may introduce a distribution mismatch. The authors should discuss its implications and further evaluate the method on unseen or OOD tasks and environments to assess its generalisability.
  • The method requires benchmark-specific transition consistency checks to detect deviations from reference trajectories.
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7rsQ (1/1)

Thank you for your thoughtful review and for recognizing the relevance of the multi-turn OPD bottleneck we address, the breadth of our evaluation across model sizes and benchmarks, and the substantial training speedups with comparable or better task performance. Below, we address your questions (Q).

Q1. Distribution mismatch and generalization.

Discussion of distribution mismatch. To clarify, we have indeed discussed the context distribution induced by fast rollouts and its implications for on-policy distillation (see §4.3 and Appendix E). We agree that fast actions can change the distribution of contexts used for distillation. Given those contexts, the student samples full responses without reference observations. Thus, full responses are sampled on-policy conditional on the collected contexts, while their context distribution need not match that induced by the student’s think-then-act policy.

Generalization. To clarify, we have indeed evaluated the method on unseen ALFWorld layouts to assess its generalizability. Table 1 evaluates generalization to unseen layouts within ALFWorld (see the “Unseen” column). Across 134 instances from held-out rooms with different layouts and receptacles, ActFirst-OPD improves the success rate relative to Vanilla OPD by 6.22, 9.20, and 3.24 percentage points for 0.6B, 1.7B, and 4B, respectively. The evaluation retains the same six task types. In the revision, we will clarify in the main text that the “Unseen” split tests generalization to held-out ALFWorld room layouts within the same six task types, and highlight the gains reported in Table 1.

Q2. Transition consistency checks.

We agree that the submitted experiments use benchmark-specific transition checks (§4.2). Their shared core is exact string matching between the normalized actual and reference next observations. The additional rules help recognize compatible observations, such as when WebShop search results differ but the reference product remains clickable, or when ScienceWorld displays the same objects in a different order.

To test whether benchmark-independent transition checks are sufficient, we trained Qwen3-1.7B on all three benchmarks with a checker that uses only exact string matching between the normalized actual and reference next observations. The table compares this benchmark-independent check with the benchmark-specific results reported in Tables 1–2. SR is mean ± SD, and time is student-training wall-clock time.

BenchmarkBenchmark-specific SR (%)Benchmark-specific time (h)Benchmark-independent SR (%)Benchmark-independent time (h)
ALFWorld77.49 ± 1.201.9574.45 ± 1.901.82
WebShop39.33 ± 1.531.7240.67 ± 1.531.69
ScienceWorld24.00 ± 1.762.1623.11 ± 1.682.11

With the benchmark-independent transition check, mean SR is 1.34 percentage points higher on WebShop and 3.04 and 0.89 points lower on ALFWorld and ScienceWorld, respectively. Student-training time is slightly lower in all three settings. ActFirst-OPD with this check still achieves higher mean SRs than Vanilla OPD, TCOD-F2B, and TurnOPD on each benchmark (Tables 1–2). These results show that a single normalized observation-matching rule can support ActFirst-OPD across the three tested environments.

We hope these responses fully address your concerns and lead to a more positive evaluation.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 7rsQ,

Thank you for your careful review. We would like to gently follow up before the discussion deadline. We would appreciate your feedback on the additional benchmark-independent transition-checking experiments and our clarification of the generalization evaluation.

Thank you again for your time and consideration.

Reviewer_meW2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ActFirst-OPD, a method for speeding up on-policy distillation for multi-turn agents. Instead of waiting for the student to finish its full reasoning before taking an action, the method predicts an action directly using a reference next observation. If the resulting transition no longer matches the reference trajectory, it switches to autonomous action prediction. Full reasoning-and-action responses are generated asynchronously and used for teacher-guided distillation. Experiments on ALFWorld, WebShop, and ScienceWorld report notable training speedups with comparable or better task performance.

Critical strengths
  1. The paper targets a real bottleneck in multi-turn OPD: the time spent generating reasoning before each environment action.
  2. The proposed design is simple and practical. The combination of reference-conditioned actions, fallback prediction, and asynchronous response generation is easy to understand.
  3. The evaluation covers three environments and three student model sizes. The ablations and serving-concurrency experiments are useful.
  4. The method achieves substantial speedups and usually maintains or improves task success.
Critical weaknesses
  1. Training contexts are collected using reference-conditioned or action-only interactions, while evaluation uses standard think-then-act interaction. Although Appendix E discusses this mismatch, its practical impact remains unclear, particularly since the full responses used for distillation are never executed.

  2. The reported speedups exclude reference preparation, which incurs substantial additional costs (Table 5). Although these references can be reused, an amortized cost comparison would help clarify when the training savings outweigh this overhead.

  3. The transition consistency checks rely on benchmark-specific rules, including hand-crafted exceptions for WebShop. It is unclear how well these checks would generalize to less structured environments or how sensitive the method is to incorrect matches.

  4. The reported variance comes from three evaluation seeds applied to the same trained checkpoint. Independent training runs are not reported, which makes it difficult to assess the reliability of smaller performance differences, particularly on WebShop.

Questions for authors
  1. How often does the executed fast action disagree with the action in the asynchronously generated full response?
  2. What fraction of rollout turns use reference-conditioned inverse dynamics versus autonomous fallback?
  3. What are the results when reference construction and teacher preparation costs are included?
  4. How robust is the method when the reference trajectories are noisy or incomplete?
  5. How would the transition-checking mechanism work for open-ended tasks such as coding?
6 comments
AuthorsOpenReview ↗

Response to Reviewer meW2 (1/3)

Thank you for your thoughtful review and for highlighting the practical rollout bottleneck, the simplicity of the design, the breadth of the experiments, and the training speedups with generally comparable or better task performance. Below, we address questions (Q) and weaknesses (W).

Q1 and W1. Action disagreement and context distribution mismatch.

Action agreement. For the 1.7B and 4B ActFirst-OPD runs on ALFWorld and WebShop, we compare the executed fast action with the action parsed from an independently generated full response at the same pre-action context. Across these four settings, the actions disagree on 40.59%–63.83% of paired turns over 250 updates. Each 50-update window pools turns from rollout batches completed in that window. The table reports exact action agreement (%). Agreement is higher in the final window than in the first in all four settings, with increases of 17.29–34.32 percentage points.

BenchmarkStudent1–5051–100101–150151–200201–2501–250
ALFWorld1.7B21.5229.6438.6750.0555.8436.17
ALFWorld4B45.8262.1562.7363.5363.1159.41
WebShop1.7B27.1242.3956.8459.7060.7446.84
WebShop4B37.2057.9460.5562.7161.7755.84

Context distribution. Our multi-turn OPD objective treats each full response as a turn-level training example. ActFirst-OPD preserves on-policy token sampling within each turn. The student samples the response from its own think-then-act policy conditional on the collected context, without access to the reference next observation. The executed fast actions determine which contexts are visited across turns. Early action disagreements can expose the student to alternative contexts when they lead to different transitions. The paired comparison measures agreement at contexts visited by ActFirst-OPD.

Effect on distillation. On ALFWorld at 1.7B, Vanilla OPD (think-then-act), w/o ID (unguided fast actions), and ActFirst-OPD (reference-guided fast actions with fallback) achieve mean success rates of 67.64%, 56.93%, and 77.49%, respectively, under the same reference-free think-then-act evaluation (Table 1). Both fast-action variants retain asynchronous full-response generation. In the fixed-checkpoint rollout comparison (§5.4 and Fig. 4), ActFirst-OPD yields more successful trajectories per 100 transitions and fewer tasks with unproductive repetition than unguided fast actions. Together, these results suggest that the rollout context source matters for distillation and that reference guidance helps collect more useful training contexts.

Q2. Reference-conditioned inverse dynamics and autonomous fallback.

To clarify, we have indeed reported turn-level reference alignment in Appendix F (Table 10 and Fig. 8). Aligned-turn coverage counts the turns that pass the reference check. Here we report actual ID and NAP usage in the same runs.

BenchmarkStudentExecuted ID (%)Executed NAP (%)
ALFWorld0.6B20.1479.86
ALFWorld1.7B35.0764.93
ALFWorld4B64.8035.20
WebShop0.6B32.9167.09
WebShop1.7B34.7665.24
WebShop4B63.4836.52

Q3 and W2. Reference construction and teacher preparation costs.

Q3. Costs including reference construction and teacher preparation. For Qwen3-1.7B, we add teacher preparation time to both methods and the measured reference construction time to ActFirst-OPD (Tables 1–2 and 5). With the same teacher and reference cache reused, the last column shows the minimum number of runs at which ActFirst-OPD’s cumulative total cost falls below Vanilla OPD’s.

BenchmarkVanilla OPD first-run total (h)ActFirst-OPD first-run total (h)Total runs to break even
ALFWorld8.6414.634
WebShop25.1528.584
ScienceWorld28.6440.313

W2. Amortized costs with reference reuse. The shared teacher preparation cost cancels in the comparison below. With the same task set and reference cache, ActFirst-OPD pays the reference candidate collection cost once and saves student-training time on each run. The last column gives the minimum total number of 250-update runs for cumulative training-time savings to offset that one-time cost.

BenchmarkVanilla OPD training per run (h)ActFirst-OPD first run incl. reference candidate collection (h)Total runs to break even
ALFWorld4.4610.454
WebShop3.106.534
ScienceWorld10.3722.043
AuthorsOpenReview ↗

Response to Reviewer meW2 (2/3)

Q4. Robustness to noisy or incomplete references.

The main experiments use carefully curated references (Appendix C.2). For ALFWorld (Table 1), all cached references solved their training tasks on replay. To test robustness,

  • we built two noisy reference sets from rollouts generated by the ALFWorld 8B teacher (80.75% rollout SR) and the base Qwen3-8B model (54.29% rollout SR). Each set retains one trajectory per training task, including unsuccessful and unfinished rollouts.
  • For incomplete references, we use the 50% Reference-Prefix Replay variant from Table 3, which replays the first half of the reference actions and then switches to NAP.
ModelCurated reference (100% replay SR)Noisy reference (80.75% rollout SR)Noisy reference (54.29% rollout SR)50% reference-prefix replay + NAP
0.6B68.61 ± 0.8959.12 ± 1.4650.24 ± 0.4263.02 ± 0.92
1.7B77.49 ± 1.2069.95 ± 0.5661.31 ± 0.6371.05 ± 1.11
4B82.60 ± 1.1279.32 ± 0.4277.86 ± 1.7272.75 ± 4.82
Avg. training wall-clock time (h)1.942.032.351.90

Student SR (%, mean ± SD across three evaluation seeds) declines at all three sizes with noisy or incomplete references. The noisy sets increase mean training time from 1.94 h to 2.03 h and 2.35 h, respectively, with larger SR drops at 0.6B and 1.7B than at 4B. With 50% prefix replay, mean training time remains similar (1.90 vs. 1.94 h), and the largest SR drop is at 4B (9.85 percentage points).

AuthorsOpenReview ↗

Response to Reviewer meW2 (3/3)

Q5 and W3. Benchmark-specific checks, sensitivity to incorrect matches, and open-ended coding.

Benchmark-specific checks. We agree that the submitted experiments use benchmark-specific transition checks (§4.2). Replacing the hand-crafted rules with a benchmark-independent checker yields modest changes in mean success rate across the three tested benchmarks. In our new Qwen3-1.7B comparison, one benchmark-independent checker uses exact string matching between the normalized actual and reference next observations on all three text benchmarks. Relative to the benchmark-specific checks, mean SR changes by −3.04, +1.34, and −0.89 percentage points on ALFWorld, WebShop, and ScienceWorld, respectively. Student training is slightly faster in all three settings, and mean SR remains above the reported results for Vanilla OPD, TCOD-F2B, and TurnOPD (Tables 1–2).

BenchmarkBenchmark-specific SR (%)Benchmark-specific time (h)Benchmark-independent SR (%)Benchmark-independent time (h)
ALFWorld77.49 ± 1.201.9574.45 ± 1.901.82
WebShop39.33 ± 1.531.7240.67 ± 1.531.69
ScienceWorld24.00 ± 1.762.1623.11 ± 1.682.11

Sensitivity to incorrect matches. Incorrect matches can change when a rollout switches from reference guidance to NAP. We use 50% Reference-Prefix Replay as a fixed-switch comparison, executing the first half of the recorded reference actions before switching to NAP (Table 3).

  • False rejection. A false rejection can end usable reference guidance prematurely. On ALFWorld at 4B, aligned-turn coverage is 64.05%, and 85.63% of reference-guided rollouts complete the reference without fallback (Fig. 8(c) and Table 10). With half-prefix replay, 4B mean SR is 9.85 percentage points below full ActFirst-OPD and 7.18 points below Vanilla OPD.
  • False acceptance. A false acceptance can keep guidance active after a deviation. If the new path remains compatible with the next reference target, continued guidance can still be useful. On ALFWorld, aligned-turn coverage is 18.19% at 0.6B and 33.59% at 1.7B (Fig. 8(a,b) and Table 10). With a valid half-prefix followed by NAP, mean SR at these sizes is 5.59 and 6.44 points below full ActFirst-OPD, respectively, while remaining 2.56 and 3.41 points above Vanilla OPD.
ModelVanilla OPDActFirst-OPD50% reference-prefix replay + NAP
0.6B60.46 ± 3.1568.61 ± 0.8963.02 ± 0.92
1.7B67.64 ± 1.5277.49 ± 1.2071.05 ± 1.11
4B79.93 ± 0.7382.60 ± 1.1272.75 ± 4.82
Avg. training wall-clock time (h)4.401.941.90

Open-ended coding. For open-ended coding tasks, we would use unit test results or tool outputs to verify intermediate progress and check whether the current repository state supports the next reference step. For example, two different edits could both pass a targeted unit test and preserve a required interface, allowing reference guidance to continue along either path.

W4. Variability across independent training runs.

We agree that independent training runs are needed to assess training-seed variability. On WebShop, we will add two independent training runs for each of Vanilla OPD, TCOD-F2B, TurnOPD, and ActFirst-OPD at all three student sizes (0.6B, 1.7B, and 4B). We will use the same three evaluation seeds for each checkpoint, yielding nine evaluations per configuration. For task score and success rate, we will average the three evaluations per checkpoint and report the mean and standard deviation across the three checkpoint averages. We will also compute wall-clock student-training speedups relative to Vanilla OPD for each matched training seed and report their mean.

Based on the reported training times and eight A100 GPUs per run, a three-seed replication of all 45 student-training configurations in Tables 1–2 (five methods across three benchmarks and three student sizes) would require approximately 4,120 A100 GPU-hours in total. Given this cost and your concern about small performance differences on WebShop, we will complete and report the WebShop runs during the discussion period.

We hope these responses fully address your concerns and lead to a more positive evaluation.

Reviewer_6ZWbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes ActFirst-OPD, which decouples environment interaction from reasoning generation, allowing full responses from different turns within a trajectory to be generated concurrently. Across ALFWorld, ScienceWorld, and WebShop, the method generally matches or exceeds baseline task success rates while substantially reducing wall-clock training time.

Critical strengths

The method enables parallel generation of full responses across turns within a prompt, which can reduce rollout latency while retaining token-level teacher supervision.

Critical weaknesses
  1. The paper does not compare against Vanilla OPD with the number of concurrent independent trajectories tuned for throughput on the same hardware. This leaves the overall throughput advantage over an optimized baseline unclear.
  2. Although high-quality references are explicitly assumed, the performance gains and speedups from fewer environment transitions may stem mainly from guided sampling rather than act-first decoupling. The proposed method should be evaluated with offline sampling costs included when comparing efficiency against methods that require no references, or compared with direct offline distillation on the same trajectories to clarify the added value of online interaction.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6ZWb (1/2)

Thank you for your thoughtful review and for highlighting that ActFirst-OPD can reduce rollout latency while retaining token-level teacher supervision. Below, we address your questions (Q).

Q1. Tuning independent trajectory concurrency.

To clarify, in our main experiments, both the rollout batch size and the total number of workflow runners are 16. Vanilla OPD therefore launches 16 independent trajectories concurrently at the start of each rollout batch, using all configured slots. Following your suggestion, we also tested 32 and 64 concurrent trajectories for both methods on ALFWorld with Qwen3-1.7B. Both methods used the same eight 80 GB A100 GPUs, and each run completed 250 updates. The results are summarized below.

Concurrent trajectoriesThroughput (env. steps/s, ActFirst / Vanilla)Env. transition count (ActFirst / Vanilla)Training speedup (Vanilla time / ActFirst time)Success rate (%) (Vanilla / ActFirst)
165.68 / 3.65 ≈1.556×0.695×2.28×67.64 ± 1.52 / 77.49 ± 1.20
328.44 / 5.44 ≈ 1.551×0.699×2.19×70.32 ± 1.11 / 70.07 ± 2.03
648.14 / 7.46 ≈ 1.091×0.607×1.75×64.96 ± 0.97 / 69.46 ± 1.87

Success rates are means ± sample SD across three evaluation seeds. Throughput and transition counts are measured from the first training-sample request to completion of update 250.

Across matched concurrency levels of 16, 32, and 64, ActFirst-OPD achieves 1.556×, 1.551×, and 1.091× the throughput (env. steps/s) of Vanilla OPD, so the throughput advantage narrows at 64. ActFirst-OPD still completes the same number of updates with fewer environment transitions at each level (0.695×, 0.699×, and 0.607× the Vanilla counts) and delivers wall-clock training speedups of 2.28×, 2.19×, and 1.75×, respectively.

We note that both methods had lower mean success rates at 64 concurrent trajectories than at either 16 or 32. In our training setup, which follows TCOD (COLM 2026), each update uses 64 turn-level responses that meet the same staleness criterion. Higher concurrency may generate more candidate responses per update, leaving a smaller fraction of generated responses to be used before they become stale. This could contribute to the observed decline. These results suggest that increasing independent-trajectory concurrency alone can improve environment-transition throughput but may lower success rates in this multi-turn OPD setup.

AuthorsOpenReview ↗

Response to Reviewer 6ZWb (2/2)

Q2. Sources of performance gains and speedups, offline sampling costs, and the added value of online interaction.

Guided sampling and act-first decoupling. To clarify, we have indeed discussed this attribution in §5.4. We agree that guided sampling is the main contributor to the performance gains and speedups from fewer environment transitions. In the fixed-checkpoint rollout comparison, guidance reduces unproductive repetition and rollout length relative to direct action prediction (Fig. 4). Removing reference-conditioned inverse dynamics also lowers mean success rate by 17.31 percentage points relative to full ActFirst-OPD (Table 3). The w/o ID variant retains act-first decoupling without reference guidance and still shortens student training relative to Vanilla OPD in all nine settings (Tables 1–2). With both components enabled, ActFirst-OPD achieves 1.56× the environment-transition throughput of Vanilla OPD while executing 30.45% fewer transitions in the matched ALFWorld 1.7B training runs (Table 9).

Offline sampling costs. To clarify, we have indeed reported offline sampling costs (reference candidate collection time) in Table 5. Below, we add the offline sampling cost to the reported ActFirst-OPD student-training time in Tables 1–2. For Qwen3-1.7B, we count the offline sampling cost once when reusing the reference cache on the same task set. The final column gives the minimum number of 250-update runs at the reported configurations needed for cumulative training-time savings over Vanilla OPD to offset that one-time cost. With the one-time collection cost included, ActFirst-OPD takes less total time than Vanilla OPD after three or four runs that share the same reference cache.

BenchmarkVanilla training (h)ActFirst training + offline sampling cost (h)Runs to break even
ALFWorld4.4610.454
WebShop3.106.534
ScienceWorld10.3722.043

Comparison with direct offline distillation. To clarify, we have indeed included 100% Reference-Prefix Replay in Table 3, a replay-based version of direct offline distillation on the same trajectories. Replay executes the recorded actions to reconstruct training contexts and takes less training time on all three benchmarks. Compared with direct offline distillation, the ActFirst-OPD in Table 3 reports higher mean success rates on both ScienceWorld (22.67% vs. 18.45%, +4.22 percentage points) and WebShop (39.78% vs. 37.89%, +1.89 percentage points). The comparison clarifies the added value of online interaction by showing that student-driven context collection improves task success while increasing mean training time.

We hope these responses fully address your concerns and lead to a more positive evaluation.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 6ZWb,

Thank you for your careful review. We would like to gently follow up before the discussion deadline. We would appreciate your feedback on the additional concurrency experiments and our offline sampling cost and direct offline distillation comparisons.

Thank you again for your time and consideration.

I thank the authors for their reply. I have raised my soundness score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

One-glance Summary of Rebuttals

We thank the Area Chair and reviewers for their feedback. The table summarizes the main concerns and the evidence in our individual responses. SR denotes success rate, and pp denotes percentage points.

ConcernEvidenceTakeawayRaised by
Rollout efficiency and sources of speedupMatched 16/32/64-trajectory sweep on ALFWorld (1.7B), unguided ablation, and fixed-checkpoint rollouts (Fig. 4).ActFirst-OPD trains 1.75–2.28× faster with 30–39% fewer environment transitions. The throughput advantage narrows to 1.09× at 64 trajectories. Act-first rollouts also speed up training without guidance, while guidance reduces unproductive interactions.6ZWb
Reference preparation costFirst-run and amortized costs at 1.7B, including shared teacher preparation and measured reference candidate collection (Table 5).The first run costs more on all three benchmarks. Reusing the same reference cache offsets collection costs after 4, 4, and 3 total runs on ALFWorld, WebShop, and ScienceWorld, respectively.6ZWb, meW2, hndD
Value of online interaction100% Reference-Prefix Replay with recorded actions (Table 3).Table 3 reports +1.89 pp on WebShop and +4.22 pp on ScienceWorld in mean SR for ActFirst-OPD relative to fixed reference-action replay.6ZWb
Training–deployment mismatchSame-context action comparisons at 1.7B and 4B on ALFWorld and WebShop, plus reference-free evaluation (Table 1).Fast and full-response actions disagree on 40.59%–63.83% of paired turns. Agreement rises 17.29–34.32 pp from the first to the last window. Under standard think-then-act evaluation on ALFWorld at 1.7B, ActFirst-OPD reaches 77.49% SR vs. 67.64% for Vanilla OPD.7rsQ, hndD, meW2
Benchmark-specific transition checksNew 1.7B runs using the same normalized observation-matching rule on all three benchmarks.Mean SR changes by −3.04 to +1.34 pp relative to the original checks. Training is slightly faster. ActFirst-OPD still achieves a higher mean SR than Vanilla OPD, TCOD-F2B, and TurnOPD on each benchmark.7rsQ, meW2
Reference quality and fallbackNew ALFWorld noisy-reference runs, executed-action logs, 50% Reference-Prefix Replay (Table 3), and ALFWorld 1.7B alignment and training results (Tables 1 and 10).Noisy references and half-prefix replay lower ALFWorld SR at all three sizes. Reference-conditioned actions account for 20.14%–64.80% of turns across the reported ALFWorld and WebShop runs.meW2, hndD
Generalization to new settingsHeld-out ALFWorld layouts (Table 1) and discussion of transition checks for coding.On unseen layouts within the same six task types, ActFirst-OPD gains 3.24–9.20 pp over Vanilla OPD across model sizes. We discuss how transition checks could be adapted to open-ended coding tasks.7rsQ, hndD, meW2
Independent training runsCompleted 24 additional WebShop training runs, yielding three training seeds per configuration.ActFirst-OPD achieved mean SRs comparable to Vanilla OPD at all three student sizes, with mean wall-clock training speedups of 1.95× (0.6B), 1.91× (1.7B), and 1.84× (4B).meW2

Full details are in the individual rebuttals. We would appreciate your consideration of the additional evidence and clarifications in your final assessment. We welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.