- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ActFirst-OPD, a framework for accelerating multi-turn agent distillation by decoupling environment interaction from full-response generation. During rollout collection, the student predicts actions using reference-conditioned inverse dynamics, with autonomous action prediction after deviations from the reference trajectory. Full reasoning-and-action responses are generated asynchronously from the collected contexts and supervised by a frozen teacher using token-level distillation. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate substantial training speedups while generally maintaining or improving task performance.
Critical strengths
-
A practical and well-motivated efficiency problem. The paper identifies reasoning generation as a significant bottleneck in multi-turn OPD and proposes a reasonable approach to removing it from the critical path of environment interaction.
-
Substantial empirical training acceleration. ActFirst-OPD achieves considerable wall-clock speedups across three environments and multiple student model scales, while generally maintaining or improving task success rates.
-
Useful ablations and efficiency analysis. The paper investigates reference-conditioned actions, autonomous fallback, reference-prefix replay, and serving concurrency. The profiling experiments also help explain the sources and limitations of the observed acceleration.
Critical weaknesses
-
Distribution mismatch between training and deployment policies. Although full responses are sampled from the student policy, the interaction contexts are generated through reference-conditioned inverse dynamics or action-only prediction. Moreover, the actions in the asynchronously generated full responses are not executed. Consequently, the training context distribution may differ substantially from that induced by the student's actual think-then-act policy during deployment, potentially undermining the on-policy benefits of distillation.
-
Incomplete accounting of offline reference preparation costs. ActFirst-OPD requires high-quality offline reference trajectories covering every training task, constructed from demonstrations, teacher rollouts, and oracle solutions. However, the reported training speedups exclude these preparation costs. Appendix Table 5 reports substantial additional computation for reference collection, making direct efficiency comparisons with methods that do not require such references potentially misleading. Although these costs can be amortized across multiple runs, their impact on end-to-end training efficiency remains unclear.
-
Limited applicability to complex, open-ended agent tasks. The method relies on aligning actual environment transitions with a fixed reference trajectory and permanently switches to autonomous action prediction after a mismatch. In modern agent tasks such as coding, where states, actions, and valid solution paths are highly diverse, maintaining such alignment may be substantially more difficult. Frequent early divergence could diminish the benefit of reference-conditioned inverse dynamics and limit the method's generalizability beyond the evaluated environments.
Questions for authors
How significant is the distribution mismatch between fast-action rollouts and actual think-then-act interaction? Could the authors measure how frequently the executed fast actions disagree with the independently generated full-response actions, and investigate whether the resulting context distribution shift affects distillation performance?
2 comments
Official Comment by Authors
Dear Reviewer hndD,
Thank you for your careful review. We would like to gently follow up before the discussion deadline. We would appreciate your feedback on the action-agreement analysis, our accounting of reference-preparation costs, and the discussion of transition checks for coding.
Thank you again for your time and consideration.
Response to Reviewer hndD (1/1)
Thank you for your thoughtful review and for recognizing the importance of the efficiency problem we address, the substantial training speedups we demonstrate, and the value of our ablations and efficiency analysis. Below, we address questions (Q) and weaknesses (W).
Frequency of action agreement. Across the four settings shown below (1.7B and 4B on ALFWorld and WebShop), the executed fast action disagrees with the action parsed from the independently generated full response on 40.59%–63.83% of paired turns over 250 updates. We compare the two actions at the same pre-action context. Each 50-update window pools turns from rollout batches completed in that window. The table reports exact action agreement (%). Agreement is higher in the final 50-update window than in the first in all four settings, with increases of 17.29–34.32 percentage points.
Context distribution. To clarify, we have indeed discussed the context distribution induced by fast rollouts and its implications for on-policy distillation (see §4.3 and Appendix E). Our multi-turn OPD objective treats each full response as a turn-level training example. ActFirst-OPD preserves on-policy token sampling within each turn. The student samples the response from its own think-then-act policy conditional on the collected context, without access to the reference next observation. The executed fast actions determine which contexts are visited across turns. Early action disagreements can expose the student to alternative contexts when they lead to different transitions. The paired comparison measures agreement at contexts visited by ActFirst-OPD.
Effect on distillation. On ALFWorld at 1.7B, Vanilla OPD (think-then-act), w/o ID (unguided fast actions), and ActFirst-OPD (reference-guided fast actions with fallback) achieve mean success rates of 67.64%, 56.93%, and 77.49%, respectively, under the same reference-free think-then-act evaluation (Table 1). Both fast-action variants retain asynchronous full-response generation. In the fixed-checkpoint rollout comparison (§5.4 and Fig. 4), ActFirst-OPD yields more successful trajectories per 100 transitions and fewer tasks with unproductive repetition than unguided fast actions. Together, these results suggest that the rollout context source matters for distillation and that reference guidance helps collect more useful training contexts.
We agree that offline reference preparation costs matter for end-to-end training efficiency when comparing against methods that operate without references. Tables 1–2 report student-training time. Starting from the same prepared teacher for both methods, we add the measured reference candidate collection time from Appendix Table 5 to ActFirst-OPD below. For Qwen3-1.7B, including the one-time reference candidate collection cost makes a single 250-update run slower than Vanilla OPD on all three benchmarks. When the reference cache is reused on the same task set, the collection cost is amortized across runs. The final column shows three or four runs are needed for the cumulative training-time savings to offset that one-time cost.
We agree that reference alignment is harder in open-ended tasks with many valid solution paths and that early divergence can limit the benefit of reference guidance. In ALFWorld with Qwen3-1.7B, 49.75% of reference-guided rollouts stayed aligned throughout, and 33.59% of all executed turns aligned with the reference (Table 10). With this partial alignment, ActFirst-OPD still achieved a 2.28× student-training speedup over Vanilla OPD and 77.49% versus 67.64% mean success under evaluation (Table 1). Extending reference guidance to open-ended coding would require usable references and validated transition checks that accommodate alternative solution paths, potentially using observable unit test or tool outcomes.
We hope these responses fully address your concerns and lead to a more positive evaluation.