- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train one-step vision-language-action policies that retain task performance while reducing the sampling cost of diffusion- or flow-based action experts. The authors formulate a sequence-level reverse-KL objective over robot trajectories and express it through a local action-distribution discrepancy and a future divergence term. DriftOPD uses a demonstration-based drifting objective for local distribution matching and a frozen, demonstration-trained action-value critic to approximate the future-gradient contribution. The resulting method optimizes the policy using offline demonstration contexts and generated action chunks, without additional environment rollouts or queries to a separate teacher during optimization. Experiments across multiple VLA architectures, simulation benchmarks, and real-world manipulation tasks compare task success and efficiency against existing one-step distillation methods and multi-step policies.
Critical strengths
- Broad empirical evaluation. The paper evaluates DriftOPD across multiple VLA architectures and fine-tuning configurations, several simulation benchmarks, and real-world manipulation tasks. It also examines critic weighting, action diversity, command oscillation, and GPU time to task success. This breadth provides useful evidence about the method’s applicability beyond a single backbone or benchmark and addresses practical aspects of deploying one-step policies.
- A promising combination with a meaningful controlled comparison. Combining drifting-based action generation with a frozen, demonstration-trained critic is a promising approach to improving one-step policies without additional environment interaction during optimization. The comparison against Drift, with the critic term disabled and other settings shared, helps assess the incremental empirical benefit of critic guidance. This is the most compelling aspect of the method to me.
Critical weaknesses
- The contribution beyond existing one-step generation methods needs sharper delineation. The paper acknowledges prior work on consistency models, MeanFlow, drifting, and one-step policy learning, but the presentation does not sufficiently separate the capabilities inherited from these methods from the contribution introduced by DriftOPD. In particular, one-step generation is already supported by the drifting formulation, while the practical algorithm adds offline critic guidance. The authors should clarify what the sequence-level formulation contributes beyond motivating this combination and distinguish the absence of separate teacher queries from initialization using a pretrained VLA checkpoint. This distinction materially affects my assessment of the methodological novelty.
- The practical critic’s connection to the sequence-level objective remains insufficiently validated. Equation (21) approximates the future divergence-cost gradient using the negative action gradient of an offline value critic. Appendix A.4 provides an ideal-critic construction and an error characterization, but does not establish that the learned critic has a small gradient mismatch. Moreover, Appendix B.3 trains the critic using successful demonstrations and includes distance-based supervision favoring actions closer to the demonstrated chunk. Consequently, the improvement over Drift does not by itself establish that the critic supplies long-horizon information beyond an additional imitation constraint. A matched comparison with direct action-distance regularization and an ablation of the critic’s distance-based supervision would help resolve this central ambiguity.
Questions for authors
Novelty and teacher-free generation. Could you clarify which components of DriftOPD are new relative to prior drifting-based one-step generation methods? In particular, what enables teacher-free optimization beyond the existing drifting objective, and what role does initialization from a pretrained VLA checkpoint play? 2. Validity of the critic surrogate. What evidence supports the approximation in Eq. (21), which replaces the future divergence-cost gradient with the negative gradient of a task-value critic? Appendix A.4 characterizes the approximation error, but does not establish that this error is small for the learned critic. Under what conditions should these gradient directions align? 3. Long-horizon guidance versus imitation regularization. Since the critic is trained on successful demonstrations and includes distance-based supervision favoring actions closer to the demonstrated chunk, how can we distinguish genuine long-horizon guidance from an additional imitation constraint? Do you have evidence from removing the distance-based supervision or comparing against a matched action-distance regularizer? 4. On-policy interpretation with offline contexts. The sequence-level gradient is defined over policy-induced states, whereas practical training uses offline demonstration states, as acknowledged in Appendix A.2. To what extent can DriftOPD address the train–test state-distribution mismatch under this substitution, and which claims require the learned policy to remain close to the demonstration distribution?
5 comments
Gentle Reminder
We have revised the manuscript and conducted additional controlled experiments to address your concerns.
We would greatly appreciate it if you could take a moment to review our updated responses. Please let us know if you have any further questions or concerns, and we would be happy to address them promptly.
Dear reviewer, we hope our rebuttal has addressed your concerns. As the discussion period draws to a close, we would appreciate it if you could take a moment to review our updated responses.
We appreciate Reviewer sE2e's recognition of our broad empirical evaluation and the meaningful controlled comparison. We respond to each comment below and have incorporated the corresponding revisions into the manuscript, highlighted in blue.
DriftQL vs. DriftOPD. We agree that a discussion of these similar objectives is warranted. We summarize the differences below and have added this discussion to Section E.1.
Long-horizon guidance signal vs. additional imitation constraint. We agree that this distinction is important, since the critic is trained on demonstrations with distance-based supervision. Following the suggestion of Reviewer v1P5, we replaced the return with zero in every critic term while keeping the Cal-QL and EGR terms with the same scales, and used the resulting critic for DriftOPD under the same setting.