- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TD-Drift, a temporal-difference objective for learning occupancy distributions with one-step drifting models from reward-free data. The method enables efficient occupancy sampling and direct policy extraction without critic distillation. Experiments demonstrate improvements in performance and policy extraction efficiency over existing flow-based occupancy methods.
Critical strengths
1.The paper introduces a temporal-difference objective that extends drifting to occupancy modeling, providing an efficient one-step alternative to flow-based methods.
Critical weaknesses
- Direct policy extraction appears to be a natural consequence of using a differentiable one-step generative model. The paper also lacks comparisons with other one-step generative approaches, such as MeanFlow [1][2], making it unclear what advantages are specific to TD-Drift.
- The paper emphasizes efficient downstream adaptation through reward-free pre-training, yet policy extraction still requires 500K reward-labeled transitions. Without an analysis of reward-label efficiency or comparisons with standard offline RL methods trained on the same labeled data [3].
[1]Nguyen X T, Yoo C. One-step flow q-learning: Addressing the diffusion policy bottleneck in offline reinforcement learning[C]//International Conference on Learning Representations. 2026.
[2]Geng Z, Deng M, Bai X, et al. Mean flows for one-step generative modeling[J]. Advances in Neural Information Processing Systems, 2026.
[3]Park S, Li Q, Levine S. Flow q-learning[J]. arXiv preprint arXiv:2502.02538, 2025.
Questions for authors
- Figure 7 reports a 2.4x speedup in policy extraction for (In)DOM-Direct over (In)FOM, while Table 3 shows that (In)DOM requires more pre-training steps. How does the total computational cost compare when both pre-training and policy extraction are considered?
I would consider raising my score if the authors could provide convincing evidence of TD-Drift's advantages over other one-step generative methods and demonstrate its reward-label efficiency under fair comparisons with offline RL baselines.
We thank the reviewer for their constructive feedback and clear re-evaluation criteria.
Comparison to other one-step generative approaches
Applying other one-step generative models such as MeanFlow to occupancy modeling is not straightforward, as standard training objectives require data samples alone, and temporal-difference variants have not been formulated yet to the best of our knowledge. MeanFlow regresses interval-averaged velocities, and we believe that deriving a Bellman-consistent bootstrap target for it would be a contribution in its own right. We thus focus on showing that a class of one-step generative models is competitive with the multi-step models so far used for occupancy modeling, and we do not claim that drifting represents the optimal model class. Having established that one-step models are competitive, we believe that an exploration of other frameworks is very interesting. We, however, consider this beyond the scope of this paper and have added it as direction for future work.
Reward-label efficiency and offline RL baselines
We thank the reviewer for these suggestions. We trained an offline RL method (TD3+BC[1] ) on reward-labeled data alone and compared it with DOM-Direct and InDOM-Direct at different data budgets [10K, 100K, 200K , 500K], and we have added this comparison to the paper (Appendix E.3). For TD3-BC we tuned the BC coefficient and found 2.5 being the best (as also reported in [2]). We find that while offline RL methods increase performance with the data budget, occupancy-based methods outperform them in all data regimes. This further motivates reward-free pretraining, and strengthens the relevance of our method.
Computational cost
We focused on timing policy extraction, as the pre-trained occupancy model can be reused for multiple rewards, and its cost amortized. As you suggested, we can nevertheless report the computational cost of pre-training for drifting models (6.37ms per training step), and compare it to that of flow-based models (2.23ms per training step). All experiments run on a single Nvidia A100. Drifting is thus generally more compute intensive, and requires more gradient steps to converge. We believe that this is a fair cost to pay in exchange for generally faster policy extraction, which is only paid once since the pre-trained occupancy model can be reused for all rewards. We have added this discussion and table with total cost as Appendix E.4. On a side note, we found and corrected an error in our timing analysis for policy extractions, and updated it accordingly. While all drifting-based approaches offer significantly faster (>2x) policy extraction than their flow-based counterparts, InDOM-Direct requires larger forward passes and is 1.2x faster.
Thank you for the clear and actionable feedback. We hope we were able to address each comment in full, and strengthen our submission accordingly. We remain available for further discussion.
[1] Fujimoto et al. “A minimalist approach to offline reinforcement learning.” NeurIPS 2021
[2] Yarats et al. "Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning."