- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GTSE, which combines Time Score Matching with TD Learning to estimate future state occupancy density. It also provides a theoretical connection to representation learning. The results on OGBench are strong, and GTSE performs much better than several baselines.
Critical strengths
- The problem is important. Combining generative models with occupancy density estimation is an interesting idea.
- The method is reasonable. Using TD Learning with Time Score Matching is a useful contribution.
- The experimental results are strong. GTSE performs well on different navigation and manipulation tasks.
Critical weaknesses
- Missing important ablations. It is unclear how much improvement comes from TD Learning or Time Score Matching. A comparison with Time Score Matching without TD would be helpful.
- The theory does not fully explain the results. Higher representation rank does not always give better performance, as shown in Appendix E.2.
- Missing strong planning baselines. GTSE uses a policy to generate actions and then ranks them. This is similar to using a value function for action selection. A world model with value-guided short-horizon planning could do something similar. I would like to see this comparison.
- Inference cost is not clear. The reported time only covers density estimation, not the full action selection process.
Questions for authors
I think the idea is interesting and the results are good. However, some important ablations and baseline comparisons are missing. I am not fully convinced that GTSE offers clear advantages over existing value learning or model-based planning methods. I lean toward rejection, but I may increase my score if these concerns are addressed.
3 comments
Dear Reviewer,
Kindly let us know if our response, which consists of multiple new sets of experiments and revisions to the paper, addresses your concerns adequately. We would greatly appreciate your engagement.
Thank you! The Authors
I thank the authors for the additional experiments and analyses. The rebuttal has addressed my concerns to a reasonable extent. I have therefore raised my score to 3.
We thank you for your review and appreciate your feedback. In our response, we provide detailed answers to the weaknesses highlighted in your review and questions you ask us: missing ablation, theory vs results, planning baseline and inference cost. Please find our response below. We also revised the paper to reflect these changes. All changes are denoted in brown color.
Missing ablation: TD vs MC
To address this concern, we compare GTSE to an ablation that uses Time Score Matching (GTSE-MC) which uses Monte-Carlo style learning, as you suggested, to Appendix E.3. We find that GTSE significantly outperforms GTSE-MC, likely because GTSE can stitch information across trajectories:
Theory vs Results
We acknowledge your concern. We want to clarify that we do not claim that higher available representation rank leads to better performance. We claim that, under a bilinear parameterization (Proposition 1, Sec 6.2), integrating the time score is equivalent to an estimator with higher available rank. In this regard, our experiments in Appendix E.2 show that the relationship between available rank and downstream performance depends on the structure of the occupancy measure of the dataset and task (Tables 10, 11 and 12), under bilinear parameterization.
In summary, our theory tries to explain what additional capacity GTSE provides and our experiments try to understand when that capacity translates to additional performance. Kindly let us know if you have any further questions.
Planning baseline
To address this concern, we compare GTSE to F-MPC (Flow-Model Predictive Control), used as a model-based planning baseline in [1]. F-MPC is a flow-matching version of Diffusion Model Predictive Control [2], and uses a flow-matching policy and learned model to propose plans which are ranked by an action-chunked value function. We tuned F-MPC by sweeping over the choice of critic (MC vs TD) and horizon (10 as used in [1] vs 32). The results added to Appendix E.4 indicate that GTSE's occupancy density estimator is able to better model long-horizon temporal effects of actions:
[1] Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. "Scalable Offline Model-Based RL with Action Chunking." arXiv preprint, 2025.
[2] Guangyao Zhou et al. "Diffusion Model Predictive Control." arXiv:2410.05364, 2024.
Inference cost not clear
We have updated Table 8 with the 3rd row showing the full inference time for selecting 16 actions during inference. We find that the ranking between algorithms is roughly similar to row 2 (critic inference time) and that GTSE and its variants remain comparable in inference time to the baselines.
We thank you again for your review. We hope that our responses adequately address your concerns and we are ready to answer any further questions.