- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether the contact outcome of a trajectory diffusion model can be judged midway through denoising. It trains probes on intermediate denoising states to read out contact information, and compares two uses of these probes: guiding trajectories and discarding samples early. Experiments in simulated scenes show that guidance helps while early discarding does not pay off. However, the conclusions are validated only in simple simulated scenes, so their generality is unclear.
Critical strengths
-
The paper gives an interesting experimental finding: mid-denoising judgments about contact can be used to guide trajectories, but not to discard samples early.
-
The criterion for when early discarding pays off, and the analysis method of copying the same intermediate state and finishing it with different noise, are simple and practical. They are useful references for "mid-way pruning" designs in diffusion models and for understanding the denoising process.
Critical weaknesses
-
The experiments are rigorous but conducted only in simulated scenes. It is unclear whether the conclusions extend to larger scenes, pretrained models, or real planning tasks.
-
The writing is hard to read. There are many self-coined terms ("mothers", "frozen/secondary envelope", "families", "uncharged oracle", etc.), key settings are scattered across the main text and multiple appendices, and the main experiments and conclusions are hard to grasp from the abstract alone.
-
The main text has too few figures and tables, while the evidence supporting the core conclusions is mostly in the appendix. I suggest condensing the prose and moving the most important figures and tables into the main text.
Questions for authors
pls see weaknesses.
Note: If the authors can add validation in real application scenarios, or convince me that the current experiments are sufficient, I will raise my score.
5 comments
Response to Reviewer 8mLM (2/2): terms, settings, figures and tables (W2, W3)
W2. "The writing is hard to read. There are many self-coined terms ("mothers", "frozen/secondary envelope", "families", "uncharged oracle", etc.), key settings are scattered across the main text and multiple appendices, and the main experiments and conclusions are hard to grasp from the abstract alone."
We revised all three points.
Terms. Every coined term is replaced throughout the paper:
| Submission | Revision |
|---|---|
| mother | intermediate state |
| branch | continuation |
| family (of comparisons) | "Bonferroni-adjusted over k comparisons" |
| frozen / secondary envelope | removed: success is verified by executing the generated initial state in the simulator |
| uncharged oracle | empirical probability from sixteen continuations, its cost reported separately |
| NFE | denoiser updates |
Settings in one place. The new Table 1 lists every main study with its models, inference units, comparison and outcome.
Abstract. The abstract now states the main experiments with their numbers: guidance raises simulator-verified success by 12.7–31.2 points where random steps change it by at most 1.2; within fixed targets a probe-scored gate yields 0.98–1.02 times the no-gate yield where an empirical probability, its own cost set aside, raises it by 6–9.5%; and the released Push-T policy shows the same division.
W3. "The main text has too few figures and tables, while the evidence supporting the core conclusions is mostly in the appendix. I suggest condensing the prose and moving the most important figures and tables into the main text."
We followed this suggestion. Each core conclusion now has its evidence in the main text:
| Core conclusion | Main-text evidence |
|---|---|
| Readout error, not residual randomness, dominates from mid-schedule | Figure 1 |
| Probe directions transfer to a geometric diagnostic | Figure 2 |
| Guidance raises simulator-verified success | Figure 3 (new): all twelve held-out cells and the three- to five-contact requests, each with its random control |
| Early rejection is limited by ranking | Figure 4 (new): gate yield within fixed targets, probe against empirical probability |
| The findings hold on a released policy | Table 2 (new), Section 4.6 |
| Studies, units and outcomes | Table 1 (new) |
The prose was condensed to make room: the main text now carries four figures and two tables within ten pages. Sections 4.1–4.5, Proposition 1 and Figures 1–2 keep their numbers.
Thanks for the rebuttal and new experiments. The paper is improved, but I still have concerns about the generality and reliability of the results.
The Push-T experiment is helpful, but it's still a simple simulated task. More importantly, the code inconsistencies pointed out by RF47 are quite concerning. Although the authors have corrected them, I'm still not fully confident about the reproducibility of the results.
Without these code issues, I would be happy to raise my score to 3.
I was considering a score between 2 and 3, but since that option isn't available, I'll keep my original score for now.
Response to Reviewer 8mLM (1/2): a released pretrained policy on a planning task (W1)
We thank the reviewer for the clear summary and for stating what would change the assessment.
We added this validation: a public pretrained model on a manipulation planning task (new Section 4.6, Table 2, Appendix L). We applied the same measurements to the released Diffusion Policy for Push-T (Chi et al., RSS 2023), with the authors' released checkpoint: a 100-update DDPM that plans a 16-step trajectory of actions and observations, conditions on the latest observations and executes eight actions per chunk in closed loop. It is a model we did not train, with its own horizon, noise schedule and conditioning. The contact events are whether the pusher touches the bar or the stem of the T during the executed chunk, and every outcome is measured by executing the actions. All four findings of the paper reappear (readout and steering on 1,024 held-out decision states from 256 episodes):
The probe's direction is what holds the pusher off a requested part of the T: requested absence rises by +13.7 (bar) and +10.9 (stem) points, which is +12.5 [8.2, 17.0] and +10.2 [6.5, 14.3] points more than random steps of the same per-update norm (Table 2 lists every arm).
Why the evidence is sufficient for the paper's claims.
The reviewer noted that our continuation analysis is a useful reference for "mid-way pruning" designs. To make it easy to reuse, the supplementary material contains the code to rerun every experiment, including the pretrained-policy study, and our trained denoisers and probes are available at an anonymous link: https://osf.io/xbjwq/overview?view_only=417b35cca1f145a9a438cdea02d040ce (also in Appendix J).
We hope the validation on a released pretrained policy and the four points above convince the reviewer that the current experiments are sufficient, and we would be grateful if the reviewer would reconsider the score.