- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DPDP, which combines a value-informed diffusion policy with a diffusion planner. The main idea is to replace the behavior prior of the planner's inverse-dynamics model with a value-informed diffusion policy using a Bayesian formulation. Based on this idea, the authors propose two methods: DPDP-inf, which combines pretrained diffusion models during inference, and DPDP-ft, which fine-tunes the inverse-dynamics model using preference alignment. The experimental results on D4RL show that the proposed methods generally improve the planning baseline, especially in long-horizon navigation tasks.
Critical strengths
- Easy to follow.
The paper is well written, and the initial motivation is easy to follow. The mathematical derivations are also understandable and logically organized.
- The motivation is intuitively reasonable.
The motivation behind the proposed method is intuitive and reasonable.
Critical weaknesses
Overall, my major concerns are related to the experimental results and evaluation design. I would be willing to raise my score if these concerns were adequately addressed.
- The experimental results do not sufficiently support the strengths of the proposed method.
I understand that planning-based algorithms in offline RL generally perform worse than reactive policy algorithms. However, the proposed method requires a pretrained DQL policy, yet its performance is lower than that of DQL itself in the MuJoCo setting. If DQL already achieves better performance, it is unclear why one should use the proposed method instead of simply using DQL. Furthermore, as discussed in the Introduction, diffusion policies are limited by their lack of explicit planning. If the proposed method aims to address this limitation by combining diffusion policies with explicit planning, why does it still underperform DQL in the MuJoCo setting?
- More ablation studies are needed to verify the proposed method.
The proposed method relies on both the inverse-dynamics denoiser and the DQL denoiser. Its theoretical justification also depends on the assumption that these models are sufficiently accurate. I understand that approximation errors are unavoidable in many diffusion-based methods. However, the proposed method is subject to approximation errors in both models. Therefore, understanding which component has a greater influence on the final performance is crucial. For example, it would be helpful to investigate the contribution of each component by using models trained for fewer optimization steps. Such experiments could reveal which component has a greater impact on the overall performance. Additionally, evaluating the method using alternative algorithms to train would provide further evidence of its generalizability.
- The evaluation is limited to relatively simple and conventional benchmarks.
Although the proposed method generally improves upon the baseline, the experiments are limited to relatively old benchmarks. The paper does not include evaluations on other challenging benchmarks, such as Adroit and OGBench [1]. Evaluations on these benchmarks would provide stronger evidence for the effectiveness and generalizability of the proposed method.
Minor Suggestions
- Is "training-free" the appropriate terminology?
While I understand that the training-free method does not require an additional training stage, it still relies on multiple pretrained models. Therefore, the term "training-free" might be confusing to readers. Would it be more appropriate to use an alternative term?
- Table readability.
Tables 1 and 2 contain too many numerical results, and the font size is too small, making them difficult to read. I suggest excluding some older baselines from the main tables and moving the complete results to the appendix.
[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
- Could you evaluate the proposed method on more diverse benchmarks, such as Adroit and OGBench?
- Could you further justify the use of the DQL denoiser, given that the proposed method underperforms DQL in some settings?
- Could you conduct additional ablation studies to investigate how the quality of the pretrained models affects the performance of the proposed method?
14 comments
Thank you for your review! (2/3)
[W2, Q3] Sensitivity to the quality of the pretrained components.
Response. Thank you for suggesting this ablation!
We deliberately used undertrained checkpoints to test whether DPDP retains useful performance when one component is substantially weakened. In each experiment, we replaced one component with an earlier checkpoint from the same training run, while keeping the other component and the planner unchanged. We then re-ran DPDP-inf and retrained the DPDP-ft student.
The reductions are substantial: Entries below are normalized returns; the full configurations are those of Tables 1–2.
Undertrained Q-denoiser (per-task results in Table A15)
| Method | MuJoCo | AntMaze | Maze2D |
|---|---|---|---|
| DQL+ with undertrained Q-denoiser | 74.8 | 50.5 | 105.2 |
| DPDP-inf | 83.5 | 73.0 | 164.3 |
| DPDP-ft | 81.5 | 80.5 | 168.8 |
Despite the substantial degradation of the standalone value policy, both DPDP variants remain effective. The undertrained Q-denoiser reduces DQL+ by 14.0–53.0 points, whereas DPDP-inf loses at most 9.4 points and DPDP-ft at most 5.2. Thus, the degraded value prior does not cause a comparable degradation of the combined system.
Undertrained inverse dynamics (per-task results in Table A16)
| Method | MuJoCo | AntMaze | Maze2D |
|---|---|---|---|
| DV+ with undertrained IDM | 81.6 | 73.0 | 78.0 |
| DPDP-inf | 81.7 | 76.5 | 143.8 |
| DPDP-ft | 80.3 | 78.0 | 167.0 |
The strongest stress case is Maze2D: weakening the IDM reduces DV+ from 159.5 to 78.0, while DPDP-inf and DPDP-ft still achieve 143.8 and 167.0. The combined executor therefore retains useful performance even when the underlying IDM is substantially degraded.
Across all tested perturbations, both variants retain at least 85% of their respective full-model normalized scores. Sensitivity is task-dependent, with the largest degradation occurring when the IDM is weakened on Maze2D. These results demonstrate that DPDP can tolerate substantial degradation of one component while the other remains unchanged; its empirical benefits do not require both components to reach their full training budgets.
We also trained the value policy with EDP [3] instead of Diffusion-QL. DPDP-inf uses this policy as its prior and DPDP-ft as its teacher (family means, normalized score):
Value policy trained with EDP (per-task results in Table A18)
| Family | EDP alone | DPDP-inf (EDP + DV+) | DPDP-ft (EDP + DV+) |
|---|---|---|---|
| MuJoCo (9) | 88.3 | 83.8 | 82.6 |
| AntMaze (4) | 70.5 | 77.5 | 83.5 |
| Maze2D (3) | 116.2 | 163.0 | 165.1 |
With an EDP-trained prior, DPDP again lifts navigation well beyond the value policy alone: 83.5 vs 70.5 on AntMaze and 165.1 vs 116.2 on Maze2D. The transfer therefore does not depend on Diffusion-QL.
Action: We have added the component-quality study and its experimental settings to the appendix (Appendix E.7, Tables A14–A16), with a summary in Sec. 4.2, and the EDP study with its per-task results (Appendix E.10, Table A18).
Thank you for your review! (3/3)
[W3, Q1] Additional benchmarks: Adroit and OGBench.
Response. Thank you for suggesting these benchmarks!
On Adroit, DPDP-inf has the best average of all methods (95.7); on five OGBench navigation tasks, DPDP-inf with a flow Q-learning prior raises a compositional diffusion planner from 86.4% to 94.8% average success, above both the value policy alone and the planner's original controller.
Adroit normalized return
| Task | DQL | DV | DV+ | DPDP-inf | DPDP-ft |
|---|---|---|---|---|---|
| pen-human | 74.9 | 72.6 | 70.2 | 77.8 (+7.6) | 63.4 (−6.8) |
| pen-cloned | 57.6 | 73.9 | 72.4 | 74.1 (+1.7) | 75.6 (+3.2) |
| pen-expert | 146.3 | 118.8 | 120.6 | 135.2 (+14.6) | 118.7 (−1.9) |
| Average | 92.9 | 88.4 | 87.7 | 95.7 (+8.0) | 85.9 (−1.8) |
DPDP-inf improves DV+ on all three tasks (+7.6, +1.7, +14.6), and its average exceeds DQL's (95.7 vs 92.9). DPDP-ft achieves the best score on pen-cloned (75.6). We ran the three pen tasks within the rebuttal period.
For OGBench [4], we build on established, publicly available training recipes for the RCD planner [5] and the GC-FQL value policy [6]. The rebuttal time and compute budget did not permit extensive OGBench-specific tuning of DV and DQL, and the existing tuned recipes provide strong initial components for evaluating DPDP's contribution. We keep the RCD planner checkpoint fixed and compare its diffusion-IDM executor with the DPDP-inf and DPDP-ft counterparts, using GC-FQL as the value prior and teacher. This evaluates whether DPDP improves an existing planning system and extends beyond the DV–DQL combination of our original experiments; the original RCD MLP controller and the standalone GC-FQL policy are included as reference baselines. As in Tables 1–2, entries are success rates (%), DPDP entries show the change from the diffusion-IDM executor in parentheses, and the best entry per row is bold:
OGBench
| Task | GC-FQL alone | RCD + MLP controller | RCD + diffusion IDM | RCD + DPDP-inf | RCD + DPDP-ft |
|---|---|---|---|---|---|
| point-medium | 100 | 100 | 100 | 100 (+0) | 100 (+0) |
| point-large | 100 | 100 | 100 | 100 (+0) | 100 (+0) |
| point-giant | 2 | 90 | 80 | 96 (+16) | 88 (+8) |
| ant-medium | 94 | 92 | 90 | 88 (−2) | 92 (+2) |
| ant-large | 92 | 86 | 62 | 90 (+28) | 80 (+18) |
| Average | 77.6 | 93.6 | 86.4 | 94.8 (+8.4) | 92.0 (+5.6) |
Relative to its base executor (the diffusion IDM), DPDP-inf adds 8.4 points of mean success and DPDP-ft 5.6, with the largest gains on the hardest tasks (point-giant +16 and ant-large +28 for DPDP-inf). DPDP-inf also exceeds the planner's original MLP controller (94.8 vs 93.6) and far exceeds the value policy alone on point-giant (96 vs 2). Because the prior here comes from flow Q-learning rather than Diffusion-QL, this also covers the alternative value learner you asked about in W2.
Action: We have added both benchmarks to the revision (Appendix F, Tables A19–A20), with the setup in Sec. 4.
[Minor] Terminology and table readability.
Response. Thank you very much for bringing this up!
We agree with both suggestions: "inference-time composition" describes DPDP-inf more accurately, since it adds no training stage beyond its pretrained components, and the full comparison with standard errors remains available in Table A3.
Action: We have replaced "training-free" with "inference-time" throughout, and the main tables now keep only the base models (DQL+, DV+), both variants and representative prior methods of each family, without the Gaussian policies (now in Table A3), so that they use a larger font.
Thank you again for the actionable review!
Following your suggestions, we ran the component-quality ablation, two new benchmarks and an alternative value learner.
We have genuinely enjoyed this discussion with you, which has definitely strengthened our manuscript!
We are confident and sincerely hope that these results resolve the concerns you identified as decisive for your assessment. If so, we would be grateful if you would consider raising your score.
Best regards,
Authors
References
[1] Bridging RL Theory and Practice with the Effective Horizon, NeurIPS 2023.
[2] On the Role of Planning in Model-Based Deep Reinforcement Learning, ICLR 2021.
[3] Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.
[4] OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.
[5] Refining Compositional Diffusion for Reliable Long-Horizon Planning, NeurIPS 2026.
[6] Flow Q-Learning, ICML 2025.
Thank you for your review! (1/3)
Dear Reviewer RcuF,
Thank you for your detailed review and for stating concretely which results would strengthen the paper!
We appreciate that you found the paper well written and easy to follow, the motivation intuitive and reasonable, and the derivations understandable and logically organized. We ran each experiment you requested and report it below with its setting.
Response. Thank you for raising this point!
DPDP makes planning a more favorable choice across task regimes: 1) it mitigates the planner's disadvantage where reactive policies are stronger, while 2) improving performance where planning and value-informed control are complementary.
MuJoCo locomotion illustrates the first case. These tasks provide dense, stepwise rewards, unlike the sparse, goal-reaching rewards of navigation; lookahead pays off mainly when the effective horizon is long [1], and shallow search matches deeper planning except in the hardest reasoning tasks [2]. Here the base planner is substantially weaker than DQL+: 83.4 versus 88.8. DPDP transfers the value policy's action preferences into the planner's executor, raising performance to 84.5 with DPDP-inf and 84.3 with DPDP-ft (+1.1 ± 0.25 and +0.9 ± 0.25, more than three standard errors). Thus, although it does not eliminate the planner's disadvantage, it bridges part of the existing gap between planning and reactive control. The DQL denoiser serves precisely this purpose: it brings value-informed action selection into the planning framework, supplying the action-level preference pQ/pB rather than the action itself; hence a much weaker Q-denoiser affects DPDP far less than DQL itself (W2, Q3).
In practice, we may not know in advance whether a new task will resemble locomotion, where direct policy execution is stronger, or navigation, where explicit planning and value-informed execution work better together. DPDP provides a more favorable way to retain planning across these settings. On locomotion, it brings the planner closer to the stronger reactive policy. On AntMaze and Maze2D, it improves over both components: DPDP-ft achieves 85.7 versus 81.4 for DQL+ and 81.2 for DV+ on AntMaze; DPDP-inf achieves 168.9 versus 158.2 and 159.5 on Maze2D. On OGBench point-giant, DPDP-inf reaches 96% success where its value prior alone reaches 2% (W3, Q1).
Our results therefore demonstrate two benefits: reducing the planner’s performance deficit in an unfavorable regime, and strengthening/outperforming its performance where the combination is advantageous.
Action: We have stated this scope and its explanation explicitly in the results section (Sec. 4.1).