acceptodds
Under review as a conference paper at ICLR 2027

DPDP: Diffusion Policy-steered Diffusion Planner in Offline Reinforcement Learning

Abstract

In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline behavior. Each is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.

Primary area:
reinforcement learning
Keywords:
Offline reinforcement learning, diffusion planner

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RcuFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DPDP, which combines a value-informed diffusion policy with a diffusion planner. The main idea is to replace the behavior prior of the planner's inverse-dynamics model with a value-informed diffusion policy using a Bayesian formulation. Based on this idea, the authors propose two methods: DPDP-inf, which combines pretrained diffusion models during inference, and DPDP-ft, which fine-tunes the inverse-dynamics model using preference alignment. The experimental results on D4RL show that the proposed methods generally improve the planning baseline, especially in long-horizon navigation tasks.

Critical strengths
  1. Easy to follow.

The paper is well written, and the initial motivation is easy to follow. The mathematical derivations are also understandable and logically organized.

  1. The motivation is intuitively reasonable.

The motivation behind the proposed method is intuitive and reasonable.

Critical weaknesses

Overall, my major concerns are related to the experimental results and evaluation design. I would be willing to raise my score if these concerns were adequately addressed.

  1. The experimental results do not sufficiently support the strengths of the proposed method.

I understand that planning-based algorithms in offline RL generally perform worse than reactive policy algorithms. However, the proposed method requires a pretrained DQL policy, yet its performance is lower than that of DQL itself in the MuJoCo setting. If DQL already achieves better performance, it is unclear why one should use the proposed method instead of simply using DQL. Furthermore, as discussed in the Introduction, diffusion policies are limited by their lack of explicit planning. If the proposed method aims to address this limitation by combining diffusion policies with explicit planning, why does it still underperform DQL in the MuJoCo setting?

  1. More ablation studies are needed to verify the proposed method.

The proposed method relies on both the inverse-dynamics denoiser ϵϕ\epsilon_\phi and the DQL denoiser. Its theoretical justification also depends on the assumption that these models are sufficiently accurate. I understand that approximation errors are unavoidable in many diffusion-based methods. However, the proposed method is subject to approximation errors in both models. Therefore, understanding which component has a greater influence on the final performance is crucial. For example, it would be helpful to investigate the contribution of each component by using models trained for fewer optimization steps. Such experiments could reveal which component has a greater impact on the overall performance. Additionally, evaluating the method using alternative algorithms to train pQp_Q would provide further evidence of its generalizability.

  1. The evaluation is limited to relatively simple and conventional benchmarks.

Although the proposed method generally improves upon the baseline, the experiments are limited to relatively old benchmarks. The paper does not include evaluations on other challenging benchmarks, such as Adroit and OGBench [1]. Evaluations on these benchmarks would provide stronger evidence for the effectiveness and generalizability of the proposed method.

Minor Suggestions

  1. Is "training-free" the appropriate terminology?

While I understand that the training-free method does not require an additional training stage, it still relies on multiple pretrained models. Therefore, the term "training-free" might be confusing to readers. Would it be more appropriate to use an alternative term?

  1. Table readability.

Tables 1 and 2 contain too many numerical results, and the font size is too small, making them difficult to read. I suggest excluding some older baselines from the main tables and moving the complete results to the appendix.

[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Could you evaluate the proposed method on more diverse benchmarks, such as Adroit and OGBench?
  2. Could you further justify the use of the DQL denoiser, given that the proposed method underperforms DQL in some settings?
  3. Could you conduct additional ablation studies to investigate how the quality of the pretrained models affects the performance of the proposed method?
14 comments
AuthorsOpenReview ↗

Thank you for your review! (1/3)

Dear Reviewer RcuF,

Thank you for your detailed review and for stating concretely which results would strengthen the paper!

We appreciate that you found the paper well written and easy to follow, the motivation intuitive and reasonable, and the derivations understandable and logically organized. We ran each experiment you requested and report it below with its setting.

[W1, Q2] Role of DPDP where DQL is stronger, and use of the DQL denoiser.

Response. Thank you for raising this point!

DPDP makes planning a more favorable choice across task regimes: 1) it mitigates the planner's disadvantage where reactive policies are stronger, while 2) improving performance where planning and value-informed control are complementary.

MuJoCo locomotion illustrates the first case. These tasks provide dense, stepwise rewards, unlike the sparse, goal-reaching rewards of navigation; lookahead pays off mainly when the effective horizon is long [1], and shallow search matches deeper planning except in the hardest reasoning tasks [2]. Here the base planner is substantially weaker than DQL+: 83.4 versus 88.8. DPDP transfers the value policy's action preferences into the planner's executor, raising performance to 84.5 with DPDP-inf and 84.3 with DPDP-ft (+1.1 ± 0.25 and +0.9 ± 0.25, more than three standard errors). Thus, although it does not eliminate the planner's disadvantage, it bridges part of the existing gap between planning and reactive control. The DQL denoiser serves precisely this purpose: it brings value-informed action selection into the planning framework, supplying the action-level preference pQ/pBp_Q/p_B rather than the action itself; hence a much weaker Q-denoiser affects DPDP far less than DQL itself (W2, Q3).

In practice, we may not know in advance whether a new task will resemble locomotion, where direct policy execution is stronger, or navigation, where explicit planning and value-informed execution work better together. DPDP provides a more favorable way to retain planning across these settings. On locomotion, it brings the planner closer to the stronger reactive policy. On AntMaze and Maze2D, it improves over both components: DPDP-ft achieves 85.7 versus 81.4 for DQL+ and 81.2 for DV+ on AntMaze; DPDP-inf achieves 168.9 versus 158.2 and 159.5 on Maze2D. On OGBench point-giant, DPDP-inf reaches 96% success where its value prior alone reaches 2% (W3, Q1).

Our results therefore demonstrate two benefits: reducing the planner’s performance deficit in an unfavorable regime, and strengthening/outperforming its performance where the combination is advantageous.

Action: We have stated this scope and its explanation explicitly in the results section (Sec. 4.1).

AuthorsOpenReview ↗

Thank you for your review! (2/3)

[W2, Q3] Sensitivity to the quality of the pretrained components.

Response. Thank you for suggesting this ablation!

We deliberately used undertrained checkpoints to test whether DPDP retains useful performance when one component is substantially weakened. In each experiment, we replaced one component with an earlier checkpoint from the same training run, while keeping the other component and the planner unchanged. We then re-ran DPDP-inf and retrained the DPDP-ft student.

The reductions are substantial: Entries below are normalized returns; the full configurations are those of Tables 1–2.

Undertrained Q-denoiser (per-task results in Table A15)

MethodMuJoCoAntMazeMaze2D
DQL+ with undertrained Q-denoiser74.850.5105.2
DPDP-inf83.573.0164.3
DPDP-ft81.580.5168.8

Despite the substantial degradation of the standalone value policy, both DPDP variants remain effective. The undertrained Q-denoiser reduces DQL+ by 14.0–53.0 points, whereas DPDP-inf loses at most 9.4 points and DPDP-ft at most 5.2. Thus, the degraded value prior does not cause a comparable degradation of the combined system.

Undertrained inverse dynamics (per-task results in Table A16)

MethodMuJoCoAntMazeMaze2D
DV+ with undertrained IDM81.673.078.0
DPDP-inf81.776.5143.8
DPDP-ft80.378.0167.0

The strongest stress case is Maze2D: weakening the IDM reduces DV+ from 159.5 to 78.0, while DPDP-inf and DPDP-ft still achieve 143.8 and 167.0. The combined executor therefore retains useful performance even when the underlying IDM is substantially degraded.

Across all tested perturbations, both variants retain at least 85% of their respective full-model normalized scores. Sensitivity is task-dependent, with the largest degradation occurring when the IDM is weakened on Maze2D. These results demonstrate that DPDP can tolerate substantial degradation of one component while the other remains unchanged; its empirical benefits do not require both components to reach their full training budgets.

We also trained the value policy with EDP [3] instead of Diffusion-QL. DPDP-inf uses this policy as its prior and DPDP-ft as its teacher (family means, normalized score):

Value policy trained with EDP (per-task results in Table A18)

FamilyEDP aloneDPDP-inf (EDP + DV+)DPDP-ft (EDP + DV+)
MuJoCo (9)88.383.882.6
AntMaze (4)70.577.583.5
Maze2D (3)116.2163.0165.1

With an EDP-trained prior, DPDP again lifts navigation well beyond the value policy alone: 83.5 vs 70.5 on AntMaze and 165.1 vs 116.2 on Maze2D. The transfer therefore does not depend on Diffusion-QL.

Action: We have added the component-quality study and its experimental settings to the appendix (Appendix E.7, Tables A14–A16), with a summary in Sec. 4.2, and the EDP study with its per-task results (Appendix E.10, Table A18).

AuthorsOpenReview ↗

Thank you for your review! (3/3)

[W3, Q1] Additional benchmarks: Adroit and OGBench.

Response. Thank you for suggesting these benchmarks!

On Adroit, DPDP-inf has the best average of all methods (95.7); on five OGBench navigation tasks, DPDP-inf with a flow Q-learning prior raises a compositional diffusion planner from 86.4% to 94.8% average success, above both the value policy alone and the planner's original controller.

Adroit normalized return

TaskDQLDVDV+DPDP-infDPDP-ft
pen-human74.972.670.277.8 (+7.6)63.4 (−6.8)
pen-cloned57.673.972.474.1 (+1.7)75.6 (+3.2)
pen-expert146.3118.8120.6135.2 (+14.6)118.7 (−1.9)
Average92.988.487.795.7 (+8.0)85.9 (−1.8)

DPDP-inf improves DV+ on all three tasks (+7.6, +1.7, +14.6), and its average exceeds DQL's (95.7 vs 92.9). DPDP-ft achieves the best score on pen-cloned (75.6). We ran the three pen tasks within the rebuttal period.

For OGBench [4], we build on established, publicly available training recipes for the RCD planner [5] and the GC-FQL value policy [6]. The rebuttal time and compute budget did not permit extensive OGBench-specific tuning of DV and DQL, and the existing tuned recipes provide strong initial components for evaluating DPDP's contribution. We keep the RCD planner checkpoint fixed and compare its diffusion-IDM executor with the DPDP-inf and DPDP-ft counterparts, using GC-FQL as the value prior and teacher. This evaluates whether DPDP improves an existing planning system and extends beyond the DV–DQL combination of our original experiments; the original RCD MLP controller and the standalone GC-FQL policy are included as reference baselines. As in Tables 1–2, entries are success rates (%), DPDP entries show the change from the diffusion-IDM executor in parentheses, and the best entry per row is bold:

OGBench

TaskGC-FQL aloneRCD + MLP controllerRCD + diffusion IDMRCD + DPDP-infRCD + DPDP-ft
point-medium100100100100 (+0)100 (+0)
point-large100100100100 (+0)100 (+0)
point-giant2908096 (+16)88 (+8)
ant-medium94929088 (−2)92 (+2)
ant-large92866290 (+28)80 (+18)
Average77.693.686.494.8 (+8.4)92.0 (+5.6)

Relative to its base executor (the diffusion IDM), DPDP-inf adds 8.4 points of mean success and DPDP-ft 5.6, with the largest gains on the hardest tasks (point-giant +16 and ant-large +28 for DPDP-inf). DPDP-inf also exceeds the planner's original MLP controller (94.8 vs 93.6) and far exceeds the value policy alone on point-giant (96 vs 2). Because the prior here comes from flow Q-learning rather than Diffusion-QL, this also covers the alternative value learner you asked about in W2.

Action: We have added both benchmarks to the revision (Appendix F, Tables A19–A20), with the setup in Sec. 4.

[Minor] Terminology and table readability.

Response. Thank you very much for bringing this up!

We agree with both suggestions: "inference-time composition" describes DPDP-inf more accurately, since it adds no training stage beyond its pretrained components, and the full comparison with standard errors remains available in Table A3.

Action: We have replaced "training-free" with "inference-time" throughout, and the main tables now keep only the base models (DQL+, DV+), both variants and representative prior methods of each family, without the Gaussian policies (now in Table A3), so that they use a larger font.


Thank you again for the actionable review!

Following your suggestions, we ran the component-quality ablation, two new benchmarks and an alternative value learner.

We have genuinely enjoyed this discussion with you, which has definitely strengthened our manuscript!

We are confident and sincerely hope that these results resolve the concerns you identified as decisive for your assessment. If so, we would be grateful if you would consider raising your score.

Best regards,

Authors


References

[1] Bridging RL Theory and Practice with the Effective Horizon, NeurIPS 2023.

[2] On the Role of Planning in Model-Based Deep Reinforcement Learning, ICLR 2021.

[3] Efficient Diffusion Policies for Offline Reinforcement Learning, NeurIPS 2023.

[4] OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.

[5] Refining Compositional Diffusion for Reliable Long-Horizon Planning, NeurIPS 2026.

[6] Flow Q-Learning, ICML 2025.

Reviewer_41ikOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors aim to combine the strengths of a diffusion policy that favors high-return actions and a diffusion planner that generates long-horizon plans. To this end, they propose Diffusion Policy–Steered Diffusion Planner (DPDP). Specifically, they derive a transition-conditioned action distribution using Bayes’ rule, combining an inverse-dynamics model learned from offline transitions with the ratio of a critic-optimized diffusion policy to an offline behavior policy. A DV planner generates state trajectories, and the planned next state serves as the local goal for the DPDP executor. The experimental results suggest that the method benefits from combining value-informed action selection with long-horizon planning.

Critical strengths
  • The idea of combining a diffusion planner with a diffusion policy is interesting, as is the proposed approach of replacing the behavior prior with a value-optimized policy prior.

  • The paper provides theoretical analysis of the proposed method, offering a strong foundation for its claims.

Critical weaknesses
  • The experimental results are mixed. On similar locomotion tasks, DPDP-inf and DPDP-ft perform differently across dataset expertise levels, and neither consistently outperforms the base DQL methods on the evaluated tasks.

  • The method only steers the executor toward the preferences of a value-optimized diffusion policy; it does not guarantee that the resulting policy maximizes either short-term or long-term return.

Questions for authors

Q1. Compared with a standard Diffuser that uses return guidance, what advantages does DPDP offer in selecting high-return, behavior-supported actions?

Q2. Since the method trains multiple diffusion models, could the authors compare its training and sampling times with those of a diffusion policy and Diffusion Veteran?

2 comments
AuthorsOpenReview ↗

Title: Thank you for your review! (1/2)

Dear Reviewer 41ik,

Thank you for your thoughtful and constructive review!

We appreciate that you found the idea of combining a diffusion planner with a diffusion policy interesting, in particular replacing the behavior prior with a value-optimized policy prior, and that you saw the theoretical analysis as a strong foundation for our claims.

We answer each weakness and question below.

[W1] Results across suites, realizations and the DQL baselines.

Response. Thank you for raising this point!

Our goal is not to replace DQL but to make planning a more favorable choice across task regimes: 1) to mitigate the planner's disadvantage where a reactive policy is stronger, and 2) to improve performance where planning and value-informed control are complementary. The locomotion results should be read in this light.

On locomotion, planning is not expected to help. The rewards are dense and stepwise, so acting on a good value estimate step by step is already close to optimal: lookahead pays off mainly when the effective horizon is long [1], and shallow search matches deeper planning except in the hardest reasoning tasks [2]. Every planner in Table 2 therefore trails DQL+, and DPDP's role there is to mitigate this disadvantage, which both realizations do (DV+ 83.4 → 84.5 and 84.3). The two realizations approximate the same clean-action distribution in different ways (Sec. 3.3), so small differences between them on individual datasets are expected; what matters is that both consistently move the planner toward the reactive policy.

In practice, one rarely knows in advance whether a new task resembles locomotion, where direct policy execution is stronger, or navigation, where planning and value-informed execution work better together. DPDP makes keeping the planner a favorable choice in both cases: it narrows the gap where planning is unnecessary, and where planning matters it improves over both components (e.g., 85.7 vs 81.4 for DQL+ and 81.2 for DV+ on AntMaze).

Our results therefore show two benefits: a smaller deficit for planning in an unfavorable regime, and stronger performance where the combination is advantageous.

Action: We have stated this scope (improving the base planner, with the largest gains on long-horizon navigation) and the suite-level gains with their standard errors explicitly in the results section (Sec. 4.1), and reported the comparison of the two realizations in the appendix (Appendix E.9).

[W2] Optimality of the executed policy.

Response. Thank you for raising this distinction!

DPDP combines value-informed action selection with explicit planning, and β\beta controls how strongly the value preference changes execution. Like Diffusion-QL [3], it uses a behavior-regularized value policy: the critic estimates accumulated discounted future return, while behavior cloning encourages actions near the offline distribution. The planner complements this estimate with an explicit future state trajectory.

At β=1\beta=1, the clean target is p⋆(a∣s,g)∝P(g∣s,a) pQ(a∣s).p^\star(a\mid s,g)\propto P(g\mid s,a)\,p_Q(a\mid s).

The transition likelihood favors actions compatible with the plan, while the value prior changes their relative preference. Smaller β\beta strengthens the value-informed reweighting; larger β\beta keeps execution closer to the original transition-conditioned executor. Thus, β\beta provides control over reliance on the learned value preference (Appendix E.2 sweeps it). Both components account for future outcomes through different mechanisms; β\beta controls transfer strength, not the temporal horizon. This makes DPDP flexible: it can lean on the value preference where the critic is reliable and stay closer to the planner's executor where it is not.

To answer your question directly: our theory establishes the optimal regularized execution distribution under these modeled preferences. This guarantee does not imply globally optimal environment return, which also depends on critic accuracy, model approximation, and the proposed plan; the same holds for all behavior-regularized methods.

Action: We have stated explicitly in Sec. 3.2 what the executor optimizes, namely the value policy's preference among plan-compatible actions, and how it relates to return.

AuthorsOpenReview ↗

Thank you for your review! (2/2)

[Q1] Comparison with return-guided Diffuser.

Response. Thank you for this excellent question!

DPDP offers three advantages over return guidance: it applies value where the action is chosen, its design is flexible and modular, and β\beta makes the transfer controllable.

Value where the action is chosen. DPDP applies value at the action that executes the plan, using a critic-trained policy that remains behavior-supported, whereas return guidance pushes the whole trajectory along a learned return gradient. Trajectory guidance changes which plan is sampled but does not rank the actions that realize the same planned transition, which is exactly where DPDP acts: among actions compatible with s^t+1\hat s_{t+1}, pQp_Q upweights those with higher estimated return. Because DV+ already selects among candidate plans with its critic (MCSS [5]), DPDP's gain over DV+ is what action-level value adds on top of trajectory-level value. The best DPDP variant exceeds Diffuser on every suite in Tables 1–2 (84.5 vs 77.5 on MuJoCo, where Diffuser uses return guidance [4]; 85.7 vs 13.3 on AntMaze; 168.9 vs 119.5 on Maze2D).

Flexible, modular design. Value can be added to an already-trained planner without retraining it or changing its sampler: DPDP-inf composes the pretrained models at inference time, and DPDP-ft fine-tunes only the inverse dynamics. The value policy is likewise interchangeable; in our experiments it comes from Diffusion-QL, EDP or flow Q-learning.

Controllable transfer. β\beta adds further flexibility. As discussed in W2, it sets how strongly the action-level value preference reweights the plan-compatible actions, from execution that stays close to the planned transition (large β\beta) to strongly value-driven execution (small β\beta), so the balance between the planner's foresight and the critic's action-level judgment can be tuned per task; with DPDP-inf, this needs no retraining.

[Q2] Training and sampling time.

Response. Thank you for suggesting this comparison!

DPDP-inf adds no training and DPDP-ft adds only 1.3–1.9 GPU-hours per task to the shared base models; at deployment, both stay within 12% of DV's time per decision, and DPDP-ft is even faster than DV on MuJoCo.

Training time per task on one RTX 6000 Ada GPU:

MethodOwn trainingFrom scratch, one GPU
DQL13.6 h (policy and critic)13.6 h
DV6.0 h (planner)6.0 h
DV+6.1 h (CFG diffusion IDM)12.1 h
DPDP-infnone25.7 h
DPDP-ft1.3–1.9 h (student)27.0–27.6 h

Because DQL and the planner train independently, two GPUs reduce the end-to-end time of DPDP-inf and DPDP-ft to about 13.6 and 14.9–15.5 hours.

Time per environment step (ms):

MethodMuJoCoAntMazeRelative to DV
DQL16250.12 / 0.01
DV1303,5381.00 / 1.00
DV+1273,5360.98 / 1.00
DPDP-inf1453,7621.12 / 1.06
DPDP-ft1203,5800.92 / 1.01

Action generation alone takes 6.6 ms for DV+, 16.4 ms for DPDP-inf (three networks per score evaluation, plus one correction step) and 7.0 ms for DPDP-ft (hopper-medium, 50 parallel environments), but plan generation dominates each decision, especially on AntMaze. DQL needs no plan and is therefore faster; relative to the planner DPDP builds on, both variants stay within 12% of DV.

Action: We have added the time-per-step table to the appendix (Table A10, Appendix E.4).


Thank you again for the constructive review!

We sincerely enjoyed the discussion and hope these clarifications address your questions!

Best regards,

Authors


References

[1] Bridging RL Theory and Practice with the Effective Horizon, NeurIPS 2023.

[2] On the Role of Planning in Model-Based Deep Reinforcement Learning, ICLR 2021.

[3] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.

[4] Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022.

[5] What Makes a Good Diffusion Planner for Decision Making?, ICLR 2025.

Reviewer_jz9cOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines a state-trajectory diffusion planner with a value-optimized diffusion policy at the inverse-dynamics execution step. It interprets the transition-conditioned inverse-dynamics distribution as a posterior: the planned next state contributes the transition likelihood, while the offline behavior policy acts as the prior. Replacing or tempering that prior with a critic-trained diffusion policy gives the proposed execution target.

DPDP-inf approximates this target by composing the policy, conditional inverse-dynamics, and null-branch scores and applying Langevin corrections. DPDP-ft instead fine-tunes a conditional executor with a diffusion preference objective and a behavior-cloning anchor. The experiments cover D4RL locomotion, AntMaze, and Maze2D using schedule-matched policy and planner baselines.

Critical strengths
  1. The posterior view gives a clean interface between planning and action-level value optimization. Rather than adding another reward signal to the trajectory sampler, the method changes how the first planned transition is executed. The derivation in Equations (3)-(11) makes this distinction precise.

  2. The paper is careful about the approximations made by both implementations. Proposition 3.1 shows that the product of separately noised marginals is generally not the forward-noised clean target. Section 3.3 and Appendix A also state that finite unadjusted Langevin correction is biased, that the denoising preference loss is a surrogate, and that the additional behavior-cloning term biases the fine-tuned executor toward the reference. This is an important qualification rather than a hidden assumption.

  3. The evaluation is reasonably thorough. The policy and planner bases are reimplemented with a shared action-noise schedule, results are averaged over five seeds, and per-task standard errors are given in Table A3. On the navigation suites, the gains over the schedule-matched planner are numerically substantial in several cases: DPDP-ft raises the AntMaze average from 81.2 to 85.7, and DPDP-inf raises the Maze2D average from 159.5 to 168.9. The paper also reports that the reactive DQL+ policy remains stronger on locomotion, which gives a more balanced picture of when planning helps.

  4. The two variants represent a useful practical tradeoff. DPDP-inf avoids additional training but uses extra denoiser evaluations at inference; DPDP-ft adds a fine-tuning stage and then deploys a single executor.

Critical weaknesses
  1. The experiments evaluate return, but do not directly test whether the executed action remains faithful to the planner's proposed next state. At the population clean-distribution level, the derivation keeps the transition likelihood in the target. The implemented algorithms, however, only approximate that target. A measurement such as the distance between the realized st+1s_{t+1} and the proposed gg, reported for DV+, DPDP-inf, and DPDP-ft, would show whether the value prior improves action choice while retaining plan adherence. The absence of this diagnostic does not show that DPDP ignores the plan; it leaves the paper's main mechanism only indirectly supported by return.

  2. The Q-reranking control is missing on AntMaze. Table A4 reports reranking with 4 and 16 inverse-dynamics samples on locomotion and Maze2D, but not on the suite where DPDP-ft has its largest gain: AntMaze-Large-Play improves from 79.5 to 92.8. Reranking is a particularly relevant control because it uses the same critic to choose among inverse-dynamics proposals without changing the executor distribution. The existing results rule out this simpler explanation on Maze2D more convincingly than on AntMaze.

  3. The statistical summary is weaker than the per-task reporting. Table A3 gives standard errors for individual tasks but none for suite averages. Several locomotion gains over DV+ are around one point and are comparable to the reported standard errors, while the text counts improvements over the full task grid. The larger navigation gains are more persuasive, but paired seed-level differences or confidence intervals for the suite averages would better support claims made at that level.

  4. Hyperparameter robustness is only partly explored where planning matters most. The value-strength sweep is single-seed and restricted to locomotion. The correction-budget sweep covers all suites and supports the default M=1M=1 at the level of suite means, but it also shows that larger correction budgets can sharply reduce AntMaze and Maze2D performance. I would like to know how β=1\beta=1 and M=1M=1 were chosen and to see at least a small multi-seed value-strength study on one navigation suite.

  5. The deployment comparison does not include the full adaptation cost. Table A9 measures evaluation wall-clock on locomotion and shows that DPDP-ft is fast at inference, but DPDP-ft also requires 200,000 additional fine-tuning steps. Reporting training time separately would make the tradeoff with training-free DPDP-inf easier to judge.

Minor Comments

  1. The introduction repeats the sentence “We realize the same posterior in two ways.”

  2. The statement that pQp_Q and pBp_B have approximately the same support is plausible because both include behavior cloning, but it would be useful to state whether this was checked empirically or is only a modeling assumption.

  3. Some wording in the introduction suggests that the two algorithms realize the same distribution. The method and limitations sections are more precise: both target the same clean distribution but introduce different approximation errors. Using the latter wording consistently would avoid ambiguity.

Questions for authors
  1. How closely does the realized next state follow the proposed gg for DV+, DPDP-inf, and DPDP-ft? Does stronger value transfer change that relationship?

  2. What does Q-reranking with 4 and 16 samples achieve on the four AntMaze tasks, especially AntMaze-Large-Play?

  3. Were β=1\beta=1 and M=1M=1 fixed before running the reported evaluation? What signal was used to choose them?

  4. Can the authors report paired seed-level uncertainty for the AntMaze and Maze2D suite averages?

  5. How sensitive is DPDP-ft to the behavior-cloning weight λ\lambda? The theory derives the unanchored target, whereas all reported runs use λ=1\lambda=1.

4 comments
AuthorsOpenReview ↗

Thank you for your review! (1/4)

Dear Reviewer jz9c,

Thank you for your time and constructive review!

We sincerely appreciate that you found our paper well-written, the posterior view a clean interface between planning and action-level value optimization, considered the schedule-matched evaluation thorough, and saw the two variants as a useful and practical tradeoff.

Below we answer each weakness together with its matching question, followed by the minor comments.

[W1, Q1] Plan adherence of the executed action.

Response. Thank you for raising this!

At the default strength, DPDP stays close to the plan: on AntMaze and Maze2D, DPDP-inf reaches the planned waypoints even more closely than DV+ (7.064 vs 7.219; 0.832 vs 0.938), and on MuJoCo its deviation is one third of the value policy's own.

We measured plan error, the L2 distance in the planner's normalized state space between the waypoint s^t+1\hat s_{t+1} that conditions the executor and the state actually reached. Planning strides (one environment step on MuJoCo, 25 on AntMaze and 15 on Maze2D; Table A1) and state dimensions differ, so values are comparable only within a suite.

L2 Plan error at the default strength, suite means (lower is closer to the plan)

SuiteDV+DPDP-infDPDP-ft
MuJoCo (9 tasks)0.1700.2140.332
AntMaze (4 tasks)7.2197.0647.129
Maze2D (3 tasks)0.9380.8321.005

L2 Plan error at the default strength, per task

TaskDV+DPDP-infDPDP-ft
halfcheetah-medium0.2830.3410.408
halfcheetah-medium-replay0.2800.3280.357
halfcheetah-medium-expert0.2380.2911.021
hopper-medium0.0600.0990.130
hopper-medium-replay0.0570.1000.086
hopper-medium-expert0.0570.1070.105
walker2d-medium0.1950.2360.283
walker2d-medium-replay0.2110.2390.325
walker2d-medium-expert0.1530.1900.274
antmaze-medium-play7.2256.4966.933
antmaze-medium-diverse7.2277.2497.228
antmaze-large-play7.1937.3257.218
antmaze-large-diverse7.2317.1867.135
maze2d-umaze1.3111.1791.479
maze2d-medium0.7440.5800.880
maze2d-large0.7570.7380.656

DPDP-inf therefore matches or improves on DV+'s adherence where planning matters most, an interesting result in its own right.

Action: We have added the plan-error table to the appendix (Appendix E.6, Table A12), with a summary in Sec. 4.2.

AuthorsOpenReview ↗

Title: Thank you for your review! (2/4)

[W2, Q2] Q-reranking on AntMaze.

Response. Thank you for suggesting this control!

Reranking recovers less than half of DPDP-ft's AntMaze gain, and about a quarter on AntMaze-Large-Play: with 16 candidates it raises the suite average from 81.2 to 83.3 (DPDP-ft: 85.7) and Large-Play from 79.5 to 83.0 (DPDP-ft: 92.8).

As in Table A4, we draw N∈{4,16}N\in\{4,16\} actions from the inverse-dynamics executor and execute the critic's argmax, keeping the planner, critic and evaluation protocol of Table 1 (mean ± s.e.).

Q-reranking with NN critic-ranked candidates, all tasks of Table A4

TaskN=1 (DV+)N=4N=16DPDP-infDPDP-ft
antmaze-medium-play84.0 ± 2.584.3 ± 2.885.0 ± 1.984.5 ± 2.684.8 ± 2.3
antmaze-medium-diverse82.9 ± 2.384.0 ± 3.885.1 ± 3.886.0 ± 2.683.5 ± 2.4
antmaze-large-play79.5 ± 2.580.0 ± 2.983.0 ± 3.278.0 ± 2.592.8 ± 2.5
antmaze-large-diverse78.3 ± 2.680.0 ± 5.780.2 ± 5.381.0 ± 2.581.7 ± 2.4
AntMaze average81.2 ± 1.282.1 ± 2.083.3 ± 1.982.4 ± 1.385.7 ± 1.2
maze2d-umaze129.2 ± 1.4133.7 ± 4.8136.8 ± 4.3142.3 ± 1.5145.7 ± 1.5
maze2d-medium146.3 ± 1.0147.1 ± 4.1152.0 ± 3.3155.5 ± 1.0152.7 ± 1.0
maze2d-large203.1 ± 1.4195.9 ± 4.5200.2 ± 6.9209.0 ± 1.5203.7 ± 1.4
halfcheetah-medium50.8 ± 0.150.8 ± 0.151.0 ± 0.150.9 ± 0.151.9 ± 0.1
halfcheetah-medium-replay46.0 ± 0.346.6 ± 0.146.7 ± 0.146.8 ± 0.147.3 ± 0.2
halfcheetah-medium-expert92.7 ± 0.292.9 ± 0.193.7 ± 0.594.6 ± 0.392.0 ± 0.5
hopper-medium83.2 ± 1.383.5 ± 1.283.8 ± 1.284.3 ± 1.283.2 ± 1.2
hopper-medium-replay91.2 ± 0.091.5 ± 0.192.0 ± 0.192.2 ± 0.091.9 ± 0.0
hopper-medium-expert109.8 ± 0.5110.0 ± 0.5110.0 ± 0.7110.7 ± 0.8110.4 ± 0.8
walker2d-medium83.4 ± 0.283.9 ± 0.884.3 ± 0.984.5 ± 0.385.0 ± 0.2
walker2d-medium-replay85.1 ± 0.485.8 ± 0.586.1 ± 0.586.8 ± 0.586.1 ± 0.3
walker2d-medium-expert108.6 ± 0.2109.1 ± 0.0109.4 ± 0.0110.0 ± 0.3110.7 ± 0.2

Reranking improves steadily with NN, but it can only select among the executor's proposals. If the useful action is rare under the original executor, it may never be selected, whereas DPDP-ft moves probability mass toward it. Our two DPDP variants are therefore competitive with Q-reranking without any test-time search, and DPDP-ft has the best suite average.

[W3, Q4] Uncertainty of suite-level comparisons.

Response. We appreciate this careful statistical suggestion!

Every suite-level gain over DV+ is positive, and five of the six exceed two standard errors.

Gain in suite-average normalized return over DV+ (± standard error of the difference)

SuiteTasksDPDP-inf gain over DV+DPDP-ft gain over DV+
MuJoCo9+1.1 ± 0.25+0.9 ± 0.25
AntMaze4+1.2 ± 1.78+4.5 ± 1.73
Maze2D3+9.4 ± 1.08+7.8 ± 1.06

Each entry is the gain in suite-average normalized return over DV+, ± the standard error of that difference, computed from the per-task standard errors of Table A3 (five seeds per method, tasks treated as independent). The Maze2D gains lie seven to nine standard errors above zero, the MuJoCo gains about three and a half to four and a half, and DPDP-ft's AntMaze gain 2.6. Per task, 17 of the 32 entries exceed DV+ by more than two standard errors of the difference, and none falls below it by that margin.

Action: We have reported these suite-level gains and their standard errors in the results section alongside the task counts (Sec. 4.1; Table A16 in Appendix E.9), and added suite-average standard errors to Table A3.

AuthorsOpenReview ↗

Thank you for your review! (3/4)

[W4, Q3] Selection of β\beta and MM.

Response. Thank you for asking how these were chosen!

Both were fixed before the reported evaluation and shared by all 16 tasks: β=1\beta=1 is the untempered target of our derivation (full prior replacement), and M=1M=1 is the smallest non-zero corrector budget; no evaluation return was used to select either.

For β\beta, Tables A5–A7 show β=1\beta=1 at or near the top for both realizations on every locomotion task, and our plan-adherence sweep (W1, Q1) varies 1/β1/\beta from 0.1 to 5 on all three suites. For MM, every reported result uses the default M=1M=1, and Table A8 shows that this default also gives the best suite average on all three suites (DPDP-inf):

Corrector budget MM for DPDP-inf, all tasks of Table A8

TaskM=0M=1 (default)M=3M=5
halfcheetah-medium50.850.950.850.8
halfcheetah-medium-replay46.446.846.846.4
halfcheetah-medium-expert94.094.694.892.9
hopper-medium83.784.384.584.4
hopper-medium-replay92.292.291.291.5
hopper-medium-expert110.3110.7110.9109.9
walker2d-medium84.084.584.284.8
walker2d-medium-replay85.686.886.286.7
walker2d-medium-expert110.0110.0110.0110.0
MuJoCo mean84.184.584.484.2
antmaze-medium-play84.084.576.072.2
antmaze-medium-diverse80.086.082.884.4
antmaze-large-play78.578.068.470.0
antmaze-large-diverse74.481.064.680.2
AntMaze mean79.282.473.076.7
maze2d-umaze145.0142.3135.8126.3
maze2d-medium153.2155.5145.9149.4
maze2d-large208.3209.0212.2200.5
Maze2D mean168.8168.9164.6158.7

The larger budgets you noted are never used, so no reported number depends on them. The reported results are also conservative: per-task tuning would raise some entries (Maze2D-Large reaches 212.2 at M=3M=3), but we report the fixed default.

[W5] Training cost of DPDP-ft.

Response. Thank you so much for the opportunity to clarify this!

DPDP-inf adds no training, and DPDP-ft adds a single fine-tuning stage of 2×1052\times10^5 steps, about 1.3–1.9 GPU-hours per task on one RTX 6000 Ada GPU including preference-pair generation; in return, DPDP-ft evaluates 1.9–3.5× faster.

We therefore report the training cost separately from the deployment cost:

Additional cost per task

Per taskDPDP-inf (M=1M=1)DPDP-ft
Additional trainingnone2×1052\times10^5 steps, batch 128
Additional training time01.3–1.9 GPU-hours
Evaluation run, locomotion (Table A9)7.0–9.4 min2.0–4.5 min

The base models (DV planner, 6.0 GPU-hours; inverse dynamics, 6.1; DQL+ policy and critic, 13.6) are shared by both variants, so this stage is the only training difference between them. DPDP-inf therefore suits few evaluations or settings where β\beta and MM must stay adjustable at test time, and DPDP-ft suits repeated deployment.

Hence, both variants are efficient: DPDP-inf needs no training at all, and DPDP-ft's one-time 1.3–1.9 GPU-hours is small next to the 25.7 GPU-hours of the shared base models and buys 1.9–3.5× faster evaluation.

Action: We have added the training time and the time per environment step to the appendix (Appendix E.4, Table A10).

AuthorsOpenReview ↗

Thank you for your review! (4/4)

[Q5] Sensitivity of DPDP-ft to the anchor weight λ\lambda.

Response. Thank you for this question!

DPDP-ft is robust to λ\lambda: the unanchored target (λ=0\lambda=0), which our derivation covers, and the deployed default (λ=1\lambda=1) are within 2.2 points on every suite, and the unanchored student already matches or improves on DV+.

λ\lambda moves the deployed student between the unanchored target and the reference executor pIp_I used by DV+ (large λ\lambda). We swept it on all three suites:

Anchor weight λ\lambda for DPDP-ft

λ\lambda0 (unanchored)0.10.51 (default)25
MuJoCo83.4 ± 0.1183.1 ± 0.1584.0 ± 0.2084.3 ± 0.1881.3 ± 0.3182.4 ± 0.27
AntMaze83.5 ± 1.3583.0 ± 1.1284.0 ± 1.3785.7 ± 1.2081.0 ± 0.9581.0 ± 1.02
Maze2D168.3 ± 0.85169.4 ± 0.77167.8 ± 0.58167.4 ± 0.76166.2 ± 0.69164.7 ± 0.85

The default λ=1\lambda=1 gives the best return on MuJoCo and AntMaze, and every λ\lambda improves on DV+ on Maze2D (164.7 to 169.4 vs 159.5). The unanchored student improves on DV+ on AntMaze and Maze2D (83.5 vs 81.2; 168.3 vs 159.5) and matches it on MuJoCo (83.4), so our analysis of the unanchored target carries over to the deployed objective.

Action: We have reported this sweep next to the derivation (Sec. 3.3; Table A11 in Appendix E.5).

[Minor] Wording in the introduction.

Response. Thank you for the careful reading!

We agree: both realizations target the same clean-action distribution and approximate it in different ways, as Secs. 3.3 and 5 state.

Action: We have removed the repeated sentence, and the introduction now uses this precise wording throughout.

[Minor] Action support of pQp_Q and pBp_B.

Response. Thank you for prompting this check!

We have now verified the assumption empirically, and it holds on all three suites: at least 94% of pQp_Q's actions fall within the support of pBp_B.

At 1,000 dataset states per task, we drew 32 actions from each model and flagged a pQp_Q action as outside the support of pBp_B if its nearest-neighbour distance to the pBp_B samples exceeded the 99th percentile of the same distance between two independent pBp_B sample sets, so 1% are flagged under identical support. The flagged share is 1.1% on MuJoCo, 1.0% on Maze2D (both at the null rate) and 6.0% on AntMaze.

Action: We have added this check to Sec. 3.2 and quantified the statement there (at least 94% of pQp_Q's actions within the support of pBp_B in every suite; details in Appendix E.8).


Thank you again for the detailed and constructive review!

We hope the plan-adherence diagnostic, the AntMaze reranking control, the suite-level uncertainty estimates and the λ\lambda sweep address your main questions, and we would be glad to discuss any remaining point.

We enjoyed this discussion, which has helped improve our submission, and are confident that we have addressed your concerns well!


Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.