acceptodds
Under review as a conference paper at ICLR 2027

TD-MTP: Coupling Structured Planning with Multi-Step Value Learning

Abstract

In model-based robot learning, a planner selects actions and collects the data used to train its value function. Its effect during learning may therefore depend on the value target. We study this dependence with TD-MTP, which combines a structured tensor-graph planner with replay-based multi-step value targets in TD-MPC2. The main comparison evaluates both planners under one-step and multi-step targets on ten continuous-control tasks. The structured planner improves mean return on one task under one-step targets and five under -returns; its effect is more favorable under -returns on eight. On h1-stair, the planner changes from reducing mean return by to increasing it by . Used separately, the multi-step target raises mean return above the baseline on eight tasks and the planner on one. Controls suggest that elite fitting contributes to this interaction, while the contribution of tensor proposals remains unresolved. For fixed policies, our error bound separates bootstrap perturbation from behavior-prior mismatch and accounts for the endpoint bootstrap retained by finite replay windows. Across 38 task-protocol pairs in four benchmarks, TD-MTP records 10 wins, 22 ties, and 6 losses against TD-MPC2 under stated tolerances. Learning curves show earlier gains on humanoid-run but slower learning on stick-pull. These results support evaluating planners and value targets together during robot learning.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, sampling-based model predictive control, multi-step value estimation, temporal-difference learning, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8Mw8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the choice of planner interacts with the value learning target in model-based reinforcement learning. The authors propose TD-MTP, which combines tensor-graph planning with multi-step λ-returns within TD-MPC2. They evaluate the method on continuous-control tasks from HumanoidBench, DMControl, Meta-World, and MyoSuite. The main finding is that a planner that performs poorly with one-step value targets can work better with multi-step targets. The paper also provides a theoretical analysis of bootstrap error and policy mismatch.

Critical strengths

This paper conducts a highly valuable study on the interplay between planning and value learning. I particularly appreciate the 2×2 experimental design, which compares the performance of two planners across single-step and multi-step goals. The results from the "stair" and "pole" tasks are especially compelling, powerfully demonstrating that an initially inefficient planning configuration can be transformed into an efficient one simply by altering the value objective. Furthermore, the authors present not only successful cases but also negative results and supplementary control experiments, thereby bolstering the credibility of their empirical findings. (This is important for my rating.)

Critical weaknesses

My primary concern is that the contribution of the "structured planner" proposed in the paper remains unclear. The paper combines existing techniques—specifically tensor-graph planning and multi-step value learning, so I expected to see compelling evidence demonstrating the efficacy of this specific combination. However, the results suggest that the majority of the performance gains likely stem from the multi-step target itself. As shown in Table 1, adding the planner alone improved average returns on only one of the ten tasks, whereas adding the multi-step target alone yielded improvements on eight tasks. More importantly, the ablation studies in Table 15 failed to clearly substantiate the benefits of the tensor proposals. For a paper proposing a novel combination of methods, I consider this a significant limitation.

Furthermore, the overall performance improvements were inconsistent. Across 38 evaluation task settings, TD-MTP achieved a record of 10 wins, 22 ties, and 6 losses. In pairwise experiments, only two of the ten "planner-target" combinations showed confidence intervals that did not include zero. Performance actually deteriorated significantly on some tasks; notably, in the "stair" task, the method's performance advantage over the baseline vanished when training was extended to 4 million steps. While the theoretical analysis is valuable, it primarily characterizes error trade-offs under a fixed policy and does not directly explain the observed improvements in training performance. Overall, although I find the interaction effects interesting, I am not convinced that TD-MTP offers a reliable advantage over simpler alternatives.

These two points are the primary reasons for my recommendation to reject the paper.

Questions for authors

Q1: Could the authors provide additional comparative experiments that isolate the effect of enabling "tensor proposals"—keeping elite fitting, action selection, the replay window, and other settings strictly constant? The current results do not clearly demonstrate the specific contribution of tensor sampling.

Q2: Could the authors provide further comparative results under identical conditions against TD-MPC2 and the "target-only" baseline? Ideally, these should use the same implementation code and training random seeds.

Q3: Is there direct evidence that multi-step targets improve the Critic's accuracy in evaluating actions selected by the structured planner?

4 comments
AuthorsOpenReview ↗

Response to 8Mw8 (1/2): proposal controls and matched comparisons

Dear Reviewer 8Mw8,

Thank you for your careful questions and for recognizing the value of the two-by-two design and negative results. We agree that the paper must distinguish the contribution of tensor proposals from that of the target.

Q1. Isolated tensor proposals. The new control holds fitting, action selection, replay length, candidate budget, seeds, and evaluation fixed. It finds a positive tensor–target interaction on stair, but the proposal benefit under the multi-step target remains uncertain (Table 30). The smaller controls are also inconclusive about a return benefit (Table 18). The one-component tests point to a role for selection pressure; they do not establish a general advantage of the full planner (Table 29).

Q2. Matched planner and target-only comparisons. The extended study uses one JAX implementation, seed-matched contrasts, and a common evaluation planner:

20 seeds per taskh1-stairh1-poleh1-hurdle
Interaction+348.9+348.9 [+253,+445][+253, +445], 20/20+129.5+129.5 [+24,+235][+24, +235], 15/20+102.7+102.7 [+28,+178][+28, +178], 14/20
Planner effect, one-step target−287.2-287.2 [−360,−215][-360, -215]−149.8-149.8 [−246,−54][-246, -54]−102.7-102.7 [−123,−83][-123, -83]
Planner effect, multi-step target+61.7+61.7 [−12,+135][-12, +135]−20.3-20.3 [−53,+13][-53, +13]−0.1-0.1 [−71,+71][-71, +71]
Interaction, seeds added after submission only+324.1+324.1 [+226,+422][+226, +422], 15/15+67.9+67.9 [−52,+188][-52, +188], 10/15+113.7+113.7 [+16,+211][+16, +211], 11/15

Paired 95% Student-t intervals; counts give positive interactions. The hurdle prediction was specified before the additional runs. Table 26 reports all target and planner contrasts.

The pooled interaction is positive on all three tasks. On stair and pole, the multi-step target raises the structured planner's return. It also lowers the reference on stair and hurdle. The target effect on the structured planner is unresolved on hurdle. This distinction matters: a positive interaction does not show that the combined method beats the baseline. The planner-effect intervals under the multi-step target include zero and do not establish equivalence.

On the additional seeds alone, the interaction intervals exclude zero on stair and hurdle, but include zero on pole. The remaining HumanoidBench and DMControl interaction intervals include zero. Humanoid-walk is an exception to a simple explanation based on the one-step planner deficit: the structured planner has lower mean return under both targets, without a positive interaction (Table 25).

Matched-window and released-planner controls retain a positive stair interaction (Table 27). The target-only suite results also show why the simpler method matters: it matches or exceeds TD-MTP's mean on most of the additional tasks where both were tested (Table 13).

Longer training. The seven-seed study uses one run group and keeps the tensor schedule fixed, with proposals stopping at 1M:

h1-stair, 7 seedsat 2Mat 4M
Planner effect, one-step target−239.9-239.9 [−303,−177][-303, -177], 0/7−142.1-142.1 [−235,−49][-235, -49], 0/7
Planner effect, multi-step target−49.6-49.6 [−109,+10][-109, +10]−12.0-12.0 [−85,+61][-85, +61]
Multi-step effect on the structured planner+186.0+186.0 [+57,+315][+57, +315], 7/7+113.3+113.3 [+14,+212][+14, +212], 5/7
Interaction+190.3+190.3 [+109,+271][+109, +271], 7/7+130.1+130.1 [−4,+264][-4, +264], 5/7

Table 24 gives the method means. Both checkpoints use the same seven runs per setting. The 2M value averages three evaluations centered on that checkpoint; the 4M value uses the last three.

At 4M, the multi-step target still raises the structured planner's return, but the interaction interval includes zero. TD-MTP, target-only, and the reference have similar means; the one-step structured planner still trails. We have removed the claim that all four settings converge.

These results support a target-dependent planning effect within the tested budgets. They do not establish a reliable advantage over the target-only agent. The revised discussion acknowledges both concerns in your review and retains the negative results that limit the interpretation.

AuthorsOpenReview ↗

Response to 8Mw8 (2/2): critic measurements

Q3. Is there direct evidence about critic accuracy on selected actions? We report calibration against each planner's own returns. The multi-step critic agrees more closely with those returns, but the comparison does not isolate estimation error against the policy value defined by the target.

We sampled states from each setting's own trajectories and compared the critic's value of the executed action with the discounted return under that setting's planner. The settings use matched replay windows and the released reference planner (Table 32).

SettingCritic estimateMeasured returnSigned errorAbsolute errorCorrelation
TD-MPC2 (reference planner, one-step)76.866.7+10.1+10.110.70.46
+Target (reference planner, multi-step)71.271.6−0.5-0.55.90.68
+Planner (structured planner, one-step)78.157.8+20.4+20.420.40.52
TD-MTP (structured planner, multi-step)77.578.5−1.0-1.02.40.74

Stair at 2M steps; ten seeds and 24 states per seed (240 states per setting), replay length 13. Returns cover 500 steps with discount 0.99. “Error” is relative to the measured own-planner return, and correlation is across states. Table 32 gives the measurement protocol.

The one-step structured critic has the largest positive gap to its planner's return. The multi-step setting has smaller mean signed and absolute gaps. The signed gap decreases on every seed. This supports closer calibration on the trajectories where that planner acts.

The comparison has two limits. First, the settings visit different states and use different continuation planners. Second, the target changes the value being estimated. The one-step operator evaluates the policy prior. The infinite-window λ\lambda operator evaluates a mixture of the prior and behavior planner. Finite windows retain the prior bootstrap and need not have that same fixed point. Agreement with own-planner returns can therefore change without a corresponding reduction in estimation error. We do not treat this measurement as the uniform bootstrap perturbation in Proposition 1.

Candidate ranking. The across-state correlation in the table is different from ranking candidates within each state. On each setting's own states, this ranking test does not show improved ranking under the multi-step target. One contrast is lower for TD-MTP than target-only; the remaining contrasts are unresolved (Appendix G).

We first specified a stricter design with common states, a common continuation planner, and candidates that differ only over the planning horizon. It could not resolve differences between settings because continuation noise was large relative to the candidate-return differences. We specified the own-trajectory measurement after this result and report it as secondary. Both designs remain in Appendix G.

The theory explains the target's policy-evaluation tradeoff under fixed policies. The component interventions test the training effect, and the critic measurements describe calibration under the tested continuations. Together they narrow the interpretation, but do not prove a unique mechanism for the learning gains.

Thank you again. Your questions led us to separate these claims and make the evidence and its limits easier to assess.

Thank you for the detailed rebuttal and additional experiments. I appreciate the expanded 20-seed comparisons and improved controls, which provide stronger evidence for the planner–target interaction.

However, my main concern remains unresolved. The new results still do not establish a reliable benefit from tensor proposals themselves, and the simpler target-only baseline matches or exceeds TD-MTP in mean performance on most additional tasks where both were evaluated. Furthermore, while the critic calibration results are interesting, they do not conclusively establish reduced value estimation error or improved candidate ranking.

Overall, I find the evidence for target-dependent planning effects more convincing, but the specific contribution and practical advantage of TD-MTP remain unclear. I therefore maintain my original rating.

Dear Reviewer 8Mw8,

Thank you for reading the new results and for telling us clearly where you stand. We agree with your three points, and the revised paper says the same. We do not claim that TD-MTP is better in practice than the target-only agent.

We have one more point. In your review, you noted that most of the gains come from the multi-step target. With 20 seeds, we see that the effect of the target also depends on the planner (Table 26).

  • With the reference planner, the target decreases the return on stair (−76.4-76.4 [−149,−3][-149, -3]) and on hurdle (−105.8-105.8 [−170,−41][-170, -41]). It increases the return on pole (+114.1+114.1 [+63,+165][+63, +165]).
  • With the structured planner, the same target increases the return on stair (+272.5+272.5 [+207,+338][+207, +338]) and on pole (+243.6+243.6 [+161,+327][+161, +327]). On hurdle, the result is not clear (−3.2-3.2 [−45,+39][-45, +39]).

On stair, the multi-step target increases the return of the structured planner, but it decreases the return of the reference planner. In the same way, the structured planner decreases the return with the one-step target, but we see no loss with the multi-step target. So the effect of the target depends on the planner, and the effect of the planner depends on the target.

The target-only agent is the simpler choice, and it helps on many tasks. But these results show that the two choices cannot be tested one at a time. We believe that this is the contribution of the paper. The evidence has matched controls and one prediction that we made before the runs. We use TD-MTP to measure this effect. We do not propose it as a method that gives a better return.

Thank you again for your time. Your questions led to these potentially insightful experiments.

Best regards, The Authors

Reviewer_em3mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes two concrete changes to TD-MPC2, a model-based reinforcement learning algorithm that learns a latent dynamics model with reward and value heads and then selects actions by planning with the learned model. The return of a given (latent) trajectory is estimated as a sum of rewards over the planning horizon + the terminal value function from the planning horizon and beyond. Ths paper proposes two changes to the algorithm: (1) a tensor graph planner, and (2) multistep (lambda) returns for value learning. Experiments are conducted on 38 tasks from 4 task suites, Results indicate that lambda returns help across some tasks while the planner is mostly the same as the baseline or worse. Overall, the proposed method TD-MTP performs similar to TD-MPC2.

Critical strengths

I appreciate that the authors take the time to produce an empirical study with multiple seeds, per-seed results reported, and are generally pretty up front about the results and empirical findings despite them not being too glamorous. The idea of using lambda returns in TD-MPC2 makes intuitive sense (it has been shown to help for a lot of other TD-learning algorithms).

Critical weaknesses

I do not believe that the paper is ready for publication in its current form. The work is incremental in nature and does not appear to improve upon TD-MPC2 according to the authors' own experiments. The paper was also a bit difficult to read. A few key things worth mentioning:

  • The motivation is not super clear to me. I think using lambda returns makes sense intuitively and seems to be supported by experiments to some extent, but I don't really understand why the authors choose the more complex tensor graph approach to planning when it seemingly does not improve results in any meaningful way. Many of the improvements seem not statistically significant either. Seems like a negative result overall?
  • The paper has very few citations throughout. I understand that the paper builds pretty directly on top of TD-MPC2 but there are no MBRL methods cited in the introduction besides TD-MPC2. The authors mention other works that build upon TD-MPC2 such as TD-M(PC)^2, BMPC, and BOOM but do not compare against these methods despite them being quite similar to the proposed method. Same goes for methods like MVE that are mentioned but not compared against. The only method that the authors compare to is TD-MPC2.
  • The writing is a bit confusing overall. For example, there are too many results and numbers in the text, especially in the abstract and introduction. Exhaustively listing all of the numbers does not add much. Please focus on the key technical contributions, insights, and takeaways. What will a curious reader learn from reading your paper? The authors also repeatedly refer to TD-MPC2 as a model-based robot learning algorithm whereas MBRL typically refers to model-based reinforcement learning, and the paper repeatedly states that MBRL algorithms are latent dynamics models that learn a value function and do planning but that is just a description of TD-MPC2, not MBRL in general. Most MBRL algorithms do not do this.
Questions for authors

I would appreciate if the authors can please respond to my comments in the "weaknesses" section above.

3 comments
AuthorsOpenReview ↗

Response to em3m (1/2): contribution, motivation, and clarity

Dear Reviewer em3m,

Thank you for your comments on the motivation, comparisons, and presentation. We agree that the paper should explain what the reader learns, with fewer numbers in the prose.

Contribution. The study shows why a planner should be evaluated with the value target used during its training. A planner selects actions and changes the data that train its critic. Changing the target can therefore change the effect of the same planner. We test this directly with two planners crossed with two targets, using one evaluation planner (Section 5.1).

The extended results support a positive interaction on stair, pole, and hurdle. They also require a more precise interpretation than our earlier response gave. On stair and pole, the multi-step target raises the structured planner's return. It also lowers the reference planner's return on stair and hurdle. In the extended stair sample, TD-MTP no longer has a higher mean than the one-step reference. The contribution is the measured dependence between planning and value learning, not a general performance advantage (Tables 1, 25, and 26).

Why the tensor planner? Tensor proposals recombine control points into action sequences. We use this existing planner as a structured alternative to Gaussian proposals. Its fitting and execution rules make the configuration sensitive to the target. Component tests help identify which changes matter, but do not establish that tensor proposals alone improve return (Tables 29 and 30).

The method comparison is mixed: the full configuration has no reliable advantage over the simpler target-only agent. The positive finding is the interaction. The target alone accounts for much of the suite gain where tested, and the paper now states this directly.

What the theory contributes. Longer returns use more observed rewards and fewer nearby bootstrap estimates. They also follow more actions from the behavior planner rather than the policy prior. The fixed-policy analysis separates these two effects and shows why a finite replay window still depends on its endpoint value. It builds on Peng's Q(λ\lambda); it does not prove that the critic becomes more accurate during nonlinear training (Section 4.3; Appendix C).

Presentation. The introduction now defines model-based reinforcement learning broadly and distinguishes online planning from policy learning on imagined trajectories. It cites representative methods before narrowing to TD-MPC-style agents. The main text follows the question, target analysis, controlled interaction, component tests, and broader results. Tables carry the numerical detail. Short explanations connect each experiment to the question it tests, including where the proposed interpretation fails.

We retain the negative results because they define the scope of the finding. Humanoid-walk does not follow a simple explanation based on the one-step planner deficit, and the longer-budget study does not establish a lasting advantage over simpler alternatives.

AuthorsOpenReview ↗

Response to em3m (2/2): related work and alternative methods

Related methods. We expanded the introduction and related work to explain how TD-MTP relates to other model-based methods. TD-M(PC)2^2 constrains policy updates to limit unsupported value queries. BMPC combines planner imitation, model-based value learning, and lazy reanalysis. BOOM aligns the policy with planner actions through a value-weighted objective. These methods address related parts of the planning–learning feedback loop.

Released TD-M(PC)2^2. We ran the released agent on the seven h1 tasks in our study. The table includes our reference and target-only agent so that implementation differences remain visible.

h1 task, 2M stepsReleased TD-M(PC)2^2 (3 seeds)Our TD-MPC2Target-onlyTD-MTP
stair542.7±190.5542.7 \pm 190.5597.3520.8582.6
pole780.5±141.9780.5 \pm 141.9785.9900.1879.7
hurdle206.0±110.0206.0 \pm 110.0626.5520.7520.6
stand848.7±44.4848.7 \pm 44.4843.9847.5854.2
run305.2±296.1305.2 \pm 296.1799.8809.4785.7
walk818.0±8.0818.0 \pm 8.0822.4832.3786.7
maze331.7±2.8331.7 \pm 2.8360.2368.4369.6

Released code: mean ± SD over three seeds at 2M. Our means use 20 seeds on stair, pole, and hurdle, and eight on the other tasks. The comparison is unpaired. Planning settings, evaluation aggregation, and endpoint timing differ across implementations (Table 34).

Our reference also exceeds the released agent on most tasks. These results therefore do not isolate the effect of TD-MTP's changes. In the released code's own comparison on humanoid-run and dog-run, we did not detect a policy-constraint gain with three seeds at 500k (Appendix H).

Policy constraint in our implementation. The constrained one-step reference has the highest stair return among the constrained settings. With the constraint, the multi-step target lowers return for both planners (Table 36). Thus, the two changes do not combine into a further gain. Comparisons with and without the constraint also span code snapshots, so the cross-column effects cannot be assigned solely to the constraint.

Pessimistic values. Following the principle studied by Chang et al., we tested a minimum over critic heads in planning. It narrows the structured planner's one-step deficit, but lowers reference-planner return on pole (Appendix H). These tests support comparing several remedies; they do not establish that one is generally best.

Multi-step alternatives. Plain finite returns at λ=1\lambda=1 do not reproduce the λ=0.8\lambda=0.8 gain on stair (Table 31). MVE uses predicted transitions to extend targets, while our target uses stored rewards and transitions before its bootstrap. Replay-based λ\lambda-returns are already established by Daley and Amato. DreamerV3 uses imagined returns and an auxiliary critic loss on replay trajectories.

We did not run BMPC, BOOM, or MVE. Their absence limits comparative claims, and we do not claim state-of-the-art performance. The related-work discussion also covers EfficientTDMPC, which changes the planning objective, and concurrent PL-MPC, which combines multi-step targets from replay and model rollouts with planner and actor changes. Its targets use model predictions only beyond the sampled replay window. Our specific focus is the controlled interaction between the data-collection planner and replay target.

Thank you for asking us to make these distinctions and comparisons explicit.

AuthorsOpenReview ↗

Gentle Reminder: Request for Feedback on Our Rebuttal

Dear Reviewer em3m,

We hope this message finds you well. We understand that this is a busy period, possibly with revisions of your own, but we would be very grateful if you could take a little time to review our rebuttal. We have done our best to address each of your concerns, and this work is very important to us.

If anything is still unclear or you would like us to elaborate on any point, please don’t hesitate to let us know. We are happy to continue the discussion and look forward to your feedback.

Best regards, The Authors

Reviewer_YWNhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a data-collection planner interacts with the value targets used to train its critic. TD-MTP combines a structured planning configuration with replay-based multi-step returns in TD-MPC2. The central experiment crosses two planners with two targets while holding the evaluation planner fixed. The paper also derives fixed-policy error bounds, specifies finite-window and episode-boundary handling, and reports a broader evaluation across 38 task–protocol settings.

Critical strengths
  1. The four-arm experiment uses a common evaluation planner, making the observed planner–target interaction a training effect rather than an immediate inference-time planner advantage. The stair case, where neither modification helps alone but their combination does, provides a concrete and useful empirical finding.

  2. The finite-window analysis carefully retains the terminal bootstrap and distinguishes the evaluated mixture policy from the policy prior. Together with the negative and longer-budget results, it gives a deeper account of what multi-step targets and positive interactions do—and do not—establish.

Critical weaknesses
  1. The planner comparison supports a configuration-level finding. The planners differ in elite count, variance fitting, terminal action selection, and execution rules, in addition to candidate generation. The tensor–target interactions in Table 15 are small and uncertain on both tested tasks. Moreover, the reported interpolation at H = M = 3 uses only two effective control-point layers, and tensor candidates disappear halfway through training. These are disclosed design choices, but they limit what can be attributed to the structured proposal. The controls point more clearly toward elite fitting, while leaving the practical reason to adopt the full TD-MTP configuration uncertain.

  2. The broader results are less directly comparable than the core study. Some DMControl comparisons cross JAX and PyTorch implementations; the reported dog-run baseline discrepancy illustrates why this matters. Table 18 also pools run sets with overlapping seed identifiers, whose independence is explicitly unresolved. The 10/22/6 summary is therefore descriptive evidence, with less weight than the controlled study. Even within the latter, the reference means differ between the main four-condition experiment and the seven-condition controls despite overlapping seed labels. A compact mapping from existing runs and configurations to the reported tables would substantially improve interpretability.

Questions for authors
  1. Which configuration or random-stream differences explain the different reference means in Tables 1/12 and 14? Can the corresponding existing runs be mapped unambiguously to the reported cells?

  2. Can the available seed-level component-control results clarify whether any independent tensor-proposal benefit is supported, beyond the interaction of the overall planning configuration with the target?

4 comments
AuthorsOpenReview ↗

Response to YWNh (1/2): run mapping and benchmark comparisons

Dear Reviewer YWNh,

Thank you for recognizing the value of the common evaluation planner and the finite-window analysis. We agree that clear run mapping is needed to interpret both the controlled study and the broader results.

Q1. Different reference means. The original four-setting and seven-condition studies used the same reported settings. Three seeds were rerun on a different machine and code revision. The other two produce identical learning curves. The table below gives the mapping; Appendix I gives the available code-state and configuration records.

SeedSame machineReference planner\qquadTarget-only\qquadFull method
1865No484.4→571.9484.4 \rightarrow 571.9 (+87.6+87.6)419.6→466.0419.6 \rightarrow 466.0 (+46.4+46.4)719.6→403.9719.6 \rightarrow 403.9 (−315.7-315.7)
3Yes408.1→408.1408.1 \rightarrow 408.1 (0)488.6→488.6488.6 \rightarrow 488.6 (0)511.2→511.2511.2 \rightarrow 511.2 (0)
4Yes674.8→674.8674.8 \rightarrow 674.8 (0)479.4→479.4479.4 \rightarrow 479.4 (0)739.3→739.3739.3 \rightarrow 739.3 (0)
5927No756.7→395.2756.7 \rightarrow 395.2 (−361.5-361.5)427.3→604.9427.3 \rightarrow 604.9 (+177.6+177.6)657.2→703.0657.2 \rightarrow 703.0 (+45.8+45.8)
6108No302.9→783.1302.9 \rightarrow 783.1 (+480.2+480.2)483.0→558.3483.0 \rightarrow 558.3 (+75.3+75.3)627.5→387.1627.5 \rightarrow 387.1 (−240.4-240.4)
Mean change [95% t-CI]+41.2+41.2 [−332,+414][-332, +414]+59.9+59.9 [−31,+151][-31, +151]−102.1-102.1 [−306,+102][-306, +102]

All three mean-difference intervals include zero, but they do not establish equality. A separate repeatability check agrees bitwise on one machine and diverges on another with the same GPU model. Seed identifiers alone therefore establish neither identical trajectories nor independence. The main conclusions use seed-matched contrasts with the run groups stated explicitly.

The revised manuscript corrects the reference planner settings, the evaluation statistic, two reported cell values, and the replay-window contrast. The main reference uses K=32K=32 and inverse temperature 11; the released default uses K=64K=64 and 0.50.5. The released-planner control retains a positive stair interaction (Table 27). For DMControl, the primary paired results use the frozen-code rerun. Its interaction intervals include zero, although some simple effects differ from the original study.

Benchmark comparability. The primary DMControl suite now compares two JAX agents with the same simulator versions. It excludes the additional TD-MTP run sets with repeated seed identifiers. The PyTorch release remains in the paper for context.

TaskJAX TD-MPC2TD-MTPDifference (%)Paired difference [95% t-CI]Seeds > 0
acrobot-swingup614.9±41.5614.9 \pm 41.5583.0±53.1583.0 \pm 53.1−5.2-5.2−31.9-31.9 [−119,+56][-119, +56]1/5
humanoid-run447.7±11.5447.7 \pm 11.5484.8±29.0484.8 \pm 29.0+8.3+8.3+37.2+37.2 [−4,+78][-4, +78]5/5
cheetah-run884.3±64.8884.3 \pm 64.8915.1±4.0915.1 \pm 4.0+3.5+3.5+41.8+41.8 [−75,+159][-75, +159]*2/4
humanoid-stand865.7±129.7865.7 \pm 129.7932.6±9.8932.6 \pm 9.8+7.7+7.7+8.1+8.1 [−226,+242][-226, +242]*1/2
humanoid-walk767.8±177.8767.8 \pm 177.8913.5±14.0913.5 \pm 14.0+19.0+19.0+145.8+145.8 [−80,+372][-80, +372]3/5
quadruped-walk964.6±9.4964.6 \pm 9.4974.4±5.0974.4 \pm 5.0+1.0+1.0+9.8+9.8 [−7,+27][-7, +27]4/5
dog-run249.4±103.2249.4 \pm 103.2560.8±57.1560.8 \pm 57.1+124.9+124.9+311.4+311.4 [+212,+411][+212, +411]5/5
finger-turn_hard979.3±4.6979.3 \pm 4.6889.7±53.6889.7 \pm 53.6−9.1-9.1−89.6-89.6 [−156,−24][-156, -24]0/5

Mean ± SD across five runs. Paired contrasts use shared seeds: four on cheetah-run, two on humanoid-stand, and five elsewhere. Intervals are 95% Student-t intervals. Table 40 lists the full seed sets.

The suite tally is descriptive, with ties defined by a tolerance rather than an equivalence test (Table 2). Most DMControl intervals include zero. Our JAX reference is also weaker than the PyTorch release on some tasks, particularly dog-run. This limits what the tally supports.

The target-only controls are essential here: their means match or exceed TD-MTP on four of the five additional suite tasks (Table 13). On acrobot-swingup, our JAX reference exceeds both multi-step methods. We state this in Section 5.2 and base the contribution on the controlled planner–target comparison.

AuthorsOpenReview ↗

Response to YWNh (2/2): tensor proposals and planner components

Q2. Is an independent tensor-proposal benefit supported? The current evidence does not establish a return gain from tensor proposals under the multi-step target. It does show a positive tensor–target interaction in the larger controlled test.

On stair, we switched tensor proposals on and off while holding the other planner rules, replay window, candidate budget, seeds, and evaluation planner fixed. The effect differs by target:

Point in trainingUnder multi-step targetUnder one-step targetTensor ×\times target interaction
End of training (2M)+95.3+95.3 [−8,+199][-8, +199], 7/10−59.4-59.4 [−119,−0.1][-119, -0.1], 2/10+154.7+154.7 [+17,+293][+17, +293], 8/10
While proposals are active (1M)+14.6+14.6 [−44,+73][-44, +73], 4/10−36.6-36.6 [−66,−7][-66, -7], 1/10—

Ten seeds; paired 95% Student-t intervals. Counts give positive differences. Tensor proposals stop at 1M steps. These results appear in Table 30; the smaller five-seed controls are in Table 18.

With the one-step target, proposals lower return. Under the multi-step target, the estimated gain is positive but its interval includes zero. The larger endpoint difference after proposals stop is consistent with a training effect; it does not establish better state coverage or a particular causal mechanism.

Other planner components. The planner–target interaction remains when tensor proposals are disabled. We therefore tested the other changes separately:

ChangeEffect on the one-step settingInteraction after change
Structured planner with the reference execution rule (sample an elite, add noise)+146.6+146.6 [+2,+291][+2, +291], 5/5+356.2+356.2 [+4,+708][+4, +708], 5/5
Reference planner with the structured execution rule (best candidate, no noise)−315.7-315.7 [−456,−176][-456, -176], 0/5+111.3+111.3 [−60,+282][-60, +282], 5/5
Structured planner without retention of the best sequence+90.2+90.2 [+15,+166][+15, +166], 4/5+363.7+363.7 [+187,+541][+187, +541], 5/5
Fitting to all candidates instead of the top K (paper, Condition D)—removes 46% [37,56][37, 56]% of the interaction, 5/5
Reference planner with K = 32 (released default otherwise)+43.7+43.7 [−239,+327][-239, +327]—
Reference planner with the variance rule (released default otherwise)+27.4+27.4 [−247,+302][-247, +302]—

Stair, replay length 13, five seeds. The elite-count and variance rows use the released reference; the others use the main reference. Each effect uses its stated control. The interaction compares the two planners after the named change. Table 29 gives the full contrasts. The interaction for transferring the structured execution rule to the reference is uncertain.

Transferring the best-candidate execution rule to the reference planner lowers its one-step return. Removing that rule, best-sequence retention, and top-K fitting together substantially reduces the structured planner's one-step deficit (Table 29). These interventions support a role for selection pressure on stair. The elite-count and variance tests remain too uncertain to conclude that they have no effect.

Why retain the structured planner? It provides a controlled example of a planning configuration whose training effect changes with the target. The tests do not establish a practical advantage over the target-only agent. Best-candidate execution avoids added action noise during collection, but candidate sampling remains random, and physical deployment was not tested.

The reported interpolation uses two active control-point layers, and proposals stop halfway through training. These facts limit the tested proposal family. We have removed the claim that this is the least favorable setting, since longer horizons were not compared.

Thank you for prompting us to separate the configuration-level finding from the more limited evidence for tensor proposals.

AuthorsOpenReview ↗

Gentle Reminder: Request for Feedback on Our Rebuttal

Dear Reviewer YWNh,

We hope this message finds you well. We understand that this is a busy period, possibly with revisions of your own, but we would be very grateful if you could take a little time to review our rebuttal. We have done our best to address each of your concerns, and this work is very important to us.

If anything is still unclear or you would like us to elaborate on any point, please don’t hesitate to let us know. We are happy to continue the discussion and look forward to your feedback.

Best regards, The Authors

I have no further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Reviewers and Area Chair,

Thank you for your careful reading and for the time you have devoted to this discussion. Your questions helped us clarify the contribution, strengthen the controls, and explain the limits of the results. The revised paper separates the observed planner–target interaction from causal explanations and distinguishes critic calibration from estimation error.

The paper asks how a planning change depends on the value target used during training. The planner collects the critic's data, and the critic scores the next plans. We test this link by crossing two training planners with two targets and using a common evaluation planner.

The revised manuscript brings the extended results into the main text. The interaction is positive on stair, pole, and hurdle, but its source differs. The multi-step target raises the structured planner's return on stair and pole. It also lowers the reference planner's return on stair and hurdle. A positive interaction therefore does not, by itself, establish a gain over the baseline or the target-only method (Tables 1 and 26).

Controls on stair support a role for the execution rule, retention of the best candidate, and elite fitting. The tensor-only test also changes the interaction, although its independent return benefit remains uncertain (Tables 29 and 30). Critic measurements show closer agreement with each planner's own returns under the multi-step target. They do not isolate estimation error against the value defined by the target (Table 32).

The theory explains what the replay target estimates under fixed policies. It separates bootstrap sensitivity from behavior–prior mismatch and retains the endpoint value in finite windows. The experiments test the learning interaction beyond those assumptions. The revised text makes this distinction explicit.

We also report the same-implementation benchmark comparison, target-only controls, the released TD-M(PC)2^2 comparison, and alternative value-learning interventions. These results show task-dependent gains and limits, rather than a reliable advantage over every simpler alternative (Section 5.2; Appendix H).

The responses below address each reviewer's questions. Appendix I gives the run mapping and corrections needed to interpret the comparisons.

Thank you again for helping us make the paper clearer and more precise.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.