- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the choice of planner interacts with the value learning target in model-based reinforcement learning. The authors propose TD-MTP, which combines tensor-graph planning with multi-step λ-returns within TD-MPC2. They evaluate the method on continuous-control tasks from HumanoidBench, DMControl, Meta-World, and MyoSuite. The main finding is that a planner that performs poorly with one-step value targets can work better with multi-step targets. The paper also provides a theoretical analysis of bootstrap error and policy mismatch.
Critical strengths
This paper conducts a highly valuable study on the interplay between planning and value learning. I particularly appreciate the 2×2 experimental design, which compares the performance of two planners across single-step and multi-step goals. The results from the "stair" and "pole" tasks are especially compelling, powerfully demonstrating that an initially inefficient planning configuration can be transformed into an efficient one simply by altering the value objective. Furthermore, the authors present not only successful cases but also negative results and supplementary control experiments, thereby bolstering the credibility of their empirical findings. (This is important for my rating.)
Critical weaknesses
My primary concern is that the contribution of the "structured planner" proposed in the paper remains unclear. The paper combines existing techniques—specifically tensor-graph planning and multi-step value learning, so I expected to see compelling evidence demonstrating the efficacy of this specific combination. However, the results suggest that the majority of the performance gains likely stem from the multi-step target itself. As shown in Table 1, adding the planner alone improved average returns on only one of the ten tasks, whereas adding the multi-step target alone yielded improvements on eight tasks. More importantly, the ablation studies in Table 15 failed to clearly substantiate the benefits of the tensor proposals. For a paper proposing a novel combination of methods, I consider this a significant limitation.
Furthermore, the overall performance improvements were inconsistent. Across 38 evaluation task settings, TD-MTP achieved a record of 10 wins, 22 ties, and 6 losses. In pairwise experiments, only two of the ten "planner-target" combinations showed confidence intervals that did not include zero. Performance actually deteriorated significantly on some tasks; notably, in the "stair" task, the method's performance advantage over the baseline vanished when training was extended to 4 million steps. While the theoretical analysis is valuable, it primarily characterizes error trade-offs under a fixed policy and does not directly explain the observed improvements in training performance. Overall, although I find the interaction effects interesting, I am not convinced that TD-MTP offers a reliable advantage over simpler alternatives.
These two points are the primary reasons for my recommendation to reject the paper.
Questions for authors
Q1: Could the authors provide additional comparative experiments that isolate the effect of enabling "tensor proposals"—keeping elite fitting, action selection, the replay window, and other settings strictly constant? The current results do not clearly demonstrate the specific contribution of tensor sampling.
Q2: Could the authors provide further comparative results under identical conditions against TD-MPC2 and the "target-only" baseline? Ideally, these should use the same implementation code and training random seeds.
Q3: Is there direct evidence that multi-step targets improve the Critic's accuracy in evaluating actions selected by the structured planner?
4 comments
Response to 8Mw8 (2/2): critic measurements
Q3. Is there direct evidence about critic accuracy on selected actions? We report calibration against each planner's own returns. The multi-step critic agrees more closely with those returns, but the comparison does not isolate estimation error against the policy value defined by the target.
We sampled states from each setting's own trajectories and compared the critic's value of the executed action with the discounted return under that setting's planner. The settings use matched replay windows and the released reference planner (Table 32).
| Setting | Critic estimate | Measured return | Signed error | Absolute error | Correlation |
|---|---|---|---|---|---|
| TD-MPC2 (reference planner, one-step) | 76.8 | 66.7 | 10.7 | 0.46 | |
| +Target (reference planner, multi-step) | 71.2 | 71.6 | 5.9 | 0.68 | |
| +Planner (structured planner, one-step) | 78.1 | 57.8 | 20.4 | 0.52 | |
| TD-MTP (structured planner, multi-step) | 77.5 | 78.5 | 2.4 | 0.74 |
Stair at 2M steps; ten seeds and 24 states per seed (240 states per setting), replay length 13. Returns cover 500 steps with discount 0.99. “Error” is relative to the measured own-planner return, and correlation is across states. Table 32 gives the measurement protocol.
The one-step structured critic has the largest positive gap to its planner's return. The multi-step setting has smaller mean signed and absolute gaps. The signed gap decreases on every seed. This supports closer calibration on the trajectories where that planner acts.
The comparison has two limits. First, the settings visit different states and use different continuation planners. Second, the target changes the value being estimated. The one-step operator evaluates the policy prior. The infinite-window operator evaluates a mixture of the prior and behavior planner. Finite windows retain the prior bootstrap and need not have that same fixed point. Agreement with own-planner returns can therefore change without a corresponding reduction in estimation error. We do not treat this measurement as the uniform bootstrap perturbation in Proposition 1.
Candidate ranking. The across-state correlation in the table is different from ranking candidates within each state. On each setting's own states, this ranking test does not show improved ranking under the multi-step target. One contrast is lower for TD-MTP than target-only; the remaining contrasts are unresolved (Appendix G).
We first specified a stricter design with common states, a common continuation planner, and candidates that differ only over the planning horizon. It could not resolve differences between settings because continuation noise was large relative to the candidate-return differences. We specified the own-trajectory measurement after this result and report it as secondary. Both designs remain in Appendix G.
The theory explains the target's policy-evaluation tradeoff under fixed policies. The component interventions test the training effect, and the critic measurements describe calibration under the tested continuations. Together they narrow the interpretation, but do not prove a unique mechanism for the learning gains.
Thank you again. Your questions led us to separate these claims and make the evidence and its limits easier to assess.
Thank you for the detailed rebuttal and additional experiments. I appreciate the expanded 20-seed comparisons and improved controls, which provide stronger evidence for the planner–target interaction.
However, my main concern remains unresolved. The new results still do not establish a reliable benefit from tensor proposals themselves, and the simpler target-only baseline matches or exceeds TD-MTP in mean performance on most additional tasks where both were evaluated. Furthermore, while the critic calibration results are interesting, they do not conclusively establish reduced value estimation error or improved candidate ranking.
Overall, I find the evidence for target-dependent planning effects more convincing, but the specific contribution and practical advantage of TD-MTP remain unclear. I therefore maintain my original rating.
Dear Reviewer 8Mw8,
Thank you for reading the new results and for telling us clearly where you stand. We agree with your three points, and the revised paper says the same. We do not claim that TD-MTP is better in practice than the target-only agent.
We have one more point. In your review, you noted that most of the gains come from the multi-step target. With 20 seeds, we see that the effect of the target also depends on the planner (Table 26).
- With the reference planner, the target decreases the return on stair ( ) and on hurdle ( ). It increases the return on pole ( ).
- With the structured planner, the same target increases the return on stair ( ) and on pole ( ). On hurdle, the result is not clear ( ).
On stair, the multi-step target increases the return of the structured planner, but it decreases the return of the reference planner. In the same way, the structured planner decreases the return with the one-step target, but we see no loss with the multi-step target. So the effect of the target depends on the planner, and the effect of the planner depends on the target.
The target-only agent is the simpler choice, and it helps on many tasks. But these results show that the two choices cannot be tested one at a time. We believe that this is the contribution of the paper. The evidence has matched controls and one prediction that we made before the runs. We use TD-MTP to measure this effect. We do not propose it as a method that gives a better return.
Thank you again for your time. Your questions led to these potentially insightful experiments.
Best regards, The Authors
Response to 8Mw8 (1/2): proposal controls and matched comparisons
Dear Reviewer 8Mw8,
Thank you for your careful questions and for recognizing the value of the two-by-two design and negative results. We agree that the paper must distinguish the contribution of tensor proposals from that of the target.
Q1. Isolated tensor proposals. The new control holds fitting, action selection, replay length, candidate budget, seeds, and evaluation fixed. It finds a positive tensor–target interaction on stair, but the proposal benefit under the multi-step target remains uncertain (Table 30). The smaller controls are also inconclusive about a return benefit (Table 18). The one-component tests point to a role for selection pressure; they do not establish a general advantage of the full planner (Table 29).
Q2. Matched planner and target-only comparisons. The extended study uses one JAX implementation, seed-matched contrasts, and a common evaluation planner:
Paired 95% Student-t intervals; counts give positive interactions. The hurdle prediction was specified before the additional runs. Table 26 reports all target and planner contrasts.
The pooled interaction is positive on all three tasks. On stair and pole, the multi-step target raises the structured planner's return. It also lowers the reference on stair and hurdle. The target effect on the structured planner is unresolved on hurdle. This distinction matters: a positive interaction does not show that the combined method beats the baseline. The planner-effect intervals under the multi-step target include zero and do not establish equivalence.
On the additional seeds alone, the interaction intervals exclude zero on stair and hurdle, but include zero on pole. The remaining HumanoidBench and DMControl interaction intervals include zero. Humanoid-walk is an exception to a simple explanation based on the one-step planner deficit: the structured planner has lower mean return under both targets, without a positive interaction (Table 25).
Matched-window and released-planner controls retain a positive stair interaction (Table 27). The target-only suite results also show why the simpler method matters: it matches or exceeds TD-MTP's mean on most of the additional tasks where both were tested (Table 13).
Longer training. The seven-seed study uses one run group and keeps the tensor schedule fixed, with proposals stopping at 1M:
Table 24 gives the method means. Both checkpoints use the same seven runs per setting. The 2M value averages three evaluations centered on that checkpoint; the 4M value uses the last three.
At 4M, the multi-step target still raises the structured planner's return, but the interaction interval includes zero. TD-MTP, target-only, and the reference have similar means; the one-step structured planner still trails. We have removed the claim that all four settings converge.
These results support a target-dependent planning effect within the tested budgets. They do not establish a reliable advantage over the target-only agent. The revised discussion acknowledges both concerns in your review and retains the negative results that limit the interpretation.