- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Squeeze-to-Plan (S2P) addresses the cost of planning over high-dimensional frozen visual features by pooling DINOv3 features and proprioception into a 512-dimensional state vector. It trains this representation using reward prediction, latent dynamics prediction, and latent norm regularization, and combines MLP world models with TD-MPC2 value learning and MPPI planning. Experiments on three modified ManiSkill3 tasks report one to two orders of magnitude faster planning than the evaluated DINO-based baselines, improved success under simulated planning delays, and robustness to three appearance perturbations.
Critical strengths
-
The planning-efficiency improvement is substantial and directly measured: Global pooling permits compact MLP rollouts instead of repeatedly predicting a patch-token state; the latency reduction is reported; observation encoding plus the complete planner call on the same L40S GPU are measured.
-
The complete system demonstrates useful closed-loop control under a time constraint. With a 10-second simulated execution budget, Table 2 reports 62.5% LiftPeg success versus 0% for all baselines, and 37.5% PickCube success versus 7.5% for the strongest baselines, averaged over four visual settings.
Critical weaknesses
-
S2P trains online, while the baselines train offline on a fixed 1.5-million-transition dataset (§5.3; Appendices A.3–A.4). S2P also explicitly includes TD-MPC2 value learning and policy-proposed trajectories, whereas the baseline adaptation documents a reward head without establishing comparable terminal-value or proposal mechanisms (§4.2; Table 6). With evaluation horizons of only 1–5 steps, these differences could materially affect control, and matched-data/planner controls and non-RT success rates are needed to separate learned control quality from latency penalties. This is a substantial attribution gap.
-
Appendix A.1 provides a valid stochastic variational lower bound, but fixed nonzero Gaussian covariance still requires expectations over encoder noise. Moreover, for Eq. (6) as written, rescaling the latent by a factor c ∈ (0,1), with compensating rescaling of the dynamics and reward predictors, preserves its information and reward predictions while reducing both dynamics MSE and the latent norm penalty by c². Thus, a smaller norm need not imply removal of irrelevant information, and the paper does not connect this surrogate to the bisimulation aggregation condition behind Eq. (3).
-
The contribution of the proposed objective beyond established compact world-model learning is insufficient. For example, DeepMDP (Gelada et al., 2019) already learns representations through reward and next-latent prediction, while TD-MPC2 (Hansen et al., 2024) combines latent prediction, reward/value learning, and online planning.
-
The reported uncertainty is insufficient for the smaller performance and ablation claims. Figure 4 and Tables 2–3 use one training seed, so their SEMs describe evaluation variability rather than variability in learning. Results also select the best among up to 54 planner configurations using only 10 evaluation episodes per baseline configuration and 30 for S2P, without a separately described confirmation evaluation.
Questions for authors
-
Do the baselines use comparable terminal values and policy proposals, and what is S2P’s total environment-interaction budget? Please offer a matched-data comparison, including success without injected delays, separate control-learning differences from the effect of planning speed.
-
Is encoder noise sampled during training, or is Eq. (6) evaluated only at encoder means? What implementation constraint or evidence rules out latent rescaling as the explanation for the norm penalty, and supports the claimed information-removal mechanism?
-
Does S2P outperform an identical pooled-DINO/MLP controller trained with conventional reward/dynamics supervision when data, value learning, planning, and optimization budgets are matched, with repeated training seeds and independently evaluated planner selection?
3 comments
Thanks for the reponse and experiments. I still have several questions:
-
S2P uses a planner to collect training data, while the baselines use only their policy networks. Why is there a difference, is it fair?
-
What benefit does the KL term bring? Now, removing the KL term gives similar or even better results.
-
The baselines still use only one training seed. Can you use more seeds?
Thank you for reviewing our newest results and experiments.
“S2P uses a planner to collect training data, while the baselines use only their policy networks. Why is there a difference, is it fair?”
We acknowledge this asymmetry. When adapting the baselines to online training, we found that collecting training data with planning would make each baseline run take ~ 27 days on an NVIDIA H100 (because planning with baselines is 1–2 orders of magnitude slower than S2P, Fig. 3). At that point, re-running all our S2P experiments under policy-head collection was not feasible within the rebuttal timeline. We acknowledge that training S2P using the policy to collect data would strengthen the comparison, and will train S2P with this method for the final version of our paper.
We do not expect that this change will substantially alter our results, and we can offer some preliminary evidence for this. We trained S2P with policy-head collection on LiftPeg, PushCube, and PickCube (two seeds, still running), evaluating periodically with the same MPPI planner used throughout our main S2P runs. On PickCube, the evaluation returns and success rates of policy-collected and planner-collected S2P closely overlap through ~500K gradient steps, reaching ~30-40 (a fully trained S2P agent reaches ~40). In PushCube and LiftPeg they appear to trend similarly, or show policy-collection potentially ahead on LiftPeg. However both PushCube and LiftPeg are earlier in training at ~250K steps, so the comparison is not yet as informative. These results suggest that, at least during early training, S2P learns comparably under both data-collection methods.
To view the training curves, please see policy_acting_vs_planning_return_2x3.png in our supplementary material.
“What benefit does the KL term bring? Now, removing the KL term gives similar or even better results.”
We agree that, on LiftPeg and Distracting LiftPeg, removing the KL term results in similar, or possibly improved success rate (Fig. 5a, Fig. 6a). However, the presence of the KL term provides a mechanism to tune the degree of compression by adjusting the value of . Our results on distracted LiftPeg suggest that may have some small, but consistent effect on decreasing the amount of distracting information contained in the S2P latent. Below, we tabulate the per-seed probe accuracies and variance shares (i.e. bar heights in Fig. 6b,c).
| Seed | Probe Accuracy (%): S2P | Probe Accuracy (%): S2P-NoKL | Variance Shares (%): S2P | Variance Shares (%): S2P-NoKL |
|---|---|---|---|---|
| 1 | 25.97 | 28.43 | 0.456 | 0.532 |
| 2 | 26.17 | 26.67 | 0.558 | 0.447 |
| 3 | 25.63 | 28.19 | 0.589 | 0.572 |
| 4 | 26.26 | 28.73 | 0.518 | 0.650 |
In probe accuracy, we see that all S2P () seeds are below all seeds of S2P-NoKL. Variance shares do not show as clear of a trend. While the magnitude of the effect is small, we suspect that it may become more pronounced in tasks with more distractors, which we plan to explore in future work.
“The baselines still use only one training seed. Can you use more seeds?”
Running multiple seeds for our DINO-based baselines is not feasible within the timeline of the rebuttal period; running one seed of all baselines requires 757 hours (~31 days) on NVIDIA H100 GPUs. We have updated our discussion in Sec. 5.3 to explicitly mention that this is the reason for running just one seed of the baselines. The original papers for our baselines likewise do not specify training seed variability (Zhou et al. 2025; Chun et al. 2026; Fu et al. 2026; Toso et al. 2026).
Response to Reviewer fuu8
We thank Reviewer fuu8 for their time and helpful feedback, in particular for the insight into the limitations of our deterministic simplification of the BIB loss. In our updated revision, we have matched training protocols, value learning and policy proposals between S2P and baselines, and implemented S2P as a fully stochastic model, with the full variational BIB loss. We also provide evidence that S2P’s BIB loss filters irrelevant information, via a probing analysis on a distracting task.
Addressing Weaknesses
To address the training-protocol confound in the comparison to DINO-based methods, we train S2P and all DINO-based baselines online, under matching training protocols. In addition, we provide the baselines with value & policy heads to match S2P. Under equivalent architectures and training protocols, S2P retains its success rate advantage against the DINO-based baselines (Table 1).
We thank the reviewer for their insight into how a rescaling of the latent can minimize the deterministic form of the BIB objective (original submission, Eq. 6). To account for this, we have updated our S2P implementation, to directly optimize the variational lower bound in Eq. 4 using a stochastic encoder.
To further support our contribution of the BIB as a compressive loss, we compare our BIB loss against several other losses used to train compact world models. This comparison shows that other losses either fail to train the S2P architecture (Fig. 5b), or, in the case of TD-MPC2, retain more distractor information (Fig. 6b,c).
To better characterize the variance across training seeds, we train S2P, BIB ablations, and the alternate compression losses under three different training seeds. In addition, we evaluate S2P and the DINO-based baselines over 25 episodes per planner configuration, and the ablations and alternate losses over 50. Due to their high online training cost (Sec. 5.3), we keep one training seed for the DINO-based baselines. Our results remain consistent when adding these additional training seeds and evaluation episodes (Table 1, Fig. 4).
Responses to Questions
Yes – in our revised comparison, we provide the baselines with a value head and policy proposal, matching S2P’s, and train all methods online with matching training protocols (1M gradient descent steps, one transition collected per step). We also report results without simulated latency effects. Our results show that S2P maintains its success rate advantage under this matched evaluation (Table 1, Fig. 4).
Our updated method samples encoder noise during training, directly optimizing our variational objective with stochastic models. Due to the now-active entropy term, shrinking the latent no longer reduces the loss, and removing the entropy term, which reintroduces this degeneracy, causes training to fail (Fig. 5a).
We additionally provide a probing analysis on S2P latents, trained over a distracting task. Our probes (Section A.6) show that the BIB loss removes task-irrelevant information better than a TD-MPC2 loss does (Fig. 6b,c).
Yes – our updated submission now includes a multi-seed comparison of training S2P with our BIB loss, vs other losses. We train each loss online, under matched training protocols and architectures. This comparison shows that other losses either fail to train the S2P architecture (Fig. 5b), or, in the case of TD-MPC2, retain more distractor information (Fig. 6b,c).
To avoid planner-selection bias, Fig. 5 reports the full distribution over all 54 planner configurations rather than the best one, and Table 1b uses a single fixed configuration.