- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ISI-WM, a training framework for learning visual world models that ignore background changes while preserving distinctions associated with divergent action outcomes. Built on TD-MPC2, it combines cross-background state alignment with outcome-gated action-fork learning, separation, and ranking objectives. Separation and ranking are activated when alternative actions produce sufficiently different simulator states. Evaluation on six DMC tasks shows improved average return under unseen video backgrounds, without adding trainable parameters or changing the RGB-only deployment interface.
Critical strengths
The strengths of the paper are:
- The insight about distinction between nuisance invariance and action sensitivity is well motivated. Aligning observations of the same physical state under different backgrounds gives a clear supervision signal for irrelevant variation. Conditioning separation on realized outcomes is also more principled than assuming that different commanded actions must always produce different states. The paper carefully describes these as controlled training interventions without claiming causal identification or a formal abstraction guarantee.
- The improvements over the TD-MPC2 model are substantial. On unseen backgrounds, the average return increases from
216to434, approximately doubling performance. Particularly large gains occur on Cup Catch, from 67 to 537, and Reacher Easy, from 231 to 630. The method also exceeds HRSSM’s macro-average of 278, although it does not outperform HRSSM on every task. - The ablations provide useful evidence that the two supervision sources complement each other. In ablation study, unseen background macro-average returns are 214 without interventions, 294 with background alignment, 264 with action forks, and 433 with both. On Cup Catch, the combined method reaches 527, compared with 166 and 169 for the individual components. These results support the combined design beyond simply showing an improvement over the base model.
- As for the model size and latency, ISI-WM retains 4.86 million parameters, and median action latency remains essentially unchanged at 25.78 ms versus 25.85 ms for TD-MPC2, demonstrating the efficiency of the proposed method. The authors also report the costs rather than presenting the method as free: training time increases from 11.9 to 14.8 hours, and peak inference allocation rises from 66.9 to 97.4 MB.
Critical weaknesses
- Basically, ISI-WM receives auxiliary trajectories constructed through simulator replay, exact background correspondences, and state derived outcome labels. Its roots are generated using a frozen policy, whereas the principal baselines do not receive this auxiliary supervision. Reporting 300k control steps for each method therefore does not establish equal total interaction or information budgets. The authors should clarify the cost of obtaining the frozen policy and constructing the auxiliary data, including repeated action prefix replay. More importantly, a baseline trained on the same additional branch trajectories using ordinary prediction objectives would help determine how much improvement comes from the proposed losses.
- The central outcome gating mechanism is not isolated by the ablation studies. The comparisons are informative, but the fork component bundles prediction, separation, ranking, and outcome gating. Consequently, they do not establish whether the distinctive contribution (the use of realized physical divergence to decide when separation is appropriate) is necessary. Comparisons against prediction-only auxiliary training, ungated separation, and a fixed-margin alternative would distinguish the proposed mechanism from more conventional auxiliary dynamics. This is particularly important because the paper’s conceptual novelty rests heavily on selective, outcome dependent supervision.
- Evidence for better action prediction is limited. Action induced latent distance increases from 0.024 to 0.089, but branch ranking accuracy improves only from 0.64 to 0.66. This leaves the connection between greater latent separation and improved predictive accuracy insufficiently demonstrated.
- The gating settings lack sensitivity analysis. The method fixes the threshold at 0.05, intervention magnitude at 0.8, and horizon at 3 steps. Acrobot produces 0 eligible comparisons in the held out representation analysis, raising questions about when the gate supplies useful supervision.
Questions for authors
- Could the authors clarify whether the reported 300k control steps include collecting the auxiliary trajectories and training the frozen clean policy? If these costs are additional, reporting them would allow a clearer comparison of the total training budgets.
- Do the authors have existing ablations comparing outcome gating with prediction only or ungated supervision using the same auxiliary data? These comparisons would help distinguish the contribution of the gating mechanism from the benefit of additional trajectories.
- Could the authors report the fraction of comparisons that passes the gate for each task and explain why none are eligible for Acrobot in the held out analysis? This would clarify how consistently the action fork objectives provide supervision across tasks.
4 comments
Thank you for asking us to isolate the outcome gate and connect the representation analysis to prediction and planning.
Full is the submitted four-loss ISI-WM. Control returns below are unseen-background means across seeds 6–8, with 20 episodes per seed.
W2
The central outcome gating mechanism is not isolated by the ablation studies. The comparisons are informative, but the fork component bundles prediction, separation, ranking, and outcome gating. Consequently, they do not establish whether the distinctive contribution (the use of realized physical divergence to decide when separation is appropriate) is necessary. Comparisons against prediction-only auxiliary training, ungated separation, and a fixed-margin alternative would distinguish the proposed mechanism from more conventional auxiliary dynamics. This is particularly important because the paper’s conceptual novelty rests heavily on selective, outcome dependent supervision.
(Physical eligibility and margin controls)
Response. Thank you for asking which part of the fork objective drives the gains. Q2 gives the same-data prediction comparison. On Walker, Full reaches 365.21 and making all recorded pairs eligible yields 327.31, with Full higher in each seed. We then fix the margin at 0.09385, the median adaptive margin over eligible training pairs. The physical gate reaches 373.82 and randomizing eligibility within each batch and horizon while preserving the count yields 372.94 (Appendix E, Table 8). These close means leave the benefit of physical selection at a matched rate open. A focused follow-up would repeat the fixed-margin, rate-matched comparison on Cup and Finger at the same three seeds.
W3
Evidence for better action prediction is limited. Action induced latent distance increases from 0.024 to 0.089, but branch ranking accuracy improves only from 0.64 to 0.66. This leaves the connection between greater latent separation and improved predictive accuracy insufficiently demonstrated.
(Predictive accuracy and shared-candidate planning)
Response. Thank you for asking for a direct prediction test. On the same held-out pairs, we count an identification as correct when the prediction is strictly closer to the executed branch than to an eligible alternative. The task-level means show the clearest gains on Cup and Finger, with variation across tasks.
| Task | Full | TD-MPC2 |
|---|---|---|
| Cartpole | 49.63% | 49.16% |
| Cup | 94.19% | 50.53% |
| Finger | 89.20% | 82.77% |
| Reacher | 52.18% | 61.33% |
| Walker | 53.13% | 51.14% |
With the same auxiliary trajectories, prediction only gives 69.72% on Cup and 51.21% on Walker (Table 10). Appendix E.2 reports correct-branch MSE in each model's own latent space. We also let the planners score the same 64 three-step candidates at 100 states per seed, using realized return discounted at 0.99 as the reference. The mean gap to the best candidate is 0.0890 for Full and 0.1220 for TD-MPC2 on Cup, and 0.3444 and 0.3343 on Walker. Candidate returns vary at 76/300 Cup states and 300/300 Walker states (Appendix E, “Ranking ablation and shared-candidate planning”). These tests show where the larger separation accompanies better branch identification or candidate selection.
W4
The gating settings lack sensitivity analysis. The method fixes the threshold at 0.05, intervention magnitude at 0.8, and horizon at 3 steps. Acrobot produces 0 eligible comparisons in the held out representation analysis, raising questions about when the gate supplies useful supervision.
(Threshold, magnitude and horizon sensitivity)
Response. Thank you for asking how the settings affect coverage. In six-task replay, raising the threshold from 0.05 to 0.10 retains 600/600 eligible pair-steps on Cup and changes Walker from 4,124/4,680 to 3,744/4,680. Extending the horizon to five steps at 0.05 gives 0/300 on Acrobot (Table 12). In matched replay, reducing the first-action offset from 0.8 to 0.4 changes Walker eligibility from 79.1% to 70.6%. Separate Walker retraining yields 379.04 at threshold 0.10 and 389.24 with a five-step auxiliary horizon, with a three-step planner throughout (Table 8). For Acrobot, a follow-up would train at 0.02 and 0.05 with the same data and three seeds to test whether the extra eligible pairs improve control.
Thank authors for the detailed rebuttal with additional clarifications and experimental results. In the first response, the matched-budget comparison addresses my concern about the extra training data on Walker. It's convenicing that ISI-WM still outperforms TD-MPC2 when both use 594,400 control steps. The same-data ablations also show that adding gated separation improves returns on Cup and Walker. The per-task results explain why no Acrobot pairs passed the gate and show that branch identification improved much more on Cup than on Walker.
My remaining concern is how the method chooses which pairs to separate. More specifically, on Walker, the physical gate performs better than using all pairs (365.21 vs. 327.31). However, as given in the additional materials, it shows that, with the same margin and number of pairs, physical and random selection give nearly identical returns (373.82 vs. 372.94). I’d appreciate a brief note or discussion or analysis in the main text acknowledging that the benefit of using physical divergence to select pairs is still unclear.
I would maintain my current score : 2 now. Thank authors for the effort again.
Thank you for your follow-up and for recognizing the matched-budget and same-data controls.
My remaining concern is how the method chooses which pairs to separate. More specifically, on Walker, the physical gate performs better than using all pairs (365.21 vs. 327.31). However, as given in the additional materials, it shows that, with the same margin and number of pairs, physical and random selection give nearly identical returns (373.82 vs. 372.94). I’d appreciate a brief note or discussion or analysis in the main text acknowledging that the benefit of using physical divergence to select pairs is still unclear.
(What physical selection adds at a matched margin and eligibility count)
Response. Thank you for suggesting this clarification. We have revised the “Outcome selection” paragraph in Section 4.3. On Walker, Full achieves a higher mean unseen-background return than all-eligible supervision (365.21 vs. 327.31). With the margin and eligibility count matched, physical and random selection yield similar means (373.82 vs. 372.94; Appendix E.2, Table 8). Accordingly, we limit the conclusion from this comparison to the benefit of selective supervision on Walker and describe the physical gate as an implementation based on realized action outcomes.
Thank you for the careful review and for asking us to account for the extra interaction and isolate what the auxiliary losses contribute.
Full is the submitted four-loss ISI-WM. Returns below are for unseen backgrounds and report the mean across seeds 6–8, with 20 episodes per seed. Where shown, ± denotes the sample SD.
Q1
(Training and data collection budgets)
Response. Thank you for asking us to make the interaction budget explicit. The reported 300,000 control steps cover online training. Training the shared collector on clean inputs takes 30,000 steps, and collecting the auxiliary training trajectories takes another 68,400–264,400 per task. Walker uses 264,400 (9,600 initial prefix, 249,600 prefix replay, 5,200 branch continuation). Charging both one-time costs in full to each seed gives 594,400 control steps, or 1,188,800 simulator steps with action repeat two. At that budget, Full reaches 365.21 ± 65.18 and TD-MPC2, which spends all 594,400 steps online, reaches 194.10 ± 5.72. We account for the 132,200 Walker validation/test collection steps separately (Appendix E, Table 7).
Q2
(Same-data and gate controls)
Response. Thank you for suggesting the same-data ablations. We train ordinary action-conditioned prediction on the same auxiliary trajectories. It reaches 44.80 on Cup and 135.33 on Walker, while Full reaches 527.18 and 365.21. With ranking off and alignment and prediction held fixed, adding gated separation raises return from 136.92 to 676.22 on Cup and from 255.40 to 432.91 on Walker, with gains in all three seeds in both tasks (Appendix E, Table 6 and Table 7). On Walker, making every recorded pair eligible in Full yields 327.31 (Table 8). W2 compares the eligibility rules at a matched margin and selection rate.
Q3
(Gate coverage and Acrobot)
Response. Thank you for asking how often the gate supplies supervision. At threshold 0.05, the revised appendix reports these counts across ten held-out roots per task and visual condition, and the mean training pass rates at steps 250,500–300,000 (Table 9).
Every Acrobot test pair stays below the standardized gap threshold over steps 1–3, so its gated prediction metrics are undefined. Lowering the threshold to 0.02 admits 18/180 pair-steps in replay (Table 12).
W1
(Interaction budget and auxiliary information)
Response. Thank you for distinguishing interaction from access to privileged supervision. The Walker comparison in Q1 matches simulator steps. The ordinary predictor in Q2 receives the same auxiliary branch trajectories and paired renders, and learns from executed-branch targets. This tests how far ordinary prediction goes with the same trajectories. Exact replay, paired rendering and physical-state labels remain requirements for constructing Full's training supervision (Appendix E, “Training supervision and interaction budget”).