- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Safe Dynamic Decision-Region Resolution (SD-DRR). The goal is to answer what a world model actually needs to learn in a situation where an agent will still have safe opportunities to gather more information later on. This is a good motivation because in many practical scenarios learning all the details of the environment is unnecessary. In such cases, the authors claim that there is no need for the agent to identify the exact world in which it is, but to narrow down the possibilities enough so that the decision it is about to make is supported by that set of worlds.
The authors perform some experiments, but the way this is written is quite confusing to me. They combine the experiments in their sentences, which makes it quite confusing for me to follow and therefore reduces the clarity a lot. Other than the settings, it seems like the results do verify the structural and sample-complexity results.
Critical strengths
To me, the obstruction characterization, i.e., identifying which remaining possibilities would prevent the agent from making a safe decision with the limited opportunities it has to gather more information, is one of the strongest parts of the paper and potentially quite impactful. This can help in determining what exactly the offline data should try to distinguish, instead of requiring the model to learn the environment in detail.
I also find that the theoretical parts of the paper are quite carefully derived. Although not written in the clearest way, as the material is quite dense, I found that the paper includes complete and correct proofs, and provides counterexamples wherever needed. This makes the theoretical part of the paper quite reliable.
Critical weaknesses
1- The main theory provided in this paper, to my understanding, describes an agent that can decide whether to act, gather more information, or refuse to act when uncertainty is high. However, the experiments do not seem to be testing this idea fully. What I believe the experiments are doing is to instead test smaller, different components of this idea. What each different task is showing is that the individual components of the idea explored in this paper can help, but the synergy between them and, therefore the full SD-DRR framework is not shown.
2- I am not sure whether some of the improvements observed in the experiments are based on small independently trained models or because of the proposed method. For example, some of the results from the TwoRoom experiment use only 3 training runs, and the improvements seem to vary quite considerably between different runs. In the PushT experiment, the confidence intervals are calculated by sampling from the same set of test tasks while having the training model fixed. They then show that this improvement is consistent across those tasks, but they do not study whether the same improvement would appear if the models were trained again under different data or random seeds. This makes it a bit difficult to assess the reliability and generality of the gains reported in the paper.
As I said above, one of the biggest weaknesses of the paper for me is that the experimental section is very difficult to follow due to the way it combines very different experiments with each other.
Questions for authors
1- Can the authors please provide an experiment in which a learned world model uses the full SD-DRR process, i.e., deciding when to act, when to gather more information, and when to refuse to act all in the same experiment? This would show that the proposed framework works as a whole and not just that individual components of it work.
2- Can I ask if the PushT models were trained multiple times using different random seeds and/or different training datasets? If this is not the case, what is the evidence that we can say that the reported improvement is reliable across different models rather than the specifc ones used here?
3- In the reduced TwoRoom experiment, the improvement reported is 5.33 percent, whereas the variation reported across runs is 6.93. This is again quite concerning, as the variation is larger than the actual improvement. Can you also please comment on what evidence there is to show that the added training loss improves performance?
4 comments
Q2 / W2. PushT with new seeds and new training data. We retrained from random initialization. Two source datasets were drawn independently from the parent corpus (2,335 episodes each, 292,059 and 291,954 frames, 293 episodes in common; the manuscript's reduced source has 291,969 frames). Each was crossed with two blocks of three seeds: 12 backbones, four three-model ensembles. The role mapper is fitted once per source draw and the factor and proposal modules once per ensemble, from that draw's frames only. Within a cell the full interface and the latent-only baseline share the ensemble, the 400 fresh tasks, task and planner seeds, and the planning budget (300 CEM candidates, 30 iterations, 50 steps).
| Source draw | Seed block | Latent-only | Full interface | Paired gain (95% task interval) |
|---|---|---|---|---|
| 0 | 0 | 35.50% | 57.00% | +21.50 pp [16.50, 26.50] |
| 0 | 1 | 33.50% | 49.75% | +16.25 pp [11.25, 21.25] |
| 1 | 0 | 37.00% | 56.50% | +19.50 pp [14.50, 24.50] |
| 1 | 1 | 29.00% | 54.50% | +25.50 pp [20.50, 30.75] |
The gain is positive in all four retrained ensembles: mean +20.69 pp [17.50, 23.88], SD across the four cells 3.87 pp, +18.875 and +22.500 pp by source draw, +20.500 and +20.875 pp by seed block. The manuscript's +15.75 pp on its original ensemble is a separate result.
Ablations on both seed blocks: removing factor scoring costs 8.125 pp [5.125, 11.250] and 10.75 pp [7.50, 14.00]. Removing the visual servo changes success by 0.125 pp [0.000, 0.375] and 0.00 pp and eliminates its extra RGB reads (about 10.4 and 11.1 per task).
We read the servo result on the first task cohort, so we generated a second cohort of 400 new tasks and ran all three controllers on the four ensembles. The full interface again beats latent-only in every ensemble: +10.50, +16.00, +15.75, +18.00 pp (mean +15.06 pp [11.88, 18.25]; 48.94% vs. 33.88%). The servo-free variant has the same success/failure outcome as the full interface on all 1,600 model-task runs (one-sided 95% bound on its success loss: 0.75 pp) and needs 11.72 fewer RGB reads per task. Over the two cohorts all eight ensemble-level gains are positive, from +10.50 to +25.50 pp.
Q3. Does the response loss help at reduced source? We enlarged the study so that the answer no longer rests on three pairs. We trained three new pretraining initializations and four fine-tuning seeds on each: 12 new control/response pairs. Within a pair the two models share initial weights, source frames, training and sampler seeds, augmentation, optimizer, the 4,000-update budget and the planner; they differ only in the response loss. All pairs are evaluated on 400 new tasks.
| Evidence (400 new tasks) | Pairs improving | Mean gain | 95% task interval |
|---|---|---|---|
| Reduced source, 12 new pairs (3 initializations × 4 seeds) | 10 / 12 | +2.67 pp | [1.60, 3.73] |
| Reduced source, the 3 original pairs | 3 / 3 | +6.33 pp | [4.42, 8.42] |
| Full source, 3 new pairs | 3 / 3 | +3.42 pp | [1.75, 5.08] |
| Full source, the 3 original pairs | 3 / 3 | +2.58 pp | [0.92, 4.25] |
In the 12-pair study the two remaining pairs are within 0.75 pp of zero, the three initialization means are +1.750, +1.125 and +5.125 pp (SD 2.15 pp between initializations, 2.37 pp between seeds within an initialization), and the pair-level SD is 2.82 pp. The original +5.33 pp rested on three pairs. The effect is now supported by 21 matched pairs, of which 19 improve, and all four intervals exclude zero. The loss adds no network passes (both arms use 24,000 batched predictor forwards per model). Task intervals are conditional on the trained models; variation across training runs is the pair-level spread above.
Thank you for the detailed response and the substantial additional experiments.
Regarding W1, the new TwoRoom experiment is useful and largely addresses my request for an integrated ACT/PROBE/ABSTAIN evaluation. However, comparing failed ACT rates with an oracle that always acts is not entirely fair, since SD-DRR can abstain. Reporting failure rates conditional on acting, alongside decision coverage, would make this comparison clearer.
Regarding W2, the additional training runs across different seeds and datasets provide convincing evidence that the improvements are not specific to the original models.
Regarding W3, the additional matched experiments substantially strengthen the evidence for the response loss, although I would encourage distinguishing uncertainty across tasks from variability across training runs.
Overall, I appreciate the additional experiments, which address some of my original feedback. I will maintain my original assessment, particularly given the remaining limitations and presentation concerns. I hope these additional results and clarifications will be incorporated into the paper.
Thank you for considering the additional experiments and for recognizing the integrated controller evaluation, the replication across seeds and datasets, and the strengthened evidence for the response loss. We have uploaded a revised manuscript that incorporates these experiments and presentation updates, and we provide the requested statistical breakdown below.
For W1, we report ACT coverage alongside failure conditional on ACT. The setting is the same as in our earlier response: three full-source TwoRoom models, N=0, B=2, 256 calibration tasks and 400 independent confirmation tasks shared across models. The table aggregates the resulting 1,200 model-task outcomes. Coverage is the fraction of tasks ending in ACT; conditional failure is the fraction of those ACTs that miss the goal.
| Controller | ACT coverage (%) | Failures / ACTs (%) | Success / all tasks (%) | Probes / task |
|---|---|---|---|---|
| SD-DRR | 32.83 | 13.96 | 28.25 | 0.468 |
| Entropy probing | 29.50 | 16.38 | 24.67 | 0.851 |
| Identification first | 29.58 | 14.93 | 25.17 | 1.192 |
| Certainty | 17.67 | 38.68 | 10.83 | 0.000 |
| Gain-informed, always ACT | 100.00 | 55.75 | 44.25 | 0.000 |
The first four controllers share the T8 calibration constraint on P(ACT and failure), evaluated over all tasks, with a risk budget of 10%. The gain-informed controller provides a reference at 100% coverage. Among the calibrated controllers in this setting, SD-DRR combines the highest coverage and success rate with the lowest conditional failure rate. It also uses fewer probes than entropy probing, 0.468 versus 0.851 per task, with an observed unconditional failed-ACT rate of 4.58%. With 1,024 calibration tasks on the same three fixed models (N=0, B=2; Section 5.3 and Table 2), SD-DRR reaches 45.17% ACT coverage, 12.36% conditional failure and 39.58% success at 0.515 probes per task, compared with 41.67% coverage, 14.60% conditional failure and 35.58% success at 1.344 probes for entropy probing.
For W3, the reduced-source response-loss study comprises twelve matched training pairs: three independent pretraining initializations, with four fine-tuning seeds per initialization. The following quantities separate task uncertainty from observed training variability; all gains and standard deviations are in percentage points.
| Quantity | Result |
|---|---|
| Mean paired gain | +2.67 |
| 95% paired task-bootstrap interval, 400 common tasks, fixed fitted models | [1.60, 3.73] |
| Mean gain for each pretraining initialization, averaged over its four fine-tuning seeds | +1.750, +1.125, +5.125 |
| Observed SD across the twelve paired training gains | 2.82 |
| SD across the three initialization means | 2.15 |
| Pooled within-initialization fine-tuning SD | 2.37 |
The mean gain is positive for every pretraining initialization, and ten of the twelve matched pairs improve. The task interval quantifies uncertainty from sampling evaluation tasks with the trained models fixed; the reported SDs describe variation across the evaluated training runs and initializations.
In the uploaded 10-page revision, every subsection in Section 5 opens with its own question and protocol and closes with its own conclusion, keeping the exact-game, full-controller, training-objective, visual-planning and test-time inference studies self-contained. Section 5.3 presents the integrated ACT/PROBE/ABSTAIN controller, placing ACT coverage, conditional failure, success and probe use side by side in Table 1 and reporting the 1,024-task calibration confirmation in Table 2 (full protocols and cost accounting in Appendices E.1-E.2 and E.5). Section 5.4 presents the response-loss objective (Table 3) and its 21-pair replication (Table 4), reporting fixed-model 95% task intervals alongside run-to-run Pair SDs in Table 4 and the initialization-level breakdown in Appendix E.3 (Tables 20-21). Section 5.5 presents the PushT interface (Table 5) and its four-ensemble replication across independent source draws and task cohorts (Appendix E.4, Table 22).
Thank you for the constructive review. We ran the experiments you asked for. This part covers Q1 and the organization of the experiments; part 2/2 covers Q2 and Q3.
Q1 / W1. ACT, PROBE and ABSTAIN in one learned system. We added a TwoRoom deployment study with hidden dynamics: one of nine actuator gains (gx,gy)∈{0.6,1.0,1.4}2 scales every command, and the agent is not told which. One controller, built on the learned world model, keeps the set of gains consistent with its RGB observations and at each decision does one of three things. It ACTs when its model predicts that one plan reaches the goal under every gain still supported. It PROBEs (a five-step move whose observed outcome prunes the gain set) when that can still lead to a justified ACT within the remaining budget of B probes. Otherwise it ABSTAINs. We cross B∈{0,1,2} with N∈{0,1,2,4,8} passive observation windows of the target world, for six trained world models (three per source budget). The controller is calibrated on 256 tasks so that failed ACTs stay below 10% of all tasks (T8), frozen, and then run on 400 independent tasks. Every task stays in the denominator.
Full source, no passive windows, two probes (N=0,B=2), averaged over the three models (1,200 model-task runs):
All three behaviours occur in the same experiment, 86% of the ACTs succeed, and most successes come before the gain is identified. For scale, a reference controller that is told the true gain and always acts reaches the goal on 44.25% of these tasks and misses on 55.75%. Without knowing the gain, the calibrated controller reaches 64% of that success rate and cuts failed ACTs from 55.75% to 4.58%.
The components interact as the theory describes. With no passive windows, success rises with the probe budget (21.83%, 26.17%, 28.25% for B=0,1,2), and the controller probes less as passive windows accumulate (0.468, 0.266, 0.209, 0.123, 0.116 probes per task for N=0,1,2,4,8 at B=2). A controller that must identify the gain before acting never acts at N=0,B=1, while SD-DRR succeeds on 26.17% of tasks there. Against information-gain probing, SD-DRR uses 0.15 to 0.41 fewer probes per task in all 20 cells with B>0 at comparable success (within 3.6 pp in every cell); in the cell above, 28.25% vs. 24.67% success with 0.468 vs. 0.851 probes. Success figures are for the three models trained on the full source budget.
W3. Organization of the experiments. Each experiment answers one question: