acceptodds
Under review as a conference paper at ICLR 2027

Safe Dynamic Decision-Region Resolution for Data-Efficient World Models

Abstract

An agent can become better informed and still lose its last opportunity to make a justified decision. We ask what a world model must learn offline when some uncertainty can still be resolved safely at deployment, and formalize this as Safe Dynamic Decision-Region Resolution (SD-DRR): offline evidence needs to remove only the decision conflicts that safe online interaction cannot resolve within budget. A decision-nuisance factorization preserves the optimal resolution cost. In finite realizable games with deterministic passive observations, minimal budget obstructions characterize the required evidence, and their covering geometry yields optimal query allocations with matching exponential failure rates. Continuation responses specify the required prediction accuracy, and calibrated cost-risk frontiers extend the analysis to continuous stochastic histories and misspecified neural predictors. In finite games, SD-DRR needs fewer passive observations under a 90% retention criterion, and convex query allocation attains the predicted exponent. Response supervision raises TwoRoom success by 3.50 to 5.50 percentage points at three source budgets. On 400 fresh PushT windows, the role interface adds 15.75 percentage points at a fixed source budget, and convex inference raises the return of a frozen Reacher agent by 54.48 points over 400 episodes.

Primary area:
reinforcement learning
Keywords:
world models, safe decision-making, active information gathering, decision-region resolution, epistemic uncertainty, model-based reinforcement learning, abstention, sample efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V3BAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Safe Dynamic Decision-Region Resolution (SD-DRR). The goal is to answer what a world model actually needs to learn in a situation where an agent will still have safe opportunities to gather more information later on. This is a good motivation because in many practical scenarios learning all the details of the environment is unnecessary. In such cases, the authors claim that there is no need for the agent to identify the exact world in which it is, but to narrow down the possibilities enough so that the decision it is about to make is supported by that set of worlds.

The authors perform some experiments, but the way this is written is quite confusing to me. They combine the experiments in their sentences, which makes it quite confusing for me to follow and therefore reduces the clarity a lot. Other than the settings, it seems like the results do verify the structural and sample-complexity results.

Critical strengths

To me, the obstruction characterization, i.e., identifying which remaining possibilities would prevent the agent from making a safe decision with the limited opportunities it has to gather more information, is one of the strongest parts of the paper and potentially quite impactful. This can help in determining what exactly the offline data should try to distinguish, instead of requiring the model to learn the environment in detail.

I also find that the theoretical parts of the paper are quite carefully derived. Although not written in the clearest way, as the material is quite dense, I found that the paper includes complete and correct proofs, and provides counterexamples wherever needed. This makes the theoretical part of the paper quite reliable.

Critical weaknesses

1- The main theory provided in this paper, to my understanding, describes an agent that can decide whether to act, gather more information, or refuse to act when uncertainty is high. However, the experiments do not seem to be testing this idea fully. What I believe the experiments are doing is to instead test smaller, different components of this idea. What each different task is showing is that the individual components of the idea explored in this paper can help, but the synergy between them and, therefore the full SD-DRR framework is not shown.

2- I am not sure whether some of the improvements observed in the experiments are based on small independently trained models or because of the proposed method. For example, some of the results from the TwoRoom experiment use only 3 training runs, and the improvements seem to vary quite considerably between different runs. In the PushT experiment, the confidence intervals are calculated by sampling from the same set of test tasks while having the training model fixed. They then show that this improvement is consistent across those tasks, but they do not study whether the same improvement would appear if the models were trained again under different data or random seeds. This makes it a bit difficult to assess the reliability and generality of the gains reported in the paper.

As I said above, one of the biggest weaknesses of the paper for me is that the experimental section is very difficult to follow due to the way it combines very different experiments with each other.

Questions for authors

1- Can the authors please provide an experiment in which a learned world model uses the full SD-DRR process, i.e., deciding when to act, when to gather more information, and when to refuse to act all in the same experiment? This would show that the proposed framework works as a whole and not just that individual components of it work.

2- Can I ask if the PushT models were trained multiple times using different random seeds and/or different training datasets? If this is not the case, what is the evidence that we can say that the reported improvement is reliable across different models rather than the specifc ones used here?

3- In the reduced TwoRoom experiment, the improvement reported is 5.33 percent, whereas the variation reported across runs is 6.93. This is again quite concerning, as the variation is larger than the actual improvement. Can you also please comment on what evidence there is to show that the added training loss improves performance?

4 comments

Thank you for the constructive review. We ran the experiments you asked for. This part covers Q1 and the organization of the experiments; part 2/2 covers Q2 and Q3.

Q1 / W1. ACT, PROBE and ABSTAIN in one learned system. We added a TwoRoom deployment study with hidden dynamics: one of nine actuator gains (gx,gy)∈{0.6,1.0,1.4}2(g_x,g_y)\in\lbrace 0.6,1.0,1.4\rbrace^2 scales every command, and the agent is not told which. One controller, built on the learned world model, keeps the set of gains consistent with its RGB observations and at each decision does one of three things. It ACTs when its model predicts that one plan reaches the goal under every gain still supported. It PROBEs (a five-step move whose observed outcome prunes the gain set) when that can still lead to a justified ACT within the remaining budget of BB probes. Otherwise it ABSTAINs. We cross B∈{0,1,2}B\in\lbrace 0,1,2\rbrace with N∈{0,1,2,4,8}N\in\lbrace 0,1,2,4,8\rbrace passive observation windows of the target world, for six trained world models (three per source budget). The controller is calibrated on 256 tasks so that failed ACTs stay below 10% of all tasks (T8), frozen, and then run on 400 independent tasks. Every task stays in the denominator.

Full source, no passive windows, two probes (N=0,B=2N=0,B=2), averaged over the three models (1,200 model-task runs):

OutcomeShare of all runs
ACT32.83%
ACT and reach the goal28.25%
ACT and miss the goal4.58%
PROBE, then ACT19.17%
PROBE, then ABSTAIN16.00%
Reach the goal with more than one gain still supported24.25%

All three behaviours occur in the same experiment, 86% of the ACTs succeed, and most successes come before the gain is identified. For scale, a reference controller that is told the true gain and always acts reaches the goal on 44.25% of these tasks and misses on 55.75%. Without knowing the gain, the calibrated controller reaches 64% of that success rate and cuts failed ACTs from 55.75% to 4.58%.

The components interact as the theory describes. With no passive windows, success rises with the probe budget (21.83%, 26.17%, 28.25% for B=0,1,2B=0,1,2), and the controller probes less as passive windows accumulate (0.468, 0.266, 0.209, 0.123, 0.116 probes per task for N=0,1,2,4,8N=0,1,2,4,8 at B=2B=2). A controller that must identify the gain before acting never acts at N=0,B=1N=0,B=1, while SD-DRR succeeds on 26.17% of tasks there. Against information-gain probing, SD-DRR uses 0.15 to 0.41 fewer probes per task in all 20 cells with B>0B > 0 at comparable success (within 3.6 pp in every cell); in the cell above, 28.25% vs. 24.67% success with 0.468 vs. 0.851 probes. Success figures are for the three models trained on the full source budget.

W3. Organization of the experiments. Each experiment answers one question:

ExperimentQuestionReplication unit
Finite games (Sec. 5.1, 5.2)Do obstructions set the passive-data requirement and the best query allocation (T5, T6)?exact games, fresh trial blocks
TwoRoom hidden-gain study (new)Does the full ACT / PROBE / ABSTAIN loop work on a learned model, and how do passive evidence and online budget trade off (T8)?6 trained models, 400 tasks
TwoRoom response loss (Sec. 5.3)Does training on continuation responses help (T7)?21 matched training pairs (part 2/2)
PushT role interface (Sec. 5.4)Does a learned decision-role interface help planning?4 retrained ensembles, 2 task cohorts (part 2/2)
Reacher (Sec. 5.5)Does convex factor inference help a frozen agent?400 paired episodes

Q2 / W2. PushT with new seeds and new training data. We retrained from random initialization. Two source datasets were drawn independently from the parent corpus (2,335 episodes each, 292,059 and 291,954 frames, 293 episodes in common; the manuscript's reduced source has 291,969 frames). Each was crossed with two blocks of three seeds: 12 backbones, four three-model ensembles. The role mapper is fitted once per source draw and the factor and proposal modules once per ensemble, from that draw's frames only. Within a cell the full interface and the latent-only baseline share the ensemble, the 400 fresh tasks, task and planner seeds, and the planning budget (300 CEM candidates, 30 iterations, 50 steps).

Source drawSeed blockLatent-onlyFull interfacePaired gain (95% task interval)
0035.50%57.00%+21.50 pp [16.50, 26.50]
0133.50%49.75%+16.25 pp [11.25, 21.25]
1037.00%56.50%+19.50 pp [14.50, 24.50]
1129.00%54.50%+25.50 pp [20.50, 30.75]

The gain is positive in all four retrained ensembles: mean +20.69 pp [17.50, 23.88], SD across the four cells 3.87 pp, +18.875 and +22.500 pp by source draw, +20.500 and +20.875 pp by seed block. The manuscript's +15.75 pp on its original ensemble is a separate result.

Ablations on both seed blocks: removing factor scoring costs 8.125 pp [5.125, 11.250] and 10.75 pp [7.50, 14.00]. Removing the visual servo changes success by 0.125 pp [0.000, 0.375] and 0.00 pp and eliminates its extra RGB reads (about 10.4 and 11.1 per task).

We read the servo result on the first task cohort, so we generated a second cohort of 400 new tasks and ran all three controllers on the four ensembles. The full interface again beats latent-only in every ensemble: +10.50, +16.00, +15.75, +18.00 pp (mean +15.06 pp [11.88, 18.25]; 48.94% vs. 33.88%). The servo-free variant has the same success/failure outcome as the full interface on all 1,600 model-task runs (one-sided 95% bound on its success loss: 0.75 pp) and needs 11.72 fewer RGB reads per task. Over the two cohorts all eight ensemble-level gains are positive, from +10.50 to +25.50 pp.

Q3. Does the response loss help at reduced source? We enlarged the study so that the answer no longer rests on three pairs. We trained three new pretraining initializations and four fine-tuning seeds on each: 12 new control/response pairs. Within a pair the two models share initial weights, source frames, training and sampler seeds, augmentation, optimizer, the 4,000-update budget and the planner; they differ only in the response loss. All pairs are evaluated on 400 new tasks.

Evidence (400 new tasks)Pairs improvingMean gain95% task interval
Reduced source, 12 new pairs (3 initializations × 4 seeds)10 / 12+2.67 pp[1.60, 3.73]
Reduced source, the 3 original pairs3 / 3+6.33 pp[4.42, 8.42]
Full source, 3 new pairs3 / 3+3.42 pp[1.75, 5.08]
Full source, the 3 original pairs3 / 3+2.58 pp[0.92, 4.25]

In the 12-pair study the two remaining pairs are within 0.75 pp of zero, the three initialization means are +1.750, +1.125 and +5.125 pp (SD 2.15 pp between initializations, 2.37 pp between seeds within an initialization), and the pair-level SD is 2.82 pp. The original +5.33 pp rested on three pairs. The effect is now supported by 21 matched pairs, of which 19 improve, and all four intervals exclude zero. The loss adds no network passes (both arms use 24,000 batched predictor forwards per model). Task intervals are conditional on the trained models; variation across training runs is the pair-level spread above.

Thank you for the detailed response and the substantial additional experiments.

Regarding W1, the new TwoRoom experiment is useful and largely addresses my request for an integrated ACT/PROBE/ABSTAIN evaluation. However, comparing failed ACT rates with an oracle that always acts is not entirely fair, since SD-DRR can abstain. Reporting failure rates conditional on acting, alongside decision coverage, would make this comparison clearer.

Regarding W2, the additional training runs across different seeds and datasets provide convincing evidence that the improvements are not specific to the original models.

Regarding W3, the additional matched experiments substantially strengthen the evidence for the response loss, although I would encourage distinguishing uncertainty across tasks from variability across training runs.

Overall, I appreciate the additional experiments, which address some of my original feedback. I will maintain my original assessment, particularly given the remaining limitations and presentation concerns. I hope these additional results and clarifications will be incorporated into the paper.

Thank you for considering the additional experiments and for recognizing the integrated controller evaluation, the replication across seeds and datasets, and the strengthened evidence for the response loss. We have uploaded a revised manuscript that incorporates these experiments and presentation updates, and we provide the requested statistical breakdown below.

For W1, we report ACT coverage alongside failure conditional on ACT. The setting is the same as in our earlier response: three full-source TwoRoom models, N=0, B=2, 256 calibration tasks and 400 independent confirmation tasks shared across models. The table aggregates the resulting 1,200 model-task outcomes. Coverage is the fraction of tasks ending in ACT; conditional failure is the fraction of those ACTs that miss the goal.

ControllerACT coverage (%)Failures / ACTs (%)Success / all tasks (%)Probes / task
SD-DRR32.8313.9628.250.468
Entropy probing29.5016.3824.670.851
Identification first29.5814.9325.171.192
Certainty17.6738.6810.830.000
Gain-informed, always ACT100.0055.7544.250.000

The first four controllers share the T8 calibration constraint on P(ACT and failure), evaluated over all tasks, with a risk budget of 10%. The gain-informed controller provides a reference at 100% coverage. Among the calibrated controllers in this setting, SD-DRR combines the highest coverage and success rate with the lowest conditional failure rate. It also uses fewer probes than entropy probing, 0.468 versus 0.851 per task, with an observed unconditional failed-ACT rate of 4.58%. With 1,024 calibration tasks on the same three fixed models (N=0, B=2; Section 5.3 and Table 2), SD-DRR reaches 45.17% ACT coverage, 12.36% conditional failure and 39.58% success at 0.515 probes per task, compared with 41.67% coverage, 14.60% conditional failure and 35.58% success at 1.344 probes for entropy probing.

For W3, the reduced-source response-loss study comprises twelve matched training pairs: three independent pretraining initializations, with four fine-tuning seeds per initialization. The following quantities separate task uncertainty from observed training variability; all gains and standard deviations are in percentage points.

QuantityResult
Mean paired gain+2.67
95% paired task-bootstrap interval, 400 common tasks, fixed fitted models[1.60, 3.73]
Mean gain for each pretraining initialization, averaged over its four fine-tuning seeds+1.750, +1.125, +5.125
Observed SD across the twelve paired training gains2.82
SD across the three initialization means2.15
Pooled within-initialization fine-tuning SD2.37

The mean gain is positive for every pretraining initialization, and ten of the twelve matched pairs improve. The task interval quantifies uncertainty from sampling evaluation tasks with the trained models fixed; the reported SDs describe variation across the evaluated training runs and initializations.

In the uploaded 10-page revision, every subsection in Section 5 opens with its own question and protocol and closes with its own conclusion, keeping the exact-game, full-controller, training-objective, visual-planning and test-time inference studies self-contained. Section 5.3 presents the integrated ACT/PROBE/ABSTAIN controller, placing ACT coverage, conditional failure, success and probe use side by side in Table 1 and reporting the 1,024-task calibration confirmation in Table 2 (full protocols and cost accounting in Appendices E.1-E.2 and E.5). Section 5.4 presents the response-loss objective (Table 3) and its 21-pair replication (Table 4), reporting fixed-model 95% task intervals alongside run-to-run Pair SDs in Table 4 and the initialization-level breakdown in Appendix E.3 (Tables 20-21). Section 5.5 presents the PushT interface (Table 5) and its four-ensemble replication across independent source draws and task cohorts (Appendix E.4, Table 22).

Reviewer_WFovOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that offline evidence for a world model only needs to resolve decision conflicts that safe online probing cannot resolve within a budget. It formalizes this as SD-DRR and gives eight theorems: compression under a decision–nuisance factorization (T3), budget obstructions that set the passive-data requirement (T5), convex query allocation (T6), and continuation-response bounds for approximate and stochastic models (T7–T8).

Critical strengths
  • The core idea is clear and well illustrated: the data requirement depends on the online budget, and pairwise disagreement is not enough.
  • CE1/CE2 show concretely that merging terminal-equivalent worlds can break resolution.
Critical weaknesses
  • Theory is fragmented and mostly standard. T1, T2, T6, T7, and T8 are standard results (trivial quotient properties, LP duality, a simulation-lemma variant, confidence-interval LPs). T3 largely restates its own assumptions. T3–T6 (finite, deterministic, realizable) and T7–T8 (continuous, stochastic) share no assumptions or objects; T8 explicitly needs no DNF. The claimed chain C* → O_B → τ_B does not extend to the learned setting.
  • The learned-model experiments do not test the central claim. None of TwoRoom, PushT, or Reacher varies the online budget B or measures a passive-data requirement.
  • The assumptions do not match the world-model setting. Under DNF, the successor x′(x,q) is the same in every world, so uncertainty about dynamics is excluded.

The most major weakness is that theoretical analysis does not support the papar's main claim and also experiments does not investigate the claim. The logical correctness is lack.

Questions for authors
  • Which result is the central contribution, and how do T7, T8, and Props 1–2 support it?
  • How does the framework apply when the transition dynamics differ across worlds?
3 comments

Thank you for the detailed review. This part answers the theory questions (W1, W3, Q1, Q2). Part 2/3 reports the new experiment that varies the online budget on a learned model (W2), and part 3/3 gives the two new proofs.

Q1. Which result is central. The central result is T5 and the object it is stated on. For a true world aa, a budget obstruction is an inclusion-minimal world set O∋aO \ni a with C∗(x0,O)>BC^\ast(x_0,O) > B, and OB(a)\mathcal O_B(a) collects these sets. T5 shows that deployment exceeds budget BB exactly when some O∈OB(a)O \in \mathcal O_B(a) survives the offline data, with matching bounds (1−sB)N≤PN(B)≤MB(1−sB)N(1-s_B)^N \le P_N(B) \le M_B(1-s_B)^N and therefore the exact rate −log⁡(1−sB)-\log(1-s_B). What offline data must achieve is thus fixed by what safe interaction can still resolve from x0x_0 within BB. In the three-world construction of App. A, one unit of online budget changes the passive sample requirement by an unbounded factor (Nη(0)/Nη(1)→∞N_\eta(0)/N_\eta(1)\to\infty as one passive distinction becomes rare), with the hypothesis class, acceptable tokens and probes unchanged. T6 allocates queries against these obstructions.

W1. Standard tools, and what T3 adds. T1, T2, T6, T7 and T8 use standard machinery on purpose: quotients, LP duality, a simulation-type bound, confidence-interval programs. What they act on is specific to this problem. The matrix RR in T6 is the obstruction-hit matrix of the sequential game, so the LP allocates evidence against budget-dependent conflicts. Pairwise disagreement would give a different and insufficient design: in the three-world example of Sec. 3.2 every pair is resolvable and the triple is not. T3's hypotheses are one-step conditions on supports, and its conclusion is global: every reachable version set stays a product and the optimal worst-case cost of the whole sequential game is preserved. CE1 and CE2 show that equal terminal profiles do not give this.

Q1. How T7, T8 and Props. 1-2 relate to T5. A new corollary (part 3/3) places T7 and T5 on the same object. The finite resolution game is a robust Bellman problem with horizon ⌊B/cmin⁡⌋\lfloor B/c_{\min}\rfloor and terminal cost above BB on unresolved branches, so T7 applies to C∗C^\ast itself. If the learned continuation value has uniform error ee and planning adds at most η\eta, the executable bound U=C^+e+ηU=\widehat C+e+\eta satisfies

C∗>B  ⟹  U>B  ⟹  C∗>B−(2e+η).C^\ast > B \implies U > B \implies C^\ast > B-(2e+\eta).

A solver that authorizes only when U≤BU \le B therefore refuses with probability between the T5 bounds at budgets BB and B−(2e+η)B-(2e+\eta), up to the probability that the error certificate fails. Model error acts as a loss of 2e+η2e+\eta units of online budget, and T6 applied at the shifted budget gives a sufficient offline design. This extends the chain C∗→OB→τBC^\ast \to \mathcal O_B \to \tau_B to an approximate model.

T8 covers models for which no uniform error certificate exists. It needs no DNF by design: it bounds the observed cost and failed-ACT risk of complete controllers from held-out episodes, and part 2/3 applies it to a neural world model. Props. 1-2 belong to the implementation layer. Prop. 2 makes the set-supervised role readout a strongly convex problem with a unique minimizer. Prop. 1 shows that projecting the fused estimate onto the admissible factor set cannot increase its error to any admissible factor; with an LL-Lipschitz continuation readout, factor error ζ\zeta becomes response error LζL\zeta, which is the δt\delta_t term in T7.

Closest prior work. Javdani et al. (AISTATS 2014, https://proceedings.mlr.press/v33/javdani14.html) cut hyperedges between overlapping decision regions with deterministic tests that are always available. Here a probe moves the physical state and changes which probes remain safe, so two histories with the same surviving worlds can have different resolution costs, and the question is how much offline evidence the remaining budget leaves necessary.

W3 / Q2. Transitions that depend on the unknown world. The game is defined with world-indexed supports σw(x,q)\sigma_w(x,q) over successor/observation pairs. The shared successor is only the sufficient condition used in T3. We now prove T3 for

σ(a,b)(x,q)={(x′,(u,v)):(x′,u)∈Da(x,q),v∈Nb(x,q)},\sigma_{(a,b)}(x,q)=\lbrace (x^\prime,(u,v)) : (x^\prime,u)\in D_a(x,q), v\in N_b(x,q)\rbrace,

where the successor x′x^\prime may depend on the decision coordinate aa (for deterministic dynamics, x′=fa(x,q)x^\prime=f_a(x,q)). Product supports still update to products and the quotient preserves C∗C^\ast exactly, because the simulation keeps the observed (x′,u)(x^\prime,u) and replaces only the nuisance observation vv. T5 and T6 use the shared successor only through this cost equality, so they hold unchanged. Actuator gains, contact modes and other dynamics parameters can therefore be decision coordinates. Part 2/3 instantiates this with nine hidden actuator gains.

W2. Varying the online budget on a learned model. We ran this experiment on TwoRoom with hidden dynamics: one of nine diagonal actuator gains (gx,gy)∈{0.6,1.0,1.4}2(g_x,g_y)\in\lbrace 0.6,1.0,1.4\rbrace^2 scales every command, so the uncertainty is in the transitions. From x=106x=106 the same five-step horizontal probe ends at x=100x=100, 9696 or 9292 for gx=0.6,1.0,1.4g_x=0.6,1.0,1.4, and the vertical probe is admissible only when x<95x < 95 or x>129x > 129, so the gain changes both the successor state and which probes remain available. We cross N∈{0,1,2,4,8}N\in\lbrace 0,1,2,4,8\rbrace passive windows of the target world (five-step before/after RGB pairs from independent resets) with B∈{0,1,2}B\in\lbrace 0,1,2\rbrace online probes, for six trained world models (three per source budget). Four controller families share the learned model and planner: SD-DRR (continuation planning over the supported gains), entropy (probe to maximize information about the gain), identification (identify the gain, then act) and certainty (commit to a point estimate, no probing). In every cell each family is calibrated on 256 tasks to a failed-ACT bound of 0.1 over all tasks (T8), frozen, and run on 400 independent tasks. Success figures below are for the three models trained on the full source budget.

Success as the online budget grows. SD-DRR success (%) with no passive windows:

ModelB=0B=0B=1B=1B=2B=2
127.5030.5035.25
216.0020.5019.25
322.0027.5030.25
Mean21.8326.1728.25

Each model does better with probes than without: +3.00, +4.50, +5.50 pp at B=1B=1 and +7.75, +3.25, +8.25 pp at B=2B=2. Passive windows are the weaker substitute. At B=0B=0, mean success for N=0,1,2,4,8N=0,1,2,4,8 is 21.83, 21.67, 23.08, 24.08 and 23.25%: eight windows add at most 2.25 pp, whereas one probe adds 4.33 pp and puts the N=0N=0 controller above every passive-only cell.

The controller spends its budget only where it is needed. At B=2B=2 it uses 0.468, 0.266, 0.209, 0.123 and 0.116 probes per task for N=0,1,2,4,8N=0,1,2,4,8. Once passive windows have shrunk the support (1.26 gains remain on average at N=4N=4), it rarely probes. Passive evidence and online probes substitute for each other, which is the exchange the obstruction analysis describes.

Passive requirement as a function of BB. Our predeclared criterion is the smallest measured NN at which all three models reach 20% success under their calibrated risk bound. For SD-DRR the target is not attained with up to eight windows at B=0B=0 and is attained with zero windows at B=1B=1 (30.50%, 20.50%, 27.50%; N=0N=0 is selected in 62.13% of task-bootstrap resamples). At B=1B=1 entropy probing needs two windows and identification one, and the certainty baseline does not attain the target. At B=2B=2 and N=0N=0 two models exceed 30% and the third is three tasks short of the cutoff.

Acting without identifying the world. At N=0,B=1N=0,B=1 the identification-first controller never acts. SD-DRR succeeds on 26.17% of tasks (paired difference 26.17 pp, simultaneous 95% band [21.50, 30.83]), and every one of these successes occurs with more than one gain still supported (7.43 of 9 gains remain on average).

Fewer probes than information-gain probing. In all 20 cells with B>0B > 0 (both source budgets) SD-DRR uses 0.15 to 0.41 fewer probes per task than entropy probing, with every pointwise 95% interval excluding zero, at comparable success (within 3.6 pp in every cell). At N=0,B=2N=0,B=2: 28.25% vs. 24.67% success with 0.468 vs. 0.851 probes per task. Probing for the decision needs fewer interactions than probing for information, and the cost is model computation (17,645 vs. 3,534 rollouts per task).

For verification, here are the two new statements with proofs. Notation follows App. A.

Proposition (product supports with world-dependent successors). Let H=A0×B0\mathcal H=A_0\times B_0 be finite, with acceptability and safety depending only on aa. Let Y=Yd×YnY=Y_d\times Y_n and suppose that at every safe (x,q)(x,q)

σ(a,b)(x,q)={(x′,(u,v)):(x′,u)∈Da(x,q),v∈Nb(x,q)}\sigma_{(a,b)}(x,q)=\lbrace (x^\prime,(u,v)) : (x^\prime,u)\in D_a(x,q), v\in N_b(x,q)\rbrace

with nonempty Da(x,q)⊆X×YdD_a(x,q)\subseteq X\times Y_d and Nb(x,q)⊆YnN_b(x,q)\subseteq Y_n. Let the decision game have worlds A0A_0, the same states, probes, costs, acceptability and safety, and supports in which (x′,u)∈Da(x,q)(x^\prime,u)\in D_a(x,q) and vv ranges over Nˉ(x,q)=⋃bNb(x,q)\bar N(x,q)=\bigcup_b N_b(x,q). Then for every xx and every nonempty product A×B1A\times B_1,

Cfull∗(x,A×B1)=Cdec∗(x,A).C^\ast_{\mathrm{full}}(x,A\times B_1)=C^\ast_{\mathrm{dec}}(x,A).

Proof. After a branch (x′,(u,v))(x^\prime,(u,v)) the support A×B1A\times B_1 becomes A′×B1′A^\prime\times B_1^\prime with A′={a∈A:(x′,u)∈Da(x,q)}A^\prime=\lbrace a\in A:(x^\prime,u)\in D_a(x,q)\rbrace and B1′={b∈B1:v∈Nb(x,q)}B_1^\prime=\lbrace b\in B_1:v\in N_b(x,q)\rbrace, so products are invariant. In the decision game the same branch gives A′A^\prime, which does not depend on vv, and safety and the existence of a common acceptable policy agree at (x,A×B1)(x,A\times B_1) and (x,A)(x,A). The decision game is the closure quotient of the full game by the fibers {a}×B0\lbrace a\rbrace\times B_0, so T2 gives Cfull∗≤Cdec∗C^\ast_{\mathrm{full}}\le C^\ast_{\mathrm{dec}}. For the converse, run a full-game strategy inside the decision game while maintaining a shadow support A×B1A\times B_1. On a branch (x′,(u,v))(x^\prime,(u,v)) keep vv if some shadow bb has v∈Nb(x,q)v\in N_b(x,q); otherwise replace it by any v~∈Nb(x,q)\tilde v\in N_b(x,q) for a shadow bb. The branch (x′,(u,v~))(x^\prime,(u,\tilde v)) is possible in the full game because some a∈Aa\in A has (x′,u)∈Da(x,q)(x^\prime,u)\in D_a(x,q) and supports are products. In both cases the physical state is the observed x′x^\prime and the decision support is A′A^\prime, so admissible probes, stopping and costs coincide along the run. A full-game strategy that resolves within cost rr thus yields a decision-game strategy within cost rr. □\square

With deterministic dynamics x′=fa(x,q)x^\prime=f_a(x,q) this is the case Da(x,q)={fa(x,q)}×Ua(x,q)D_a(x,q)=\lbrace f_a(x,q)\rbrace\times U_a(x,q), and the manuscript's condition is the special case of a common ff. The proof of T5 uses the DNF structure only through this equality and version-set monotonicity (App. A, valid for every game), so T5 and T6 hold as stated.

Corollary (budget shift under a response certificate). Let cmin⁡>0c_{\min} > 0 be the smallest probe cost, H≥⌊B/cmin⁡⌋H\ge\lfloor B/c_{\min}\rfloor and M>BM > B. Consider the horizon-HH robust problem with terminal cost 00 after resolution and MM otherwise (an abort of cost MM is allowed), and value CHM(V)C_H^M(V). Since an unresolved branch costs at least MM and each probe at least cmin⁡c_{\min}, CHM(V)>bC_H^M(V) > b iff C∗(x0,V)>bC^\ast(x_0,V) > b for all 0≤b≤B0\le b\le B. Suppose that on an event of probability at least 1−δ1-\delta the assumptions of T7 hold uniformly over the version spaces considered, with total error E0≤eE_0\le e, and the returned policy has total optimization error at most η\eta. Put U=C^+e+ηU=\widehat C+e+\eta and κ=2e+η≤B\kappa=2e+\eta\le B, and authorize iff U(VN)≤BU(V_N)\le B. Then on that event

C∗(x0,VN)>B  ⟹  U(VN)>B  ⟹  C∗(x0,VN)>B−κ,C^\ast(x_0,V_N) > B \implies U(V_N) > B \implies C^\ast(x_0,V_N) > B-\kappa,

and under the passive-observation model of T5, with qa,Oq_{a,O} the per-sample probability of breaking OO,

max⁡O∈OB(a)(1−qa,O)N−δ≤Pr⁡(refuse)≤∑O∈OB−κ(a)(1−qa,O)N+δ.\max_{O\in\mathcal O_B(a)}(1-q_{a,O})^N-\delta \le \Pr(\text{refuse}) \le \sum_{O\in\mathcal O_{B-\kappa}(a)}(1-q_{a,O})^N+\delta.

Empty maxima and sums are zero. An authorization whose executed policy exceeds the budget or stays unresolved has probability at most δ\delta.

Proof. T7 gives ∣C^−CHM∣≤e\lvert\widehat C-C_H^M\rvert\le e, and its stagewise comparison bounds the true cost JπJ^\pi of the returned policy by C^+e+η\widehat C+e+\eta. Hence CHM≤Jπ≤U≤CHM+2e+ηC_H^M\le J^\pi\le U\le C_H^M+2e+\eta on the event. The two implications follow from the threshold identity, and the probability bounds from T5 applied at budgets BB and B−κB-\kappa, adding δ\delta for the complement of the event. If U≤BU\le B, the bound on JπJ^\pi excludes every unresolved branch because such a branch costs M>BM > B, so a wrong authorization can only occur outside the event. □\square

Reviewer_YJ2AOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework, SD-DRR, where an agent can save the data access for offline training and uses the observation during deployment to conduct the task safely. The paper formalizes theoretically how such a process can be carried out safely and show which details can be ignored, how observations should be allocated, and how prediction errors affect decision quality. The empirical experiments also demonstrated that the data requirement for a given job can be significantly reduced with such a method.

Critical strengths
  1. The authors point out that the need for data depends on what the agent can still find out safely during deployment. In particular, Theorem 5 concretizes the size of data needed offline into a bound with respect to the failure probability. Theorem 6 further characterizes how we can distribute data collection across the available observation types. The empirical results in Section 5.1 also demonstrate that the data efficiency can be improved up to 10x.

  2. The paper also supplies necessary trade-off analyses. T7 investigates the prediction accuracy and decision quality trade-off, whereas T8 characterizes the deployment risk. I appreciate that the authors looked into such details rigorously.

Critical weaknesses
  1. It is less clear how the theoretical guarantees can be obtained or even estimated in practice. The framework assumes a truth-containing world set, sound transition supports, safety predicates, and acceptable-policy sets.

  2. It seems that, in general, this framework saves the use of real observations/datapoints by conducting more computations beforehand. But it is less clear what the cost of such computation is and how it can be compared with the data collection process.

  3. This paper is not easily accessible for people coming from a general TCS background without much knowledge of world models.

Questions for authors
  1. Can the authors provide more insights on how the theoretical guarantees are useful when used in real world settings?

  2. see weakness 2, can the authors provide more insights on that?

2 comments

Thank you for the review. This part covers W1/Q1 (how the guarantees are obtained and used in practice) and W3 (a worked example for readers outside world models). Part 2/2 covers W2/Q2 (computation versus data collection).

W1 / Q1. How the assumed objects are obtained in practice. We added a learned-model deployment study on TwoRoom with hidden dynamics: one of nine actuator gains scales every command, and the agent must reach an image goal within 50 steps. The four objects you list are obtained as follows.

  • Truth-containing world set. Declared by the task designer: the nine gains (gx,gy)∈{0.6,1.0,1.4}2(g_x,g_y)\in\lbrace 0.6,1.0,1.4\rbrace^2. The framework needs a family that contains the truth, not a prior over it.
  • Sound transition supports. After each passive window or probe, a gain stays in the set if propagating the observed RGB position through the declared motion model (with the border clamp at every step) lands within one pixel of the observed outcome. On the 400 confirmation tasks this update never removed the true gain, in any of the 360 model/controller/budget cells.
  • Safety predicates. Declared probe geometry: five-step probes of fixed amplitude with stated admissible regions, enforced by the execution layer. Nothing about probe safety is learned.
  • Acceptable-policy sets. This is the learned part. Whether one plan reaches the goal under every gain still in the set is predicted by a neural world model and an RGB position readout, both trained only on source data. These predictions can be wrong, so the guarantee does not rest on them.

Guarantee for a learned model (T8). We calibrate the complete controller (support tracking, probe choice, planning, admission) on 256 held-out tasks. For every candidate policy we compute a one-sided Clopper-Pearson bound on the frequency of "ACT and miss the goal", counted over all tasks, and a Hoeffding bound on cost, valid jointly at level 0.05 over all 1,890 policy/model/budget combinations. We deploy the mixture of at most two policies (always-ABSTAIN is a candidate) with the lowest certified cost among those whose certified failed-ACT rate is at most 0.1. On 400 independent tasks the observed failed-ACT rate stays below 0.1 in all 360 cells (maximum 9.75%; 9.00% for SD-DRR). With the full source budget, no passive windows and two probes, SD-DRR acts on 32.83% of tasks, reaches the goal on 28.25% (86% of its ACTs) and misses on 4.58%. A reference controller that is told the true gain and always acts misses on 55.75%.

Use in practice. T5 and T6 say which offline observations to collect and how many, once the family and the probes are declared. T7 says how accurate continuation predictions must be relative to the decision margin. T8 is the statement one can certify at deployment for a neural model: a finite-sample bound on harmful ACTs, obtained from held-out episodes, with no assumption on model accuracy.

W3. A three-world example. Take worlds a0,a1,a2a_0,a_1,a_2. Policy π1\pi_1 is acceptable in a0a_0 and a1a_1; policy π2\pi_2 is acceptable in a1a_1 and a2a_2. From the start state x0x_0, one safe probe tells "a0a_0 or a1a_1" apart from "a2a_2". With one probe allowed (B=1B=1) the agent probes and commits to π1\pi_1 or π2\pi_2. It never learns whether the world is a0a_0 or a1a_1 and does not need to, and no offline data are required. With no probe allowed (B=0B=0) the pair {a0,a2}\lbrace a_0,a_2\rbrace has no common acceptable policy. It is a budget obstruction, and offline data must rule out one of the two. If each passive observation separates them with probability pp, deployment fails with probability (1−p)N(1-p)^N after NN observations. This is T5 in miniature: the online budget decides which distinctions the data must make, and T6 picks the observations that make them fastest. A probe can also move the agent to a state where the separating probe is no longer safe. The set of surviving worlds is then unchanged but the decision can no longer be resolved, which is why the physical state and the remaining budget enter the definition of an obstruction.

W2 / Q2. Computation versus data collection. We now account for each stage separately: source data, training compute, calibration episodes, online interaction, and model computation at deployment. For the learned models the extra computation is not in training. It appears at deployment (planning over the supported worlds) and in calibration.

Deployment. TwoRoom hidden-gain study, full source, no passive windows, two probes; per-task means over three models and 400 tasks:

ControllerSuccessProbesPhysical stepsModel rolloutsModel time (s)
SD-DRR28.25%0.46810.2517,64512.91
Entropy probing24.67%0.85111.093,5342.79
Identify, then act25.17%1.19212.763,8693.06
Point estimate, no probing10.83%0.0006.041560.16

SD-DRR has the highest success and the fewest probes among the three probing controllers. Relative to entropy probing it saves 0.383 probes per task (95% task interval [0.322, 0.446]; a probe is five physical steps plus one camera observation). The price is about 14,100 additional model rollouts per task, roughly 10 s of planning on a shared A800 GPU. Over the whole grid SD-DRR uses fewer probes than entropy probing in all 20 cells with a positive probe budget (0.15 to 0.41 fewer per task) at comparable success (within 3.6 pp in every cell). The method trades interaction for model computation at an explicit rate, which is the favourable direction when interaction is slow, costly or risky. Model time is the accounted planning and encoding time of each policy on shared hardware, not an isolated latency benchmark.

Training. The continuation-response loss needs no extra data and no extra network passes: both arms use 4,000 updates and 24,000 batched predictor forwards per model and reuse the same rollout predictions. Measured fine-tuning time without and with the loss is 2,070 s and 2,208 s (reduced source, 12 pairs) and 1,407 s and 1,366 s (full source, 3 pairs). Source budgets are 148,678 frames (reduced) and 2,397,936 frames (full), including pretraining.

Calibration. The T8 certificate requires held-out episodes, and we count them as data collection. Calibrating every candidate policy in every budget cell for six models on 256 tasks executed 757,971 physical steps and 101,912 probe observations. The 400-task confirmation executed a further 364,108 steps and 22,584 probe observations. These totals cover the whole comparison; calibrating a single deployed configuration uses a fraction of them.

PushT. Four retrained ensembles, 400 tasks; per-task means:

ResourceLatent-onlyFull interface
Success33.75%54.44%
Physical control steps41.8436.53
Planning calls4.703.74
CEM candidate sequences42,29433,677
Extra RGB reads outside planning steps010.76
One-step factor predictionsnot used106,476

The interface raises success by 20.69 pp with fewer control steps and fewer planning calls, because episodes end earlier. Fitting the four factor/proposal interfaces takes 759 s in total, against 225,971 s for the 12 backbones that both controllers share. The extra RGB reads come from the visual servo and can be dropped: on a second cohort of 400 new tasks the servo-free interface has the same outcome as the full interface on all 1,600 runs and keeps a +15.06 pp gain over latent-only [11.88, 18.25].

Comments on the paper

3 comments

We have uploaded the revised manuscript (10-page main text) and updated supplementary archive incorporating the experiments, proofs, and presentation updates discussed with the reviewers:

  1. World-dependent transitions and approximate resolution margin (WFov, YJ2A): Section 3.1 and Proposition A.15 (Appendix A.15) extend T3 to product supports with decision-dependent physical successors; Section 3.4 and Corollary A.8 (Appendix A.16) connect T7 continuation error directly to T5 budget-obstruction bounds via an explicit 2e+η2e+\eta budget shift.
  2. Integrated learned ACT/PROBE/ABSTAIN controller and calibration study (WFov, V3BA, YJ2A): Section 5.3 presents the hidden-gain TwoRoom controller study across passive windows and online probe budgets, with ACT coverage, conditional failure, success, and probe use reported side by side in Table 1 and the 1,024-task calibration confirmation in Table 2 (full protocols and model-level results in Appendices E.1-E.2).
  3. Response-loss training replication and uncertainty separation (V3BA): Section 5.4 and Table 4 report the 21 matched training pairs alongside the original TwoRoom benchmark (Table 3), placing fixed-model 95% paired task intervals alongside run-to-run Pair SDs in Table 4 and giving the initialization-level breakdown in Appendix E.3 (Tables 20-21).
  4. PushT cross-seed and cross-dataset replication (V3BA): Section 5.5 reports the four retrained three-model ensembles across two independent source draws and two independent 400-task cohorts, including component contributions and the servo-free confirmation (full results in Appendix E.4, Table 22).
  5. Self-contained experimental presentation and cost accounting (V3BA, YJ2A): Every subsection in Section 5 (Sections 5.1-5.6) opens with its own question and protocol and closes with its own conclusion; Section 5.3, Table 10 (Appendix D), and Appendix E.5 (Tables 23-25) report source frames, training compute, calibration episodes, online probes, and deployment rollouts separately across all studies.

We have uploaded the revised manuscript (10-page main text) and updated supplementary archive incorporating the experiments, proofs, and presentation updates discussed with the reviewers:

  1. World-dependent transitions and approximate resolution margin (WFov, YJ2A): Section 3.1 and Proposition A.15 (Appendix A.15) extend T3 to product supports with decision-dependent physical successors; Section 3.4 and Corollary A.8 (Appendix A.16) connect T7 continuation error directly to T5 budget-obstruction bounds via an explicit 2e+η2e+\eta budget shift.
  2. Integrated learned ACT/PROBE/ABSTAIN controller and calibration study (WFov, V3BA, YJ2A): Section 5.3 presents the hidden-gain TwoRoom controller study across passive windows and online probe budgets, with ACT coverage, conditional failure, success, and probe use reported side by side in Table 1 and the 1,024-task calibration confirmation in Table 2 (full protocols and model-level results in Appendices E.1-E.2).
  3. Response-loss training replication and uncertainty separation (V3BA): Section 5.4 and Table 4 report the 21 matched training pairs alongside the original TwoRoom benchmark (Table 3), placing fixed-model 95% paired task intervals alongside run-to-run Pair SDs in Table 4 and giving the initialization-level breakdown in Appendix E.3 (Tables 20-21).
  4. PushT cross-seed and cross-dataset replication (V3BA): Section 5.5 reports the four retrained three-model ensembles across two independent source draws and two independent 400-task cohorts, including component contributions and the servo-free confirmation (full results in Appendix E.4, Table 22).
  5. Self-contained experimental presentation and cost accounting (V3BA, YJ2A): Every subsection in Section 5 (Sections 5.1-5.6) opens with its own question and protocol and closes with its own conclusion; Section 5.3, Table 10 (Appendix D), and Appendix E.5 (Tables 23-25) report source frames, training compute, calibration episodes, online probes, and deployment rollouts separately across all studies.

We thank the Area Chair and reviewers (YJ2A, V3BA, WFov). As discussion closes, we summarize how our proofs, confirmation studies, and uploaded 10-page revision resolve each review, including V3BA's Nov 17 follow-up.

  1. Theory and world-dependent dynamics (WFov W1/W3/Q1/Q2, YJ2A W3). T5 is central: offline data requirements are set by minimal budget obstructions OB(a)\mathcal O_B(a) under safe probe budget BB. Prop. A.15 (Sec. 3.1, App. A.15) extends T3 to product supports with decision-dependent physical successors, carrying T3, T5, and T6 to unknown dynamics (actuator gains or contact modes). Cor. A.8 (Sec. 3.4, App. A.16) places T7 on OB(a)\mathcal O_B(a): continuation error ee and planning tolerance η\eta bound an approximate solver's refusal rate between T5 bounds at BB and B−(2e+η)B-(2e+\eta). Without a uniform error certificate, T8 certifies complete neural controllers from held-out episodes without DNF.

  2. Integrated ACT/PROBE/ABSTAIN loop and coverage breakdown (WFov W2, V3BA W1/Q1 and Nov 17 follow-up, YJ2A W1/Q1). On TwoRoom with nine hidden actuator gains altering transitions and future safe probes, six trained models are evaluated across N∈{0,1,2,4,8}N\in\lbrace 0,1,2,4,8\rbrace passive windows and B∈{0,1,2}B\in\lbrace 0,1,2\rbrace probes (Sec. 5.3, Tables 1-2, Apps. E.1-E.2), calibrated via T8 to a 10% unconditional failed-ACT bound (met in all 360 confirmation cells; max 9.75%) on 400 independent tasks. V3BA noted this "largely addresses [the] request for an integrated ACT/PROBE/ABSTAIN evaluation" and asked to pair coverage with conditional failure (Table 1, full source, N=0,B=2N=0, B=2, 256 calibration tasks, 1,200 runs):

ControllerACT cov. (%)Failures/ACT (%)Success (%)Probes
SD-DRR32.8313.9628.250.468
Entropy29.5016.3824.670.851
Identify first29.5814.9325.171.192
Certainty17.6738.6810.830.000
Gain-informed100.0055.7544.250.000

Among T8-calibrated controllers (rows 1-4), SD-DRR combines the highest coverage and success with the lowest conditional failure and fewer probes than entropy (unconditional failed ACT: 4.58%; row 5 is a 100%-coverage always-ACT reference). In all 20 cells with B>0B > 0, SD-DRR uses 0.15-0.41 fewer probes/task than entropy, and at N=0,B=1N=0, B=1 reaches 26.17% success (multiple gains supported) where identification-first never acts.

  1. 1,024-task calibration confirmation without retraining (WFov W2, V3BA Q1, YJ2A W1/W2). On 400 new confirmation tasks with the same fixed models at N=0N=0 (Sec. 5.3, Table 2, App. E.2), all three full-source models certify at least 20% success at 1,024 calibration tasks and B=1B=1 (lowest bound 22.34%). Success for B=0,1,2B=0,1,2 reaches 29.83%, 37.58%, 39.58% at full source (+13.25 pp from 256 to 1,024 calibration tasks at B=1B=1 [10.83, 15.75]; +5.33 pp over entropy, band [0.67, 10.00], 0.246 vs. 0.601 probes; at B=2B=2, 45.17% coverage and 12.36% conditional failure vs. entropy's 41.67% and 14.60%) and 21.58%, 28.08%, 29.08% at reduced source (B=2B=2 vs. B=0B=0 gain +7.50 pp [5.25, 9.83], band [4.67, 10.33]; +5.08 pp over entropy at B=2B=2, band [1.00, 9.17], 0.366 vs. 1.085 probes).

  2. Model replication and statistical separation (V3BA W2/W3/Q2/Q3 and Nov 17 follow-up). On PushT (Sec. 5.5, App. E.4, Table 22), 12 retrained backbones (four ensembles, two source draws ×\times two seed blocks) beat latent-only planning in all four ensembles on 400 fresh tasks (+20.69 pp [17.50, 23.88], SD 3.87) and on 400 new confirmation tasks (+15.06 pp [11.88, 18.25], SD 3.20), matching on all 1,600 runs without visual servo (-11.72 RGB reads/task); V3BA noted "convincing evidence that the improvements are not specific to the original models." On TwoRoom response loss (Sec. 5.4, Table 4, App. E.3, Tables 20-21), 19/21 matched pairs improve with zero extra forward passes, "substantially strengthen[ing] the evidence" (V3BA). Table 4 pairs fixed-model 95% task intervals with run-to-run Pair SDs: 12 new reduced-source pairs (3 initializations ×\times 4 seeds) gain +2.67 pp (task interval [1.60, 3.73], pair SD 2.82, positive mean in every initialization at +1.750, +1.125, +5.125 pp, init SD 2.15, within-init SD 2.37), and new full-source pairs gain +3.42 pp ([1.75, 5.08], pair SD 1.26).

  3. Cost accounting and presentation (YJ2A W1/W2/W3, V3BA presentation). Source frames, compute, calibration, probes, and rollouts are accounted separately (Sec. 5.3, Table 10, App. E.5 Tables 23-25; 17,645 vs. 3,534 rollouts/task for 0.468 vs. 0.851 probes at N=0,B=2N=0, B=2). Every Section 5 subsection (5.1-5.6) opens with its own question/protocol and closes with its own conclusion while preserving all original baselines, ablations, and proofs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.