acceptodds
Under review as a conference paper at ICLR 2027

From Partial Identification to Directed Exploration in World Models

Abstract

World models support planning by predicting action consequences, but data collected by a controller may lack sufficient action variation, leaving different predictions for untried actions. We study world models with linear Gaussian latent dynamics and recoverable states. We fully characterize which action effects existing data determine and which remain ambiguous, and quantify how prediction accuracy on observed trajectories constrains predictions under changed actions. This characterization guides the identification of action directions requiring additional exploration. We determine the minimum number of directions needed to resolve the ambiguity and optimize probing strength across these directions to improve the accuracy of action effect estimates. The resulting method combines existing trajectories and new interventions to estimate action effects. We further analyze how errors in learning state representations and selecting probe directions affect the final estimation accuracy. With matched interaction counts and action perturbation budgets, directed acquisition reduces control cost by about 53% against balanced full-space exploration under missing and weak coverage. Robot manipulation and pixel experiments further show improvements in control success and action response prediction, respectively.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, partial identification, system identification, optimal exploration, linear dynamical systems

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HJ9aOpenReview ↗
Rating
2(−1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the setting in which data from an old policy predict observed transitions accurately but do not identify the consequences of new actions. In recoverable-state linear Gaussian models, it characterizes a compatible dynamics set, uses conditional action residuals to locate missing directions, and allocates an exploration budget to targeted interventions. The motivation is practical, and the theory and controlled experiments support sampling to close specific information gaps. The task relevance of the selected directions and the incremental benefit in more complex settings, however, require stronger evidence.

Critical strengths
  1. A coherent identification-to-exploration chain. Theorems 1, 2, 4, and 6 connect state recovery, compatible dynamics sets, counterfactual prediction bounds, and minimum-intervention directions in a way that is clear and checkable within the declared model class.

  2. Careful separation of design objectives. Appendix E distinguishes random Gaussian designs, fixed-Gram designs, and expected-information allocation objectives. Propositions 34-35 also analyze estimation error and representation transfer under interventions, avoiding an overstatement that these distinct notions of optimality coincide.

  3. Relatively transparent empirical evidence. Appendices G, I, and J include repeated systems, alternative fitters and controllers, assumption-deviation tests, and cases without an improvement. This helps make both the benefits and the failure boundary of directed sampling visible.

Critical weaknesses
  1. The allocation criterion is not yet sufficiently task-aware. Section 6 mainly allocates budget by action-effect estimation error, while Proposition 40 explicitly shows that task weights alter the optimal design. The paper does not establish when a low-coverage direction is a control bottleneck rather than an irrelevant but difficult-to-identify direction.

  2. The principal fitting comparison may underuse old data. In Appendix G.1, PairedTwoStage estimates the probed directions from new data and then fixes their coefficients, so FullBalanced cannot fully reuse old-data constraints on those coefficients. Although another fitter shows gains, roughly 53 percent of the primary improvement magnitude should be rechecked with information-weighted fusion of old and new data.

  3. The complex-task gains over simpler marginal diagnostics are not yet stable. Directed and GlobalMean differ by only 0.14 percentage points in Table 1, and GlobalGap attains the best small-budget mean return in the pixel experiment. The evidence supports the feasibility of directed sampling, but not a consistent additional decision benefit from the conditional-residual diagnostic.

  4. The representation-learning extension lacks an empirical bridge. Section 4 and Theorem 22 connect representation error to identification, but Appendix G mainly reports final selection and control outcomes. An analysis varying spectral gaps or alignment errors would clarify the practical value of this extension.

Questions for authors
  1. If there are action directions that are poorly covered but irrelevant to the current task, does the method still prioritize them, and can a task-weighted design avoid ineffective interventions?

  2. If all methods use a shared estimator that preserves passive information matrices and combines all old and new data, how do the main control gains and the advantage of NoNewData under full coverage change?

  3. In real tasks where marginal actions are rich but conditional actions are inadequate, can the conditional-residual diagnostic consistently outperform GlobalMean, GlobalGap, and uncertainty sampling?

2 comments
AuthorsOpenReview ↗

Task relevance, passive-information reuse, and the representation bridge

W1/Q1. Can weak but irrelevant directions receive probes, and how can this be avoided?

Yes. Unweighted identification can spend probes on directions irrelevant to the current task. For a fixed goal, Proposition 40 uses tr(H_task J^(-1)) to prioritize uncertain effects that matter to that goal; unweighted identification supports reuse across unknown future goals.

We added the diagonal rule j_j=(√(h_j/(ηw_j))−λ_j)_+, with η set by the budget. In a common diagonal basis, λ_j is passive information, h_j>0 task importance and w_j>0 action cost. A direction receives zero new information exactly when λ_j≥√(h_j/(ηw_j)); otherwise its task weight determines the amount added. This combines Proposition 40's task weighting with Eq. 18's reuse of passive information. TaskFill estimates task relevance using frozen-model Jacobians; the revision distinguishes this proxy-weight implementation from the fixed-weight design result.

W2/Q2. What changes when all methods reuse old data through a shared estimator?

With every old and new row retained at equal weight, the missing/weak-coverage control comparison changes to favor FullBalanced, and NoNewData's isotropic full-coverage advantage disappears. Following your suggestion, we compared all methods using shared estimators at the original budget of 24. The original approximately 53% reduction belongs to PairedTwoStage. Mean paired control-cost reductions of ConditionalGap versus FullBalanced are:

CoveragePairedTwoStageEqualMassJointOriginalJoint
Missing+53.34%+40.61%−50.38%
Weak full+53.33%+40.65%−31.57%
Anisotropic+57.19%+44.47%+0.77%
Isotropic0%0%0%

Positive values favor ConditionalGap. EqualMassJoint assigns each dataset equal total weight; OriginalJoint weights every row equally. Under isotropic full coverage, NoNewData's advantage over FullBalanced changes from +55.29% to −0.87%, slightly favoring FullBalanced with joint fitting.

This comparison also separates estimation from control. Under OriginalJoint, ConditionalGap is closer to the aligned physical action-matrix reference in all 32 cases of each missing/weak regime, with mean relative error reductions of 19.16%/17.45%. Action-effect estimation improves, but the original control reduction does not carry over. The revision specifies both acquisition and final estimation; task-weighted design further accounts for which estimation errors affect the current goal.

W3/Q3. Does conditional screening consistently improve realistic-task decisions over marginal and uncertainty baselines?

The control ranking varies across tasks and budgets; the added natural-data study quantifies gains in action-response prediction. In the added natural-data comparison with joint fitting, Eq. 18 has lower mean action-response error in 8/8 Square/Transport groups versus FullBalanced and GlobalGap, 7/8 versus GlobalEq18, and 3/8 versus uncertainty sampling. Reductions are 3.98–15.70% versus FullBalanced and −.34% to +2.26% versus GlobalEq18 (Table 21).

In PickCube, GlobalMean already identifies the useful near-vertical direction, so conditioning adds little directional information. Directed/GlobalMean success is 69.66%/69.52%; their distance difference is .038 mm, with a 98.75% interval [−.067,.169]. EnsembleDisagreement gives 60.91%. The original feedback-masking result explains when conditioning matters: rich marginal actions can still leave the conditional coverage span unidentified. When a marginal method finds the same useful direction, similar performance is expected.

In pixels, Directed lowers response error against FullBalanced by 12.04%/10.89% at 10/30 episodes. Ensemble has lower response error and GlobalGap the highest return at 10; Directed has lower response error at 30. The new natural-demonstration control checks did not improve success over BC, so their gains are reported as action-response prediction. These comparisons identify the measured benefits while keeping the control endpoint specific to its task and budget.

W4. How is representation error connected to the observed results?

Representation error affects direction recovery through two margins: Theorem 11 bounds encoder MSE by 2ε/γ_rep; Theorem 22 propagates encoder and sampling errors relative to the residual spectral gap. Smaller margins make the diagnostic more sensitive.

Table 5 crosses exact/learned coordinates with true/estimated spans, giving MSE .191523–.193246. The added measurements across the 128 frozen cases give γ_rep=.03943–.09346 and aligned encoder MSE=.15413–.57078. These ranges describe error scales where systems and encoders vary together; Table 5 provides the crossed comparison. The revision connects both measurements to the bounds.

Thank you for the response and clarifications. However, my concerns have not been fully resolved. I therefore maintain my current rating.

Reviewer_2ZywOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how insufficient action variation limits the prediction from world models. For linear Gaussian latent dynamics, it characterizes which action effects are identifiable and which remain ambiguous. Based on this analysis, it proposes an exploration method that removes the state-dependent component of actions in the dataset and probes directions with remaining variation. Experiments evaluate the method on controlled systems, robot manipulation, and a pixel-based control task.

Critical strengths
  • The paper's distinction between raw action variation and action variation conditional on state is a real challenge in learning the world model.

  • The theoretical analysis connects unidentified action effects to exploration directions and provides estimation guarantees. The controlled experiments provide empirical evidence of the proposed method on standard offline RL benchmarks.

Critical weaknesses
  • The main theory relies on linear Gaussian dynamics and strong assumptions for recovering the underlying state. It is unclear how well these results apply to practical world models that learn nonlinear dynamics.

  • Sec. 4 and Sec. 5 contain a lot of theoretical results, but it is unclear how the practical algorithm works. Providing an overview of the practical algorithm, e.g., pseudocode, would clarify the method.

  • The main exploration experiments remove variation in certain action directions on purpose. For example, the PickCube data omit vertical exploration, and evaluation begins after the object has been grasped. These experiments illustrate the idea, but its connection with realistic datasets remains unclear. In practice, the data-collection policy might not remove those variations in advance.

Questions for authors
  • When should the proposed method work reliably with nonlinear dynamics or action coverage that changes across states?

  • How does the practical algorithm work?

  • Can the authors show exploration gains on datasets where missing action directions were not introduced on purpose?

2 comments
AuthorsOpenReview ↗

The practical procedure, nonlinear reliability, and natural coverage

W2/Q2. How does the practical algorithm work?

The practical algorithm finds action variation missing after conditioning on state, collects probes in those directions, and combines the observations with old data to estimate action effects. The original Figure 2 and Section 6 give this workflow; the revision collects its implementation into four steps:

  1. Learn and freeze centered, whitened features, and fit the behavior action mean.
  2. Estimate residual action covariance on a separate split; use spectral separation and split stability to identify weak directions.
  3. Collect antipodal probes around behavior actions, matching added transitions and perturbation energy. Use full-space exploration when a weak span cannot be identified stably.
  4. Estimate action effects using old and new data, then update the model.

The fitting choices are explicit: Eq. 17 analyzes regression with known nuisance effects; the original PairedTwoStage fixes paired-difference estimates before fitting the remaining effects; joint fitting retains all old and new rows' constraints on the coefficients. This connects the implementation, theoretical quantities and estimator used in each comparison.

W1/Q1(a). When is the nonlinear extension reliable?

Given a suitable state representation, nonlinear identification requires a correctly specified feature or local response model and sufficient action information. Theorem 41 covers controlled means linear in known nonlinear features with spanning information Grams. Theorem 42 covers a local tangent model with a positive Gram and bounded quadratic remainder. The practical checks therefore concern local model error, residual spectral separation and stability across data splits.

The representation theorem uses stationary linear Gaussian dynamics, sufficient observations, affine realizability, whitening and a positive representation margin. Its perturbation result shows how encoder and sampling errors affect the recovered action span. In the assumption-deviation tests, Directed lowers mean control cost against FullBalanced in all 22 tested conditions at 96 added transitions; rankings vary at smaller budgets. These tests evaluate specified departures from the model alongside the extension's conditions.

Q1(b). What if coverage changes across states?

When action responses must be estimated separately across regions, coverage should be diagnosed within each region. Reliability requires an accurate local model, sufficient samples and a stable weak span. Horizontal variation in one region and vertical variation in another can cover both directions in aggregate without identifying either regional effect fully.

For G₁=diag(1,0) and G₂=diag(0,1), a shared action-effect matrix is identifiable from their full-rank sum. Distinct regional matrices instead require diag(G₁,G₂), which is singular. This separates varying coverage from effects that require local identification. The regional implementation fits local behavior means and residual covariances, then probes reachable regions with stable directions and sufficient samples; otherwise it retains broad exploration. The constructed rotating-gap experiment demonstrates a response-prediction improvement from this localization.

W3/Q3. What evidence comes from gaps that were not introduced deliberately?

Yes. On Square and Transport human demonstrations without deliberately removed action directions, the continuous allocation in Eq. 18 lowers action-response prediction error by 3.98–15.70% versus uniform probing, with equal added budgets and joint fitting (Appendix K, Table 21).

The original experiments deliberately vary action coverage to test the identification mechanism. Following your suggestion, the added experiments use human demonstrations directly and show that targeted action variation also improves response prediction on naturally collected behavior.

I thank the authors for the responses. For the practical algorithm, I would expect a formal pseudocode block to clarify how the algorithm works in practice and the input and output of the algorithm. For the nonlinear dynamics and exploration experiments, I also expect more empirical evidence to ablate each component individually. Given that my main concerns remain unresolved, I would like to keep my score.

Reviewer_SxM4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies directed exploration for world models. The authors characterize identifiable dynamics in linear Gaussian systems and propose an exploration method that targets poorly excited action directions using conditional action covariance. Theoretical guarantees and experiments on synthetic systems, robot manipulation, and pixel-based control evaluate improvements in model estimation and control.

Critical strengths

The paper provides a detailed theoretical analysis of partial identification and action coverage, supported by experiments across several simulation environments.

Critical weaknesses
  1. Restrictive theoretical assumptions: The theoretical analysis relies on linear Gaussian latent dynamics, recoverable state representations, and strong spectral separation assumptions. These conditions are far removed from practical world models, substantially limiting the applicability of the theoretical guarantees.

  2. Insufficient experimental evaluation: The experiments primarily focus on synthetic systems with artificially constructed action coverage gaps. The evaluation on more realistic environments is limited and does not convincingly demonstrate the effectiveness of the method in general world-model learning settings.

  3. Limited empirical improvements: The improvements in realistic environments are relatively modest and inconsistent. For instance, in PickCube, the proposed method achieves 69.66% success compared with 69.52% for a simple global covariance baseline. In pixel-based control, it does not consistently outperform alternative exploration strategies.

Questions for authors
  1. More challenging evaluation: Can the authors demonstrate the effectiveness of the proposed method in more challenging environments, such as high-dimensional robotic manipulation tasks with complex nonlinear dynamics and diverse behavior policies, where exploration gaps are not artificially constructed?

  2. State-dependent exploration: How does the method handle settings where poorly explored action directions vary across the state space? Since the proposed approach relies on global conditional action covariance, could it overlook important state-dependent coverage gaps? How could the method be extended to address this limitation?

  3. Lack of task awareness: The exploration strategy prioritizes poorly excited action directions without considering their relevance to downstream planning or control. Could authors discuss more about why they prefer a task-agnostic exploration strategy?

2 comments
AuthorsOpenReview ↗

Natural-data evaluation, local gaps, and task-aware exploration

W2/Q1. What does evaluation on harder, unmodified manipulation data show?

Yes. On Square and Transport human-manipulation data without deliberately removed action directions, the continuous allocation in Eq. 18 lowers action-response prediction error by 3.98–15.70% versus uniform probing, with equal added budgets and joint fitting (Appendix K, Table 21).

The added evaluation covers proficient and mixed human behavior and tests whether the world model predicts the consequences of changing actions more accurately. It provides response-learning evidence on complex manipulation tasks with diverse behavior and natural coverage, complementing the original experiments with imposed gaps.

W1. When do the theoretical guarantees apply to nonlinear world models?

Given a suitable representation, the nonlinear action-effect guarantees cover two model classes: controlled means linear in known nonlinear features with spanning information Grams (Theorem 41), and local tangent models with a positive Gram and bounded quadratic remainder (Theorem 42). Stable direction recovery requires residual separation large enough relative to encoder and sampling errors.

Representation recovery has separate linear-Gaussian, sufficiency, affine-realizability, whitening and margin assumptions. The original experiments test nonlinear dynamics/behavior, heavy-tailed noise, changing excitation and unpaired acquisition: at 96 added transitions, Directed lowers mean control cost against FullBalanced in all 22 conditions. The revision maps each assumption and comparison to the result it supports.

W3. Why is the advantage over simpler baselines small or inconsistent?

In PickCube, GlobalMean already finds almost the same useful vertical direction, so conditioning supplies little additional directional information. Directed/GlobalMean success is 69.66%/69.52%; the distance difference is .038 mm, with a 98.75% interval [−.067,.169]. The original robot experiment notes GlobalMean's near-vertical choice.

Conditioning addresses action variation masked by state feedback. Section 6 shows that full-rank marginal covariance can still fail to determine the conditional coverage span. This explains its identification value under feedback masking, and why the incremental benefit can be small when a simpler baseline already selects the useful direction.

The pixel comparison depends on budget and endpoint: Directed lowers response error against FullBalanced by 12.04%/10.89% at 10/30 episodes, whereas GlobalGap has the highest mean return at 10. The revision reports response learning and control return separately.

Q2. Can global covariance miss state-dependent gaps, and how should the method handle them?

Yes, when action effects differ across regions. Coverage should then be diagnosed and probed locally. Varying coverage alone need not cause a problem: for G₁=diag(1,0) and G₂=diag(0,1), their full-rank sum identifies a shared action-effect matrix. Distinct regional matrices instead require diag(G₁,G₂), which is singular.

The regional implementation fits local behavior means and residual covariances, checks stability across splits, and probes reachable regions with sufficient samples; otherwise it uses broad exploration. The constructed rotating-gap experiment demonstrates response-prediction gains from localization. Task weights below account for each effect's relevance to the control goal.

Q3. Why prefer task-agnostic exploration, and can task relevance be included?

Task-agnostic identification serves models reused across unknown future goals. For a specified goal, Proposition 40 supplies the task-weighted objective tr(H_task J^(-1)). The added diagonal allocation combines task importance, passive information and action costs, assigning probes where task-weighted uncertainty warrants them.

TaskFill estimates relevance from frozen-model Jacobians. In the separate Pusher development comparison, it lowers mean within-case control cost by 2.57% versus FullBalanced: +5.16%/−.02% in expert/medium cohorts. InformationFill is 2.13% worse. These comparisons illustrate how incorporating the current goal changes allocation and control performance.

I thank the authors for their response. However, my main concerns remain unresolved. I encourage the authors to evaluate their approach on more challenging tasks and explore how local covariance information could be incorporated into the framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.