acceptodds
Withdrawn from ICLR 2027

What Should I Try Next? Decision-Relevant Counterfactual Information Gain for Active World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models enable agents to evaluate actions through imagined futures, yet these predictions are useful only when the underlying environment dynamics are sufficiently identified. Existing exploration methods typically seek novel states, large prediction errors, local model disagreement, or generic information gain. Such objectives may spend costly interactions resolving uncertainty that has little effect on subsequent decisions. We introduce Active Epistemic World Modeling (AEWM), a framework that asks which real interaction should be performed now to make relevant counterfactual futures more reliable. AEWM distinguishes irre- ducible outcome noise from reducible uncertainty about latent mechanisms and targets decision-relevant counterfactual information gain. Its practical Counterfac- tual Disagreement Reduction (CDR) score estimates query-set variance reduction by sampling hypothetical outcomes and reweighting ensemble members without inner-loop retraining. In a matched five-seed study on 60 held-out zero-delay hidden-physics worlds, CDR lowers one-step prediction RMSE at 32 interactions by 19.7% relative to local disagreement. It modestly improves budget-curve area relative to random selection, but matches random at the endpoint and uses roughly 3.2× more scoring time. Historical model ablations expose a delayed-action training-target mismatch, and an all-candidate audit finds weak fidelity of the cheap CDR rankings. The completed study establishes a one-step acquisition result; planner-generated queries and downstream control remain to be tested.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, active exploration, epistemic uncertainty, counterfactual prediction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aD3uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Active Epistemic World Modeling (AEWM), which selects informative interactions to improve world model predictions. And its CDR module estimates counterfactual uncertainty reduction through ensemble reweighting.

Critical strengths
  1. Interesting research motivation: The paper tries to address a vital problem, selecting informative interactions to improve world model predictions rather than simply maximizing the local uncertainty.
  2. An attempt at efficient information acquisition: CDR provides an approximation to counterfactual uncertainty reduction without requiring repeated model retraining.
Critical weaknesses
  1. Poor writing, hard to read: The paper is difficult to follow due to its complicated organization and insufficient explanations. For example, the introduction section needs much improvement.
  2. Overly simplistic experimental setting and evaluation: The experiments are restricted to a single synthetic environment with low-dimensional states and actions. The primary evaluation focuses on one-step prediction accuracy in only 60 zero-delay worlds, without experiments on standard control benchmarks. More experiments are necessary.
  3. What is the specific advantage of the proposed framework for decision-relevant exploration when the current experiments rely on randomly sampled queries rather than planner-generated ones? I guess this should be clarified.
Questions for authors

Can the authors provide experiments on more challenging or established benchmarks, including evaluations of multi-step prediction and downstream control performance?

1 comment
AuthorsOpenReview ↗

Reviewer aD3u: Prediction Through 16 Steps, Queries Generated by a Planner, and Downstream MPC Evaluation

We apologize for the delayed response. Thank you for the suggestions on prediction over multiple steps, downstream control, and queries generated by a planner. We have added these evaluations and clarified their relationship to the original experiment.

Prediction over multiple steps. We evaluated H=1, 4, 8, and 16 on 12 fixed worlds, with 16 action sequences per world and eight repeats with different simulator noise. Each rollout uses a fixed context. Predictions are compared with mean simulator trajectories. At B=64, state RMSEs are:

SelectorH=1H=4H=8H=16
Random0.0025640.0058480.0092160.015736
Local disagreement0.0042000.0090050.0140240.023723
Global CDR0.0028040.0063530.0098190.016557
Planner CDR0.0027570.0061960.0096280.016188

At H=16, global CDR reduces state RMSE by 30.2% relative to local. The paired difference is −0.007166 [−0.009293, −0.005040]. RMSE for predicted returns is 0.160701 versus 0.236797, a 32.1% reduction, with difference −0.076096 [−0.102355, −0.049837]. Random remains competitive, with lower mean errors and intervals for its differences from CDR that include zero. The appendix reports the complete budget and horizon results.

Queries generated by a planner. CEM generates 32 action sequences of length four from the learned models. Each member's discounted predicted return defines a query output. We test uniform weights and weights based on gaps in predicted returns. Query generation uses only the learned models, without hidden mechanisms or simulator outcomes. This directly tests the proposed use of a planner to supply the prediction targets, instead of relying only on randomly sampled actions. At B=64/H=16, planner CDR has state RMSE 0.016188, compared with 0.016557 for global CDR using random queries. The revised paper describes both query sources and reports their measured performance.

Downstream control. After B=0/32/64 identification interactions, we run CEM MPC with horizon eight for up to 32 control steps on the same 12 worlds with matched evaluation seeds. At B=64, mean returns are −23.276 for random, −23.080 for local, −23.121 for global CDR, and −23.358 for planner CDR. Success rates are 35.0%, 36.7%, 33.3%, and 31.7%, respectively. The interval for the return difference between global CDR and local is [−1.760, 1.676]. This provides a direct downstream evaluation, with no clear return advantage detected under the tested protocol. The paper distinguishes this result from the improvement in prediction accuracy.

Benchmark scope and presentation. These experiments extend the evaluation within the simulator with hidden physical parameters and no action delays. They do not replace an established second benchmark. We have clarified this scope, retained the original exposition where applicable, and updated the method and results to explain the query sources and downstream evaluation.

Reported intervals use five training seeds as the replication unit and are unadjusted for multiple comparisons.

Should you have any other concerns, we would be willingly to answer in further discussions.

Reviewer_fVJaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose Active Epistemic World Modeling (AEWM), a framework for choosing which interactions to perform that will reduce epistemic uncertainty “over a target set of counterfactual predictions”. This separates the irreducible outcome noise from the uncertainty about the hidden mechanisms that are reducible. Their method, which they call Counterfactual Disagreement Reduction (CDR), is a cheap-to-compute approximation of expected query-set variance reduction. The main motivation is that current exploration objectives may spend time on reducing uncertainty that doesn't affect decisions. CDR approximates the expected reduction in variance over a set of query predictions through couterfactual reasoning and reweighing members of an ensemble. This doesn't require retraining in any inner loop as many meta-RL models require.

In the tests, it is claimed that CDR reduces one-step prediction RMSE at 32 interactions by 19.7% relative to local disagreement. The gain is smaller when compared with random selection. The paper also shows that ablations of earlier models show a mismatch between the training target and delayed actions, and that the cheap CDR rankings agree only somewhat with a full audit over all candidates.

What is claimed in the abstract does not seem to hold up against what is actually tested.

Critical strengths

The authors are upfront about the limitations.

Critical weaknesses

There are no clear contributions beyond established prior work. The “framework” restates value-of-information literature, as well as (goal-oriented) Bayesian experimental design work. The authors point out the overlap with Bayesian experimental design themselves. Strongly related work on decision relevance in model-based RL is not discussed (Mehta et al., ICLR 2022, and follow-on work). This is critical because the framework is presented as the primary contribution.

The main potentially strong contributions are seemingly untested. The paper uses a single environment. There is not enough evidence to make a generality or robustness claim. The core results should contain comparisons to the relevant baselines, including from work authors already cite.

Questions for authors

What does AEWM provide that goal-oriented Bayesian experimental design (applied to your settings) does not? How does your work relate to Mehta et al. (2022)? Can access to the code repository be restored? It is currently unreachable.

2 comments
AuthorsOpenReview ↗

Reviewer fVJa: Contribution Relative to Bayesian Experimental Design and BARL, Expanded Evaluation, and Code Access

We apologize for the delayed response. Thank you for pointing out the connection to Bayesian experimental design and Mehta et al. (2022). We have added this discussion and expanded the experiments to clarify the contribution and its empirical scope.

Relationship to Bayesian experimental design and Mehta et al. (2022). AEWM applies targeted experimental design to predictions made by a world model. BARL, introduced in An Experimental Design Perspective on Model-Based Reinforcement Learning (ICLR 2022), selects queries about state and action pairs for information about the optimal MDP solution. CDR targets the expected reduction in variance over a specified set of predicted quantities. The revised related work now discusses this relationship explicitly.

The practical contribution is a batched ensemble score that evaluates candidate interactions through hypothetical outcomes and likelihood updates. Query predictions remain fixed during these virtual updates, which avoids retraining or encoding the context again for every hypothetical outcome. The query set can contain rewards for sampled actions or returns for sequences generated by a planner. This provides a concrete implementation of targeted acquisition for a context dependent world model. The contribution is the approximation and its empirical characterization within this setting, rather than a new principle of Bayesian experimental design.

Additional baselines and empirical evidence. We expanded the matched comparison to seven selectors, five seeds, 60 worlds without action delays, and budgets through 64. The added ensemble IG comparator uses the same predictor and likelihood machinery as CDR, which isolates the difference between generic information gain and reduction of uncertainty over query predictions.

At B=64, global CDR has RMSE 0.002398 versus 0.003615 for local disagreement. The paired difference is −0.001216 [−0.001685, −0.000747]. Random reaches 0.002167, and the difference between CDR and ensemble IG is −0.000050 [−0.000138, 0.000038]. CDR and random have nearly identical normalized AUC over 0–64. The results support the advantage over local at larger budgets without suggesting that CDR outperforms every acquisition criterion. Ensemble IG is a controlled comparator on the shared predictor, rather than a reproduction of BARL, VIME, or OPAX.

Scope of the evaluation. The paper now includes queries generated by a planner, prediction through H=16, downstream MPC, and sensitivity to ensemble size and query count. These additions test the method beyond the original prediction experiment over one step. They remain within the same simulator, so the paper presents the evidence as a study of this setting rather than a claim of generality across benchmarks.

Code access. We have fixed the original anonymous repository link: AEWM anonymous repository. We have updated the reproduction package to cover the experiments reported in the revised paper.

The revised abstract, related work, and results distinguish the design motivation, the implemented approximation, and the measured outcomes. Reported intervals use five training seeds as the replication unit and are unadjusted for multiple comparisons.

Please feel free to leave a comment if you would like to discuss any of these points further.

References. An Experimental Design Perspective on Model-Based Reinforcement Learning, Mehta et al. (2022)

Reviewer_fVJaOpenReview ↗

Response to rebuttal

Thank you for engaging in my questions, and for carrying out further evaluations. It seems to me that CDR is still worse than random here, so my concern remains. It also looks like the difference between CDR and ensemble IG is not statistically significant.

In terms of the novelty, it's still not really clear to me how this differs from established posterior-sample and importance-weighting approaches to Bayesian ED.

Reviewer_8DAnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Active Epistemic World Modeling (AEWM) framework for improving model-based reinforcement learning techniques. To do so, the exploration strategy proposed by the authors is on that reduces Counterfactual disagreement. The authors argue that this helps to make relevant counterfactual futures more reliably thus leading to efficient exploration.

The problem setting is of standard RL in MDP/POMDP settings with a simulator, an observation function and a reward function. The typical loop is similar to Dyna or other model-based RL methods with a few exploration steps, followed by model-learning, offline updates using the learned model and so on. The authors only focus on the exploration aspect in this work. Their objective is to use the limited interaction budget for model learning to identify good trajectories for learning world models. To do so, they use a surrogate decision-relevant information gain metric (Eqn. 6) over fixed prediction maps, queries and weights, which is used to synthesize a score for a given experiment (a partial episode upto budget).

The authors then showcase an empirical evaluation using a newly created benchmark with two simple baselines and a few ablations.

Critical strengths

The problem setting is very relevant to the AI community and the proposed approach of using active learning for epistemic exploraiton by counterfactual disagreement reduction is an intuitive approach.

Critical weaknesses

Unfortunately, i do not think this paper is ready for publication just yet. While interesting, it seems there are a few gaps in both the overall work and the evaluation. Im providing some points which i think can be useful in a revised version to improve chances of acceptance.

Firstly, the paper writing is a bit robotic i think. Its accessible enough to researches well-versed in the area but not really easy to read otherwise.

Next, the paper repeatedly says that the focus is on reducing disagreement but actually using their system in a full end-2-end framework is not performed. This in conjunction with the points below dilutes the key pitch of the paper.

There are no baselines except for local disagreement. I think more baselines or e2e performance needs to be shown to truly understand the value of the work.

Next, there are some constants like K=5, L=32.. why are these selected as such. What about others. Are these artifacts of the environment?

Fig 2. is only plotted at the inflection point. Why not go to 64. It is not clear if the trends continue. Valuable data is missing to make an informed decision here.

Questions for authors

Please address my weaknesses. Happy to discuss and reconsider if i have misunderstood the work.

1 comment
AuthorsOpenReview ↗

Reviewer 8DAn: Additional Baselines, Evaluation Through 64 Interactions, Sensitivity to K and L, and Downstream Control

We apologize for the delayed response. Thank you for the concrete suggestions on baselines, downstream evaluation, parameter choices, and the budget curve. We have added experiments addressing each of these points and updated the manuscript.

Additional comparisons and budgets through 64. We now compare seven selectors on the same 60 evaluation worlds and five training seeds. All selectors share the same checkpoint within each seed. The added ensemble IG baseline measures information gain about which ensemble member generated an observation. Two CDR variants use return queries generated by a planner.

SelectorRMSE at B=32RMSE at B=64Normalized AUC, 0–64
Random0.0022630.0021670.002495
Local disagreement0.0028390.0036150.003026
Ensemble IG0.0023090.0024480.002508
Global CDR0.0022790.0023980.002495
Weighted CDR0.0022690.0024050.002493
Planner CDR0.0022960.0023430.002481
Planner weighted CDR0.0022980.0023490.002483

At B=64, global CDR reduces RMSE by 33.6% relative to local disagreement. The paired difference is −0.001216, with a 95% interval of [−0.001685, −0.000747]. Its AUC over 0–64 is lower than local's. The full curve shows where this benefit occurs: local performs better at the earliest budgets, while CDR performs better at larger budgets. Random has the lowest endpoint error at B=64, and its AUC difference from CDR includes zero. Figure 2 and the complete budget table now show the curve through 64 rather than stopping at 32.

Choices of K and L. We retrained ensembles with K=3, 5, and 10 using the same training protocol. CDR reduces B=64 RMSE relative to local by 33.8%, 40.7%, and 37.1%, respectively. All three paired intervals exclude zero in CDR's favor. Varying L over 8, 32, and 128 gives B=64 RMSEs of 0.002401, 0.002398, and 0.002381. The paired differences from L=32 include zero. These results show that the advantage over local is not specific to K=5. They support L=32 as a practical choice with similar measured accuracy to the other tested query counts.

Downstream evaluation. We added CEM MPC after identification on 12 fixed worlds, using matched reset states and seeds for simulator noise and planning. At B=64, global CDR's mean return is −23.121, compared with −23.080 for local and −23.276 for random. The interval for the return difference between CDR and local is [−1.760, 1.676], so this experiment does not show a clear return advantage. Prediction over longer horizons gives a clearer benefit: at B=64/H=16, state RMSE is 0.016557 for CDR versus 0.023723 for local, a 30.2% reduction. The paper reports prediction and realized control return separately.

We have clarified the query definitions and the distinction between the Bayesian objective and the implemented CDR score. The added experiments appear in the main results and appendix. Intervals use five training seeds as the replication unit and are unadjusted for multiple comparisons.

Should you have any other concerns, we would be willingly to answer in further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.