acceptodds
Under review as a conference paper at ICLR 2027

OnlineWM: Causality-Aware Active Online Learning for Effective World Modeling

16 views

Abstract

Generative world models aim to predict future states conditioned on actions, where action controllability is fundamental for reliable dynamics modeling. While recent efforts leverage simulator-generated data to enhance this capability, existing training pipelines face two fundamental limitations. First, static offline data collection leads to a distribution misalignment between training sets and the model’s evolving error patterns, failing to resolve critical long-tail scenarios where dynamics predictions remain unreliable. Second, the standard objective of minimizing observational discrepancy often encourages the model to exploit spurious correlations instead of capturing the underlying action-effect causality. To address these limitations, we propose OnlineWM, an online training framework that continuously improves world modeling through active simulator interaction and causality-aware optimization. OnlineWM introduces two key innovations: (1) Active Online Learning: Instead of using fixed datasets, OnlineWM adaptively queries the simulator for new interaction sequences that target the model's current predictive weaknesses, ensuring high-utility data acquisition. (2) Causality-Aware Fine-Tuning: We propose a counterfactual learning strategy that contrasts the outcomes of different actions from identical states, forcing the model to attribute state transitions to specific actions rather than ambient environmental evolution, thereby grounding its predictions in reliable causal mechanisms. By integrating active data acquisition with causal optimization, OnlineWM establishes a closed-loop refinement process that ensures the model is both robust to diverse scenarios and precise in its causal attribution. Extensive experiments demonstrate that OnlineWM significantly enhances action controllability and generalizes effectively to unseen domains, suggesting the learning of physically-grounded causal dynamics rather than simple visual patterns.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Models, Active Online Learning, Causality-Aware

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K7yrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes OnlineWM, an online post-training framework for action-conditioned video world models, instantiated on HY-World 1.5 (8B, initialized from the WorldCompass checkpoint) with 32 parallel Minecraft simulators. It has two components: Active Online Learning (AOL) and Causality-Aware Fine-Tuning (CFT).

Critical strengths

The problem is relevant. Poor action following for long-tail action–scene combinations is a real failure mode of current interactive world models.

The engineering is substantial and well documented: closed-loop training of an 8B model with 32 parallel simulators.

The paper’s best idea is the coupling between its two components.

Critical weaknesses

The main tables are ablations, not comparisons. No alternative training strategy is evaluated, no other action condition world model compared.

All results use a self-curated test set. Make it hare to the performance relative to existing Minecraft world models.

The paper characterizes selected samples as “novel yet learnable,” but the score is the current loss only; nothing measures learnability or progress.

Questions for authors

Why did the authors choose not to evaluate with the public protocols already established for Minecraft world models, such as the IDM-based action-following metrics of MineWorld or GameWorld Score from Matrix-Game?

How well does Depth Anything 3 recover camera poses on this Minecraft data?

How does OnlineWM relate to RLIR (https://arxiv.org/abs/2509.23958), which also post-trains video world models for action-following?

8 comments
AuthorsOpenReview ↗

Response to Reviewer K7yr (Part 1/2)

Thank you for recognizing the relevance of the problem, the documented engineering effort, and the coupling between AOL and CFT. Your concerns and questions are organized below.

W1 & W2 & Q3: External comparisons and the relationship to RLIR

W1. The main tables are ablations, not comparisons. No alternative training strategy is evaluated, no other action condition world model compared.

W2. All results use a self-curated test set. Make it hare to the performance relative to existing Minecraft world models.

Q3. How does OnlineWM relate to RLIR (https://arxiv.org/abs/2509.23958), which also post-trains video world models for action-following?

A1. We thank the reviewer for the comments and for pointing out relevant work.

The main goal of this paper is to improve action controllability through Active Online Learning (AOL) and Causality-Aware Fine-Tuning (CFT). In Tables 2 and 7, we compare HY-World 1.5 8B, initialized from the WorldCompass [4] checkpoint, under Random online SFT and other variants under fair comparison settings. Relative to Random online SFT, AOL reduces RPErot\mathrm{RPE}_ {\mathrm{rot}} and RPEtrans\mathrm{RPE}_ {\mathrm{trans}} by 21.27% and 16.58%, respectively; adding CFT to AOL yields further reductions of 19.09% and 4.22%. These results confirm the effectiveness of our OnlineWM approach.

Following your suggestion, we add comparisons with two action-conditioned world model baselines, Matrix-Game [1] and Matrix-Game 3.0 [2]:

MethodSSIM ↑LPIPS ↓Combined Acc. ↑Fine Acc. ↑RPErot\mathrm{RPE}_ {\mathrm{rot}} ↓RPEtrans\mathrm{RPE}_ {\mathrm{trans}} ↓
WorldCompass (Baseline)0.4580.5060.2400.4900.69060.0235
Matrix-Game0.4850.5270.2930.5050.77990.0189
Matrix-Game 3.00.4310.5350.2160.5140.54830.0243
OnlineWM0.5310.4110.4250.6400.40720.0159

These results show that OnlineWM outperforms all selected baselines.

Relation to other post-training methods for action following. RLIR [3] and WorldCompass [4] optimize world models through reward-based reinforcement learning on model-generated rollouts. Their performance depends heavily on rollout quality and reward-model reliability. Generating these rollouts and obtaining reliable rewards introduce substantial design complexity and computational cost. OnlineWM directly aligns the world model with ground-truth simulator outputs through active selection and causality-aware optimization, providing a more efficient, accurate, and controllable training approach. We will expand the related-work discussion to cite [3, 4] and clarify the relationship to these post-training methods.


Q1 & Q2: Public Minecraft evaluation protocols and comparability

Q1. Why did the authors choose not to evaluate with the public protocols already established for Minecraft world models, such as the IDM-based action-following metrics of MineWorld or GameWorld Score from Matrix-Game?

Q2. How well does Depth Anything 3 recover camera poses on this Minecraft data?

A2. We thank the reviewer for the comment, and we would like to make the following clarifications.

Evaluation protocols. IDM-based metrics typically rely on discrete bins to assess action correctness rather than directly estimating camera poses. Our evaluation compares action following under both discrete action and continuous camera-pose conditioning to assess the effectiveness of OnlineWM training. Estimating camera poses directly from video is therefore more reliable and straightforward than recovering them from discrete inverse-dynamics predictions. For a more comprehensive evaluation, we also use VBench and established video-quality metrics (SSIM and LPIPS) to compare our method with the baselines.

Depth Anything 3 on Minecraft data. Depth Anything 3 is trained on a large-scale dataset and can accurately recover actions directly in Minecraft (>95% accuracy for both view and movement actions). For camera-pose error evaluation, we apply closed-form Sim(3) Umeyama alignment to the camera centers to account for the estimator’s per-scene scale and any global coordinate-frame offset.

AuthorsOpenReview ↗

Response to Reviewer K7yr (Part 2/2)

W3: Basis for describing selected samples as novel and learnable

W3. The paper characterizes selected samples as “novel yet learnable,” but the score is the current loss only; nothing measures learnability or progress.

A3. We thank the reviewer for the comment. The training loss, evaluated at a fixed timestep of the flow-matching objective, can serve as an efficient and stable proxy for identifying samples that are challenging but not impossible to learn [5, 6]: it requires only one forward pass, and its stability is supported by Table 8. In OnlineWM, we further decompose this loss into action- and scene-level difficulty to guide the learning curriculum of the generative world model, an approach whose effectiveness is also demonstrated empirically in Table 7.

We will include this discussion and the additional references in the final version.


We hope these clarifications and additional experiments address the reviewer's concerns, and we would appreciate the reviewer's consideration of these points in the final evaluation.

References

[1] Zhang, Yifan, et al. "Matrix-game: Interactive world foundation model." arXiv, 2025.

[2] Wang, Zile, et al. "Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory." arXiv, 2026.

[3] Ye, Yang, et al. "Reinforcement learning with inverse rewards for world model post-training." arXiv, 2025.

[4] Wang, Zehan, et al. "Worldcompass: Reinforcement learning for long-horizon world models." ICML, 2026.

[5] Mindermann, Sören, et al. "Prioritized training on points that are learnable, worth learning, and not yet learnt." ICML, 2022.

[6] Kumar, M., Benjamin Packer, and Daphne Koller. "Self-paced learning for latent variable models." NeurIPS, 2010.

Dear Reviewer K7yr,

Thank you once again for your valuable feedback on our submission. As the discussion period is approaching its end, we would be grateful to know whether our rebuttal has adequately addressed your concerns. Should any questions remain or any points require further clarification, please do not hesitate to let us know, and we will respond promptly.

If you find that our responses have resolved the main concerns, we would greatly appreciate it if you could take the rebuttal and additional evidence into account when updating your assessment.

We sincerely look forward to your feedback.

Best regards,

Authors of Paper 7472

Reviewer_3vv5OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Training action-conditioned world models is prone to a failure mode in which the model ignores the action conditioning and simply predicts the most probable next state. To mitigate this, the paper proposes an online training strategy that actively collects data. The strategy balances data collection so that the model learns accurate transition dynamics, and it adds a counterfactual fine-tuning loss that contrasts the outcomes of different actions taken from the same state. The method is validated by training a diffusion transformer on Minecraft, where it improves action controllability while preserving visual quality.

Critical strengths
  • The paper addresses an important problem. Poor action controllability is a known failure mode of current world models, and it degrades performance on downstream tasks: planners that use world models as predictive models, and policies trained inside world models, often produce inaccurate or hallucinated trajectories. The paper clearly motivates two plausible causes of poor action controllability: action–state pairs that are rare in the training data, and a prediction loss that can be minimized without using the action.
  • The approach is described in detail, including implementation details and negative results. It is evaluated both on Minecraft and on more general domains, although the latter evaluation is limited.
Critical weaknesses
  • The paper suggests that these issues may stem from the reconstruction loss used during training, here combined with a frozen VAE. It would be valuable to study whether the issues persist under other training recipes. For instance, JEPA-style world models such as LeWM or PLDM do not use any pixel reconstruction loss, and world models such as Dreamer reconstruct other signals, such as rewards, in addition to pixels.
  • The extra EMA forward passes add training compute, and the paper does not report this overhead relative to the random baseline. A compute-matched comparison against the baselines would strengthen the evaluation.
  • The paper frames the approach around generative world models, but the problems it addresses matter just as much for world models used for planning or for training control policies. It would be interesting to measure how much poor action controllability contributes to failures on planning tasks, and how far the online training recipe can fix them.
Questions for authors
  • Does the diagnosis of poor action controllability also apply to training recipes that do not rely on generative objectives? Would the proposed approach still be relevant in those settings?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3vv5 (Part 1/2)

Thank you for recognizing the importance of the action-controllability problem, the motivation for the approach, and the detailed presentation of implementation choices and negative results. Your concerns and question are organized below.

W1 & Q1: Applicability beyond reconstruction-based generative training

W1. The paper suggests that these issues may stem from the reconstruction loss used during training, here combined with a frozen VAE. It would be valuable to study whether the issues persist under other training recipes. ...

Q1. Does the diagnosis of poor action controllability also apply to training recipes that do not rely on generative objectives? Would the proposed approach still be relevant in those settings?

A1. We thank the reviewer for the thoughtful comments. OnlineWM currently focuses on generative world models, where scene generation is coupled with accurate action control. Its active acquisition and counterfactual-training principles may also extend to other types of world models.

Following your suggestion, we conduct a preliminary adaptation of OnlineWM to LeWM [1] on a TwoRoom navigation task, where an agent should pass through a doorway to reach an image-specified goal. Door locations, starting positions, and goals vary across scenes. All methods start from the same official checkpoint and train for 1,500 updates with batch size 32, using two seeds and matched collection budgets.

Random online SFT selects action branches randomly. AOL instead selects branches with high EMA-model prediction error, retaining 20% random exploration. CFT additionally uses mixed latent inputs of paired outcomes under their respective actions. Training data are collected online throughout optimization.

We measure world-modeling performance using 15-step autoregressive latent prediction error, normalized by target-feature variance (NMSE), over 512 held-out scenes. Planning success is evaluated on 128 held-out goals using the same CEM planner for every model: a 15-step planning horizon, replanning every five steps, and a 30-step execution budget. Success requires reaching within the environment’s standard 16-pixel goal threshold.

MethodWM NMSE ↓ Seed 1WM NMSE ↓ Seed 2Planning success ↑ Seed 1Planning success ↑ Seed 2
Random online SFT0.13160.129343.75%47.66%
AOL0.10710.111265.62%52.34%
AOL + CFT0.08820.088367.19%71.88%

The adapted CFT objective reduces mean NMSE by 19.1% and improves mean planning success by 10.55 percentage points over the adapted AOL baseline, with improvements in both seeds. These preliminary results support the benefits of active acquisition and counterfactual training in a latent-prediction world model without pixel reconstruction.


W2: Additional EMA computation and comparison with the random baseline

W2. The extra EMA forward passes add training compute, and the paper does not report this overhead relative to the random baseline. A compute-matched comparison against the baselines would strengthen the evaluation.

A2. Per your request, a detailed breakdown of relative computational cost (compared to the supervised training stage) is shown below. We have included all sub-stages in the corresponding training stage. For example, CFT time includes input preparation, EMA weight swapping, reference–EMA and current–model forward passes, loss computation, and backpropagation.

Training StageCollectEncodeScoreFMCFTGrad UpdateTotal
Relative Time0.194×0.388×0.155×1.000×0.317×0.187×2.241×

Offline training also requires data collection and VAE encoding. The additional OnlineWM-specific stages are model scoring and CFT optimization, which account for approximately 20% of total runtime. With appropriate engineering, the rollout, encoding, and scoring stages can be further parallelized across simulator workers independently of training.

We use the same number of optimization steps for all variants to ensure a fair comparison. Our ablations show that both AOL and CFT clearly improve performance on the fine-grained camera-pose metrics: AOL yields relative reductions of 21.27% and 16.58% in rotational and translational error, respectively, while CFT yields corresponding reductions of 19.09% and 4.22%. These results confirm the effectiveness of OnlineWM.

AuthorsOpenReview ↗

Response to Reviewer 3vv5 (Part 2/2)

W3: Downstream effects on planning and control

W3. The paper frames the approach around generative world models, but the problems it addresses matter just as much for world models used for planning or for training control policies. It would be interesting to measure how much poor action controllability contributes to failures on planning tasks, and how far the online training recipe can fix them.

A3. We thank the reviewer for the comment. We agree that accurate action following is important for planning and for training policies in world models. In A1, the preliminary LeWM results show that lower prediction error for action control yields higher planning success, which is consistent with our expectations. We plan to explore downstream applications of generative world models with improved action following in future work.


We hope these clarifications address the remaining questions and concerns. We appreciate the reviewer's positive assessment of the framework and hope the additional details are helpful.

References

[1] Maes, Lucas, et al. "Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv, 2026.

Reviewer_EC5wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OnlineWM trains a video world model to follow actions through repeated interaction with a simulator. Active Online Learning (AOL) generates several action branches from a simulator state and scores the model's prediction errors. It uses these scores to balance scene and action difficulty when selecting further trajectories. Causality-Aware Fine-Tuning (CFT) compares the outcomes of different actions from the same starting state. Its modified flow-matching objective aims to make predictions more sensitive to the requested action. The Minecraft experiments measure action following and camera trajectory errors, alongside visual quality. The paper also presents examples of transfer beyond Minecraft.

Critical strengths
  1. I find the use of simulator interventions interesting. A world model can generate plausible video while failing to follow the requested action. Running different actions from the same starting state provides a controlled way to study that problem. Updating the collected data as the model changes also gives the method a way to target its current errors.

  2. Table 7 provides reasonable evidence that the combined procedure improves action control, albeit with only modest gains. Combined action accuracy rises from 0.356 with random collection to 0.393 with AOL and 0.425 when CFT is added. Rotation and translation errors improve as well. These results support combining AOL and CFT, with mixed changes in visual quality across the reported metrics.

Critical weaknesses
  1. My main concern is whether Causality-Aware Fine-Tuning (CFT) adds anything beyond learning from the additional simulator examples. Removing CFT also removes training on the short counterfactual pairs. Changes to the model can then change which data the online procedure collects. Table 7 therefore leaves the contribution of the objective somewhat unresolved.

I expected a different baseline, one that uses ordinary flow matching on both branches of the same collected pairs, with each branch conditioned on its correct action. Use the same starting model and simulator data for both methods. Then compare them with the same amount of training data and, separately, with the same compute budget. Giving both methods the same simulator examples still controls for the extra data, while matching compute controls for the additional training cost. Any remaining gain would provide stronger evidence that the proposed loss helps.

  1. I also need stronger evidence that the gains address the cases motivating the paper. Aggregate action accuracy can hide failures on rare actions and unseen action–scene combinations. Collision cases need separate evaluation because correctly blocked movement can look like failure to follow a command. Table 7 provides no variability across training runs. Appendix A.4 describes how to select a translation threshold for each method, but it doesn't make clear whether that selection uses an independent calibration set.

The reported control metrics support better action following. Claims about reliable causal mechanisms require tests of the effects of changing an action while holding the starting conditions fixed. The examples outside Minecraft also need an aggregate evaluation with component ablations to support the stated transfer claims.

My provisional recommendation is weak rejection because the contribution of the CFT loss and the reliability of the key gains remain unresolved. A convincing same-data supervised comparison, with gains that hold across training runs and the motivating evaluation cases, could change my recommendation. The authors should also align the causal claims with the evidence.

Cost and related work are secondary concerns. Eight local branches avoid an exponentially growing rollout tree, but their practical cost still needs measurement. Include simulator collection, encoding, scoring, and training in the comparison. The related work should explain how the acquisition rule differs from Plan2Explore (Sekar et al., ICML 2020) and Self-Supervised Exploration via Disagreement (Pathak et al., ICML 2019). It should also identify what CFT adds to the cited DiffusionNFT objective.

Questions for authors
  1. Does CFT outperform ordinary flow matching on the same fixed simulator data when both methods train on both action branches? Please use the same initialization and condition each branch on its correct action. Report equal-data and equal-compute comparisons, including variability across training runs. This result would have the largest effect on my recommendation.

  2. Were thresholds calibrated independently of the test set, and were test scenes or maps held out from training? Do gains hold for rare actions and unseen action–scene combinations? How does the evaluation distinguish correctly blocked movement from failed action following?

  3. When generating CFT pairs, which parts of the simulator state and randomness do you hold fixed? Which action tokens and camera poses condition each prediction? How do you handle different actions that produce equivalent visible outcomes, such as movement blocked by an obstacle?

  4. Given CFT’s 3.2-point accuracy gain over AOL alone, how much additional compute does it require, including data collection and training? Does the gain persist under the same total compute budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EC5w (Part 1/3)

Thank you for recognizing the value of simulator interventions, adaptive data collection, and the observed improvements in action control. Your concerns and questions are organized below.

W1 & Q1: Isolating the contribution of the CFT objective

W1. My main concern is whether Causality-Aware Fine-Tuning (CFT) adds anything beyond learning from the additional simulator examples. ...

Q1. Does CFT outperform ordinary flow matching on the same fixed simulator data when both methods train on both action branches? ...

A1. We thank the reviewer for suggesting a more controlled comparison of the CFT objective. We first clarify that, although CFT introduces an additional loss term, the total number of gradient updates remains the same, allowing us to isolate its effect on the overall gradient direction by removing this term.

Following your suggestion, we train an FM-only variant with the same amount of training data, also applying the flow-matching loss to the short branches conditioned on their respective actions. We report the results below.

MethodCombined Acc. ↑Fine Acc. ↑RPErot\mathrm{RPE}_ {\mathrm{rot}} ↓RPEtrans\mathrm{RPE}_ {\mathrm{trans}} ↓
AOL only0.3930.6100.50330.0166
 + FM_only (matched negative)0.3930.6140.49380.0167
 + CFT0.4250.6400.40720.0159

The results above show that additional supervision on the short branches has little effect on model performance, with slight improvements in fine-grained action accuracy and RPErot\mathrm{RPE}_ {\mathrm{rot}}. However, this variant clearly underperforms the model trained with CFT, providing direct evidence for the effectiveness of our proposed loss beyond standard supervised fine-tuning.

For training variability, we apologize that we currently are not able to report variability across independent training runs because of the substantial computational cost of training an 8B video DiT (32 GPUs with 80 GiB of memory each in our experiments). Our ablations show that both core components clearly improve model performance over baselines: AOL yields relative reductions of 21.27% and 16.58% in rotational and translational error, respectively, while CFT yields corresponding reductions of 19.09% and 4.22%. These results confirm the effectiveness of OnlineWM.

We hope this additional comparison clarifies CFT's contribution and would appreciate your consideration of these points in the final evaluation.


W2 & Q2 & Q3: Evaluation and CFT data construction

W2. I also need stronger evidence that the gains address the cases motivating the paper. ...

Q2. Were thresholds calibrated independently of the test set, and were test scenes or maps held out from training? ...

Q3. When generating CFT pairs, which parts of the simulator state and randomness do you hold fixed? ...

A2. We thank the reviewer for the comment, and we would like to make the following clarifications.

Evaluation. During training, each action is held for 4–16 latent frames, compared with 2–3 latent frames during testing. This results in more frequent action changes and challenging evaluation action sequences that are unseen during training. The test scenes are all held out from training and generated independently with different random seeds.

The candidate thresholds are set empirically without using the test set, and we apply the same evaluation protocol to all models following [2]. Action accuracy provides a coarse-grained measure of action following [1, 2], while translational and rotational camera-pose errors provide complementary fine-grained, continuous measures that accurately characterize the model’s ability to follow commands under challenging test scenes. Regarding variability across training runs, we follow the practice of related work [1, 2], given the prohibitively high training cost and the clear performance improvements observed in our ablations.

CFT pair construction and blocked movement. We restore the exact player position and orientation in the simulator to collect CFT pairs. We also remove all living entities from Minecraft to ensure that the rollouts can be exactly reproduced. We randomly sample compound action tokens as described in Appendix A.2 and record the resulting camera poses directly from the simulator. These records provide the executed-action and the accurate corresponding realized-motion annotations for each rollout, including cases where different actions produce the same visible outcome.

AuthorsOpenReview ↗

Response to Reviewer EC5w (Part 2/3)

W2: Evidence for causal claims

W2. The reported control metrics support better action following. ...

... The authors should also align the causal claims with the evidence.

A3. The term causality-aware reflects CFT’s loss construction: it contrasts simulator continuations under different actions while keeping the initial conditioning state fixed, encouraging the model to capture action-induced visual changes. This design aims to improve precise action following, a central capability for action-conditioned world models.

To illustrate the connection with action following, consider camera motion represented as vectors in a shared coordinate system. Let mam_a and m^a\widehat m_a denote the ground-truth and predicted motion vectors under control aa at a shared starting state. By the triangle inequality, the error in predicting the effect of changing an action is bounded by the sum of the two motion-prediction errors:

||(\widehat m_ a-\widehat m_ b)-(m_ a-m_ b)|| }_ {\text{error in predicting the action effect}} \leq \underbrace{ ||\widehat m_ a-m_ a||+||\widehat m_ b-m_ b|| }_ {\text{sum of motion-following errors}}.$$ Thus, small prediction errors under both controls bound the error in their predicted motion difference in this representation. Our results on held-out action sequences provide behavioral evidence consistent with the motivation of the causality-aware objective. We thank the reviewer for pointing this out, and we will revise and expand the discussion of causal claims in our final version. ------ ### W2: Aggregate evaluation of transfer beyond Minecraft > W2. The examples outside Minecraft also need an aggregate evaluation with component ablations ... **A4.** Using the same evaluation protocol, we evaluate the Minecraft-trained checkpoint variants on the non-Minecraft scenes used for the transfer study. We use 6 initial scenes, each paired with 20 action sequences of 15 latent steps (61 video frames). We report the aggregate results below. | Method | Combined Acc. (%) ↑ | Fine Acc. (%) ↑ | $\mathrm{RPE}_ {\mathrm{rot}}$ ↓ | $\mathrm{RPE}_ {\mathrm{trans}}$ ↓ | | ----------------------- | ------------------: | --------------: | -------------------------------: | ---------------------------------: | | Baseline | 35.10 | 59.69 | 1.8861 | 0.3469 | |  Random online SFT | 46.41 | 66.90 | 1.0739 | 0.2656 | |  AOL Only | 50.10 | 69.09 | 0.9850 | 0.2411 | |  AOL + CFT | 52.81 | 71.07 | 0.9508 | 0.2283 | Fine-tuning on Minecraft improves action following on these evaluated domains, supporting that accurate action control can transfer to other domains. Further improvements may require training on more advanced simulators with fine-grained camera control capability and bridging the domain gap by realistic rendering techniques. ------ ### W3 & Q4: End-to-end computational cost > W3. Cost and related work are secondary concerns. ... > > Q4. Given CFT’s 3.2-point accuracy gain over AOL alone, how much additional compute does it require, including data collection and training? Does the gain persist under the same total compute budget? **A5.** Per your request, a detailed breakdown of relative computational cost (compared to the supervised training stage) is shown below. We have included all sub-stages in the corresponding training stage. For example, CFT time includes input preparation, EMA weight swapping, reference–EMA and current–model forward passes, loss computation, and backpropagation. | Training Stage | Collect | Encode | Score | FM | CFT | Grad Update | Total | | -------------: | ------: | -----: | ---------: | -------: | ---------: | ----------: | -----: | | Relative Time | 0.194× | 0.388× | **0.155×** | *1.000×* | **0.317×** | 0.187× | 2.241× | We would like to clarify that offline training also requires data collection and VAE encoding. The additional OnlineWM-specific stages are **model scoring** and **CFT optimization**, which account for **approximately 20%** of total runtime. With appropriate engineering, the rollout, encoding, and scoring stages can be further parallelized across simulator workers independently of training. We use the same number of optimization steps for all variants to ensure a fair comparison. Our ablations show that both AOL and CFT clearly improve performance on the fine-grained camera-pose metrics: AOL yields relative reductions of 21.27% and 16.58% in rotational and translational error, respectively, while CFT yields corresponding reductions of 19.09% and 4.22%. These results confirm the effectiveness of OnlineWM.
AuthorsOpenReview ↗

Response to Reviewer EC5w (Part 3/3)

W4: Relationship to exploration methods and DiffusionNFT

W4. The related work should explain how the acquisition rule differs from Plan2Explore (Sekar et al., ICML 2020) and Self-Supervised Exploration via Disagreement (Pathak et al., ICML 2019). It should also identify what CFT adds to the cited DiffusionNFT objective.

A6. We thank the reviewer for pointing out these relevant works.

Relation to Plan2Explore and exploration via disagreement. Both works use ensemble disagreement to guide a learned exploration policy [3, 4]. Plan2Explore further optimizes this policy over imagined trajectories in a learned latent world model [3]. AOL shares the motivation of adapting data acquisition to the model's current limitations. Our acquisition rule in Sec. 3.2 directly evaluates short simulator forks using the target world model's flow-matching prediction error at a fixed noise level. We decompose this error into a scene-level mean and an action-level residual, then select complementary difficulty ranks: harder scenes are paired with easier actions and vice versa. AOL therefore performs explicit, difficulty-balanced branch selection for world-model training, without an ensemble-based intrinsic reward or an additional exploration policy.

Relation to DiffusionNFT. As stated in Sec. 3.3, CFT adopts DiffusionNFT's forward-process optimization and implicit positive/negative policies with an EMA reference [5]. DiffusionNFT weights model-generated samples using rewards. CFT uses simulator counterfactuals from the same starting state: z+\mathbf{z}_ {+} follows the selected action, while z−\mathbf{z}_ {-} is the most visually similar outcome of a different action. The negative is a valid simulator outcome generated under a different action and is treated as a counterfactual negative under the selected action. CFT additionally injects γk(z−−z+)\gamma_ k(\mathbf{z}_ {-}-\mathbf{z}_ {+}) into the forward-noised latent (Eq. (4)) and uses the two corresponding reconstruction targets with similarity-based weights (Eq. (6)). Our contribution is the controlled action supervision and counterfactual input/target construction within the DiffusionNFT framework.

We will expand the related-work discussion to cite [3, 4] and make the connection to these exploration methods, the components inherited from DiffusionNFT [5], and the additional CFT design more explicit.


We hope that our clarification and additional experiments can help address the reviewer's concerns, and we would appreciate the reviewer's consideration of these points in the final evaluation.

References

[1] Zhu, Yixuan, et al. "Astra: General interactive world model with autoregressive denoising." ICLR, 2026.

[2] Wang, Zehan, et al. "Worldcompass: Reinforcement learning for long-horizon world models." ICML, 2026.

[3] Sekar, Ramanan, et al. "Planning to explore via self-supervised world models." ICML, 2020.

[4] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. "Self-supervised exploration via disagreement." ICML, 2019.

[5] Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." ICLR, 2026.

Dear Reviewer EC5w,

Thank you once again for your valuable feedback and constructive discussion on our submission. We are glad to know that our additional experiments have addressed your main concern, and we appreciate the opportunity to further clarify the experimental details.

As the discussion period is approaching its end, we would greatly appreciate it if you could take our rebuttal, additional experimental evidence, and further clarifications into account when updating your assessment.

Should you have any further questions, please do not hesitate to let us know, and we will respond promptly. Thank you again for your time and consideration.

Best regards,

Authors of Paper 7472

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.