acceptodds
Under review as a conference paper at ICLR 2027

MineLoop: Inner-Time Computation for Autoregressive Visual World Models

Abstract

Autoregressive visual world models turn each sampled token into evidence for later predictions. Standard feed-forward predictors sample each token after a single pass, without an explicit stage for refining uncertain predictions under the same history. We introduce MineLoop for inner-time world modeling, giving each prediction an internal trajectory before it enters the world history. MineLoop encodes the committed visual-action prefix as causal evidence, initializes a target-indexed future state, and recurrently updates it with a shared prediction core while the evidence remains fixed. Only the terminal state emits a visual token. On action-conditioned Minecraft rollout, MineLoop improves FVD, LPIPS, SSIM, and PSNR against MineWorld controls at Small, Medium, and Large model tiers, with FVD reductions of 16.3%, 10.5%, and 7.3%, respectively. Counterfactual camera-action tests show stronger action alignment, and an inverse-dynamics evaluator recovers conditioning actions with comparable macro-F1. Across 64 held-out sources from the Small model, 23.13% of visual tokens change from wrong at K1 to correct at K4, while 0.093% change in the opposite direction. The advantage over a standard autoregressive model is concentrated in motion and occlusion regions. Continuing beyond the trained terminal step raises confidence while reducing accuracy, showing that MineLoop learns a finite-depth prediction program with a specific terminal state. These results identify token commitment as a productive locus for recurrent computation in visual world models.

Primary area:
generative models
Keywords:
visual world models, recurrent inference, inner-time computation, action-conditioned generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zb4FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work propose MineLoop, for inner-time world modeling, to improve the token prediction performance based on historical evidence. To verify the performance, this work compare MineLoop with other methods on MineCraft benchmark.

Critical strengths
  • This work leverage historical content to refine and improve the token prediction.
  • The method part presents well for understanding the detailed approach.
Critical weaknesses
  • The word "inner-time" appears in the title, abstract and other essential parts, but seems lack the explaination of this term. What "inner-time" actually is?
  • The comparison is limited. Would the method also work fine on scenarios except MineCraft benchmark, like the real-world task? Also, the amount of compared method is not enough. More world model methods are needed to be compared.
Questions for authors

Please see "Critical Weaknesses" part for reference.

2 comments

We sincerely thank the reviewer for their constructive comments and valuable feedback. Below, we use C to denote the reviewer's questions or concerns, and R to present our corresponding responses.


C: The word "inner-time" appears in the title, abstract and other essential parts, but seems lack the explaination of this term. What "inner-time" actually is?

R: Thank you; we agree the term should be defined precisely at first use. Formally:

  • World time is the index nn of the autoregressive sequence. It advances by one when a visual token is sampled and appended to the history s<n\mathbf{s}_{<n}.
  • Inner time is the index k=0,1,…,Kk = 0, 1, \dots, K of the recurrent updates qnk+1=Dϕ(q≤nk,E<n)\mathbf{q}_n^{k+1} = D_\phi(\mathbf{q}^{k}_{\le n}, \mathbf{E}_{<n}) applied to the prediction state of one target position nn. During these updates, the evidence E<n\mathbf{E}_{<n} is fixed and no token is sampled.

In plain terms, inner time counts the recurrent prediction steps for one token while the history stays fixed. After the K-th step, a token is sampled and world time advances. The revision gives this definition at the first mention in the introduction and in Section 3, and uses "recurrent prediction steps" elsewhere.


C: The comparison is limited. Would the method also work fine on scenarios except MineCraft benchmark, like the real-world task?

R: Thank you for raising this; we acknowledge it as a limitation and state it explicitly in the revision. We chose MineWorld because it is an open, action-conditioned, token-autoregressive world model with a public tokenizer and data pipeline. This allowed us to train the baselines from scratch on identical data at three scales, with matched estimated training FLOPs at two of them. That control is what isolates the effect of the architectural change.

MineLoop makes no Minecraft-specific assumption: it requires only an interleaved visual-action token sequence and a causal backbone. Real-world domains such as driving video are an important next step.


C: Also, the amount of compared method is not enough. More world model methods are needed to be compared.

R: Thank you. Our comparison is designed to isolate a single change. MineLoop and MineWorld share the tokenizer, training data, frame resolution, action representation, and training recipe, and differ only in how each token's prediction is computed; at Small and Medium they are also matched in estimated training FLOPs. World models from other works differ in several of these factors at once, so differences in their reported numbers could not be attributed to the prediction architecture. For this reason, we prioritized controlled comparisons at three scales, complemented by counterfactual action tests and an independent inverse-dynamics evaluator.

Following the reviewer's suggestion, we added one more controlled comparison: MineLoop Small trained with a single recurrent step (K = 1) and otherwise identical. Under the Table 1 rollout protocol:

ModelParamsFVD ↓LPIPS ↓
MineWorld (25 layers, training-FLOP-matched)397.1M1040.7070.31330
MineLoop K = 1339.2M1052.3380.31512
MineLoop K = 4339.2M870.6060.30169

The improvement over MineWorld appears only with the recurrent prediction steps.

Thank for the authors' feedback. However, as the evaluation is limited (no expansion to other world model benchmarks, and no comparison to existing world model approaches), I think it is not available for a higher rating.

Reviewer_uaZiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MineLoop introduces recurrent computation before token sampling in autoregressive visual world models, aiming to refine predictions before they become part of the generated history. It repeatedly updates a latent prediction state using a shared core and fixed causal evidence, with supervision applied to the terminal readout. On Minecraft, it improves rollout quality across three model scales.

Critical strengths

Refining predictions through shared recurrent computation under fixed causal evidence is an interesting architectural contribution, with consistent empirical improvements over MineWorld on the evaluated rollout metrics.

Critical weaknesses
  1. The presentation is often terminology-heavy, insufficiently explicit, and overly AI-written. For example, K1 and K4 appear in the abstract without explanation.
  2. Evaluation is restricted to Minecraft, leaving the applicability of the proposed architecture to other environments and visual dynamics unclear.
  3. The main paper lacks controlled ablation study, and analysis in Section 5 is not insightful enough.
Questions for authors
  1. Why is the Large MineWorld baseline not matched to MineLoop in training FLOPs, and would the gains persist under a matched budget?
  2. Could you provide controlled ablations of key components of MineLoop?
  3. Since the prediction loss is applied only at K4, why should the 23.132% wrong-at-K1/right-at-K4 transition rate be interpreted as evidence of a distinctive correction mechanism?
  4. What architectural or algorithmic insight follows from the experimental results in Section 5.4 and 5.5?
  5. Missing related work: iVideoGPT on autoregressive, action-conditioned visual world modeling.
3 comments

We sincerely thank the reviewer for their constructive comments and valuable feedback. Below, we use C to denote the reviewer's questions or concerns, and R to present our corresponding responses.


C: The presentation is often terminology-heavy, insufficiently explicit, and overly AI-written. For example, K1 and K4 appear in the abstract without explanation.

R: Thank you for this direct feedback. We agree, and we have revised the manuscript as follows:

  • Notation. K1/K4 no longer appear in the abstract. In the main text, KkK_k is defined at first use as the prediction read out after kk applications of the shared prediction cell.
  • Terminology. "Inner time" is defined once, formally, in Section 3 (see also our response to Reviewer zb4F). Elsewhere we use plain terms such as "recurrent prediction steps," and we have removed other coined phrases.
  • Explicitness. Each analysis in Section 5 now states what is measured, on which data, and under which protocol before reporting its result.
  • Style. We have rewritten the introduction and the analysis section to remove repeated conceptual summaries, so that each claim is stated once, next to its evidence.

C: Evaluation is restricted to Minecraft, leaving the applicability of the proposed architecture to other environments and visual dynamics unclear.

R: Thank you for raising this; we acknowledge it as a limitation and state it explicitly in the revision. We chose MineWorld because it is an open, action-conditioned, token-autoregressive world model with a public tokenizer and data pipeline. This allowed us to train every control from scratch on identical data at three scales, with matched estimated training FLOPs at two of them. That control is what isolates the architectural change.

MineLoop makes no Minecraft-specific assumption. It requires only an interleaved visual-action token sequence and a causal backbone, and it changes only the per-token prediction computation. Extending the evaluation to other environments is an important next step.


C: Why is the Large MineWorld baseline not matched to MineLoop in training FLOPs, and would the gains persist under a matched budget?

R: Thank you for this question. The three tiers are designed to answer two complementary questions. At Small and Medium, the MineWorld controls are deepened to match MineLoop's estimated training FLOPs. These tiers test whether the recurrent predictor is a better use of the same budget than additional backbone layers, and MineLoop improves all four rollout metrics with fewer parameters. At Large, both models share the identical 20-layer, 2048-width backbone. This tier tests whether adding the recurrent predictor to a fixed large backbone still helps, and it improves FVD by 7.3% with consistent gains on LPIPS, SSIM, and PSNR; its decode FLOPs ratio is reported in Table 5. In the revision, the Large row is labeled consistently as a same-backbone comparison in all table captions.


C: Since the prediction loss is applied only at K4, why should the 23.132% wrong-at-K1/right-at-K4 transition rate be interpreted as evidence of a distinctive correction mechanism?

R: Thank you; we agree with this point. Because only K4 is supervised, readouts at K1–K3 pass intermediate states through a head trained for the terminal state. The transition rate therefore describes how the terminal readout of the trained computation takes shape across steps, and it should not be read as identifying a separate correction mechanism. In the revision, we have removed this statistic from the abstract and contributions, presented it in Section 5 as a descriptive trace of the trained computation, and removed the "rescue/repair" wording. The main claims of the paper, rollout quality and action response under matched controls, do not rely on this analysis.


C: What architectural or algorithmic insight follows from the experimental results in Section 5.4 and 5.5?

R: Thank you for this question. We have added an explicit takeaway to each subsection.

  • Section 5.4 (prediction depth, margin, and confidence). Under a uniform four-step budget, positions in the same frame settle at different steps. This identifies per-token adaptive depth as a natural efficiency direction. Together with the terminal-depth result (Table 7), it also identifies the prerequisite: adaptive or extended depth must be trained, for example through intermediate supervision or randomized depth, because terminal-only training does not produce usable intermediate exits.
  • Section 5.5 (evidence retrieval). Across steps, the same target state reads a progressively wider history: mean evidence lag grows from 0.98 to 3.15 frames, and attention to action tokens grows from 0.15 to 0.34. Because the evidence is encoded once and its keys and values are reused, this wider read requires no re-encoding of the history. The design implication is to loop the predictor over a fixed evidence cache rather than the whole backbone: additional steps then change what is read from history at no extra encoding cost.

C: Missing related work: iVideoGPT on autoregressive, action-conditioned visual world modeling.

R: Thank you for pointing this out. We have added iVideoGPT (Wu et al., NeurIPS 2024) to the related work. iVideoGPT scales interactive autoregressive video world models through compressive tokenization of observations, actions, and rewards. MineLoop is orthogonal to the choice of tokenizer and backbone: it modifies the per-token prediction computation and is applicable to iVideoGPT-style backbones.


C: Could you provide controlled ablations of key components of MineLoop?

R: Thank you. The key component is the recurrent prediction step. We trained MineLoop Small with K = 1 and kept everything else identical, including the 339.2M parameters, data, and training recipe. Rollout under the Table 1 protocol:

ModelParamsFVD ↓LPIPS ↓
MineWorld (25 layers, training-FLOP-matched)397.1M1040.7070.31330
MineLoop K = 1339.2M1052.3380.31512
MineLoop K = 4339.2M870.6060.30169

Without recurrence, MineLoop does not improve on the FLOP-matched control. With the same parameters and K = 4, FVD falls from 1052.3 to 870.6. The gain therefore comes from the recurrent steps rather than from the predictor architecture or parameter count. Teacher-forced NLL and accuracy show the same ordering (see our response to Reviewer CLi8). We add this ablation to the revision.

I appreciate the authors' response. However, my major concerns remain insufficiently addressed. The evaluation is still limited in terms of domain diversity, and the analysis does not provide sufficiently novel insights into the looped architecture. Furthermore, although the authors state that they have revised the manuscript, I do not appear to have access to the revised version (e.g., the undefined terms K1 and K4 still appear in the abstract).

I will keep my score.

Reviewer_CLi8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitation that autoregressive visual world models commit each sampled token after a single feed-forward pass, so prediction errors immediately become evidence for later steps. The key idea is to separate "inner time" from "world time": each prediction is refined internally before it is committed to the history. This is implemented via a causal evidence encoder that encodes the committed visual-action prefix once, and a target-indexed query that is updated K = 4 times by a shared recurrent cell (query self-attention, cross-attention to the fixed evidence, and an MLP), with only the terminal state supervised and decoded. Experiments on action-conditioned Minecraft rollout show improved FVD, LPIPS, SSIM, and PSNR over MineWorld controls at Small, Medium and Large scales, with training-FLOP-matched baselines at Small and Medium, along with better response to counterfactual camera actions and lower teacher-forced NLL.

Critical strengths
  1. The paper highlights an important issue in autoregressive world models: each token is committed after one pass and immediately becomes evidence for later predictions.
  2. Against the MineWorld controls, MineLoop consistently improves rollout quality and action controllability across three scales, with fewer parameters and matched compute at Small and Medium.
  3. The analysis suggests that the gains concentrate in motion and occlusion regions, although its representativeness should be clarified (see Question 3).
Critical weaknesses
  1. Overclaimed motivation. The paper motivates MineLoop by the problem that each token is committed after a single feed-forward pass and its errors then propagate to later predictions. However, MineLoop does not change this mechanism: tokens are still committed irrevocably in causal order, and a committed error propagates exactly as in a standard AR model.

  2. "Inner-time refinement" is a fixed-depth forward pass. Since K is fixed to 4 at both training and inference and only the terminal state is supervised, the unrolled model is a single forward pass through a fixed-depth, weight-tied decoder rather than a refinement process. Prior looped and recurrent-depth models instead supervise outputs at multiple iterations, as in PonderNet [1], or randomize the number of iterations during training, as in Geiping et al. [2], the latter explicitly allowing more test-time iterations to improve performance. Discussing these strategies would clarify whether MineLoop's inner-time computation can genuinely be scaled.

  3. Missing ablations. The paper does not report a K = 1 MineLoop or models trained with different K, so it is unclear how much of the gain depends on recurrence itself. The K1–K4 readouts analyzed in Section 5 do not substitute for this, since they come from a model whose prediction head is trained only at K4.

  4. Missing simple baselines. An untied 4-layer decoder with the same target query and cross-attention at matched compute would directly test whether weight-shared recurrence is needed. Moreover, since the paper is motivated by error accumulation, simple mitigations applicable to MineWorld, such as training with noisy context frames as in GameNGen [3], should be considered.

References:

[1] Banino et al., 2021. PonderNet: Learning to Ponder. arXiv:2107.05407.

[2] Geiping et al., 2025. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS 2025.

[3] Valevski et al., 2024. Diffusion Models Are Real-Time Game Engines. arXiv:2408.14837.

Questions for authors
  1. Would supervising intermediate steps or randomizing K during training allow additional inner steps at inference to improve, rather than degrade, performance?
  2. Under true history, Section 5.1 reports that MineLoop lowers NLL by 0.246 nat over the Small control, roughly 8× the gain in Table 3 (about 0.03 nat), and its absolute K4 NLL on the analysis set (3.66, Table 7) is also much higher than in Table 3 (about 2.93–3.00). How do the 64 analysis sources and the evaluation protocol differ from Table 3, and do the Section 5 findings hold on the main evaluation set?
  3. What is the wall-clock latency per generated frame compared with MineWorld?
3 comments

We sincerely thank the reviewer for their constructive comments and valuable feedback. Below, we use C to denote the reviewer's questions or concerns, and R to present our corresponding responses.


C: Overclaimed motivation. The paper motivates MineLoop by the problem that each token is committed after a single feed-forward pass and its errors then propagate to later predictions. However, MineLoop does not change this mechanism: tokens are still committed irrevocably in causal order, and a committed error propagates exactly as in a standard AR model.

R: Thank you for this precise observation. We agree, and we have revised the motivation accordingly.

MineLoop leaves commitment and error propagation unchanged: tokens are committed irrevocably in causal order, exactly as in a standard AR model. What MineLoop changes is the computation that produces the distribution from which each token is sampled. The intended effect is a lower per-token error rate under the same history, so that fewer errors enter the history in the first place. The paper measures both sides of this argument. Under true history, MineLoop lowers NLL and raises top-1 accuracy at every tier and context length (Table 3). Under generated history, it improves all four rollout metrics at every tier (Table 1).

The revised introduction states this directly: MineLoop reduces the error of each committed token, and the commitment mechanism itself is unchanged.


C: "Inner-time refinement" is a fixed-depth forward pass. Since K is fixed to 4 at both training and inference and only the terminal state is supervised, the unrolled model is a single forward pass through a fixed-depth, weight-tied decoder rather than a refinement process. Prior looped and recurrent-depth models instead supervise outputs at multiple iterations, as in PonderNet [1], or randomize the number of iterations during training, as in Geiping et al. [2] [...]. Discussing these strategies would clarify whether MineLoop's inner-time computation can genuinely be scaled.

R: Thank you. We agree with this characterization: at a fixed K, the unrolled predictor is a weight-tied, four-step computation. We do not claim adaptive test-time scaling, and Table 7 is consistent with that: a fifth step raises confidence but lowers accuracy.

The contribution lies in where this computation is placed. Recurrent-depth language models re-apply their core block to the full sequence representation. MineLoop instead encodes the visual-action history once and loops only the target-indexed prediction state against this fixed evidence. The keys and values of the evidence are computed once and reused across all K steps (Section 3, causal execution), so additional steps change what the prediction reads from history without re-encoding that history. At Small and Medium, this allocation outperforms spending the same estimated training FLOPs on additional backbone layers (Table 1), while using 14.6% and 12.5% fewer parameters.

In the revision, we (i) use "recurrent prediction" instead of "refinement" wherever the latter could suggest test-time improvability; (ii) state explicitly that K is a trained depth; and (iii) discuss PonderNet's per-step supervision with learned halting and Geiping et al.'s randomized recurrence depth as the two established routes to depth that can be extended at test time. Both are training strategies and can be applied to MineLoop's predictor without architectural changes.


C: Missing simple baselines: a compute-matched untied decoder would test whether the gain requires weight sharing.

R: Thank you for this suggestion. Our claim concerns where computation is placed, not weight sharing as such. An untied predictor would place its compute in the same position with more parameters; weight sharing is what lets MineLoop do so with 14.6% and 12.5% fewer parameters than the matched controls. The K = 1 ablation, which keeps the parameters and removes only the repeated steps, eliminates the gain (see below).


C: Moreover, since the paper is motivated by error accumulation, simple mitigations applicable to MineWorld, such as training with noisy context frames as in GameNGen [3], should be considered.

R: Thank you for this suggestion. Context-noise augmentation is an effective mitigation of exposure bias and is relevant here. It acts on the training-time distribution of the evidence, while MineLoop acts on the predictor that reads this evidence. The two are therefore complementary rather than alternatives: noise augmentation can be applied to MineLoop's evidence encoder unchanged. We have added GameNGen to the related work and state this relation explicitly.


C: Missing ablations: without a K = 1 MineLoop, it is unclear how much of the gain depends on recurrence itself.

R: Thank you. We trained MineLoop Small with K = 1 and kept every other setting identical: 339.2M parameters, the same data, and the same training recipe. This is a separately trained model, not the K = 4 model truncated after one step.

Teacher-forced prediction (Table 3 protocol, 16 held-out trajectories):

ModelParamsShort NLL ↓Extra-long NLL ↓Extra-long Acc ↑
MineWorld (25 layers, training-FLOP-matched)397.1M3.02572.96010.3067
MineLoop K = 1339.2M3.04182.97680.3043
MineLoop K = 4339.2M3.00042.93010.3100

Rollout (Table 1 protocol, 256 held-out initial states):

ModelFVD ↓LPIPS ↓
MineWorld (25 layers, training-FLOP-matched)1040.7070.31330
MineLoop K = 11052.3380.31512
MineLoop K = 4870.6060.30169

With a single step, MineLoop does not improve on the FLOP-matched control; it is slightly worse, as expected from its lower training compute. With the same parameters and K = 4, FVD falls from 1052.3 to 870.6, and teacher-forced NLL and accuracy improve. The gain therefore comes from the recurrent prediction steps, not from the predictor architecture or the parameter count. We add this ablation to the revision.


C: Would supervising intermediate steps or randomizing K during training allow additional inner steps at inference to improve, rather than degrade, performance?

R: Thank you for this question. We tested intermediate supervision. We trained MineLoop Small at K = 4 with a weighted cross-entropy loss on the readout of every step, and evaluated K = 1 to 8 under the protocol of Table 7:

K1234 (trained)5678
NLL ↓4.01253.78413.71023.67903.68343.69573.71203.7301
Top-1 acc ↑0.17860.21350.22510.22980.22910.22730.22500.2226

Intermediate supervision removes the degradation beyond the trained depth. From K4 to K5, NLL changes by +0.0044 instead of +1.4267 under terminal-only training (Table 7), and accuracy by −0.0007 instead of −0.0921. Additional steps still do not improve the prediction: NLL rises slowly from K5 to K8. Intermediate supervision therefore makes the recurrence stable beyond its trained depth, but on its own it does not turn extra steps into gains. We report this result in the revision. We do not claim test-time depth scaling.


C: The Section 5 NLL gain (0.246) is about 8× the gain in Table 3, and its absolute NLL (3.66) does not match Table 3 (≈2.93). What explains the difference, and do the Section 5 findings hold on the main evaluation set?

R: Thank you for checking this carefully. The two results come from different evaluation sets and protocols. Section 5 analyzes 64 held-out sources with one teacher-forced forward pass over the full 16-frame clip, scoring every visual token in frames 1–15. Table 3 evaluates 16 held-out trajectories under the short, long, and extra-long context windows.

To answer the second question directly, we recomputed the Section 5 comparison on the Table 3 trajectories and windows (Small, 25-layer MineWorld control versus MineLoop K4). The paired NLL gain is 0.0253, 0.0297, and 0.0300 for the short, long, and extra-long windows, which matches Table 3. The regional decomposition is:

RegionNLL gain, main evaluation setNLL gain, Section 5 (Table 8)
Motion0.0610.427
Motion boundary0.0440.302
Occlusion / newly revealed0.0580.429
Static0.0120.073

The regional pattern of Section 5 holds on the main evaluation set. Motion and newly revealed regions gain the most, motion boundaries are intermediate, and static regions gain the least. Each regional gain scales down in proportion to the overall gain of the evaluation set. In the revision, Section 5 states its protocol explicitly and reports these results on the main evaluation set.


C: What is the wall-clock latency per generated frame compared with MineWorld?

R: Thank you. We measured both models on the same GPU with the same decoding implementation (static KV cache and torch.compile for both), batch size 1, generating 15 frames after one observed frame:

TierMineWorld (ms/frame)MineLoop (ms/frame)Latency ratioDecode FLOPs ratio (Table 5)
Small8579001.05×1.00×
Medium1,1861,2481.05×1.00×
Large1,3521,6381.21×1.23×

At Small and Medium, where decode FLOPs are matched, MineLoop is 5% slower because its K predictor steps run sequentially. At Large, the latency ratio follows the FLOPs ratio. We add this table to the revision.

Thank you for the detailed response and the new experiments. However, I have two key concerns remaining. First, there is still no untied-decoder baseline to show that recurrence itself matters. The K = 1 ablation does not fill this gap, since it also reduces compute and therefore only shows that more predictor depth helps. Second, the method is a fixed-depth predictor rather than an extendable refinement process. I will therefore keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.