acceptodds
Under review as a conference paper at ICLR 2027

When Generative Models Act as Simulators

Abstract

A simulator summarizes an environment in a state that users can read and change. Generative models learn representations of the world without explicit specification, but offer no built-in way to edit that world's state. We investigate the conditions under which a generative model can act as a simulator through : whether changing a model's latent representation produces the next-step outputs we get by editing a simulator's state. Plainly stated, our goal is to find a mapping between a generative model's latent state and the simulator state, where the model is trained on observations alone and the mapping is learned afterward. We test two kinds of mappings, both learned from observations paired with simulator states. In one, we train probes to decode the simulator state from the model, then steer along them. In the other, we learn the mapping directly with an inverse map, which can succeed even where steering fails. We study both in two contrasting environments: the discrete board game Othello, and Rayworld, our own continuous world of moving discs seen through partial renderings, closer to physical scenes and video. We find that which mapping succeeds depends strongly on the environment, even though the state is decodable in every variant. In Othello, editability falls smoothly as the game is simplified, and even small rule changes have large consequences. In Rayworld, probes fail to steer along the simulator's continuous state, yet an inverse map sets the model from that state anyway. A model can therefore be driven by a simulator it may have never learned.

Primary area:
generative models
Keywords:
generative models, simulators, interpretability, model steering, representation probing, latent interventions

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rfowOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

We are studying a class of problems in which we train a transformer on a sequence of observations of a partially-observable, stateful system, and try to understand the information content and encoding of the latent states of the learned representation. Previous work has shown, in some cases (e.g. basic Othello) that the latent representation is "editable" in the sense that it can be altered to correspond to a different true hidden world state in such a way that future predictions are sensible.

This paper goes farther, studying variations on Othello and another prediction problem, Rayworld, and studying a different way of probing the information content of the latent representation: that is, by learning to map the true hidden world state to the learned latent state, and then "editing" the learned latent to the one corresponding to some different desired world state. This paper asks when will this technique work and how does it compare to pre-existing methods that work by more directly "steering" the latent representation in a desired direction.

The paper asks the question and formulates tests for decodability and editability of the latent state. It the compares different probe types on different versions of the ground-truth state machines.

Critical strengths

This paper asks an interesting question and I learned a lot from reading it. (I am far from an expert in this area, though I was familiar with the basic Othello editability results, so I did have a lot to learn).

It asks questions that are close to the original heart of ICLR: what is in the nature of the representations that latent-variable models learn?

Critical weaknesses

In the end, I don't exactly know what to make of this paper. Although the direct editing method and the results are interesting, I'm not sure whether they shed deep light on the underlying learning or representation processes.

The "morals" didn't come through clearly. In particular the idea that "simplifying an environment's rules can smoothly reduce its editability" is I guess demonstrated in these problems, by example. But what are the principled reasons why this might be true? What makes us believe that this is a general principle?

What makes these two domains and the changes you made to their rules particularly revealing? I'm really looking for some general principles.

Very small points:

  • At first I thought this was a typo of some sort: " in Othello editability falls smoothly" but now I guess we mean it decreases. But calling something smooth that's based on 4 discrete changes seems overstating the case (or I misunderstand).
  • The writing is harder to understand and more Claude-ish ("IM lands everywhere") toward the end
  • Is the transformer width of 512 the dimension of Z?
Questions for authors

What makes these two domains and the changes you made to their rules particularly revealing?

Could you construct a domain in which there is a good encoder/decoder model but for which it would not (for some interesting reason) be able to map a true hidden world state into the model's latent space in a way that would preserve the semantics? Maybe this has to be possible.

5 comments
AuthorsOpenReview ↗

Response (1/2): The general principle and a new test

Thank you for a thoughtful review, and for calling the paper close to the heart of ICLR. Your question about the principles behind our results led us to expand the general theory in a new subsection on forcing simulator-like states (Sec. 5.1), along with a new experiment that tests it.

What is the general principle? A model acts as a simulator when prediction pressures it to compress its history into the simulator's state. The submission originally linked this to computational mechanics (Shalizi & Crutchfield, 2001), and Sec. 5.1 now elaborates on that connection. Since the observations are generated from the simulator, the simulator's state is always a sufficient statistic of the observations. The question is about the minimal sufficient statistic (the causal state), and how much coarser it is than the simulator's state.

  • In Othello, we hypothesize that more complex game rules leave fewer alternative representations which are both (1) simpler than the board and (2) as successful at predicting as the board state is. So the causal state is closer to the board. Empirically, this manifests as each simplification of the rules lowering editability.
  • In Rayworld we hypothesize that the causal state is coarser than the simulator's state. In plain wording, this is what we mean by a shortcut. Our new results suggest that the model shifts its recent frames in observation space first, and then refines a rendering of the next frame, bypassing any 2D state. Since it already predicts nearly optimally, more scale gives it no reason to represent distinctions the causal state discards.

The principle also predicts a fix. One way to picture this is to think of the causal state as a bag wrapped loosely around the simulator’s state. Anything that refines the causal state draws the air out, so the sheet ends up conforming more closely to the simulator’s state, meaning the model must represent more of it (App. A.17). In a new experiment, we make the Rayworld model commit to a latent state before it knows the next viewpoint, and then render the next observation from that state alone. A state that must serve every viewpoint can only be finer than (or equal to) the causal state for one fixed viewpoint, and thus closer to the simulator's state. This meaningfully increases editability on the 8-ray model, raising the Edit Index of PI and GS to +0.55 and +0.42 (from +0.21 and -0.07 before), with a smaller gain on 128 rays (App. A.17). If the rendering step can instead read the whole history, the gain disappears, as expected from the theory. We consider this a promising recipe for improving the simulator behavior of generative models.

Q1. What makes these two domains revealing? They sit at opposite ends of our principle. Othello's rules make the board both necessary for prediction and complexly related to the moves, and it is the setting where editable world models were first shown (Li et al., 2023). Rayworld is continuous and partially observed (closer to video) but its frames reveal a lot of its state, which leaves room for a shortcut. The variants then simplify the rules in Othello and change what the observation reveals in Rayworld, while the model and analysis stay fixed. Such an analysis helps to reveal the principles governing the representations that a model learns based on the observations and training environment it is exposed to.

AuthorsOpenReview ↗

Response (2/2): Semantic mappings and other notes

Q2. Can a good model fail to admit a semantic mapping? Yes, only once the inverse map is limited in size. An inverse map of unlimited size could technically even rerun the model itself, so a mapping would always exist and would say nothing about the model. We therefore match the inverse map's size to the probes, so it is a shallow MLP (App. A.14). With that limit, the paper already contains a case where the inverse map fails. In adjacent-noflip, a good model decodes the board at every step, yet an inverse map fit across all time steps fails to edit it. Our reading is that its representations never hold the board in one consistent way across time steps, so only a map fit to a single time step succeeds.

By this view, we can predict another setting where the inverse mapping would fail in an informative way. In Rayworld, the inverse map appears to learn something like the rendering itself, since the model works in observation space and rendering is cheap. Now imagine an interactive video world model, where rendering is expensive. If that model also works in observation space, a mapping from the world state into its latent space needs to implement the rendering process, and a small inverse map may fail. If the model works with something like a 3D state, the mapping should be simpler. We have begun follow-up work testing related ideas on a large video world model.

The inverse map as a tool. Beyond editing, the inverse map also lets us see how a model represents and generates its world. Tracking where it succeeds across residual points revealed Othello's board bottleneck and Rayworld's gradual rendering (Fig. 6). Fitting it at a single time step revealed that adjacent-noflip represents the board differently from step to step (App. A.16).

"Smoothly." We agree that "smoothly" does not best capture the Othello editability results. We now explain what we mean by "simpler," and show that each simplification lowers editability monotonically (Sec. 3.1, Sec. 5).

Writing. We rewrote the Discussion and Conclusion for clarity, state when an editor succeeds (Sec. 3.5), and use "the inverse map" in place of "IM" in the revised prose.

Is the width 512 the dimension of zz? Each residual point is a 512-dimensional vector, and ztz_t is comprised of all nine, one per point (Sec. 3.3). In the main results, the editors write one point at a time, except GS, which also writes every later point (following the original OthelloGPT implementation).

As the discussion period closes soon, we wanted to check whether the general principle in the new Sec. 5.1 and the accompanying experiment (App. A.17) address your question about what the results show. We’d be glad to clarify anything. Thank you again for the thoughtful review.

Reviewer_11foOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Large video/world models, such as Sora, Genie, and Cosmos, are called "world simulators", but a real simulator has an explicit state that you can set, while a model trained on observations has no such control. Interpretability work on Othello-GPT showed that a board can be read out of a transformer with a probe and sometimes even edited, but reading a variable does not guarantee that it can also be set.

The paper asks when a trained model can be set from a simulator's state after training, which the authors call editability - writing a target state into the model's activations and checking whether its next prediction matches what the simulator produces after the same edit. The model, training and analysis stay fixed and only the environment changes.

The authors test this on Othello with a few rule changes, and on Rayworld, a new toy world of two moving discs.

They compare two ways to edit a model (trained on the same data):

  1. Probe steering changes the model's internals until a probe, a small model trained to read the world's state from those internals, reports the matching state.
  2. Via an inverse map that learns to map a world state to the matching internals and swaps them in directly.

The world's state can be read out of every model, but editing only sometimes works. In standard Othello both kinds of edits work. With changed rules only the inverse map works, or nothing does. In Rayworld, probe steering mostly fails, while the inverse map mostly works.

Critical strengths

The setup makes editability something you can measure. Both simulators can be edited directly, so every edit is checked against the exact outcome the simulator gives, not judged by eye or by another probe. The architecture and training are fixed. Every model predicts close to optimally, so a failed edit can't be blamed on a weak model. The released weights, edit sets and per-setting results are unusually complete and reproduce the main tables.

The Rayworld result is a clean, well-tested example that reading a variable is not the same as being able to set it. The probe edit moves the probe's reading exactly onto the target, yet the predicted frame barely changes. This holds with smooth shading, five viewpoints and a token interface. The authors also report results that cut against them, such as edits reverting after one step unless the history is rewritten.

Critical weaknesses

The Othello results seem to come from impossible edits, not from simpler rules.

  • Without flipping, a token can never change colour so the test edit, flipping one token, creates a board that no real game could produce. When the edit is one a real game could produce (swapping two tokens of opposite colour, Table 6), the inverse map works on a no-flip variant at least as well as on normal Othello.
  • The four versions don't form a scale from complex to simple. They are two rule changes combined, and results drop in two jumps rather than gradually. In one version a token's colour is fixed by its square, so there is nothing real to edit.
  • MetaOthello (Chawla et al., 2026) already trains separate models on Othello rule variants, including a similar placement rule, so this part is less new than the paper suggests.

The inverse map's success may say little about the model. ALso, the evaluation favours it.

  • Much simpler edits already work. Copying in the average internals of the ten most similar training examples, with nothing learned, gets a big part of the inverse map's effect (Table 12). Writing just before the output works almost as well as the best layer (Table 10), and on the coarsest version the best result comes from writing at the input. So the inverse map may be learning to draw the next frame from the state, or doing the same as swapping in a new last image. The paper doesn't test these simpler explanations.
  • The settings are tuned on the same test edits that are reported.
  • Edits last one step. After that the model goes back to the original world unless its whole history is rewritten.
Questions for authors
  • Does the inverse map beat replacing the last input frame with the edited render, patching activations from that run, or a direct state-to-frame regressor? If these match it, the third contribution needs reframing.
  • What exactly falls "smoothly" in Othello, given that the inverse map works on reachable edits in the adjacent no-flip variant? Do single-flip edits still fail if the edit is also written at earlier positions, such as where the flipped token was placed?
  • Were the layer and step size chosen on the evaluation cases? If so, can you report results on held-out cases?
  • Why does the released benchmark keep only edits that change at least two rays? How does the coarse-ray trend look with one-ray edits included?
  • Is the inverse map fit across all sequence positions? If so, how well does it fit at the edit position alone? Position may explain its poor fit to Rayworld activations.
  • Does probe steering on continuous positions still fail when the edit is written at every later layer, as gradient steering is, or along a difference-of-means direction?

Smaller fixes:

  • Sec. 5 says the inverse map sets unreachable boards "nearly as well" as reachable ones, while A.4 says it does better on reachable boards (Table 5).
  • "Landing" is never defined. Pseudoinverse injection on adjacent-flip is called failing, while a modestly better categorical injection result at 5 rays is treated as a success.
  • The Fig. 5b caption says it covers all editors, but gradient steering and standard-noflip are missing.
6 comments
AuthorsOpenReview ↗

Response (1/3): Othello reachability and variants

Thank you for a considered and careful review. Your questions led directly to the new controls, the layer-wise analysis, and the single-time-step fits in the revision. These results address both weaknesses you raise, and in each case they support the paper's claims with more precise evidence. Revisions are marked in blue, and the table and appendix numbers from your review are unchanged.

W1. Do the Othello results come from impossible edits? (Q2) No, and the new results give evidence against it. Reachability only separates the results on one of the four variants.

  • In standard, every editor edits unreachable boards as well as reachable ones (Table 5).
  • In adjacent-flip, only the IM succeeds, and it edits unreachable boards too, though somewhat less well (+0.61 against +0.78).
  • In standard-noflip, each cell determines its color, so the data never reveals how the rules would treat a token of the other color. Prediction never requires the model to represent color apart from the cell. This is the limiting case of the principle in Sec. 5.1, and every edit fails, as it predicts.
  • In adjacent-noflip, the IM fit across all time steps edits the reachable pairs (Table 6) but not the single flips. Following your Q5, an IM fit at the edit step alone edits those unreachable single flips, at an Edit Index of +0.86 (App. A.16), but doesn't work across all timesteps. So unreachable boards are not the barrier here either, and the editability is not general.

What adjacent-noflip shows instead is brittle editability. The step-fit IM still works at nearby steps where the same player moves, but fails at steps where the other player moves. Describing the board by absolute color instead of mine and theirs does not fix this, and probes fit at the edit step still fail to edit. So the model can be edited under the right circumstances, but not robustly, unlike standard and adjacent-flip.

Writing at earlier positions (Q2). Writing a single-flip edit where the flipped token was placed does not edit on its own. Writing it there and at the current step raises the IM's Edit Index to +0.40, still below adjacent-flip even when written at the current step alone (App. A.4).

W1. Do the variants form a scale? (Q2) We agree that "smoothly" overstated the results, and removed it. We now explain what we mean by "simpler" (Sec. 3.1). The variants combine two rule changes, and each makes the game simpler to play. Adjacent placement replaces a search for runs of the opponent's tokens along lines across the board with a local color check. Removing flips means a token never changes once placed. standard-noflip keeps the standard placement rule, but its fixed checkerboard makes the board easy to track and legal moves easy to calculate. Each simplification lowers editability.

W1. MetaOthello. Thank you for the pointer, which we now discuss in Sec. 2. Chawla et al. ask a different question with a different experiment. They train transformers on Othello variants, alone and in mixtures, to study how one model organizes several world models, and find a largely shared board representation through which probes trained on one variant can intervene on another. We ask when a single model's state can be set from a simulator's state, and test this with editability metrics, an inverse map, and a second environment, Rayworld. Their variants are designed to agree or conflict within one model, while ours make the game simpler to play. Their DelFlank variant uses a placement rule like our adjacency rule, but deletes enclosed tokens, and NoMidFlip still flips the outermost enclosed tokens. Neither removes flipping, which is the change with the largest effect in our results.

AuthorsOpenReview ↗

Response (2/3): What the inverse map’s success shows

W2. Does the IM's success say little about the model? (Q1, Tables 10 and 12) No. The new controls show that the inverse map (IM) isn't just editing the inputs, and they reveal how the Rayworld model generates (Sec. 4, Fig. 6, App. A.14). The submission proposed that the Rayworld model never forms a compact 2D state, and works with its observations instead. The new experiments make this more precise. Our hypothesis is that the early layers shift the recent frames in observation space, bypassing any 2D state, and the later layers refine this rendering of the next frame. The clearest evidence comes from passing the latent state at each residual point through the decoder. A rough version of the next frame appears by point 3, and most of it by point 5. The IM edits well by point 3 and plateaus around point 5. In this view, the IM is close to a rendering operation, but it renders in the model's latent space and starts from what the model has already computed.

All results below are on standard Rayworld (Table 14).

EditEdit IndexEdit Fidelity
Splice the edited render as the last frame (oracle)-0.090.37
Patch activations from that run (oracle)-0.090.37
Splice the last two frames (oracle)+0.360.55
Splice the last three frames (oracle)+0.630.68
Learned renderer, three frames spliced+0.090.41
Learned renderer, ten frames spliced+0.460.46
IM+0.590.66
Counterfactual patch (oracle)+0.910.84
  • Splicing. Splicing true renders of the edited world needs three frames to match the IM.
  • Patching from the spliced run. Under causal attention, every position before the edit is the same in the spliced and original runs. Patching the edit position at any layer therefore recomputes exactly what the spliced run computes, so this control equals the one-frame splice.
  • The state-to-frame regressor. We trained an MLP of the IM's size on the same state and frame pairs to render observations directly. Scored on its own outputs, it reaches an Edit Index close to the IM's, but its loss is about ten times the Bayes floor, against about 1.3 times for the transformer (Table 3). Spliced into the history, its frames fall meaningfully short of the IM even with ten frames, so the IM is not just editing the model's inputs.
  • The counterfactual patch runs the model on the edited world's full history and writes its latent state at one residual point, which is exactly what the IM learns to predict. It fails at the embedding, where it amounts to replacing the last frame, and edits the prediction almost exactly from the middle of the network on (Table 14).

Tables 10 and 12. We agree that success just before the output says little, since the decoder alone maps that point to the prediction. The informative points are in the middle of the network, where the IM and the counterfactual patch already succeed with several blocks to go. The nearest-neighbor control also writes the model's own latent states, averaged over nearby worlds, so its partial success fits this account. On the coarsest variant, writing at the input does act much like swapping in the edited last frame. With only 5 rays an edit changes few values, and the next frame often matches the last one, so the last frame is close to the answer. By 16 rays this effect is mostly gone, and writing at point 6 edits far better than at the input. On standard, swapping in the edited last frame fails outright (-0.09).

What changes. The controls do not match the IM, but they sharpen what it does, so we reframed contribution (3) around this. Contribution (3), Sec. 4, and Sec. 5 now describe where edits succeed in each model. In Othello the model compresses its history into a board at the middle layers. In Rayworld it computes an observation-space update, then gradually refines a rendering of the next frame, which the IM approximates. The IM still drives the model from a simulator the model may never have learned, but we now see the reasons Rayworld forms no simulator-like state as the more informative result (Sec. 5.1).

AuthorsOpenReview ↗

Response (3/3): Settings, robustness, and other questions

W2. Settings tuned on the test edits (Q3). Yes, in the submission. We repeated the selection on a separate set of validation edits and scored the reported cases at the chosen settings (App. A.16). The same setting is chosen in 87% of cells, and no Edit Index or Edit Fidelity changes by more than 0.1, so every conclusion stands.

W2. Edits last one step. We agree this is a limitation, and Appendix A.10 shows that an edit persists only when the history agrees with it. The new results suggest why. The Rayworld model builds its prediction from its recent frames, so a write at one position is overridden by the frames that contradict it.

Q4. One-ray edits. The Rayworld benchmarks kept only edits that change at least two rays, which the submission should have stated. We now state it in Sec. 3.4. Keeping the one-ray edits changes no conclusion, and the trends with ray count remain (App. A.16).

Q5. Is the inverse map fit across all positions? Yes. Refitting it at the edit step alone changes its Edit Index by at most 0.12, with one exception (App. A.16). In Rayworld the inverse map R2R^2 rises only modestly on the 128-ray variants and stays below 0.4, while it rises on the coarser ones, to near 1 on 5 rays at the embedding layer. Its edits barely change on any variant except adjacent-noflip, where the refit IM succeeds (when evaluated at that step) although the IM fit across all time steps fails (see Q2). Fitting across positions therefore explains neither the IM's success in Rayworld nor its weak fit on the 128-ray variants.

Q6. Other steering directions. We tested steering along a difference-of-means direction fit on our state-activation pairings, finding the average change in activations per unit change in the continuous state variables. We also tested writing the PI edit at every later residual point, as in GS, and tested the difference of means direction in this way as well. None succeeds on continuous positions in standard Rayworld, with Edit Indices of at most -0.02 (App. A.15). So in Rayworld, the state stays decodable but not steerable across these choices.

Smaller fixes

  • The Discussion now gives the reachable and unreachable results for adjacent-flip directly (+0.78 and +0.61), consistent with Appendix A.4.
  • We now state when an editor succeeds, which is an Edit Index of at least +0.4 with an Edit Fidelity of at least 0.25 (Sec. 3.5), and use it throughout in place of "landing." PI on adjacent-flip (+0.35 at 0.17) does not succeed, while PI on the 5-ray categorical target (+0.51 at 0.30) does.
  • The caption of Fig. 5 now says it shows a subset of variants and editors and points to Appendix A.13, which shows every variant and editor, including GS and standard-noflip.

Thank you again. We hope the new controls and the reframed claims address your concerns, and we are glad to run further checks during the discussion.

Reviewer_LmbsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a research problem of finding a mapping between a generative model's latent state and the simulator state, so that a generative model can be used as a simulator by setting its latent state. There are two methods: train probes to decode the simulator state from the model and steer it, or map directly with an inverse map. They use two environments: Othello, a discrete board game, and Rayworld, a continuous world of moving discs. Their core finding is that mapping success depends strongly on the environment, even when the state is decodable. For Othello, when the game is simplified, the model becomes harder to control by editing its internal representations. For Rayworld, when probes can't steer the state, an inverse map works well. These findings suggest a promising future research direction, which is using generative models as simulators, along with the unique challenges in steerability.

Critical strengths
  1. The research problem is quite interesting and novel. Unlike previous work in world models and generative models, which typically uses either an action-conditioned model or an instruction/language-conditioned model, this work trains the generative model solely from observations and uses probes or inverse mapping to steer the environment's internal state, enabling state editing. The use of different probes and inverse maps is interesting to the model interpretability community and opens up more directions in this field.

  2. Experiments are solid and reveal a counterintuitive but interesting finding, which are especially important. The paper's experiments have two different environments, one is discrete and the other one is continuous. Overall, the experiments and results support the central claims of the paper. I especially like the finding that decodability does not imply editability, and the finding that editability drops when rules are simplified in the Othello environment. The second finding is a bit counterintuitive, but it suggests that harder, more complex rules let the model better understand the environment's internal state, so editability improves.

Critical weaknesses
  1. Both environments are overly synthetic and do not align with real-world physical dynamics. Therefore, it is unclear whether the findings generalize to more realistic environments.
  2. The model is relatively small compared to existing foundation generative models (~25M parameters). Question 1 is related to this weakness.
Questions for authors
  1. What is the reason for not using foundation generative models in experiments like Stable Diffusion 3.5?
2 comments

Thank you for the positive assessment. We are glad you found the research problem interesting and novel, and the core findings on decodability and editability especially important.

On the Othello finding. We agree with your reading, that harder rules let the model better understand the environment's internal state. The revision makes this explicit, with an explanation of what we mean by "simpler" (Sec. 3.1). Sec. 5.1 then grounds the idea in computational mechanics, and we use that theory to predict a change to Rayworld's training that meaningfully increases its editability.

W1. Synthetic environments. Editability can only be scored against a simulator whose state we can set and whose next observation we can compute after the edit. Recorded natural video has no such ground truth on its own. A simulator of realistic video, such as a game engine or renderer, can provide one, and this is the direction of our follow-up work (see Q1). For this paper, we chose environments where every edit has an exact answer, and where we can change one property of the environment while the model, training, and analysis stay fixed. Rayworld is closer to physical scenes, with continuous state, partial observation, occlusion, and rendering, and its variants test several of these factors directly.

W2. Model size. Our models match the 8-layer, 512-wide transformer of Li et al. (2023), the setting where Othello world models were first shown to be editable. Size also does not limit prediction here. Every model closes at least 99% of the gap between a trivial predictor and the Bayes floor (Table 3), so a failed edit cannot be blamed on a weak model. In Rayworld, the model already predicts nearly optimally while bypassing a 2D state, so more scale gives it no reason to represent the parts of the simulator's state that prediction does not need (Sec. 5.1).

Q1 (W2). Why not a foundation model such as Stable Diffusion 3.5? For three reasons. SD3.5 generates images from text and has no world state or dynamics to set, so there is no next-step outcome to score an edit against. Our method also varies the training environment and retrains the same model on each, which is not possible for a pretrained foundation model. And our editors are fit to pairs of latent states and simulator states, which requires the simulator that generated the training data. The natural route to scale is a large video world model trained on scenes from a game engine or other simulator we control, so that every edit again has an exact answer. We have begun this follow-up work, and our editors, metrics, and controls carry over to such models directly.

Thanks for the rebuttal. I maintain my original positive assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions and new experiments

We thank all three reviewers for the careful and constructive reviews. The requested experiments confirmed and sharpened the paper’s main claims, and we believe they made the paper substantially more informative. Revisions are marked in blue in the updated PDF.

How the claims changed

  1. Rayworld: revealing how the model generates. The submission proposed that the Rayworld model never forms a compact state like the Othello board. The controls Reviewer 11fo asked for now show how it generates instead (Sec. 4, Fig. 6, App. A.14). Our hypothesis is that the early layers shift the recent frames in observation space, bypassing any 2D state, and the later layers refine this rendering. In this view, the inverse map (IM) succeeds by approximating the partly formed latent rendering.
    • Splicing edited frames into the observation stream needs three ground-truth frames to match the IM. A learned renderer with the IM’s size and training data can't match IM edits even with 10 frames spliced in, so the IM is not just editing the model's inputs.
    • Passing the latent at each residual point through the decoder shows an approximation of the next frame by residual point 3 and most of the final rendering by point 5, which keeps improving afterward. The IM edits well by point 3, and plateaus around point 5. Patching in the exact latent states from a counterfactual run follows a similar profile.
    • In contrast, in standard Othello every editor peaks at the middle residual points and fails after them.
  2. Othello: clarifying simplicity and how it affects editability. We now explain what we mean by “simpler,” and show that each simplification lowers editability, replacing our claim that editability “falls smoothly.” We agree with Reviewer Lmbs that simpler rules demand less of prediction, so the model has less reason to form an editable state. Each rule change makes the game simpler to play. Adjacent placement replaces a complex search (runs of the opponent’s tokens along lines across the board) with a local color check. Removing flips means a token never changes once placed. standard-noflip keeps the standard placement rule, but its fixed checkerboard gives the simplest board to track with legal moves easy to calculate.
    • With the standard rules, every editor succeeds on any board, using one probe or map fit across all time steps. In adjacent-flip, only the IM succeeds.
    • In adjacent-noflip, a single probe still decodes the board at every step, but editing it is brittle in ways the others are not. Probe steering fails, and the IM edits single flips only when it is fit at a single time step (App. A.16). This came out of Reviewer 11fo’s question about fitting at the edit position. It also shows that unreachable boards are not the barrier here, since the step-specific IM edits them.
    • In standard-noflip, each cell determines its color, so all edits fail, as expected.
  3. A unifying explanation that predicts a fix. The general principle that Reviewer rfow asked for is that a model acts as a simulator when prediction pressures it to compress its history into the simulator’s state. We originally linked this to computational mechanics (Shalizi & Crutchfield, 2001). We now make that connection more precise (Sec. 5.1) and demonstrate how it predicts a fix for Rayworld that meaningfully increases editability on the 8-ray model, with a smaller gain on 128 rays (App. A.17).
    • Since the observations are generated from the simulator in each variant, the simulator’s state is always a sufficient statistic of the observations. Therefore, the question is about the minimal sufficient statistic (the causal state), and how coarse it is relative to the simulator’s state.
    • In Othello we hypothesize that more complex game rules leave fewer representations simpler than the board that predict equally well, so the causal state is less coarse. Thus, rule complexity refines the minimal sufficient statistic toward the simulator, which the model is decreasingly pressured to learn as rules are simplified.
    • In Rayworld we hypothesize that the causal state is coarser than the simulator’s state. In plain wording, this is what we mean by a shortcut in the predictive task. Since the model already predicts nearly optimally, more scale gives it no reason to represent distinctions in our simulator that the causal state discards.
    • In a new experiment, we make the model commit to a latent state before it knows the next viewpoint, and render the next frame from that state alone. A state that must serve every viewpoint can only be finer than (or equal to) the causal state for one fixed viewpoint, and so closer to the simulator’s state. If the rendering step can instead read the whole history, the gain disappears, as expected from the theory. We consider this a promising recipe for improving the simulator behavior of generative models.
AuthorsOpenReview ↗

Other changes

  • In standard Rayworld, probe steering on continuous positions still fails along encoding and difference-of-means directions, and at every later layer (App. A.15).
  • Choosing settings on held-out edits, or keeping one-ray Rayworld edits, changes no conclusion (App. A.16).
  • Decodability is now summarized in Table 1, and we state when an editor succeeds (Sec. 3.5).
  • We clarified the positioning against MetaOthello (Sec. 2), and rewrote the Discussion and Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.