- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
We are studying a class of problems in which we train a transformer on a sequence of observations of a partially-observable, stateful system, and try to understand the information content and encoding of the latent states of the learned representation. Previous work has shown, in some cases (e.g. basic Othello) that the latent representation is "editable" in the sense that it can be altered to correspond to a different true hidden world state in such a way that future predictions are sensible.
This paper goes farther, studying variations on Othello and another prediction problem, Rayworld, and studying a different way of probing the information content of the latent representation: that is, by learning to map the true hidden world state to the learned latent state, and then "editing" the learned latent to the one corresponding to some different desired world state. This paper asks when will this technique work and how does it compare to pre-existing methods that work by more directly "steering" the latent representation in a desired direction.
The paper asks the question and formulates tests for decodability and editability of the latent state. It the compares different probe types on different versions of the ground-truth state machines.
Critical strengths
This paper asks an interesting question and I learned a lot from reading it. (I am far from an expert in this area, though I was familiar with the basic Othello editability results, so I did have a lot to learn).
It asks questions that are close to the original heart of ICLR: what is in the nature of the representations that latent-variable models learn?
Critical weaknesses
In the end, I don't exactly know what to make of this paper. Although the direct editing method and the results are interesting, I'm not sure whether they shed deep light on the underlying learning or representation processes.
The "morals" didn't come through clearly. In particular the idea that "simplifying an environment's rules can smoothly reduce its editability" is I guess demonstrated in these problems, by example. But what are the principled reasons why this might be true? What makes us believe that this is a general principle?
What makes these two domains and the changes you made to their rules particularly revealing? I'm really looking for some general principles.
Very small points:
- At first I thought this was a typo of some sort: " in Othello editability falls smoothly" but now I guess we mean it decreases. But calling something smooth that's based on 4 discrete changes seems overstating the case (or I misunderstand).
- The writing is harder to understand and more Claude-ish ("IM lands everywhere") toward the end
- Is the transformer width of 512 the dimension of Z?
Questions for authors
What makes these two domains and the changes you made to their rules particularly revealing?
Could you construct a domain in which there is a good encoder/decoder model but for which it would not (for some interesting reason) be able to map a true hidden world state into the model's latent space in a way that would preserve the semantics? Maybe this has to be possible.
5 comments
Response (2/2): Semantic mappings and other notes
Q2. Can a good model fail to admit a semantic mapping? Yes, only once the inverse map is limited in size. An inverse map of unlimited size could technically even rerun the model itself, so a mapping would always exist and would say nothing about the model. We therefore match the inverse map's size to the probes, so it is a shallow MLP (App. A.14). With that limit, the paper already contains a case where the inverse map fails. In adjacent-noflip, a good model decodes the board at every step, yet an inverse map fit across all time steps fails to edit it. Our reading is that its representations never hold the board in one consistent way across time steps, so only a map fit to a single time step succeeds.
By this view, we can predict another setting where the inverse mapping would fail in an informative way. In Rayworld, the inverse map appears to learn something like the rendering itself, since the model works in observation space and rendering is cheap. Now imagine an interactive video world model, where rendering is expensive. If that model also works in observation space, a mapping from the world state into its latent space needs to implement the rendering process, and a small inverse map may fail. If the model works with something like a 3D state, the mapping should be simpler. We have begun follow-up work testing related ideas on a large video world model.
The inverse map as a tool. Beyond editing, the inverse map also lets us see how a model represents and generates its world. Tracking where it succeeds across residual points revealed Othello's board bottleneck and Rayworld's gradual rendering (Fig. 6). Fitting it at a single time step revealed that adjacent-noflip represents the board differently from step to step (App. A.16).
"Smoothly." We agree that "smoothly" does not best capture the Othello editability results. We now explain what we mean by "simpler," and show that each simplification lowers editability monotonically (Sec. 3.1, Sec. 5).
Writing. We rewrote the Discussion and Conclusion for clarity, state when an editor succeeds (Sec. 3.5), and use "the inverse map" in place of "IM" in the revised prose.
Is the width 512 the dimension of ? Each residual point is a 512-dimensional vector, and is comprised of all nine, one per point (Sec. 3.3). In the main results, the editors write one point at a time, except GS, which also writes every later point (following the original OthelloGPT implementation).
As the discussion period closes soon, we wanted to check whether the general principle in the new Sec. 5.1 and the accompanying experiment (App. A.17) address your question about what the results show. We’d be glad to clarify anything. Thank you again for the thoughtful review.
Response (1/2): The general principle and a new test
Thank you for a thoughtful review, and for calling the paper close to the heart of ICLR. Your question about the principles behind our results led us to expand the general theory in a new subsection on forcing simulator-like states (Sec. 5.1), along with a new experiment that tests it.
What is the general principle? A model acts as a simulator when prediction pressures it to compress its history into the simulator's state. The submission originally linked this to computational mechanics (Shalizi & Crutchfield, 2001), and Sec. 5.1 now elaborates on that connection. Since the observations are generated from the simulator, the simulator's state is always a sufficient statistic of the observations. The question is about the minimal sufficient statistic (the causal state), and how much coarser it is than the simulator's state.
The principle also predicts a fix. One way to picture this is to think of the causal state as a bag wrapped loosely around the simulator’s state. Anything that refines the causal state draws the air out, so the sheet ends up conforming more closely to the simulator’s state, meaning the model must represent more of it (App. A.17). In a new experiment, we make the Rayworld model commit to a latent state before it knows the next viewpoint, and then render the next observation from that state alone. A state that must serve every viewpoint can only be finer than (or equal to) the causal state for one fixed viewpoint, and thus closer to the simulator's state. This meaningfully increases editability on the 8-ray model, raising the Edit Index of PI and GS to +0.55 and +0.42 (from +0.21 and -0.07 before), with a smaller gain on 128 rays (App. A.17). If the rendering step can instead read the whole history, the gain disappears, as expected from the theory. We consider this a promising recipe for improving the simulator behavior of generative models.
Q1. What makes these two domains revealing? They sit at opposite ends of our principle. Othello's rules make the board both necessary for prediction and complexly related to the moves, and it is the setting where editable world models were first shown (Li et al., 2023). Rayworld is continuous and partially observed (closer to video) but its frames reveal a lot of its state, which leaves room for a shortcut. The variants then simplify the rules in Othello and change what the observation reveals in Rayworld, while the model and analysis stay fixed. Such an analysis helps to reveal the principles governing the representations that a model learns based on the observations and training environment it is exposed to.