- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper asks what happens to a model‘s internal representations when you make it wider, deeper, or change its feed-forward layer (dense vs. MoE), and whether those changes explain why it performs better. The authors train 128 small Transformers on random legal move sequences from Othello and Chess. Board games are used because the “true” board state is known, which gives 128 yes/no properties per board (whether each square holds the player’s own piece or the opponent‘s). For each property, they use regression to estimate the direction in the model’s activation space that encodes it. They then compute the summed feature dimensionality, r, and relate it to performance.
The paper has five main findings. First, width mostly spreads existing features apart rather than adding new ones. Second, depth and FFN capacity matter for information that must be computed from move order. Third, r explains performance differently in the two games. Fourth, the estimated directions causally affect the model’s outputs. Fifth, excess loss falls roughly linearly in log r.
Critical strengths
-
The experiment design is clean and controlled. The authors vary width, depth, and FFN structure as a full grid, and they use board properties as a ground-truth reference for comparing all models. Including dense models matched to the MoE models on total and active parameters is a nice control.
-
The analysis goes beyond a single performance number. Splitting errors into state errors (wrong idea of the board) and rule errors (right board but an illegal move) shows where depth actually helps in Othello. The authors also report failed predictions and limitations honestly, which makes the results easier to trust.
Critical weaknesses
-
The causal evidence could be stronger. The main intervention adds a direction after the final block, right before the output layer. At that point, the change in output is mostly a linear readout, so a large effect is expected. The appendix notes that injecting directions at middle or early blocks has almost no effect. I think that is the more informative test, because there the model still has to compute with the direction. It would help if the authors could discuss this result in the main text, and if possible, try interventions at middle blocks with a scaled magnitude to show that the model uses these directions in its computation.
-
The evidence for the empirical law is limited. The fit uses four widths (8 to 64), one seed per setting, and models that may not be fully converged. Also, in Othello, log width explains loss slightly better than log r (0.97 vs. 0.94), so r only adds clear value in Chess. I suggest the authors either soften the “empirical law” framing or add more seeds and a wider range of widths to support it.
-
The link between r and performance may be explained by training quality. Models that train better may have both cleaner representations and lower loss, so the two could move together without r explaining the loss. It would be helpful to track r and loss across training checkpoints of the same model, or to compare models with similar loss but different r. This would make the claim much more convincing.
-
The novelty over prior work could be made clearer. Linear board representations in Othello and Chess models are well studied. The feature dimensionality measure comes from Elhage et al. (2022), and links between superposition and loss have also been studied (Liu et al., 2025). I think the systematic sweep across width, depth, and FFN structure is useful, but the paper would benefit from stating more clearly what is new, and what “Model Space” adds beyond the activation distribution itself.
-
The introduction is long and abstract before reaching the actual question. Several key details live only in the appendix: training steps, the fact that intervention at middle layers fails, and that the pre-registered predictions came from earlier model sets. It would be better to make the research question more straightforward and the method details more accessible throughout the main text.
References:
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., & Olah, C. (2022). Toy models of superposition (arXiv:2209.10652). arXiv. https://doi.org/10.48550/arXiv.2209.10652
Liu, Y., Liu, Z., & Gore, J. (2025). Superposition yields robust neural scaling (arXiv:2505.10465). arXiv. https://doi.org/10.48550/arXiv.2505.10465
Questions for authors
-
How does r relate to loss within a single model across training checkpoints? This would help separate r from overall training quality.
-
Can you show causal effects from interventions at middle layers, where the model still has to compute with the direction?
-
How much do the main fits change across seeds, given that each grid cell uses only seed 0?
2 comments
We believe all your questions have been fully resolved through our additional explanations. If you agree, we earnestly hope you will reconsider and raise the score of our paper.
Thank you for your careful reading and constructive feedback. Your questions motivated the training-trajectory checks and helped us sharpen the presentation of our contribution and intervention results.
Q1
Loss decreased even when r decreased within a model. Between 25k and 50k steps, loss fell in all 12 runs, while r fell by 0.028–0.335 in four of six Othello runs and by 0.743–1.524 in two of six Chess runs. This limits the claim that training improves r and loss together, but does not isolate training quality’s contribution to the relationship across final models.
Q2
Early- and middle-layer interventions had almost no output effect at the magnitudes used (Appendix A). Direction norms were small, but without controlling intervention magnitude we cannot attribute the weak effects to those norms. We will report these results and magnitudes in the main text, and describe final-layer interventions as effects on the corresponding outputs.
Q3
Held-out FFN evaluation in existing four-layer Chess models yielded R² = .935/.954/.973 across three seeds, supporting repeatability over four widths and four FFN structures. Seed variation in fits across the entire main grid remains unmeasured. We will use “empirical relationship” for the r–loss relationship.
Additional responses to critical weaknesses
Our contribution is to distinguish representation changes across resources and clarify how measured geometry relates to performance. Increasing width was primarily associated with greater separation of existing state directions, while the representation of order-dependent state also depended on depth and FFN capacity. In Chess, r closely corresponded to performance; in Othello, differences in candidate rankings across depths remained even at similar values of r. Model Space organizes these comparisons and interpretations around common state properties.
We will state the research question early in the introduction: “How do model resources change state representations, and how do these changes relate to candidate selection and performance?” We will report the 50,000 training steps, single seed per main-grid configuration, early- and middle-layer intervention results, and pre-registered tests on earlier model sets in the main text, making each result’s conditions directly accessible.