acceptodds
Under review as a conference paper at ICLR 2027

Model Space and Features: Analysis of Internal Representations in Language Models

Abstract

How do model resources shape internal representations, and how does their organization relate to performance? We study feature geometry and superposition in decoder-only Transformers trained on random legal move sequences from Othello and Chess, using rule-defined board-state properties as a common reference. Across 128 Dense and Mixture-of-Experts models varying in width, depth, and FFN structure, increasing width primarily separates already represented state directions, whereas order-dependent state properties depend more on depth and FFN capacity. The relationship between summed feature dimensionality and performance differs across tasks: Chess source-square prediction closely tracks this measure, whereas deeper Othello models improve candidate rankings at similar values. Interventions along estimated feature directions causally change the corresponding output probabilities. Excess loss is approximately log-linear in summed feature dimensionality, and the fitted relationship transfers to FFN structures excluded from fitting. These findings characterize how feature geometry and performance vary with model resources, and locate the performance differences that the measured dimensionality does not account for.

Primary area:
interpretability and explainable AI
Keywords:
Language Model, Feature, Superposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t6brOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks what happens to a model‘s internal representations when you make it wider, deeper, or change its feed-forward layer (dense vs. MoE), and whether those changes explain why it performs better. The authors train 128 small Transformers on random legal move sequences from Othello and Chess. Board games are used because the “true” board state is known, which gives 128 yes/no properties per board (whether each square holds the player’s own piece or the opponent‘s). For each property, they use regression to estimate the direction in the model’s activation space that encodes it. They then compute the summed feature dimensionality, r, and relate it to performance.

The paper has five main findings. First, width mostly spreads existing features apart rather than adding new ones. Second, depth and FFN capacity matter for information that must be computed from move order. Third, r explains performance differently in the two games. Fourth, the estimated directions causally affect the model’s outputs. Fifth, excess loss falls roughly linearly in log r.

Critical strengths
  1. The experiment design is clean and controlled. The authors vary width, depth, and FFN structure as a full grid, and they use board properties as a ground-truth reference for comparing all models. Including dense models matched to the MoE models on total and active parameters is a nice control.

  2. The analysis goes beyond a single performance number. Splitting errors into state errors (wrong idea of the board) and rule errors (right board but an illegal move) shows where depth actually helps in Othello. The authors also report failed predictions and limitations honestly, which makes the results easier to trust.

Critical weaknesses
  1. The causal evidence could be stronger. The main intervention adds a direction after the final block, right before the output layer. At that point, the change in output is mostly a linear readout, so a large effect is expected. The appendix notes that injecting directions at middle or early blocks has almost no effect. I think that is the more informative test, because there the model still has to compute with the direction. It would help if the authors could discuss this result in the main text, and if possible, try interventions at middle blocks with a scaled magnitude to show that the model uses these directions in its computation.

  2. The evidence for the empirical law is limited. The fit uses four widths (8 to 64), one seed per setting, and models that may not be fully converged. Also, in Othello, log width explains loss slightly better than log r (0.97 vs. 0.94), so r only adds clear value in Chess. I suggest the authors either soften the “empirical law” framing or add more seeds and a wider range of widths to support it.

  3. The link between r and performance may be explained by training quality. Models that train better may have both cleaner representations and lower loss, so the two could move together without r explaining the loss. It would be helpful to track r and loss across training checkpoints of the same model, or to compare models with similar loss but different r. This would make the claim much more convincing.

  4. The novelty over prior work could be made clearer. Linear board representations in Othello and Chess models are well studied. The feature dimensionality measure comes from Elhage et al. (2022), and links between superposition and loss have also been studied (Liu et al., 2025). I think the systematic sweep across width, depth, and FFN structure is useful, but the paper would benefit from stating more clearly what is new, and what “Model Space” adds beyond the activation distribution itself.

  5. The introduction is long and abstract before reaching the actual question. Several key details live only in the appendix: training steps, the fact that intervention at middle layers fails, and that the pre-registered predictions came from earlier model sets. It would be better to make the research question more straightforward and the method details more accessible throughout the main text.

References:

Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., & Olah, C. (2022). Toy models of superposition (arXiv:2209.10652). arXiv. https://doi.org/10.48550/arXiv.2209.10652

Liu, Y., Liu, Z., & Gore, J. (2025). Superposition yields robust neural scaling (arXiv:2505.10465). arXiv. https://doi.org/10.48550/arXiv.2505.10465

Questions for authors
  1. How does r relate to loss within a single model across training checkpoints? This would help separate r from overall training quality.

  2. Can you show causal effects from interventions at middle layers, where the model still has to compute with the direction?

  3. How much do the main fits change across seeds, given that each grid cell uses only seed 0?

2 comments

Thank you for your careful reading and constructive feedback. Your questions motivated the training-trajectory checks and helped us sharpen the presentation of our contribution and intervention results.

Q1

Loss decreased even when r decreased within a model. Between 25k and 50k steps, loss fell in all 12 runs, while r fell by 0.028–0.335 in four of six Othello runs and by 0.743–1.524 in two of six Chess runs. This limits the claim that training improves r and loss together, but does not isolate training quality’s contribution to the relationship across final models.

Q2

Early- and middle-layer interventions had almost no output effect at the magnitudes used (Appendix A). Direction norms were small, but without controlling intervention magnitude we cannot attribute the weak effects to those norms. We will report these results and magnitudes in the main text, and describe final-layer interventions as effects on the corresponding outputs.

Q3

Held-out FFN evaluation in existing four-layer Chess models yielded R² = .935/.954/.973 across three seeds, supporting repeatability over four widths and four FFN structures. Seed variation in fits across the entire main grid remains unmeasured. We will use “empirical relationship” for the r–loss relationship.

Additional responses to critical weaknesses

The novelty over prior work could be made clearer. Linear board representations in Othello and Chess models are well studied. The feature dimensionality measure comes from Elhage et al. (2022), and links between superposition and loss have also been studied (Liu et al., 2025). I think the systematic sweep across width, depth, and FFN structure is useful, but the paper would benefit from stating more clearly what is new, and what “Model Space” adds beyond the activation distribution itself.

Our contribution is to distinguish representation changes across resources and clarify how measured geometry relates to performance. Increasing width was primarily associated with greater separation of existing state directions, while the representation of order-dependent state also depended on depth and FFN capacity. In Chess, r closely corresponded to performance; in Othello, differences in candidate rankings across depths remained even at similar values of r. Model Space organizes these comparisons and interpretations around common state properties.

The introduction is long and abstract before reaching the actual question. Several key details live only in the appendix: training steps, the fact that intervention at middle layers fails, and that the pre-registered predictions came from earlier model sets. It would be better to make the research question more straightforward and the method details more accessible throughout the main text.

We will state the research question early in the introduction: “How do model resources change state representations, and how do these changes relate to candidate selection and performance?” We will report the 50,000 training steps, single seed per main-grid configuration, early- and middle-layer intervention results, and pre-registered tests on earlier model sets in the main text, making each result’s conditions directly accessible.

We believe all your questions have been fully resolved through our additional explanations. If you agree, we earnestly hope you will reconsider and raise the score of our paper.

Reviewer_gmuhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how width, depth, and FFN structure affect what Transformers represent internally and how this relates to performance. Using models trained on Othello and Chess move sequences, the authors find that wider models mainly separate features they already represent, while features that depend on move history rely more on depth and FFN capacity. They also find that feature dimensionality explains some of the performance differences.

Critical strengths
  1. The finding that width mainly separates existing features, while depth and FFN capacity help with history-dependent properties, gives a useful distinction between their roles.
  2. The relationship between feature dimensionality and loss also works on FFN structures that were not used to fit it, making the finding more convincing.
Critical weaknesses
  1. The findings come from two board games and a limited range of model sizes. It is unclear whether the same patterns would hold for natural language or tasks requiring planning. This is my main concern.

  2. The dimensionality measure may miss useful information. For example, much of the improvement with depth comes from learning movement rules. Does the measure capture this knowledge, or mainly board-state information? How sensitive are the findings to the features included?

  3. The interventions show that feature directions affect predictions, but do not explain why some models perform better than others. Can the authors test whether changes in these features account for the performance differences?

  4. Could differences in training settings or compute budgets partly explain the results across width, depth, and FFN structure?

Questions for authors

See the weaknesses!

2 comments

Thank you for your thoughtful assessment of our experimental design and findings. Your comments helped us clarify the scope of the results and distinguish the evidence from model comparisons and interventions.

Q1

Generalization to natural language or strategic planning remains untested. Board games provide rule-defined state properties for comparing representations and performance across resource settings within a task. The differing geometry–performance relationships show why extensions must account for each task’s state requirements and prediction stages.

Q2

r measures board-state direction geometry; we assess rule-related behavior through error decomposition and rankings of legal and illegal candidates. Together, these measurements distinguish performance differences associated with state geometry from candidate selection requiring further examination. Encoding affects r: one Chess configuration yielded 3.0 or 6.7 depending on the encoding (Appendix E).

Q3

We have not tested whether feature changes causally mediate performance differences across models. Figure 2b shows that changing a final-layer direction changes the corresponding output probability. Cross-model associations among geometry, candidate rankings, and performance are comparative evidence, distinct from this intervention effect.

Q4

Compute and convergence differences may contribute to the results. We controlled task data and the 50,000-step learning-rate schedule, and included Dense models matched to MoE models on total or active parameters. Equal steps guarantee neither equal FLOPs nor equal convergence. The results compare resources, representations, and performance under this shared schedule.

We believe all your questions have been fully resolved through our additional explanations. If you agree, we earnestly hope you will reconsider and raise the score of our paper.

Reviewer_ixQbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors train 128 small decoder-only Transformers on random legal Othello and Chess games and study how board-state features are arranged in the residual stream. The models span widths 8-64, depths 1-8, and dense and MoE FFNs, with one seed per configuration. Board-state directions are estimated by ridge regression, and their geometry is summarized by r, the summed feature dimensionality of Elhage et al. (2022). The main claims are:

  • Width mostly separates features that are already represented rather than adding new ones.
  • Order-dependent state (ownership in Othello, own-piece positions in Chess) needs depth and FFN capacity.
  • Candidate-ranking AUC tracks r in Chess but separates by depth in Othello.
  • Adding a feature direction changes output log-odds more than a norm-matched random direction does.
  • Excess loss fits E = a - b log r with R^2 = 0.94, and the fit transfers to held-out FFN types.
Critical strengths

Design

Using rule-defined board state as a shared reference across 128 models makes feature geometry comparable across architectures. The decomposition into state errors and rule errors (App. C) is a useful bridge from geometry to behavior.

Transparency:

The failed pre-registered predictions are reported alongside the successful ones.

Critical weaknesses

Some key technical limitations and suggested improvements:

  1. The r-loss relationship may reflect model quality rather than explain it. Both r and excess loss are measured on the trained model, and both should improve as the model gets better. A strong correlation across a grid that spans an 8x range in width is therefore somewhat expected, and it can't be used before training. When the paper compares against simple size measures (Sec. 4.3), log d alone gives R^2 = 0.97 in Othello, slightly above log r at 0.94. In Chess, r beats parameter count (0.94 vs 0.78), but this rests on 32 points with one seed each. As far as I can tell, no seed variance is reported for the main grid, so it is hard to judge how much of the held-out fit exceeds noise. Please let me know if I'm missing it. Couple of suggestions:

    • Run 3-5 seeds per configuration and report the seed variance.
    • Run a joint regression of loss on log d, log L, parameter count and FFN type, to show whether r adds explanatory power.
    • Find one prediction that r makes and model size does not, either early in training or outside the grid.
  2. Several findings are expected or overlap with prior work, and the experiments are narrower than the title suggests. Width separating existing features, order-dependent state needing depth, and steering along a probe direction shifting the matching logit are all consistent with Li et al. (2023) and Nanda et al. (2023). The models are small (width at most 64) and trained on random-play board-game transcripts, so "Language Models" in the title is broader than the evidence covers. The MoE conclusions also rest on top-1-of-4 routers at small widths, and App. A notes that r depends on the ridge penalty and feature encoding. Couple of suggestions:

    • Clarify what the "Model Space" framing (Eq. 2) adds beyond the standard residual-stream view.
    • Either narrow the title or add an experiment at larger scale or on natural language.
    • Move the sensitivity of r to the ridge penalty and encoding into the main text.

I think the experimental framework is a good start. The paper needs a clearer case that r explains something that model size does not.

Questions for authors
  1. How much do r and excess loss change across random seeds, compared with how much they change across configurations? The main grid has one seed per configuration, so I can't tell how much of the R^2 = 0.94 fit is above noise. Even 3 seeds per configuration would matter most for my assessment.

  2. Does log r still explain excess loss once width, depth, parameter count and FFN type are all in the regression? In Othello, log d alone already gives R^2 = 0.97, so a joint regression is needed to show that r adds information in both games.

  3. Can r predict anything that model size cannot? For example, does r measured early in training predict final loss, or does the fit extrapolate to a configuration outside the training grid?

2 comments

Thank you for your detailed review and concrete suggestions. Your questions helped us clarify what r adds beyond model size and strengthen the evidence for repeatability across seeds.

Q1

Across the 16 selected configurations, between-configuration differences generally exceeded seed variation. Median within-configuration SDs of r were .4886/.1808 for Othello/Chess, versus 25.8548/13.9111 across configuration means. For excess loss, the corresponding values were .0051/.0216 and .5361/.6927. Separately, held-out FFN evaluation in four-layer Chess models yielded R² = .935/.954/.973 across seeds.

Q2

In-sample, r added more explanatory power in Chess after controlling for resources. With log d, log L, log total parameter count, and FFN type included, adding log r increased R² from .852 to .958 in Chess and from .9817 to .9832 in Othello. In cross-validation across seeds, overall RMSE decreased only in Chess, with improvements on two of the three evaluation seeds.

Q3

In Chess, r measured at the end of training provided additional information for predicting loss beyond the resource baseline. RMSE on held-out seeds decreased from .08139 to .07606, with improvements on two of the three evaluation seeds. In Othello, RMSE increased from .01653 to .02004. Thus, r summarizes learned state geometry and adds task-dependent predictive information beyond resources. These checks did not test whether early r predicts final loss or outperforms size baselines outside the training grid.

Additional responses to critical weaknesses

Clarify what the “Model Space” framing (Eq. 2) adds beyond the standard residual-stream view.

Equation 2 defines the conditional representation distribution being compared. The contribution lies in linking it to common state properties and behavior across models. Figure 1 links width to direction separation, and depth and FFN capacity to order-dependent state representation. In Figures 2–3, r closely tracks Chess performance; Othello candidate rankings differ across depths even at similar r.

Either narrow the title or add an experiment at larger scale or on natural language.

We will specify board-game move prediction in the title and abstract. Our experiments cover Transformers with widths 8–64 and depths 1–8 in Othello and Chess, comparing Dense models with top-1-of-4 Learned/Hash MoE models.

Move the sensitivity of r to the ridge penalty and encoding into the main text.

We will present the measurement settings and encoding sensitivity in the main text. In one Chess configuration in Appendix E, r was 3.0 or 6.7 depending on the encoding. This shows sensitivity of r itself, not robustness of the main r–loss relationship to encoding or ridge changes.

Reviewer_ixQbOpenReview ↗

Response to authors

Thank you for the extra seed runs and the joint regression. The seed variation is small compared with the differences across configurations, so that concern is resolved. I found the Chess result interesting: r still explains a fair amount of the loss after controlling for width, depth and FFN type (R2 increases from 0.852 to 0.958). Narrowing the title to board games and moving the encoding sensitivity into the main paper also seem like good changes.

My remaining reservation is that r adds little or nothing in Othello, and in some runs loss keeps falling while r goes down. The paper should say clearly that the r–loss link beyond model size holds in Chess only. I’m raising my score to 2 (weak reject).

Comments on the paper

1 comment

Common Response

The contribution of Model Space

Using common board-state properties, we compare how representations vary with model resources and relate to candidate selection and performance. The main grid contains 128 models: two games × four FFN structures (Dense-total, Dense-active, Learned, and Hash) × four widths (8, 16, 32, and 64) × four depths (1, 2, 4, and 8).

Figure 1 shows that increasing width is primarily associated with greater separation of existing state directions, while the representation of order-dependent state also depends on depth and FFN capacity. Figures 2–3 show that the measured geometry closely relates to performance in Chess, whereas in Othello, candidate rankings and performance differ across depths even at similar values of r.

Model Space distinguishes resource-dependent changes in representations and their relation to performance. Measured state geometry closely tracks performance differences in Chess; Othello also requires examining candidate selection across depths.

Additional validation

The three checks below use joint regression on the main 128 models, seed replication and training trajectories from 48 additional runs, and repeated fitting and evaluation on existing four-layer Chess models.

Validation 1: Variation across seeds and consistency of the fitted relationship

Variation across seeds in Othello and Chess

We compared within- and between-configuration variation using 48 runs: three seeds for each of 16 configurations covering both games, Dense-total/Hash, widths {8, 64}, and depths {1, 8}. Initialization and data-order seeds both vary. The standard deviations below are newly calculated; loss denotes excess loss.

GameMedian within-configuration SD of rSD of configuration means of rMedian within-configuration SD of lossSD of configuration means of loss
Othello0.488625.85480.00510.5361
Chess0.180813.91110.02160.6927

Within-configuration SD measures variation across three seeds; configuration means average the three runs. The table compares within-configuration seed variation with variation across means for the 16 selected configurations.

Repeatability across seeds on held-out FFN structures

The existing four-layer Chess models cover four widths, four FFN structures, and three seeds, using the same random60 data, square-level tokens, 50,000-step training, and final-layer measurements at move 33 as the main experiments. Fitting on Dense-total/Hash and evaluating on Dense-active/Learned for each seed yielded R² = .935/.954/.973, supporting repeatability under these conditions.

Validation 2: The additional explanatory power of r after controlling for resources

For the 64 models per game in the main experiments, we performed joint regression including log d, log L, log total parameter count, and FFN type. Adding log r increased the in-sample R² from .852 to .958 in Chess and from .9817 to .9832 in Othello.

For the 48 additional runs, we fitted regressions using two seeds and evaluated on the held-out seed. The baseline included log d, log L, FFN type, and (log d)(log L) to account for width–depth interactions. With only two widths and two depths, and identical total parameter counts for Dense-total and Hash at each d and L, log parameter count is a linear combination of these terms and the intercept. We omitted this redundant predictor.

Adding log r changed RMSE from .08139 to .07606 in Chess and from .01653 to .02004 in Othello. Error decreased on two of three held-out seeds in Chess and increased on all three in Othello. Additional predictive value varied by task and seed.

Validation 3: Changes in r and loss during training within the same model

Among the 48 additional runs, we compared the 12 runs with d = 64 and L = 8 at 5k, 10k, 25k, and 50k steps. Between 25k and 50k steps, excess loss decreased by 0.024–0.040 nats in all 12 runs. Meanwhile, r decreased by 0.028–0.335 (0.06–0.66%) in four of the six Othello runs and by 0.743–1.524 (2.10–4.76%) in two of the six Chess runs.

Loss therefore decreased even when r decreased. Changes in r were small in Othello. These observations do not isolate training quality’s contribution to the correlation across final models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.