acceptodds
Under review as a conference paper at ICLR 2027

How Does Scaling Reshape Skill Structure in Language Model Representations?

Abstract

Scaling language models consistently improves downstream performance, yet the representation-level changes associated with this improvement remain poorly understood. We study this question through skill structure, using skills as an abstraction of latent capabilities reflected in representations and relevant to downstream prediction. To make skill structure comparable across scales, we develop a task- and template-conditioned analysis based on RSS, with common dimensions and matched random-label calibration. Our theory shows that an increase in the positive calibrated between-skill component raises the guaranteed floor on fitted prototype separation, which in turn can tighten the upper bound on held-out skill-recovery error when prototype-estimation error and within-skill variation are controlled. Across Pythia and Qwen2.5, larger models generally exhibit stronger skill structure. Geometric decomposition shows that this trend is driven primarily by increased separation among skill-conditioned representation centers, accompanied by larger held-out prototype margins. We further test whether this geometry contributes functionally to answer formation. Interventions on skill-aligned representation subspaces produce larger behavioral effects than relative-energy-matched random interventions, supporting functional relevance. The same representation diagnostics provide useful signals for selecting adaptation data under limited training budgets. Together, these results show that scaling is accompanied by stronger skill structure, greater between-skill separation, and more reliable held-out skill recoverability, while the associated skill-aligned subspaces are functionally relevant to answer formation.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Model Scaling, Representation Geometry, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9nLNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language-model scaling changes the representation structure associated with externally annotated skills. The authors introduce a task- and template-conditioned RSS measure with random-label calibration and decompose it into between-skill separation and covariance components. Experiments across Pythia and Qwen2.5 suggest that larger models generally exhibit stronger separation between skill-conditioned representation centers and larger held-out prototype margins. The paper further examines the functional relevance of the identified subspaces through representation interventions and applies the resulting diagnostics to adaptation-data selection under limited training budgets.

Critical strengths

S1: Meaningful Research Problem. The paper studies how model scaling changes internal representation structure, which is a meaningful question for understanding capability formation in large language models.

S2: Preliminary Practical Potential. The representation-guided data-selection experiment provides a useful proof of concept that a simple skill-diagnosis signal can inform model adaptation under a limited training budget, connecting representation analysis with practical training decisions.

Critical weaknesses

W1: Limited Generalizability of the Skill Interpretation. The central analysis depends on a fixed skill taxonomy generated by an external language model. The reported trend may therefore reflect separability under this particular label system rather than a general property of language-model representations. Without validation across independently defined skills, broader domains, or model-derived capability structures, the finding remains narrow and not fully convincing.

W2: Limited Link to Scaling-Related Behavioral Change. The decomposition, held-out, and intervention results show that the identified geometry is non-random and functionally relevant. However, the paper does not show whether its strengthening predicts task performance, generalization, or error patterns. The adaptation evidence is narrow, covering two models and showing only a small gain for Qwen2.5.

Questions for authors

Q1: The study covers models up to 32B parameters. What evidence suggests that the reported trend extends to substantially larger models rather than saturating or changing beyond the evaluated range?

Q2: Does the proposed skill geometry predict performance gains, error patterns, or transfer across templates and model scales beyond what can already be explained by model size?

Reviewer_VJCxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the role of model scale on the separability of skill-level representations in language models. They analyze the activations of language models under different tasks and prompting templates and propose a coding rate reduction-based approach for measuring the extent of skill structure. They additionally measure the strength of representation-level skill structure by evaluating held-out prototype-based skill recovery (as well as the corresponding classification margins), justifying this theoretically. They evaluate Pythia models ranging in scale from 410M to 12B parameters, as well as the Qwen model and show that show that skill structure increases linearly with model scale. They also provide evidence that the skill geometry that they uncover is causally responsible for how the model computes its answers. Finally, they demonstrate that their approach can enable improvements in data selection.

Critical strengths
  • Less Explored Area: I think that understanding how model interpretability and internal geometry shifts across model scales could in principle be interesting and could answer some basic questions about whether we can expect models to become easier/harder to control as they scale up. I think the “skill-based” analysis done in this submission is also an interesting addition to existing work on personas representations/neuron-level analyses/etc. I also think that the authors have made some effort to decouple from prompt surface-form.

  • Potential For Impact:I think that the underlying results explored here also have the potential for impact (though the authors don’t currently examine the most impactful possibilities in my opinion). For example, what are the implications of this geometry on the model’s continual training behavior. Does more separated skill geometry imply less catastrophic forgetting during downstream training? What is the relationship between downstream fine-tuning generalization and the separability of representations? I feel like understanding the scaling behaviors of these properties could leverage some of the analysis tools that the authors have built here but answer some very interesting questions in the community.

  • Rigorous Evaluation of Skill Identifiability: I think that the authors do make efforts to rigorously evaluate the identifiability of skills. They use tools from coding rate reduction and justify the relationships between the quantities that they measure (RSS, Prototype margin, etc). They also control for formats and tasks and do held-out evaluations, as well as performing a randomized control. Traditional representation-based analysis can often have these aspects as confounds and so it is commendable that these authors perform a more rigorous study.

Critical weaknesses
  • Significance of the Result-Novelty: The core contribution: that skill-structured representations emerge with scale is somewhat unsurprising to me. In particular, in larger models with larger dimensionalities and more layers, it seems natural that (a) representations may become more separable simply, similar to the findings of superposition in prior works. Similarly, some latent skills might not even be learned in smaller models and they may correspondingly not show any structure. While the results intuitiveness is certainly not a reason to reject this paper, combined with the unclear practical implications (see below), I have a somewhat harder time understanding the contributions of this paper to the community.

  • Significance of the Result-Applicability: The authors also do not articulate what the practical implications of their findings are. They provide one example of data-selection based off of the representation-level metrics of a skill. However, it is not entirely clear to me why the separability/identifiability of a skill should be necessarily correlated with whether it is implemented in the model. For example, a model may still be able to implement a skill well even if the capability is relatively entangled with other related skills. Aside from this, the authors do not justify other implications of their skill-recovery results. Although this alone is not a reason to reject the paper, in combination with my point (1), it is difficult for me to understand what is the contribution of this work.

  • Limited Evaluation of the Data Selection Application: The evaluation of the data-selection pipeline is relatively narrow, examining only a small number of basic natural language benchmarks: ARC-Challenge, ARC-Easy, and SciQ. These benchmarks are traditionally used for evaluating general language modelling capabilities of base-models and thus do not represent fine-tuning or adaptation settings of typical interest (such as instruction tuning, fine-tuning for reasoning, safety training etc). Thus, the relevance of these benefits are unclear to me. I see no reason why the authors could not extend their evaluation to these slightly more realistic tasks (which in principle should share some skills with what they are looking at) and it would significantly improve the claims of the paper if they would do so.

  • Unclear Details: Many important details needed to understand the paper are not present. For example what exactly are the skill labels generated? The appendix only mentions the following:

We treat skills as an abstraction of latent capability variations reflected in representations and relevant to solving downstream examples. Each example is assigned a single primary skill that captures its main functional requirement. The skill vocabulary is constructed in a separate annotation stage before the cross-scale analysis and is then frozen for all subsequent experiments.

But does not list exact examples of the skills that were used. This makes evaluating the applicability and significance of the results challenging and hence is a reason that I would reject the paper. Similarly,  from the main paper it is relatively difficult to identify what are the exact settings under which the models are being trained or evaluated. While I observe these details, the current prose in the main paper makes it quite challenging to understand what exactly is going on in this evaluation.
  • Readability: The paper is often a bit hard to read (in my opinion). Terms and abbreviations are sometimes not defined prior to their use, for example what does RSS stand for? Similarly, they list one of the their contributions as “Establish the functional relevance of the identified representation… relative-energy-matched interventions”, this terminology is unclear to me. I think this is an important problem in terms of bringing the paper up to a publishable quality.
Questions for authors

Many of these questions will be drawn from existing questions/points I have raised in the previous sections. I would also appreciate if the authors directly respond to any Weaknesses I raise that they don't agree with.

  1. Can the authors comment on the relationship of their result to existing studies on superposition/monosemanticity/representational convergence. The authors have only a perfunctory description of these prior works as the paper stands and don't specifically comment on the relationship of their findings to these existing results. I think that this should definitely be addressed in the rebuttal and revision of the paper.

  2. Can authors comment on whether/if they have any hypotheses on what their results reveal about how models would behave under fine-tuning? My hypothesis is that the structure of the representation could be an important inductive bias over the generalization of fine-tuning, as well as issues such as catastrophic forgetting. For ex. do the authors expect that more seperable skill representations would result in less catastrophic forgetting?

  3. Can the authors provide specific examples of what are "skills" in their setting? Ideally a table of skills studied would be nice.

  4. Could the authors use their proposed skill-centric data selection approach on more realistic fine-tuning tasks?

Reviewer_BXpyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work concentrates on how the model size affects the hidden structure of skills. The skills here are labels for tasks. RSS, as a mathmatical tool, is used to quantify the skill structures. Two series of models are used for experiments.

Theoretical analysis is given to show that RSS is composed of the between-skill part and the covariance part. The between-skill part is theoretical useful for labeling new tasks.

Empirical results show that adjusted RSS is roughly improving with the model size and the between-skill part is the main reason for RSS improvement. Then new label prediction are tested and is still improving with the model size. Finally, skill spaces are interuputed to see the effect on answer accurcies and the skill is also useful for fine-tuning data chosen.

Critical strengths
  1. This work uses RSS as a useful tool to quantify the skill structure and connects geometric measurement, model probing and the data selection.
  2. Theoretical analysis gives more detailed decomposition for RSS and shows that the The between-skill part is theoretical useful for labeling new tasks. Empirical results support that the larger models have better between-skill separation component.
  3. A series of experiments are done and give relatively complete results for the scale effect of skill structure and how to use the skill structure.
Critical weaknesses
  1. The interpretation of the labels as skills is insufficiently validated. It might not be clear whether the labels can represent some general skills or some specific features of some tasks.
  2. The intervention experiments show that intervention can damage the answer. I think it might be helpful to see the specific intervention is useful for specifical skill. It is possible that the intervention is conducted on some general feature, rather than skill-related features.
  3. The mechanism explanation is limited, mainly on the gemometric level. If there can be more analysis for deeper mechanisms on model scale or answer generation, the work can be more interesing.
Questions for authors

See weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.