acceptodds
Withdrawn from ICLR 2027

Linking Representations to Function across Neural Network Populations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How do differences in internal representations relate to differences in what intelligent systems can do? The growing diversity of trained neural networks creates an opportunity to address this question by treating representations as a data modality with population-level structure of its own. We introduce a metamodeling framework that learns relationships between representational structure and behavior across systems, using representational measurements as predictive markers analogous to biomarkers in medicine. To compare representations across models with different dimensionalities and coordinate systems, we use symmetry-aware descriptions based on invariant geometric summaries and alignment to shared spaces. Across vision and language, representations elicited by shared stimuli predict variation in visual recognition, robustness to distribution shifts, and linguistic capabilities. These relationships generalize to held-out architecture families in vision and model scales in language, revealing regularities in the mapping from representational organization to function. Our analyses further show that accessing this predictive information depends on how representations are measured: natural images provide more informative probes of visual performance, while linguistic predictions depend on the match between probe stimuli and the capability being assessed. Together, these findings establish internal representations as transferable markers of model capabilities and provide a foundation for a comparative science linking representational structure to function across intelligent systems.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representational geometry, structure--function, meta-modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Py6yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether representations measured on a small shared set of inputs can predict performance across neural networks. The authors compare three representation summaries and use regression to predict behavioral scores, with experiments on vision models, language model checkpoints, and mice. The main results concern generalization to held-out architectures or model sizes, and how prediction depends on the inputs used to measure representations.

Critical strengths

I find the vision experiment interesting. Training on CNNs and predicting the accuracy of held-out non-CNN models is a meaningful test, and the reported performance using only 200 images is positive. The experiments varying stimulus number and content are also useful. In particular, the difference between natural images and noise suggests that the choice of probe inputs matters.

Critical weaknesses
  1. I think the paper needs stronger baselines to establish what the representation measurements add. For vision, how well would model size and training information predict accuracy? For OOD performance, I would particularly like to see a comparison against ImageNet accuracy alone. For language, model size, training step, and validation loss seem important. The checkpoints come from nine sizes of the same Pythia models, so I am unsure how much of the prediction reflects shared training progress (there could be confounders). In Figure 4, some subject–verb agreement predictions have reasonably high R^2 but almost zero or negative rank correlation. This makes me wonder whether they mainly capture the initial improvement and subsequent plateau. The finding in Appendix C.6 that nearest neighbors often outperform ridge makes this question more relevant.

  2. I am also unsure how much methodological novelty to attribute to the framework. Williams et al. (2021) already used regression on representation embeddings to predict network test accuracy. The authors cite this work, but I would appreciate a more explicit account of what the proposed approach adds. I think the contribution here may lie mainly in the broader experiments and transfer settings. If so, comparisons against prior approaches would help establish the significance of those results.

Questions for authors
  1. Does representation geometry improve prediction beyond model size and training progress, and beyond ImageNet accuracy when predicting OOD performance? For the language experiments, does useful predictive performance remain when the earliest checkpoints are excluded?

  2. What is the main advance over the representation-based performance prediction in Williams et al. (2021)? Can the authors compare against that approach under the same architecture or scale splits?

Reviewer_VNRyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether differences in internal representations can predict differences in model capabilities across neural network populations. The authors propose a population-level meta-modeling framework that uses symmetry-aware representation descriptors, including RDM-based intrinsic summaries, CKA-based anchor-relative summaries, and Procrustes-based canonicalization, to predict model performance through ridge regression.

Experiments on vision models, language models, and biological neural recordings suggest that representational geometry contains information predictive of recognition accuracy, out-of-distribution robustness, linguistic capabilities, and behavioral sensitivity. The authors further examine how prediction depends on stimulus selection, representation descriptors, and generalization across model architectures or scales.

Critical strengths
  1. Broad empirical evaluation. The paper evaluates the proposed framework across vision models, language models, and biological neural recordings. The cross-architecture prediction results in vision are particularly encouraging, suggesting that representational measurements can provide useful information about model performance beyond the training population.

  2. Interesting observations about stimulus selection. The experiments demonstrate that predictive performance depends substantially on the choice of stimuli. In particular, structured natural inputs tend to be more informative than random or unstructured inputs, and different behavioral capabilities may require different probing stimuli. These observations could be useful for future studies of representation analysis.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from prior work. The proposed framework primarily combines established techniques, including RDM, CKA, Procrustes alignment, and ridge regression. Moreover, previous studies have already explored relationships between representational geometry and model performance or generalization. While the population-level evaluation is useful, I am not fully convinced that the framework introduces a sufficiently distinct methodological or conceptual contribution beyond existing representation-based prediction approaches. I would encourage the authors to clarify what fundamentally new insights or capabilities their framework provides compared with prior work.

  2. Limited scientific insight beyond statistical prediction. Although the experiments demonstrate correlations between representational geometry and model performance, it remains unclear what these relationships reveal about the underlying mechanisms connecting representations to function. In particular, model scale, training progress, and other shared factors may influence both representational structure and behavioral performance. Without stronger controls, it is difficult to determine whether the proposed descriptors capture functionally meaningful properties or primarily serve as proxies for overall model quality. Furthermore, the empirical evidence is somewhat uneven: the language results vary substantially across model scales and tasks, and cross-scale generalization is evaluated only within the Pythia family. These limitations make the broader claims about transferable structure–function relationships less convincing.

Questions for authors
  1. Novelty and scientific contribution. Could the authors clarify what distinguishes the proposed framework from existing approaches that use representational geometry to predict model accuracy or generalization? Beyond demonstrating predictive correlations, what new understanding of representation–function relationships does the framework provide?

  2. Confounding factors and independent predictive information. Could the authors investigate whether representational geometry provides substantial predictive information beyond simpler model-level characteristics, such as parameter count, training progress, or basic activation statistics? Such comparisons would help establish whether the learned relationships reflect meaningful geometric structure rather than general differences in model quality.

  3. Generalization across language model families. Since the language experiments are restricted to different scales and checkpoints within Pythia, could the authors clarify how strongly the current evidence supports generalization across language model families? Additional cross-family evaluation, if feasible, would significantly strengthen the broader claims.

Reviewer_9B6ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a meta-modeling framework that predicts a network's behavior from its representational geometry. Each trained network is one data point. Its responses to a shared stimulus set are summarized with RDMs, CKA similarities, or Procrustes-aligned activations, and ridge regression or kNN then predicts behavioral metrics. The framework is tested on vision models, language-model checkpoints and biological datasets, including held-out architecture families and model scales.

Critical strengths

The scale of the evaluation. The authors test thousands of models and checkpoints across vision, language and biological data, and the held-out architecture and scale splits check that the predictors work beyond the model families they were fit on. This is the main reason to take the empirical results seriously.

Critical weaknesses

1. Without the obvious baselines, the practical value is unclear. The method still needs forward passes on the shared stimuli plus behavioral labels for hundreds of training models. Two comparisons are missing: (i) evaluating the target model directly on the same number of labeled stimuli, and (ii) linear probing. For example, measuring ImageNet accuracy on 200 images gives a standard error of about 2.8 points at 80% accuracy. If accuracy varies across the model population with a standard deviation of 7.5 points or more, that alone gives an R² near the reported 0.86. Without compute-matched baselines it is hard to tell what the framework buys. Prior work on predicting generalization and estimating accuracy without labels (e.g. Jiang et al., 2020; Garg et al., 2022) should also be compared.

2. The main findings have simpler explanations, and the paper does not say which geometric properties matter. The descriptors (RSA, CKA, generalized Procrustes) are standard and Proposition 3.1 is a classical result about Gram matrices, so the contribution has to come from what the predictors reveal. The paper never identifies which properties of the geometry drive the predictions, and several results have more mundane explanations:

  • Ridge beating kNN is read as evidence for learned cross-model structure, but Fig. 12 shows that changing the kNN distance metric alone raises R² from about 0.40 to 0.76.
  • The stimulus-count effects may just reflect high-dimensional regression: at n = 200 the RDM has 19,900 features for 723 training models, so canonicalization getting worse at larger n is not surprising. The stimulus-type ablations may simply reflect distribution shift.
  • The OOD results may be "accuracy on the line" (Taori et al., 2020; Miller et al., 2021). The authors should compare against ImageNet accuracy alone and predict effective robustness after controlling for ID accuracy.

Comparing against interpretable quantities with known links to readout, such as manifold capacity (Chung et al., 2018) or few-shot SNR (Sorscher et al., 2022), would show what the regression adds beyond them.

References: Chung, Lee & Sompolinsky (2018), Phys. Rev. X 8, 031003. Garg et al. (2022), ICLR. Jiang et al. (2020), ICLR. Miller et al. (2021), ICML. Sorscher, Ganguli & Sompolinsky (2022), PNAS 119, e2200800119. Taori et al. (2020), NeurIPS.

Questions for authors
  1. For vision models the representation is the pooled penultimate-layer activation; for language models it is the last non-padding token in the penultimate transformer block. Are these comparable? For language models, does this skip a final block or layernorm, or is it, as in vision, the input to a linear readout? (clarification)
  2. How does the method compare with direct evaluation on the same number of labeled stimuli, and with linear probing, at matched compute?
  3. Does OOD prediction hold up after controlling for ID accuracy, i.e. when predicting effective robustness?

Comments on the paper

1 comment

Dear authors, it looks like you haven't responded to reviewer concerns yet. As the rebuttal phase is nearing its end, this is your last chance to respond - alternatively, I would suggest to consider withdrawing the paper if you don't believe it has a chance of acceptance.

Thank you,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.