- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
The authors apply an interesting idea from neuroscience of noise ceiling estimation to study scale-related trends of representational alignment. They treat independently trained seeds of identical training recipes as repeated measurements. Using this, they show that for mutual -NN and linear CKA, the expected alignment between two recipes can be broken down into a consensus and reliability term, akin to how one defines noise ceilings in e.g., fMRI recordings. They test this on PolyPythia and DataDecide/OLMo against fixed vision models and find that approximately half of the increase in language-vision alignment with scale comes from inter-seed consistency in the same model. Last, they examine training checkpoints, run simulations with known ground truth and test whether raw or corrected alignment better predicts held-out retrieval.
Critical strengths
The paper asks an interesting question, in light of increasing interest in the Platonic representation hypothesis, which posits that scale drives convergence. It is great that people are trying to question where this convergence comes from and whether alignment metrics which are commonplace in the community have confounds.
- Using simulations is always great, since we know the exact ground-truth structure and allows one to test where a method breaks down. The authors use this to demonstrate that an exact identity does not guarantee good corrected noise ceiling estimates with only a handful of model seeds.
- I like that the authors explore an alternative explanation for rising alignment curves -- that seeds of the same recipe become more consistent with one another, rather than models converging to a shared representation through scale alone.
- The math for splitting alignment into consensus and reliability terms, and chance-level floors of -NN and linear CKA is fairly straightforward and looks correct.
Critical weaknesses
- My biggest concern is that a lot of reliability appears even before the beginning of training. In Table 5, Pythia M for example already has a reliability of , which increases to at the end of training. This means of the final reliability is already present before any training takes place. In this vein, reliability does increase over training, but only modestly, and for both Pythia models it peaks mid-training. Another caveat here is that while the authors describe " recipes", they differ in much more than scale (e.g., training data, training length). Another thing to note here is that across recipes, at step , the larger Pythia model is already more reliable (M ; M ). This would suggest that part of the increase in reliability can be attributed to scale that is intrinsically present in untrained models, rather than to training being more reproducible at scale. There are a few ways I can think of baseline-ing against step (beyond just looking at raw values at the beginning and end of training) to separate learned reliability from reliability present at initialization:
(i) Percentage increase from step , although this risks looking exaggerated when starting from lower reliability, as in the smaller Pythia model.
(ii) The share of headroom gained, i.e., how much of the remaining gap to "perfect reliability" is covered by training. This would handle the ceiling at and mirror the paper’s floor calibration . Doing so would calibrate reliability to an "initialization floor" over and above the chance floor.
(iii) from the log decomposition in Eq. 3, since it would directly give how much of the reliability contribution is learned.
-
I am not sure that "roughly half" holds across the scale range. The figure comes from fitting a single line to log alignment across all Pythia sizes. However, the authors’ own exploratory split at M (L279–282) suggests two different regimes. Below M, reliability accounts for of the slope, and the corrected alignment scores presented in Table 4 barely change. That is, almost all of the increase in raw alignment at small scales likely comes from seeds being more consistent with each other. Above M, reliability contributes only of the slope while corrected alignment rises clearly , suggesting that most of the increase at larger scales does indeed come from representational convergence, not from seeds becoming more consistent. There is also a drop in reliability between M and M . If this is the case, the result supports PRH more at the larger end of the model scale spectrum than implied by the authors. Relatedly, since reliability cannot exceed one, its contribution is bounded as models scale. Take for instance the M model, where reliability is . Here, reliability alone can increase raw alignment by at most a factor of , which makes it hard to see how reliability could explain a trend that is central to PRH. I think the authors should discuss this separately for small and large models, rather than as a single number.
-
The reliability share also depends strongly on the choice of metric and readout. For example, the OLMo replication is sensitive to the choice of readout. With independently selected layers, the OLMo share closely matches Pythia, but at fixed final layers the two diverge substantially ( and ), even though the Pythia share does not seem to be affected by the choice of readout (Table 1). Do the authors have insights on why this could be the case? Also, the abstract only reports the pairing where the two lineages agree. I would suggest reporting both readouts in the abstract, or at least qualifying that “roughly half” is specific to mutual -NN and selected-layer readouts, for a more complete account of the presented results.
-
The paper unfortunately reads very terse and hard to follow. Most results are followed by disclaimers, some of which are about claims the paper never makes. For example, the main scaling result is followed by "This is evidence that both factors matter in the measured suite, rather than a population significance test" (L271-272). Also, the training-checkpoint section ends with "these five-checkpoint paths establish neither a universal learning phase nor an information bottleneck" (L321-322), and the M split makes the following note: "does not identify a universal transition" (L282). However, no claims of an information bottleneck or universal learning phase or transition are proposed anywhere in the main text. I would suggest stating each finding succinctly, and perhaps adding caveats as an addendum to the existing discussion section, such that a reader can access all such relevant details in a single place.
Questions for authors
-
At L102, is used as a random variable over training runs, but Lemma 1 only defines for a fixed model. Is this the same object?
-
At L106, what does "fresh training randomness" vary exactly? Is it initialization, data order, or both, and is this the same for PolyPythia and DataDecide? Also, are "reproducibility" in the introduction and "reliability" in Section 2 the same quantity?
-
I could only find "record" and "reference" (L165–169) defined in the appendix. What do they mean here?
-
How does the decomposition change if layers are selected by corrected rather than raw alignment?
-
Does corrected alignment predict retrieval for seed-averaged representations? I would expect raw alignment to win for individual models.