- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes DATS, a framework to test whether a property of LLM representation geometry comes from data, architecture, or training. The same documents are represented in 3 ways: bag-of-words (data only), an untrained network with same config and tokenizer (data + architecture), and the trained model. A property is credited to the first level where its value is >= half of the trained value. Testing 7 models from 4 families, the paper finds: (1) ordering of 5 English document types by peak intrinsic dimension (ID) is already present in BoW; (2) RSA between Chinese characters and pinyin is already 0.46 in TF-IDF, and the tokenizer decides if untrained network keeps it; (3) only trained models can match a pinyin sentence to its character version (hit@10 is 0.34-0.99 vs 0.10-0.20 for untrained). In Pythia-410M, this identity matching reaches 1.00 by step 4,000 on a mixed/upper case pair.
Critical strengths
-
Important question and simple baseline idea. Using BoW and an untrained twin as default controls, without any retraining, is very useful for anyone who uses geometry as evidence of learning. The paper applies this setup consistently across 4 model families, with a held-out Japanese replication and a base-vs-instruct check.
-
Careful measurement and honest reporting. App. A shows that raw and standardized vectors give opposite conclusions for cross-form alignment, this is a good warning for future alignment work. Also, failed predictions are reported honestly in Table 5.
Critical weaknesses
-
The attribution rule does not support the causal claims. (a) Scalar vs structure: The rule only compares scalar values. For Qwen, RSA goes 0.46 (TF-IDF) -> 0.07-0.14 (untrained) -> 0.53-0.72 (trained). The structure is lost and rebuilt by training, however the rule still credits it to data. Mistral having 0.46 before and after also does not show it keeps the same structure as BoW. (b) Baseline sensitivity: Paper's own char n-gram BoW gives 0.13, then property (2) is not "data" for any model. (c) Data definition: "Data" here only means probe documents, not training corpus, so DATS cannot separate training data from training procedure.
-
The identity result is close to by-design, and the main claim is too strong. (a) Token overlap: Characters and pinyin share no content tokens (only punctuation), and untrained embeddings are random, so untrained cross-form neighbors stay close to chance (hit@10 0.10-0.20 vs 0.05). When tokens are shared (simplified vs traditional, Table 4), untrained hit@10 is already 0.75-1.00. (b) Other training changes: Paper's own results show training changes more than identity. In Pythia the cross-form RSA rises from 0.48 to 0.91 (Sec. 5.3), trained models equalize the pinyin/character ID ratio (Fig. 4B), and for the same Qwen2.5-0.5B the ID response to within-sentence shuffle goes from 74 -> 96 (untrained) to 29 -> 28 (trained) (Table 11). (c) Saturation: In Pythia, identity reaches hit@10 = 1.00 by step 4k of 143k and stays there, so it cannot track most of training. This is in tension with the advice in Sec. 6 that "progress in training should be read from identity, not from shape".
Questions for authors
Clarifications:
-
Can you report within-form RSA between the trained model and BoW, and cross-form RSA after controlling for BoW similarity? This will directly test whether the geometry shape really comes from data.
-
With n=200, running TWO-NN on pure Gaussian noise (Eq. 1 with 10% discard) gives ~381 at H=3584 and ~423 at H=4096 in my simulation. Untrained 7-8B values in the paper are 366-397. Is the ordering in these untrained networks only driven by legal and code documents? Also, how are code files split into 200 sentences?
-
Can you provide confidence intervals with resampling at the book level? Since the 24 Chinese documents only come from 6 books, document-level resampling may not be independent.
Suggestions:
-
Add a token-level BoW built with each model's own tokenizer. This can directly test the claim that "the tokenizer decides".
-
For untrained networks, fit an orthogonal map on half of the sentences and test hit@10 on the other half. This will test whether training only aligns 2 point clouds that already have the same shape.
8 comments
W2(a). Token overlap
- The low untrained value is the control, and the paper presents it as one. Section 4.1 chooses this pair because the two forms "share no content token, so correspondence between their clouds cannot come from shared tokens". Appendix B introduces simplified-traditional as "the Chinese shared-token reference" for the reason you give. The finding is the trained side: with no shared content token, trained models reach 0.34 to 0.99 against a length null of 0.11 to 0.13.
- You are right that punctuation is shared. New: we removed all punctuation and symbols from both forms, which eliminates shared punctuation as an explanation for the identity result (Table R3).
| hit@10, submitted forms | hit@10, no punctuation | length null, no punctuation | |
|---|---|---|---|
| Untrained networks (7) | 0.10 to 0.20 | 0.06 to 0.15 | 0.05 to 0.12 |
| Trained, five larger models | 0.86 to 0.99 | 0.83 to 0.98 | 0.11 to 0.13 |
| Trained Qwen2.5-0.5B | 0.73 | 0.66 | 0.12 |
| Trained Mamba-130M | 0.34 | 0.24 | 0.11 |
Untrained networks fall to their length null. Trained identity does not rest on punctuation.
- The result is not by design, and your S5 test shows why. Whether an untrained network contains the correspondence is an empirical question. With an orthogonal map fitted on 100 sentence pairs, untrained networks recover 0.52 to 0.86 of held-out counterparts (chance 0.10; Table R7 under S5). The untrained networks already contain the cross-form correspondence, although they do not place a sentence next to its counterpart. Training adds identity.
- Planned revision.
- Clarification. Section 4.1 will state that the two forms share punctuation. Finding (3) is unchanged.
- Additional evidence. Table R3 will be added to Appendix F. It supports Finding (3): with punctuation removed, trained identity remains and untrained networks fall to their length null.
W2(b). Other training changes
- These three results are in the paper because the paper makes this point. The abstract, finding (3): "while the shape of the representation keeps changing to the end". Section 5.3, last sentence: "shape is present before training and keeps changing for the rest of it". Appendix H, which contains Table 11, is titled "Boundaries of what training adds".
- What the paper claims. Under the rule of Section 3, a property is attributed to the first representation that shows it. Section 6: identity is "the one property among those we measured that neither the data nor the architecture supplies". A property that is present before training can still change in size during training; the Pythia RSA you cite is 0.48 at step 0. The claim is about which properties first appear with training. It is not a claim that training changes nothing else.
- Planned revision.
- Clarification. The claim is unchanged. So that its scope stays attached when the sentence is quoted alone, "Training adds one thing" in the abstract will read "Training adds one thing that neither the data nor the architecture supplies", which is the wording of Section 6. The abstract and Section 1 will state at first use that "present before training" and "changed by training" are two different judgments. In Section 5.3, "training added identity and nothing else that these measurements see" will become "Training adds cross-form identity, which neither the BoW representation nor any untrained network shows, while the shape of the representation keeps changing during training." Findings (1) to (3) are unchanged.
W2(c). Saturation
- Agree. Early saturation is the finding ("this ability appears early in training"), and a measure that saturates at 3% of training cannot track the rest of it. Finding (3) is unaffected; only the recommendation in Section 6 is withdrawn.
- New: a second trajectory, Pythia-1.4B, with the length null for both (Table R4).
| Step | Pythia-410M hit@10 | length null | Pythia-1.4B hit@10 | length null |
|---|---|---|---|---|
| 0 | 0.38 | 0.07 | 0.42 | 0.08 |
| 1,000 | 0.81 | 0.12 | 0.91 | 0.11 |
| 2,000 | 0.97 | 0.14 | 1.00 | 0.13 |
| 4,000 | 1.00 | 0.14 | 1.00 | 0.13 |
| 143,000 | 1.00 | 0.14 | 1.00 | 0.13 |
(Re-extracted embeddings; step 0 of Pythia-410M is 0.38 here and 0.39 in the submitted text.)
- Planned revision.
- Correction. The sentence "progress in training should be read from identity, not from shape" will be removed from Section 6. This affects one recommendation in the Discussion and no finding.
- Additional evidence. Table R4 will be added. It supports Finding (3): identity reaches 1.00 by step 2,000 to 4,000 in two model sizes.
Q1. Within-form RSA, and cross-form RSA controlling for BoW
(i) Within one form, RSA between the similarity matrix of BoW and that of the untrained (U) and trained (T) network. Table R1; median over middle layers, then documents.
| Model | English BoW-U / BoW-T | Characters BoW-U / BoW-T | Pinyin BoW-U / BoW-T |
|---|---|---|---|
| Qwen2.5-0.5B | 0.09 / 0.32 | 0.05 / 0.25 | 0.10 / 0.29 |
| Qwen2.5-3B | 0.12 / 0.38 | 0.06 / 0.35 | 0.17 / 0.32 |
| Qwen2.5-7B | 0.14 / 0.29 | 0.08 / 0.23 | 0.23 / 0.30 |
| Llama-3.1-8B | 0.31 / 0.42 | 0.21 / 0.37 | 0.24 / 0.40 |
| Mistral-7B | 0.30 / 0.42 | 0.25 / 0.36 | 0.26 / 0.34 |
| Mamba-130M | 0.43 / 0.43 | 0.38 / 0.39 | 0.62 / 0.34 |
| Mamba-2.8B | 0.45 / 0.44 | 0.41 / 0.40 | 0.65 / 0.42 |
- Trained models agree with BoW at 0.23 to 0.44 in every model and form.
- Untrained networks start anywhere from 0.05 (Qwen) to 0.65 (Mamba); after training all seven are at a similar level of agreement with word counts.
(ii) Character-pinyin RSA after regressing out, from both network matrices, the BoW matrices of both forms, then punctuation, then sentence length and token counts. Table R5; partial Spearman on ranks.
| Network | RSA | given BoW | + punctuation | + length, token counts |
|---|---|---|---|---|
| Untrained Qwen2.5-0.5B / 3B / 7B | 0.07 / 0.11 / 0.14 | 0.07 / 0.10 / 0.13 | 0.07 / 0.10 / 0.13 | 0.07 / 0.10 / 0.12 |
| Untrained Llama-3.1-8B | 0.44 | 0.41 | 0.38 | 0.30 |
| Untrained Mistral-7B | 0.46 | 0.43 | 0.40 | 0.36 |
| Untrained Mamba-130M / 2.8B | 0.24 / 0.30 | 0.12 / 0.15 | 0.12 / 0.15 | 0.11 / 0.14 |
| Qwen2.5-0.5B / 3B / 7B | 0.69 / 0.53 / 0.71 | 0.65 / 0.47 / 0.69 | 0.59 / 0.42 / 0.64 | 0.45 / 0.33 / 0.52 |
| Llama-3.1-8B | 0.58 | 0.48 | 0.46 | 0.44 |
| Mistral-7B | 0.46 | 0.39 | 0.36 | 0.31 |
| Mamba-130M / 2.8B | 0.32 / 0.44 | 0.25 / 0.36 | 0.21 / 0.32 | 0.15 / 0.29 |
(Median over layers within each document first; the RSA column differs from Table 10 by at most 0.02.)
- Reading. Controlling for BoW removes half of the cross-form RSA of untrained Mamba and a smaller part elsewhere. A network's cross-form similarity is therefore not a copy of the BoW similarity matrix. The attribution in Section 5.2 does not say that it is: it is by first appearance (Section 3), and it rests on the fact that word counts alone already show cross-form similarity of this size (0.46; 0.76 with matched units).
- Planned revision.
- Additional evidence. Tables R1 and R5 will be added to Appendix F. They answer both parts of this question. Finding (2) is unchanged: it is an attribution by first appearance and rests on the BoW value.
Q2(i). Estimator noise
- Agree, and we reproduce your simulation. With n = 200 and the 10% discard, 20 draws: 378 (sd 25) at H = 3584 and 401 (sd 32) at H = 4096. Your 381 and 423 are within one sd. The last-readout peaks of untrained Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B on narrative and essay (366 to 397) are at this level. The caption of Table 9 notes that these networks "sit near the estimator's ceiling"; we now give the noise level per width (Table R6a). Finding (1) is unaffected: it holds under the mean readout, where every untrained network is far below this level (Q2(ii)).
| Width H | TWO-NN on noise, mean (sd) | Untrained network | Its narrative / essay peak |
|---|---|---|---|
| 768 | 175 (12) | Mamba-130M | 56 / 57 |
| 896 | 180 (14) | Qwen2.5-0.5B | 74 / 74 |
| 2048 | 276 (21) | Qwen2.5-3B | 203 / 208 |
| 2560 | 322 (26) | Mamba-2.8B | 92 / 89 |
| 3584 | 378 (25) | Qwen2.5-7B | 366 / 369 |
| 4096 | 401 (32) | Llama-3.1-8B; Mistral-7B | 397 / 383; 377 / 379 |
We thank the reviewer for the detailed technical questions and concrete experimental suggestions. We ran all of them. The paper's three findings remain unchanged.
Summary
W1(a). Scalar vs structure
W1(b). Baseline sensitivity
W1(c). Data definition