acceptodds
Under review as a conference paper at ICLR 2027

Attributing Representational Geometry to Data, Architecture, and Training

Abstract

Any observable property of a large language model (LLM) stems from three sources: the data it reads, the model architecture (the network and tokenizer), and the training process. We argue that training produces little of the representational structure we can measure in an LLM. Instead, most of that structure comes from the data and the architecture. Training adds one thing: the model's ability to tell what a text says from how it is written, and this ability appears early in training. Existing evaluation methods such as scaling laws and ablations compare trained models with one another and do not separate the contributions of these three sources, so claims about what a model has learned, and comparisons between models based on their representations, may rest on properties that training did not produce. To identify the contributions of these three sources, we propose DATS (Data–Architecture–Training Separation), a framework that measures the same documents in three ways and attributes each property to its source: a property already shown by bag-of-words (BoW) representations is attributed to the data, a property first shown by untrained networks to the architecture, and a property shown only by trained models to training. Applied to the geometry of document representations across four model families, DATS finds that (1) the ordering of document types by intrinsic dimension is already present in the data; (2) the similarity between two ways of writing the same text is already present in the word counts, and the tokenizer decides whether an untrained network keeps it; (3) only trained models place two ways of writing the same text together, and in Pythia they do so within the first few percent of training, while the shape of the representation keeps changing to the end.

Primary area:
interpretability and explainable AI
Keywords:
representation geometry, intrinsic dimension, representational similarity, random-initialization baselines, language models, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RZzDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes DATS, a framework to test whether a property of LLM representation geometry comes from data, architecture, or training. The same documents are represented in 3 ways: bag-of-words (data only), an untrained network with same config and tokenizer (data + architecture), and the trained model. A property is credited to the first level where its value is >= half of the trained value. Testing 7 models from 4 families, the paper finds: (1) ordering of 5 English document types by peak intrinsic dimension (ID) is already present in BoW; (2) RSA between Chinese characters and pinyin is already 0.46 in TF-IDF, and the tokenizer decides if untrained network keeps it; (3) only trained models can match a pinyin sentence to its character version (hit@10 is 0.34-0.99 vs 0.10-0.20 for untrained). In Pythia-410M, this identity matching reaches 1.00 by step 4,000 on a mixed/upper case pair.

Critical strengths
  1. Important question and simple baseline idea. Using BoW and an untrained twin as default controls, without any retraining, is very useful for anyone who uses geometry as evidence of learning. The paper applies this setup consistently across 4 model families, with a held-out Japanese replication and a base-vs-instruct check.

  2. Careful measurement and honest reporting. App. A shows that raw and standardized vectors give opposite conclusions for cross-form alignment, this is a good warning for future alignment work. Also, failed predictions are reported honestly in Table 5.

Critical weaknesses
  1. The attribution rule does not support the causal claims. (a) Scalar vs structure: The rule only compares scalar values. For Qwen, RSA goes 0.46 (TF-IDF) -> 0.07-0.14 (untrained) -> 0.53-0.72 (trained). The structure is lost and rebuilt by training, however the rule still credits it to data. Mistral having 0.46 before and after also does not show it keeps the same structure as BoW. (b) Baseline sensitivity: Paper's own char n-gram BoW gives 0.13, then property (2) is not "data" for any model. (c) Data definition: "Data" here only means probe documents, not training corpus, so DATS cannot separate training data from training procedure.

  2. The identity result is close to by-design, and the main claim is too strong. (a) Token overlap: Characters and pinyin share no content tokens (only punctuation), and untrained embeddings are random, so untrained cross-form neighbors stay close to chance (hit@10 0.10-0.20 vs 0.05). When tokens are shared (simplified vs traditional, Table 4), untrained hit@10 is already 0.75-1.00. (b) Other training changes: Paper's own results show training changes more than identity. In Pythia the cross-form RSA rises from 0.48 to 0.91 (Sec. 5.3), trained models equalize the pinyin/character ID ratio (Fig. 4B), and for the same Qwen2.5-0.5B the ID response to within-sentence shuffle goes from 74 -> 96 (untrained) to 29 -> 28 (trained) (Table 11). (c) Saturation: In Pythia, identity reaches hit@10 = 1.00 by step 4k of 143k and stays there, so it cannot track most of training. This is in tension with the advice in Sec. 6 that "progress in training should be read from identity, not from shape".

Questions for authors

Clarifications:

  1. Can you report within-form RSA between the trained model and BoW, and cross-form RSA after controlling for BoW similarity? This will directly test whether the geometry shape really comes from data.

  2. With n=200, running TWO-NN on pure Gaussian noise (Eq. 1 with 10% discard) gives ~381 at H=3584 and ~423 at H=4096 in my simulation. Untrained 7-8B values in the paper are 366-397. Is the ordering in these untrained networks only driven by legal and code documents? Also, how are code files split into 200 sentences?

  3. Can you provide confidence intervals with resampling at the book level? Since the 24 Chinese documents only come from 6 books, document-level resampling may not be independent.

Suggestions:

  1. Add a token-level BoW built with each model's own tokenizer. This can directly test the claim that "the tokenizer decides".

  2. For untrained networks, fit an orthogonal map on half of the sentences and test hit@10 on the other half. This will test whether training only aligns 2 point clouds that already have the same shape.

8 comments

We thank the reviewer for the detailed technical questions and concrete experimental suggestions. We ran all of them. The paper's three findings remain unchanged.

Summary

  • New results. Q1, Q2, Q3, S4, and S5 are answered with new tables (R1 to R10 below). The orthogonal-map test you proposed (S5) is included in our planned revisions.
  • Two objections concern statements the paper does not make. W1 reads the attribution as causal; Section 3 states that it is "by first appearance, not a causal decomposition". W2(b) lists three changes produced by training; all three are results the paper itself reports. We point to the text in each case and make the definition explicit where it was easy to miss.
  • Points we agree with. W1(c), W2(c), and the first half of Q2. The planned revisions are listed under each point.

W1(a). Scalar vs structure

  • The paper does not make a causal claim. Section 3, first paragraph: "the attribution is by first appearance, not a causal decomposition." The rule attributes each property to the first of the three representations of the same documents in which it appears. If the BoW representation already shows a property, observing it in a trained representation does not by itself show that training produced it. This is the inference the paper draws, and it does not require the structure to be passed from one representation to the next.
  • Qwen: Section 5.2 reports the same three numbers you cite. "untrained Qwen2.5 stays at 0.07-0.14" and "Training raises the cross-form RSA to 0.53-0.72 in Qwen2.5". The attribution to data says that word counts already show cross-form similarity (0.46; 0.76 with matched units, see W1(b)), so its presence in trained Qwen does not by itself indicate a learned invariance.
  • Mistral: we now compare each network with BoW directly (Table R1 under Q1). Within the character form, untrained Mistral agrees with BoW at 0.25 and trained Mistral at 0.36.
  • Planned revision.
    • Clarification. The paper does not claim that an untrained network retains the BoW structure. To remove that reading, "keeps" will become "also shows" in the abstract, Contribution 2, and Section 5.2: the tokenizer decides whether an untrained network also shows the similarity that the word counts show. In Section 5.2, "leaves it unchanged in Mistral" will become "leaves the cross-form RSA equal in Mistral (0.46)". Finding (2) is unchanged.
    • Additional evidence. Table R1 will be added to Appendix F. It answers Q1(i). Findings (1) to (3) do not depend on it.

W1(b). Baseline sensitivity

  • Section 5.2 reports this value and its cause: "the character n-gram version, whose units cut across syllables in pinyin, keeps less of it (0.13)". A 3-to-5-gram of Latin letters on the pinyin side does not correspond to a 3-to-5-gram of characters, so this variant compares mismatched units.
  • New: the result under every BoW we could construct (Table R2; 24 documents, median [min, max]).
Bag-of-wordsCharacter-pinyin RSA
Submitted BoW representation (TF-IDF)0.46 [0.40, 0.71]
Same, punctuation removed0.45 [0.39, 0.70]
Same, sentence length and punctuation regressed out0.46 [0.40, 0.71]
Matched units: one character against one syllable0.76 [0.71, 0.92]
Token-level, with the Qwen / Llama / Mistral / GPT-NeoX tokenizer0.34 / 0.38 / 0.39 / 0.35
Character 3-to-5-grams (the 0.13 variant)0.13
  • Reading. With the units that Lemma 1 is stated for, BoW gives 0.76, above every trained model (0.46 to 0.72). The value is not punctuation and not sentence length. The 0.13 variant is the only one that compares mismatched units.
  • Planned revision.
    • Additional evidence. Table R2 will be added to Appendix F, and Section 5.2 will cite the matched-unit value. It supports Finding (2): the cross-form similarity is in the word counts, and with matched units it is larger (0.76) than in the submitted BoW representation (0.46).

W1(c). Data definition

  • Agree. The BoW representation is computed on the documents we measure. Everything that reaches the trained model through pretraining, including its corpus, is on the training side of the comparison. This does not change any of the three attributions, which are stated for the documents we measure.
  • Planned revision.
    • Clarification. Section 3.2 will state in one sentence that the BoW representation is computed on the documents we measure, and the definition of "data" in Section 1 will be aligned with it. Findings (1) to (3) are unchanged.

W2(a). Token overlap

  • The low untrained value is the control, and the paper presents it as one. Section 4.1 chooses this pair because the two forms "share no content token, so correspondence between their clouds cannot come from shared tokens". Appendix B introduces simplified-traditional as "the Chinese shared-token reference" for the reason you give. The finding is the trained side: with no shared content token, trained models reach 0.34 to 0.99 against a length null of 0.11 to 0.13.
  • You are right that punctuation is shared. New: we removed all punctuation and symbols from both forms, which eliminates shared punctuation as an explanation for the identity result (Table R3).
hit@10, submitted formshit@10, no punctuationlength null, no punctuation
Untrained networks (7)0.10 to 0.200.06 to 0.150.05 to 0.12
Trained, five larger models0.86 to 0.990.83 to 0.980.11 to 0.13
Trained Qwen2.5-0.5B0.730.660.12
Trained Mamba-130M0.340.240.11

Untrained networks fall to their length null. Trained identity does not rest on punctuation.

  • The result is not by design, and your S5 test shows why. Whether an untrained network contains the correspondence is an empirical question. With an orthogonal map fitted on 100 sentence pairs, untrained networks recover 0.52 to 0.86 of held-out counterparts (chance 0.10; Table R7 under S5). The untrained networks already contain the cross-form correspondence, although they do not place a sentence next to its counterpart. Training adds identity.
  • Planned revision.
    • Clarification. Section 4.1 will state that the two forms share punctuation. Finding (3) is unchanged.
    • Additional evidence. Table R3 will be added to Appendix F. It supports Finding (3): with punctuation removed, trained identity remains and untrained networks fall to their length null.

W2(b). Other training changes

  • These three results are in the paper because the paper makes this point. The abstract, finding (3): "while the shape of the representation keeps changing to the end". Section 5.3, last sentence: "shape is present before training and keeps changing for the rest of it". Appendix H, which contains Table 11, is titled "Boundaries of what training adds".
  • What the paper claims. Under the rule of Section 3, a property is attributed to the first representation that shows it. Section 6: identity is "the one property among those we measured that neither the data nor the architecture supplies". A property that is present before training can still change in size during training; the Pythia RSA you cite is 0.48 at step 0. The claim is about which properties first appear with training. It is not a claim that training changes nothing else.
  • Planned revision.
    • Clarification. The claim is unchanged. So that its scope stays attached when the sentence is quoted alone, "Training adds one thing" in the abstract will read "Training adds one thing that neither the data nor the architecture supplies", which is the wording of Section 6. The abstract and Section 1 will state at first use that "present before training" and "changed by training" are two different judgments. In Section 5.3, "training added identity and nothing else that these measurements see" will become "Training adds cross-form identity, which neither the BoW representation nor any untrained network shows, while the shape of the representation keeps changing during training." Findings (1) to (3) are unchanged.

W2(c). Saturation

  • Agree. Early saturation is the finding ("this ability appears early in training"), and a measure that saturates at 3% of training cannot track the rest of it. Finding (3) is unaffected; only the recommendation in Section 6 is withdrawn.
  • New: a second trajectory, Pythia-1.4B, with the length null for both (Table R4).
StepPythia-410M hit@10length nullPythia-1.4B hit@10length null
00.380.070.420.08
1,0000.810.120.910.11
2,0000.970.141.000.13
4,0001.000.141.000.13
143,0001.000.141.000.13

(Re-extracted embeddings; step 0 of Pythia-410M is 0.38 here and 0.39 in the submitted text.)

  • Planned revision.
    • Correction. The sentence "progress in training should be read from identity, not from shape" will be removed from Section 6. This affects one recommendation in the Discussion and no finding.
    • Additional evidence. Table R4 will be added. It supports Finding (3): identity reaches 1.00 by step 2,000 to 4,000 in two model sizes.

Q1. Within-form RSA, and cross-form RSA controlling for BoW

(i) Within one form, RSA between the similarity matrix of BoW and that of the untrained (U) and trained (T) network. Table R1; median over middle layers, then documents.

ModelEnglish BoW-U / BoW-TCharacters BoW-U / BoW-TPinyin BoW-U / BoW-T
Qwen2.5-0.5B0.09 / 0.320.05 / 0.250.10 / 0.29
Qwen2.5-3B0.12 / 0.380.06 / 0.350.17 / 0.32
Qwen2.5-7B0.14 / 0.290.08 / 0.230.23 / 0.30
Llama-3.1-8B0.31 / 0.420.21 / 0.370.24 / 0.40
Mistral-7B0.30 / 0.420.25 / 0.360.26 / 0.34
Mamba-130M0.43 / 0.430.38 / 0.390.62 / 0.34
Mamba-2.8B0.45 / 0.440.41 / 0.400.65 / 0.42
  • Trained models agree with BoW at 0.23 to 0.44 in every model and form.
  • Untrained networks start anywhere from 0.05 (Qwen) to 0.65 (Mamba); after training all seven are at a similar level of agreement with word counts.

(ii) Character-pinyin RSA after regressing out, from both network matrices, the BoW matrices of both forms, then punctuation, then sentence length and token counts. Table R5; partial Spearman on ranks.

NetworkRSAgiven BoW+ punctuation+ length, token counts
Untrained Qwen2.5-0.5B / 3B / 7B0.07 / 0.11 / 0.140.07 / 0.10 / 0.130.07 / 0.10 / 0.130.07 / 0.10 / 0.12
Untrained Llama-3.1-8B0.440.410.380.30
Untrained Mistral-7B0.460.430.400.36
Untrained Mamba-130M / 2.8B0.24 / 0.300.12 / 0.150.12 / 0.150.11 / 0.14
Qwen2.5-0.5B / 3B / 7B0.69 / 0.53 / 0.710.65 / 0.47 / 0.690.59 / 0.42 / 0.640.45 / 0.33 / 0.52
Llama-3.1-8B0.580.480.460.44
Mistral-7B0.460.390.360.31
Mamba-130M / 2.8B0.32 / 0.440.25 / 0.360.21 / 0.320.15 / 0.29

(Median over layers within each document first; the RSA column differs from Table 10 by at most 0.02.)

  • Reading. Controlling for BoW removes half of the cross-form RSA of untrained Mamba and a smaller part elsewhere. A network's cross-form similarity is therefore not a copy of the BoW similarity matrix. The attribution in Section 5.2 does not say that it is: it is by first appearance (Section 3), and it rests on the fact that word counts alone already show cross-form similarity of this size (0.46; 0.76 with matched units).
  • Planned revision.
    • Additional evidence. Tables R1 and R5 will be added to Appendix F. They answer both parts of this question. Finding (2) is unchanged: it is an attribution by first appearance and rests on the BoW value.

Q2(i). Estimator noise

  • Agree, and we reproduce your simulation. With n = 200 and the 10% discard, 20 draws: 378 (sd 25) at H = 3584 and 401 (sd 32) at H = 4096. Your 381 and 423 are within one sd. The last-readout peaks of untrained Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B on narrative and essay (366 to 397) are at this level. The caption of Table 9 notes that these networks "sit near the estimator's ceiling"; we now give the noise level per width (Table R6a). Finding (1) is unaffected: it holds under the mean readout, where every untrained network is far below this level (Q2(ii)).
Width HTWO-NN on noise, mean (sd)Untrained networkIts narrative / essay peak
768175 (12)Mamba-130M56 / 57
896180 (14)Qwen2.5-0.5B74 / 74
2048276 (21)Qwen2.5-3B203 / 208
2560322 (26)Mamba-2.8B92 / 89
3584378 (25)Qwen2.5-7B366 / 369
4096401 (32)Llama-3.1-8B; Mistral-7B397 / 383; 377 / 379
Reviewer_zsXmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an interesting question: Where do certain geometric properties in the representation space of large language models originate? The authors propose a framework called DAST, which constructs bag-of-words representations from a set of texts, randomly initializes these representations, and pre-trains the model. The framework attributes these geometric properties to one of three sources: the data, the framework, or the training process. The authors conducted experiments on four large language model backbones and presented a series of conclusions that demonstrate a certain degree of robustness.

Critical strengths
  1. The research question is interesting, and the chosen research methods are sound.
  2. The experimental framework is intuitive and easy to understand, making it easy to reproduce the results.
  3. The experiment covers multiple models, and the conclusions have some generalizability.
Critical weaknesses

Overall, this paper has presented some interesting conclusions, but I believe it lacks sufficient depth. This is because the paper on interpretability is grounded in the fundamental problem of characterizing the geometric properties of learning, rather than explaining an unusual phenomenon such as “double descent” or “attention sink.” This makes it difficult to directly assess the paper’s practical value.

If this paper is intended to be an empirical study, I believe the current analysis lacks sufficient depth, primarily due to two concerns:

  1. The paper’s current conclusions require more rigorous counterfactual testing. Taking architectural attribution as an example, certain architectural components should be modified while holding other conditions constant to further pinpoint which module gives rise to the geometric properties. At a coarse granularity, the Transformer could be directly replaced with another architecture; at a fine granularity, aspects such as the network structure and number of layers of a randomly initialized Transformer could be fixed, while only the tokenizer is changed to verify the authors’ conclusion that “the tokenizer determines cross-form geometric structure.” Similar issues exist regarding the data and training sections.
  2. Some properties still face comparability issues, yet no solutions have been proposed. For example, the authors define “Identity” as the ability to recover nearest neighbors across writing forms and note that this metric cannot be defined on BoW. However, the fact that Hit@K cannot be directly computed on BoW does not mean that the input text itself cannot provide cross-form correspondences. The authors could easily construct a simple, non-learning-based comparison: first, convert Chinese characters to Pinyin; then represent both forms in a common character, syllable, or n-gram feature space; and finally perform the same nearest-neighbor matching. Similarly, differences in English uppercase and lowercase letters can be directly eliminated through deterministic preprocessing.

Alternatively, if the paper does not aim to advance the field in terms of depth, I believe it could dedicate 1–2 pages to demonstrating a practical application—such as guiding model design and optimization, as mentioned by the authors, or monitoring model training status—which would significantly enhance the paper’s practical potential.

Questions for authors
  1. Can the author directly compare the representation similarity matrices between BoW, a randomly initialized model, and the trained model? This is because identical RSA values do not prove that BoW and the trained model retain the same geometric structure.
6 comments

We appreciate the reviewer's encouraging assessment and the constructive suggestions. We ran all three experiments you proposed (W1, W2, Q1). The paper's three findings remain unchanged.

W1. Counterfactual tests

  • We thank the reviewer for this constructive suggestion. Holding the network fixed while swapping the tokenizer provides a direct test of Finding (2).
  • Fine granularity: we ran the test you describe (Table R10). An untrained network is built from one model's configuration with the other model's tokenizer. Depth, width, and the initialization rule are fixed; only the tokenizer, and with it the size of the embedding matrix, changes. The measure is the character-pinyin RSA of the untrained network.
Untrained networkOwn tokenizerOther tokenizerPaired difference [95% CI over books]
Llama-3.1-8B0.440.16 (Qwen tokenizer)0.25 [0.20, 0.33]
Qwen2.5-7B0.140.40 (Llama tokenizer)0.23 [0.17, 0.28]

The value follows the tokenizer. A second seed gives 0.15 and 0.40.

  • The same design holds the tokenizer fixed and changes the network. With the Qwen tokenizer, the Llama network gives 0.16 and the Qwen network 0.14. With the Llama tokenizer, the Qwen network gives 0.40 and the Llama network 0.44. The network changes the value little.
  • Coarse granularity. The submission includes a state-space architecture, Mamba, at two sizes (Section 4.2). All three findings hold for it (Tables 9 and 10).
  • Data and training. For data, the paper modifies the documents in five ways with a known effect on the BoW representation (Appendix G, Table 11), and we add the character and pinyin forms with all punctuation and symbols removed (trained identity 0.24 to 0.98; untrained networks at their length null). For training, we add a second trajectory, Pythia-1.4B, to Pythia-410M: identity reaches 1.00 by step 2,000 and step 4,000. We did not retrain models with altered data or objectives; the design of the paper is to attribute without retraining (Section 2).
  • Planned revision.
    • Table R10 will be added to Section 5.2. It supports Finding (2): the tokenizer decides whether an untrained network also shows the similarity that the word counts show. In Section 6, the tokenizer explanation will be described as a controlled result and will no longer be listed as "an observation".
    • The second trajectory and the punctuation-free forms will be added to Appendix B. They support Finding (3).

W2. Comparability of identity

  • Thank you for this insightful observation. We built the baseline you describe (Table R11). hit@10 as in Eq. 3, with TF-IDF vectors and no model. "With the map" applies the known conversion first (characters to pinyin; uppercase to lowercase) and then matches in one feature space. "Without the map" matches the two forms as written, with punctuation removed.
PairBoW, with the mapBoW, without the mapTrained models, no map given
Characters and pinyin1.000.04 (chance 0.05)0.24 to 0.98
Mixed and upper case1.000.281.00

(Trained models on characters and pinyin are measured on the same punctuation-free forms; on the submitted forms, they reach 0.34 to 0.99 (Table 10).)

  • Reading. You are right that the text carries the correspondence: once the conversion is given, matching is perfect, and for case it is removed by deterministic preprocessing as you say. This makes Finding (3) more precise. The conversion is what the baseline is given and what a model is not given. Without it, word counts match characters and pinyin at chance, and trained models match them at 0.24 to 0.98 (length null 0.11 to 0.13).
  • Planned revision.
    • Table R11 will be added to Section 5.3 as an additional comparison. Table 10 and Figure 2C keep the BoW entry for identity as in the submission, because this baseline uses a matching construction that is not part of the submitted measurement. It supports Finding (3): identity is not available from the text without the conversion, and only trained models show it.

Q1. Direct comparison of similarity matrices

  • Agree, and we ran the comparison (Table R1). Within one form and one set of sentences, the RSA between the similarity matrices of two representations. B = BoW, U = untrained network, T = trained model. Median over middle layers, then documents.
ModelEnglish B-U / B-T / U-TCharacters B-U / B-T / U-TPinyin B-U / B-T / U-T
Qwen2.5-0.5B0.09 / 0.32 / 0.150.05 / 0.25 / 0.040.10 / 0.29 / 0.11
Qwen2.5-3B0.12 / 0.38 / 0.240.06 / 0.35 / 0.080.17 / 0.32 / 0.15
Qwen2.5-7B0.14 / 0.29 / 0.340.08 / 0.23 / 0.130.23 / 0.30 / 0.27
Llama-3.1-8B0.31 / 0.42 / 0.540.21 / 0.37 / 0.320.24 / 0.40 / 0.37
Mistral-7B0.30 / 0.42 / 0.560.25 / 0.36 / 0.380.26 / 0.34 / 0.42
Mamba-130M0.43 / 0.43 / 0.460.38 / 0.39 / 0.370.62 / 0.34 / 0.35
Mamba-2.8B0.45 / 0.44 / 0.500.41 / 0.40 / 0.390.65 / 0.42 / 0.38
  • Reading. Your point holds in the case the paper highlights. In Mistral the cross-form RSA is 0.46 before and after training, and within one form the untrained and trained matrices agree at 0.38 (characters) and 0.42 (pinyin). Equal values do not mean the same structure. Trained models agree with BoW at 0.23 to 0.44.
  • What the paper claims. The attribution is "by first appearance, not a causal decomposition" (Section 3). Finding (2) says that word counts already show cross-form similarity of this size. It does not say that the trained model retains the BoW structure, and two words in the submission suggested that it does.
  • Planned revision.
    • Clarification. "keeps" will become "also shows" in the abstract, Contribution 2, and Section 5.2: the tokenizer decides whether an untrained network also shows the similarity that the word counts show. In Section 5.2, "leaves it unchanged in Mistral" will become "leaves the cross-form RSA equal in Mistral (0.46)". Finding (2) is unchanged.
    • Table R1 will be added to Appendix F. It answers this question. Findings (1) to (3) do not depend on it.

Depth and practical application

  • Thank you for raising this, and for the two directions you offer. We answer both.
  • On depth: what the paper explains. The paper takes regularities that prior work reports in representation geometry and asks where each one comes from. Four of its answers are explanations of this kind, and each is now a tested statement.
    • Why intrinsic dimension differs across kinds of text. It follows the lexical freedom of the text. Per-document unigram entropy predicts peak intrinsic dimension at rho = 0.94 in the BoW representation and at 0.87 to 0.90 in trained models (Table M1, in our response to Reviewer Mp4g). This is Finding (1).
    • Why a text and its other way of writing are similar in the untrained networks of one family and not of another. The tokenizer. With the network fixed, swapping the tokenizer moves the character-pinyin RSA from 0.44 to 0.16 and from 0.14 to 0.40 (W1, Table R10). This is the second half of Finding (2).
    • Why similarity on raw vectors and identity disagree. In several trained models one coordinate carries more than 99% of the variance. On raw vectors the character-pinyin RSA is 0.89 to 0.93 while hit@10 is 0.27 to 0.51; after standardization the RSA is 0.46 to 0.72 and hit@10 is 0.73 to 0.99 (Appendix A).
    • What training adds. With an orthogonal map fitted on 100 sentence pairs, held-out matching is 1.00 for BoW and 0.52 to 0.86 for untrained networks (chance 0.10). Six of the seven trained models reach 0.85 to 0.99 with no map (Table R7, in our response to Reviewer RZzD). The correspondence is already in the word counts and in untrained networks; trained models place a sentence next to its counterpart without being given a map. This is Finding (3), and it is a test of the statement in Section 6 that identity "aligns two clouds that already have the same shape".
  • How this relates to the kind of work you describe. The examples you give, double descent and attention sink, are single anomalies. The subject of this paper is different: the evidence that geometry provides about what a model learned. Its results are of the form "this regularity comes from this source", and the four above are stated at that level.
  • Practical application. We did not add an application experiment. The results support the following practical guidance, each with the result it rests on.
    • Measuring geometry before reading it as learned. Standardize each coordinate first (Appendix A, numbers above). Report a length null with any matching score: sentence length alone matches characters and pinyin at 0.68 and the case pair at 1.00. Compare with the BoW representation and with an untrained network of the same architecture: the ordering of document types has rho = 0.93 in BoW and 0.87 to 0.91 in untrained networks.
    • Model design. The tokenizer decides whether an untrained network also shows the cross-form similarity (Table R10).
    • Monitoring training. Identity reaches 1.00 by step 2,000 to 4,000 of 143,000 in two model sizes. It marks an early stage of training and cannot track what follows, so the broader recommendation in Section 6 is withdrawn.
  • Planned revision.
    • Table R7 will be added to Section 5.3. It supports Finding (3) and the statement in Section 6 quoted above.
    • Clarification. "Use and limits" in Section 6 will list the measurement practices above, each with the result it rests on. Findings (1) to (3) are unchanged.
    • Correction. The sentence "progress in training should be read from identity, not from shape" will be removed from Section 6. This affects one recommendation in the Discussion and no findings.
Reviewer_Mp4gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DATS, which compares bag-of-words, untrained networks, and trained language models in an attempt to distinguish contributions to “representational geometry”.

Critical strengths

Highlights the value of keeping simple and untrained baselines in measuring qualities of the representations.

Critical weaknesses

Intrinsic dimension and representational similarity are to my knowledge established metrics that have been well studied. Prior work even studies how geometry changes during training. The paper does not clearly establish what the DATS framework adds.

Section 5.1 reports Spearman correlations against an author-specified ordering of document types. Although motivated by vocabulary entropy, the significance is unclear and seemingly arbitrary. Broad claims that training adds little representational structure far exceed the evidence from selected metrics and narrow transformations.

The relationship between individual empirical results and the headline claims is difficult to follow. It is not clear why the experiments are chosen the way they are.

The dense writing makes the central question, framework, and implications unnecessarily difficult to understand. It remains unclear what conclusions the framework supports.

Questions for authors
  1. What does DATS establish beyond existing work (such as Layer by Layer ... [Skean, 2025]) on representational geometry and comparisons with untrained baselines?

  2. Why is the author-specified ordering of document types the appropriate reference for Section 5.1, rather than each document’s measured vocabulary entropy?

  3. How do the selected metrics and writing transformations support the broad claim that training contributes little representational structure?

6 comments

We thank the reviewer for the opportunity to clarify DATS's contribution and its relationship to prior work. We answer the three questions and the two comments in order. The paper's three findings remain unchanged.

Q1. What DATS adds beyond existing work

  • We thank the reviewer for bringing Skean et al. (2025) to our attention. We will add a discussion of this work to Section 2 to clarify how its research question differs from that of DATS.
  • We agree that intrinsic dimension and RSA are established metrics. In the last paragraph of Section 2, we note that DATS uses "the measurements standard in this literature" (TWO-NN intrinsic dimension and RSA), and we do not propose a new one. Our contribution is the comparison and the rule: the same documents are represented in three ways (bag-of-words with no model, an untrained network, the trained model) and measured with the same instruments, and each property is attributed to the first of the three in which it appears. This turns "the trained model has property X" into a test of whether X is already observable before training.
  • Beyond comparisons with untrained baselines. In Section 2 (paragraph "Properties present before training"), we cite this line of work (Zhang & Bowman, 2018; Wieting & Kiela, 2019; Zhong & Andreas, 2024) and describe the difference: DATS "applies the same measurements to the same documents before and after training". DATS also adds a representation that involves no model at all, which those comparisons do not have.
  • Beyond Skean et al. (2025). That paper asks which layer gives the best representation: it computes a set of representation metrics at every layer, relates them to downstream tasks, and follows them over training checkpoints. It does not include a representation without a model or an attribution step, so whether a reported value is already observable before training is outside its scope. The two papers answer different questions.
  • An example of what the comparison changes (Table 9). The ordering of the five document types by intrinsic dimension has rho = 0.88 to 0.94 in the seven trained models. An analysis of trained models alone would report this as a property of the models. The bag-of-words representation, which has no model, gives 0.93, and the untrained networks give 0.87 to 0.91. The ordering is a property of the text.
  • Planned revision.
    • Clarification. Skean et al. (2025) will be added to Section 2 with the comparison above. Findings (1) to (3) are unchanged.

Q2. Type ordering and measured entropy

  • We thank the reviewer for this constructive suggestion; we ran it. The answer has two parts: what the submission states, and the per-document version you ask for.
  • The ordering of document types is based on their measured median unigram entropy. In Section 4.1, we write: "The ordering is measured, not assumed: we operationalize lexical freedom as the unigram entropy of a document", and we report the median entropy of the five types (8.62, 8.59, 8.26, 7.78, 6.67 bits). Table 8 lists the entropy of each of the 56 documents. The ordering was fixed before any model was run (Appendix D).
  • New: the per-document version you ask for (Table M1). Spearman correlation across the 56 documents between each document's unigram entropy and its peak intrinsic dimension.
Representationrho with type order (Table 9)rho with per-document entropysame, sentence length regressed out
TF-IDF (no model)0.930.940.90
Character n-gram (no model)0.930.940.90
Untrained networks (7)0.87 to 0.910.86 to 0.950.78 to 0.92
Trained models (7)0.88 to 0.940.87 to 0.900.78 to 0.84
  • Reading. Per-document entropy predicts peak intrinsic dimension as well as the type ordering does, in every representation, and sentence length does not explain it. The conclusion of Section 5.1 is unchanged.
  • Planned revision.
    • Table M1 will be added next to Table 9, and Section 5.1 will report both correlations. It supports Finding (1): the ordering follows the measured entropy of each document.

Q3. Metrics, transformations, and the claim

  • We thank the reviewer for the opportunity to clarify the scope of the claim. We answer with the text of the paper and with new checks.
  • The claim is stated for what is measured. In the abstract, we state the claim for "the representational structure we can measure". In Section 1, we describe the ability that training adds as "invariance to the way of writing, tested here on case, script, and romanization". In the Conclusion, we write "Most of the document-level structure we measure". The paper makes three statements, one per property (abstract, Section 4.3): (1) the ordering of document types by intrinsic dimension is already present in the data; (2) the similarity between two ways of writing the same text is already present in the word counts, and the tokenizer decides whether an untrained network keeps it; (3) only trained models place two ways of writing the same text together.
  • Why these measurements are sufficient for the conclusion. We do not argue that intrinsic dimension and similarity structure capture everything about a representation. Instead, the paper tests readings that prior work has drawn from these same measurements (Section 4.3): intrinsic dimension read as a measure of how much a model abstracts from its input (Cheng et al., 2025), and alignment across two ways of writing the same text read as a learned invariance (Muller et al., 2021; Wendler et al., 2024). DATS applies the identical measurement to the identical documents in all three representations. If a measurement is informative enough to support "the trained model has property X", the same measurement on word counts or on an untrained network is informative enough to show that X is already there. The conclusion is as strong as the measurements those readings rely on, and it is a conclusion about those readings.
  • The ways of writing in the submission. Five pairs of ways of writing the same text: characters and pinyin, characters and toneless pinyin, simplified and traditional characters, Japanese and romaji (a held-out replication in a second language), and mixed and upper case (Tables 4 and 10, Section 5.3). Seven models from four families, including a state-space architecture (Mamba).
  • New checks, all with the same outcome. A second seed for every untrained network (values within 0.02). A second training trajectory, Pythia-1.4B (identity reaches 1.00 by step 2,000). Character and pinyin forms with all punctuation and symbols removed, which rules out shared punctuation as an explanation (trained identity 0.24 to 0.98; untrained networks at their length null).
  • Planned revision.
    • Clarification. The claim is unchanged. So that its scope stays attached when the sentence is quoted alone, "Training adds one thing" in the abstract will read "Training adds one thing that neither the data nor the architecture supplies", which is the wording of Section 6. The abstract and Section 1 will state at first use that "present before training" and "changed by training" are two different judgments. Findings (1) to (3) are unchanged.
    • The three new checks above will be reported in Appendix B. They support Findings (2) and (3).

Experiment choice

  • Thank you for this comment. The mapping from experiments to claims is in Section 4.3 and Figure 2; we now show it in one table.
  • Each experiment tests one published reading of representation geometry. In Section 4.3, we give the reading for each property; the mapping to results is one to one.
Property (Section 4.3)Prior reading it testsResultWhere
(1) Ordering of document types by intrinsic dimensionA measure of how much a model abstracts from its input (Cheng et al., 2025)Already present in the data (BoW representation, rho = 0.93)Abstract (1), Fig. 2A, Section 5.1
(2) Similarity between two ways of writing the same textA learned invariance (Muller et al., 2021; Wendler et al., 2024)Already present in the word counts (RSA = 0.46); the tokenizer decides whether an untrained network keeps itAbstract (2), Fig. 2B, Section 5.2
(3) Placing two ways of writing the same sentence togetherSameOnly trained models do so (hit@10 0.34 to 0.99 against 0.10 to 0.20 for untrained networks)Abstract (3), Fig. 2C, Section 5.3
  • Planned revision.
    • Clarification. This table will be added at the start of Section 5. It cross-references what Section 4.3 and Figure 2 already state. Findings (1) to (3) are unchanged.

Writing

  • Thank you for the frank comment on the writing.
  • To make the mapping from properties to results easier to follow, the revision will open Section 5 with the table above and add Table M1 next to Table 9.
  • The conclusions the framework supports, as stated in the Conclusion. The ordering of document types and the shape shared by two ways of writing the same text are present before training; only the ability to place two ways of writing a sentence together is not (Section 7). In Pythia, this ability appears within the first few percent of training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Area Chair [Update of PDF & Summary of Revisions]

Dear Reviewers and Area Chair,

We thank the Area Chairs and all Reviewers for their time and effort in reviewing our work. We have uploaded a revised PDF and posted a point-by-point response to each reviewer, with 12 new analyses.

We are encouraged by the reviewers' recognition of the following aspects of our work:

  • An important question and a simple control. "The research question is interesting, and the chosen research methods are sound" (Reviewer zsXm); "Important question and simple baseline idea. Using BoW and an untrained twin as default controls, without any retraining, is very useful for anyone who uses geometry as evidence of learning" (Reviewer RZzD); "Highlights the value of keeping simple and untrained baselines in measuring qualities of the representations" (Reviewer Mp4g).
  • Breadth and reproducibility. "The experiment covers multiple models, and the conclusions have some generalizability" and "easy to reproduce the results" (Reviewer zsXm); "applies this setup consistently across 4 model families, with a held-out Japanese replication and a base-vs-instruct check" (Reviewer RZzD).
  • Careful measurement. "Careful measurement and honest reporting. App. A shows that raw and standardized vectors give opposite conclusions for cross-form alignment, this is a good warning for future alignment work" (Reviewer RZzD).

Summary of additions (table numbers refer to the revised PDF)

Controlled tests of the attribution

  • Tokenizer swap on a fixed untrained network: Table 2 in Section 5.2, full values in Table 25 (Reviewers zsXm, RZzD).
  • Matching baseline that uses no model, with and without the known conversion: Table 24 (Reviewer zsXm).
  • Orthogonal map fitted on half of the sentences, tested on the other half: Table 23 (Reviewer RZzD).

Structure, not only scalar values

  • Within-form RSA between BoW, untrained, and trained similarity matrices: Table 17 (Reviewers zsXm, RZzD).
  • Cross-form RSA after controlling for BoW, punctuation, and length: Table 19 (Reviewer RZzD).
  • BoW constructions, including matched units and each model's own tokenizer: Tables 18 and 22 (Reviewer RZzD).

Robustness

  • Per-document entropy against peak intrinsic dimension: Table 16 (Reviewer Mp4g).
  • Estimator noise level per width, and the mean readout: Tables 7 and 8 (Reviewer RZzD).
  • Punctuation and symbols removed from both forms: Table 20 (Reviewer RZzD).
  • A second training trajectory, Pythia-1.4B: Table 9 (Reviewer RZzD).
  • Bootstrap over books: Table 21 (Reviewer RZzD).
  • A second seed for every untrained network: Appendix B (Reviewer Mp4g).

Clarifications

  • Section 5 opens with a table mapping each property to the prior reading it tests and the result (Table 1).
  • Definitions are numbered in Section 3, with a remark that first appearance and subsequent change are two different judgments.
  • Related work now discusses Skean et al. (2025) (Section 2).

Corrections

  • Figure 3B is redrawn on the 56 English documents used in Section 5.1; the submitted panel pooled documents across languages. The document types are ordered at step 0 as at the end of training (rho = 0.89 and 0.90).
  • Appendix E now states six books, as in Table 12.

The three findings of the paper are unchanged, and each now rests on controlled evidence. DATS asks, for each geometric property of a trained model, whether the same measurement on the same documents already shows it in a bag-of-words representation or in an untrained network. The answer separates what training adds, the ability to place two ways of writing the same text together, from what the data and the tokenizer already supply.

We hope this summary helps in reviewing the revision. Thank you again for your time.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.