acceptodds
Under review as a conference paper at ICLR 2027

Relative Generalization Invariance of LLM Pretraining

Abstract

Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Pretraining, Optimizers, Training Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EKMMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates a metric, the Relative Generalization Invariance (RGI), which is defined on a pair of two models whose every NTP loss on a validation set has a constant difference. The paper empirically shows when and how strongly such an invariance holds by varying the training conditions, e.g., optimizers, architectures, and data stream. As the RGI tends to hold when the data stream is fixed, it may be a metric for analyzing the effect of other factors.

Critical strengths
  • RGI seems an interesting observation.
  • Experiments seems reasonable and sound enough.
Critical weaknesses
  • The paper may not emphasize practical contributions about RGI. It would be strong if interesting or useful discussions beyond "just finding something" are provided.
  • In my feeling, RGI is possibly not very novel. Practitioners may know that two models show very similar training curve with similar zigzags when training them using the same seed (basically, the same init. and data stream), which might be a partial observation of RGI (especially for the first epoch). I also found some related literature (e.g., Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability, https://arxiv.org/abs/2308.15419).
Questions for authors

(abst. and similarly for intro.) We take a first step toward isolating their effects by studying relative generalization. ... our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.

What is the "effect" concretely? Can we rephrase it more concretely?

Furthermore, does this phenomenon really help distinguish the effects of optimizers and architectures from those of training data (stream?)? In other words, what is "to distinguish the effects ..." or "isolating their effects" concretely? And, is it something practical or important?

4 comments

W1. The paper may not emphasize practical contributions about RGI. It would be strong if interesting or useful discussions beyond "just finding something" are provided.


RGI provides practical insights for LLM pretraining: optimizer and moderate architectural choices primarily shift overall validation loss while approximately preserving relative token losses, whereas changing the training data can alter the relative loss structure across tokens. It leads to the following practical implications.

  1. Actionable guidance for model development: Because changing the optimizer (Section 4.1, Figures 1 and 2) or making moderate architectural changes (Section 4.2, Figures 3 and 4) shifts validation losses by a token-independent constant C(T,T′,t)C(\mathfrak{T}, \mathfrak{T}', t), one can tune optimizers or attention/FFN modules to lower the global validation loss without worrying about trading off performance across different token subsets. Conversely, if a practitioner wants to improve generalization on specific domains or token types relative to others, tuning the optimizer will not change the relative loss structure; one must modify the training data stream (Section 4.3, Figure 5).

  2. Explaining training-curve diagnostics: RGI also explains why different optimizers and architectures trained on the same data stream exhibit similar training-loss curves and shared local spikes (Appendix D, Figure 35). Specifically, we decompose adjacent-step training-loss changes into optimizer-update effects and batch-induced effects, finding that the latter are approximately 50 times larger (Figure 35(b)). Since RGI implies that these batch-induced loss differences are approximately invariant across optimizers and architectures, their training-loss curves exhibit similar local fluctuations.


W2. In my feeling, RGI is possibly not very novel. Practitioners may know that two models show very similar training curve with similar zigzags when training them using the same seed (basically, the same init. and data stream), which might be a partial observation of RGI (especially for the first epoch). I also found some related literature (e.g., Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability, https://arxiv.org/abs/2308.15419).


RGI goes substantially beyond the observation of similar training-curve zigzags under the same seed and the correlation analysis in Chang et al.

  1. Beyond same-seed training zigzags: Rather than being a restatement of parallel training curves, RGI is the underlying mechanism that explains why those parallel zigzags occur after the early stage of training (Appendix D, Figure 35). In addition, the notion RGI is beyond the typical setting, where models share the same initialization. It holds across most optimizers and a wide range of moderate architecture variations.

  2. Distinction from Chang et al.: Chang et al. characterize token-level learning curves and their reproducibility across random initializations and data shuffles under a fixed architecture and training recipe. Our work differs in both the invariance criterion and the training configurations investigated.

    • A more stringent invariance criterion: Pearson correlation equals 11 whenever LB(d)=aLA(d)+bL_B(d)=aL_A(d)+b for any a>0a>0, whereas RGI requires a unit-slope additive shift, LB(d)=LA(d)+CL_B(d)=L_A(d)+C, with CC independent of dd (Proposition 1). We provide empirical evidence for approximate RGI through the unit-slope fit y=x−0.07y=x-0.07 (R2=0.99R^2=0.99, Figure 1(a)) and our RGI diagnostics.

    • Invariance across optimizers and architectures: While Chang et al. fix the optimizer and architecture, we show that approximate RGI persists across distinct optimizer geometries, including Adam, Muon, and SOAP, and moderate architectural variations. In contrast, changing the training data stream can substantially alter relative generalization.

We thank the reviewer for this reference and haved discussed Chang et al. [1] in related works (Lines 900-908) of the revised version.


(abst. and similarly for intro.) We take a first step toward isolating their effects by studying relative generalization. ... our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.

Q1. What is the "effect" concretely? Can we rephrase it more concretely?


Concretely, "effect" means the influence of each training component on generalization, as measured by the token-wise validation losses {L(θtT,d)}d∈Dval\{L(\theta_t^{\mathfrak{T}}, d)\}_{d \in \mathcal{D}_{\mathrm{val}}}. By Proposition 1 (Section 3), a component's influence on validation generalization can take two forms:

  1. A uniform shift C(T,T′,t)C(\mathfrak{T}, \mathfrak{T}', t) in the overall validation-loss level across all tokens d∈Dvald \in \mathcal{D}_{\mathrm{val}}, or
  2. A change in relative generalization, namely the validation-loss differences L(θtT,d)−L(θtT,d′)L(\theta_t^{\mathfrak{T}}, d) - L(\theta_t^{\mathfrak{T}}, d') between any two tokens dd and d′d'.

We have revised the paper's abstract and introduction accordingly (Lines 15, 24, 38, and 66).


Q2. Furthermore, does this phenomenon really help distinguish the effects of optimizers and architectures from those of training data (stream?)? In other words, what is "to distinguish the effects ..." or "isolating their effects" concretely?


Yes. RGI distinguishes their influences by testing whether changing a single component preserves pairwise token-loss differences or alters the relative loss structure across tokens. By Proposition 1 (Section 3), RGI between two training triplets T\mathfrak{T} and T′\mathfrak{T}' is equivalent to a token-independent constant gap: L(θtT,d)−L(θtT′,d)=C(T,T′,t)∀d∈Dval.L(\theta_t^{\mathfrak{T}}, d) - L(\theta_t^{\mathfrak{T}'}, d) = C(\mathfrak{T}, \mathfrak{T}', t) \quad \forall d \in \mathcal{D}_{\mathrm{val}}. Concretely, we vary one component at a time in Sections 4.1 to 4.3 and test whether this additive relationship holds:

  1. Optimizers and moderate architectural variations preserve relative generalization: Across all six tested optimizers (Section 4.1, Figure 1, e.g., y=x−0.07y = x - 0.07 with R2=0.99R^2 = 0.99 for Adam vs. Muon) and moderate architectural variations in attention, FFN, and width (Section 4.2, Figures 3 and 4), approximate RGI holds (rCCC>0.95r_{\mathrm{CCC}} > 0.95 and rdiff<0.05r_{\mathrm{diff}} < 0.05). Their influence on token-wise validation losses is therefore captured by an approximately uniform shift C(T,T′,t)C(\mathfrak{T}, \mathfrak{T}', t) across tokens.

  2. Training data alters relative generalization: In contrast, changing the training data source (C4, ArXiv, or CodeParrot vs. FineWeb in Section 4.3, Figure 5, even with Adam and the architecture held fixed) substantially weakens or breaks RGI (rCCC<0.95r_{\mathrm{CCC}} < 0.95), showing that the training data reshapes the relative loss structure across tokens.

Q3. And, is it something practical or important?

Distinguishing these two forms of influence is both conceptually important for understanding pretraining generalization and practically useful for model development and diagnostics.

  1. Conceptual importance: Although the optimizer, architecture, and training data jointly determine validation performance, RGI shows that they govern different aspects of generalization. Even when different optimizers move far from initialization and converge to parameters and hidden states with low similarity (Section 5.1, Figure 7), they preserve the same relative token-wise loss structure, which is instead shaped by the training data.

  2. Practical relevance: First, for model evaluation and development, because optimizer and moderate architectural improvements act as an approximately uniform shift across tokens (which also transfers across moderate validation shifts in Section 4.4, Figure 6), lowering validation loss via optimizer or architectural tuning improves performance consistently across token subsets without hidden trade-offs, whereas changing relative performance on specific domains or token types requires modifying the training data. Second, for training diagnostics, RGI explains why different optimizers and architectures trained on the same data stream exhibit parallel training-loss curves with shared local spikes (Appendix D, Figure 35).


We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.


References

[1] Chang T A, Tu Z, Bergen B K. Characterizing learning curves during language model pre-training: Learning, forgetting, and stability[J]. Transactions of the Association for Computational Linguistics, 2024, 12: 1346-1362.

Thank you for the response. It has helped me better understand the study. However, my concern about its significance remains.

The proposed practical implications are still abstract. For optimizer tuning, it is unclear when token-subset trade-offs are a practical concern that RGI helps resolve. Likewise, modifying training data to improve a specific domain is a natural strategy even without RGI. What concrete difficulty does this finding address, or what decision does it change?

An immediate application is not necessary, but I would like to understand what we learn beyond the loss relationship itself. Could you give a concrete example of a useful prediction or a large change in our understanding of pretraining that follows from RGI?

We sincerely thank you for your reading and replying to our response, as well as for raising insightful follow-up comments.

The proposed practical implications are still abstract. For optimizer tuning, it is unclear when token-subset trade-offs are a practical concern that RGI helps resolve. Likewise, modifying training data to improve a specific domain is a natural strategy even without RGI. What concrete difficulty does this finding address, or what decision does it change?

An immediate application is not necessary, but I would like to understand what we learn beyond the loss relationship itself. Could you give a concrete example of a useful prediction or a large change in our understanding of pretraining that follows from RGI?

Our main contributions are the formal definition and empirical characterization of RGI, together with demonstrating that RGI persists beyond the NTK and MF regimes. Although a comprehensive investigation of downstream applications is beyond the scope of this conference paper, we highlight several practical implications of RGI below.

Prediction:

  • A new decomposition of LLM representations. Existing interpretability methods, such as sparse autoencoders, decompose LLM representations into features associated with different semantic concepts or domains. Motivated by RGI, we conjecture that LLM representations may admit a complementary decomposition into token-relative and token-average components. Specifically, RGI shows that changes in optimizers and moderate architectural choices approximately preserve relative token-wise losses while shifting their average. We hypothesize that a similar phenomenon may occur in representation space: the token-relative component remains approximately invariant across different optimizers and architectures, while the token-average component captures the effects of these training choices. Although our results in Section 5.1 show low overall similarity between representations learned using different optimizers, they do not examine whether such a decomposition could reveal an invariant component. Formally defining these components and empirically testing this hypothesis remain important directions for future work.

Practical implications:

  • Separating architecture and optimizer tuning from data curation. In practice, LLM developers often introduce new architectures, optimizers, and curated domain-specific data mixtures simultaneously. A natural question is whether changes in architectures or optimizers introduce topic-specific biases that could undermine the intended benefits of data curation. Our RGI results provide evidence against such biases when the training data stream is held fixed. Specifically, changes in optimizers and moderate architectural choices shift token-wise validation losses approximately uniformly, rather than systematically favoring or penalizing tokens from particular topics. This suggests that architecture and optimizer tuning can improve overall performance without substantially altering the relative loss structure across different token subsets.

  • Evaluation guidance for new optimizers and architectures. How should practitioners design evaluation datasets when comparing new optimizers and architectures? Our RGI findings suggest that a relatively small in-distribution validation set can be sufficient to estimate overall performance improvements, because the loss differences across models are approximately uniform across tokens. Therefore, practitioners can allocate more evaluation resources to diverse Out-Of-Distribution (OOD) datasets, where RGI may be weaker and domain-specific performance differences are more likely to emerge.

Reviewer_k57UOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the performance of LLM pre-training in relation to three components: the optimizer, model architecture, and training data. The authors emphasize considering these components jointly, in contrast to previous studies, and introduce the concept of Relative Generalization Invariance (RGI). RGI refers to the invariance of validation loss differences between any two tokens across models, essentially describing the preservation of differences between individual examples' losses.

The authors demonstrate approximate RGI across different optimizers, architectural variations, and training data, with experiments apparently involving models of up to approximately 4.5B parameters.

Critical strengths

The empirical aspect of the paper is relatively strong. In particular, considering different components of LLM pre-training within a common experimental setup is useful.

Critical weaknesses

The theoretical explanation appears somewhat weaker than the empirical component of the paper.

My main concerns, however, are twofold.

First, the paper's motivation rests heavily on the argument that three important components of pre-training have not previously been considered jointly, and that examining them together constitutes a significant contribution. Yet, the findings seem to suggest that two of these components, namely the optimizer and model architecture, have relatively limited impact, while training data plays a more decisive role.

Second, these observations appear to be largely consistent with what is already known in the community. While the paper provides experimental evidence for these findings, I am not convinced that the results offer substantially new insights beyond the existing understanding.

Questions for authors

1- What specific new insights do the authors believe RGI provides beyond what is already understood in the community about the respective roles of optimizers, model architectures, and training data?

2- To what extent does the theoretical analysis independently predict the empirical observations? In particular, does the theory establish these relationships without relying on the experimental findings?

6 comments

We sincerely thank the reviewer for reading our paper.


W1. First, the paper's motivation rests heavily on the argument that three important components of pre-training have not previously been considered jointly, and that examining them together constitutes a significant contribution.


We do not claim that the three components “have not previously been considered jointly” or that “examining them together” is itself a significant contribution. Indeed, Appendix A (Lines 888–908) explicitly acknowledges prior work studying these factors “separately and jointly.” Rather, our motivation in Lines 14 and 38 is to understand how these components influence performance “in distinct ways,” and our contribution is to identify and characterize this distinction through RGI.

W2. Yet, the findings seem to suggest that two of these components, namely the optimizer and model architecture, have relatively limited impact, while training data plays a more decisive role.

Our findings do not suggest that optimizers and architectures have limited impact on performance; rather, but they show the specific form of their impact. By Proposition 1 (Section 3), RGI is equivalent to a token-independent constant generalization gap:

L(θtT,d)−L(θtT′,d)=C(T,T′,t)∀d∈Dval.L(\theta_t^{\mathfrak{T}}, d) - L(\theta_t^{\mathfrak{T}'}, d) = C(\mathfrak{T}, \mathfrak{T}', t) \quad \forall d \in \mathcal{D}_{\mathrm{val}}.

The constant CC equals the difference in average validation loss, and RGI does not require this gap to be small. Thus, RGI does not imply that optimizers or architectures have a “limited impact” on overall performance. Figure 1(a), for example, shows that Muon’s and Adam’s validation-batch losses closely follow y=x−0.07y=x-0.07 (R2=0.99R^2=0.99), illustrating improved overall loss alongside approximately preserved relative losses. Our empirical finding is that changes among the tested optimizers and moderate architectural variations induce approximately uniform shifts in token-wise losses (Figures 1–4), whereas changes in the training data stream can alter their relative structure (Figure 5). This characterizes the distinct forms of their effects on generalization, rather than establishing that optimizer or architecture effects are “small” or “limited.”


W3. Second, these observations appear to be largely consistent with what is already known in the community. While the paper provides experimental evidence for these findings, I am not convinced that the results offer substantially new insights beyond the existing understanding.


We respectfully disagree that RGI is already known in the community, and we kindly ask the reviewer to show the specific prior works they have in mind. The general understanding that training data matters or that better optimizers achieve lower average loss does not imply that changing the optimizer (e.g., coordinate-wise Adam vs. matrix-based Muon in Section 4.1) or architecture (e.g., RoPE vs. NoPE/ALiBi/SWA, or GELU vs. SReLU/SiLU in Section 4.2) approximately preserves the validation-loss differences L(θ,d)−L(θ,d′)L(\theta,d)-L(\theta,d') across tokens. Moreover, we observe this invariance even when models move substantially from initialization and exhibit low cross-optimizer cosine similarity in both parameters and hidden-state representations (Section 5.1, Figure 7).

If the reviewer is aware of prior works establishing this generalization invariance, we would appreciate the references and gladly compare our findings directly.

Q1. What specific new insights do the authors believe RGI provides beyond what is already understood in the community about the respective roles of optimizers, model architectures, and training data?


Beyond defining RGI (Section 3), our work provides two concrete new insights into LLM pretraining:

  1. Separating uniform loss shifts from relative token structure (Sections 4.1 to 4.4): Across six coordinate-wise and matrix-based optimizers (Figure 1), optimizer hyperparameters (Figure 2), independent initializations and moderate model-size changes (Figure 3), attention and FFN variants (Figure 4), and related validation-distribution shifts (Figure 6), we observe approximately uniform shifts in token-wise validation losses. In contrast, changing the training data stream (C4, ArXiv, and CodeParrot in Figure 5) can substantially weaken or prevent RGI, while an extreme reduction in model width (768 vs. 64 in Figure 3c) also breaks it. This structural distinction in token-wise generalization cannot be inferred from existing works.

  2. Showing that RGI persists beyond standard NTK and Mean-Field explanations, motivating new theoretical directions for LLM pretraining (Sections 5.1–5.2): Figure 7 shows that parameters and hidden states deviate substantially from initialization and exhibit low similarity across optimizers, providing evidence against an NTK-style lazy-learning explanation. Figure 8 further shows that RGI persists down to width 192, where the validation-loss–learning-rate profile differs qualitatively from wider models (Figure 8(d)), suggesting that the same Mean-Field approximation is no longer adequate.


Q2. To what extent does the theoretical analysis independently predict the empirical observations? In particular, does the theory establish these relationships without relying on the experimental findings?


Theorem 2 establishes the emergence of RGI in an overparameterized quadratic model directly from mathematical assumptions, without using the empirical findings as premises. Specifically, under assumptions, Theorem 2 proves from the update rules of GD, Adam, and Muon that exact RGI (LtA(x)−LtA(x′)=LtA′(x)−LtA′(x′)L_t^{\mathsf{A}}(x) - L_t^{\mathsf{A}}(x') = L_t^{\mathsf{A}'}(x) - L_t^{\mathsf{A}'}(x')) holds across different initializations and step-size bounds, while the cosine similarity between their learned parameters decays as O(1/t)O(1/t) and O(1/t2)O(1/t^2).

Thus, the proof does not use the empirical observations in Sections 4 and 5 as premises; instead, it independently demonstrates that exact RGI can emerge across optimizers and coexist with decaying parameter similarity beyond the NTK and Mean-Field regimes.


We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.


Thank you for the detailed response. I appreciate the clarifications, particularly the distinction between the magnitude of an optimizer's effect and the preservation of relative token-wise losses. I agree that RGI represents a more specific observation than the general understanding that training data plays an important role.

My main remaining concern is the theoretical contribution. While Theorem 2 establishes RGI independently of the experiments, it does so in a highly structured quadratic setting, and its explanatory power for the broader empirical observations remains limited.

Overall, your response has helped clarify the novelty and significance of the empirical findings, although I remain somewhat unconvinced by the theoretical explanation and broader implications. On balance, I am willing to raise my score from 1 to 2 to reflect these clarifications.

Reviewer_wKvYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

the authors introduce RGI: a phenomenon when two models trained differently have the same loss difference between any two tokens, which they show to be equivalent to a constant generalization gap. interestingly, RGI is observable for different optimizers and model architectures, but not necessarily for different data.

Critical strengths

S1: extensive experiments, 6 optimizers, massive sweeps, different architecture modifications tested

S2: the authors complement their empirical findings with theory. for example, they provide nice two-layer quadratic model in which GD, Adam and Muon reach exact RGI while their parameters stay far apart

Critical weaknesses

W1: both metrics reduce to the same statistic. rdiff=12E[Vark(e)/Vark(LA)+Vark(e)/Vark(LB)]r_{diff} = \tfrac12\mathbb E[\mathrm{Var}k(e)/\mathrm{Var}k(L_A) + \mathrm{Var}k(e)/\mathrm{Var}k(L_B)], and 1−rCCC=E[Vark(e)/(Vark(LA)+Vark(LB))]1 - r_{CCC} = \mathbb E[\mathrm{Var}k(e)/(\mathrm{Var}k(L_A) + \mathrm{Var}_k(L_B))], where e=LA(k)−LB(k)e = L_A(k) - L_B(k). so both metrics do not normalize by the mean gap.

W2: data ablations are questionable. models trained on arxiv or code and evaluated on fineweb should rank fineweb text differently. a more informative test would compare training mixtures matched in val loss, or mixtures that differ only partially (e.g., fineweb vs fineweb-edu)

W3: i dont think that thresholds for rCCCr_{CCC} are calibrated.

Questions for authors

Q1: could the authors give a setting where per-sample losses differ and RGI still holds non-trivially under different optimizers? otherwise, what does theorem 2 explain beyond symmetry?

Q2: calibrate rCCCr_{CCC} and rdiffr_{diff} against a weak reference model (e.g., a smaller chekcpoint), to show that the metric can separate models that clearly learned different things

3 comments

We thank the reviewer for your positive review and suggestions, as well as many insightful suggestions.


W1: both metrics reduce to the same statistic. rdiff=1/2E[Vark(e)/Vark(LA)+Vark(e)/Vark(LB)]r_{diff}=1/2 \mathbb{E}[Var_{k}(e)/Var_{k}(L_A)+Var_{k}(e)/Var_{k}(L_B)], and 1−rCCC=E[Vark(e)/(Vark(LA)+Vark(LB))]1-r_{CCC}=\mathbb{E}[Var_{k}(e)/(Var_{k}(L_A)+Var_{k}(L_B))], where e=LA(k)−LB(k)e= L_A(k)-L_B(k). so both metrics do not normalize by the mean gap.


Our metrics rCCCr_{\mathrm{CCC}} and rdiffr_{\mathrm{diff}} assess RGI without requiring normalization by the mean loss gap. Classical Concordance Correlation Coefficient (CCC) measures agreement with the identity line y=xy=x [1]. By applying CCC to centered loss vectors, our rCCCr_{\mathrm{CCC}} extends this criterion to y=x+Cy=x+C, directly matching the RGI definition LA(d)=LB(d)+CA,BL_A(d)=L_B(d)+C_{A,B} for any token-independent constant CA,BC_{A,B}. Similarly, rdiffr_{\mathrm{diff}} uses pairwise differences to eliminate the constant shift and measure the remaining variation across tokens. Both metrics thus quantify departures from a constant gap independently of its magnitude. Normalizing by the mean gap would introduce a dependence on its magnitude that RGI does not require.

W2: data ablations are questionable. models trained on arxiv or code and evaluated on fineweb should rank fineweb text differently. a more informative test would compare training mixtures matched in val loss, or mixtures that differ only partially (e.g., fineweb vs fineweb-edu)


Our experiments already compare closely related web-text corpora, FineWeb and C4, and show that changing the training data can weaken RGI even in this setting. Both datasets are derived from Common Crawl but use different data-curation pipelines. Section 4.3 (Figure 5(a)) and Appendix C.5.3 (Figure 29(a)) show that the FineWeb–C4 comparison exhibits an intermediate rise in rCCCr_{\mathrm{CCC}}, consistent with their shared web-text origin, before declining later in training. Moreover, these experiments include comparisons where both models use Adam and identical architectures, ruling out optimizer and architecture differences as explanations for the observed reduction in concordance.


W3: i dont think that thresholds for rCCCr_{CCC} are calibrated.


Our evidence for approximate RGI is supported by worst-case metric values across random seeds, persistent trends throughout training, and unit-slope regression analysis, rather than calibrated statistical cutoffs.

  1. We assess RGI using worst-case metric values and their trends across training. Specifically, we report the minimum rCCCr_{\mathrm{CCC}} and maximum rdiffr_{\mathrm{diff}} across random seeds (Section 4, Line 186). Their full trajectories show that rCCCr_{\mathrm{CCC}} increases and remains high, while rdiffr_{\mathrm{diff}} decreases and remains low after the initial training phase. These persistent trends support approximate RGI independently of any particular threshold crossing. Thus, rCCC=0.95r_{\mathrm{CCC}}=0.95 and rdiff=0.05r_{\mathrm{diff}}=0.05 serve only as descriptive reference lines.

  2. The high-agreement interpretation is further supported by unit-slope regression analysis. Figures 1(a) and 10 report R2≥0.99R^2\geq 0.99 for fits y=x+Cy=x+C, providing additional quantitative evidence of high agreement.

Q1: could the authors give a setting where per-sample losses differ and RGI still holds non-trivially under different optimizers? otherwise, what does theorem 2 explain beyond symmetry?


Empirically, we demonstrate approximate RGI with heterogeneous per-sample losses; theoretically, Theorem 2 provides a stylized setting in which exact RGI coexists with vanishing parameter cosine similarity across optimizers.

  1. Our experiments already demonstrate approximate RGI with heterogeneous per-sample losses. Each sample is a prefix–target-token pair (Section 2). Figures 1(a) and 10 show broadly varying validation-batch losses that nevertheless closely follow y=x+Cy=x+C across optimizers. Appendix C.3 (Figure 13) further verifies approximate RGI directly on 5M individual tokens with ∣Dval,k∣=1|D_{\mathrm{val},k}|=1, showing that the agreement persists without batch averaging.

  2. Theorem 2’s additional contribution is to show that exact RGI need not imply alignment of raw parameter vectors. Under the stated structured initializations and zero-gradient tie-breaking rules, the theorem constructs trajectories for GD and the simplified Adam/Muon updates whose pairwise parameter cosine similarities decay as O(1/t)O(1/t) or O(1/t2)O(1/t^2) while exact RGI holds. Its scope is therefore a stylized existence result, rather than a full characterization of heterogeneous token lossesy.


Q2: calibrate rdiffr_{diff} and rCCCr_{CCC} against a weak reference model (e.g., a smaller chekcpoint), to show that the metric can separate models that clearly learned different things


Our results already include both a substantially smaller-model reference and data-shift controls, with both metrics distinguishing these settings from the strong-RGI comparisons.

  1. The width-64 ablation provides a substantially weak-model reference. Section 4.2 compares width-64 models against the width-768 baseline in Figure 3c, with corresponding rdiffr_{\mathrm{diff}} results in Appendix C.5.2 (Figure 26c). At the final checkpoint, rCCCr_{\mathrm{CCC}} is approximately 0.40.4–0.90.9 across optimizers, while rdiffr_{\mathrm{diff}} is substantially larger than in the width-384 comparisons. Thus, both metrics distinguish the severe capacity reduction from the moderate size changes under which strong RGI persists.

  2. The metrics also detect training-data shifts with architecture and optimizer held fixed. Section 4.3 includes Adam-versus-Adam comparisons between models trained on FineWeb and models trained on C4, ArXiv, or CodeParrot. Figure 5 and Appendix C.5.3 (Figure 29) show lower late-training rCCCr_{\mathrm{CCC}} and higher rdiffr_{\mathrm{diff}} than the same-data comparisons, particularly for ArXiv and CodeParrot. These controls demonstrate that the metrics detect changes in the relative loss profile induced by different training data.


We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.


References

[1] Lawrence I, Lin K. A concordance correlation coefficient to evaluate reproducibility[J]. Biometrics, 1989: 255-268.

Reviewer_wKvYOpenReview ↗

acknowledged the rebuttal

i thank the authors for the detailed responses, and i apologise for the delayed reply

W3 and Q2 are resolved.

W1. i think, we have a bit of misunderstanding here. my points is that rdiffr_{\text{diff}} and 1−rCCC1 - r_{\text{CCC}} differ only in how the two denominators. so they are not independent

about W2. for two datasets remaining, the validation set is FineWeb, so the loss of RGI conflates a change in relative generalization with OOD eval of the compared model. a sweep of FineWeb vs FineWeb-Edu, would be more meaningful there. i do not consider this a major issue, but it would make the section with experiments considerably stronger

Q1 and Theorem 2. i accept the authors response and share the Reviewer k57U’s view that that its a limited explanation for empirical observations

i am still leaning towards the weak acceptance of the work

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion (Part I)

Dear AC and Reviewers,

We sincerely appreciate your time and effort in reviewing our work. Below, we summarize our discussions with the reviewers.


Reviewer wKvY's main concerns and our responses are summarized as follows.

  • Normalization of metrics (W1): We clarify that our metrics do not require normalization by the mean loss gap, as they directly measure deviations from the constant-gap relationship required by RGI. Specifically, our metric rCCCr_{CCC} is adapted from the classical Concordance Correlation Coefficient (CCC), which is widely used to measure agreement with the identity line. By applying CCC to the centered token-wise losses, we remove the constant mean shift and directly measure deviations from RGI, without requiring further normalization by the mean gap.

  • Calibration of the rCCCr_{CCC} threshold (W3): We use rCCCr_{CCC} thresholds as interpretive guidelines, while our conclusions are supported by multiple complementary diagnostics, including worst-case rCCCr_{CCC} values across random seeds, persistent trends during training, and unit-slope regression analysis. Specifically, the unit-slope regression yields R2>0.99R^2 > 0.99, providing strong additional evidence for the constant-gap relationship required by RGI.

  • RGI with heterogeneous per-sample losses (Q1): Our empirical experiments already demonstrate approximate RGI in settings with heterogeneous per-sample losses. Theorem 2 provides a complementary theoretical result by constructing a stylized setting in which exact RGI coexists with vanishing parameter cosine similarity across optimizers. The key contribution of Theorem 2 is to establish that RGI does not require alignment of the learned parameter vectors. Thus, the theorem serves as a stylized existence result rather than a general characterization of heterogeneous per-sample losses.

  • Validation of metrics against reference baselines (Q2): Our manuscript already includes substantially smaller-model and data-shift controls, with both metrics distinguishing these settings from strong-RGI comparisons. Specifically, Section 4.2 (Figure 3(c)) compares width-64 models against the width-768 baseline, while Section 4.3 compares Adam-trained models on FineWeb against those trained on C4, ArXiv, or CodeParrot. These experiments demonstrate that our metrics can effectively distinguish settings where RGI holds strongly from those where it weakens or fails.

Discussion results:

Despite our active responses to the raised concerns, the reviewer did not engage in the discussion.


Reviewer k57U's main concerns and our responses are summarized as follows.

  • Motivation Regarding Joint Consideration of Training Components (W1). We clarified that we do not claim the three components have never studied jointly. Rather, our goal is to characterize their distinct effects on performance via RGI.

  • Impact of Optimizers and Architectures vs. Training Data (W2). We explained that RGI does not imply optimizers or architectures have limited impact (e.g. Muon vs. Adam follows y=x−0.07y=x-0.07 with R2=0.99R^2=0.99, Figure 1a). Instead, they induce an approximatedly uniform shift across tokens, whereas training data alters relative token-wise losses.

  • Novelty and Specific New Insights (W3&Q1). We highlighted our two insights: (1) seperating uniform loss shifts (across six optimizers, hyperparameters, initializations, moderate architectural/size changes, and validation shifts in Section 4.1-4.4) from relative token structure. (2) showing that RGI persists beyond NTK and Mean-Field regimes (Section 5).

Theoretical Independence, Quadratic Model, and Empirical Verification (Q2 & Follow-ups). We clarified that Theorem 2 derives exact RGI across GD, Adam, and Muon solely from mathematical assumptions, independent of empirical findings. Addressing the reviewer's follow-up, we noted that Assumptions 1 and 2 are empirically supported in LLMs (Appendix F), Theorem 2's conclusions align with Sections 4–5, and quadratic models offer a minimal tractable framework beyond NTK and Mean-Field regimes. Finally, to address the reviewer's Round 3 concern regarding conditions for emergence/breakdown and broader predictive power, we added Appendix G (Figure 37) testing the overparameterization mechanism via hidden dimensions d∈{32,64,192}d \in \{32, 64, 192\} across five optimizers: as predicted, rCCCr_{\text{CCC}} recovers above 0.950.95 at d=192d = 192, but weakens progressively and stays below 0.950.95 at d=64d = 64 and 3232.

Discussion results:

The reviewer actively engaged in the discussion, acknowledged that our response clarified the novelty and significance of the empirical findings, and raised the score. In Round 3, the reviewer asked remaining questions about the conditions for RGI emergence/breakdown in realistic pretraining and the predictive power of Theorem 2. We responded with the new empirical verification in Appendix G (Figure 37) to address this.

AuthorsOpenReview ↗

Summary of Rebuttal and Discussion (Part II)

Reviewer EKMM's main concerns and our responses are summarized as follows.

  • Practical Contributions, Predictions, and Implications (W1, Q3, and follow-up). We explained that RGI provides actionable guidance for model development and accounts for parallel training-loss fluctuations across optimizers and architectures. In response to the reviewer's follow-up requesting concrete predictions and practical decisions, we further highlighted: (1) a conjectured decomposition of LLM representations into token-relative and token-average components; (2) empirical evidence that optimizer and architecture choices do not substantially alter relative topic-wise performance when the training data is fixed; and (3) a practical recommendation to prioritize diverse out-of-distribution benchmarks over simply expanding in-distribution validation sets when comparing optimizers and architectures.

  • Novelty Relative to Same-Seed Zigzags and Chang et al. (W2): We clarified that RGI explains the parallel training-loss fluctuations observed after early training (Appendix D, Figure 35). Unlike Chang et al., who study correlations in token-level learning curves under a fixed optimizer and architecture, RGI requires a stricter unit-slope additive relationship and is observed across different random initializations, optimizers, and architectures. We have added a discussion of these distinctions in Lines 900–908.

  • Clarifying "Effect" and "Isolating Effects" (Q1 & Q2): By "effect," we mean how each training component influences token-wise validation losses, which Proposition 1 decomposes into uniform shifts and changes in relative generalization. Our revision clarified this terminology in the abstract and introduction (Lines 15, 24, 38, and 66). By varying one component at a time (Sections 4.1–4.3), we distinguish the largely uniform shifts associated with optimizer and architectural changes from the changes in relative loss structure induced by training data.

Dissusion Results:

The reviewer acknowledged that our response improved their understanding of the study but requested more concrete predictions and practical implications. We subsequently provided a detailed follow-up addressing these points, but have not received a further response from the reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.