- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates a metric, the Relative Generalization Invariance (RGI), which is defined on a pair of two models whose every NTP loss on a validation set has a constant difference. The paper empirically shows when and how strongly such an invariance holds by varying the training conditions, e.g., optimizers, architectures, and data stream. As the RGI tends to hold when the data stream is fixed, it may be a metric for analyzing the effect of other factors.
Critical strengths
- RGI seems an interesting observation.
- Experiments seems reasonable and sound enough.
Critical weaknesses
- The paper may not emphasize practical contributions about RGI. It would be strong if interesting or useful discussions beyond "just finding something" are provided.
- In my feeling, RGI is possibly not very novel. Practitioners may know that two models show very similar training curve with similar zigzags when training them using the same seed (basically, the same init. and data stream), which might be a partial observation of RGI (especially for the first epoch). I also found some related literature (e.g., Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability, https://arxiv.org/abs/2308.15419).
Questions for authors
(abst. and similarly for intro.) We take a first step toward isolating their effects by studying relative generalization. ... our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.
What is the "effect" concretely? Can we rephrase it more concretely?
Furthermore, does this phenomenon really help distinguish the effects of optimizers and architectures from those of training data (stream?)? In other words, what is "to distinguish the effects ..." or "isolating their effects" concretely? And, is it something practical or important?
4 comments
(abst. and similarly for intro.) We take a first step toward isolating their effects by studying relative generalization. ... our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.
Q1. What is the "effect" concretely? Can we rephrase it more concretely?
Concretely, "effect" means the influence of each training component on generalization, as measured by the token-wise validation losses . By Proposition 1 (Section 3), a component's influence on validation generalization can take two forms:
- A uniform shift in the overall validation-loss level across all tokens , or
- A change in relative generalization, namely the validation-loss differences between any two tokens and .
We have revised the paper's abstract and introduction accordingly (Lines 15, 24, 38, and 66).
Q2. Furthermore, does this phenomenon really help distinguish the effects of optimizers and architectures from those of training data (stream?)? In other words, what is "to distinguish the effects ..." or "isolating their effects" concretely?
Yes. RGI distinguishes their influences by testing whether changing a single component preserves pairwise token-loss differences or alters the relative loss structure across tokens. By Proposition 1 (Section 3), RGI between two training triplets and is equivalent to a token-independent constant gap: Concretely, we vary one component at a time in Sections 4.1 to 4.3 and test whether this additive relationship holds:
-
Optimizers and moderate architectural variations preserve relative generalization: Across all six tested optimizers (Section 4.1, Figure 1, e.g., with for Adam vs. Muon) and moderate architectural variations in attention, FFN, and width (Section 4.2, Figures 3 and 4), approximate RGI holds ( and ). Their influence on token-wise validation losses is therefore captured by an approximately uniform shift across tokens.
-
Training data alters relative generalization: In contrast, changing the training data source (C4, ArXiv, or CodeParrot vs. FineWeb in Section 4.3, Figure 5, even with Adam and the architecture held fixed) substantially weakens or breaks RGI (), showing that the training data reshapes the relative loss structure across tokens.
Q3. And, is it something practical or important?
Distinguishing these two forms of influence is both conceptually important for understanding pretraining generalization and practically useful for model development and diagnostics.
-
Conceptual importance: Although the optimizer, architecture, and training data jointly determine validation performance, RGI shows that they govern different aspects of generalization. Even when different optimizers move far from initialization and converge to parameters and hidden states with low similarity (Section 5.1, Figure 7), they preserve the same relative token-wise loss structure, which is instead shaped by the training data.
-
Practical relevance: First, for model evaluation and development, because optimizer and moderate architectural improvements act as an approximately uniform shift across tokens (which also transfers across moderate validation shifts in Section 4.4, Figure 6), lowering validation loss via optimizer or architectural tuning improves performance consistently across token subsets without hidden trade-offs, whereas changing relative performance on specific domains or token types requires modifying the training data. Second, for training diagnostics, RGI explains why different optimizers and architectures trained on the same data stream exhibit parallel training-loss curves with shared local spikes (Appendix D, Figure 35).
We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.
References
[1] Chang T A, Tu Z, Bergen B K. Characterizing learning curves during language model pre-training: Learning, forgetting, and stability[J]. Transactions of the Association for Computational Linguistics, 2024, 12: 1346-1362.
Thank you for the response. It has helped me better understand the study. However, my concern about its significance remains.
The proposed practical implications are still abstract. For optimizer tuning, it is unclear when token-subset trade-offs are a practical concern that RGI helps resolve. Likewise, modifying training data to improve a specific domain is a natural strategy even without RGI. What concrete difficulty does this finding address, or what decision does it change?
An immediate application is not necessary, but I would like to understand what we learn beyond the loss relationship itself. Could you give a concrete example of a useful prediction or a large change in our understanding of pretraining that follows from RGI?
We sincerely thank you for your reading and replying to our response, as well as for raising insightful follow-up comments.
The proposed practical implications are still abstract. For optimizer tuning, it is unclear when token-subset trade-offs are a practical concern that RGI helps resolve. Likewise, modifying training data to improve a specific domain is a natural strategy even without RGI. What concrete difficulty does this finding address, or what decision does it change?
An immediate application is not necessary, but I would like to understand what we learn beyond the loss relationship itself. Could you give a concrete example of a useful prediction or a large change in our understanding of pretraining that follows from RGI?
Our main contributions are the formal definition and empirical characterization of RGI, together with demonstrating that RGI persists beyond the NTK and MF regimes. Although a comprehensive investigation of downstream applications is beyond the scope of this conference paper, we highlight several practical implications of RGI below.
Prediction:
- A new decomposition of LLM representations. Existing interpretability methods, such as sparse autoencoders, decompose LLM representations into features associated with different semantic concepts or domains. Motivated by RGI, we conjecture that LLM representations may admit a complementary decomposition into token-relative and token-average components. Specifically, RGI shows that changes in optimizers and moderate architectural choices approximately preserve relative token-wise losses while shifting their average. We hypothesize that a similar phenomenon may occur in representation space: the token-relative component remains approximately invariant across different optimizers and architectures, while the token-average component captures the effects of these training choices. Although our results in Section 5.1 show low overall similarity between representations learned using different optimizers, they do not examine whether such a decomposition could reveal an invariant component. Formally defining these components and empirically testing this hypothesis remain important directions for future work.
Practical implications:
-
Separating architecture and optimizer tuning from data curation. In practice, LLM developers often introduce new architectures, optimizers, and curated domain-specific data mixtures simultaneously. A natural question is whether changes in architectures or optimizers introduce topic-specific biases that could undermine the intended benefits of data curation. Our RGI results provide evidence against such biases when the training data stream is held fixed. Specifically, changes in optimizers and moderate architectural choices shift token-wise validation losses approximately uniformly, rather than systematically favoring or penalizing tokens from particular topics. This suggests that architecture and optimizer tuning can improve overall performance without substantially altering the relative loss structure across different token subsets.
-
Evaluation guidance for new optimizers and architectures. How should practitioners design evaluation datasets when comparing new optimizers and architectures? Our RGI findings suggest that a relatively small in-distribution validation set can be sufficient to estimate overall performance improvements, because the loss differences across models are approximately uniform across tokens. Therefore, practitioners can allocate more evaluation resources to diverse Out-Of-Distribution (OOD) datasets, where RGI may be weaker and domain-specific performance differences are more likely to emerge.
RGI provides practical insights for LLM pretraining: optimizer and moderate architectural choices primarily shift overall validation loss while approximately preserving relative token losses, whereas changing the training data can alter the relative loss structure across tokens. It leads to the following practical implications.
Actionable guidance for model development: Because changing the optimizer (Section 4.1, Figures 1 and 2) or making moderate architectural changes (Section 4.2, Figures 3 and 4) shifts validation losses by a token-independent constant C(T,T′,t), one can tune optimizers or attention/FFN modules to lower the global validation loss without worrying about trading off performance across different token subsets. Conversely, if a practitioner wants to improve generalization on specific domains or token types relative to others, tuning the optimizer will not change the relative loss structure; one must modify the training data stream (Section 4.3, Figure 5).
Explaining training-curve diagnostics: RGI also explains why different optimizers and architectures trained on the same data stream exhibit similar training-loss curves and shared local spikes (Appendix D, Figure 35). Specifically, we decompose adjacent-step training-loss changes into optimizer-update effects and batch-induced effects, finding that the latter are approximately 50 times larger (Figure 35(b)). Since RGI implies that these batch-induced loss differences are approximately invariant across optimizers and architectures, their training-loss curves exhibit similar local fluctuations.
RGI goes substantially beyond the observation of similar training-curve zigzags under the same seed and the correlation analysis in Chang et al.
Beyond same-seed training zigzags: Rather than being a restatement of parallel training curves, RGI is the underlying mechanism that explains why those parallel zigzags occur after the early stage of training (Appendix D, Figure 35). In addition, the notion RGI is beyond the typical setting, where models share the same initialization. It holds across most optimizers and a wide range of moderate architecture variations.
Distinction from Chang et al.: Chang et al. characterize token-level learning curves and their reproducibility across random initializations and data shuffles under a fixed architecture and training recipe. Our work differs in both the invariance criterion and the training configurations investigated.
A more stringent invariance criterion: Pearson correlation equals 1 whenever LB(d)=aLA(d)+b for any a>0, whereas RGI requires a unit-slope additive shift, LB(d)=LA(d)+C, with C independent of d (Proposition 1). We provide empirical evidence for approximate RGI through the unit-slope fit y=x−0.07 (R2=0.99, Figure 1(a)) and our RGI diagnostics.
Invariance across optimizers and architectures: While Chang et al. fix the optimizer and architecture, we show that approximate RGI persists across distinct optimizer geometries, including Adam, Muon, and SOAP, and moderate architectural variations. In contrast, changing the training data stream can substantially alter relative generalization.
We thank the reviewer for this reference and haved discussed Chang et al. [1] in related works (Lines 900-908) of the revised version.