- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that differences in internal representations arising from variations in random seeds during pretraining significantly determine out-of-domain (OOD) generalization during subsequent fine-tuning. The authors observe that when multiple seeds are used for pretraining under the same settings, the pretraining (PT) loss remains nearly identical, yet generalization performance after fine-tuning (FT) varies significantly. They refer to this phenomenon as the representation lottery.
To verify this, they conducted experiments on the World-Data-Model synthetic setup (Park, 2026), either by injecting ground-truth geometry as a representation or by distilling the hidden representations from a “good” seed into a “bad” seed. The results showed that transferring representations improved fine-tuning generalization, whereas distilling only the teacher’s output behavior had almost no effect. Furthermore, when the changes in pretrained representations during FT were regularized, only models with good representations tended to learn and generalize well. Experiments on a synthetic two-hop relational QA task using Pythia-160M/410M also showed that distilling deep-layer representations from a strong seed to a weak seed improved synthetic two-hop out-of-domain (OOD) performance.
Critical strengths
- The experiment is highly reliable, as large-scale multi-seed experiments were conducted under the given setting. Sufficient additional information regarding the experiments is provided in the Appendix.
- The paper reinforces its claims by conducting experiments not only with Qwen2-based models but also with the pretrained Pythia model on synthetic setups and synthetic two-hop relation tasks.
- Through a comparison showing that FT generalization does not improve when only the teacher’s output behavior is matched, but does improve when the hidden representation is transferred, the paper demonstrates effects beyond simple teacher imitation.
Critical weaknesses
- The paper’s claims are too broad given the scope of the experiments. The claims are overly generalized relative to the scope of the experiments, and the scope of the claims should be narrowed. For example, all experiments were conducted on synthetic data. It is necessary to verify whether this “lottery” phenomenon generally occurs with real data as well. Furthermore, since the paper deals only with Transformer architectures and language inputs, experimental results involving architectures such as CNNs and image inputs are also required to generalize the claims.
- Although the paper rigorously verifies the seed variance in PT representations and the resulting changes in downstream task performance through experiments, the phenomenon itself does not feel novel or new. Performance fluctuations due to seeds are a common occurrence in ML model training. For the paper to demonstrate greater novelty, it would require advanced detection methods—such as the ability to select good representation seeds at the PT stage or to determine (without comparison) whether the model achieves absolutely good or bad performance at the FT stage.
- While the phenomenon is diagnosed, the proposed solution is not impressive. The paper introduces a method to improve performance by making the representations of a “bad seed” model similar to those of a “good seed” model; however, this approach is only feasible if a “good seed” model is already available. Paradoxically, if a “good seed” model already exists, there is no reason to use this solution. Therefore, the paper merely highlights the “bad seed” phenomenon without proposing a practically applicable solution.
Questions for authors
- Which weight initialization method did authors use? Gaussian, He, or Xavier? Does the seed lottery phenomenon for weight initialization seed occurs similar across these methods?
6 comments
this approach is only feasible if a ‘good seed’ model is already available. Paradoxically, if a ‘good seed’ model already exists, there is no reason to use this solution.
Yes, representation distillation requires an existing good seed as a teacher, serving strictly as a scientific tool to establish how a favorable pretrained representation geometry facilitates fine-tunability. However, we also provide a practical, general approach that does not require a good seed: representation regularization (Section 5) requires no good seed and uses the model's own states as its regularization target. It increases the OOD accuracy of Pythia-160M from 0.222 to a range of 0.244–0.291, and Pythia-410M from 0.278 to a range of 0.36–0.42. The absolute test in the final row of the table above also relies on this method.
Representation distillation serves as our causal intervention, and even then, it does more than merely copy the good seed: the distilled weak seed ultimately surpasses its teacher (0.403 vs. 0.338 in Pythia-160M). A single teacher can improve 7 out of 8 recipients in just 50k steps, whereas pretraining from scratch requires 656k steps. For a model that represents the two groups separately during pretraining, it can also instill a joint representation, which standard fine-tuning cannot achieve (Section 4.3).
Which weight initialization method did authors use? Gaussian, He, or Xavier? Does the seed lottery phenomenon for weight initialization seed occurs similar across these methods?
The synthetic model uses a Gaussian initialization with zero mean and a standard deviation of 0.1 for all weight matrices and embeddings, with biases initialized to zero. It uses neither He scaling nor Xavier scaling. Across every initialization scheme for which we have data, the seed effect consistently emerges:
| Model | Initialization | Share of Outcome Variance Explained by Pretraining Seed |
|---|---|---|
| Synthetic model, main setup | Gaussian, SD 0.1 | 74% and 90% |
| Synthetic model, second implementation | Gaussian, SD 0.02 | 51% () |
| Pythia-160M and 410M | GPT-NeoX small_init, wang_init on output layers | 30% and 40% |
Calculated using our layer widths, He scaling corresponds to standard deviations of 0.125 and 0.0625, and Xavier scaling corresponds to 0.088 and 0.056, placing our choice of 0.1 squarely within this standard range.
More importantly, after crossing ten initializations with ten data orders, we found that whether a run succeeds or fails depends on the specific combination of the two (App. C.1.3), rather than any particular set of initial weights alone. Thus, the initialization scheme itself is not the driving factor behind this phenomenon. We have now specified the initialization details in the experimental setup section and added this table to the appendix.
In summary, the paper already includes results on real-world data and both types of detection methods you described, and our practical method does not require a good seed.
We appreciate that the reviewer found our experiments highly reliable and recognized that the comparison between behavior and representation transfer demonstrates effects beyond simple teacher imitation, which is one of the core evidences of our paper. The current score appears to rest on three remaining concerns that we believe are largely due to misunderstandings of the text: the paper already includes real-data experiments (Pythia pretrained on The Pile, Section 6), provides a practical method that requires no good seed (representation regularization, Section 5), and presents both types of detection methods requested. We kindly ask the reviewer to reconsider the score in light of these clarifications. If any specific concerns remain that prevent raising the score, please let us know so we can address them directly during the discussion period.
Please feel free to let us know if there are any other questions or concerns!
Thank you for the detailed response. I would like to clarify which points I believe have been addressed and which concerns remain.
- Scope and generality - Major concern.
The experiments are indeed limited to Transformers/language inputs. While the contribution bullets are appropriately scoped, the Introduction still frames the findings broadly in terms of “modern deep learning” and even motivates the question with “language or vision models.” Narrowing down the claim to proper scope would positively affect my assessment.
- Novelty beyond seed variance
I now understand the intent of the work: it separates the sources of variance between PT and FT and causally investigates this phenomenon. I am therefore neutral on my original novelty concern. However, I remain uncertain about the significance of this finding.
- Practical solution
I appreciate the authors’ additional evaluation of good-seed detection during the discussion period. However, apart from the scientific analysis of PT-seed variance, the proposed methods still seem far from a practical solution. For good/bad seed detection to provide practical benefits, it should ideally reduce PT cost by identifying seeds early during PT, or reliably assess a single completed PT checkpoint. The methods in Figure R1 instead operate after the relevant PT computation has already been performed. Representation regularization also does not identify good PT seeds; it preserves representations already acquired during PT while fine-tuning. Moreover, closely related representation-preserving techniques have already been studied in the field of continual learning and adaptation [1,2]. [1] Hua, Hang, et al. "Improving pretrained language model fine-tuning with noise stability regularization." IEEE Transactions on Neural Networks and Learning Systems 36.1 (2023): 1898-1910. [2] Jung, Heechul, et al. "Less-forgetting learning in deep neural networks." arXiv preprint arXiv:1607.00122 (2016).
Overall, I believe the paper would benefit from a narrower and sharper statement of its central claim. The mechanistic finding itself may be sufficient; framing it more precisely around what the experiments actually establish, rather than implying broader generality or practical applicability.
Dear reviewer,
Thank you for your careful review!
We are very grateful that the reviewer found our experiments "highly reliable" and noted that the comparison between behavioral transfer and representation transfer demonstrates "effects beyond simple teacher imitation."
The reviewer primarily raised three points: the claims are overly broad relative to the scope of the experiments, the phenomenon requires detection methods to be novel, and the method appears to require a good seed. We believe some of these points stem from misunderstandings of the paper. Our detailed responses are below.
Yes, our core experiments are based on synthetic data, as this provides an ideal controlled starting point for understanding this question and sweeping across seeds. However, in Section 6, we do use Pythia-160M and 410M, which are pretrained on The Pile, a real-world text corpus; only the fine-tuning task is synthetic. Even in this real-data setting, the pretraining seed still accounts for 30% and 40% of the outcome variance, and both representation distillation and representation regularization improve out-of-distribution (OOD) generalization.
Regarding CNNs and image inputs: the core question of this paper is how language-model pretraining affects generalization after fine-tuning. Switching to another modality and architecture alters the research question itself, not merely its scope. We explicitly note in the limitations that other architectures and image inputs have not yet been tested, and we clearly specify the corresponding experimental setting wherever a claim is made.
We do in fact provide both types of detection methods you described. None of them use any OOD data, and all are evaluated across 277 pretraining seeds of the synthetic model:
In contrast, pretraining loss, zero-shot accuracy on the task (Fig. R1a), ten static statistics of the representation, loss-landscape sharpness, and gradient alignment all fail to identify good seeds.
Moreover, this phenomenon differs fundamentally from standard seed noise. Seed variance is typically treated as noise fluctuating around a single mean, whereas we find that outcome variance stems primarily from the pretraining seed rather than the fine-tuning seed (accounting for 74% and 90% in the two grids). While the pretraining loss across the 278 runs differs by no more than 5%, FT generalization differs by a factor of seven. Furthermore, the cause can be localized directly to the representation, as distilling a good seed's representation transfers generalization ability, whereas distilling its behavior does not.
To be continued [1/2]