acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Representation Lottery Determines Fine-Tuning Generalization

11 views

Abstract

Large scale pretraining (PT) followed by fine-tuning (FT) is the central paradigm of modern deep learning, yet how PT shapes FT generalization remains poorly understood. Here, we use a synthetic setup in which the ground-truth concept geometry of entities is known, so we can study how representational geometry affects model behavior. We first show that models differing only in training seed can closely match in PT loss yet vary substantially in FT generalization. We hypothesize that a representational lottery underlies this phenomenon: the random seed dependent geometry formed during PT determines what FT can reach. We confirm this causally. First, injecting the ground-truth concept geometry into the model yields near-ceiling FT generalization, showing that good representations directly drive generalization. Then, distilling a lucky seed’s learned representation into an unlucky seed unlocks FT generalization, while distilling the same model’s behavior does not. Furthermore, representation distillation can merge entities that never interacted during PT, which FT alone cannot. Finally, we show that regularizing representational drift during FT enables naturalistic learning: only models with good representations learn the task, and they generalize, while models with poor representations fail to fit it. Under plain FT, by contrast, models with poor representations fit the training objective by destroying their pretrained representations, causing catastrophic forgetting. We reproduce the distillation result in Pythia-160M and Pythia-410M, which were each pretrained multiple times with different random seeds, and show that the transferable representation lives in the deep layers at both scales. Overall, PT acts as a lottery over representations, and the representation a model draws determines how it adapts during FT in a way that behavior alone does not explain.

Primary area:
interpretability and explainable AI
Keywords:
pretraining, fine-tuning, representational geometry, seed variance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aS23OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that differences in internal representations arising from variations in random seeds during pretraining significantly determine out-of-domain (OOD) generalization during subsequent fine-tuning. The authors observe that when multiple seeds are used for pretraining under the same settings, the pretraining (PT) loss remains nearly identical, yet generalization performance after fine-tuning (FT) varies significantly. They refer to this phenomenon as the representation lottery.

To verify this, they conducted experiments on the World-Data-Model synthetic setup (Park, 2026), either by injecting ground-truth geometry as a representation or by distilling the hidden representations from a “good” seed into a “bad” seed. The results showed that transferring representations improved fine-tuning generalization, whereas distilling only the teacher’s output behavior had almost no effect. Furthermore, when the changes in pretrained representations during FT were regularized, only models with good representations tended to learn and generalize well. Experiments on a synthetic two-hop relational QA task using Pythia-160M/410M also showed that distilling deep-layer representations from a strong seed to a weak seed improved synthetic two-hop out-of-domain (OOD) performance.

Critical strengths
  1. The experiment is highly reliable, as large-scale multi-seed experiments were conducted under the given setting. Sufficient additional information regarding the experiments is provided in the Appendix.
  2. The paper reinforces its claims by conducting experiments not only with Qwen2-based models but also with the pretrained Pythia model on synthetic setups and synthetic two-hop relation tasks.
  3. Through a comparison showing that FT generalization does not improve when only the teacher’s output behavior is matched, but does improve when the hidden representation is transferred, the paper demonstrates effects beyond simple teacher imitation.
Critical weaknesses
  1. The paper’s claims are too broad given the scope of the experiments. The claims are overly generalized relative to the scope of the experiments, and the scope of the claims should be narrowed. For example, all experiments were conducted on synthetic data. It is necessary to verify whether this “lottery” phenomenon generally occurs with real data as well. Furthermore, since the paper deals only with Transformer architectures and language inputs, experimental results involving architectures such as CNNs and image inputs are also required to generalize the claims.
  2. Although the paper rigorously verifies the seed variance in PT representations and the resulting changes in downstream task performance through experiments, the phenomenon itself does not feel novel or new. Performance fluctuations due to seeds are a common occurrence in ML model training. For the paper to demonstrate greater novelty, it would require advanced detection methods—such as the ability to select good representation seeds at the PT stage or to determine (without comparison) whether the model achieves absolutely good or bad performance at the FT stage.
  3. While the phenomenon is diagnosed, the proposed solution is not impressive. The paper introduces a method to improve performance by making the representations of a “bad seed” model similar to those of a “good seed” model; however, this approach is only feasible if a “good seed” model is already available. Paradoxically, if a “good seed” model already exists, there is no reason to use this solution. Therefore, the paper merely highlights the “bad seed” phenomenon without proposing a practically applicable solution.
Questions for authors
  1. Which weight initialization method did authors use? Gaussian, He, or Xavier? Does the seed lottery phenomenon for weight initialization seed occurs similar across these methods?
6 comments

Dear reviewer,

Thank you for your careful review!

We are very grateful that the reviewer found our experiments "highly reliable" and noted that the comparison between behavioral transfer and representation transfer demonstrates "effects beyond simple teacher imitation."

The reviewer primarily raised three points: the claims are overly broad relative to the scope of the experiments, the phenomenon requires detection methods to be novel, and the method appears to require a good seed. We believe some of these points stem from misunderstandings of the paper. Our detailed responses are below.


The paper’s claims are too broad given the scope of the experiments. ... For example, all experiments were conducted on synthetic data. ... experimental results involving architectures such as CNNs and image inputs are also required to generalize the claims.

Yes, our core experiments are based on synthetic data, as this provides an ideal controlled starting point for understanding this question and sweeping across seeds. However, in Section 6, we do use Pythia-160M and 410M, which are pretrained on The Pile, a real-world text corpus; only the fine-tuning task is synthetic. Even in this real-data setting, the pretraining seed still accounts for 30% and 40% of the outcome variance, and both representation distillation and representation regularization improve out-of-distribution (OOD) generalization.

Regarding CNNs and image inputs: the core question of this paper is how language-model pretraining affects generalization after fine-tuning. Switching to another modality and architecture alters the research question itself, not merely its scope. We explicitly note in the limitations that other architectures and image inputs have not yet been tested, and we clearly specify the corresponding experimental setting wherever a claim is made.

the phenomenon itself does not feel novel or new. ... it would require advanced detection methods—such as the ability to select good representation seeds at the PT stage or to determine (without comparison) whether the model achieves absolutely good or bad performance at the FT stage.

We do in fact provide both types of detection methods you described. None of them use any OOD data, and all are evaluated across 277 pretraining seeds of the synthetic model:

Detection MethodTypeCostResult
Linear probe on the pretrained modelSelects seeds at the pretraining stageOne forward passAUC 0.91 for seeds that will generalize under regularized FT; the top decile of seeds captures 10 of the 13 good seeds
In-distribution accuracy after 4% of FTAbsolute test with a fixed threshold4% of the FT budgetAUC 0.94; precision 0.64 and recall 0.61 on held-out seeds
In-distribution accuracy under regularized FTAbsolute test with a fixed thresholdOne FT runAUC 0.95; precision 0.83 and recall 0.71 on held-out seeds

In contrast, pretraining loss, zero-shot accuracy on the task (Fig. R1a), ten static statistics of the representation, loss-landscape sharpness, and gradient alignment all fail to identify good seeds.

Moreover, this phenomenon differs fundamentally from standard seed noise. Seed variance is typically treated as noise fluctuating around a single mean, whereas we find that outcome variance stems primarily from the pretraining seed rather than the fine-tuning seed (accounting for 74% and 90% in the two grids). While the pretraining loss across the 278 runs differs by no more than 5%, FT generalization differs by a factor of seven. Furthermore, the cause can be localized directly to the representation, as distilling a good seed's representation transfers generalization ability, whereas distilling its behavior does not.


To be continued [1/2]

this approach is only feasible if a ‘good seed’ model is already available. Paradoxically, if a ‘good seed’ model already exists, there is no reason to use this solution.

Yes, representation distillation requires an existing good seed as a teacher, serving strictly as a scientific tool to establish how a favorable pretrained representation geometry facilitates fine-tunability. However, we also provide a practical, general approach that does not require a good seed: representation regularization (Section 5) requires no good seed and uses the model's own states as its regularization target. It increases the OOD accuracy of Pythia-160M from 0.222 to a range of 0.244–0.291, and Pythia-410M from 0.278 to a range of 0.36–0.42. The absolute test in the final row of the table above also relies on this method.

Representation distillation serves as our causal intervention, and even then, it does more than merely copy the good seed: the distilled weak seed ultimately surpasses its teacher (0.403 vs. 0.338 in Pythia-160M). A single teacher can improve 7 out of 8 recipients in just 50k steps, whereas pretraining from scratch requires 656k steps. For a model that represents the two groups separately during pretraining, it can also instill a joint representation, which standard fine-tuning cannot achieve (Section 4.3).

Which weight initialization method did authors use? Gaussian, He, or Xavier? Does the seed lottery phenomenon for weight initialization seed occurs similar across these methods?

The synthetic model uses a Gaussian initialization with zero mean and a standard deviation of 0.1 for all weight matrices and embeddings, with biases initialized to zero. It uses neither He scaling nor Xavier scaling. Across every initialization scheme for which we have data, the seed effect consistently emerges:

ModelInitializationShare of Outcome Variance Explained by Pretraining Seed
Synthetic model, main setupGaussian, SD 0.174% and 90%
Synthetic model, second implementationGaussian, SD 0.0251% (p=2×10−6p = 2 \times 10^{-6})
Pythia-160M and 410MGPT-NeoX small_init, wang_init on output layers30% and 40%

Calculated using our layer widths, He scaling corresponds to standard deviations of 0.125 and 0.0625, and Xavier scaling corresponds to 0.088 and 0.056, placing our choice of 0.1 squarely within this standard range.

More importantly, after crossing ten initializations with ten data orders, we found that whether a run succeeds or fails depends on the specific combination of the two (App. C.1.3), rather than any particular set of initial weights alone. Thus, the initialization scheme itself is not the driving factor behind this phenomenon. We have now specified the initialization details in the experimental setup section and added this table to the appendix.


In summary, the paper already includes results on real-world data and both types of detection methods you described, and our practical method does not require a good seed.

We appreciate that the reviewer found our experiments highly reliable and recognized that the comparison between behavior and representation transfer demonstrates effects beyond simple teacher imitation, which is one of the core evidences of our paper. The current score appears to rest on three remaining concerns that we believe are largely due to misunderstandings of the text: the paper already includes real-data experiments (Pythia pretrained on The Pile, Section 6), provides a practical method that requires no good seed (representation regularization, Section 5), and presents both types of detection methods requested. We kindly ask the reviewer to reconsider the score in light of these clarifications. If any specific concerns remain that prevent raising the score, please let us know so we can address them directly during the discussion period.

Please feel free to let us know if there are any other questions or concerns!

Thank you for the detailed response. I would like to clarify which points I believe have been addressed and which concerns remain.

  1. Scope and generality - Major concern.

The experiments are indeed limited to Transformers/language inputs. While the contribution bullets are appropriately scoped, the Introduction still frames the findings broadly in terms of “modern deep learning” and even motivates the question with “language or vision models.” Narrowing down the claim to proper scope would positively affect my assessment.

  1. Novelty beyond seed variance

I now understand the intent of the work: it separates the sources of variance between PT and FT and causally investigates this phenomenon. I am therefore neutral on my original novelty concern. However, I remain uncertain about the significance of this finding.

  1. Practical solution

I appreciate the authors’ additional evaluation of good-seed detection during the discussion period. However, apart from the scientific analysis of PT-seed variance, the proposed methods still seem far from a practical solution. For good/bad seed detection to provide practical benefits, it should ideally reduce PT cost by identifying seeds early during PT, or reliably assess a single completed PT checkpoint. The methods in Figure R1 instead operate after the relevant PT computation has already been performed. Representation regularization also does not identify good PT seeds; it preserves representations already acquired during PT while fine-tuning. Moreover, closely related representation-preserving techniques have already been studied in the field of continual learning and adaptation [1,2]. [1] Hua, Hang, et al. "Improving pretrained language model fine-tuning with noise stability regularization." IEEE Transactions on Neural Networks and Learning Systems 36.1 (2023): 1898-1910. [2] Jung, Heechul, et al. "Less-forgetting learning in deep neural networks." arXiv preprint arXiv:1607.00122 (2016).

Overall, I believe the paper would benefit from a narrower and sharper statement of its central claim. The mechanistic finding itself may be sufficient; framing it more precisely around what the experiments actually establish, rather than implying broader generality or practical applicability.

Reviewer_WcHhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work aims at understand how pretraining (PT) loss shapes downstream fine-tuning (FT) generalization through representations. The authors specifically show that the representations a model learns during PT determines what FT can achieve and that such good representations can be distilled into another model, to move improve its generalization capabilities under finetuning. The study is demonstrated on a toy task that considers diverse geometric tasks on a model of city world coordinates and treats an LLM setup as well.

Critical strengths
  • S1: This is a well motivated problem, understanding how representations of different PT models that only vary in seed (i.e data ordering and model initialization) are affecting FT generalization is important. It connects two areas that are not always described together, but influence each other strongly.
Critical weaknesses
  • W1: While the overall questions the authors are trying to answer are interesting and important, I think the experiments could be more thorough to support the claims being made.
  • W2: The LLM experiments appear to differ from the setups described in the toy experiments, making it difficult to establish a direct connection between the two. This weakens the extent to which the LLM experiments support the paper's broader claims. More experimental results on the LLM setup would be helpful.
  • W3: The clarity of the paper could be improved, especially in Sections 4, 5, and 6. I found some of the experimental setups difficult to follow, even after reading the relevant details in the appendix.
Questions for authors
  • Q1: I would intuitively predict that pretraining OOD accuracy might be a better measure of fine-tuning generalization than loss, because the representations needed to perform a task (that we wish to FT) might have already formed. Distance task error was reported in Figure 9(b), but did you also do any analysis of how PT models perform on the sangle task? A good representation in a PT model probably means that the model already has some information needed to perform other OOD tasks. In a real-world setting, I would actually test whether my model is already capable of performing the task before fine-tuning it. It would be really nice to see the accuracies of PT models on the sangle task without any fine-tuning and whether their performance is already sufficient to perform the task.
  • Q2: After learning a linear map to obtain a 128-dimensional representation, in Sec 4.1 line 243-244 the authors mention, “which we add to the residual stream at the city’s last id token, at the input of every layer after the first, throughout PT and FT.”
  • Q2.1: Could you clarify this further? How are the model weights being updated during PT and FT while also performing interventions at a particular layer? Are some parts of the model frozen?
  • Q2.2: Is the type of intervention with the oracle representation an additive intervention or does it completely replace the existing representation? How are hyperparameters frozen?
  • Q2.3: If the weights are being updated while the same vector is injected at every transformer layer, what is the role of these weight updates, especially if the input is being overridden?
  • Q3: The intervention experiment in sec 4.1 shows that by simply intervening with an oracle representation the FT accuracy improves. Using the learned representations via distillation in the subsequent steps (sec 4.2, 4.3) supports the claim that the teacher contains similarly useful representations if they come from a good seed. However, it should be shown whether this is better/worse than using the oracle representation for distillation directly, to understand the advantage of the teacher data better.
  • Q4: In Sec 4.2 distilling is done from a FT model representation to PT model. And from my understanding you distill this representation to a student (bad seed) model during the pretraining process as well as fine-tuning process. From App. C.4.1, it looks like this ablation performed the best. However, if distillation is needed during fine-tuning as well, rather than just during pretraining, wouldn't this weaken the initial claim in Section 4.1 that having a good representation in the PT model is sufficient for good fine-tuning generalization?
  • Q5: For results with pythia models (in sec 6), distillation is done only during FT not during PT, but the method that worked best in toy models is when both PT and FT models were distilled. Do you have any intuition why distilling only the FT model didn’t work as well in toysetup but seems to work well in LLM setup?

Minor:

  1. Figure 2c, I cannot find the super tier points and in Fig 2(d) “Boundary” and “Bad” tiers have three and two different groups of points respectively, what do they mean?
  2. Throughout the paper when pretraining loss is mentioned is training loss or validation loss?
  3. Evaluation accuracy within 10 deg seems a little arbitrary.
  4. “It is generally believed that the rich representations built during PT are the main driver of FT performance.” - it would be nice to add citations to support this.
3 comments

Dear reviewer,

Thank you for the reviewer’s careful review!

We are very glad that the reviewer thinks the problem is “well motivated”, and thinks the paper “connects two areas that are not always described together”.

The reviewer mainly pointed out that the experiments could be more thorough, the language-model experiments are hard to match with the synthetic experiments, and Sections 4 to 6 are hard to understand. Our replies are below.


I think the experiments could be more thorough to support the claims being made.

For each of your questions (Q1, Q3, Q4 and Q5), we did one new experiment, and we also added two new language-model results. The results are listed under each point below.

The LLM experiments appear to differ from the setups described in the toy experiments, making it difficult to establish a direct connection between the two. This weakens the extent to which the LLM experiments support the paper’s broader claims. More experimental results on the LLM setup would be helpful.

Thank you for pointing this out. Now the experiment protocol of each setting has been run once in the other setting, and the results are consistent. The one-stage protocol of Section 6 is run in the synthetic model (see Q4 and Q5 below), and the behavior-distillation control of Section 4.2 is run in Pythia.

We also added two new language-model results. One is about behavior distillation in Pythia. We fine-tune a weak seed and add an extra KL loss, which pushes it toward the next-token outputs of the teacher used in the RD experiments (three recipients, three loss weights, four FT seeds each). Under every loss weight, the student model in the end agrees with the teacher on 92% to 96% of next tokens, while originally it was only about 20%. But its OOD accuracy is still close to plain FT: at 160M it is 0.26 to 0.28, plain FT is 0.22, and RD is 0.40 (Fig. R2). In Section 6, each pretraining seed now has three more FT seeds. Under 80 runs, the mean of each seed is 0.223 to 0.292 at 160M, and 0.286 to 0.475 at 410M.

The clarity of the paper could be improved, especially in Sections 4, 5, and 6. I found some of the experimental setups difficult to follow, even after reading the relevant details in the appendix.

Totally agree. The review also let us see the problems in our writing. The review summary describes the paper as a study about how PT loss affects FT generalization, but our point is that runs with the same PT loss have very different fine-tuning results. Also, Q4 understands the teacher of Section 4.2 as a fine-tuned model, but it is actually the pretrained model. Both of these are problems of our writing.

It would be really nice to see the accuracies of PT models on the sangle task without any fine-tuning and whether their performance is already sufficient to perform the task.

Thank you for this suggestion. We now did this test on all 278 seeds. The zero-shot accuracy of every seed is at chance level, so it cannot tell the seeds apart. The mean is 0.061, chance level is 0.056, no seed is above 0.125, and it is unrelated to FT generalization (r = 0.06). This task never appears in pretraining, so the model’s behavior cannot reflect what is in its representation.

There are three measures that can actually predict generalization, and none of them uses OOD data (Fig. R1):

  • Representation. A linear probe trained on the frozen pretrained model can predict generalization under regularized FT, r = 0.64 (App. C.2.1). This probe is a ridge regression, which predicts the answer from the hidden state at the “=” token in sangle prompts, and it is only fit on in-distribution prompts.
  • Early learning speed. The in-distribution accuracy at the first 4% of FT correlates with FT generalization at r = 0.73, and it can separate the 15 good seeds from the other 262 with AUC 0.94.
  • Using regularized FT as a test. The in-distribution accuracy under regularized FT, with a fixed threshold of 0.70, can separate the seeds that will generalize with AUC 0.95.

To be continued [1/3]

Q2.1: How are the model weights being updated during PT and FT while also performing interventions at a particular layer? Are some parts of the model frozen?

No part is frozen. A linear map maps the true coordinates of a city to a 128-dimensional vector. This vector is added to the hidden state at the city’s last id token, at the input of every layer after the first layer. The transformer and this map are trained together with the next-token loss, in both PT and FT.

Q2.2: Is the type of intervention with the oracle representation an additive intervention or does it completely replace the existing representation? How are hyperparameters frozen?

The intervention is additive: the state h becomes h + W(x, y). The three conditions share all hyperparameters.

Q2.3: If the weights are being updated while the same vector is injected at every transformer layer, what is the role of these weight updates, especially if the input is being overridden?

The input is not overridden. The injected vector tells the model where each city is, but the weights still need to learn the computations that use it. With shuffled coordinates, FT generalization is 0.17, the same as no injection; with true coordinates it is 0.90. The protocol table at the start of Section 4 now lists these details.

it should be shown whether this is better/worse than using the oracle representation for distillation directly, to understand the advantage of the teacher data better.

We did this comparison. The representation learned by a good seed is the better target.

We used the two-stage protocol of Section 4.2 to distill the oracle representation (a linear map of each city’s true coordinates) into the two recipients of Fig. 4. The true map is successfully installed into the recipients (fidelity 0.98 and 0.86), but FT generalization still stays at the level of plain FT: 0.096 and 0.090, while plain FT is 0.096 and 0.075. The good seed’s representation can reach 0.28 and 0.27.

This is consistent with Section 4.1. In Section 4.1, the coordinates are an input from the start of pretraining, so the model learned to compute with them. But a map written into one layer of a pretrained model does not come with these computations.


To be continued [2/3]

if distillation is needed during fine-tuning as well, rather than just during pretraining, wouldn’t this weaken the initial claim in Section 4.1 that having a good representation in the PT model is sufficient for good fine-tuning generalization?

This does not weaken the claim. First, one clarification: the teacher in Section 4.2 is the pretrained model of the good seed, not a fine-tuned model.

In Section 4.1, the coordinates are an input at every step, so FT cannot overwrite them. But the distilled representation lives in the weights, and plain FT rewrites it. The loss during FT keeps it unchanged. We measured this directly: we took the two recipients of Fig. 4 after the install stage, fine-tuned them with and without the RD loss, two FT seeds each, and after each FT segment, computed the cosine similarity between the recipient’s and the teacher’s layer-5 states of all cities. Without the loss, the similarity drops from 0.95 to 0.39 within the first 1,750 steps; with the loss it ends at 0.98. FT generalization ends at 0.16 and 0.11 without the loss, and 0.28 and 0.26 with the loss. This is consistent with the stage ablation in App. C.4.1, where plain FT after the install stage gives 0.13 and 0.10.

The loss itself also does not produce this effect. Two controls both stay at the level of plain FT: RD toward the recipient’s own states, and behavior distillation. Under the two-stage protocol, this is Fig. 4b of the paper. Under the one-stage protocol of Section 6 (which we also ran in the synthetic model), both controls give 0.06 to 0.09, plain FT gives 0.08 to 0.10, and with the teacher’s states as the target it gives 0.18 and 0.29.

Do you have any intuition why distilling only the FT model didn’t work as well in toysetup but seems to work well in LLM setup?

In the synthetic model, one stage also works for one of our two recipients. Two stages work for both, so the paper used two stages in the synthetic model.

The two experiments use different protocols. Section 4.2 (synthetic model) is two-stage: first RD on pretraining text, then FT while keeping the RD loss, and the teacher is a pretrained model. Section 6 (Pythia) is one-stage: one FT run with the RD loss on the whole time, and the teacher itself has already been fine-tuned on the task. We ran two one-stage forms in the synthetic model:

RD protocol in the synthetic modelRecipient ARecipient B
Plain FT, no RD0.100.08
(a) One stage, fine-tuned teacher (Section 6 protocol)0.180.29
(b) One stage, pretrained teacher, same total steps as two stages0.110.27
Two stages, pretrained teacher (Section 4.2)0.280.27

The numbers in the table are FT generalization, averaged over two to five runs. Recipient B already reaches the two-stage level with one stage, same as the weak seeds in Pythia; Recipient A does not.

The reason is in what stage 1 does. It moves the recipient’s representation toward the teacher’s representation, and at the same time also lets the layer above the distilled layer adapt to reading this representation. One stage has to do both of these things while learning the task. We tested this on recipient A: starting stage 2 from the stage-1 model but only resetting the last layer to the pretrained weights, FT generalization drops from 0.27 to 0.13; giving a one-stage run only the last layer of stage 1 and keeping everything else the same, FT generalization rises from 0.09 to 0.19. Recipient B depends much less on this preparation (from 0.25 to 0.20 under the same reset), so one stage is enough for it.

Figure 2c, I cannot find the super tier points and in Fig 2(d) ‘Boundary’ and ‘Bad’ tiers have three and two different groups of points respectively, what do they mean?

In Fig. 2c, the y axis shows the accuracy at the end of FT, but the tiers use the peak, so the three super runs were drawn in other colors. This is now corrected. In Fig. 2d, each column is one pretraining run and its 14 FT seeds, and now each column is labeled.

Throughout the paper when pretraining loss is mentioned is training loss or validation loss?

The loss in the figures used 384 training sequences. We recomputed the loss on all 278 runs with 4,000 new sequences. The gap between the two is 0.06%, the correlation is 0.998, all conclusions hold under the held-out loss, and Fig. 2b now uses the held-out loss.

Evaluation accuracy within 10 deg seems a little arbitrary.

Under 5, 15, 20 and 30 degrees, and under the mean angular error, the order of the tiers is the same (rank correlation with the 10-degree accuracy is 0.95 to 0.99).

‘It is generally believed that the rich representations built during PT are the main driver of FT performance.’ - it would be nice to add citations to support this.

Added, thank you.


If the above replies address the reviewer’s concerns, we kindly ask you to consider raising the score!

If you have any other questions or concerns, please let us know.

Reviewer_ceVfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the seed-by-seed variability of pretrained models with respect to their fine-tuning performance by drawing on a synthetic dataset of city relations in the world. The authors find that for identical pretraining procedure, only varying the random seed results in substantially different downstream fine-tuning generalization out-of-distribution. They find that this is driven by the model's representation, with representational distillation transferring successful performance from a bad model into a good model. They validate these findings in a larger setting, with Pythia.

Critical strengths

I thought this was an interesting paper!

  • I found the core findings very insightful, both that there is such variance in FT OOD generalization at all on this synthetic task, and that this is mediated by the representation rather than, say, the output token distributions.
  • The experiments are careful and interesting. The authors clearly thought carefully about important controls and overall provided a comprehensive assessment of the overall phenomenon. They also discuss extensively experiments that didn't work including attempts to predict the generalizability of a model given its representations. I really appreciated this discussion.
Critical weaknesses
  • While I think this paper provides an interesting insight, it focuses on diagnosing a phenomenon without providing a closer understanding of what drives this variance or how representations determine subsequent OOD generalization. While the careful description of this phenomenon is valuable in its own right, this limits the depth of understanding from this paper. Similarly, the authors do not provide any concrete recipes for how one might think about improving pretrained models in practice (after all, if one already has a good model, one would just want to use it for fine-tuning directly rather than using it for representation distillation). So I think this is a valuable paper but these two gaps limit the paper somewhat. Notably, the authors discuss the limitations raised here, which I appreciate.
Questions for authors
  • Do you have any thoughts on how one could build on this work to either develop a closer theoretical understanding of these phenomena or to build more concrete tools?
  • Why do you think keeping the representation distillation loss turned on during fine-tuning is necessary?

Is there any diversity between the data samples on which different good seeds are performing well? I.e., is the per-sample correlation highly correlated between different good seeds? If not, could you run representational distillation from multiple models to incorporate the best of all of them? Note that this is more of a suggestion for future work (and something I found myself curious about while reading the paper). It is not necessary for the current paper, which I think could be accepted in its current state.

Minor comments:

  • l. 53: “but not isolate the cause” -> “do not”
  • l. 197: missing space
  • l. 267: “installs” or “instills”?
  • Text in “Data Generation Process” of the figure is too small
2 comments

Dear reviewer,

Thank you for your thorough review!

We are glad to hear that the reviewer found our core findings very insightful and our experiments careful and interesting.

The reviewer mainly pointed out that the paper diagnoses the seed effect without explaining what drives it, and gives no concrete recipe. Our response is as follows.


it focuses on diagnosing a phenomenon without providing a closer understanding of what drives this variance or how representations determine subsequent OOD generalization.

the authors do not provide any concrete recipes for how one might think about improving pretrained models in practice (after all, if one already has a good model, one would just want to use it for fine-tuning directly rather than using it for representation distillation).

Regarding these two questions, we actually have substantial content in the paper explaining the underlying causes. The key to generalization is whether fine-tuning can learn the task primarily by reading it out from the pretrained representation, or whether it must reshape that representation entirely.

As for why the variance is so large, it stems from the combination of initialization and data order. We crossed ten initialization seeds with ten data-order seeds and found that a single seed's score cannot actually predict its score under other data-order seeds, nor can it predict the score of the same data order under other initializations (r=0.02r = 0.02). Before the pretraining loss even converges, the ranking among seeds is already determined (r=0.84r = 0.84 under matched loss).

As for how representations determine generalization, good seeds and bad seeds represent the map of individual cities equally well (R=0.86R = 0.86 to 0.920.92 in every tier), so the difference does not lie there. Rather, in good seeds, the answer to the four-city task can already be read out directly from the pretrained states. A linear probe prior to fine-tuning (FT) already predicts generalization (r=0.64r = 0.64), and good seeds learn the task faster with smaller parameter changes. Bad seeds reshape their representations to fit the training data, and this reshaping fails to transfer across groups. Once regularization is added to the representation, seeds that cannot read out the task fail to fit it altogether (in-distribution accuracy ranges from 0.100.10 to 0.860.86, correlating with generalization at r=0.67r = 0.67).

Regarding the concrete method proposed in this paper—representation regularization, it can even be used to evaluate pretrained models: a linear probe requiring only a single forward pass can identify seeds that will generalize under regularized FT (AUC 0.910.91) without using any OOD data. Furthermore, we fine-tune with regularization toward the model's own states. In the real-world LLM setup, the OOD accuracy of Pythia-160M increases from 0.2220.222 to between 0.2440.244 and 0.2910.291, and that of Pythia-410M increases from 0.2780.278 to between 0.360.36 and 0.420.42. At the same time, we found that regularized training serves as a diagnostic test: a model that cannot learn the task without altering its representation will not generalize (0.830.83 precision and 0.710.71 recall on held-out seeds).

As for our representation distillation, as noted in the paper, it achieves more than simply using a good model directly. In Pythia, the distilled weak seed ultimately surpasses its teacher (0.4030.403 vs. 0.3380.338 at 160M), and a single teacher can improve 7 out of 8 recipients in just 50k steps, whereas pretraining from scratch requires 656k steps.

Do you have any thoughts on how one could build on this work to either develop a closer theoretical understanding of these phenomena or to build more concrete tools?

Thank you for raising this question. We see concrete next steps in both directions and have added an outlook paragraph to the Discussion.

On the theoretical front, we plan to investigate which properties of the representation are left unconstrained by the pretraining loss (which can be derived in the orthogonal model; see App. C.5.3), and under what conditions gradient descent reads the task out from the representation versus reshaping it.

On the tooling front, since seed rankings are determined early, we can terminate weak seeds early during pretraining, use regularized FT to test individual pretrained models, and select or combine multiple teachers (see the final question).


To be continued [1/2]

Why do you think keeping the representation distillation loss turned on during fine-tuning is necessary?

Because standard fine-tuning pushes the installed representation away from the teacher within the first few thousand steps. Generalization arises when FT reads the task out from the representation rather than reshaping it. After the installation stage, the recipient carries the teacher's representation within the bad seed's weights, but under standard FT, it reverts to the bad seed's trajectory. Keeping this loss active keeps FT in the "read-out" regime. We measured this directly on the two recipients in Fig. 4:

MeasureRecipient ARecipient B
Similarity of city states to the teacher after the installation stage0.950.96
Similarity after 1,750 steps of standard FT (5% of the FT stage)0.390.40
Similarity at the end of FT with the distillation loss active0.980.98
FT generalization without the loss: peak, then final0.18, then 0.160.14, then 0.11
FT generalization with the loss0.280.26

Is there any diversity between the data samples on which different good seeds are performing well? I.e., is the per-sample correlation highly correlated between different good seeds? If not, could you run representational distillation from multiple models to incorporate the best of all of them?

Only partially: the agreement between two different good seeds is substantially lower than that between two fine-tuning runs of the same seed. We compared the good seeds prompt by prompt across 2,400 cross-group test prompts:

Pair of fine-tuned modelsCorrelation of per-prompt correctness
Same good seed, two FT seeds0.59
Two different good seeds0.30
Two bad seeds0.10

This variation is structured. Cross-group prompts fall into four composition types (App. B.3), and different good seeds excel at different types. On prompts containing one segment within each group, performance across the 15 good seeds ranges from 0.090.09 to 0.690.69. Two super-seeds achieve 0.65±0.080.65 \pm 0.08 and 0.35±0.040.35 \pm 0.04 respectively across 14 FT seeds each, demonstrating that this performance profile is intrinsic to the pretraining seed itself. It also transfers along with the representation: when 260 teachers are each distilled into the same recipient, the recipient's performance profile tracks that of its teacher (r=0.46r = 0.46).

Consequently, combining multiple good seeds should yield clear gains. Selecting the best good seed for each composition type reaches 0.450.45, whereas the best single seed reaches only 0.350.35. Simply averaging model outputs cannot capture this gain, and averaging target representations fails completely because different teachers' maps reside in different coordinate systems. Distilling with a separate target per teacher is the natural next step, which we have now included as future work in the paper.

l. 53: 'but not isolate the cause' -> 'do not'. l. 197: missing space. l. 267: 'installs' or 'instills'? Text in 'Data Generation Process' of the figure is too small.

Thank you for catching these. All four issues have been fixed. Regarding l. 267, "installs" is indeed what we intended, and we have now explicitly defined it upon first use.


Overall, the paper now explains the difference between good and bad seeds, provides a practical method that does not require a good model, and reports two new results prompted by your questions.

If the responses above address your concerns, we would kindly ask you to consider raising your score!

Please let us know if you have any other questions or concerns.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your thorough and constructive reviews! We are glad the reviewers found the problem "well-motivated" (R2), the core finding "very insightful" (R1), and the experiments "highly reliable" (R3).

The reviews primarily raised four points: evidence beyond synthetic models, how to detect good seeds, a method that does not require a good seed, and the clarity of Sections 4 through 6. Our response is as follows:


R3: "all experiments were conducted on synthetic data"

R2: "More experimental results on the LLM setup would be helpful."

Pythia-160M and 410M used in Section 6 are pretrained on The Pile, a real-world text corpus, and we have also added two new language model results. In Pythia, the pretraining seed accounts for 30% and 40% of the outcome variance, respectively. Each Pythia seed now has four fine-tuning (FT) seeds (80 runs in total), with seed means ranging from 0.223 to 0.292 at 160M and 0.286 to 0.475 at 410M. We also conducted behavioral distillation in Pythia: while the student model ends up agreeing with the teacher on 92% to 96% of next tokens, its out-of-distribution (OOD) accuracy remains close to standard FT—0.26 to 0.28 at 160M, compared to 0.40 achieved by representation distillation (Fig. R2).

R3: "it would require advanced detection methods"

R2: "It would be really nice to see the accuracies of PT models on the sangle task without any fine-tuning"

Good seeds can be detected without any OOD data. Zero-shot accuracies across all 278 seeds are at chance level and thus cannot distinguish them. However, three metrics achieve this (Fig. R1):

  • A linear probe on the pretrained model, requiring only a single forward pass (AUC 0.91).
  • In-distribution accuracy after the first 4% of FT (AUC 0.94).
  • In-distribution accuracy under regularized FT (AUC 0.95).

R1: "the authors do not provide any concrete recipes"

R3: "this approach is only feasible if a 'good seed' model is already available"

Representation regularization does not require a good seed. Instead, it targets the model's own states (Section 5). It improves OOD accuracy on Pythia-160M from 0.222 to a range of 0.244–0.291, and on Pythia-410M from 0.278 to a range of 0.36–0.42.

R2: "The clarity of the paper could be improved, especially in Sections 4, 5, and 6."

Section 4 now opens with a pipeline table covering all intervention experiments. We also ran each setting's pipeline on the other setting, obtaining consistent results.


The revised PDF includes the following updates:

  • Added two new figures in App. C: Fig. R1 (detecting good seeds) and Fig. R2 (behavioral distillation in Pythia).
  • Section 4 now begins with a pipeline table.
  • Expanded Section 6 so each pretraining seed has four FT seeds, and added a behavioral distillation baseline.
  • Updated Fig. 2b to use held-out loss, and corrected and annotated Figs. 2c and 2d.
  • Specified initialization details in the experimental setup section, and added an outlook paragraph in the Discussion.
  • Fixed all typographical errors.

In summary, the seed effect holds equally in language models pretrained on real-world text, good seeds can be detected without OOD data, and our practical method does not require a good seed.

Please feel free to let us know if there are any other questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.