acceptodds
Under review as a conference paper at ICLR 2027

Is More Always Better? Unveiling the Impact of Pre-training Iterations on Fine-tuning Performance in Text-to-Image Models

Abstract

As pre-training progresses, checkpoints at different stages may adapt to downstream tasks at different rates. Selecting the optimal pre-trained checkpoint is crucial for efficient downstream adaptation. Conventional wisdom suggests that more pre-training generally leads to better downstream performance, while findings from cognitive science suggest that younger learners often adapt more quickly. Motivated by this contrast, we investigate whether text-to-image (T2I) models exhibit a similar "youth advantage" during fine-tuning. Understanding this phenomenon could enable more compute-efficient adaptation strategies. Through experiments on two T2I backbones and three downstream datasets across ten pre-trained checkpoints, we reveal that when fine-tuning tasks conflict with the pre-training data, moderately pre-trained ("Youth") models adapt most quickly in full fine-tuning, whereas under LoRA, the most extensively pre-trained ("Expert") models converge the fastest. We explain this discrepancy with a fluid/crystallized intelligence framework from cognitive science, suggesting that full fine-tuning benefits from the fluid intelligence of young models, whereas LoRA leverages the crystallized intelligence of mature models. Our findings offer a unique perspective on practical checkpoint selection, enabling computational savings compared to using the most pre-trained model.

Primary area:
generative models
Keywords:
Text-to-Image Model, Model Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_vbMGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper attempt to figure out trade-offs in terms of how much pre-training may affect the fine-tuning later on, specifically for text-to-image models. The authors link this to cognitive science where youths tend to adapt to new things faster. Throughout the experiments, the authors found that under FFT, moderately pretrained models adapt fastest, and they link it to more reliant on fluid intelligence, whereas the most extensively pretrained model benefit from PEFT. The authors also provided some practical guidelines regarding computational costs.

Critical strengths

The part on dataset divergence is well conducted and interesting. The authors gradually change the identifier from non-semantic token to original word, and they provide a controlled manipulation to show the Youth model advantage under FFT depends on conflict with pre-training data.

Critical weaknesses
  1. The entire claim/attempt to link the research findings to fluid/crystallized intelligence is quite post-hoc, and I am not sure if it adds anything substantial from prior work. For example, the proposed metrics are not validated as measuring some form of model plasticity or knowledge. Also, the Gc metrics would likely improve with any amount of training.

  2. I think there is a crucial difference between maturity and over-trained/over-fitting. In the paper, the authors define expert models as extensively pre-trained. But pre-training uses only about 212k images, so this corresponds to roughly 19 epochs for BK-SDM and 77 epochs for SD-V2. Isn’t the extensive pre-training here confounded with data repetition? Because of this, it appears the work overlaps a lot with the findings of: “Overtrained Language Models Are Harder to Fine-Tune” from Springer et al. This supports, albeit this paper is regarding LLM, the authors’ overtraining explanation as for why the expert models fine-tune slowly.

Questions for authors
  1. Is it possible to distinguish, or rather, how can you distinguish the Youth advantage from overtraining/overfitting to repeated data? Does the effect persist with larger pre-training set?

  2. All checkpoints fine-tuned with the same LR. Would expert models perform better with their own LR? In your figure 12, it appears that the LR affects the model quite a lot i.e., Expert’s fine-tuning speed improves more at lower LR. Its unclear how much of the gap remains under properly tuned per-checkpoint LR.

Reviewer_JXg2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how checkpoints taken along the pre-training trajectory adapt to downstream tasks. It focuses on text-to-image (T2I) diffusion models (BK-SDM and SD-V2) fine-tuned on three downstream tasks (Flower, Cat and Unsafe). Based on the number of pre-training iterations, checkpoints are classified into Infant, Youth and Expert models; Infant models have seen the fewest pre-training iterations (0 in the main experiments) and Expert models the most. Two fine-tuning strategies are compared: full fine-tuning, which updates the entire model, and LoRA fine-tuning, where the pre-trained weights are frozen and only low-rank adapters are trained.

The results are as follows: a) Under full fine-tuning, Youth models adapt faster than Expert models. b) Under LoRA fine-tuning, Expert models adapt fastest.

Critical strengths

The observation that Youth models adapt better under full fine-tuning, while Expert models adapt better under LoRA fine-tuning, is interesting, and connecting it to the fluid/crystallized intelligence distinction from cognitive science is an interesting contribution.

Critical weaknesses

The evaluation is the weakest aspect of the paper. The paper introduces many metrics with very little description and without any motivation or background. Below is the list of the metrics and their problems.

a) G_f and G_c, dubbed fluid and crystallized intelligence: their formulation (which is not included in the main paper) and their motivation are missing. Without these, it is not clear what these quantities are intended to measure. Moreover, the plots are not consistent with the definitions given in the appendix (Section A.3.3):

  • KRR-grad is defined as a ratio in [0, 1] (Eq. 5), but in Figure 5a it reaches values of about 120.
  • Gc_spec is a fraction of spectral energy and hence at most 1 (Eq. 7), and for a random (Infant) weight matrix it should be close to k/min(d_out, d_in); yet in Figure 5b it starts at 1.0 and rises to about 1.9.
  • If a normalization is applied, it is not described.

b) Area Ratio (area under the normalized performance curve): it is not clear why this is chosen instead of the performance at the end of the fine-tuning budget.

Questions for authors
  1. For Expert models, does full fine-tuning eventually catch up with LoRA? If so, after how many iterations?

  2. For language models, it has already been reported that overtrained models are harder to fine-tune. It has also been reported that how fine-tuning is performed matters, for example the choice of hyperparameters or algorithm [1, 2]. In this paper, the same fine-tuning hyperparameters (learning rate and batch size, Table 5) are used for all pre-training checkpoints, i.e., they are not tuned per checkpoint. The observed differences in fine-tuning speed could therefore partly reflect how well a fixed learning rate suits each checkpoint, rather than the adaptability of the checkpoint itself. I do not ask the authors to do an exhaustive search over hyperparameters and algorithms, but they should clearly report the hyperparameters and training details, state that fine-tuning can be sensitive to these choices, and scope their contributions to the regimes they study.

[1] SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs (ICLR 2026, arXiv:2509.20758) [2] (How) Learning Rates Regulate Catastrophic Overtraining (arXiv:2604.13627)

1 comment

Dear Reviewer,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

Reviewer_FVGcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how pre-training progress affects adaptation speed in text-to-image diffusion models. The authors evaluate ten checkpoints across two backbones and three downstream tasks, finding that moderately pre-trained models often adapt faster under full fine-tuning, whereas more mature models adapt faster under LoRA. Based on these observations, the authors propose a fluid/crystallized intelligence interpretation and derive checkpoint-selection recommendations from an analysis of adaptation costs.

Critical strengths

The interaction between pre-training progress and fine-tuning methods is an interesting and practically relevant observation.

Critical weaknesses
  1. The experiments in Sec. 4.3 do not isolate task conflict from unfamiliar-identifier effects. Specifically, distribution divergence is varied by replacing “flower” with different strings, which also changes the input representation produced by the frozen text encoder. This does not establish that the ordering in Fig. 4 measures increasing conflict with pre-training knowledge. Learning a new label for an existing concept should be distinguished from contradicting an established concept mapping.

  2. The explanation in Sec. 5.3 does not separate knowledge reuse from adaptation progress. In Fig. 6, the full-fine-tuning outputs still resemble their initial images, whereas LoRA already generates flowers. This difference may reflect unequal adaptation success rather than overwriting versus redirecting knowledge.

  3. The practical efficiency recommendation relies on a limited notion of successful adaptation. To be specific, GPT-Eval and Human-Eval mainly assess whether the target concept is recognizable. The CLIP-based checks provide additional evidence, but they do not establish that Youth checkpoints achieve comparable visual quality and detailed prompt adherence at lower cost. This distinction matters for interpreting the savings reported in Sec. 6.

Questions for authors
  1. Can the authors distinguish unfamiliar-label learning from conflict with existing knowledge, for example, by comparing a new identifier with an existing concept word deliberately assigned a different meaning?

  2. Does the similarity contrast between full fine-tuning and LoRA persist when they are compared at matched downstream concept accuracy? This would help determine whether Fig. 6 reflects different learning mechanisms or different stages of adaptation.

  3. How do Youth and Expert outputs compare in a blinded evaluation of visual quality and prompt adherence at the performance thresholds used in Table 2?

Reviewer_WmLLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how pretraining progress affects adaptation in text-to-image models. The paper finds that earlier checkpoints often favor full finetuning on token remapping tasks, while later checkpoints favor LoRA. It explains this reversal through plasticity versus accumulated knowledge and proposes checkpoint selection guidelines to reduce adaptation costs.

Critical strengths
  1. Comparing ten checkpoints across two backbones provides a systematic examination of how pretraining progress affects adaptation.

  2. The reversal between full fine-tuning and LoRA highlights an important interaction between checkpoint selection and adaptation method.

  3. Multiple evaluation metrics and hyperparameter ablations strengthen the evidence for the observed empirical trend.

Critical weaknesses
  1. Without matched total FLOPs comparisons, the analysis cannot distinguish a maturity dependent adaptation effect from a tradeoff in compute allocation between pretraining and finetuning.

  2. Artificial token remapping with frozen text encoders provides insufficient evidence for broad claims about adaptation or the proposed plasticity based mechanism.

  3. The reported savings rely on retrospective checkpoint comparisons, without a clearly specified independent selection protocol or accounting for checkpoint search costs.

Questions for authors
  1. Do the performance differences persist at matched total pretraining and finetuning FLOPs, using a common downstream quality target?

  2. Does the Youth advantage generalize to tasks beyond token remapping when checkpoint selection uses an independent validation set and final evaluation uses held out test data?

1 comment

Dear Reviewer,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.