- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper attempt to figure out trade-offs in terms of how much pre-training may affect the fine-tuning later on, specifically for text-to-image models. The authors link this to cognitive science where youths tend to adapt to new things faster. Throughout the experiments, the authors found that under FFT, moderately pretrained models adapt fastest, and they link it to more reliant on fluid intelligence, whereas the most extensively pretrained model benefit from PEFT. The authors also provided some practical guidelines regarding computational costs.
Critical strengths
The part on dataset divergence is well conducted and interesting. The authors gradually change the identifier from non-semantic token to original word, and they provide a controlled manipulation to show the Youth model advantage under FFT depends on conflict with pre-training data.
Critical weaknesses
-
The entire claim/attempt to link the research findings to fluid/crystallized intelligence is quite post-hoc, and I am not sure if it adds anything substantial from prior work. For example, the proposed metrics are not validated as measuring some form of model plasticity or knowledge. Also, the Gc metrics would likely improve with any amount of training.
-
I think there is a crucial difference between maturity and over-trained/over-fitting. In the paper, the authors define expert models as extensively pre-trained. But pre-training uses only about 212k images, so this corresponds to roughly 19 epochs for BK-SDM and 77 epochs for SD-V2. Isn’t the extensive pre-training here confounded with data repetition? Because of this, it appears the work overlaps a lot with the findings of: “Overtrained Language Models Are Harder to Fine-Tune” from Springer et al. This supports, albeit this paper is regarding LLM, the authors’ overtraining explanation as for why the expert models fine-tune slowly.
Questions for authors
-
Is it possible to distinguish, or rather, how can you distinguish the Youth advantage from overtraining/overfitting to repeated data? Does the effect persist with larger pre-training set?
-
All checkpoints fine-tuned with the same LR. Would expert models perform better with their own LR? In your figure 12, it appears that the LR affects the model quite a lot i.e., Expert’s fine-tuning speed improves more at lower LR. Its unclear how much of the gap remains under properly tuned per-checkpoint LR.
Dear Reviewer,
Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.