- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the effectiveness of model updates by comparing generalists, models trained i.i.d., with specialists, models trained on temporally-ordered data. It observes that specialists converge faster during model updates compared to generalists. It proposes a theoretical explanation for this phenomenon and applies generalists in different empirical settings.
Critical strengths
- Generally, the question of how to optimize for forward transfer is still open and fundamental. If well supported theoretically and empirically (which currently it is not), the argument of the paper would be of high interest
Critical weaknesses
- limited related work section. There is a lot of work on pre-training, continual finetuning, and continual model editing. It should be appropriately discussed and compared against
- Big claims with limited supporting evidence
- the paper claims to be about "pre-training", but it studies factual knowledge, which is a limited form of knowledge. It's unclear whether the claims will generalize to standard pretraining setups
- the claim about convergence speed is measured via the number of Adam updates. However, this quantity depends on the learning rate. Could it be fixed by using a bigger learning rate for the generalist?
- the theoretical work is not very well supported. It's mostly a theory to explain the empirically observed phenomena.
- Limited baselines in the experiments: Fig.6 is the only experiment that compars against baselines from the literature, which is not sufficient.
Questions for authors
- if the advantage is mostly gradient magnitude, can it be recovered by setting a proper learning rate?
- more than specialist, they seem to be time-aware vs time-agnostic?
- "...specialists and generalists, trained under CD, perform equally well on the year for which the specialist is optimized. Yet, a specialist clearly has not learned the year information [...]. Why, then, can a model that has not learned the year information perform as well as one that has?"
- I'm confused by this statement. The specialist is only correct about the current year and it's going to answer with that knowledge regardless of the request. Isn't this obvious? The model completely ignores year information.