- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies how models generalize functions in the presence of varying degrees of exceptions / noise, as well as under varying training orders. It finds (consistent with prior work) that both factors can affect generalization.
Critical strengths
- This work runs reasonable experiments for what it sets out to test, and qualifies its results according to its limitations.
Critical weaknesses
-
These findings simply aren't new. Works the authors cite such as Arpit et al., as well as others they don't like https://aclanthology.org/2020.blackboxnlp-1.21/ or https://arxiv.org/abs/2412.04619 or https://arxiv.org/abs/2103.04514 have clearly shown that 1) fit on (different) training data will obviously affect generalization (and in particular that even if models can memorize training data, they will learn more generalizing structure) and 2) fit on the same training data, including induced by changes in ordering, can substantially change generalization. It is not obvious to me how this paper contributes a publishable new result beyond these, and the authors never make a case for it.
-
The descriptions are extremely hard to follow and the entire work reads like a language model performed all the experiments and wrote it. Of course, having LM experiment and writing assistance is reasonable if the paper makes a useful research contribution, but its unclear that this does.
Questions for authors
- What new things do we learn from this relative to prior works?