- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission seeks to investigate the question of how generalization and memorization evolve as learning unfolds in neural language models. The submission contains two studies based on training networks to learn synthetic language datasets.
The first set of experiments seeks to address the question of whether learning affects the continua global structure formed by the model. They assess the performance of the models in predicting the category of the next token, when provided with tokens along the path defined by the centroids of the representations of the two categories. They compared the results from models of different sizes and lengths of training, and found that there were some differences.
In the second set of experiments, they seek to investigate “the temporal relationship between generalization and memorization along the learning trajectory” and whether observed sequences leads to stronger compression. The compression was evaluated using a quantify from information theory.
Overall, while the paper conducts several numerical experiments on synthetic datasets and reported some notable observations, the results are too preliminary. It wasn’t clear how these experimental results speak to or against the bigger questions they claim to study. The quality of the writing would need substantial improvement to meet the bar for ICLR.
Critical strengths
- The question of generalization vs. memorization is an important problem in language modeling, particularly in the context of large language models (LLMs). I thought the creation of the two synthetic datasets and some of the task designs are thoughtful.
Critical weaknesses
-
The results are preliminary. The experimental results were not carefully interpreted. It was not clear how they indeed support the main claims on generalization and memorization. It is uncertain to me if the observations reported could indeed speak in favor (or against) the issues of memorization and generalizations in language modeling in general.
-
It is unclear how these results would generalize to other regimes, in particular for large language models trained on a large amounts of text. Even if the authors’ interpretations on the generalization and memorization on these synthetic datasets are correct, there seem still a large gap for understanding generalization and memorization in LLMs.
Questions for authors
The paper claims that, early in training, there is evidence for the emergence of geometric structure that would support generalization. As training progresses for much longer, memorization start to dominate. Does this mean that models trained for longer would not generalize or generalize worse?
Line 52-32, “early stage optimization constitutes a comprehensive update on the whole continuous representation space.” The paper seems to imply that this means generalization. Can the authors unpack the ideas? What does variable c represent in Section 2?
Would the results generalize to large-scale datasets with larger models?
Can the authors better explain what “pure generalization” and “pure memorization” would predict in the context of their numerical experiments? I think explaining these early on before showing the results would help the readers.
The authors did not engage in a discussion, so my concerns were not addressed and thus I am keeping my score.