- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors analyze how LLM outputs drift when they are finetuned on a shared corpus of each other's text. In their experiments, the models start outputting fewer "function words" (the most frequent words in English) and do not repeat themselves. Furthermore, the authors report that increasing Phi-2's share of the corpus does not necessarily "align" the other models' outputs closer to Phi-2's. For more concrete analysis, they introduce definitions of "delay" and "departure" from a reference drift trajectory, and use these tools to analyze drift while varying how much of the corpus each model contributes. Their conclusions also vary based on the share percentage and the dominant model.
Critical strengths
The authors address an interesting and timely problem, related to model drift and collapse due to finetuning on the mixed outputs of other models. The authors use diverse metrics like drift, separation, delay, departure to analyze how each model's outputs change.
Critical weaknesses
I did not see justification for why the authors' results shed insight on the real-world problem they're motivated by. Some reasons why this is not obvious: (a) the models are relatively small (1B-3B) base models, not instruction-tuned, and the corpus consists of short (<=128 token) continuations; (b) each generation resets to base weights, and the authors never experiment with a continual learning setup; (c) sampling involves non-standard frequency and repetition penalties. I don't think the authors' experiments have to replicate real-world settings to shed insight on their real-world problem, but I need justification/convincing why insight transfers.
Questions for authors
- Please improve the writing in Section 3: Setup, I found the paper hard to read. In particular, the setup is a large number of definitions and metrics with little justification/intuition for where they come from, and many of the specifics can be moved to the appendix. Also, terms like "departure" and "delay" appear in the abstract/main text before they are defined.
- The authors chose DeBERTa as their sequence encoder because it "separates arms more sharply from seed noise", i.e. based on the same results it is being used to report. Results would be improved by varying the sequence encoder. As it stands, two issues: (a) greater separation does not mean the results are more valid. For example, greater separation may simply be due to difference in surface-level features like function-word rate (which is somewhat artificially lowered by the authors' frequency and repetition penalties); (b) The authors' key result about steering towards Phi-2 seemingly only holds under DeBERTa and not under e.g. all-mpnet-base-v2. Varying the sequence encoder lets us extract more generalizable insight.
- In the experiments, the authors only used two seeds per arm and a seed floor estimated from two runs. The between-seed variation in results (e.g., in Table 1) can be quite large, so ideally the results have confidence intervals and more seeds.
- Many of the claims sound like over-simplifications. For example, the authors claim the 90% arm ends at an earlier stage of the uniform arm. But across metrics: the 90% arm matches the uniform arm's generation ~3 on perplexity, ~4 on function-word rate, ~5 on word length, etc. Can we make the claims more precise?