- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper works as a diagnostic study and explores the root cause of performance degradation of LoRA iterative self-training using answer verified data. The authors designed a 40 rounds LoRA iterative self-training with both high and low learning rate on GSM8K benchmark and found that the model self trained with high learning rate loses 5–8 points on a fixed test set while the model self trained with low learning rate almost lost nothing. To separate the update strength from the data the loop generates, the authors then designed replay training with high/low rate replayed training data on high/low rate model. They found that the low rate helps through the data it produces rather than through gentler updates. However, the authors didn't provide any reason or find out any mechanism under it.
Critical strengths
This paper provide an explorational result on what will cause the performance degradation on LoRA self training on verified answer. It designed detailed and clear experiment to separate the update strength from the data the loop generates. The The authors are honest by disclosing a mid-study change to the protocol and show it doesn’t affect the result, and report the failures.
Critical weaknesses
-
This paper only work on one incremental experiment without any further explanation or exploration. The main contribution of this work is a single experiment that freeze the training data produced by high and low rate self-training runs and show that the data from the high rate run accounts for most of the forgetting. Its core observation that training on the outputs of a drifted model transmits the drift to a fresh model is already the central result of the model collapse literature [1] [2]. The paper offers no explanation for the effect and present with low novelty
-
The abstract’s conclusion is inconsistent with the paper’s own results. The abstract concludes that “verification leaves a data-source component of forgetting that rate tuning does not remove,”. Following this logic, in round 1, both donors generate from the same base model with the same seed and problem order, so they train on the same (or near identical) examples and differs only in learning rate. If rate does not matter, the two models after round 1 are the same. The same logic holds by induction from round 1 to the final round 40. This makes the resulting Low and High banks should be identical after these 40 self training rounds. However, they are not in reality. The high bank data degrade more with 7 points. So “rate does not matter” cannot be literally true.
[1]Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv:2305.17493, 2023. [2] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go MAD. arXiv:2307.01850, 2023.
Questions for authors
-
An abstract should stand on its own, but this paper's abstract relies on terms and quantities that are only defined later in the paper. "Generator" and "receiver" can be inferred from context. But the term "bank" is not standard jargon and may confuse with other concepts. The reported numbers have the same problem: “final-five-round near-domain accuracy change,” “source contrast,” “fixed-bank receiver-rate contrasts,” and “round-zero spread” are all metrics whose definitions appear
-
The authors didn't discuss the only successful training recipe in the paper in details. How was the only performance-improving bank D1 subsampled, were steps held fixed, and does D1’s advantage over LL survive the anchor-batch noise band?
-
The abstract claims training data is the source of performance degradation instead of learning rate. If receiver leraning rate barely moves near accuracy at fixed data, what creates the divergence between the H- and L-banks? When and how the H and L bank begin to diverge.