- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper compares using fresh data with reusing data during training and analyzes how reusing data affects model training.
Critical strengths
There appear to be many experiments that analyze reusing data versus using fresh data and different aspects of reusing data.
Critical weaknesses
The paper is very difficult to read with many ideas being referenced without introduction. The paper uses words such as "battery", "redirection", and concepts like "fresh direction" without definition making it difficult to understand what the results are stating. As a result, it seems unlikely to contribute value to the community.
From what I could understand, it seems the main analysis is how reusing data varies from using fresh data under a fixed training budget and configuration, and it seems the main claims are that the model changes differently between epochs when reusing data, reusing data is different from fresh data, and that across epochs using the same data, loss and reward change in steps. The core findings primarily seem to be widely known empirical patterns and therefore not a meaningful contribution.
Questions for authors
No questions.
Thank you for your review. We answer both points in order.
W1 (Ideas referenced without introduction): We removed "battery", "redirection" and "fresh direction". The revised paper rests on one main measure: the share of 10,273 held-out human-preference pairs on which two models choose differently. Section 1 defines the design terms, and we define each measure before any result uses it (Sections 2.2 and 4).
W2 (Widely known empirical patterns): Our result is that, at a fixed budget, reuse changes the same held-out choices on two pools that share no pair. The known pattern, that more DPO epochs lower scores, comes from studies that add compute with every epoch (Section 1). Fresh training sees each pair once, and reuse makes k passes over a pool k times smaller.
The effect also holds under standard DPO at beta = 0.1, and it is not the known length bias. Under summed scoring, reuse's changed choices favor neither length, and a second pool still changes 72% of them (Sections 6 and 7). None of the DPO studies we cite holds compute fixed, compares pools that share no pair, or matches KL divergence.
Thank you again. We would be glad to answer any further questions.