- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how to allocate a fixed budget of human data across training generations and distributional modes to mitigate recursive-training degradation. It introduces four budget-matched allocation policies and evaluates them under controlled training exposure. The results suggest benefits from human-data injection and concentrated selection, while revealing that the joint policy effectively reduces to selection-only due to an inactive scheduling mechanism. The main contribution lies in the budget-controlled experimental framework and the reconstruction of actual allocation decisions, rather than a demonstrated advantage of joint optimization.
Critical strengths
- The paper addresses a practical problem in recursive training: how to allocate limited human data rather than simply increasing its proportion. The distinction between temporal allocation and data selection provides a useful perspective.
- Matching both human-origin tokens and total optimizer tokens improves the fairness of policy comparisons. Replacing synthetic examples instead of appending human examples also removes an important training-volume confound.
Critical weaknesses
- My main concerns lie in the gap between the proposed joint allocation framework and what the experiments actually evaluate. The joint policy never activates its adaptive spending mechanism and purchases the same examples as Selection-only, so the primary comparison cannot establish whether jointly optimizing timing and selection is beneficial. More importantly, the executed reciprocal-NLL score reverses the intended degradation-aware prioritization, fundamentally changing the method being tested. Although the authors carefully document these issues, such transparency does not compensate for the absence of a valid experiment addressing the paper's central hypothesis.
- The reported 9.59% improvement mixes mode selection with repeated exposure to a small subset of examples. Without replay-matched and diversity-controlled baselines, the contribution of targeted selection cannot be isolated.
- The back-loaded policy spends substantially more human tokens in the final generation, confounding timing with terminal training composition. Moreover, only one fixed nonuniform schedule is evaluated, leaving the broader scheduling hypothesis insufficiently explored.
- Figure 2 labels the joint policy as underspent and shows a visible trajectory difference from Selection-only, despite the nearly identical results reported in Table 1. A similar discrepancy appears between Figure 3 and Table 7 for SmolLM2-135M, raising concerns about artifact consistency.
Questions for authors
- What actually determines the value of a human example?
- How much of the allocation benefit propagates through future synthetic generations?
Reply to Reviewer pWF2
Thank you for the review,
Joint policy and the score: We ran the comparison with the preregistered relative-gap score and thresholds the monitored signal crosses (Section 6.5). Joint's spending now departs from selection-only's, and joint lowers integrated test regret by 1.69% (95% CI [−0.062, −0.006] in outcome units, p = 0.027), lower in all five seeds.
Figure 2: Thank you for catching this. The submitted Figure 2 was rendered from a superseded execution of the grid, which is where the "underspent" label comes from. Table 1 and all reported numbers come from the reported execution. The revision uses the figure generated from those chains, in which selection-only and joint coincide, consistent with Table 1.
Figure 3 of the submission (Figure 4 in the revision): We re-derived the SmolLM2-135M rows of the submission's Table 4 from the 25 archived chain results and they reproduce exactly. The figure's endpoints equal each policy's generation-0 NLL plus its final regret. (The submission's Table 7 lists policy parameters, so we take Table 4 to be the one meant.)
The selection gain: In the follow-up no article is presented twice. A replay-matched control against random is the planned next step for the main grid's 9.59% contrast.
Schedules: The follow-up matches the final-generation allocation for the two policies it compares. Front-loaded, uniform and back-loaded schedules with matched final allocations are the planned schedule comparison.
Q: What determines the value of a human example? In the main grid, the largest gain came from concentrating the budget on a small set of tail articles presented repeatedly. In the follow-up, with no repetition, the degradation-ranked score directs almost the whole budget to the mid mode, and when that budget is spent also matters.
Q: How far does the benefit propagate?
In the main GPT-2 grid the benefit accumulates while spending continues. The gap between random and the control, and between selection-only and random, widens at every generation, and schedule-only stays near the control until it starts spending. Appendix D.2 reports every contrast at every generation, and Figure 3 does the same for the follow-up.