acceptodds
Under review as a conference paper at ICLR 2027

Where Should a Fixed Budget of Human Data Go? Allocating Across Time and Modes in Recursive Training

Abstract

Recursive training on model-generated text degrades language models, and human-written data arrests that degradation. We study how to spend a finite human-data budget: one fixed lifetime stock of human-origin tokens, allocated both across recursive generations and across monitored regions of the human distribution. We formalise this as a constrained allocation problem and decompose it into four budget-matched policies that isolate each decision, spending randomly, on timing alone, on selection alone, or on both jointly. We enforce matching on two axes at once, lifetime human-origin tokens and total optimizer tokens, realising 0.0381% and 0.00% spreads against a 0.2% bound, and we assemble each corpus by displacement so every arm trains on identical data volume. We show that spending any human budget improves held-out regret by 4.04% over a control trained on that same volume, that concentrating the budget by mode is worth a further 9.59%, and that a back-loaded schedule improves endpoint regret by 6.51% on an interval excluding zero while leaving integrated regret unchanged, so the integral hides a timing effect the endpoint reveals. We then reconstruct every arm's realised purchases from the frozen manifests and verify them to the token against every archived chain total. That audit establishes which comparisons the executed run licenses: it shows the joint policy never left its base schedule, so the joint-versus-selection comparison measures how far two identical policies drift apart, and the value of coupling timing with selection remains open. We report 25 certified GPT-2 chains and 50 further chains on Pythia-160M and SmolLM2-135M as a descriptive replication.

Primary area:
datasets and benchmarks
Keywords:
recursive training, self-consuming loops, model collapse, synthetic data, human data budget, data allocation, data selection, data scheduling, budget-matched comparison, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pWF2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to allocate a fixed budget of human data across training generations and distributional modes to mitigate recursive-training degradation. It introduces four budget-matched allocation policies and evaluates them under controlled training exposure. The results suggest benefits from human-data injection and concentrated selection, while revealing that the joint policy effectively reduces to selection-only due to an inactive scheduling mechanism. The main contribution lies in the budget-controlled experimental framework and the reconstruction of actual allocation decisions, rather than a demonstrated advantage of joint optimization.

Critical strengths
  • The paper addresses a practical problem in recursive training: how to allocate limited human data rather than simply increasing its proportion. The distinction between temporal allocation and data selection provides a useful perspective.
  • Matching both human-origin tokens and total optimizer tokens improves the fairness of policy comparisons. Replacing synthetic examples instead of appending human examples also removes an important training-volume confound.
Critical weaknesses
  • My main concerns lie in the gap between the proposed joint allocation framework and what the experiments actually evaluate. The joint policy never activates its adaptive spending mechanism and purchases the same examples as Selection-only, so the primary comparison cannot establish whether jointly optimizing timing and selection is beneficial. More importantly, the executed reciprocal-NLL score reverses the intended degradation-aware prioritization, fundamentally changing the method being tested. Although the authors carefully document these issues, such transparency does not compensate for the absence of a valid experiment addressing the paper's central hypothesis.
  • The reported 9.59% improvement mixes mode selection with repeated exposure to a small subset of examples. Without replay-matched and diversity-controlled baselines, the contribution of targeted selection cannot be isolated.
  • The back-loaded policy spends substantially more human tokens in the final generation, confounding timing with terminal training composition. Moreover, only one fixed nonuniform schedule is evaluated, leaving the broader scheduling hypothesis insufficiently explored.
  • Figure 2 labels the joint policy as underspent and shows a visible trajectory difference from Selection-only, despite the nearly identical results reported in Table 1. A similar discrepancy appears between Figure 3 and Table 7 for SmolLM2-135M, raising concerns about artifact consistency.
Questions for authors
  • What actually determines the value of a human example?
  • How much of the allocation benefit propagates through future synthetic generations?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer pWF2

Thank you for the review,

Joint policy and the score: We ran the comparison with the preregistered relative-gap score and thresholds the monitored signal crosses (Section 6.5). Joint's spending now departs from selection-only's, and joint lowers integrated test regret by 1.69% (95% CI [−0.062, −0.006] in outcome units, p = 0.027), lower in all five seeds.

Figure 2: Thank you for catching this. The submitted Figure 2 was rendered from a superseded execution of the grid, which is where the "underspent" label comes from. Table 1 and all reported numbers come from the reported execution. The revision uses the figure generated from those chains, in which selection-only and joint coincide, consistent with Table 1.

Figure 3 of the submission (Figure 4 in the revision): We re-derived the SmolLM2-135M rows of the submission's Table 4 from the 25 archived chain results and they reproduce exactly. The figure's endpoints equal each policy's generation-0 NLL plus its final regret. (The submission's Table 7 lists policy parameters, so we take Table 4 to be the one meant.)

The selection gain: In the follow-up no article is presented twice. A replay-matched control against random is the planned next step for the main grid's 9.59% contrast.

Schedules: The follow-up matches the final-generation allocation for the two policies it compares. Front-loaded, uniform and back-loaded schedules with matched final allocations are the planned schedule comparison.

Q: What determines the value of a human example? In the main grid, the largest gain came from concentrating the budget on a small set of tail articles presented repeatedly. In the follow-up, with no repetition, the degradation-ranked score directs almost the whole budget to the mid mode, and when that budget is spent also matters.

Q: How far does the benefit propagate?

In the main GPT-2 grid the benefit accumulates while spending continues. The gap between random and the control, and between selection-only and random, widens at every generation, and schedule-only stays near the control until it starts spending. Appendix D.2 reports every contrast at every generation, and Figure 3 does the same for the follow-up.

Reviewer_X95ZOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to allocate a fixed budget of human-written data during recursive training, where each generation of a language model is trained on synthetic data generated by the previous one. The authors investigate when to introduce human data across generations, and which data to select across modes (regions of the human data distribution). To do so, they compare four policies (random, schedule-only, selection-only and joint) against a control with no human data, with all policies matched on both lifetime human tokens and total optimiser tokens.

Using GPT-2 (124M) on 400 WikiText-103 articles over 10 generations and 5 seeds, the authors find that the random policy improves integrated NLL regret by ~4% over the control, and that selection-only improves on random by a further ~9.6%. The schedule-only policy has no detectable effect on integrated regret, although it improves final-generation regret by ~6.5%.

Lastly, the authors find no detectable difference between joint and selection-only. They show that the joint policy never adapts its spending and therefore selects exactly the same examples as selection-only. They also acknowledge that the executed mode score differs from the preregistered one, so selection-only prioritises the mode the model already performs best on. Given these limitations, they conclude that whether jointly deciding when and where to allocate human data is beneficial remains an open question.

Critical strengths
  1. The authors are unusually transparent about what went wrong in their study. Deviations between the planned and executed procedure are listed item by item (Appendix J), and the limitations summarised in the main body, including the policies they did not build, are fully documented (Appendices G.3, G.4). They even state plainly that the question the paper set out to answer remains open (L474-475). This makes it clear which comparisons the experiments do and do not support.

  2. The authors pose an interesting idea, and make a clear effort to control how much data each policy is trained on, which I think is an important aspect of the experimental setup. In particular, policies are matched on both lifetime human-origin tokens and total optimiser tokens (L173-178), with repeated examples counted each time they are presented (L171). Moreover, purchased human examples replace synthetic records rather than being added on top, ensuring that every policy trains on the same amount of data (L311-315). This is consequential, as it helps isolate the effects of the different policies and makes their comparison more meaningful.

Critical weaknesses
  1. Methodology: I think the main issue with the paper is that the experiments, as implemented, do not test the central research question of whether jointly deciding when and where to spend human data is beneficial (L60-61), as the authors themselves acknowledge (L474-475).

a) The selection-only policy ranks modes using 1/NLLm1/\mathrm{NLL}_m rather than the preregistered relative gap (L242-244), so it prioritises the mode the model already performs best on, rather than the one which has degraded the most. Moreover, even with the sign reversed, I think ranking by raw NLL is problematic, since NLLm=H(pm)+KL(pm∣q)\mathrm{NLL}_m = H(p_m) + \mathrm{KL}(p_m | q), meaning that differences in NLL can largely reflect the inherent entropy of each mode rather than how much the model has degraded. The policy also reuses the same 75 articles at every generation (L376-377).

b) The joint policy never adapts, as all measured NLL values remain within its base band (L263-266). It therefore behaves identically to selection-only (L353-363), so the paper's central comparison is uninformative.

c) The schedule-only policy uses a single, coarse, back-loaded schedule, which goes against the authors' own motivation (L238-239), and its final performance is confounded by a larger final allocation (L409-411). Testing several schedules with the same total budget would have been more informative.

d) Final-generation regret (L192-194) reduces, but does not eliminate, cancellation effects.

This is consequential because none of the reported comparisons isolates the effect it is meant to measure, so the paper provides no evidence on its central question. Moreover, given the limited scale of the experiments (a single 124M-parameter model, 400 WikiText-103 articles and 5 seeds, retrained at every generation from a pretrained checkpoint which already contains knowledge from human data, L716-718), it is unclear whether even the secondary findings would generalise.

  1. Presentation: I found the paper very difficult to follow and understand.

a) The writing is often unnecessarily dense, with frequent use of semicolons and sentences which convey very little information (e.g. L181-183). Terminology is often introduced without being defined (e.g. "adaptive pacing", "rescue partition", "arm") or is used with different meanings (e.g. "partition", L159, L166), notation is overloaded (e.g. rr, L187, L216), and rules are referenced without being stated (L172, L180).

b) Important experimental details are only introduced late in the paper: GPT-2 first appears in the main text in the results (L374), the data source only in the Limitations section (L443), and the definition of the modes only in Appendix A.3 (L729-737).

This is consequential because it makes it difficult to establish what was actually done and to assess the paper's claims, and addressing it would require a substantial rewrite of the paper.

Beyond these points, the paper could benefit from some editing, although these issues aren't major and don't influence my score: certain information is repeated unnecessarily (e.g. L49-51, L171, L215), and numerical results are sometimes reported with inconsistent and unnecessarily high precision (e.g. L22, L391-392, L437).

Questions for authors

Clarifications:

  1. Which checkpoint is meant at L160-161? Appendix A.3 suggests a pretrained GPT-2 at a pinned revision (L709, L714), but the revision is not given. Please state this in the main text.

  2. What are the rules referred to at L172 and L180, and where are they defined?

  3. How and when was the reciprocal NLL substituted for the preregistered relative gap? Was this detected before or after the outcomes were opened?

  4. Why was a back-loaded schedule tested when the motivation in Section 1 points to front-loading, given that a front-loaded schedule would have cost the same (L1386-1387)?

Requested analyses:

  1. Could the authors rerun the selection-only and joint policies with the preregistered score, with bought examples removed from the pool (or ties broken by a seeded shuffle), and with thresholds that the monitored signal actually crosses?

  2. Could the authors test several schedules with the same total budget (e.g. front-loaded, uniform and back-loaded), with the final allocation matched across policies, to establish which aspects of timing matter?

The central issues (Weakness 1) cannot be addressed without new experiments, and the writing requires a severe change to improve the clarity of the exposition (Weakness 2). I do not believe they can be satisfactorily resolved within the rebuttal period.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer X95Z

Thank you for the detailed review and the precise line references, which made each point straightforward to address.

Q1: openai-community/gpt2, revision 607a30d783dfa663caf39e06633721c8d4cfcd7e (model and tokenizer). The revision states it in Section 3.

Q2: L172: the human-token ledger counts each purchased article's own GPT-2 tokens without padding, while the optimizer ledger counts the 512-token blocks the articles are packed into. L180: within each generation, schedule-only takes candidates in a seeded random order regardless of mode, as random does, until the allowance is filled. Both rules are now stated in the text.

Q3: The reciprocal transform was added on 18 August 2026, before any reported chain ran, when the tail-retention metric was oriented so that higher is better. The runner supplied the same transformed scores to the policies. This was identified after the results were available.

Q4: The back-loaded vector is the schedule fixed on 31 July 2026, before any chain ran, and the main grid kept it. Front-loaded and uniform schedules are part of the planned schedule comparison (Q6).

Q5: Done; see Section 6.5 and Appendix F. Selection-only and joint were rerun with:

the preregistered relative-gap score; purchased articles removed from the pool, so nothing is presented twice; joint thresholds set from a separate development seed, which the monitored signal crosses; matched lifetime, optimizer and final-generation budgets; outcomes on the held-out test partition. Joint's controller skips generations 1 and 8 and doubles its spending at generations 5 and 7. It lowers integrated test regret by 1.69% (paired difference −0.034, 95% CI [−0.062, −0.006], p = 0.027), lower in all five seeds, and the policies are level at the final generation.

Q6: Not yet run. Front-loaded, uniform and back-loaded schedules with matched final-generation allocations are the planned schedule comparison. The follow-up already matches the final allocation for the two policies it compares.

W1a: The relative-gap score normalises each mode by its own generation-0 NLL, which addresses differences in inherent mode entropy. In the follow-up it ranks the mid mode as most degraded.

W1c and W1d: The revision reports the paired difference at every generation (Appendix D.2) alongside the per-generation allocations. The follow-up matches the final allocation, and Figure 3 shows its per-generation difference.

Scale and the pinned checkpoint: Retraining every generation from one pretrained checkpoint is a deliberate choice. The recursion then runs only through the data, so differences between policies come from what each policy bought.

The revision makes these changes: defines the model, data, modes and partitions in Section 3; replaces internal shorthand ("arm", "rescue partition") with plain terms; uses "partition" only for the data splits; removes the Rg and T notation overlaps and aligns summation bounds with g = 0, …, G−1; states every rule where it is used; shortens sentences and removes repeated passages.

Reviewer_3prYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to allocate a fixed lifetime budget of human-written data across recursive training generations and predefined data modes. This is a constrained allocation problem and compare four policies: random allocation, schedule-only allocation, selection-only allocation, and joint time-and-mode allocation, together with a no-rescue control.

Critical strengths
  • Careful budget-matched experimental design
  • Transparent experimental auditing and interpretation.
Critical weaknesses

As pointed out by the purchase reconstruction in Section 6.3 and Appendix D, joint and selection-only purchase identical human examples at identical generations. Thus, the primary comparison cannot determine whether jointly adapting timing and mode selection is better.

Although selection-only improves cumulative regret by 9.59% relative to random allocation, this effect cannot be attributed specifically to better mode selection. The selection policy repeatedly purchases a small set of tail-mode examples, including 75 examples reused in all nine spending generations.

The tail-retention measure uses a reciprocal-NLL statistic closely related to the selection criterion, which is not good for its independence as supporting evidence.

The executed mode score differs from the preregistered relative-NLL-gap score. The implementation uses reciprocal NLL, which prioritizes the best-performing mode rather than the most degraded one. This changes the intended behavior of both selection and pacing.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 3prY

Thank you for highlighting the budget-matched design and the transparency of the audit.

Joint and selection-only: We ran the comparison with a trigger that fires (Section 6.5). With the preregistered score and no repeated purchases, joint's spending departs from selection-only's at four generations. Joint lowers integrated test regret by 1.69% (95% CI [−0.062, −0.006] in outcome units, p = 0.027) and is lower in all five seeds.

Attributing the selection gain: The revision reports the main grid's 9.59% gain together with how it was achieved, the budget was concentrated on a small set of tail articles that were presented repeatedly. In the follow-up no article is presented twice. A replay-matched control against random is the planned next step for separating targeting from repetition.

Tail-retention measure: The main grid's tail retention is now presented as a descriptive outcome. The follow-up measures outcomes on the held-out test partition, and tail-mode regret there moves with the overall result (−0.032, 95% CI [−0.062, −0.002]).

Mode score: The follow-up uses the preregistered relative-gap score, under which the most degraded mode is ranked first. Joint's thresholds are set from a separate development seed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. The reviewers agree on the core of the design:

Every policy is matched on both lifetime human tokens and total optimizer tokens; purchased human data displaces synthetic data, so every policy trains on the same volume; every policy's purchases are reconstructed and verified to the token.

New experiment: joint allocation with a trigger that fires (Section 6.5, Appendix F). We reran joint and selection-only on GPT-2 with the design the reviewers asked for:

Both policies use the preregistered relative-gap score, so the most degraded mode is ranked first; purchased articles leave the pool, so nothing is presented twice; selection-only spends near-uniformly, and both policies get the same final-generation allocation; joint's thresholds are set from a separate development seed; outcomes are on the held-out test partition, over the same five paired seeds; both policies spend exactly the same human tokens and optimizer tokens.

Joint's spending now responds to the signal: it skips generations 1 and 8 and doubles its allowance at generations 5 and 7. Joint lowers integrated test regret by 1.69% relative to selection-only (paired difference −0.034, 95% CI [−0.062, −0.006], p = 0.027) and is lower in all five seeds. On the validation partition, where the main grid measured its outcomes, the difference is −0.032 (1.63%, p = 0.021), again lower in all five seeds.

Other changes in the revised PDF:

Multiplicity: Every test now has an exact p-value, and secondary tests are Holm-corrected. All three main-grid effects (human data, selection, the back-loaded endpoint) remain significant after correction (adjusted p ≤ 0.002). Three model families: Section 6.6 now reports the Pythia-160M and SmolLM2-135M grids (50 chains) in full. On SmolLM2-135M the back-loaded endpoint effect recurs (6.29%) and joint and selection-only again buy identical data. Main-grid joint comparison: It is now described by what it measured: joint's urgency stayed within its base band, so it spent like selection-only. Figure 2 is regenerated from the reported chains. Every contrast is reported at every generation (Appendix D.2). Appendix consolidated from nineteen sections to ten. Model, data, modes and the evaluation data are now in Section 3. Terms are defined at first use, rules are stated where they are used, and notation is consolidated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.