acceptodds
Under review as a conference paper at ICLR 2027

Reusing Preference Data Changes What DPO Learns

Abstract

Direct preference optimization (DPO) is often run for several passes over a smaller preference pool. We show that, at a fixed training budget, this reuse changes the direction of what the model learns. We compare fresh and reused data with identical steps, batch sizes, and pair presentations, measure each trained model’s change from its parent on a frozen 200-scenario battery and 10,273 held-out human-preference pairs, and calibrate every comparison against variation between training seeds. In OLMo-2 1B, reuse and fresh training agree at a reliability-corrected r* = 0.798 [0.675, 0.892] (about 37°), outside seed-to-seed variation, while a second, disjoint reused pool lands in the same place (r* = 0.966 with the first). At a fixed budget, the turn grows with the number of passes, from 20° at two passes to 38° at 6.4, as predicted by a finite-pool model of DPO, and a third seed confirms the effect (r* = 0.770 [0.637, 0.881]). The redirection replicates across four DPO model families and on held-out human preferences. It also reaches behavior: reuse changes the preferred response on one-fifth of held-out human-preference pairs, five to eight times the seed-to-seed rate, without improving agreement with human labels, and no comparison finds its generations better; at greedy decoding, fresh training’s response wins on 71–74% of prompts. Held-out preference accuracy is also lower under reuse (0.604/0.616 versus 0.651/0.660). How often each pair is reused is therefore part of a preference-training recipe: equal budgets can produce systematically different models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hePDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares using fresh data with reusing data during training and analyzes how reusing data affects model training.

Critical strengths

There appear to be many experiments that analyze reusing data versus using fresh data and different aspects of reusing data.

Critical weaknesses

The paper is very difficult to read with many ideas being referenced without introduction. The paper uses words such as "battery", "redirection", and concepts like "fresh direction" without definition making it difficult to understand what the results are stating. As a result, it seems unlikely to contribute value to the community.

From what I could understand, it seems the main analysis is how reusing data varies from using fresh data under a fixed training budget and configuration, and it seems the main claims are that the model changes differently between epochs when reusing data, reusing data is different from fresh data, and that across epochs using the same data, loss and reward change in steps. The core findings primarily seem to be widely known empirical patterns and therefore not a meaningful contribution.

Questions for authors

No questions.

1 comment

Thank you for your review. We answer both points in order.

W1 (Ideas referenced without introduction): We removed "battery", "redirection" and "fresh direction". The revised paper rests on one main measure: the share of 10,273 held-out human-preference pairs on which two models choose differently. Section 1 defines the design terms, and we define each measure before any result uses it (Sections 2.2 and 4).

W2 (Widely known empirical patterns): Our result is that, at a fixed budget, reuse changes the same held-out choices on two pools that share no pair. The known pattern, that more DPO epochs lower scores, comes from studies that add compute with every epoch (Section 1). Fresh training sees each pair once, and reuse makes k passes over a pool k times smaller.

  1. A different pool changes most of the same choices. Reuse on a second pool that shares no pair with the first changes 72% to 73% of the choices the first pool changes, against 23% by chance (Section 4). Overfitting to a sample would change other choices.
  2. Matching distance does not reproduce the change. Fresh training at reuse's KL divergence changes only 22% to 24% of those choices, against 14% to 15% by chance (Section 5). Over-optimization would change the same choices.
  3. Accuracy against the human labels hides the change. Reuse changes 21% of held-out choices but trails fresh training by only 0 to 2 accuracy points (Section 3).

The effect also holds under standard DPO at beta = 0.1, and it is not the known length bias. Under summed scoring, reuse's changed choices favor neither length, and a second pool still changes 72% of them (Sections 6 and 7). None of the DPO studies we cite holds compute fixed, compares pools that share no pair, or matches KL divergence.

Thank you again. We would be glad to answer any further questions.

Reviewer_A2BEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares fresh and reused preference data under matched DPO presentation budgets. It measures each trained model's change from its parent using response-score profiles, compares displacement directions against seed-based reliability estimates, and tests disjoint reused pools, other instruments, and several model families. A finite-pool logistic model is proposed as an explanatory account. The experiments find reproducible differences between fresh and reused recipes, alongside poorer held-out outcomes for reuse in the main setting.

Critical strengths
  1. Matching optimizer steps, batch size, and total pair presentations is useful. The disjoint reused pool and held-out human-preference measurements also provide evidence that the observed recipe difference is not just one subset's accidental behavior.

  2. The paper includes informative counterchecks, including the pass-count ladder, norm- and KL-matched comparisons, and explicit reporting of mixed outcomes. These give readers material for examining alternative explanations.

Critical weaknesses
  1. The presentation is seriously overloaded and needs substantial editorial work. The abstract packs correlations, confidence intervals, angles, pass counts, win rates, and accuracy pairs into a single paragraph, with little room to explain the central contribution. The main text then introduces many arm codes, corrected correlations, ceiling rules, and separate verdicts before the reader has a simple picture of the design. This is more than a wording issue: important qualifications become difficult to track. The abstract and experiment narrative should be rebuilt around the question and a small number of main results.

  2. The design does not isolate repetition from reduced data coverage. Fresh training uses a larger pool, while reused training repeatedly visits a smaller one. The fixed-budget pass ladder also changes the pool size, since pool size equals presentations divided by passes. The paper acknowledges this limitation, but still describes the ladder as varying only pass count and treats pass count as carrying the redirection. It supports an exposure-recipe effect, not a separate causal effect of repetition. Distinguishing that claim requires a different control design.

  3. The stronger interpretation of a distinct reuse-induced direction is not adequately separated from different optimization progress. At the primary endpoint, reuse has roughly seven times fresh training's sampled KL. The nearest KL-matched reuse checkpoint has much higher agreement with fresh and receives a mixed verdict; the norm-targeted control also receives a mixed verdict (Section 6). These are important limitations, not decisive exclusions of the alternative explanation. The evidence establishes different endpoints under the chosen recipes more clearly than it establishes a distinct mechanism beyond stronger fitting or over-optimization.

  4. The finite-pool analysis does not explain the central empirical pattern closely enough. The field identity is the difference between empirical and population gradients, while the divergence result concerns separable logistic flow with frozen features and a fixed metric. Neither result derives a monotone pass-count effect at the finite transformer training budget, or a common redirection across disjoint pools. Most proposed signatures are identified as post hoc, and the authors explicitly test the modelling assumption rather than establish it. The account is suggestive, but its explanatory strength is overstated.

Questions for authors
  1. What conclusion about repetition remains after explicitly treating unique-pair coverage as a jointly changed factor? Is any existing comparison able to vary repetition while holding the unique pool fixed without changing other relevant training factors?

  2. How should the mixed KL- and norm-matched results change the claim that reuse produces a distinct direction beyond greater optimization progress? Please explain what these controls rule out and what they leave unresolved.

  3. Which proposition predicts the observed pass-count trend or agreement between disjoint reused pools under the actual training conditions? Please distinguish a derived prediction from an empirical signature motivated by the simplified model.

1 comment

Thank you for your review. We answer each point in your order.

W1 (Overloaded presentation and abstract): We rebuilt both around the question and a small number of main results. The abstract now gives the question, the design and the main results, each with its comparator. The main text uses one main measure, the choice difference on 10,273 held-out human-preference pairs, and one supporting measure, the angle between two runs' updates. Arm codes appear only in Table A1, and each term is defined where it first appears. We also retitled the paper to name the design: "Fewer Pairs, More Passes: Reusing Preference Data Changes What DPO Learns".

W2 / Q1 (Repetition against reduced data coverage): A narrower conclusion about repetition holds: at equal extra budget, repeating pairs changes more held-out choices than adding new ones. One comparison, new in this revision, varies repetition with the pool and the batch size fixed.

  • A second pass at a fixed batch size. Every run uses 20 pairs per step. Against one pass over 29,120 pairs, a second pass over the same pairs changes 5.2% to 6.1% of held-out choices. The second pass doubles the optimizer steps, so its control spends the same steps on new pairs, one pass over the 58,251-pair pool that contains them. That control changes 3.4% to 4.1%, close to the 2.9% between reruns (Appendix D.10).
  • The same pairs, once or 6.35 times. One pass and 6.35 passes over the same 58,251 pairs choose differently on 20.9% to 21.0% of pairs. This comparison also raises the batch size, and a control that makes the same change with new pairs differs on 4.0% to 4.4% (Section 3).

The main comparison measures the full exposure recipe, as you say, and Section 2.1 now states it that way.

W3 / Q2 (Distinct direction against optimization progress, and the mixed controls): We reran both controls on the 10,273 held-out pairs, our main probe set, and added two more. We report the scenario results in Appendix D and base the main claims on the held-out pairs. We measure direction by the angle between two runs' per-pair margin changes, corrected so that reruns sit at 0 degrees. On the held-out pairs, all four controls separate the recipes.

  • Reuse stopped at fresh training's KL divergence. At step 800, reuse is at 0.173 nats per token against fresh training's 0.197. It already lies 25.2 degrees from fresh training, against 17.7 degrees between fresh runs on two pools.
  • A higher learning rate. At eight times the rate, fresh training reaches reuse's KL divergence. It changes 22% to 24% of reuse's changed choices, against a chance level of 14% to 15%.
  • The lengthened update. We scaled fresh training's own weight change until its KL divergence matched reuse's. It changes 28% of reuse's changed choices, against a chance level of 14%.
  • A weaker anchor. At beta = 0.4, fresh training overshoots reuse's KL divergence fourfold. It changes 40% to 43% of reuse's changed choices, against a chance level of 26% to 30%.

These controls rule out one reading: that reuse changes the choices fresh training would change if it moved as far. Each changes far fewer of them than reuse on a second pool, which changes 72% to 73% (Section 5). They leave one thing open. Each control still changes more of reuse's choices than chance, so distance explains a part of the change. The three controls that move fresh training also change its learning rate, its weights or beta, so we also stopped reuse early, which changes only the stopping step.

W4 / Q3 (Which proposition predicts the pass-count trend or the agreement between pools): Neither is a derived prediction. Both are empirical results, established by measurement in Sections 3 and 4, and the revision makes it clear. Section 8 now presents the finite-pool analysis as a simple model of one mechanism. In that model, a pool that some parameter ranks entirely correctly has no finite minimizer, so training keeps moving the parameters. Appendix E states the model's scope. The effect also appears without this mechanism. IPO's loss has a finite minimizer on every pool, and reuse still lies 41.1 degrees from fresh training under IPO (Section 7).

Thank you again. We would be glad to answer any further questions.

Reviewer_MtBbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks the question that if given a fixed training budget for DPO, does it matter whether one uses lots of different preference examples once each or repeats of the examples? The authors answer this question by first training a Language Model both ways, and then measure how much each of the model's preference changed on a set of made-up decisions. They also test on real human-preference data. They find that the repeated data changed in a different direction compared to the separate examples. They tested this on separate models and data subsets, and they all had similar enough results to come to the same general observation. The conclusion from their results seems to be that repetitive training means the model learns differently, so how often another work re-uses the training pairs should be part of reported training parameters.

Critical strengths
  1. The authors report the drawbacks and negative results too, which is a very refreshing change.
  2. The experiments are thorough.
Critical weaknesses
  1. While experiments are thorough, it is not clear that it is actually the repeated data that makes the model learn a different direction. What is a concern is that the model just might be overfitting. This was not really tested in any way. For the KL-matched control experiment, the two models have the same amount of change and their results look almost the same.
  2. No new insights or novel claims are made. There have been works in the past that have done sensitivity analysis as part of bigger claims on DPO where they show that DPO on the same data multiple times can lead to learning direction change (can be due to overfitting here too). This is quite well known in practice, and standard knowledge in ML. It is unclear what the actual novelty here is.
  3. The paper is very hard to read, in addition to having significant formatting issues.
Questions for authors
  1. When they compare the re-used vs. non-repeating data-trained models with a similar amount of change (ie. when the reused model has moved from the starting model as far as the non-repeating model has), the two changes in the same direction it seems. can the authors please explain how this fits with the claim that reusing the data changes the direction too, and not just the same amount?
  2. Did the authors test if the number of repeats and the number of unique examples separately. In other words, what would be good to see is whether one pass vs. six passes with a smaller set (same budget) impacts the direction. That would clearly show the difference due to repetition.
  3. If the training stops at or.a little after the validation loss saturates, does it still show this difference? This would help rule out impact of overfitting.
  4. It seems that the reused subsets are from the same data mix. Could it be that the two reused models simply have the same results since they come trained from the same source?
  5. The non-repeating data model trained with a weaker beta ends up farther from the normal fresh direction than the reused data model. Can the authors clarify that when this happens, does this mean the model's direction changes due to this larger distance?
  6. In practice and in most standards, DPO can be run with a single or 2-3 passes. How impactful is this work if the actual real-world settings are far below what the authors test for?
  7. How does different training hyperparameters such as batch size and learning rate impact this difference?
  8. Papers like Tulu 3 have already discussed that DPO on same data multiple times result in overfitting. What are the major contributions that this paper have that is different from other observations and how can that be applied to practical settings?
  9. Please check for multiple formatting issues, as well as unclear terminology in the paper. There are also quite a few missing symbols.
1 comment

Thank you for your review. We answer each point in order. Fresh training sees each pair once, and reuse makes k passes over a pool k times smaller, with the same optimizer steps and batch size.

W1 (Is it the repeated data, or overfitting?): Reuse does overfit its pool, and we tested what that overfitting does on pairs the model never saw. If the change came from particulars of the pool, a different pool would change different choices. Reuse on a second pool that shares no pair with the first changes 72% to 73% of the same choices, against 23% by chance (Section 4). If reuse only pushed the model too far from the starting model, fresh training pushed just as far would change the same choices. Fresh training at reuse's KL divergence changes only 22% to 24% of them, against 14% to 15% by chance (Section 5). Each reading explains a part of the change, and neither explains most of it.

W2 / Q8 (Novelty, and contributions beyond Tulu 3): Tulu 3 and other epoch studies show that more DPO epochs lower scores, but every extra epoch adds compute. We hold compute fixed and show three things those studies do not: which held-out choices change, that a pool sharing no pair changes most of the same ones, and that accuracy against the human labels hides the change. In practice, DPO recipes should report passes and distinct pairs and compare held-out choices against a one-pass run (Section 10).

W3 / Q9 (Readability, formatting, unclear terminology and missing symbols): We rebuilt the abstract and the paper around one main measure, the share of 10,273 held-out human-preference pairs on which two models choose differently. We define each term where it first appears, and we rebuilt every table, figure and symbol.

Q1 (Same direction at a similar amount of change): Reuse changes the direction as well as the amount. We stopped reuse at step 800, where its KL divergence is closest to fresh training's. We measure direction by the angle between two runs' per-pair margin changes, corrected so that reruns sit at 0 degrees. At matched distance the gap is small, but reuse already lies 25.2 degrees from fresh training, against 17.7 degrees between fresh runs on two pools. Both of our tests separate the two. The gap widens to 44.6 degrees by 6.4 passes (Section 5).

Q2 (Repeats and unique examples separately): Yes, in two comparisons that hold the distinct pairs fixed. The first keeps the batch size at 20. A one-pass run sees 29,120 pairs once. Two passes over the same pairs change 5.2% to 6.1% of its held-out choices. One pass over the 58,251-pair pool that contains them changes 3.4% to 4.1%, and reruns of the one-pass run differ on 2.9%. The second comparison makes one pass and 6.35 passes over the same 58,251 pairs. Those models choose differently on 20.9% to 21.0% of pairs. It also raises the batch size, so its control makes the same change and spends the extra budget on new pairs. Those models differ on 4.0% to 4.4% (Section 3).

Q3 (Stopping when validation loss saturates): The difference holds at the validation-accuracy stop. Reuse's validation accuracy peaks at 2.6 passes, where the recipes choose differently on 12% to 13% of the 2,016 human validation pairs. Validation loss is lowest much earlier, at step 100 for both recipes, 3% into training. There neither recipe has moved from the starting model, so they differ on only 0.25% to 0.50% (Section 3).

Q4 (Same source for both reused pools): No. Fresh training on the second pool, from the same source, changes only 10% to 11% of reuse's changed choices. Reuse on that pool changes 72% to 73%. So the overlap comes from reusing a pool (Section 4).

Q5 (Weaker beta and larger distance): Larger distance does change fresh training's direction, but it moves away from reuse too. At beta = 1.25, fresh training lands between the two recipes' KL divergences and lies 48.6 degrees from reuse, farther than standard fresh training's 44.6 degrees. At beta = 0.4, it overshoots reuse's KL divergence fourfold and lies 63 degrees from both recipes (Section 5).

Q6 (Real-world settings of one to three passes): The change is clear at the three-epoch default in TRL. At 3.2 passes, reuse chooses differently from fresh training on 15.6% to 15.8% of held-out pairs, against 3.0% between fresh reruns. It starts at two passes, at 6.1% to 6.7%, against 4.6% to 4.7% between fresh runs on two pools (Section 3).

Q7 (Batch size and learning rate): The fixed-batch comparison in Q2 holds the batch size at 20. The difference also holds over a 16-fold range of learning rates. At a quarter of the learning rate, reuse changes 8.2% to 8.9% of choices, against 0.7% between fresh reruns. Under standard DPO at beta = 0.1, reuse changes 11.4% to 12.1%, against 1.3% between fresh reruns (Section 7).

Thank you again. We would be glad to answer any further questions.

Comments on the paper

1 comment

We thank the reviewers for their careful reviews. The reviews share two concerns: readability, and whether reuse does more than overfit or optimize further. The revised PDF makes these changes:

  • Title. We changed the title from "Reusing Preference Data Changes What DPO Learns" to "Fewer Pairs, More Passes: Reusing Preference Data Changes What DPO Learns". The new title names what the design compares at a fixed budget.
  • Abstract and main measure. We rebuilt the abstract around the question and the main results. The paper now rests on one main measure, the choice difference: the share of 10,273 held-out human-preference pairs on which two models choose differently. The 200 decision scenarios moved to the appendix, and each term is defined where it first appears.
  • New controls. We trained reuse on a second pool that shares no pair with the first. We stopped reuse at fresh training's KL divergence from the starting model, and we moved fresh training to reuse's KL divergence in three ways. We tested repetition with the distinct pairs held fixed, including at a fixed batch size. We added a third seed with its own pool, IPO with two seeds, standard DPO at beta = 0.1 and full-parameter Qwen2.5 3B.
  • Theory. Section 8 presents the finite-pool analysis as a simple model of one mechanism. Sections 3 and 4 establish the pass-count trend and the agreement between pools by measurement.
  • Values. We re-derived every value from stored outputs, and Appendix F lists every value that changed. We replaced the UltraFeedback accuracy comparison, whose prompts overlap the training data, with accuracy on the held-out human pairs. We also rescored by summed log-probability. Under that rule the recipes differ on fewer pairs, 6.8% to 7.4% at the large budget, and the overlap between pools still holds. The tilt toward longer responses appears only under per-token scoring.

Our design holds compute fixed. Fresh training sees each pair once, and reuse makes k passes over a pool k times smaller, with the same optimizer steps and batch size. Our main results on OLMo-2 1B are:

  1. Reuse changes what the model prefers. At 6.4 passes, reuse chooses differently from fresh training on 21% of held-out pairs, against 3.0% between fresh reruns. At a fixed batch size of 20, a second pass over the same pairs changes 5.2% to 6.1% of choices, and new pairs change 3.4% to 4.1% (Section 3).
  2. Most of the change recurs on a pool that shares no pair. Reuse on a second pool changes 72% to 73% of the same choices. Chance is 23%, and a rerun on the same pool reaches 82% to 83% (Section 4). So the particulars of one pool explain only part of the change.
  3. Moving fresh training as far does not reproduce the change. Fresh training at eight times the learning rate reaches reuse's KL divergence. It changes 22% to 24% of reuse's changed choices, against 14% to 15% by chance (Section 5). The other three distance controls agree.
  4. Accuracy against the human labels hides the change. Reuse trails fresh training by only 0 to 2 accuracy points, because where the recipes disagree, fresh training matches the human label on just 51% to 55% of pairs (Section 3). On four zero-shot benchmarks, reuse scores 2.2 to 9.0 points lower (Section 6).

The recipes also differ on TinyLlama 1.1B and Qwen2.5 3B, under IPO and under standard DPO at beta = 0.1 (Section 7). Each reply answers its reviewer's points in order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.