acceptodds
Under review as a conference paper at ICLR 2027

Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coverage

Abstract

In verifiable domains, such as math and coding, where multiple solutions can be generated in parallel and scored by a verifier, getting just one correct solution among many attempts can be more important than the individual pass rate of each attempt, especially for difficult tasks where a correct solution might be a needle-in-the-haystack. Post-training methods such as supervised fine-tuning (SFT) and reinforcement learning based preference fine-tuning in Large Language Models (LLMs) have been shown to concentrate their outputs around a few modes. A common method to increase LLM output diversity is to increase a “temperature” parameter, but the effectiveness of this has been shown to be limited (Banayeeanzade et al., 2026; Holtzman et al., 2020). To address this issue we introduce Don't Repeat Yourself Supervised Fine-Tuning (DRY-SFT): a post-training method that increases output diversity and improves coverage — coverage being how likely it is for there to be at least one correct solution among many generated attempts. DRY-SFT has two simple stages: first, for each problem, sequentially generate K solutions, where, for each subsequent generation, the prompt contains the problem plus all of the prior problem attempts and asks for a different solution. Second, fine-tune the LLM on each solution attempt as if it was generated independently by stripping away the prior attempts from the context. This process involves no reward, verifier, or correctness filter. The result is a model that is trained to produce many diverse solutions. We evaluate DRY-SFT on three coding benchmarks and show that it raises pass@100+ by 10.8, 12.5, and 12.4 points across HumanEval+, MBPP+, and DS-1000 at a small cost to pass@1+. We also measure the structural diversity using abstract syntax tree (AST) edit distance of passing solutions and find that this rises significantly across all three benchmarks. Additionally, DRY-SFT solves 244 of 600 problems the base model was not capable of solving in the same 200 attempts. Finally, across nine open-weight models, lower structural diversity of the base model significantly predicts higher DRY-SFT performance gains, indicating that our method is especially effective on more mode-collapsed LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mode collapse, output diversity, post-training, supervised fine-tuning, pass@k, coverage, test-time scaling, self-generated data, code generation, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P3LTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DRY-SFT, a method for improving the probability that at least one of several sampled solutions is correct. It sequentially generates multiple solutions conditioned on earlier attempts, then removes that conditioning and fine-tunes on each solution independently. No correctness filter or reward is used during the default training procedure. On three coding benchmarks, the method improves high-budget pass rates while reducing single-sample accuracy. Matched independent-generation controls, temperature experiments, held-out and cross-dataset evaluations, and structural-diversity measurements support the approach. A nine-model study additionally relates lower initial structural diversity to larger coverage gains.

Critical strengths
  1. A simple intervention produces substantial coverage gains with a relevant control. Section 4.1 reports approximately 10.8–12.5-point pass@100 improvements across HumanEval+, MBPP+, and DS-1000, whereas the matched independent-generation control changes performance much less. Because the optimization recipe and number of targets are matched, this comparison directly supports the importance of conditioning target generation on prior attempts. Amortizing sequential generation into ordinary independent inference is a useful practical contribution.

  2. The paper provides evidence of transfer and reports its trade-offs. Section 4.4 shows held-out pass@100 gains on all three benchmarks and transfer between HumanEval+ and MBPP+. Appendix E shows that gains persist with or without correctness filtering, while the paper reports the pass@1 decline and failures on already-diverse models. These qualifications make the contribution more credible and support acceptance as a method for improving coverage in the stated regime.

Critical weaknesses
  1. The headline comparisons use training-problem adaptation rather than a fully held-out protocol. Section 4.4 states that the main experiments generate training targets and evaluate on the same problems. Absence of correctness labels does not make such exposure irrelevant. The later held-out results address transfer, but Figure 4 does not repeat the full independent-generation, P-SFT, and temperature comparisons there. Clarifying separation of chain-depth selection, checkpoint validation, and final evaluation would strengthen the claim that the comparative advantage generalizes to unseen problems.

  2. The diversity evidence needs a more precise interpretation. Section 4 measures AST distance only on problems with at least two passing solutions among ten samples, potentially changing the evaluated problem subset between methods. Syntax-tree distance also does not directly establish algorithmic diversity. These issues particularly affect the nine-model correlation in Section 4.5, which is presented as a practical predictor of benefit. Shared-problem measurements and implementation details on identifier normalization would make that interpretation more convincing; the pass-rate results remain the more direct evidence of usefulness.

Questions for authors
  1. Clarification: For the 50/50 held-out experiments, were chain depth, checkpoint selection, and all other choices determined without evaluating the held-out half? Are matched independent-generation or P-SFT results on those same splits already available?

  2. Clarification: Are AST-diversity comparisons computed on a shared set of eligible problems across methods? How are identifiers, literals, and tree size handled, and do existing common-subset results preserve the reported diversity increase?

  3. Clarification: Do the reported bootstrap intervals cover one training seed per configuration, and are any generation/training repeats available? This matters particularly for the nine-model predictive relationship and close comparisons with temperature tuning.

1 comment

Thank you for your review and for recognizing the coverage gains and the importance of reporting the honest trade-offs. We have run additional experiments based on your comments and have uploaded the revised manuscript to OpenReview, in which the changes are marked in blue.

W1.

The headline comparisons use training-problem adaptation rather than a fully held-out protocol.

Thank you for making this point! We agree that excluding evaluation problems from fine-tuning matters even when training does not use correctness labels. We added a section stating which experiments use the same problems for training-target generation and evaluation, and which hold out evaluation problems from fine-tuning and checkpoint selection (Section 4, p. 5, lines 232–235).

We also added the matched held-out i.i.d. SFT and P-SFT comparisons. Consistent with our previous findings, DRY-SFT exceeds i.i.d. SFT / P-SFT by +6.62 / +5.41, +5.54 / +5.74, and +9.66 / +12.97 pass@100+ points on HumanEval+, MBPP+, and DS-1000 (paired 95% CIs in Table H.1, p. 19, lines 973–979). We added these results to Section 4.4 (p. 7, lines 359–367) and Figure 4 (p. 8, lines 391–395).

For temperature, we updated Figure 3 to use two-fold cross-validation on HumanEval+. Consistent with our previous findings, DRY-SFT exceeds this temperature baseline by 4.72 held-out pass@100+ points (paired 95% CI [1.79, 8.34]). Its AST diversity is 0.206 versus the base model's maximum of 0.158 on the same 104 held-out problems. We report these results in Section 4.3 (p. 7, lines 348–351) and the setup in Appendix I (pp. 19–20, lines 1005–1030).

W2.

The diversity evidence needs a more precise interpretation.

Thank you for identifying this. It is a great point. We recomputed the diversity comparisons on the same 123 HumanEval+, 244 MBPP+, and 294 DS-1000 problems for all four methods where each method had two or more correct solutions. Under this specification, DRY-SFT maintains significantly higher AST diversity over all three baselines on each benchmark. We updated Figure 2 and Section 4.2 with these results (p. 6, lines 295–299 and 307–313).

To address your other point, we also recomputed the nine-model comparison using the same 81 problems to measure baseline diversity. Lower baseline diversity remains significantly associated with larger coverage gains over base (R²=0.665, p=0.0074). We updated Figure 5 and Section 4.5 (p. 8, lines 413–420 and 422–427).

We agree that AST distance does not establish algorithmic diversity. We added this limitation explicitly: “This measures structural variation, and does not guarantee algorithmic or semantic diversity.” (Section 4, p. 5, lines 229–230). We also describe the nine-model result as an association in the models studied (Section 4.5, p. 7, line 377).

Q1.

For the 50/50 held-out experiments, were chain depth, checkpoint selection, and all other choices determined without evaluating the held-out half?

Thank you for the question. The evaluation problems were excluded from fine-tuning and checkpoint selection. However, K=5 was chosen using the broader benchmark ablation. We added this distinction explicitly to Section 4.4: “these results are not a fully independent evaluation of hyperparameter selection.” (p. 7, lines 363–365).

We added the training, validation, and evaluation counts and the checkpoint-selection rule to Appendix H (p. 18, lines 956–961), clarifying which problems were used for each step.

Q2.

Are AST-diversity comparisons computed on a shared set of eligible problems across methods?

We retain node types and child structure but omit identifier names and literal values. Each pair's edit distance is divided by the larger tree's node count, then averaged within and across problems.

We added these measurement details to Appendix J (p. 20, lines 1034–1042), along with the sampling procedure for the nine-model comparison (p. 20, lines 1075–1078).

Q3.

Do the reported bootstrap intervals cover one training seed per configuration, and are any generation/training repeats available?

Yes, each fine-tuned model has one training run. We do not report repeated training runs or regenerated training datasets. We added this clarification: “The intervals measure variation across problems, not repeated training runs.” (Section 4, p. 5, lines 222–223).

We also clarified that the temperature comparison resamples problems within each fold (Appendix I, p. 19, lines 1013–1015), while the nine-model regression calculates confidence intervals over models (Appendix J, p. 21, lines 1080–1081). We added 95% bootstrap confidence bands to the individual models' coverage curves in Figure G.1 (p. 18, lines 948–951).

Thank you again for your feedback! We hope our comments and additional experiments have sufficiently addressed your held-out data and diversity concerns and clarified model selection and what the confidence intervals measure. If so, we hope you'll consider increasing your score.

Reviewer_D8KwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DRY-SFT generates solutions sequentially, removes previous-solution context, and fine-tunes on the resulting problem–solution pairs without correctness filtering. It aims to improve pass@k under independent inference-time sampling. Code-generation experiments include matched self-training controls, held-out and cross-dataset evaluations, and analysis across models with different initial diversity.

Critical strengths
  1. The method transfers sequential-generation benefits into model weights while retaining parallel inference. Its training examples require neither correctness labels nor a reward model.
  2. The primary model gains approximately 11–13 percentage points in pass@100 in the main setting, where training and evaluation problems overlap, and outperforms matched self-training controls. Separate held-out and cross-dataset gains support transfer, while failures on some initially diverse models help delimit applicability.
Critical weaknesses
  1. The comparative advantage on unseen problems remains unclear. Figure 2 compares DRY-SFT with i.i.d. SFT and P-SFT on the problem sets used to construct training data. Figure 4 provides valuable held-out evidence, but only against the base model. The study therefore supports transfer without establishing whether sequential conditioning outperforms the matched self-training alternatives on unseen problems.
Questions for authors
  1. On the existing 50/50 held-out split, how do matched i.i.d. SFT and P-SFT compare with DRY-SFT under the same checkpoint-selection protocol?
1 comment

Thank you for your review and for recognizing the value of transferring sequential generation into independent sampling. We have run additional experiments based on your comments and have uploaded the revised manuscript to OpenReview, in which the changes are marked in blue.

W1.

The comparative advantage on unseen problems remains unclear. Figure 2 compares DRY-SFT with i.i.d. SFT and P-SFT on the problem sets used to construct training data. Figure 4 provides valuable held-out evidence, but only against the base model. The study therefore supports transfer without establishing whether sequential conditioning outperforms the matched self-training alternatives on unseen problems.

Thank you for identifying this missing comparison. We have now trained i.i.d. SFT and P-SFT on the same 50/50 train-test splits, with the same number of training targets and checkpoint-selection protocol as DRY-SFT.

Consistent with our previous findings, DRY-SFT exceeds i.i.d. SFT / P-SFT by +6.62 / +5.41, +5.54 / +5.74, and +9.66 / +12.97 pass@100+ points on HumanEval+, MBPP+, and DS-1000. The gains are statistically significant against both controls on every benchmark (paired 95% CIs in Table H.1, p. 19, lines 973–979). We added these results to Section 4.4 (p. 7, lines 359–367) and Figure 4 (p. 8, lines 391–395), showing that the coverage advantage also holds on unseen problems.

Q1.

On the existing 50/50 held-out split, how do matched i.i.d. SFT and P-SFT compare with DRY-SFT under the same checkpoint-selection protocol?

Thank you for raising this question. The comparisons above use the existing splits and the same checkpoint-selection protocol. We reserve 10% of the training split for validation and select the checkpoint with minimum validation cross-entropy.

We added these details to Appendix H, including: “The evaluation problems are excluded from fine-tuning and checkpoint selection.” (p. 18, lines 956–965).

Thank you again for your feedback! We hope the additional experiments have sufficiently addressed your held-out data concerns and clarified the checkpoint-selection setup. If so, we hope you'll consider increasing your score.

Reviewer_ZUxxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DRY-SFT, a data-engine paradigm concentrated on generating diverse training data. The method mainly includes two stages: sequentially generating diverse solutions through prompting the base model to deviate from prior solution(s) and fine-tune the model using different solutions. Throughout this process, the method aims to improve the generation diversity and the pass@K performance. Experiments demonstrate that DRY-SFT improves the pass@K (K>100) performance on several verified domains such as code generation compared to the base model and baseline.

Critical strengths
  1. The motivation of this method is clear and reasonable. Preserving the generation diversity is an important topic during model fine-tuning.
  2. The method and content are easy to follow and clear.
Critical weaknesses
  1. Hard to scale: given the context limit and the length of solution, this method is hard to scale when the task requires significant reasoning or agentic task.
  2. Using DRY-SFT leads to a consistent drop in pass@1, which is an important metric. Since in practice, it is not always feasible to generate multiple candidates for every task and select a correct solution.
  3. The proposed method is heuristic and lacks rigorous analysis. For example, an intuitive baseline is to prompt the base model to generate multiple diverse solutions at once.
Questions for authors
  1. What if the original task is extremely difficult for base model, for example a very low pass@100?
  2. If we apply RL to both models, does the DRY-SFT model retain any advantage over the base model?
  3. Can you explain the scalability of DRY-SFT?
1 comment

Thank you for your review and for recognizing the clarity of the method and the importance of preserving generation diversity. We have run additional experiments based on your comments and have uploaded the revised manuscript to OpenReview, in which the changes are marked in blue.

W1.

Hard to scale: given the context limit and the length of solution, this method is hard to scale when the task requires significant reasoning or agentic task.

Thank you for identifying this limitation. We agree that fitting multiple solution attempts into the context window could be an issue for tasks with long reasoning traces or agent trajectories. However, our K-sweep ablation demonstrates that even two solutions are sufficient to see meaningful coverage gains: K=2 improves pass@100+ by 7.9, 10.2, and 10.1 points on HumanEval+, MBPP+, and DS-1000.

We added this limitation stating that “fitting multiple prior attempts into the context window may not be feasible” and cite the K=2 results (Section 5, p. 9, lines 447–449).

W2.

Using DRY-SFT leads to a consistent drop in pass@1, which is an important metric. Since in practice, it is not always feasible to generate multiple candidates for every task and select a correct solution.

We agree that pass@1 matters in many settings, especially in settings where only one response can be generated. We instead focus the application of this method for settings where the goal is to obtain at least one correct solution and/or where many attempts can be generated and verified, such as math or coding.

We added this distinction explicitly to the limitations section: “DRY-SFT may be unsuitable for settings where only one response can be generated or single-sample accuracy is more important.” (Section 5, p. 9, lines 440–442). We also directly report the pass@1+ decrease in Section 4.1 (p. 5, lines 262–265).

W3.

The proposed method is heuristic and lacks rigorous analysis. For example, an intuitive baseline is to prompt the base model to generate multiple diverse solutions at once.

We would like to point the reviewer to Figure 2 of the original submission, which compares DRY-SFT with the base model, i.i.d. SFT, and P-SFT under matched training settings. The temperature comparison in Section 4.3 and the K-sweep ablation in Appendix C provide additional empirical analysis.

As for the recommended baseline, prompting the base model to generate multiple diverse solutions at once is nearly the same process as Stage 1 of DRY-SFT.

As the closest comparison to the proposed baseline, we also added coverage curves for the sequentially generated solutions alongside independent sampling from the base model and DRY-SFT (Appendix E, Figure E.1, p. 16, lines 833–837). The sequential curve uses the first one to five solutions generated during Stage 1.

Q1.

What if the original task is extremely difficult for base model, for example a very low pass@100?

Thank you for the question. In Section 4.1 we specifically analyze 600 problems that the base model could not solve in 200 attempts. Our results show DRY-SFT solves 244 of those 600 problems using the same 200-attempt budget.. We added a motivational sentence to Section 4.1 connecting back to your question (Section 4.1, p. 5, lines 266–269).

Q2.

If we apply RL to both models, does the DRY-SFT model retain any advantage over the base model?

We agree that this is an interesting question, and we believe it would be a natural future extension of this work. The scope of our contribution is primarily that DRY-SFT improves coverage without rewards or verifiers. Whether these gains persist after subsequent RL is a future direction we do not test in this paper.

We added this explicitly to the future directions: “Whether DRY-SFT provides a better model initialization for reinforcement learning remains an open empirical question.” (Section 5, p. 9, lines 460–461).

Q3.

Can you explain the scalability of DRY-SFT?

Thank you for bringing up this question. As mentioned in W1, sequential generation and the resulting large context usage are costs of training-data generation. However, inference remains independent and solutions can be generated in parallel. In order to clarify that this limitation concerns only training-data generation, we added the long-context limitation and the K=2 results in Section 5 (p. 9, lines 447–449).

Thank you again for your feedback! We hope our comments and additional experiments have sufficiently clarified the intended setting, the limitations, and how fine-tuning transfers diversity into independent sampling. If so, we hope you'll consider increasing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.