- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DRY-SFT, a method for improving the probability that at least one of several sampled solutions is correct. It sequentially generates multiple solutions conditioned on earlier attempts, then removes that conditioning and fine-tunes on each solution independently. No correctness filter or reward is used during the default training procedure. On three coding benchmarks, the method improves high-budget pass rates while reducing single-sample accuracy. Matched independent-generation controls, temperature experiments, held-out and cross-dataset evaluations, and structural-diversity measurements support the approach. A nine-model study additionally relates lower initial structural diversity to larger coverage gains.
Critical strengths
-
A simple intervention produces substantial coverage gains with a relevant control. Section 4.1 reports approximately 10.8–12.5-point pass@100 improvements across HumanEval+, MBPP+, and DS-1000, whereas the matched independent-generation control changes performance much less. Because the optimization recipe and number of targets are matched, this comparison directly supports the importance of conditioning target generation on prior attempts. Amortizing sequential generation into ordinary independent inference is a useful practical contribution.
-
The paper provides evidence of transfer and reports its trade-offs. Section 4.4 shows held-out pass@100 gains on all three benchmarks and transfer between HumanEval+ and MBPP+. Appendix E shows that gains persist with or without correctness filtering, while the paper reports the pass@1 decline and failures on already-diverse models. These qualifications make the contribution more credible and support acceptance as a method for improving coverage in the stated regime.
Critical weaknesses
-
The headline comparisons use training-problem adaptation rather than a fully held-out protocol. Section 4.4 states that the main experiments generate training targets and evaluate on the same problems. Absence of correctness labels does not make such exposure irrelevant. The later held-out results address transfer, but Figure 4 does not repeat the full independent-generation, P-SFT, and temperature comparisons there. Clarifying separation of chain-depth selection, checkpoint validation, and final evaluation would strengthen the claim that the comparative advantage generalizes to unseen problems.
-
The diversity evidence needs a more precise interpretation. Section 4 measures AST distance only on problems with at least two passing solutions among ten samples, potentially changing the evaluated problem subset between methods. Syntax-tree distance also does not directly establish algorithmic diversity. These issues particularly affect the nine-model correlation in Section 4.5, which is presented as a practical predictor of benefit. Shared-problem measurements and implementation details on identifier normalization would make that interpretation more convincing; the pass-rate results remain the more direct evidence of usefulness.
Questions for authors
-
Clarification: For the 50/50 held-out experiments, were chain depth, checkpoint selection, and all other choices determined without evaluating the held-out half? Are matched independent-generation or P-SFT results on those same splits already available?
-
Clarification: Are AST-diversity comparisons computed on a shared set of eligible problems across methods? How are identifiers, literals, and tree size handled, and do existing common-subset results preserve the reported diversity increase?
-
Clarification: Do the reported bootstrap intervals cover one training seed per configuration, and are any generation/training repeats available? This matters particularly for the nine-model predictive relationship and close comparisons with temperature tuning.
Thank you for your review and for recognizing the coverage gains and the importance of reporting the honest trade-offs. We have run additional experiments based on your comments and have uploaded the revised manuscript to OpenReview, in which the changes are marked in blue.
W1.
Thank you for making this point! We agree that excluding evaluation problems from fine-tuning matters even when training does not use correctness labels. We added a section stating which experiments use the same problems for training-target generation and evaluation, and which hold out evaluation problems from fine-tuning and checkpoint selection (Section 4, p. 5, lines 232–235).
We also added the matched held-out i.i.d. SFT and P-SFT comparisons. Consistent with our previous findings, DRY-SFT exceeds i.i.d. SFT / P-SFT by +6.62 / +5.41, +5.54 / +5.74, and +9.66 / +12.97 pass@100+ points on HumanEval+, MBPP+, and DS-1000 (paired 95% CIs in Table H.1, p. 19, lines 973–979). We added these results to Section 4.4 (p. 7, lines 359–367) and Figure 4 (p. 8, lines 391–395).
For temperature, we updated Figure 3 to use two-fold cross-validation on HumanEval+. Consistent with our previous findings, DRY-SFT exceeds this temperature baseline by 4.72 held-out pass@100+ points (paired 95% CI [1.79, 8.34]). Its AST diversity is 0.206 versus the base model's maximum of 0.158 on the same 104 held-out problems. We report these results in Section 4.3 (p. 7, lines 348–351) and the setup in Appendix I (pp. 19–20, lines 1005–1030).
W2.
Thank you for identifying this. It is a great point. We recomputed the diversity comparisons on the same 123 HumanEval+, 244 MBPP+, and 294 DS-1000 problems for all four methods where each method had two or more correct solutions. Under this specification, DRY-SFT maintains significantly higher AST diversity over all three baselines on each benchmark. We updated Figure 2 and Section 4.2 with these results (p. 6, lines 295–299 and 307–313).
To address your other point, we also recomputed the nine-model comparison using the same 81 problems to measure baseline diversity. Lower baseline diversity remains significantly associated with larger coverage gains over base (R²=0.665, p=0.0074). We updated Figure 5 and Section 4.5 (p. 8, lines 413–420 and 422–427).
We agree that AST distance does not establish algorithmic diversity. We added this limitation explicitly: “This measures structural variation, and does not guarantee algorithmic or semantic diversity.” (Section 4, p. 5, lines 229–230). We also describe the nine-model result as an association in the models studied (Section 4.5, p. 7, line 377).
Q1.
Thank you for the question. The evaluation problems were excluded from fine-tuning and checkpoint selection. However, K=5 was chosen using the broader benchmark ablation. We added this distinction explicitly to Section 4.4: “these results are not a fully independent evaluation of hyperparameter selection.” (p. 7, lines 363–365).
We added the training, validation, and evaluation counts and the checkpoint-selection rule to Appendix H (p. 18, lines 956–961), clarifying which problems were used for each step.
Q2.
We retain node types and child structure but omit identifier names and literal values. Each pair's edit distance is divided by the larger tree's node count, then averaged within and across problems.
We added these measurement details to Appendix J (p. 20, lines 1034–1042), along with the sampling procedure for the nine-model comparison (p. 20, lines 1075–1078).
Q3.
Yes, each fine-tuned model has one training run. We do not report repeated training runs or regenerated training datasets. We added this clarification: “The intervals measure variation across problems, not repeated training runs.” (Section 4, p. 5, lines 222–223).
We also clarified that the temperature comparison resamples problems within each fold (Appendix I, p. 19, lines 1013–1015), while the nine-model regression calculates confidence intervals over models (Appendix J, p. 21, lines 1080–1081). We added 95% bootstrap confidence bands to the individual models' coverage curves in Figure G.1 (p. 18, lines 948–951).
Thank you again for your feedback! We hope our comments and additional experiments have sufficiently addressed your held-out data and diversity concerns and clarified model selection and what the confidence intervals measure. If so, we hope you'll consider increasing your score.