- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates recursive self-training of code large language models (LLMs) under different filtering conditions, ranging from no filtering to fixed exogenous filtering and model-coupled filtering. Through experiments covering four model families and multiple benchmarks, the authors compare compilation checks, static quality rules, perplexity-based filtering, and binary self-scoring. The results show that all evaluated approaches lead to a decline in model performance. The paper further explores the impact of coupling between the evaluator and the generator and provides a theoretical analysis of gated training and representation-level covariance concentration.
Critical strengths
-
The paper addresses a practically significant question: when a continuously evolving model is used both to generate code and to filter data for its own subsequent training, can the filtering process still provide a reliable quality signal?
-
The experiments span four families of code-generation models and multiple execution-based benchmarks. The round-by-round results demonstrate that, under the recursive training protocols used, these filtering rules fail to maintain the models’ initial performance levels.
Critical weaknesses
-
There are mathematical errors in the spectral analysis presented in the main text. Proposition 2.1 incorrectly calculates repeated covariance updates by taking powers of the singular values of a general matrix , and it incorrectly derives an exact limit from a lower bound. Appendix E and Section F.3 provide a revised result that requires to be a symmetric positive-semidefinite matrix and assumes the initial covariance is diagonal in the same eigenbasis, but the proposition in the main text omits these restrictions. The authors should revise the proposition and its proof and adjust the theoretical conclusions accordingly.
-
The experimental design fails to isolate the impact of coupling between the evaluator and the generator. Compile, Quality, and AI self-gates differ in both their acceptance criteria and whether the evaluator evolves alongside the generator. Therefore, the observed performance differences cannot be attributed solely to the exogeneity of the filtering process. The authors should compare frozen and updated versions of the same evaluator using the same scoring rule and matched sampling and training budgets. Furthermore, the claim that “stable recursive training requires external verification” should be qualified.
Questions for authors
-
The claim that compilation checks prevent the spread of runtime errors lacks sufficient support from the execution-based analysis. In Table 8, the 89.4% relaxed pass rate for Compile R10 includes 78.0% ImportError cases, while the strict pass rate is only 11.4%. Failed imports leave subsequent code untested. Could the authors address these dependency issues or label these samples as execution-unresolved, and base claims about executable validity on completed execution results?
-
The experiments do not distinguish between the effects of recursive feedback and those of continued fine-tuning. In Table 13, MBPP+ performance for Qwen2.5-Coder under the Compile condition drops from 0.582 to 0.045 in the first round before recovering to 0.127 by Round 5. Could the authors include a control experiment using a frozen initial generator to produce fresh samples, while keeping the filtering criteria and training budget constant, to isolate the additional effect of recursive feedback?
-
Key training details remain incomplete. Appendix N primarily covers generation batches and hardware. Could the authors provide the full training configuration and the actual amount of data used?
-
There is an inconsistency between the described PPL filtering criteria and the reported pass rates. The main text and Appendix H.1 state that the 25% of candidates with the lowest perplexity are retained from a pool of 20,000. However, Table 9 shows pass rates varying from 0.167 to 0.235. Additionally, Figure 4 depicts both Compile and Quality pass rates as 1.0, whereas Table 3 reports a raw gate pass rate of 0.898 for Compile. Could the authors clarify the denominators used for these metrics, the stages at which they are calculated, and whether any additional filtering steps are applied, and ensure that these descriptions are consistent throughout the paper?
-
An increase in pass rates or scores is not, by itself, sufficient evidence that filtering capability has degraded. The experiments retain the top 25% of samples based on their scores. The effectiveness of this approach depends on whether the ranking distinguishes between correct and incorrect code. However, Theorem 2.3 assumes that the acceptance function is constant on the support of the generator’s distribution. Could the authors clarify the connection between this assumption and the actual ranking-based filtering process, and provide details on the relationship between filtering scores and code correctness?
-
Table 2 states that the best-performing AI self-gate is selected based on Round 5 HumanEval+ results. For Code Llama, PPL was selected with a score of 0.134. Yet in Table 14, the corresponding score for Binary is 0.1585, which is higher. Could the authors explain this selection and verify the associated summaries and comparisons?
-
Appendix C.3 discusses additional runs, but Table 7 is labeled as “single-seed,” and the six sets of HumanEval and MBPP results match the corresponding Round 5 results in Tables 11 and 13. Are these results derived from independent runs? Could the authors report variability across repeated training runs to better support comparisons between different filtering strategies?
-
The layout and clarity of the figures and tables require improvement. On page 8, Figure 2 obscures the final row of Table 2. The double horizontal lines and spacing in Table 1 could be simplified, and the small text and formulas in Figure 1 should be made more legible. Please adjust the figure and table layouts, standardize the table formatting, and improve the readability of the labels in Figure 1.
4 comments
rebuttal to Reviewer sD9B
8. Question 6. Code Llama summary row.
Paper Table 14 confirms that Binary achieves the highest R5 HumanEval+ score among AI-based gates (0.1585), exceeding PPL (0.1341). Compile remains the strongest overall at 0.1707. We will correct Paper Table 2 to reflect these results. This correction does not change our main findings or conclusions.
Rebuttal Table R5. Code Llama-7B R5 pass@1 for the Paper Table 2 correction
| Gate | HumanEval | HumanEval+ | MBPP | MBPP+ | LiveCodeBench |
|---|---|---|---|---|---|
| Compile | 0.1890 | 0.1707 | 0.1243 | 0.0899 | 0.003 |
| PPL | 0.1646 | 0.1341 | 0.2063 | 0.1746 | 0.000 |
| Binary | 0.1829 | 0.1585 | 0.1667 | 0.1481 | 0.003 |
9. Question 7. Independence and variability.
Paper Table 7 summarizes the same single-seed runs reported in Tables 11 and 13, rather than independent repetitions. We will clarify this in Appendix C.3. Importantly, our findings are supported by comparisons across four code LLM families, multiple benchmarks, five selection strategies, and five training rounds. These results provide broad evidence of recursive self-training degradation across different experimental settings.
10. Question 8. Layout.
We will improve the presentation by separating Paper Figure 2 from Table 2, simplifying the rules in Table 1, and enlarging the labels and equations in Figure 1. These changes will make the experimental results and theoretical analysis easier to follow.
As the rebuttal period ends tomorrow, please let us know if you have any remaining concerns. We would be grateful if you could revisit your assessment in light of our rebuttal and additional experiments. Thank you again for your review and consideration!
We sincerely appreciate your time and thoughtful feedback. We hope you will carefully reconsider our contributions and the additional evidence provided. Our code is open-source under the MIT license to support reproducibility and future research. We would be truly grateful if you could revisit your assessment and consider raising your score.
rebuttal to Reviewer sD9B
Thank you for the detailed review. Our implementation is open source under MIT.
1. Weakness 1. Spectral assumptions.
Proposition E.1 and §F.3 establish spectral concentration under a symmetric positive-semidefinite update, a strict leading eigenvalue gap, and a shared covariance eigenbasis with positive leading variance. We will make these assumptions explicit in §2.2. This provides a rigorous theoretical basis for representation concentration under repeated updates.
2. Weakness 2. Reviewer coupling.
Our experiments provide both policy-level and controlled comparisons. Paper Table 1 and §3.2 compare complete training policies across five rounds. Rebuttal Table R1 (YWuj, Question 2) directly compares frozen and current reviewers using identical candidates, scoring rules, and selection quotas. At R5, frozen reviewers achieve higher AUROC and selection lift for both Binary and PPL. These matched results show that reviewer updates reduce filtering effectiveness, supporting our analysis of reviewer coupling.
3. Question 1. Import failures.
We distinguish dependency failures from successful execution. Paper Table 8 reports the original environment, while Table R4 adds strict execution results with pinned dependencies and a five-second timeout. Compile consistently achieves higher strict completion than Vanilla. At R5, Compile reaches 17.2% versus 9.4% for Vanilla, showing that compilation-based filtering improves execution reliability even under strict evaluation.
Rebuttal Table R4. Strict execution in the original and pinned environments
Both rates use all samples as the denominator. Paper Table 3 separately reports raw gate acceptance and accepted-set execution.
4. Question 2. Frozen generator.
Our design separates the initial fine-tuning effect from later recursive feedback. A frozen R0 generator producing fresh samples under the same Compile gate and training budget provides a direct control for recursive generation. Paper Table 13 (Rebuttal Table R2, VBf1) shows that the R1 decline occurs before any updated generator produces training data. Later rounds track the effect of repeatedly using updated models to generate training samples. This distinction allows us to examine initial training effects and recursive feedback separately.
5. Question 3. Training and data volume.
Our experiments follow a consistent training setup across models and gates. §3.1 and Appendices I and N specify five rounds with 3,000 updates per round. Vanilla, Compile, Quality, and Binary each retain 25,000 rows per run. PPL retains 21,046 for SantaCoder, 23,637 for StarCoder2, 20,221 for Qwen, and 24,104 for Code Llama. We use The Stack Dedup Python, 1,024-token prefixes and generation cap, temperature 0.8, top-p 0.95, 2,048-token packing, effective batch 64, peak LR 5×10−5, cosine decay, weight decay 0.05, and 500 R1 warmup steps. No generations or benchmark tasks were cherry-picked. We will release exact settings, prompt IDs, seeds, revisions, sample counts, hashes, and logs for full reproducibility.
6. Question 4. Pass-rate denominators.
The reported rates measure different stages and are consistent. The legacy PPL selector retains the lowest 25% of finite scores, giving 3,347 and 4,698 accepted samples out of 20,000 candidates at R1 and R5 for SantaCoder (Paper Table 9). The current selector targets 5,000 when enough finite scores are available. Paper Figure 4 measures execution among 5,000 accepted samples, while Table 3 measures raw acceptance (5,000/5,568 = 89.8%). Quality also limits repeated nonempty lines to 30% and requires at least 50 completion tokens. We will clarify these denominators.
7. Question 5. Constant acceptance versus ranking.
Theorem 2.3 establishes when self-gating loses its filtering benefit under candidate-independent acceptance. Our experiments provide direct evidence of declining ranking quality under fixed selection quotas. Table R1 (YWuj, Question 2) shows that current reviewers have lower AUROC and selection lift than frozen reviewers at R3 and R5 on identical candidates. This directly supports our finding that reviewer updates weaken code selection, beyond changes in scores alone.