- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes an execution-free data generation pipeline to reduce hallucinations in small code completion models. It mines fill-in-the-middle (FIM) examples from real repositories, uses developer-written code as positive samples, and generates synthetic incorrect completions as hard negatives. The authors further introduce a fool-rate metric based on LLM judges to select challenging negative samples. Experiments across multiple models and benchmarks show that targeted supervised fine-tuning significantly reduces code hallucinations, while contrastive training with hard negatives provides additional improvements on certain completion tasks.
Critical strengths
- The paper addresses a practical problem in code completion and proposes a scalable, execution-free pipeline that constructs training pairs from real code and synthetic mistakes.
- Targeted supervised fine-tuning substantially improves code completion accuracy and reduces verified hallucinations, demonstrating the effectiveness of carefully selected training examples.
- The paper provides extensive experiments and ablations across multiple model families, languages, and benchmarks. The analysis of negative sample difficulty, including human evaluations, is also valuable.
Critical weaknesses
- The contribution of synthetic hard negatives is relatively limited. Most performance gains come from standard SFT on carefully selected real code. On Delulu, ORPO achieves comparable or even slightly worse results than SFT, while hard-negative selection provides only marginal additional improvements.
- The benefits of contrastive training are inconsistent across tasks. Although hard negatives improve control-flow completion, they degrade performance on multi-line completion. Moreover, the random-hole SFT baseline achieves competitive or better control-flow results, raising questions about the necessity of the proposed negative generation pipeline.
- The quality and effectiveness of the generated negatives are not sufficiently established. Structural and category-level validation does not guarantee that all negatives are semantically incorrect. Furthermore, the advantage of high-fool-rate negatives becomes much smaller after controlling for edit distance and completion length, weakening the justification for the proposed difficulty metric.
Questions for authors
- Given that standard SFT accounts for most of the improvements, could the authors provide stronger evidence that synthetic hard negatives offer substantial benefits beyond carefully selected positive training examples?
- Why does contrastive training improve control-flow completion while degrading multi-line completion? Could the authors further investigate this trade-off and compare against stronger SFT baselines, particularly random-hole SFT?
- How do the authors ensure that generated hard negatives are genuinely incorrect rather than valid alternative completions? Could the authors further validate the fool-rate metric while controlling for factors such as edit distance and completion length?
4 comments
Random-hole SFT leads on SAFIM control flow (33.8%) because uniform spans directly sample if/for/while headers, yet it loses 5.3 EM points on Delulu and trails on multi-line as well. Mixing 50% API holes with 50% random spans closes the control-flow coverage gap (34.7%) while keeping Delulu EM at 59.7%. At the same time, adding random spans leaves Delulu call-site accuracy unchanged (59.7%), whereas contrastive pairs push Delulu EM higher on the exact same API holes (+1.9 EM, Table 5)—showing a clear division between SFT span mixture (syntactic coverage) and hard negatives (call-site discrimination).
3. [W3, Q3] Semantic Incorrectness of Negatives and Fool Rate Controlled for Edit Distance
W3 / Q3: "Structural and category-level validation does not guarantee that all negatives are semantically incorrect... How do the authors ensure that generated hard negatives are genuinely incorrect rather than valid alternative completions? Could the authors further validate the fool-rate metric while controlling for factors such as edit distance and completion length?"
Thank you for raising both points; our category constraints, registry checks, compiler verification, and controlled strata address them directly:
- Why generated negatives are genuinely incorrect:
- Scope-aware construction (Section 3.2, Appendices B–C, Table 8): Each prompt sees prefix , developer code , and suffix , and injects one targeted error (
92.3%conformance on a 36K audit). ForUNDEFINED VARIABLEandMETHOD, the prompt requires the injected identifier or method name to be absent from and , which excludes in-scope locals, parameters, and symbols defined or imported in the file; forPARAMETER, the callee and existing arguments stay fixed while an invented argument is added. - Registry verification across (Table 9): Across all 789 checkable package-level
IMPORTedits in six public registries (PyPI, npm, crates.io, Go, RubyGems, NuGet), 95.4% of hard () packages do not exist, and accidental matches to real packages fall from 9.5% () to 4.6% (). High- mistakes fool judges—and professional developers up to 41% of the time in our blind 400-item study (Appendix H)—because their names fit the code context, even though the symbols are fabricated. - Downstream compiler check (Tables 5 & 17): Delulu's containerized compiler/type-checker (
verify patch) accepts any valid alternative completion that compiles and type-checks. If high- negatives were mostly valid alternatives, pushing them down would hurt compiler pass rates; instead, hard-pair ORPO lowers verified hallucinations and raises EM across all four categories (Table 17).
- Scope-aware construction (Section 3.2, Appendices B–C, Table 8): Each prompt sees prefix , developer code , and suffix , and injects one targeted error (
- Fool rate under edit-distance and length control: Matching C# strata on character edit distance (bins of 4) and completion length (bins of 16) narrowed the Delulu gap (
+1.4for vs.+1.3for , Section 4.3) because many raw outputs were large rewrites, meaning naturally filters out non-local edits. Hard pairs still beat SFT after matching (+1.4), so we read as capturing both plausibility and edit locality, and we will state this explicitly in the revision. Small edit distance alone, however, does not make a useful negative: rule-based single-identifier edits are minimal by construction, yet they have and add +0.0 on Delulu (Table 4). Independent of edit size, also tracks how convincing a mistake is to people and to other model families: in our blind study, professional developers accept 9% of negatives and 41% of negatives, and Claude Haiku 4.5 plus a third-provider judge rank pairs consistently with the GPT panel (Spearman ; Appendix H).
We really appreciate reviewers constructive feedback and comments and would love to continue this conversation, so all issues and concerns are resolved. We would like to kindly ask the reviewer to revise their score if they believe that their concerns are resolved and the paper has merit to be published at ICLR.
Additional References
- Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., & Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024.
- Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., & White, C. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv:2402.13228.
- Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025.
- Saeidi, A., Verma, S., Uddin, M. N., & Baral, C. (2025). Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. ACL 2025.
Dear Reviewer,
We wanted to briefly follow up as the discussion period wraps up to see if our responses and additional experiments have addressed all of your questions and concerns.
If there are any remaining points that need further clarification or analysis, please let us know, we would be glad to address them right away. If everything is resolved, we would sincerely appreciate it if you would consider updating your evaluation and increasing your score.
Thank you again for your time and constructive feedback!
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the clarification of the control-flow and multi-line trade-off. However, I remain concerned about the limited gains of synthetic hard negatives over strong SFT baselines and the effectiveness of fool-rate after controlling for confounding factors. While some of my concerns have been addressed, I will maintain my original score. Nevertheless, I believe this paper is worthy of acceptance once the remaining concerns are adequately addressed.
Thank you for your careful review and for highlighting our execution-free pipeline, cross-family ablations, and developer study. Regarding Presentation, we appreciate your feedback and are streamlining Section 4.3 in the revision so the progression from unfiltered pairs to negative writers to fool-rate selection is much easier to follow. Below we respond directly to each of your weaknesses (
W1–W3) and questions (Q1–Q3).1. [W1, Q1] Role of SFT, Comparison with Preference Optimization, and Stronger Evidence for Hard Negatives
Thank you for raising this question. Three points explain why our SFT baseline is so strong, why unfiltered preference learning ties it, and when synthetic negatives add substantial value:
2. [W2, Q2] Control-Flow vs. Multi-Line Trade-Off and Comparison to
Random-Hole SFTThank you for asking us to examine this trade-off further; below we break down the multi-line regressions, note how objective normalization affects them, and compare directly against
random-hole SFTand a mixed-span SFT control:DPO+NLL(Section 3.3) preserves multi-line accuracy (29.4% vs. 29.9% 3-seed mean) while still lifting SAFIM control flow by +3.9 points (31.9% vs. 28.0%, Table 16).random-hole SFTand a 50:50 SFT mix: Comparing span selection on the exact same 5K files and updates isolates syntactic coverage from call-site accuracy:Random-hole SFT(Tables 3 & 16)API-hole SFT(Table 6)50:50 API + random SFT(new)