acceptodds
Under review as a conference paper at ICLR 2027

Real Code, Synthetic Mistakes: Teaching Small Code Models Not to Hallucinate

Abstract

Code autocomplete runs on small open models: every keystroke needs an answer within milliseconds, and much source code is not allowed to leave the company that owns it. These models often hallucinate. They call methods that do not exist, pass arguments an API does not accept, or import packages that were never published. The usual remedies do not fit autocomplete: execution-based filtering needs tests that a half-written file does not have, and preference learning needs labelled pairs that nobody collects for individual cursor positions. We start from a simple observation. Every fill-in-the-middle (FIM) example mined from a real repository already contains a correct answer, the code the developer wrote, so a frontier model only has to write a convincing mistake. We turn this into an execution-free pipeline that mines 400K realistic FIM holes across eight languages and produces 2.47M hard negatives in four hallucination categories. With this data we study two questions. First, fine-tuning a 7B model on only 5K curated holes raises exact match on the Delulu hallucination benchmark from 49.6% to 61.2% for a FIM-pretrained model (42.9% to 59.7% for its Instruct variant) and more than halves the rate of completions that fail Delulu's compiler check. Randomly placed holes and cutting outputs to the right length fall well short of this, and the gains transfer to three model families and to languages never seen in training. Second, the synthetic negatives add value when chosen well. Contrastive training on them improves control-flow completion by 1.5–3.5 points over fine-tuning alone, although it does not help long multi-line completions. A cheap, label-free difficulty signal, the fraction of blind LLM judges a negative fools, finds the pairs that help: ORPO trained on the pairs that fool the judges beats fine-tuning on Delulu by 1.5–1.9 points at the same budget, while easy or rule-based negatives add almost nothing. We release the code for every stage of the pipeline after hole mining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fill-in-the-middle Code Completion, Large Language Models, Code Agents, Hallucination detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oaFNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an execution-free data generation pipeline to reduce hallucinations in small code completion models. It mines fill-in-the-middle (FIM) examples from real repositories, uses developer-written code as positive samples, and generates synthetic incorrect completions as hard negatives. The authors further introduce a fool-rate metric based on LLM judges to select challenging negative samples. Experiments across multiple models and benchmarks show that targeted supervised fine-tuning significantly reduces code hallucinations, while contrastive training with hard negatives provides additional improvements on certain completion tasks.

Critical strengths
  1. The paper addresses a practical problem in code completion and proposes a scalable, execution-free pipeline that constructs training pairs from real code and synthetic mistakes.
  2. Targeted supervised fine-tuning substantially improves code completion accuracy and reduces verified hallucinations, demonstrating the effectiveness of carefully selected training examples.
  3. The paper provides extensive experiments and ablations across multiple model families, languages, and benchmarks. The analysis of negative sample difficulty, including human evaluations, is also valuable.
Critical weaknesses
  1. The contribution of synthetic hard negatives is relatively limited. Most performance gains come from standard SFT on carefully selected real code. On Delulu, ORPO achieves comparable or even slightly worse results than SFT, while hard-negative selection provides only marginal additional improvements.
  2. The benefits of contrastive training are inconsistent across tasks. Although hard negatives improve control-flow completion, they degrade performance on multi-line completion. Moreover, the random-hole SFT baseline achieves competitive or better control-flow results, raising questions about the necessity of the proposed negative generation pipeline.
  3. The quality and effectiveness of the generated negatives are not sufficiently established. Structural and category-level validation does not guarantee that all negatives are semantically incorrect. Furthermore, the advantage of high-fool-rate negatives becomes much smaller after controlling for edit distance and completion length, weakening the justification for the proposed difficulty metric.
Questions for authors
  1. Given that standard SFT accounts for most of the improvements, could the authors provide stronger evidence that synthetic hard negatives offer substantial benefits beyond carefully selected positive training examples?
  2. Why does contrastive training improve control-flow completion while degrading multi-line completion? Could the authors further investigate this trade-off and compare against stronger SFT baselines, particularly random-hole SFT?
  3. How do the authors ensure that generated hard negatives are genuinely incorrect rather than valid alternative completions? Could the authors further validate the fool-rate metric while controlling for factors such as edit distance and completion length?
4 comments

Thank you for your careful review and for highlighting our execution-free pipeline, cross-family ablations, and developer study. Regarding Presentation, we appreciate your feedback and are streamlining Section 4.3 in the revision so the progression from unfiltered pairs to negative writers to fool-rate selection is much easier to follow. Below we respond directly to each of your weaknesses (W1–W3) and questions (Q1–Q3).

1. [W1, Q1] Role of SFT, Comparison with Preference Optimization, and Stronger Evidence for Hard Negatives

W1 / Q1: "On Delulu, ORPO achieves comparable or even slightly worse results than SFT, while hard-negative selection provides only marginal additional improvements. Given that standard SFT accounts for most of the improvements, could the authors provide stronger evidence that synthetic hard negatives offer substantial benefits beyond carefully selected positive training examples?"

Thank you for raising this question. Three points explain why our SFT baseline is so strong, why unfiltered preference learning ties it, and when synthetic negatives add substantial value:

  1. Over half of the paper is dedicated to whether and how SFT on public code reduces FIM hallucinations: Our study first asks how far SFT on public repositories can curb hallucinations without execution, devoting Section 4.2 and Appendices E–G to row and update scaling, hole placement, and transfer across model families and languages. For example, 5K third-party API holes (~140 updates) cut verified hallucinations from 29.3% to 12.7% versus 19.2% for random FIM spans on the exact same files and updates (Table 3), and already approach our 100K run (Table 15). We would like to emphasize the value of this contribution alongside our hard-negative studies, not just as a baseline to beat.
  2. Showing when robust SFT matches DPO/ORPO is a deliberate takeaway: While post-training papers often expect DPO or ORPO (Rafailov et al., 2023; Hong et al., 2024) to beat SFT automatically, recent analyses show preference optimization often yields small gains over a properly tuned SFT, or degrades outputs through likelihood displacement, unless pairs add strong signal (Pal et al., 2024; Ivison et al., 2024; Razin et al., 2025; Saeidi et al., 2025). Giving SFT the exact same curated holes and update exposure shows unfiltered ORPO ties it (59.6% vs. 59.7% EM, Table 2; unanchored DPO drops to 29.4%, Section 4.3) because crafting hard negatives is difficult even with frontier teachers: 60% of raw mistakes fool zero blind judges (τ=0\tau = 0, Appendix H). Reporting this candidly clarifies when curated SFT suffices versus when negative mining is worth adding.
  3. Substantial gains when pairs reach high fool rate (τ≥2/3\tau \ge 2/3): In Table 6, taking the max over only 2–3 candidates per hole raised mean τ\tau (fraction of blind LLM judges accepting the negative) modestly from 0.22 to 0.29, yielding +0.7 on Delulu alongside +3.6 points on SAFIM control flow (see also Table 17). When pairs actually reach τ≥2/3\tau \ge 2/3, ORPO beats SFT trained on the same holes by +1.9 EM, whereas τ=0\tau = 0 pairs add +0.0 (Table 5).

2. [W2, Q2] Control-Flow vs. Multi-Line Trade-Off and Comparison to Random-Hole SFT

W2 / Q2: "Why does contrastive training improve control-flow completion while degrading multi-line completion? Could the authors further investigate this trade-off and compare against stronger SFT baselines, particularly random-hole SFT?"

Thank you for asking us to examine this trade-off further; below we break down the multi-line regressions, note how objective normalization affects them, and compare directly against random-hole SFT and a mixed-span SFT control:

  • Mechanism of the trade-off: Most negatives alter the end of the developer's completion gg, concentrating contrastive push-down at the span boundary. In 79% of ORPO's SAFIM control-flow fixes over SFT, ORPO corrects an SFT boundary overrun (Section 4.3). On HumanEval multi-line (118 items), ORPO's length-normalized odds ratio loses 9 items to SFT (and gains 2), shortening outputs on those items (median 2 lines vs. 3 for SFT and the reference; one third stop after a valid prefix and the rest diverge inside the block). Replacing mean log-probability in ORPO with sum-based anchored DPO+NLL (Section 3.3) preserves multi-line accuracy (29.4% vs. 29.9% 3-seed mean) while still lifting SAFIM control flow by +3.9 points (31.9% vs. 28.0%, Table 16).
  • Comparison to random-hole SFT and a 50:50 SFT mix: Comparing span selection on the exact same 5K files and updates isolates syntactic coverage from call-site accuracy:
5K SFT span designDelulu EMSAFIM CtrlHE Multi
Random-hole SFT (Tables 3 & 16)54.433.828.0
API-hole SFT (Table 6)59.728.830.5
50:50 API + random SFT (new)59.734.729.7

Random-hole SFT leads on SAFIM control flow (33.8%) because uniform spans directly sample if/for/while headers, yet it loses 5.3 EM points on Delulu and trails on multi-line as well. Mixing 50% API holes with 50% random spans closes the control-flow coverage gap (34.7%) while keeping Delulu EM at 59.7%. At the same time, adding random spans leaves Delulu call-site accuracy unchanged (59.7%), whereas τ≥2/3\tau \ge 2/3 contrastive pairs push Delulu EM higher on the exact same API holes (+1.9 EM, Table 5)—showing a clear division between SFT span mixture (syntactic coverage) and hard negatives (call-site discrimination).

3. [W3, Q3] Semantic Incorrectness of Negatives and Fool Rate Controlled for Edit Distance

W3 / Q3: "Structural and category-level validation does not guarantee that all negatives are semantically incorrect... How do the authors ensure that generated hard negatives are genuinely incorrect rather than valid alternative completions? Could the authors further validate the fool-rate metric while controlling for factors such as edit distance and completion length?"

Thank you for raising both points; our category constraints, registry checks, compiler verification, and controlled strata address them directly:

  • Why generated negatives are genuinely incorrect:
    1. Scope-aware construction (Section 3.2, Appendices B–C, Table 8): Each prompt sees prefix pp, developer code gg, and suffix ss, and injects one targeted error (92.3% conformance on a 36K audit). For UNDEFINED VARIABLE and METHOD, the prompt requires the injected identifier or method name to be absent from pp and ss, which excludes in-scope locals, parameters, and symbols defined or imported in the file; for PARAMETER, the callee and existing arguments stay fixed while an invented argument is added.
    2. Registry verification across τ\tau (Table 9): Across all 789 checkable package-level IMPORT edits in six public registries (PyPI, npm, crates.io, Go, RubyGems, NuGet), 95.4% of hard (τ≥2/3\tau \ge 2/3) packages do not exist, and accidental matches to real packages fall from 9.5% (τ=0\tau = 0) to 4.6% (τ≥2/3\tau \ge 2/3). High-τ\tau mistakes fool judges—and professional developers up to 41% of the time in our blind 400-item study (Appendix H)—because their names fit the code context, even though the symbols are fabricated.
    3. Downstream compiler check (Tables 5 & 17): Delulu's containerized compiler/type-checker (verify patch) accepts any valid alternative completion that compiles and type-checks. If high-τ\tau negatives were mostly valid alternatives, pushing them down would hurt compiler pass rates; instead, hard-pair ORPO lowers verified hallucinations and raises EM across all four categories (Table 17).
  • Fool rate under edit-distance and length control: Matching C# strata on character edit distance (bins of 4) and completion length (bins of 16) narrowed the Delulu gap (+1.4 for τ≥2/3\tau \ge 2/3 vs. +1.3 for τ=0\tau = 0, Section 4.3) because many raw τ=0\tau = 0 outputs were large rewrites, meaning τ>0\tau > 0 naturally filters out non-local edits. Hard pairs still beat SFT after matching (+1.4), so we read τ\tau as capturing both plausibility and edit locality, and we will state this explicitly in the revision. Small edit distance alone, however, does not make a useful negative: rule-based single-identifier edits are minimal by construction, yet they have τˉ=0.09\bar\tau = 0.09 and add +0.0 on Delulu (Table 4). Independent of edit size, τ\tau also tracks how convincing a mistake is to people and to other model families: in our blind study, professional developers accept 9% of τ=0\tau = 0 negatives and 41% of τ=1\tau = 1 negatives, and Claude Haiku 4.5 plus a third-provider judge rank pairs consistently with the GPT panel (Spearman ρ=0.50\rho = 0.50; Appendix H).

We really appreciate reviewers constructive feedback and comments and would love to continue this conversation, so all issues and concerns are resolved. We would like to kindly ask the reviewer to revise their score if they believe that their concerns are resolved and the paper has merit to be published at ICLR.

Additional References

  • Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., & Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024.
  • Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., & White, C. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv:2402.13228.
  • Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025.
  • Saeidi, A., Verma, S., Uddin, M. N., & Baral, C. (2025). Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. ACL 2025.

Dear Reviewer,

We wanted to briefly follow up as the discussion period wraps up to see if our responses and additional experiments have addressed all of your questions and concerns.

If there are any remaining points that need further clarification or analysis, please let us know, we would be glad to address them right away. If everything is resolved, we would sincerely appreciate it if you would consider updating your evaluation and increasing your score.

Thank you again for your time and constructive feedback!

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, particularly the clarification of the control-flow and multi-line trade-off. However, I remain concerned about the limited gains of synthetic hard negatives over strong SFT baselines and the effectiveness of fool-rate after controlling for confounding factors. While some of my concerns have been addressed, I will maintain my original score. Nevertheless, I believe this paper is worthy of acceptance once the remaining concerns are adequately addressed.

Reviewer_3HE8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an execution-free synthetic data pipeline to reduce hallucinations in small code models. It mines realistic fill-in-the-middle (FIM) examples from real repositories, generates plausible incorrect completions using frontier LLMs, and trains smaller models using SFT and preference optimization. Experiments show improvements in code completion accuracy, with additional gains from selecting high-quality synthetic hard negatives.

Critical strengths
  1. The paper introduces an approach to constructing code completion training data without execution-based verification, leveraging real developer-written code as positive examples and LLM-generated mistakes as negative examples.

  2. The paper evaluates multiple training objectives, model families, programming languages, and FIM benchmarks. The results demonstrate substantial improvements in targeted code completion accuracy and provide useful ablations on data selection and negative sample difficulty.

Critical weaknesses
  1. The paper does not clearly distinguish hallucination from general code generation failures. Inventing nonexistent APIs or using incorrect arguments may simply reflect insufficient coding knowledge or model capability, rather than a distinct hallucination problem. This weakens the motivation and interpretation of the reported improvements.

  2. Most performance gains come from targeted SFT on real FIM examples, while synthetic negatives provide only marginal additional improvements on the primary Delulu benchmark. This raises questions about whether the proposed synthetic mistake generation pipeline is necessary.

  3. The evaluation focuses primarily on FIM and hallucination-related benchmarks. Although some cross-language and cross-benchmark transfer is demonstrated, it remains unclear whether the improvements extend to broader code generation, or SWE tasks. The results may largely reflect task-specific adaptation rather than general improvements in coding capability.

Questions for authors

How do the authors distinguish code hallucination from general code generation errors caused by insufficient model knowledge or capability?

Can the authors evaluate the method on broader code generation benchmarks, such as LiveCodeBench, or SWE-bench verified, to demonstrate that the improvements extend beyond targeted FIM completion?

3 comments

Thank you for taking the time to review our paper and for recognizing our execution-free pipeline and ablations across objectives, model families, and languages. Below we address each of the weaknesses (W1–W3) and questions (Q1–Q2) you raised.

1. [W1, Q1] Distinguishing Hallucination from Missing Model Knowledge

W1 / Q1: "Inventing nonexistent APIs or using incorrect arguments may simply reflect insufficient coding knowledge or model capability... How do the authors distinguish code hallucination from general code generation errors caused by insufficient model knowledge or capability?"

Thank you very much for your constructive feedback, we agree that clarifying how FIM hallucinations differ from general knowledge gaps will strengthen Sections 1 and 4.2, and we will revise the text accordingly. At the same time, we would like to elaborate on how our verification setup and transfer results separate context-grounding failures from missing library knowledge.

Following Delulu (Erfanian et al., 2026), we study referential hallucinations: syntactically valid completions that invoke a nonexistent method, argument, variable, or package, checked statically in Delulu's compiler and type-checker containers. While errors on external APIs absent from both the file and pretrained weights do reflect missing parametric knowledge (contributing to the ~12.5% residual error floor), if the base model's 29.3% failure rate came solely from missing knowledge or weak coding capacity, ~140 updates on short FIM spans could only fix the specific APIs seen in training. Instead, over half of those failures disappear because the model already has the needed knowledge in the file or in pretrained weights, yet fails to ground on surrounding FIM context. We would like to point out three notes pointing to this finding:

  1. In-file context is ignored: In our example in Figure 1, terminal.ReadPassword(...) appears three lines above the cursor, yet the base 7B model invents CapturePassword(...).
  2. Unseen APIs improve: 5K SFT gains +14.8 EM points on the 71% of Delulu holes whose called method never appears in training (Appendix A).
  3. Unseen languages improve: With zero C++ or TypeScript training rows, 100K SFT lifts C++ EM by +40.0 points (and TypeScript as well; Section 4.2, Table 7).

2. [W2] Role of SFT, Comparison with Preference Optimization, and When Negatives Are Needed

W2: "Most performance gains come from targeted SFT on real FIM examples, while synthetic negatives provide only marginal additional improvements on the primary Delulu benchmark. This raises questions about whether the proposed synthetic mistake generation pipeline is necessary."

Thank you for raising this concern. Clarifying how targeted SFT and synthetic hard negatives fit together goes to the core message of our paper:

  1. Over half of the paper is dedicated to whether and how SFT on public code reduces FIM hallucinations: Our study first asks how far SFT on public repositories can curb hallucinations without execution, devoting Section 4.2 and Appendices E–G to row and update scaling, hole placement, and transfer across model families and languages. For example, 5K third-party API holes (~140 updates) cut verified hallucinations from 29.3% to 12.7% versus 19.2% for random FIM spans on the exact same files and updates (Table 3), and already approach our 100K run (Table 15). We would like to emphasize the value of this contribution alongside our hard-negative studies, not just as a baseline to beat.

  2. Showing when robust SFT matches DPO/ORPO is itself an important finding: Post-training pipelines commonly expect DPO or ORPO (Rafailov et al., 2023; Hong et al., 2024) to beat SFT automatically, yet recent studies show preference optimization often yields marginal gains over a well-tuned SFT, or hurts via likelihood displacement, unless pairs add strong contrastive signal (Pal et al., 2024; Ivison et al., 2024; Razin et al., 2025; Saeidi et al., 2025). On FIM, when SFT receives curated holes and adequate updates, unfiltered ORPO ties it (59.6% vs. 59.7% EM, Table 2; unanchored DPO drops to 29.4%, Section 4.3) because 60% of raw frontier mistakes still fool zero blind judges (τ=0\tau = 0, Appendix H). Reporting this candidly helps practitioners know when curated SFT already suffices and when to be cautious before building a negative-mining pipeline.

  3. When synthetic negatives are worth adding: On the 5K multilingual pool, taking the best of only 2–3 candidates per hole raised mean τˉ\bar\tau modestly from 0.22 to 0.29, yielding +0.7 EM on Delulu across all four categories alongside +3.6 points on SAFIM control flow (Tables 6 & 17). When pairs all reach τ≥2/3\tau \ge 2/3 on the same holes, ORPO beats matched SFT by +1.9 EM, whereas τ=0\tau = 0 pairs add +0.0 (Table 5).

3. [W3, Q2] Broader Code Generation (LiveCodeBench and SWE-bench)

W3 / Q2: "Can the authors evaluate the method on broader code generation benchmarks, such as LiveCodeBench, or SWE-bench verified, to demonstrate that the improvements extend beyond targeted FIM completion?"

Thank you for suggesting LiveCodeBench; we agree that checking general code generation is helpful and report those results below. Regarding SWE-bench Verified, our recipe is a lightweight adaptation (~140 updates on 5K single-span holes) for sub-second local FIM autocomplete, whereas SWE-bench evaluates multi-turn repository agents (bash execution, file search, and multi-file edits) requiring agentic post-training trajectories.

Our goal is FIM completion, so we do not expect training on 5K infilling holes to improve prompt-to-code benchmarks, and most FIM models (e.g., base checkpoints) cannot be evaluated on them at all. Our main model, however, is Qwen2.5-Coder-7B-Instruct, which supports both FIM and chat. This lets us run our FIM-tuned checkpoints in LiveCodeBench's standard chat setting, with the same prompts and decoding as the untuned model, and check whether FIM training harms general code generation. Beyond our cross-family FIM gains and out-of-taxonomy CoderEval transfer (+10.1 pairwise accuracy across 11 full-function error categories; Figure 4, Table 21), we evaluated our 5K checkpoints on LiveCodeBench (release_v6, all 1,055 problems, greedy pass@1):

Qwen2.5-Coder-7B-InstructLiveCodeBench pass@1 ↑Delulu Hall. ↓
Base29.729.3
SFT 5K29.512.7
Hard-pair ORPO 5K29.012.5

As expected, FIM training leaves general prompt-to-code generation unchanged (all three are within 0.7 points, below one standard error of about 1.4 points), while verified API hallucinations drop by more than half. We will add this table to the appendix.

We really appreciate reviewers constructive feedback and comments and would love to continue this conversation, so all issues and concerns are resolved. We would like to kindly ask the reviewer to revise their score if they believe that their concerns are resolved and the paper has merit to be published at ICLR.

Additional References

  • Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., & Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024.
  • Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., & White, C. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv:2402.13228.
  • Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025.
  • Saeidi, A., Verma, S., Uddin, M. N., & Baral, C. (2025). Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. ACL 2025.

Dear Reviewer,

We wanted to briefly follow up as the discussion period wraps up to see if our responses and additional experiments have addressed all of your questions and concerns.

If there are any remaining points that need further clarification or analysis, please let us know, we would be glad to address them right away. If everything is resolved, we would sincerely appreciate it if you would consider updating your evaluation and increasing your score.

Thank you again for your time and constructive feedback!

Reviewer_y3iPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method for preparing datasets for fine-tuning LLMs to reduce FIM API hallucinations. The method involves using an LLM to identify locations to split a code file into prefix/suffix/completion tuples. The paper also proposes using an LLM to prepare contrastive pairs, where the original developer code is a positive/preferred response, and LLM written fabricated responses are hard negatives. Experiments show that SFT over the developer written code alone is a very strong baseline, with limited gains using the contrastive pairs for narrow settings such as code for control flow.

Critical strengths

The paper aims to reduce hallucinations for code auto-completion for small models, which is a narrow, yet important problem.

Critical weaknesses
  1. The paper starts with the sentence: "Most requests to a code assistant are autocomplete requests" with no evidence for this. In fact, with the rise of coding agents, its more pressing on the authors to back a claim such as this. Overall, I believe the authors should demonstrate that code auto-completion continues to be an important use case in the real-world as a motivation for their study. At the moment, this is not substantiated.

  2. The paper uses old LLMs such as Qwen-2.5-Coder-7B-Instruct, etc. in their evaluations. This model was released roughly two years ago. LLM development is very fast moving, and I'd suspect newer LLMs in the same size class to be less hallucinatory than their evaluations indicate. As such, the validity of their claims for the newer LLMs is questionable.

  3. My overall takeaway from the paper was that SFT on the original code is largely sufficient to reduce hallucinations for small LLMs. The authors identified niche scenarios (like control flow) where their main contribution - LLM based contrastive pair mining - is helpful. It's unclear if this was a special case, or if there is some underlying pattern that can help me predict if contrastive pair mining and subsequent preference fine-tuning will help reduce hallucinations.

  4. [not a major concern] The proposed method for contrastive pair mining (asking an LLM to modify GT code) is straightforward. However, if the results and analyses are strong on their own, this won't be a major concern. I'm not certain, at the moment, this is the case. I've also not rigorously validated the novelty of this work.

Questions for authors
  1. How do your evaluations hold up against more recent LLMs in the same size class?
  2. Can we predict when contrastive pair mining should help in a more general way?
5 comments

Thank you for your thoughtful feedback on our paper, and we appreciate your evaluation of the paper's soundness. Below we address each of your raised weaknesses (W1–W4) and questions (Q1–Q2).

1. [W1] Why Autocomplete Remains Important Alongside Coding Agents

W1: "The paper starts with the sentence: 'Most requests to a code assistant are autocomplete requests' with no evidence for this... Overall, I believe the authors should demonstrate that code auto-completion continues to be an important use case in the real-world as a motivation for their study."

First, we would like to thank you for this valid concern and valuable feedback. We agree that our opening sentence could use citations for clarification, and we will revise it accordingly. At the same time, we would like to elaborate on why that statement holds in practice. Even in 2026 alongside coding agents, inline completions remain a major, high-volume workload in everyday development, and we share a couple of supporting points here:

  • Coexisting with agents at scale: A 2026 study of over 500,000 developers (Demirer et al., 2026) analyzes autocomplete, interactive agents, and autonomous agents as three active, coexisting tool classes in production workflows. Companies continue to invest actively in this area; for example, JetBrains released Mellum-4B specifically as a 4B model for production in-IDE completion (Pavlichenko et al., 2025), and Meta's CodeCompose serves sub-second suggestions to tens of thousands of engineers, accounting for ~8% of all code typed (Murali et al., 2024).

  • Direct fit to API recall: Surveys show developers rely on inline completion mainly to save keystrokes and recall unfamiliar API names and signatures while typing (Liang et al., 2024), matching our call-site focus.

Given these points, we believe that research on FIM code completion remains a well-motivated problem for both academia and industry, and in light of your comment, we will add these citations at the start of Section 1 so this motivation is well supported from the opening line.

2. [W2, Q1] Choice of Qwen2.5-Coder and Results on Newer Small Models

W2 / Q1: "The paper uses old LLMs such as Qwen-2.5-Coder-7B-Instruct, etc. in their evaluations... How do your evaluations hold up against more recent LLMs in the same size class?"

Thank you for raising this question. We are glad to add newer 4B–8B models to the revision, and we would first like to explain why Qwen2.5-Coder-3B/7B (alongside StarCoder2-7B and CodeLlama-7B) remains a strong, representative choice for local FIM autocomplete today.

  • Purpose-built for FIM: Qwen2.5-Coder was pretrained on 5.5T code-focused tokens with explicit file- and repo-level FIM objectives (Hui et al., 2024), achieving top open-weight FIM accuracy at its scale on HumanEval-Infilling, CrossCodeEval, and SAFIM. However, among successor models released afterward, dedicated -Coder models such as Qwen3-Coder (30B-A3B, 480B-A35B) moved to MoE architectures requiring 60 GB+ in bf16 for cloud-hosted agents and are impractical for local FIM tasks, while dense Qwen3-4B/8B focus on general reasoning with thinking modes. Sub-second tab completion needs a fast, non-reasoning, FIM-pretrained model fitting in 4–16 GB of memory, which is why Qwen2.5-Coder-3B/7B remain default local autocomplete models in tools like Continue, Tabby, and Ollama, and why our submission already shows +15 to +35 EM gains across three 7B model families (Figure 3, Table 19).

In addition, this study includes a very comprehensive set of experiments designed to evaluate different training paradigms and hyperparameters, which took months to set up, run, and verify. Given the rapid pace of model releases, we tried our best to use the latest strong models available, though our resources are too limited to rerun all experiments on every new release, as even these newer models may become dated by the time of publication.

Still, we definitely agree that having results on newer models helps show that our findings are not bound to one specific model. To test newer 2025 models directly under the exact same 5K holes and updates, we evaluated Seed-Coder-8B-Base (2025, native FIM) and Mellum-4B-Base (2025, JetBrains' IDE completion model), alongside Qwen3-8B-Base (2025, general-purpose):

FIM-pretrained model (5K holes, matched updates)Delulu EM ↑ (Base → SFT 5K)Base Hall. ↓SFT 5K Hall. ↓
Qwen2.5-Coder-7B-Instruct (Table 6)42.9 → 59.7 (+16.8)29.312.7
Seed-Coder-8B-Base (2025)52.9 → 61.1 (+8.2)8.27.5
Mellum-4B-Base (2025)45.3 → 51.0 (+5.7)14.210.1

Newer 2025 FIM-pretrained models start stronger than Qwen2.5-Coder yet still miss roughly half of Delulu's API call sites out of the box (8.2% and 14.2% verified hallucinations) because a 4B–8B model cannot memorize every third-party signature across seven languages, and our 5K SFT holes improve both metrics on each model (+8.2 and +5.7 EM, cutting hallucinations to 7.5% and 10.1%).

We excluded Qwen3-8B from the table above because our initial baseline evaluation showed it is not a viable FIM model out of the box. it lacks dedicated FIM pretraining, it ignores the <|fim_middle|> marker and scored 0.1% EM on Delulu (1/1,950 holes) and 0.0% EM on SAFIM control and HumanEval multiline, either appending unrelated code past the end of the file or emitting conversational explanations of the snippet. Because its base outputs are not candidate infills, splicing them into the hole fails at parsing rather than testing API recall, making Base Hall. undefined. Even after our 5K SFT holes teach it the FIM format and lift Delulu EM to 44.3%, a few thousand post-training steps cannot replace trillions of tokens of FIM pretraining,its fine-tuned accuracy barely reaches where FIM-pretrained coders start before adaptation (43%–53%), confirming that general-purpose Qwen3 models are not suitable candidates here and why purpose-built FIM models remain the right testbed.

3. [W3, W4, Q2] Role of SFT, Honest Comparison with Preference Optimization, and Predicting When Contrastive Pairs Help

W3 / W4 / Q2: "My overall takeaway from the paper was that SFT on the original code is largely sufficient to reduce hallucinations for small LLMs. The authors identified niche scenarios (like control flow) where their main contribution - LLM based contrastive pair mining - is helpful... Can we predict when contrastive pair mining should help in a more general way?"

Thank you for sharing this takeaway; we appreciate the opportunity to clarify why our SFT study is itself a central contribution and when contrastive pair mining pays off.

1. Over half of the paper is dedicated to whether and how SFT on public code reduces FIM hallucinations. Our paper first asks whether SFT on public repositories can reduce API hallucinations without execution, and only then asks when synthetic negatives add further gains. Section 4.2 and Appendices E–G systematically study SFT across row counts, update exposure, hole positions, categories, and languages: showing that 5K third-party API holes (~140 updates) cut verified hallucinations from 29.3% to 12.7% versus 19.2% for random spans on the same files (Table 3), already approach our 100K run whereas 1-epoch training undertrains stopping (Section 4.1, Table 15), and transfer across model families and unseen languages (Figure 3, Table 7, Appendix G). We would like to emphasize the value of this contribution alongside our hard-negative studies, not just as a baseline to beat.

2. Showing when robust SFT matches DPO/ORPO and why naive LLM edits (W4) fall short is a deliberate finding. Post-training work often expects DPO or ORPO (Rafailov et al., 2023; Hong et al., 2024) to beat SFT automatically, yet recent studies show preference optimization often yields small gains over a properly tuned SFT, or degrades outputs via likelihood displacement, unless pairs carry strong signal (Pal et al., 2024; Ivison et al., 2024; Razin et al., 2025; Saeidi et al., 2025). On FIM, when SFT receives curated holes and adequate updates, unfiltered ORPO ties it (59.6% vs. 59.7% EM, Table 2; unanchored DPO collapses to 29.4%, Section 4.3) because straightforwardly prompting frontier teachers to edit ground-truth code (W4) still leaves 60% of raw mistakes fooling zero blind judges (τ=0\tau = 0, Appendix H). Reporting this candidly, without any prior bias against SFT, helps researchers know when curated SFT already suffices and why blind-judge filtering is needed before adding contrastive pairs.

3. Predicting in general when contrastive pair mining beats SFT (Q2). Once filtered by our blind-judge fool rate τ\tau, ORPO beats SFT trained on the same holes on Delulu itself (+1.9 EM for τ≥2/3\tau \ge 2/3 pairs, Table 5; see also Tables 6 & 17) and adds +3.6 points on SAFIM control flow (Table 6). Across all experiments, two general conditions predict when contrastive pairs help:

  • Hard negatives are genuinely plausible (τ≥2/3\tau \ge 2/3): Easy pairs and rule-based edits add +0.0 on Delulu even though rule-based edits are minimal by construction (Tables 4–5), whereas plausible pairs add +1.5 to +1.9 over SFT on the same holes (Table 5).
  • The completion has a tight stopping boundary: Pushing down near-miss endings sharpens stopping on single-call APIs and control-flow conditions (79% of ORPO's fixes over SFT correct boundary overruns, Section 4.3), with a predictable trade-off on open-ended multi-line blocks (Tables 2 & 6).

In short, properly trained SFT on mined API-call holes is the right default recipe, and contrastive training pays off specifically when you filter for τ≥2/3\tau \ge 2/3 negatives and target tight-boundary completions.

We really appreciate reviewers constructive feedback and comments and would love to continue this conversation, so all issues and concerns are resolved. We would like to kindly ask the reviewer to revise their score if they believe that their concerns are resolved and the paper has merit to be published at ICLR.

Thanks, Best, Authors

Additional References

  • Demirer, M., Musolff, L., & Yang, L. (2026). Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools. NBER Working Paper 35275.
  • Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., & Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024.
  • Liang, J. T., Yang, C., & Myers, B. A. (2024). A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. ICSE 2024.
  • Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2024). AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation. Proc. ACM Softw. Eng. (FSE 2024).
  • Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., & White, C. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv:2402.13228.
  • Pavlichenko, N., et al. (2025). Mellum: Production-Grade in-IDE Contextual Code Completion with Multi-File Project Understanding. arXiv:2510.05788.
  • Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025.
  • Saeidi, A., Verma, S., Uddin, M. N., & Baral, C. (2025). Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks. ACL 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.