acceptodds
Under review as a conference paper at ICLR 2027

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Abstract

Coding agents must integrate external tool returns into ongoing reasoning. We draw a structural analogy between the action → observation → continuation loop of a coding agent and a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This analogy motivates learning from function-level dependencies in ordinary code through function-aware fill-in-the-middle (FIM) mid-training: an infilling objective that masks functions selected via program dependency graph analysis and a complexity–inferability double criterion, with teacher-generated rationale-plus-implementation targets. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.3 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and extends to Qwen3-8B with SWE-Lego. Beyond in-domain gains, mid-training improves final-agent performance on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (τ-bench, BFCL). A single Python-derived corpus thus improves the resulting agents across coding and tool-use tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
midtraining, coding agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VmMROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a code-specific mid-training recipe centered on where to place the mask: they target functions that are complex but still inferable from the surrounding context. It gives roughly +3 on SWE-Bench-Verified across three models and (at 14B) recovers part of the general coding and tool-use ability that agent SFT tends to erode.

Critical strengths

The gain holds across three base models (Qwen2.5-Coder 7B/14B, Qwen3-8B), three post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego), two agent harnesses, and both SWE-Bench splits, using a single corpus and a single set of mid-training hyperparameters.

All baselines are re-evaluated or retrained in the authors’ own environment with three evaluation runs and reported std, and the official numbers are shown side by side

Appendix B lists every constant and threshold, gives pseudocode for both selection algorithms, and walks through a numerical example that reproduces Figure 2 to four decimals; the generation and judging prompts, full hyperparameter tables, and compute accounting (including discarded runs) are all provided.

Critical weaknesses

Weakness 1. The paper’s own ablations do not show that its novel components carry the result. The dependency-based selection criterion adds only 0.58 over letting Gemini pick targets and 0.71 over AST-FIM, both within the reported run-to-run std, and the inverted-mask ablation fails to support the directionality of the function-call-as-agent-step analogy that motivates the whole design. What remains is a data recipe assembled from known pieces (FIM, mid-training, teacher CoT distillation), and the evidence that the specific selection rule matters is weak.

Weakness 2. The gains are modest and the most striking result lacks a control. The main improvement is about +3 on SWE-Bench-Verified at roughly one std over three evaluation runs, with no significance testing, and the 7B mid-trained model (17.8) still falls below the officially reported R2E-Gym baseline (19.0). The +11 LiveCodeBench recovery at 14B has no ablation at all, so the paper cannot distinguish its function-aware selection from the simpler explanation that any CoT code SFT before agentic post-training regularizes against forgetting.

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for recognizing the consistency of the gains across models and post-training pipelines, and for the detailed questions about the training recipe, masking construction, and transfer results. Below, we clarify these contributions and report the added training-budget controls.

Contribution of the training recipe

Our contribution is a function-aware mid-training recipe that constructs reconstruction tasks from program dependencies and provides an initialization for agentic post-training. Its selection rule is explicit, interpretable, and reusable across models. With the teacher and 200K-target budget fixed in Table 3(B), full selection scores 15.61 on average, compared with 13.91 for random selection, 15.03 for Gemini selection, and 14.81 for PDG-only selection: differences of +1.70, +0.58, and +0.80 points. Appendix B specifies the scoring formulas and selection algorithms, making the target-construction procedure explicit and reproducible.

The new non-FIM code continued-pretraining (CPT) control tests the benefit of the complete recipe against additional code training. It uses the same repositories and commits as the full 80%/15%/5% FIM mixture, and permits additional code files beyond FIM's self-contained-file filtering. At a controlled 2.6B-token budget, followed by the same 7B R2E-Gym setup, CPT obtains 15.67±0.98 on Verified and 12.56±1.10 on Lite. The full FIM recipe obtains 17.80±1.40 and 15.00±1.09, improving by +2.13 and +2.44 points. This comparison supports the benefit of the complete function-aware recipe over additional code CPT in the tested setting.

We also added a token-budget-matched no-CoT AST-FIM control at 7B, followed by R2E-Gym. At 1.08B tokens, function-aware FIM obtains 16.07±1.09 on Verified and 12.56±1.03 on Lite, while AST-FIM obtains 15.60±1.27 and 13.33±1.25; the differences are +0.47 and -0.77 points, respectively.

Function-call analogy and masking

The function-call analogy motivates learning from the dependencies around a function: callers specify its use, visible callees expose available operations, and downstream code constrains how its return is consumed. Masking the function body creates a reconstruction task that brings these sources of context together.

Table 3(E) tests the alternative of retaining callee bodies while masking call sites and downstream continuations. It obtains 15.80 on Verified and 13.00 on Lite, compared with 17.00 and 14.22 for single-function body infilling. The +1.20 and +1.22-point improvements support the choice of function-body infilling in the tested recipe. The analogy motivates the design, and final-agent performance measures the value of this training intervention.

Consistency across training settings

All reported means and population standard deviations are computed over three evaluations of each fixed final checkpoint. This protocol also applies to the added CPT and matched AST-FIM controls.

Table 1 provides four comparisons on Verified. Qwen2.5-Coder-7B with R2E-Gym improves from 15.00±1.50 to 17.80±1.40 (+2.80); the same base model with SWE-Smith improves from 12.33±1.20 to 17.60±1.30 (+5.27); Qwen2.5-Coder-14B with R2E-Gym improves from 26.20±1.40 to 29.20±1.50 (+3.00); and Qwen3-8B with SWE-Lego improves from 31.80±1.02 to 35.00±1.50 (+3.20). Each comparison uses the same named post-training pipeline and our common evaluation setup.

The measured improvements have the same direction across all four configurations, covering two model generations, two sizes in the Qwen2.5-Coder series, and three post-training pipelines. This breadth of within-setting results supports the usefulness of inserting the mid-training stage across these agent-training configurations. Section 3.1 specifies our evaluation protocol.

Comparison under a common evaluation protocol

Table 1 includes the officially reported R2E-Gym 7B result of 19.00±1.00, alongside our evaluation of its released checkpoint. As specified in Section 3.1, the released checkpoint evaluated with the official scripts in our environment obtains 15.00±1.50 on Verified. Under our evaluation setup, FIM mid-training followed by R2E-Gym obtains 17.80±1.40, a +2.80-point improvement over the same-environment baseline. Reporting the published reference and the common-environment comparison side by side gives both the original reference score and the comparison used to measure the training intervention.

Transfer beyond repository repair

Table 2 studies the benefit of the complete 14B training pipeline across benchmark families. On LiveCodeBench, R2E-Gym post-training alone scores 24.10 and FIM mid-training followed by the reported R2E-Gym post-training scores 35.20, an improvement of +11.10 points; the Instruct reference scores 37.20.

All five other benchmarks in Table 2 also improve over post-training alone: +1.94 on OJBench, +0.53 on FullStackBench-EN, +1.25 on Terminal-Bench, +3.90 on τ-bench, and +2.40 on BFCL. The six-benchmark average rises from 16.04 to 19.56 (+3.52). These results show that a single Python-derived mid-training corpus improves the resulting agents across repository repair, other coding tasks, and non-coding tool use.

We evaluate function-aware mid-training as a transferable initialization within a completed agent-training pipeline. The new 2.6B-token CPT control further supports the benefit of this complete recipe: on 7B followed by R2E-Gym, it improves over additional code CPT by +2.13 on Verified and +2.44 on Lite. Together, the controlled training comparison and the broader-capability results provide evidence for the practical value of this initialization across the tested settings. Section 3.3 and Table 2 report the broader-capability evaluations.

Dear Reviewer VmMR,

As the discussion period is ending soon, we wanted to follow up on our response. We added a budget-matched continued-pretraining (CPT) control: our recipe beats it by +2.13 / +2.44 on Verified / Lite. We also added a token-matched AST-FIM comparison, explained the gap to the official R2E-Gym number, which comes from the evaluation environment, and pointed to the consistent gains across all four model–pipeline settings. We believe these address your concerns about how much the selection recipe contributes and about the “more training” explanation. If you have any remaining questions, we would be glad to answer them. If our response has resolved your concerns, we would appreciate it if you could consider updating your score.

Thank you again for your time and feedback.

Reviewer_9Rf5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Fill in the middle training involves inferring the logic (body) of a function given the functional signature and its invocation in code around it (eg other functions before and after it).

This paper champions for Fill-in-the-middle training as a Mid-training step for training coding and agentic models. They find that doing FIM mid training lifts post-training gains more, compared to without.

they provide a mechanism to generate such mid training data.

They also specify the training setup, SFT - chat completion.

Critical strengths
  1. The data generation pipeline is quite nice. It is deterministic and builds on actual logical dependencies between code. It's also scalable and verified by design.

  2. Fill-in-the-middle mid training increasing semantic understanding of the model for:

  • functional signatures
  • logical dependencies between functions

Is quite nice. it also models well how large repos are worked upon with coding agents, where quite often existing abstractions have to be utilized and only the business logic is updated.

Critical weaknesses
  1. Weak Baseline: I feel the obvious baseline of training on the same github repos (or at least same sized corpus) with non-FIM style templating is missing and is important to highlight why 'FIM' is the right choice of design for the mid-training.
  2. No interpretation and analysis of the gains. An analysis of how the model's behavior changes immediately after the FIM mid-training and during post training is not done. It would be good to see if the model now actually understand logical dependencies and functional signatures better. Both qualitative (examples) and quantitative analysis of this is required. Just downstream gains are not a signal for this, as they can benefit from just more training overall.
Questions for authors

Lines 157–161: The inferability weights in Appendix B.2.2 (0.30/0.25/0.20/0.15) and the internal constants appear hand-set. How were they chosen, and is there evidence the score actually predicts how recoverable a function is from context?

Lines 221–223: Why the difference in post-training techniques? Lines 231–232 explain why Qwen2.5 can’t use SWE-Lego (32K context), but not why Qwen3-8B isn’t also run with R2E-Gym, which would allow a cleaner comparison across models.

Lines 228–229: Please also add results after FIM mid-training, before post-training, if possible, to gauge if intermediate performance drops or improves. Both would be useful signals for interpreting what is happening with FIM in the latent space.

Lines 244–247: Are the gains an artifact of just “more training” on new data? What is FIM actually contributing here? Table 3 partly address this, but both are still FIM. A good baseline would be standard SFT / next-token training on a similar token-size corpus (maybe even the same GitHub curated corpus) and then measuring post-training gains. This will help separate out “more training” from FIM templating gains.

4 comments

Thank you for the constructive comments and for recognizing the dependency-based data construction. We have added the non-FIM training control and clarify the behavioral evidence and evaluation design below.

Non-FIM training control

We have added a non-FIM continued-pretraining (CPT) control using the same repository list and commit snapshots as our full FIM recipe, with a 2.6B-token budget and controlled training compute, followed by R2E-Gym post-training on the same 7B starting model. CPT additionally includes code files excluded by FIM's self-contained-file eligibility filter, giving this control broader file coverage within the same source repositories.

CPT followed by R2E-Gym obtains 15.67±0.98 on SWE-Bench-Verified and 12.56±1.10 on SWE-Bench-Lite. Our full 80%/15%/5% FIM mixture followed by R2E-Gym obtains 17.80±1.40 and 15.00±1.09, respectively, gains of 2.13 and 2.44 percentage points over CPT. These are the full-corpus FIM results from Table 1. Each mean and population standard deviation comes from three evaluations of a fixed final checkpoint, without retraining between these evaluations. This new control confirms the benefit of the function-aware FIM recipe beyond additional code training at a comparable budget.

Behavioral evidence

Section 4 and Appendix D provide both quantitative behavioral analysis and a qualitative repair example for the final 14B R2E-Gym agents. On Verified, recovery after detected negative tool observations increases from 24.8% to 28.8%, edits per solved task from 3.3 to 7.4, and steps per solved task from 15.1 to 23.6 (Table 4). The action distribution also shifts toward execution and verification: the share of execute_bash actions rises from 19.5% to 24.6% (Appendix D.2.2).

The gains also align with the function-level construction. On Verified's 88 multi-function, single-file tasks, success rises from 13.6% to 22.7% (Section 4.2). Appendix D.2.4 gives a concrete example on scikit-learn-26323: the baseline terminates with an empty patch, while our agent continues through negative tool returns and produces a passing patch. We claim that function-aware mid-training provides a more effective initialization for downstream agent learning. We support this claim with the improved task scores, the sustained edit-and-verify behavior documented above, and the new same-source, budget-matched CPT control.

Inferability-based selection

Inferability is a structural heuristic for selecting targets whose behavior is supported by the surrounding code. Appendix B.2.2 combines four cues: caller specificity, in-file callees, signature information, and class context. The formula is

I_hat = 0.30 C_caller + 0.25 C_callee + 0.20 C_sig + 0.15 C_class.

These coefficients specify the relative contributions directly, without renormalization; Table 6 gives the internal constants and Appendix B provides the component formulas. Combining this score with complexity selects substantive implementations with supporting contextual cues.

Table 3(B) evaluates the practical utility of these cues with the teacher and 200K-target budget fixed. PDG-only selection obtains a two-benchmark average of 14.81; adding inferability raises it to 15.33, and the full complexity–inferability rule reaches 15.61. This progression supports the role of contextual information in target construction. The score is used as an explicit, reusable ranking rule, and its effectiveness is assessed through the resulting downstream models.

Post-training pipeline design

The comparison unit is a fixed model–pipeline pairing: we estimate the gain from mid-training within that pairing and evaluate both variants in our common environment. Table 1 tests Qwen2.5-Coder-7B with both R2E-Gym and SWE-Smith, Qwen2.5-Coder-14B with R2E-Gym, and Qwen3-8B with SWE-Lego. For Qwen3-8B, Verified improves from 31.80 to 35.00 and Lite from 27.33 to 32.67; these are comparisons within Qwen3/SWE-Lego.

This design checks robustness along two axes: the same 7B base benefits under two post-training pipelines, and the same mid-training corpus also benefits the Qwen3-8B/SWE-Lego setting. Section 3.1 explains the 32K constraint on using long SWE-Lego trajectories with Qwen2.5. Our within-pair comparisons measure the benefit of inserting mid-training across the evaluated model–pipeline settings.

Role of the intermediate checkpoint

The goal of mid-training is to provide an initialization that transfers through subsequent agent post-training. The latter stage adapts the model to agent actions, tool interfaces, and trajectory formats. Accordingly, the primary evaluation in Section 3.1 compares the completed pipelines: post-training alone versus FIM mid-training followed by the reported agent post-training. This directly measures the value of the initialization for its intended downstream use.

Intermediate-checkpoint evaluation is possible and can offer another view of training-stage behavior. For an agent benchmark such as SWE-Bench, evaluating before agent adaptation also measures readiness for interfaces and formats addressed by the subsequent stage. The downstream-transfer question is therefore best answered after that adaptation. Tables 1–2 and the new CPT control show the benefit at this intended endpoint, and Section 4 and Appendix D analyze the resulting agents' repair behavior.

My concerns on continual pre-training have been resolved.

I am updating my score.

Thank you very much for your positive feedback and constructive comments! We are happy to hear that our rebuttal addressed your questions well.

Reviewer_9VFNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces function-aware fill-in-the-middle mid-training for coding agents. It selects individual functions and connected function groups using code dependencies, complexity, and contextual inferability, then trains models on rationales and implementations before post-training. Experiments cover three Qwen models and three post-training pipelines with gains on SWE-Bench and broader coding and tool-use benchmarks.

Critical strengths
  1. The SWE-Bench gains appear across all four settings, including both Qwen generations.
  2. The no-CoT comparison keeps the inputs and implementation targets fixed, which makes the rationale comparison easy to interpret.
Critical weaknesses
  1. The gains from complexity scoring and multi-function masking are small, and token or compute matching with AST-FIM is unclear. This leaves uncertainty about how much the proposed design adds beyond a simpler infilling recipe..
  2. All experiments use Qwen2.5-Coder or the original Qwen3 checkpoints, leaving the benefits on more recent coding models untested. Evaluating a newer model would help establish whether the gains persist with a stronger starting point..
Questions for authors
  1. Are the small gains from complexity scoring and multi-function masking consistent across runs? Please report paired uncertainty for these comparisons and clarify whether the repeats involve retraining or only evaluation..
  2. Do AST-FIM and your method use comparable numbers of supervised tokens and similar training compute? If the budgets differ, could you provide a matched comparison to assess the benefit of the proposed design?.
  3. Does the method also improve a more recent coding model under the same agent post-training setup?.
2 comments

Thank you for recognizing the gains across the four experimental settings and our controlled CoT comparison, and for the questions on design contributions, training budgets, and model coverage.

Evaluation protocol and design choices

Each trained checkpoint is evaluated three times, including all ablation configurations and the new controls. These repeats involve evaluation only, with no retraining between runs. We report the mean and population standard deviation across the three evaluation runs.

The design comparisons in Table 3 show the contribution of the selection and masking choices under its fixed CoT setting and 200K-target budget. Adding complexity to PDG+inferability increases the two-benchmark average from 15.33 to 15.61 (+0.28 points). Replacing single-function masking with the 80%/15%/5% mixture increases the average from 15.61 to 16.09 (+0.48 points). The selector prioritizes substantive functions supported by their context, while joint masking constructs reconstruction tasks involving interface and shared-state consistency (Sections 2.3.4–2.3.5). These design choices refine the complete function-aware mid-training recipe.

We also added a 2.6B-token continued-pretraining (CPT) control using the same source repositories and commits as the full 80%/15%/5% FIM corpus. CPT permits additional files beyond the self-contained code selected for FIM. With the same 7B/R2E-Gym pipeline, CPT obtains 15.67 (±0.98) on Verified and 12.56 (±1.10) on Lite. The full-corpus FIM model in Table 1 obtains 17.80 (±1.40) and 15.00 (±1.09), respectively. Thus, the complete recipe improves over same-source, token-budget-matched CPT by +2.13 and +2.44 percentage points after agentic post-training. This new comparison supports the value of the complete recipe beyond additional next-token code training.

Token-matched AST-FIM and CoT supervision

Target selection and CoT supervision are integral parts of our mid-training recipe. The selector defines challenging, context-supported reconstruction tasks, while the rationale provides a coherent, step-by-step reasoning process before the implementation. Good masked-function targets and explicit reasoning supervision work together in this design.

We added an AST-FIM comparison at the same 1.08B-token budget as the no-CoT function-aware FIM variant, using the 7B model followed by R2E-Gym post-training. Both variants omit rationale supervision. Under this matched budget, function-aware FIM reaches 16.07 on Verified versus 15.60 for AST-FIM, a +0.47-point improvement. The table reports means and population standard deviations from three evaluations of each fixed checkpoint.

Method (1.08B-token budget)VerifiedLite
Function-aware FIM, no CoT16.07 (±1.09)12.56 (±1.03)
AST-FIM, token-matched15.60 (±1.27)13.33 (±1.25)

Table 3(A) directly examines the contribution of the reasoning supervision. With the masked inputs and implementation targets held fixed, adding Gemini-3-Flash CoT raises Verified from 16.07 to 17.00 (+0.93 points) and Lite from 12.56 to 14.22 (+1.66 points). The two-benchmark average increases by 1.30 points. This comparison highlights the value of pairing carefully chosen masked functions with an explicit reasoning process. With mixed masking granularity, the 200K-target recipe reaches an average of 16.09 in Table 3(C).

Model and pipeline coverage

We evaluate a training intervention: inserting FIM mid-training before agentic post-training for a given starting checkpoint. Table 1 studies this effect across three checkpoints and four settings: Qwen2.5-Coder-7B-Instruct with R2E-Gym and SWE-Smith, Qwen2.5-Coder-14B-Instruct with R2E-Gym, and Qwen3-8B with SWE-Lego. The Verified gains are +2.80, +5.27, +3.00, and +3.20 points, respectively; the Lite gains are +3.67, +0.44, +4.00, and +5.33 points.

Within each setting, the baseline and FIM model share the same named post-training pipeline and our common evaluation environment. We measure the value of the inserted stage across model generations, model sizes, and training pipelines. Our claim concerns the downstream value of function-aware mid-training across these settings, measured through within-setting comparisons after agentic post-training.

Dear Reviewer 9VFN,

As the discussion period is ending soon, we wanted to follow up on our response. We added a token-matched AST-FIM comparison, a budget-matched continued-pretraining (CPT) control (+2.13 / +2.44 over CPT on Verified / Lite), and explained how the repeated runs were done. We believe these address your concerns about the design’s contribution and about compute matching. If you have any remaining questions, we would be glad to answer them. If our response has resolved your concerns, we would appreciate it if you could consider updating your score.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.