- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a code-specific mid-training recipe centered on where to place the mask: they target functions that are complex but still inferable from the surrounding context. It gives roughly +3 on SWE-Bench-Verified across three models and (at 14B) recovers part of the general coding and tool-use ability that agent SFT tends to erode.
Critical strengths
The gain holds across three base models (Qwen2.5-Coder 7B/14B, Qwen3-8B), three post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego), two agent harnesses, and both SWE-Bench splits, using a single corpus and a single set of mid-training hyperparameters.
All baselines are re-evaluated or retrained in the authors’ own environment with three evaluation runs and reported std, and the official numbers are shown side by side
Appendix B lists every constant and threshold, gives pseudocode for both selection algorithms, and walks through a numerical example that reproduces Figure 2 to four decimals; the generation and judging prompts, full hyperparameter tables, and compute accounting (including discarded runs) are all provided.
Critical weaknesses
Weakness 1. The paper’s own ablations do not show that its novel components carry the result. The dependency-based selection criterion adds only 0.58 over letting Gemini pick targets and 0.71 over AST-FIM, both within the reported run-to-run std, and the inverted-mask ablation fails to support the directionality of the function-call-as-agent-step analogy that motivates the whole design. What remains is a data recipe assembled from known pieces (FIM, mid-training, teacher CoT distillation), and the evidence that the specific selection rule matters is weak.
Weakness 2. The gains are modest and the most striking result lacks a control. The main improvement is about +3 on SWE-Bench-Verified at roughly one std over three evaluation runs, with no significance testing, and the 7B mid-trained model (17.8) still falls below the officially reported R2E-Gym baseline (19.0). The +11 LiveCodeBench recovery at 14B has no ablation at all, so the paper cannot distinguish its function-aware selection from the simpler explanation that any CoT code SFT before agentic post-training regularizes against forgetting.
Questions for authors
See Weaknesses.
3 comments
Comparison under a common evaluation protocol
Table 1 includes the officially reported R2E-Gym 7B result of 19.00±1.00, alongside our evaluation of its released checkpoint. As specified in Section 3.1, the released checkpoint evaluated with the official scripts in our environment obtains 15.00±1.50 on Verified. Under our evaluation setup, FIM mid-training followed by R2E-Gym obtains 17.80±1.40, a +2.80-point improvement over the same-environment baseline. Reporting the published reference and the common-environment comparison side by side gives both the original reference score and the comparison used to measure the training intervention.
Transfer beyond repository repair
Table 2 studies the benefit of the complete 14B training pipeline across benchmark families. On LiveCodeBench, R2E-Gym post-training alone scores 24.10 and FIM mid-training followed by the reported R2E-Gym post-training scores 35.20, an improvement of +11.10 points; the Instruct reference scores 37.20.
All five other benchmarks in Table 2 also improve over post-training alone: +1.94 on OJBench, +0.53 on FullStackBench-EN, +1.25 on Terminal-Bench, +3.90 on τ-bench, and +2.40 on BFCL. The six-benchmark average rises from 16.04 to 19.56 (+3.52). These results show that a single Python-derived mid-training corpus improves the resulting agents across repository repair, other coding tasks, and non-coding tool use.
We evaluate function-aware mid-training as a transferable initialization within a completed agent-training pipeline. The new 2.6B-token CPT control further supports the benefit of this complete recipe: on 7B followed by R2E-Gym, it improves over additional code CPT by +2.13 on Verified and +2.44 on Lite. Together, the controlled training comparison and the broader-capability results provide evidence for the practical value of this initialization across the tested settings. Section 3.3 and Table 2 report the broader-capability evaluations.
Dear Reviewer VmMR,
As the discussion period is ending soon, we wanted to follow up on our response. We added a budget-matched continued-pretraining (CPT) control: our recipe beats it by +2.13 / +2.44 on Verified / Lite. We also added a token-matched AST-FIM comparison, explained the gap to the official R2E-Gym number, which comes from the evaluation environment, and pointed to the consistent gains across all four model–pipeline settings. We believe these address your concerns about how much the selection recipe contributes and about the “more training” explanation. If you have any remaining questions, we would be glad to answer them. If our response has resolved your concerns, we would appreciate it if you could consider updating your score.
Thank you again for your time and feedback.
We thank the reviewer for recognizing the consistency of the gains across models and post-training pipelines, and for the detailed questions about the training recipe, masking construction, and transfer results. Below, we clarify these contributions and report the added training-budget controls.
Contribution of the training recipe
Our contribution is a function-aware mid-training recipe that constructs reconstruction tasks from program dependencies and provides an initialization for agentic post-training. Its selection rule is explicit, interpretable, and reusable across models. With the teacher and 200K-target budget fixed in Table 3(B), full selection scores 15.61 on average, compared with 13.91 for random selection, 15.03 for Gemini selection, and 14.81 for PDG-only selection: differences of +1.70, +0.58, and +0.80 points. Appendix B specifies the scoring formulas and selection algorithms, making the target-construction procedure explicit and reproducible.
The new non-FIM code continued-pretraining (CPT) control tests the benefit of the complete recipe against additional code training. It uses the same repositories and commits as the full 80%/15%/5% FIM mixture, and permits additional code files beyond FIM's self-contained-file filtering. At a controlled 2.6B-token budget, followed by the same 7B R2E-Gym setup, CPT obtains 15.67±0.98 on Verified and 12.56±1.10 on Lite. The full FIM recipe obtains 17.80±1.40 and 15.00±1.09, improving by +2.13 and +2.44 points. This comparison supports the benefit of the complete function-aware recipe over additional code CPT in the tested setting.
We also added a token-budget-matched no-CoT AST-FIM control at 7B, followed by R2E-Gym. At 1.08B tokens, function-aware FIM obtains 16.07±1.09 on Verified and 12.56±1.03 on Lite, while AST-FIM obtains 15.60±1.27 and 13.33±1.25; the differences are +0.47 and -0.77 points, respectively.
Function-call analogy and masking
The function-call analogy motivates learning from the dependencies around a function: callers specify its use, visible callees expose available operations, and downstream code constrains how its return is consumed. Masking the function body creates a reconstruction task that brings these sources of context together.
Table 3(E) tests the alternative of retaining callee bodies while masking call sites and downstream continuations. It obtains 15.80 on Verified and 13.00 on Lite, compared with 17.00 and 14.22 for single-function body infilling. The +1.20 and +1.22-point improvements support the choice of function-body infilling in the tested recipe. The analogy motivates the design, and final-agent performance measures the value of this training intervention.
Consistency across training settings
All reported means and population standard deviations are computed over three evaluations of each fixed final checkpoint. This protocol also applies to the added CPT and matched AST-FIM controls.
Table 1 provides four comparisons on Verified. Qwen2.5-Coder-7B with R2E-Gym improves from 15.00±1.50 to 17.80±1.40 (+2.80); the same base model with SWE-Smith improves from 12.33±1.20 to 17.60±1.30 (+5.27); Qwen2.5-Coder-14B with R2E-Gym improves from 26.20±1.40 to 29.20±1.50 (+3.00); and Qwen3-8B with SWE-Lego improves from 31.80±1.02 to 35.00±1.50 (+3.20). Each comparison uses the same named post-training pipeline and our common evaluation setup.
The measured improvements have the same direction across all four configurations, covering two model generations, two sizes in the Qwen2.5-Coder series, and three post-training pipelines. This breadth of within-setting results supports the usefulness of inserting the mid-training stage across these agent-training configurations. Section 3.1 specifies our evaluation protocol.