- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes two program-state reasoning tasks for training code LLMs that act as software-engineering (SWE) agents for bug fixing and regression-test generation: (a) Buggy I/O Reasoning. Given a buggy program, the model must infer unit tests that expose the bug, without access to execution tools. (b) Pre/Post-Condition Reasoning. At the repository level, the model must write a regression test for a bug fix. It has shell access to explore the repository. It must also produce a separate reasoning file that records the symbolic conditions under which the bug is triggered and what the bug causes. These two tasks are added to the standard patch-generation task in an SFT + RL training pipeline. The resulting model, Comet-9B, is competitive with other agentic baselines on bug fixing and test generation in SWE-Bench Pro, SWT-Bench Verified, and CyberGym.
Critical strengths
-
Intuitive task formulation. Inferring bug-exposing inputs and the pre/post-conditions of a fix are intuitive ways to make a model reason about runtime program state, a skill that patch-only training does not target directly.
-
Clear data collection and training setup. The pipeline for building task data and the SFT + RL setup are well described, which supports reproducibility.
-
Clear gains over the base model. Adding the two tasks produces a model that is noticeably better at bug patching and regression-test generation than the base model.
Critical weaknesses
W1. Missing empirical comparison with models trained on SWE-Gym (Pan et al., 2025), R2E-Gym (Jain et al., 2025), SWE-smith (Yang et al., 2025) provided executable repository environments and trajectory data for agent SFT and RL. This will isolate the impact of adding training tasks for reasoning about runtime program states which existing works don't provide.
W2. Missing cost–performance comparison with the strongest baselines. The best agentic baselines clearly outperform Comet-9B: Gemini-3 Flash by 4.12 points on SWE-Bench Pro, and GPT-5-mini by 16.9 points on SWT-Bench Verified. The 16.9-point gap in particular weakens the case for Comet-9B as a standalone contribution. Please report the inference cost of running the agent with each model, for example API dollars per instance, tokens, number of agent turns, and wall-clock time. Please compare these with Comet-9B’s self-hosted serving cost. Can you justify the accuracy loss with the advantages offered by Comet-9B?
W3. Bug detection is not addressed. Both tasks assume that a bug is already known to exist. In practice, an agent must first decide whether a bug exists in a program or codebase, and where, which requires scanning the code and reasoning about program states without executing it. The paper does not address this setting. It should either evaluate it, for instance as bug detection or localization, or state it explicitly as a limitation.
W4. Possible bias toward finding bugs that are not there. Do the two tasks bias the model toward reporting bugs even when none exist? Can you compute Comet-9B’s false-positive rate. One test: give the model a bug report paired with the already fixed program and measure how often it still generates a “bug-exposing” test or a patch. A model that always finds a bug would be of limited use in practice.
W5. Size of fixes and depth of repository/codebase exploration. Please break down results by the size of the gold patch, e.g., lines changed, number of chunks, and number of files touched. Is Comet-9B mainly fixing small, local bugs, or can it handle multi-location fixes? It is also unclear how thoroughly the agent explores the repository while reasoning about a bug. Please report exploration statistics, such as files opened, distinct functions inspected, and number of shell commands, with and without the two auxiliary tasks in training. This would show whether the tasks actually change how the agent navigates code.
W6. Missing per-task ablation. The paper does not clearly isolate the contribution of each task. The RL variants in Tables 1 and 2 have all seen both tasks during SFT, so the effect of each one cannot be separated. Please compare Comet-9B against models trained with SFT + RL on (i) patch generation plus only Buggy I/O Reasoning, and (ii) patch generation plus only Pre/Post-Condition Reasoning. This would show whether both tasks are necessary.
Questions for authors
See Weaknesses section.
3 comments
W5: Fix complexity and repository exploration.
Thank you for suggesting this analysis of fix complexity and repository exploration. We compare Pure SWE SFT, Mixed SFT, and Comet-9B. Pure SWE SFT provides an issue-resolution baseline with the agentic capabilities needed to use the evaluation scaffold, which the earlier mathematics/code RL checkpoint has not yet acquired. Mixed SFT adds both program-state reasoning tasks during SFT; Comet-9B further applies program-state reasoning RL.
Performance across fix complexity. We group all 731 SWE-bench Pro tasks by their official gold patches. Changed lines count additions plus deletions, hunks are unified-diff hunks, and files are distinct modified paths.
| Gold-patch metric | Group | Tasks | Pure SWE SFT | Mixed SFT | Comet-9B |
|---|---|---|---|---|---|
| Changed lines | 11–25 | 12 | 0.0% | 25.0% | 41.7% |
| Changed lines | 26–50 | 171 | 26.3% | 38.6% | 49.1% |
| Changed lines | 51–100 | 199 | 21.1% | 29.6% | 36.2% |
| Changed lines | >100 | 349 | 8.6% | 12.0% | 17.8% |
| Hunks | 0–1 | 19 | 5.3% | 10.5% | 21.1% |
| Hunks | 2 | 38 | 23.7% | 42.1% | 39.5% |
| Hunks | 3–5 | 129 | 17.8% | 28.7% | 41.1% |
| Hunks | 6+ | 545 | 15.4% | 21.1% | 27.7% |
| Files | 1 | 105 | 17.1% | 29.5% | 41.0% |
| Files | 2 | 126 | 18.3% | 29.4% | 36.5% |
| Files | 3+ | 500 | 15.2% | 20.4% | 26.8% |
Overall success is 16.01%, 23.26%, and 30.51%, respectively. Tasks involving at least three files contribute 58 of the 106 net additional successes over Pure SWE SFT and account for 134/223 (60.1%) of Comet's successes. On patches exceeding 100 changed lines, Comet resolves 62 tasks versus 30 for Pure SWE SFT. The gains therefore extend to larger, multi-location fixes.
Repository exploration and patch revision. We analyze shell commands on matched tasks. For early exploration, we use a fixed window of the first 50 shell calls and measure the proportion of successful explicit source-reading calls that access at least one file modified by the gold patch. We compute this proportion per task and then average across tasks. All other statistics below use the complete trajectories.
| Exploration and revision metric | Pure SWE SFT | Mixed SFT | Comet-9B |
|---|---|---|---|
| Shell calls per task | 204.1 | 205.8 | 171.9 |
| Distinct source files explicitly read per task | 12.1 | 10.8 | 8.4 |
| Share of source-reading calls involving gold-modified files, first 50 shell calls | 56.5% | 59.3% | 69.0% |
Inspections of current changes (git diff) per task | 17.7 | 7.9 | 4.1 |
| Source re-edits after checking, per task | 8.3 | 6.5 | 3.3 |
| Source rollbacks after editing, per task | 3.3 | 2.8 | 1.7 |
A source re-edit after checking is a later edit to the same source file following an earlier edit and an intervening diff inspection, test, or static check. Each later editing call is counted at most once. Source rollbacks count calls that restore previously edited source files using git checkout, git restore, or git reset --hard. We identify these events from explicit source-writing and restoration commands, excluding test files. Source-reading statistics count successful calls that explicitly display source files, including repeated reads. Diff inspections include summaries and exclude patch exports and git diff --check.
Adding the auxiliary tasks during SFT reduces source re-edits after checking from 8.3 to 6.5 per task, and subsequent program-state reasoning RL further reduces this to 3.3. Source rollbacks also decrease from 3.3 to 2.8 and 1.7. Comet's early reading is more concentrated on gold-modified files, reaching 69.0% compared with 56.5% for Pure SWE SFT and 59.3% for Mixed SFT, while it reads fewer distinct source files overall.
On tasks successfully solved by all three models, source re-edits after checking also decrease from 3.0 to 1.9 and 1.1 per task. Across the matched tasks, normalizing by trajectory length gives 3.37, 2.55, and 1.74 such re-edits per 100 shell calls, respectively. These observations are consistent with more focused early repository exploration and fewer back-and-forth patch revisions.
W6: Contribution of the individual training tasks.
Training separate SFT-plus-RL pipelines for every task combination would require substantial computational resources, particularly for agentic RL involving repository interaction and execution. We therefore use incremental evaluation: retain the best overall configuration selected in preceding stages and measure the additional benefit of subsequent components. Stage-by-stage comparisons are also used in Nemotron-Cascade [7] and Qwen3 [8].
Including the same task types in SFT and subsequent RL is common practice in recent model training. DeepSeek-V4 trains domain specialists through domain-specific SFT followed by GRPO [1]. GLM-5, Kimi K3, Qwen3-Coder-Next, and Nemotron 3 Super similarly include reasoning or agentic tasks during SFT before further optimizing those capabilities through RL [2–5]. Nemotron 3.5 Super additionally branches its agentic and reasoning teachers from the same intermediate RLVR checkpoint [6]. These designs use preceding training to establish a capable initial policy, then assess further improvements through specialized optimization.
We follow this practice by initializing both RL branches from Mixed SFT. This fixes the preceding training while measuring the additional benefit of each RL task. Buggy-I/O RL and Pre–Post RL improve SWE-bench Pro performance from 23.26% to 26.54% and 26.81%, respectively, and SWT-Bench Verified performance from 18.71% to 24.71% and 44.80%. Sequentially combining them further improves patch-generation performance to 30.51%. Our existing comparisons therefore already support our main claim: adding explicit program-state reasoning training improves downstream software-engineering capabilities.
[1] DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026.
[2] GLM-5: from Vibe Coding to Agentic Engineering. 2026.
[3] Kimi K3: Open Frontier Intelligence. 2026.
[4] Qwen3-Coder-Next Technical Report. 2026.
[5] Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. 2026.
[6] Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI. 2026.
[7] Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models. 2025.
[8] Qwen3 Technical Report. 2025.
Thank you for your detailed review and constructive suggestions. We appreciate your recognition of the task formulations, the reproducibility of the training setup, and the gains over the base model. We address each of your concerns below.
W1: Comparison with SWE-Gym, R2E-Gym, and SWE-smith.
We report additional comparisons on SWE-bench Verified using Pass@1 only, excluding results that use multiple candidate patches and verifier-based selection.
Comet-9B resolves 290 of 500 tasks, exceeding the strongest of these published Pass@1 baselines by 17.8 percentage points while using fewer parameters.
[1] Training Software Engineering Agents and Verifiers with SWE-Gym.
[2] R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents.
[3] SWE-smith: Scaling Data for Software Engineering Agents.
W2: Cost and performance.
We agree that cost–performance comparisons are valuable for deployment decisions. However, our main contribution is the training method: designing explicit program-state reasoning tasks that improve downstream software-engineering capabilities. Comet-9B serves as an empirical validation of this method; we do not claim that it offers a superior deployment cost–accuracy trade-off over commercial models.
Our central comparisons therefore use the same model family and agent settings to study the benefits of program-state reasoning training. A comprehensive deployment comparison would require additional measurements under specified serving conditions. Although API expenditure can be measured, self-hosted cost depends on hardware, utilization, batching, and workload, making a meaningful comparison more involved than comparing API prices with model size. Such a deployment study is beyond the scope of this work. The reported commercial baselines provide performance context, rather than support a claim that Comet-9B should replace the strongest available agents.
W3 and W4: Issue-resolution setting and bug existence.
We address W3 and W4 together because both concern whether the agent must first determine that a reported bug actually exists.
Our main claim is that adding explicit program-state reasoning tasks improves downstream software-engineering capabilities. We study issue resolution under the established SWE-bench setting [1], where a user reports an issue and asks the agent to resolve it. The same setting is used to construct training tasks and environments in SWE-Gym [2], R2E-Gym [3], and SWE-smith [4], and to train models in SWE-RL [5], Co-PatcheR [6], and SWE-Fixer [7]. These works, accepted at top-tier venues, share a focus on resolving reported issues. We follow this established scope. Asking an agent to independently determine whether and where bugs exist is an important but distinct research problem, which falls outside the scope of our study.
To examine whether the benefits extend beyond issue resolution, we also study regression-test generation and generalization to vulnerability reproduction without security-specific post-training. Together, these three downstream tasks provide broad evidence that our program-state reasoning tasks improve software-engineering capabilities across different applications.
[1] SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
[2] Training Software Engineering Agents and Verifiers with SWE-Gym. ICML 2025.
[3] R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. COLM 2025.
[4] SWE-smith: Scaling Data for Software Engineering Agents. NeurIPS 2025, Datasets and Benchmarks Track.
[5] SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. NeurIPS 2025.
[6] Co-PatcheR: Collaborative Software Patching with Component-specific Small Reasoning Models. NeurIPS 2025.
[7] SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution. Findings of ACL 2025.