- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Dev-Primitives, where each repository file is paired with its own LLM that edits that file and sends messages to related files. HERMES adds an activation step that picks which files take part and a diagnosis step that reads execution output and decides what to revise, looping for a few rounds. Experiments cover SWE-bench Verified, SWE Refactor Bench, Terminal-Bench and DevOps-Gym against several existing harnesses, plus ablations and a study that mixes small and large models across components.
Critical strengths
- The evaluation is broad, and the single-editor control in the appendix (same selected files, same diagnosis loop, then a compute-matched variant) is the right test for the main claim. It is the most convincing experiment in the paper.
- The analysis of where to spend a stronger model is useful in practice. Putting it on diagnosis helps more than putting it on activation, which is a concrete takeaway for cost-aware agent design.
Critical weaknesses
- Some results are hard to accept and do not agree with each other. Qwen3-8B with a 32K context reaching 80.6 on SWE-bench Verified on its own is far beyond what I would expect from a model of that size, and beyond a single case study the paper gives no per-task evidence for it. In the scaling table, swapping only the Claude Sonnet 5 primitives for Qwen3-8B gives clearly higher scores than running HERMES with Sonnet 5 everywhere, on both SWE-bench Verified and Terminal-Bench. This goes against the paper’s own claim that larger primitives help. I would need released patches and trajectories, and an explanation of this inversion, before trusting the headline numbers.
- The comparison with MAGIS and CodeTeam in Table 5 is not informative. Both numbers are copied from the original papers under their own backbones and settings, and the MAGIS one comes from a much older model generation, so the gap mostly reflects the model, not the framework. Either rerun them with the same backbone or drop the table.
- The novelty is limited. Per-file agents that message each other along dependencies are close to the file-level developer agents in MAGIS and CodeTeam, and to the agent-native primitive framing of Jin et al. Appendix F.5 says model state is not kept across calls; a primitive is simply called again with the current file and the accumulated messages. So resident LLM and active participant are mostly framing. Activation is localization and diagnosis is a reflection loop over test output, both standard.
- The comparisons are not controlled. Baselines are a mix of leaderboard numbers and the authors’ own runs (Appendix F.8).
Questions for authors
- Can you release per-instance patches and logs for the Qwen3-8B runs on SWE-bench Verified? How are files longer than the 32K window handled?
- Why does replacing only the Sonnet 5 primitives with Qwen3-8B beat HERMES with Sonnet 5 everywhere? Were the effort setting and runs the same?
- How often does diagnosis return PASS when the hidden tests later fail? In the case study it passes without any fail-to-pass reproducer.
4 comments
W2. The comparison with MAGIS and CodeTeam in Table 5 is not informative. Both numbers are copied from the original papers under their own backbones and settings, and the MAGIS one comes from a much older model generation, so the gap mostly reflects the model, not the framework. Either rerun them with the same backbone or drop the table.
R2. Thank you for suggesting these re-runs. We re-ran both frameworks with our backbones and have replaced Table 5 with these re-runs. In the original submission, Table 5 listed each framework's reported score in the setting of its own paper. In the re-runs, MAGIS and CodeTeam (PE) used GPT-5.6 Luna on the two SWE-bench sets and GPT-5.6 Sol on SWE Refactor Bench. On SWE Refactor Bench, we capped their budget at HERMES's cost. On NL2Repo-Bench, they used HERMES's backbone. The table reports the re-runs, where SWE-bench is the split of the original Table 5.
| Method | SWE-bench | SWE-bench Verified | SWE Refactor | NL2Repo-Bench |
|---|---|---|---|---|
| MAGIS | 28.7 | 91.6 | 14.5 | - |
| CodeTeam (PE) | - | 92.8 | 17.5 | 40.2 |
| HERMES | 36.6 | 95.6 | 31.0 | 44.9 |
With GPT-5.6 Luna, MAGIS reached 28.7 on the SWE-bench split, compared with 13.9 in the original Table 5 and 36.6 for HERMES. On SWE-bench Verified, MAGIS and CodeTeam (PE) reached 91.6 and 92.8, compared with 95.6 for HERMES. On SWE Refactor Bench, they reached 14.5 and 17.5, compared with 31.0. On NL2Repo-Bench, CodeTeam (PE) reached 40.2, compared with 34.6 in the original Table 5 and 44.9 for HERMES. CodeTeam (SFT) relies on its own fine-tuned weights, so Table 5 keeps its reported result.
W3. The novelty is limited. Per-file agents that message each other along dependencies are close to the file-level developer agents in MAGIS and CodeTeam, and to the agent-native primitive framing of Jin et al. Appendix F.5 says model state is not kept across calls; a primitive is simply called again with the current file and the accumulated messages. So resident LLM and active participant are mostly framing. Activation is localization and diagnosis is a reflection loop over test output, both standard.
R3. We would like to clarify that the novelty of HERMES lies in binding each reasoning unit to one repository artifact and in routing requirements and failures between these units. Each Dev-Primitive pairs one artifact with a resident LLM, so the repository induces the set of primitives. MAGIS and CodeTeam assign file-level tasks to developer agents and coordinate them through a manager, a shared plan, or a design contract. The designer thus fixes in advance which developers exist and along which paths they talk. The Agent Primitives and Tool Primitives of Jin et al. wrap reusable, task-agnostic behaviors or tools, so they hold no state of a particular file.
This leaves two difficulties for repository tasks. First, requirements arise while files are being edited, and each one may concern a caller, a configuration, or a test. Second, every edit changes the repository, so a central agent has to rebuild and pass on the state of every file. We address these difficulties with three mechanisms. Each primitive addresses its messages to the components its change implicates, so requirements follow dependency edges as they are found. Dynamic activation expands the localized files along their dependencies, including configurations and tests, and starts only these primitives, each with its own objective. Bug diagnosis names the components behind an execution failure, and activation reactivates only their primitives. A single agent needs no such routing, because it edits every file itself.
"Resident" refers to the binding of a primitive to its artifact. Across the revision rounds of a task, a primitive keeps the current version of its artifact and the messages it has received. Its LLM is called again on this state at each activation (Appendix F.5). We have added this description to Sec. 3.2.
In Table 9 of the original submission, the w/o Diagnosis Feedback variant kept execution and let activation read the raw output. Its SWE Refactor Bench score fell from 31.0 to 20.0. We also built a file-agent baseline with one agent per file and a manager that relays messages and reflects on the raw execution output. We then added HERMES's mechanisms one at a time at the same call budget. Replacing the manager's reflection with bug diagnosis was the largest step, from 22.5 to 31.0 on SWE Refactor Bench and from 45.8 to 51.8 on Terminal-Bench 4.0. We report this ladder in Appendix K.5, Table 22. The differences are also stated in Sec. 2.
W4. The comparisons are not controlled. Baselines are a mix of leaderboard numbers and the authors' own runs (Appendix F.8).
R4. Thank you for raising this point. In the original submission, our baselines came from two sources (Appendix F.8). We used a public result when it matched the model, harness, reasoning effort, and benchmark version. Otherwise, we ran the baseline harness ourselves with HERMES's snapshot and protocol, as for every medium-effort baseline on Terminal-Bench 4.0.
To make every headline comparison controlled, we re-ran the baseline of each of the nine headline pairs ourselves. These re-runs used the same task snapshots and containers as HERMES, the benchmark's default wall-clock cap, and API retries on transient errors only. We ran each baseline at its native budget and with restarts up to HERMES's mean cost per task, and we paired the outcomes per task. SWE Refactor Bench scores are means over three paired runs. The table compares the baselines of Tables 1–4 with our re-runs.
| Setting | Harness | Tables 1–4 | Native re-run | Matched re-run | HERMES |
|---|---|---|---|---|---|
| SWE-bench Verified, GPT-5.6 Sol | mini-SWE-agent | 96.2 | 96.0 | 96.4 | 97.0 |
| SWE-bench Verified, Claude Opus 4.8 | Claude Code | 85.8 | 86.2 | 87.6 | 91.0 |
| SWE-bench Verified, GPT-5.5 | Codex | 76.4 | 77.0 | 80.8 | 85.6 |
| SWE Refactor Bench, GPT-5.6 Sol | Codex | 6.5 | 7.0 | 10.5 | 31.0 |
| SWE Refactor Bench, Claude Opus 5 | Claude Code | 28.5 | 27.5 | 29.5 | 36.3 |
| Terminal-Bench 4.0, GPT-5.6 Sol | Codex | 33.0 | 32.4 | 36.4 | 51.8 |
| Terminal-Bench 4.0, Claude Opus 5 | Claude Code | 47.0 | 47.3 | 47.6 | 48.5 |
| DevOps-Gym, GPT-5.6 Sol | Codex | 49.89 | 49.65 | 51.33 | 55.41 |
| DevOps-Gym, Claude Opus 5 | Claude Code | 52.06 | 52.36 | 54.12 | 57.10 |
At the native budget, the re-runs were within 1.0 point of the baselines in Tables 1–4 in all 9 pairs. At the matched budget, HERMES remained ahead in all 9 pairs, and the paired 95% CI excluded zero in 6 of them. We report these re-runs in Appendix K.1, Table 18.
Q3. How often does diagnosis return PASS when the hidden tests later fail? In the case study it passes without any fail-to-pass reproducer.
A3. Thank you for asking about this. We matched every final diagnosis verdict of our default runs with the held-out evaluation tests. The table reports false accepts as a share of PASS verdicts, and the PASS verdicts among the tasks or trials that fail the held-out tests.
| Setting | False accepts | PASS on failing tasks |
|---|---|---|
| SWE-bench Verified, GPT-5.6 Sol | 1.4% | 7 of 15 |
| SWE-bench Verified, Qwen3-8B | 8.9% | 39 of 97 |
| Terminal-Bench 4.0, GPT-5.6 Sol | 13.8% | 27 of 159 trials |
With GPT-5.6 Sol on SWE-bench Verified, 1.4% of PASS verdicts were false accepts. With Qwen3-8B, the case-study backbone, the rate was 8.9%.
Diagnosis cannot return PASS when a task-visible test, the build, or another observed execution fails (Appendix B.4). A false accept can therefore occur only when every retained visible check passes, as in the case study. With GPT-5.6 Sol, PASS without a working reproducer occurred on 44 tasks, and 5 of the 7 false accepts are among them. As an additional check, we tested a policy that requires a working reproducer before PASS. It reduced false accepts from 1.4% to 0.8% of PASS verdicts at 7% more cost. We report this analysis in Appendix K.9, Tables 28 and 29, and have added the policy to Appendix B.4 as an option.
Dear Reviewer Jo51,
Thank you again for your thoughtful review and for highlighting the single-editor control as a direct test of our main claim. We also appreciate your comments on the practical value of using a stronger model for diagnosis.
In our rebuttal, we have carefully addressed your key concerns and provided the per-task evidence you requested.
- (W1/Q1) The anonymous repository now includes the per-task Qwen3-8B patches and trajectories for SWE-bench Verified. HERMES resolved 80.3% of tasks on average over three seeds. The longest input in the audited run was 28.9K tokens, and none was truncated.
- (W1/Q2) At the same reasoning effort, 22.9% of Sonnet 5’s edits fell outside the assigned region, compared with 6.4% for Qwen3-8B. Restricting edits to that region narrowed the SWE-bench Verified gap from 6.6 to 0.8 percentage points.
- (W2–W4) On SWE Refactor Bench with the same backbone and cost cap, MAGIS scored 14.5, CodeTeam (PE) scored 17.5, and HERMES scored 31.0. We also reran all nine headline comparisons at matched budgets and tested the individual contributions of messaging, activation, and diagnosis in Table 22.
- (Q3) On SWE-bench Verified, 8.9% of Qwen3-8B’s PASS verdicts and 1.4% of GPT-5.6 Sol’s PASS verdicts failed the hidden tests. For GPT-5.6 Sol, requiring a working reproducer reduced this rate to 0.8%.
We hope these clarifications and additional results help address your concerns. With two days left in the rebuttal period, we would be happy to provide further details on any points you would like to discuss.
Best regards,
The Authors
Thank you for the thorough review. We are glad that you regard the single-editor control as a direct test of our main claim and as the most informative experiment in the paper, and that you find placing a stronger model on diagnosis a concrete takeaway for cost-aware agent design. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We have carefully considered your concerns and respond to them individually below.
W1/Q1 (1/2). Some results are hard to accept and do not agree with each other. Qwen3-8B with a 32K context reaching 80.6 on SWE-bench Verified on its own is far beyond what I would expect from a model of that size, and beyond a single case study the paper gives no per-task evidence for it. Can you release per-instance patches and logs for the Qwen3-8B runs on SWE-bench Verified? How are files longer than the 32K window handled?
R1/A1 (1/2). Thank you for asking for per-task evidence. Yes, we've updated the anonymous repository of Appendix J with the per-instance patches and trajectories of HERMES with Qwen3-8B on SWE-bench Verified. We also ran three further checks of the 80.6 result. Under mini-SWE-agent with the same deployment, Qwen3-8B resolved 11.8%. Two more HERMES seeds gave 80.3 ± 0.8% over three seeds. On 200 tasks created after the model's release, HERMES resolved 47.5%, compared with 5.0% for mini-SWE-agent.
Each primitive call holds its own artifact with its objective and messages, so the 32K window has to fit one file rather than the repository. The original submission did not describe how an artifact longer than the window would be handled. We therefore audited every Qwen3-8B call of HERMES in the Table 1 run. We compared the token count of each prompt with the number of tokens that Ollama evaluated. The largest input had 28.9K tokens, and 0 calls were truncated. Inputs longer than 24K tokens occurred for 43 of the 2,473 activated primitives. We report these checks and the audit in Appendix K.7, and have added mini-SWE-agent with Qwen3-8B to Tables 1–4.
W1/Q2 (2/2). In the scaling table, swapping only the Claude Sonnet 5 primitives for Qwen3-8B gives clearly higher scores than running HERMES with Sonnet 5 everywhere, on both SWE-bench Verified and Terminal-Bench. This goes against the paper's own claim that larger primitives help. I would need released patches and trajectories, and an explanation of this inversion, before trusting the headline numbers. Why does replacing only the Sonnet 5 primitives with Qwen3-8B beat HERMES with Sonnet 5 everywhere? Were the effort setting and runs the same?
R1/A2 (2/2). Yes, the effort setting and the pipeline were the same. Both configurations used the default medium reasoning effort and B = 3 (Sec. 4.1), and both were separate runs of the same pipeline. We would like to clarify that the benefit of scaling the Dev-Primitives in Sec. 4.3 was measured with GPT-5.6 Sol in the other roles. There it added 1.2 to 4.5 points, for example from 94.2 to 97.0 on SWE-bench Verified (Table 6). Table 6 has no homogeneous Claude Sonnet 5 configuration, so its reference is the Claude Sonnet 5 result of Tables 1–4. This configuration reached 85.8 on SWE-bench Verified and 35.8 on Terminal-Bench 4.0, compared with 92.4 and 48.8 with Qwen3-8B Dev-Primitives (Table 6).
To check that this gap is stable, we ran two more SWE-bench Verified seeds and a new five-trial Terminal-Bench 4.0 run of both configurations. Over three seeds, the configuration with Qwen3-8B Dev-Primitives resolved 92.3%, compared with 85.7% for homogeneous Claude Sonnet 5. On Terminal-Bench 4.0, the new runs resolved 47.9% and 36.4%.
To find the cause, we measured how often primitives edit code outside the region named in their objective. A hunk counts as outside when it does not overlap that region. Claude Sonnet 5 Dev-Primitives placed 22.9% of their hunks outside their objective, compared with 6.4% for Qwen3-8B Dev-Primitives. We then re-ran both configurations once with an edit-scope guard. The guard rejects such hunks and returns them to activation as a message. It is an analysis tool and is not used in Tables 1–4. The table reports both configurations, with Claude Sonnet 5 in the other roles.
With the guard, homogeneous Claude Sonnet 5 rose from 85.8 to 91.8 on SWE-bench Verified and from 35.8 to 46.7 on Terminal-Bench 4.0. The configuration with Qwen3-8B Dev-Primitives stayed at 92.6 and 48.5. These results point mainly to how Claude Sonnet 5 Dev-Primitives follow their objective. We report these runs in Appendix K.8, Tables 26 and 27, and have revised Sec. 4.3 to state that the scaling benefit is measured with GPT-5.6 Sol in the other roles.