acceptodds
Under review as a conference paper at ICLR 2027

Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

Abstract

Large language models (LLMs) equipped with terminal access have demonstrated strong capabilities in automating software engineering tasks. However, existing agents remain brittle on long-horizon workflows, where they must repeatedly reconstruct program state scattered across source files, configurations, tests, dependencies, and runtime behavior, leading to increasingly long interaction histories, context explosion, and semantic drift. Large repositories further complicate the identification of task-relevant components. To address these challenges, we introduce Dev-Primitives (Development Primitives), a modular and executable abstraction that transforms repository components from passive software artifacts into active participants in software engineering. Each Dev-Primitive pairs a repository artifact with a resident LLM, which gives the artifact an agent-native interface grounded in its own implementation and dependencies, enabling natural-language reasoning, inter-component communication, and localized self-modification. Building on Dev-Primitives, we propose HERMES, a Harness Engineering framework for software engineeRing via Modular Executable Dev-PrimitiveS, which instantiates these primitives at repository scale through a dependency-aware dynamic activation mechanism and a bug diagnosis mechanism that maps execution evidence back to the components that must be revised. Extensive experiments on four software engineering benchmarks demonstrate that HERMES outperforms matched baseline harnesses by 12.4% on average. Moreover, when paired with strong activation and diagnosis models, HERMES, even with Qwen3-8B Dev-Primitives, remains within 4.5% of the homogeneous GPT-5.6 Sol configuration across all four benchmarks, while reducing inference cost by 26.2% on Terminal-Bench 4.0, highlighting the importance of harness design in software engineering agents.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Harness Engineering, Large Language Models, Software Engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Jo51OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Dev-Primitives, where each repository file is paired with its own LLM that edits that file and sends messages to related files. HERMES adds an activation step that picks which files take part and a diagnosis step that reads execution output and decides what to revise, looping for a few rounds. Experiments cover SWE-bench Verified, SWE Refactor Bench, Terminal-Bench and DevOps-Gym against several existing harnesses, plus ablations and a study that mixes small and large models across components.

Critical strengths
  1. The evaluation is broad, and the single-editor control in the appendix (same selected files, same diagnosis loop, then a compute-matched variant) is the right test for the main claim. It is the most convincing experiment in the paper.
  2. The analysis of where to spend a stronger model is useful in practice. Putting it on diagnosis helps more than putting it on activation, which is a concrete takeaway for cost-aware agent design.
Critical weaknesses
  1. Some results are hard to accept and do not agree with each other. Qwen3-8B with a 32K context reaching 80.6 on SWE-bench Verified on its own is far beyond what I would expect from a model of that size, and beyond a single case study the paper gives no per-task evidence for it. In the scaling table, swapping only the Claude Sonnet 5 primitives for Qwen3-8B gives clearly higher scores than running HERMES with Sonnet 5 everywhere, on both SWE-bench Verified and Terminal-Bench. This goes against the paper’s own claim that larger primitives help. I would need released patches and trajectories, and an explanation of this inversion, before trusting the headline numbers.
  2. The comparison with MAGIS and CodeTeam in Table 5 is not informative. Both numbers are copied from the original papers under their own backbones and settings, and the MAGIS one comes from a much older model generation, so the gap mostly reflects the model, not the framework. Either rerun them with the same backbone or drop the table.
  3. The novelty is limited. Per-file agents that message each other along dependencies are close to the file-level developer agents in MAGIS and CodeTeam, and to the agent-native primitive framing of Jin et al. Appendix F.5 says model state is not kept across calls; a primitive is simply called again with the current file and the accumulated messages. So resident LLM and active participant are mostly framing. Activation is localization and diagnosis is a reflection loop over test output, both standard.
  4. The comparisons are not controlled. Baselines are a mix of leaderboard numbers and the authors’ own runs (Appendix F.8).
Questions for authors
  1. Can you release per-instance patches and logs for the Qwen3-8B runs on SWE-bench Verified? How are files longer than the 32K window handled?
  2. Why does replacing only the Sonnet 5 primitives with Qwen3-8B beat HERMES with Sonnet 5 everywhere? Were the effort setting and runs the same?
  3. How often does diagnosis return PASS when the hidden tests later fail? In the case study it passes without any fail-to-pass reproducer.
4 comments

Thank you for the thorough review. We are glad that you regard the single-editor control as a direct test of our main claim and as the most informative experiment in the paper, and that you find placing a stronger model on diagnosis a concrete takeaway for cost-aware agent design. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We have carefully considered your concerns and respond to them individually below.


W1/Q1 (1/2). Some results are hard to accept and do not agree with each other. Qwen3-8B with a 32K context reaching 80.6 on SWE-bench Verified on its own is far beyond what I would expect from a model of that size, and beyond a single case study the paper gives no per-task evidence for it. Can you release per-instance patches and logs for the Qwen3-8B runs on SWE-bench Verified? How are files longer than the 32K window handled?
R1/A1 (1/2). Thank you for asking for per-task evidence. Yes, we've updated the anonymous repository of Appendix J with the per-instance patches and trajectories of HERMES with Qwen3-8B on SWE-bench Verified. We also ran three further checks of the 80.6 result. Under mini-SWE-agent with the same deployment, Qwen3-8B resolved 11.8%. Two more HERMES seeds gave 80.3 ± 0.8% over three seeds. On 200 tasks created after the model's release, HERMES resolved 47.5%, compared with 5.0% for mini-SWE-agent.

Each primitive call holds its own artifact with its objective and messages, so the 32K window has to fit one file rather than the repository. The original submission did not describe how an artifact longer than the window would be handled. We therefore audited every Qwen3-8B call of HERMES in the Table 1 run. We compared the token count of each prompt with the number of tokens that Ollama evaluated. The largest input had 28.9K tokens, and 0 calls were truncated. Inputs longer than 24K tokens occurred for 43 of the 2,473 activated primitives. We report these checks and the audit in Appendix K.7, and have added mini-SWE-agent with Qwen3-8B to Tables 1–4.

W1/Q2 (2/2). In the scaling table, swapping only the Claude Sonnet 5 primitives for Qwen3-8B gives clearly higher scores than running HERMES with Sonnet 5 everywhere, on both SWE-bench Verified and Terminal-Bench. This goes against the paper's own claim that larger primitives help. I would need released patches and trajectories, and an explanation of this inversion, before trusting the headline numbers. Why does replacing only the Sonnet 5 primitives with Qwen3-8B beat HERMES with Sonnet 5 everywhere? Were the effort setting and runs the same?
R1/A2 (2/2). Yes, the effort setting and the pipeline were the same. Both configurations used the default medium reasoning effort and B = 3 (Sec. 4.1), and both were separate runs of the same pipeline. We would like to clarify that the benefit of scaling the Dev-Primitives in Sec. 4.3 was measured with GPT-5.6 Sol in the other roles. There it added 1.2 to 4.5 points, for example from 94.2 to 97.0 on SWE-bench Verified (Table 6). Table 6 has no homogeneous Claude Sonnet 5 configuration, so its reference is the Claude Sonnet 5 result of Tables 1–4. This configuration reached 85.8 on SWE-bench Verified and 35.8 on Terminal-Bench 4.0, compared with 92.4 and 48.8 with Qwen3-8B Dev-Primitives (Table 6).

To check that this gap is stable, we ran two more SWE-bench Verified seeds and a new five-trial Terminal-Bench 4.0 run of both configurations. Over three seeds, the configuration with Qwen3-8B Dev-Primitives resolved 92.3%, compared with 85.7% for homogeneous Claude Sonnet 5. On Terminal-Bench 4.0, the new runs resolved 47.9% and 36.4%.

To find the cause, we measured how often primitives edit code outside the region named in their objective. A hunk counts as outside when it does not overlap that region. Claude Sonnet 5 Dev-Primitives placed 22.9% of their hunks outside their objective, compared with 6.4% for Qwen3-8B Dev-Primitives. We then re-ran both configurations once with an edit-scope guard. The guard rejects such hunks and returns them to activation as a message. It is an analysis tool and is not used in Tables 1–4. The table reports both configurations, with Claude Sonnet 5 in the other roles.

Dev-PrimitivesSWE-bench VerifiedTerminal-Bench 4.0
Claude Sonnet 5 (Tables 1 and 3)85.835.8
Claude Sonnet 5, with guard91.846.7
Qwen3-8B (Table 6)92.448.8
Qwen3-8B, with guard92.648.5

With the guard, homogeneous Claude Sonnet 5 rose from 85.8 to 91.8 on SWE-bench Verified and from 35.8 to 46.7 on Terminal-Bench 4.0. The configuration with Qwen3-8B Dev-Primitives stayed at 92.6 and 48.5. These results point mainly to how Claude Sonnet 5 Dev-Primitives follow their objective. We report these runs in Appendix K.8, Tables 26 and 27, and have revised Sec. 4.3 to state that the scaling benefit is measured with GPT-5.6 Sol in the other roles.

W2. The comparison with MAGIS and CodeTeam in Table 5 is not informative. Both numbers are copied from the original papers under their own backbones and settings, and the MAGIS one comes from a much older model generation, so the gap mostly reflects the model, not the framework. Either rerun them with the same backbone or drop the table.
R2. Thank you for suggesting these re-runs. We re-ran both frameworks with our backbones and have replaced Table 5 with these re-runs. In the original submission, Table 5 listed each framework's reported score in the setting of its own paper. In the re-runs, MAGIS and CodeTeam (PE) used GPT-5.6 Luna on the two SWE-bench sets and GPT-5.6 Sol on SWE Refactor Bench. On SWE Refactor Bench, we capped their budget at HERMES's cost. On NL2Repo-Bench, they used HERMES's backbone. The table reports the re-runs, where SWE-bench is the split of the original Table 5.

MethodSWE-benchSWE-bench VerifiedSWE RefactorNL2Repo-Bench
MAGIS28.791.614.5-
CodeTeam (PE)-92.817.540.2
HERMES36.695.631.044.9

With GPT-5.6 Luna, MAGIS reached 28.7 on the SWE-bench split, compared with 13.9 in the original Table 5 and 36.6 for HERMES. On SWE-bench Verified, MAGIS and CodeTeam (PE) reached 91.6 and 92.8, compared with 95.6 for HERMES. On SWE Refactor Bench, they reached 14.5 and 17.5, compared with 31.0. On NL2Repo-Bench, CodeTeam (PE) reached 40.2, compared with 34.6 in the original Table 5 and 44.9 for HERMES. CodeTeam (SFT) relies on its own fine-tuned weights, so Table 5 keeps its reported result.

W3. The novelty is limited. Per-file agents that message each other along dependencies are close to the file-level developer agents in MAGIS and CodeTeam, and to the agent-native primitive framing of Jin et al. Appendix F.5 says model state is not kept across calls; a primitive is simply called again with the current file and the accumulated messages. So resident LLM and active participant are mostly framing. Activation is localization and diagnosis is a reflection loop over test output, both standard.
R3. We would like to clarify that the novelty of HERMES lies in binding each reasoning unit to one repository artifact and in routing requirements and failures between these units. Each Dev-Primitive pairs one artifact with a resident LLM, so the repository induces the set of primitives. MAGIS and CodeTeam assign file-level tasks to developer agents and coordinate them through a manager, a shared plan, or a design contract. The designer thus fixes in advance which developers exist and along which paths they talk. The Agent Primitives and Tool Primitives of Jin et al. wrap reusable, task-agnostic behaviors or tools, so they hold no state of a particular file.

This leaves two difficulties for repository tasks. First, requirements arise while files are being edited, and each one may concern a caller, a configuration, or a test. Second, every edit changes the repository, so a central agent has to rebuild and pass on the state of every file. We address these difficulties with three mechanisms. Each primitive addresses its messages to the components its change implicates, so requirements follow dependency edges as they are found. Dynamic activation expands the localized files along their dependencies, including configurations and tests, and starts only these primitives, each with its own objective. Bug diagnosis names the components behind an execution failure, and activation reactivates only their primitives. A single agent needs no such routing, because it edits every file itself.

"Resident" refers to the binding of a primitive to its artifact. Across the revision rounds of a task, a primitive keeps the current version of its artifact and the messages it has received. Its LLM is called again on this state at each activation (Appendix F.5). We have added this description to Sec. 3.2.

In Table 9 of the original submission, the w/o Diagnosis Feedback variant kept execution and let activation read the raw output. Its SWE Refactor Bench score fell from 31.0 to 20.0. We also built a file-agent baseline with one agent per file and a manager that relays messages and reflects on the raw execution output. We then added HERMES's mechanisms one at a time at the same call budget. Replacing the manager's reflection with bug diagnosis was the largest step, from 22.5 to 31.0 on SWE Refactor Bench and from 45.8 to 51.8 on Terminal-Bench 4.0. We report this ladder in Appendix K.5, Table 22. The differences are also stated in Sec. 2.

W4. The comparisons are not controlled. Baselines are a mix of leaderboard numbers and the authors' own runs (Appendix F.8).
R4. Thank you for raising this point. In the original submission, our baselines came from two sources (Appendix F.8). We used a public result when it matched the model, harness, reasoning effort, and benchmark version. Otherwise, we ran the baseline harness ourselves with HERMES's snapshot and protocol, as for every medium-effort baseline on Terminal-Bench 4.0.

To make every headline comparison controlled, we re-ran the baseline of each of the nine headline pairs ourselves. These re-runs used the same task snapshots and containers as HERMES, the benchmark's default wall-clock cap, and API retries on transient errors only. We ran each baseline at its native budget and with restarts up to HERMES's mean cost per task, and we paired the outcomes per task. SWE Refactor Bench scores are means over three paired runs. The table compares the baselines of Tables 1–4 with our re-runs.

SettingHarnessTables 1–4Native re-runMatched re-runHERMES
SWE-bench Verified, GPT-5.6 Solmini-SWE-agent96.296.096.497.0
SWE-bench Verified, Claude Opus 4.8Claude Code85.886.287.691.0
SWE-bench Verified, GPT-5.5Codex76.477.080.885.6
SWE Refactor Bench, GPT-5.6 SolCodex6.57.010.531.0
SWE Refactor Bench, Claude Opus 5Claude Code28.527.529.536.3
Terminal-Bench 4.0, GPT-5.6 SolCodex33.032.436.451.8
Terminal-Bench 4.0, Claude Opus 5Claude Code47.047.347.648.5
DevOps-Gym, GPT-5.6 SolCodex49.8949.6551.3355.41
DevOps-Gym, Claude Opus 5Claude Code52.0652.3654.1257.10

At the native budget, the re-runs were within 1.0 point of the baselines in Tables 1–4 in all 9 pairs. At the matched budget, HERMES remained ahead in all 9 pairs, and the paired 95% CI excluded zero in 6 of them. We report these re-runs in Appendix K.1, Table 18.

Q3. How often does diagnosis return PASS when the hidden tests later fail? In the case study it passes without any fail-to-pass reproducer.
A3. Thank you for asking about this. We matched every final diagnosis verdict of our default runs with the held-out evaluation tests. The table reports false accepts as a share of PASS verdicts, and the PASS verdicts among the tasks or trials that fail the held-out tests.

SettingFalse acceptsPASS on failing tasks
SWE-bench Verified, GPT-5.6 Sol1.4%7 of 15
SWE-bench Verified, Qwen3-8B8.9%39 of 97
Terminal-Bench 4.0, GPT-5.6 Sol13.8%27 of 159 trials

With GPT-5.6 Sol on SWE-bench Verified, 1.4% of PASS verdicts were false accepts. With Qwen3-8B, the case-study backbone, the rate was 8.9%.

Diagnosis cannot return PASS when a task-visible test, the build, or another observed execution fails (Appendix B.4). A false accept can therefore occur only when every retained visible check passes, as in the case study. With GPT-5.6 Sol, PASS without a working reproducer occurred on 44 tasks, and 5 of the 7 false accepts are among them. As an additional check, we tested a policy that requires a working reproducer before PASS. It reduced false accepts from 1.4% to 0.8% of PASS verdicts at 7% more cost. We report this analysis in Appendix K.9, Tables 28 and 29, and have added the policy to Appendix B.4 as an option.

Dear Reviewer Jo51,

Thank you again for your thoughtful review and for highlighting the single-editor control as a direct test of our main claim. We also appreciate your comments on the practical value of using a stronger model for diagnosis.

In our rebuttal, we have carefully addressed your key concerns and provided the per-task evidence you requested.

  • (W1/Q1) The anonymous repository now includes the per-task Qwen3-8B patches and trajectories for SWE-bench Verified. HERMES resolved 80.3% of tasks on average over three seeds. The longest input in the audited run was 28.9K tokens, and none was truncated.
  • (W1/Q2) At the same reasoning effort, 22.9% of Sonnet 5’s edits fell outside the assigned region, compared with 6.4% for Qwen3-8B. Restricting edits to that region narrowed the SWE-bench Verified gap from 6.6 to 0.8 percentage points.
  • (W2–W4) On SWE Refactor Bench with the same backbone and cost cap, MAGIS scored 14.5, CodeTeam (PE) scored 17.5, and HERMES scored 31.0. We also reran all nine headline comparisons at matched budgets and tested the individual contributions of messaging, activation, and diagnosis in Table 22.
  • (Q3) On SWE-bench Verified, 8.9% of Qwen3-8B’s PASS verdicts and 1.4% of GPT-5.6 Sol’s PASS verdicts failed the hidden tests. For GPT-5.6 Sol, requiring a working reproducer reduced this rate to 0.8%.

We hope these clarifications and additional results help address your concerns. With two days left in the rebuttal period, we would be happy to provide further details on any points you would like to discuss.

Best regards,
The Authors

Reviewer_AmDYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HERMES, a framework that helps coding agents maintain consistency during long tasks involving many files. Its core idea is Dev-Primitives, which pair repository components, such as source files, configurations, and tests, with LLM agents that understand their own component, communicate requirements to related components, and make local edits. HERMES activates relevant components, runs the modified software, and uses execution feedback to diagnose failures and guide further revisions. The authors report an average improvement of 12.4 percentage points over matched baseline harnesses across four benchmarks. They also find that smaller models can handle component edits effectively when stronger models manage activation and diagnosis, reducing costs while retaining much of the performance. The main takeaway is that organizing agents around software dependencies and execution feedback can improve coding performance, though success still depends on accurate component selection and informative tests.

Critical strengths

One major strength of HERMES is that it splits coding work into clear responsibilities. Each AI agent looks after a particular part of the project, such as a code file, configuration file, or test. When an agent makes a change that affects another part, it communicates what needs to change there too. This is useful for large tasks because it helps agents keep related files consistent without requiring one agent to remember every detail of the project.

Another strength is that HERMES checks whether the changes work together and directs fixes to the right places. It starts with the parts of the project that seem relevant, runs the modified software, and examines test results and errors. It then uses that evidence to decide which agents should make further changes. The paper’s experiments show that removing testing feedback, error diagnosis, or communication between agents lowers performance. Activating agents only when needed also helps control costs.

The paper also provides evidence that the way agents are organized contributes to better results. A single editing agent performed worse even when it received the same relevant files. Giving that agent roughly the same computing budget improved its results, but HERMES still performed better on Terminal-Bench. The authors also found that smaller, cheaper models could handle individual files effectively when stronger models selected the relevant files and diagnosed failures. This suggests a practical way to reduce costs while keeping much of the performance of a system that uses strong models throughout.

Critical weaknesses

A key weakness is that the paper does not clearly explain its contribution compared with several relevant earlier systems. LongHorizon-Harness already addresses growing context and unreliable progress tracking by separating task management, execution, and independent checking. CodePlan uses software dependencies to coordinate changes across files, while RPG/ZeroRepo uses a structured graph to keep planning and implementation consistent. These works are missing from the paper’s references and should be discussed in Section 2 and Appendix A. Their designs differ from HERMES, but comparing them would help explain why assigning an agent to each component offers benefits beyond existing approaches to planning, memory, and verification.

A second weakness is that better performance often comes at a much higher cost. On SWE-bench Verified, GPT-5.6 Sol improves from 96.2% to 97.0%, while spending per task increases from $1.15 to $1.80. On SWE Refactor Bench, the improvement is larger, but spending rises from $6.00 to $24.80 per task. The paper includes a useful comparison with a single editing agent given roughly the same computing budget, but only on Terminal-Bench. Extending this comparison to all four benchmarks would make the practical value clearer. The refactoring benchmark also contains only 20 tasks, so broader testing is needed before assuming similar gains across other projects.

A third weakness is that the system can accept a repair without directly checking that the reported bug is fixed. In the Django example in Appendix H, three attempts to reproduce the bug fail during setup. HERMES still accepts the patch because existing tests pass and the changes appear consistent with the request. The hidden evaluation later confirms success, but the system itself lacked a working test of the original problem. The paper should measure how often its diagnosis incorrectly accepts patches and how often bug reproduction fails. Comparing this process with LongHorizon-Harness’s independent auditing would also help clarify the strengths and limits of its completion checks.

Missing citations:

  • LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks (2026).
  • CodePlan: Repository-level Coding using LLMs and Planning (2023).
  • RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation (2025).
Questions for authors

What does assigning an agent to each component add beyond earlier approaches? Could you compare HERMES with LongHorizon-Harness, CodePlan, and RPG/ZeroRepo to show which benefits specifically come from component agents and their communication?

Does HERMES still outperform simpler approaches at the same cost? Could you extend the comparison using equal computing budgets to all four benchmarks, including spending on planning, communication, diagnosis, and revision?

How reliably does HERMES decide that a task is complete? When it cannot reproduce the reported bug, what evidence supports accepting the patch, and how often does its diagnosis accept patches that later fail the hidden evaluation?

4 comments

We appreciate your thoughtful review and are encouraged by your recognition of our work: HERMES splits coding work into clear responsibilities, checks whether changes work together, and offers a practical way to reduce cost with small primitives. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We believe the remaining concerns can be addressed, and we provide detailed responses below.


W1/Q1. A key weakness is that the paper does not clearly explain its contribution compared with several relevant earlier systems. LongHorizon-Harness already addresses growing context and unreliable progress tracking by separating task management, execution, and independent checking. CodePlan uses software dependencies to coordinate changes across files, while RPG/ZeroRepo uses a structured graph to keep planning and implementation consistent. These works are missing from the paper's references and should be discussed in Section 2 and Appendix A. Their designs differ from HERMES, but comparing them would help explain why assigning an agent to each component offers benefits beyond existing approaches to planning, memory, and verification. Missing citations: LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks (2026). CodePlan: Repository-level Coding using LLMs and Planning (2023). RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation (2025). What does assigning an agent to each component add beyond earlier approaches? Could you compare HERMES with LongHorizon-Harness, CodePlan, and RPG/ZeroRepo to show which benefits specifically come from component agents and their communication?
R1/A1. We would like to clarify that the novelty of HERMES lies in where cross-file requirements are found and how they travel. Each Dev-Primitive pairs a repository artifact with a resident LLM, which works out what a change to its artifact requires of other artifacts. The systems you cite coordinate this through a central structure. LongHorizon-Harness keeps the task state outside execution. A manager maintains this state and picks the next subtask, a fresh-context executor performs it, and a read-only auditor verifies the environment before the next round. Cross-file requirements thus pass through the manager's task state. CodePlan has a central planner schedule one LLM over code blocks, with a dependency graph from static analysis that is updated after each edit. RPG/ZeroRepo builds a repository planning graph during planning, which then guides generation.

In a repository, each edit creates new requirements for its callers, configurations, and tests. A central plan or graph therefore has to be brought up to date after every edit. In HERMES, the primitive that edits an artifact sends what its change requires directly to the artifacts that must satisfy it, within the same round. Implementation state stays with each artifact and its messages rather than in one central trajectory. CodePlan derives the obligations on dependent code from program analysis. A Dev-Primitive writes them in natural language, so they can carry semantic constraints, such as preserving non-ASCII output along the JSON paths of the case study (Table 17).

We have added LongHorizon-Harness (Ma et al., 2026), CodePlan, and RPG/ZeroRepo to Sec. 2 and Appendix A.

Following your suggestion, we compared HERMES with LongHorizon-Harness in the Terminal-Bench 2.1 setting of its paper. HERMES used Qwen 3.7-Plus in every role, with the sampling parameters of that paper. It also used Harbor with the Docker backend, a five-hour timeout, and three trials per task. The paper reports Claude Code and LongHorizon-Harness with the same model and setting. The table reports the mean score over the three trials of each of the 89 tasks.

MethodTerminal-Bench 2.1
Claude Code (Ma et al., 2026)69.7
LongHorizon-Harness (Ma et al., 2026)77.2
HERMES78.3

HERMES reached 78.3%, compared with 77.2% for LongHorizon-Harness and 69.7% for Claude Code. We also re-implemented a CodePlan-style system with GPT-5.6 Sol and a per-task budget capped at HERMES's cost. It reached 18.7 on SWE Refactor Bench, averaged over three runs, and 36.7 on Terminal-Bench 4.0, compared with 31.0 and 51.8 for HERMES. On NL2Repo-Bench, an RPG/ZeroRepo-style planner with HERMES's backbone reached 41.6, compared with 44.9 for HERMES. To isolate the component agents and their communication, we also built a file-agent baseline with one agent per file and a manager that relays every message. With GPT-5.6 Sol on Terminal-Bench 4.0, it reached 39.1. Direct inter-primitive communication raised it to 43.9, on-demand activation to 45.8, and bug diagnosis to 51.8. We report these comparisons in Appendix K.6, Tables 23 and 24, and the file-agent ladder in Appendix K.5, Table 22.

W2/Q2. A second weakness is that better performance often comes at a much higher cost. On SWE-bench Verified, GPT-5.6 Sol improves from 96.2% to 97.0%, while spending per task increases from $1.15 to $1.80. On SWE Refactor Bench, the improvement is larger, but spending rises from $6.00 to $24.80 per task. The paper includes a useful comparison with a single editing agent given roughly the same computing budget, but only on Terminal-Bench. Extending this comparison to all four benchmarks would make the practical value clearer. The refactoring benchmark also contains only 20 tasks, so broader testing is needed before assuming similar gains across other projects. Does HERMES still outperform simpler approaches at the same cost? Could you extend the comparison using equal computing budgets to all four benchmarks, including spending on planning, communication, diagnosis, and revision?
R2/A2. Thank you for raising this concern. Yes, with GPT-5.6 Sol, HERMES scored higher than simpler approaches at the same cost on all four benchmarks. Our tokens and costs sum every call of a trajectory, including activation, the Dev-Primitives, and diagnosis (Appendix F.8). An equal budget therefore covers planning, communication, diagnosis, and revision.

Following your suggestion, we ran two simpler approaches at HERMES's cost per task on all four benchmarks. First, we re-ran the baseline harness of each GPT-5.6 Sol pair with restarts until its cost per task reached HERMES's mean. This harness is mini-SWE-agent on SWE-bench Verified and Codex on the other benchmarks. Second, we extended the compute-matched single editor of Table 14 from Terminal-Bench 4.0 to all four benchmarks. This editor keeps HERMES's activation, execution, and diagnosis and edits all selected files in one context. The table reports both with GPT-5.6 Sol.

MethodSWE-bench VerifiedSWE RefactorTerminal-Bench 4.0DevOps-Gym
Baseline harness, matched budget96.410.536.451.33
Single editor, matched budget94.621.045.550.19
HERMES97.031.051.855.41

On SWE Refactor Bench, budget-matched Codex cost \24.5 per task, compared with \24.8 for HERMES. For the 20 tasks of this benchmark, we ran each baseline three times at matched budget. Both paired 95% CIs exclude zero, with sign-test p = 0.001 for Codex and 0.039 for Claude Code. The matched gains with GPT-5.6 Sol on Terminal-Bench 4.0 and DevOps-Gym also have 95% CIs that exclude zero. Terminal-Bench 4.0 has 66 tasks with five trials each. For applications where cost matters more, the heterogeneous configurations of Sec. 4.3 offer a cheaper option. On Terminal-Bench 4.0, GPT-5.6 Sol activation and GPT-5.5 diagnosis with Qwen3-8B Dev-Primitives reached 49.7% at \2.13k,comparedwith51.82.13k, compared with 51.8% at \\3.40k for homogeneous GPT-5.6 Sol (Table 7). We report the equal-cost comparisons in Appendix K.1, Table 18, and Appendix K.4, Table 21, and have added the paired runs to Sec. 4.2.

W3/Q3. A third weakness is that the system can accept a repair without directly checking that the reported bug is fixed. In the Django example in Appendix H, three attempts to reproduce the bug fail during setup. HERMES still accepts the patch because existing tests pass and the changes appear consistent with the request. The hidden evaluation later confirms success, but the system itself lacked a working test of the original problem. The paper should measure how often its diagnosis incorrectly accepts patches and how often bug reproduction fails. Comparing this process with LongHorizon-Harness's independent auditing would also help clarify the strengths and limits of its completion checks. How reliably does HERMES decide that a task is complete? When it cannot reproduce the reported bug, what evidence supports accepting the patch, and how often does its diagnosis accept patches that later fail the hidden evaluation?
R3/A3. Thank you for raising this concern. To measure how reliably diagnosis decides that a task is complete, we matched every final diagnosis verdict of our default runs with the held-out evaluation. We also recorded whether a working reproducer was built. We counted false accepts as a share of PASS verdicts and reproduction failures as a share of tasks. The table reports both.

SettingFalse acceptsReproduction failures
SWE-bench Verified, GPT-5.6 Sol1.4%9.4%
SWE-bench Verified, Qwen3-8B8.9%17.6%
Terminal-Bench 4.0, GPT-5.6 Sol13.8%n/a

Of the 15 SWE-bench Verified tasks that GPT-5.6 Sol left unresolved, diagnosis returned PASS on 7. With Qwen3-8B, it returned PASS on 39 of the 97 unresolved tasks. On Terminal-Bench 4.0, it returned PASS on 27 of the 159 failing trials.

When no reproducer works, PASS rests on the remaining checks. Diagnosis cannot turn a failing task-visible test, a failed build, or another observed execution failure into PASS (Appendix B.4). In the case study, the three reproduction attempts stopped while Django was being configured and were discarded. PASS then rested on a 407-test regression run without regressions and on the model's judgment that the diff matches the issue. Appendix H of the original submission treats this PASS as consistency with the issue rather than as proof of the hidden benchmark behavior. With GPT-5.6 Sol, diagnosis returned PASS without a working reproducer on 44 tasks, and 5 of the 7 false accepts are among them.

Following your suggestion, we also tested two stricter completion checks with GPT-5.6 Sol. The first requires a working reproducer before PASS. It reduced false accepts from 1.4% to 0.8% of PASS verdicts at 7% more cost, and resolution rose to 97.2%. The second adds an independent auditor, modeled on the read-only auditor of LongHorizon-Harness. The auditor sees the issue, the final diff, and the logs, and its approval is required for PASS. It reduced false accepts to 0.6% on SWE-bench Verified and from 13.8% to 9.9% on Terminal-Bench 4.0, with resolution at 96.8%. We report these analyses in Appendix K.9, Tables 28 and 29, and have added the reproducer-required option to Appendix B.4.

Dear Reviewer AmDY,

Thank you again for your thoughtful review and for highlighting how assigning each agent responsibility for one component helps keep related files consistent.

In our rebuttal, we have carefully addressed your three questions about prior work, cost, and the reliability of completion checks.

  • (W1/Q1) We added comparisons with LongHorizon-Harness and with implementations modeled on CodePlan and RPG/ZeroRepo. To isolate the contribution of direct communication, we also tested file agents that exchanged messages through a manager. Letting them message each other directly raised their Terminal-Bench 4.0 score from 39.1 to 43.9.
  • (W2/Q2) We extended the cost-matched single-editor comparison to all four benchmarks, counting planning, communication, diagnosis, and revision. With GPT-5.6 Sol, HERMES scored higher on all four, including 31.0 versus 21.0 on SWE Refactor Bench.
  • (W3/Q3) On SWE-bench Verified, 1.4% of GPT-5.6 Sol’s PASS verdicts failed the hidden tests. Five of the seven false accepts occurred without a working reproducer. Requiring approval from an independent auditor reduced the false-accept rate to 0.6%.

We hope these clarifications and additional results help address your concerns. With two days left in the rebuttal period, we would be happy to provide further details on any points you would like to discuss.

Best regards,
The Authors

Reviewer_XeESOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that coding agents struggle on long tasks because they must rebuild program state that is scattered over many files. It proposes Dev-Primitives, where each repository file is wrapped by its own LLM that can edit the file and send natural-language messages to the files it depends on. HERMES builds on this with an LLM that selects which files to activate and expands along dependencies, an execution step that runs the modified repository, and an LLM diagnosis step that maps failures back to files and starts another round. On SWE-bench Verified, SWE Refactor Bench, Terminal-Bench and DevOps-Gym, HERMES is compared with harnesses such as mini-SWE-agent, Claude Code and Codex on the same backbones, and is reported to improve on almost every backbone. The authors also show that a small Qwen3-8B model can serve as the Dev-Primitives when strong models do activation and diagnosis.

Critical strengths
  1. The evaluation is broad. It covers four benchmarks of different types and many backbones, and each comparison pairs a baseline harness with HERMES on the same model and reasoning effort.
  2. The analysis of which component needs a strong model is useful in practice. Strong models help most in diagnosis and activation, while the file-level edits can run on a small model with moderate loss. The link between selection recall and task success also supports that localization is the bottleneck.
Critical weaknesses
  1. It is unclear where the large gain over existing harnesses comes from. The ablation removes one part at a time, and the parts unique to HERMES, message passing and on-demand activation, cost only a few points each. The largest drops come from test execution and diagnosis, which Codex and Claude Code also do. This does not explain why HERMES beats these harnesses by close to twenty points on some benchmarks.
  2. The comparison is not fair. Many baseline numbers are copied from leaderboards, so they are not evaluated under the same setup as HERMES, for example the same token budget. HERMES spends more tokens and runs several revision rounds with execution. Under this setup, the small Qwen3-8B model with HERMES scores higher on SWE-bench Verified than many frontier models, which is hard to believe without a matched comparison.
Questions for authors
  1. Qwen3-8B is not evaluated under any standard harness, so the gain from HERMES on this model cannot be measured. Small models usually score low on hard software tasks, but here Qwen3-8B with HERMES scores higher on SWE-bench Verified than many frontier models. Please report Qwen3-8B under a standard harness such as mini-SWE-agent. If the jump is really this large, please explain which mechanism produces it, since this would be a surprising result.
  2. Please compare with a single agent that has the same localization, execution and diagnosis loop and the same budget, but edits all selected files in one context. This isolates the value of per-file LLMs and message passing.
  3. The paper does not show how performance changes with the token budget. HERMES uses clearly more tokens than the baselines on most backbones. Please plot performance against tokens for HERMES and the baselines on the same axis.
5 comments

Thank you for the review. We are encouraged that you found our evaluation broad, with each comparison on the same model and effort, and that you see the analysis of which component needs a strong model as useful in practice. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We answer each concern below with the relevant tables and numbers.


W1/Q2. It is unclear where the large gain over existing harnesses comes from. The ablation removes one part at a time, and the parts unique to HERMES, message passing and on-demand activation, cost only a few points each. The largest drops come from test execution and diagnosis, which Codex and Claude Code also do. This does not explain why HERMES beats these harnesses by close to twenty points on some benchmarks. Please compare with a single agent that has the same localization, execution and diagnosis loop and the same budget, but edits all selected files in one context. This isolates the value of per-file LLMs and message passing.
R1/A2. We would like to clarify that the part unique to HERMES is the Dev-Primitive abstraction, and the original submission already isolates it. The single-editor variant of Appendix G keeps HERMES's activation, selected files, execution, diagnosis, and revision budget (B = 3). A single editor then changes all selected files in one context. This is the most damaging change in our ablations (Sec. 4.5). On SWE Refactor Bench, the score fell from 31.0 to 16.8, and on Terminal-Bench 4.0 from 51.8 to 38.8 (Table 13). Table 9 removes one mechanism at a time and keeps the Dev-Primitives in every variant. Its message-passing and on-demand-activation variants therefore measure these mechanisms on top of the Dev-Primitives, not the abstraction itself.

Bug diagnosis in HERMES is also not the test loop of Codex and Claude Code. Each edit is local to one artifact, so a failure observed in the whole repository has to be traced to the artifacts that own the faulty code. Diagnosis names these components, and activation revises only their primitives. A single agent edits every file itself and has no owner to route a failure to.

The single agent requested in Q2 is also in the original submission. The compute-matched single editor of Table 14 has HERMES's localization, execution, and diagnosis loop and the same budget. On Terminal-Bench 4.0, it reached 45.5, compared with 51.8 for HERMES. We have extended this comparison to the other three benchmarks, adding candidate revisions until the editor's budget matched HERMES's. The table reports both editors and HERMES with GPT-5.6 Sol.

MethodSWE-bench VerifiedSWE RefactorTerminal-Bench 4.0DevOps-Gym
Single editor (Table 13)93.416.838.844.21
Single editor + compute matching94.621.045.550.19
HERMES97.031.051.855.41

At the same budget, the single editor reached 21.0 on SWE Refactor Bench, 94.6 on SWE-bench Verified, and 50.19 on DevOps-Gym, compared with 31.0, 97.0, and 55.41 for HERMES. On Terminal-Bench 4.0, the gain of close to twenty points over Codex comes in three steps. Re-running Codex with restarts until its cost matched HERMES's raised it from 32.4 to 36.4. The compute-matched single editor inside HERMES's pipeline reached 45.5. The Dev-Primitives then raised the score to 51.8. We report these results in Appendix K.4, Table 21, and the budget-matched Codex runs in Appendix K.1, Table 18.

W2/Q3. The comparison is not fair. Many baseline numbers are copied from leaderboards, so they are not evaluated under the same setup as HERMES, for example the same token budget. HERMES spends more tokens and runs several revision rounds with execution. Under this setup, the small Qwen3-8B model with HERMES scores higher on SWE-bench Verified than many frontier models, which is hard to believe without a matched comparison. The paper does not show how performance changes with the token budget. HERMES uses clearly more tokens than the baselines on most backbones. Please plot performance against tokens for HERMES and the baselines on the same axis.
R2/A3. We would like to clarify that every pair in Tables 1–4 shares the backbone, reasoning effort, and benchmark version. The original submission used a leaderboard or published result only when it matched the model, harness, reasoning effort, and benchmark version of the HERMES configuration (Appendix F.8). Otherwise, we ran the baseline harness ourselves on HERMES's benchmark snapshot and evaluation protocol, as for every medium-effort baseline on Terminal-Bench 4.0.

To match the budget as well, we re-ran the baseline of each of the nine headline pairs ourselves, on HERMES's task snapshots and at HERMES's mean cost per task. A baseline restarted from a clean snapshot until it reached this cost. HERMES remained ahead in all 9 budget-matched pairs, and the paired 95% CI excluded zero in 6 of them. On Terminal-Bench 4.0 with GPT-5.6 Sol, for example, budget-matched Codex reached 36.4, compared with 51.8 for HERMES.

The comparison of Qwen3-8B with frontier models in W2 is across harnesses. The frontier models that Qwen3-8B with HERMES exceeds on SWE-bench Verified run under their baseline harnesses, such as Claude Sonnet 5 at 79.6 under mini-SWE-agent (Table 1). With HERMES, every other backbone in Table 1 except GPT-5 Mini (80.2) scores above Qwen3-8B (80.6). Under mini-SWE-agent, Qwen3-8B resolved 11.8%.

The original submission reports the cost of each configuration in Tables 1–3, the total tokens in Table 3, and HERMES's score at each revision budget in Table 12. We also plotted resolution against total tokens for HERMES and the baselines on Terminal-Bench 4.0. Each system kept its backbone and reasoning effort and changed only its own budget. HERMES varied its revision budget B, and Codex and Claude Code varied their number of restarts. We also added the max-effort points of Table 3. The table reports the points with GPT-5.6 Sol.

MethodTokensResolution
Codex3.5B32.4
Codex, matched budget6.5B36.4
Codex, more restarts11.1B37.9
Codex, max effort (Table 3)4.4B37.3
HERMES, B=03.1B39.7
HERMES, B=14.6B45.2
HERMES, B=3 (Table 3)6.8B51.8

With GPT-5.6 Sol, HERMES at B = 1 resolved 45.2% with 4.6B tokens. Codex resolved at most 37.9%, with 11.1B tokens. Without revision, HERMES used 3.1B tokens and resolved 39.7%, compared with 3.5B tokens and 32.4% for Codex at its native budget. The next table reports the points with Claude Opus 5, whose Claude Code result is the highest baseline in Table 3.

MethodTokensResolution
Claude Code5.2B47.3
Claude Code, matched budget7.2B47.6
Claude Code, max effort (Table 3)6.5B51.8
HERMES (Table 3)7.3B48.5
HERMES, max effort (Table 3)9.4B53.0

With Claude Opus 5, HERMES resolved 48.5% with 7.3B tokens, compared with 47.6% for Claude Code at a matched 7.2B tokens. At max effort, Claude Code resolved 51.8% with 6.5B tokens, and HERMES resolved 53.0% with 9.4B tokens. We report these points in Appendix K.2, Table 19 and Figure 2.

Q1. Qwen3-8B is not evaluated under any standard harness, so the gain from HERMES on this model cannot be measured. Small models usually score low on hard software tasks, but here Qwen3-8B with HERMES scores higher on SWE-bench Verified than many frontier models. Please report Qwen3-8B under a standard harness such as mini-SWE-agent. If the jump is really this large, please explain which mechanism produces it, since this would be a surprising result.
A1. Under mini-SWE-agent, Qwen3-8B resolved 11.8% of SWE-bench Verified, compared with 80.6% with HERMES. We ran mini-SWE-agent with Qwen3-8B on all four benchmarks, with the same Ollama deployment, 32K window, and sampling as HERMES (Appendix E). The table reports both harnesses.

MethodSWE-bench VerifiedSWE RefactorTerminal-Bench 4.0DevOps-Gym
mini-SWE-agent11.80.06.14.27
HERMES (Tables 1–4)80.610.527.637.48

To locate the mechanism, we went from mini-SWE-agent to HERMES one step at a time on SWE-bench Verified, with Qwen3-8B in every role. Configuration (b) served the model with vLLM and YaRN for a 128K window. Configuration (c) added the files selected by HERMES's activation as hints. Configuration (d) placed a single editor inside HERMES's activation, execution, and diagnosis loop with three revision rounds. Configurations (d) to (f) used the Ollama 32K deployment of Appendix E. The table reports the resolved rates.

ConfigurationResolved
(a) mini-SWE-agent, 32K window11.8
(b) mini-SWE-agent, 128K window15.0
(c) mini-SWE-agent, 128K window, file hints27.8
(d) Single editor61.2
(e) HERMES, B=071.6
(f) HERMES, B=380.6

The ladder identifies three mechanisms. First, 41.3% of the failures of mini-SWE-agent in (a) were history overflows in the 32K window, while no primitive call in HERMES holds more than one artifact. Second, the Dev-Primitives without revision (e) resolved 71.6%, compared with 61.2% for the single editor with three revision rounds (d). Third, diagnosis-driven revision raised the score from 71.6% to 80.6% (f). Over three seeds, HERMES with Qwen3-8B resolved 80.3 ± 0.8%. On 200 tasks created after the model's release, HERMES resolved 47.5%, compared with 5.0% for mini-SWE-agent. We have added mini-SWE-agent with Qwen3-8B to Tables 1–4 and report the ladder in Appendix K.7, Table 25.

Reviewer_UsxVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Dev-Primitives, file- or artifact-associated LLM reasoning units that can edit their own components and exchange natural-language requirements along repository dependencies. HERMES activates relevant primitives on demand, executes their combined changes, and uses an LLM-based diagnosis mechanism to attribute failures and schedule revisions.

Experiments on SWE-bench Verified, SWE Refactor Bench, Terminal-Bench 4.0, and DevOps-Gym report gains over selected agent harnesses and investigate component ablations and heterogeneous allocation of model capacity. A configuration with small Qwen3-8B primitives and stronger activation/diagnosis models approaches the performance of a homogeneous stronger-model configuration at lower reported inference cost.

Critical strengths

Well-motivated systems problem. Localizing context and routing cross-file requirements are credible strategies for avoiding centralized context growth in long software-engineering workflows (Secs. 1 and 3).

Relevant component-level evidence. Communication, execution feedback, diagnosis, activation, and revision-budget ablations provide a useful breakdown of the architecture (Table 9; Appendices F and G). Notably, a compute-matched centralized editor reaches 45.5% versus HERMES's 51.8% on Terminal-Bench 4.0 (Table 14), suggesting that the difference is not exclusively due to token count in this comparison.

Practical heterogeneous-model analysis. Separating model choices for activation, primitives, and diagnosis yields an interesting performance–cost trade-off (Tables 6–7).

Critical weaknesses

The headline harness comparisons are not compute matched. For GPT-5.6 Sol on Terminal-Bench 4.0, HERMES uses 6.8B tokens and $3.4k versus the Codex baseline's 3.6B tokens and $1.9k, alongside a resolution gain from 33.0% to 51.8% (Table 3). This does not isolate the role of the harness from additional inference and revision opportunities. The compute-matched centralized-editor control in Table 14 is encouraging but does not replace compute-matched comparisons with the actual baseline harnesses across benchmarks.

Baseline provenance complicates comparability. Appendix F.8 explicitly mixes publicly reported leaderboard results with new controlled baseline runs. Matching model name, effort, and benchmark version is useful, but it does not establish identical execution budgets, tool affordances, retry policy, environment settings, or run-level pairing. The manuscript's broad framing that the harness is the only experimental difference should be qualified or supported with direct controlled re-runs.

Some claimed gains are small relative to uncertainty, and replication is limited. On SWE-bench Verified, the flagship GPT-5.6 Sol result improves from 96.2% to 97.0%, a difference of four tasks out of 500; the reported standard errors are 0.86 and 0.76 percentage points (Table 1). The paper does not provide a paired significance test. HERMES is usually run once per backbone (Sec. 4.1), and the 20-task SWE Refactor Bench has three runs for one HERMES configuration but no corresponding repeated, paired baselines.

Novelty relative to existing file-level multi-agent approaches is not fully isolated. Attaching an LLM to a file and exchanging natural-language dependency messages resembles prior file-assigned multi-agent editors. The direct MAGIS/CodeTeam comparison (Table 5) uses reported results from different model/evaluation configurations and is not a controlled method comparison. A same-backbone, same-compute file-agent baseline without dynamic activation/diagnosis would sharpen the novelty claim.

Scalability and cost accounting need more support. The average number of distinct activated primitives is only 4.7–12.6 per task (Table 8), leaving repository-scale coordination costs under much larger dependency sets untested. Since Qwen3-8B is run on local A40 hardware while other models use APIs (Appendix E), the cost definition should clarify whether local compute, hosting, and latency are included in the reported savings.

Questions for authors

Can the authors rerun Codex, Claude Code, and a strong contemporary file-based multi-agent baseline under matched total tokens/cost, identical wall-clock and retry budgets, and the same task snapshots? Please report paired task-level outcomes.

On SWE-bench Verified and SWE Refactor Bench, which gains remain statistically convincing over repeated trajectories? Can the authors provide paired bootstrap intervals or McNemar-style tests for binary outcomes and corresponding intervals for composite scores?

How much of the gain remains after comparing HERMES against a file-agent baseline with identical activation-selected files, per-file contexts, communication rounds, and model-call budget, differing only in the proposed mechanisms?

How are locally hosted Qwen3-8B GPU costs and serving latency incorporated into the economic comparison? What happens as the number of simultaneously activated primitives or dependency depth grows?

6 comments

We thank the reviewer for the constructive comments. We appreciate the recognition that HERMES addresses a well-motivated systems problem and that our ablations give a useful breakdown of the architecture, as well as the acknowledgment that the heterogeneous-model analysis yields an interesting performance-cost trade-off. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of the raised weaknesses and questions below.


W1/Q1. The headline harness comparisons are not compute matched. For GPT-5.6 Sol on Terminal-Bench 4.0, HERMES uses 6.8B tokens and \3.4kversustheCodexbaseline′s3.6Btokensand3.4k versus the Codex baseline's 3.6B tokens and \\1.9k, alongside a resolution gain from 33.0% to 51.8% (Table 3). This does not isolate the role of the harness from additional inference and revision opportunities. The compute-matched centralized-editor control in Table 14 is encouraging but does not replace compute-matched comparisons with the actual baseline harnesses across benchmarks. Can the authors rerun Codex, Claude Code, and a strong contemporary file-based multi-agent baseline under matched total tokens/cost, identical wall-clock and retry budgets, and the same task snapshots? Please report paired task-level outcomes.
R1/A1. Thank you for suggesting these controls. Yes, we re-ran Codex, Claude Code, and mini-SWE-agent for the headline pairs of all four benchmarks, at their native budget and at a budget matched to HERMES. Each re-run used HERMES's task snapshots and containers, the benchmark's default per-task wall-clock cap, and API retries on transient errors only. For the matched budget, a baseline restarted from a clean snapshot until its cost per task reached HERMES's mean. We submitted the last attempt that passed the task-visible tests, without using the held-out evaluation tests. We paired the outcomes per task and computed a paired bootstrap 95% CI on each benchmark's own metric. SWE Refactor Bench scores are means over three paired runs, and Table 2 reports the first of these runs with Claude Opus 5 (36.5). The exact tests are McNemar tests on SWE-bench Verified and DevOps-Gym and sign tests over the three runs on SWE Refactor Bench. The table reports the paired results.

SettingHarnessNativeMatchedHERMESGain vs. matched [95% CI]Exact test p
SWE-bench Verified, GPT-5.6 Solmini-SWE-agent96.096.497.0+0.6
−0.6,+1.8-0.6, +1.8

| 0.55 | | SWE-bench Verified, Claude Opus 4.8 | Claude Code | 86.2 | 87.6 | 91.0 | +3.4

+0.8,+6.0+0.8, +6.0

| 0.016 | | SWE-bench Verified, GPT-5.5 | Codex | 77.0 | 80.8 | 85.6 | +4.8

+1.8,+7.8+1.8, +7.8

| 0.002 | | SWE Refactor Bench, GPT-5.6 Sol | Codex | 7.0 | 10.5 | 31.0 | +20.5

+11.7,+29.7+11.7, +29.7

| 0.001 | | SWE Refactor Bench, Claude Opus 5 | Claude Code | 27.5 | 29.5 | 36.3 | +6.8

+0.8,+12.8+0.8, +12.8

| 0.039 | | Terminal-Bench 4.0, GPT-5.6 Sol | Codex | 32.4 | 36.4 | 51.8 | +15.4

+7.6,+23.0+7.6, +23.0

| - | | Terminal-Bench 4.0, Claude Opus 5 | Claude Code | 47.3 | 47.6 | 48.5 | +0.9

−4.5,+6.4-4.5, +6.4

| - | | DevOps-Gym, GPT-5.6 Sol | Codex | 49.65 | 51.33 | 55.41 | +4.08

+0.5,+7.7+0.5, +7.7

| 1.1e-4 | | DevOps-Gym, Claude Opus 5 | Claude Code | 52.36 | 54.12 | 57.10 | +2.98

−0.8,+6.7-0.8, +6.7

| 0.005 |

HERMES remained ahead in all 9 matched pairs, and the paired 95% CI excluded zero in 6 of them. With GPT-5.6 Sol on Terminal-Bench 4.0, budget-matched Codex reached 36.4, compared with 51.8 for HERMES. On SWE Refactor Bench, Codex at nearly HERMES's cost ($24.5 vs. $24.8 per task) reached 10.5, compared with 31.0 for HERMES. Over the four benchmarks, the average GPT-5.6 Sol gain at matched budget was 10.1 points, compared with 12.4 points in the original submission. We also re-ran MAGIS and CodeTeam (PE), two file-based multi-agent frameworks, with HERMES's backbones. With GPT-5.6 Sol on SWE Refactor Bench and the budget capped at HERMES's cost, they reached 14.5 and 17.5, compared with 31.0 for HERMES. We report these re-runs in Appendix K.1, Table 18, and have revised Sec. 4.2 to report the average gain at matched budget.

W2. Baseline provenance complicates comparability. Appendix F.8 explicitly mixes publicly reported leaderboard results with new controlled baseline runs. Matching model name, effort, and benchmark version is useful, but it does not establish identical execution budgets, tool affordances, retry policy, environment settings, or run-level pairing. The manuscript's broad framing that the harness is the only experimental difference should be qualified or supported with direct controlled re-runs.
R2. Thank you for pointing this out. We have revised Sec. 4.1 to state that each group pairs a baseline harness and HERMES on the same backbone and reasoning effort. In the original submission, Appendix F.8 described the two sources of baseline results. We used a reported result when it matched the model, harness, reasoning effort, and benchmark version. Otherwise, we ran the baseline harness ourselves on HERMES's benchmark snapshot and evaluation protocol. Every medium-effort baseline on Terminal-Bench 4.0 is such a controlled run.

Following your suggestion, we also re-ran the baseline of each of the nine headline pairs ourselves. These re-runs used HERMES's task snapshots and containers, the benchmark's default wall-clock cap, and API retries on transient errors only. We ran each baseline at its native budget and with restarts up to HERMES's mean cost per task, and we paired the outcomes per task. SWE Refactor Bench scores are means over three paired runs. The table compares the baselines of Tables 1–4 with our re-runs.

SettingHarnessTables 1–4Native re-runMatched re-runHERMES
SWE-bench Verified, GPT-5.6 Solmini-SWE-agent96.296.096.497.0
SWE-bench Verified, Claude Opus 4.8Claude Code85.886.287.691.0
SWE-bench Verified, GPT-5.5Codex76.477.080.885.6
SWE Refactor Bench, GPT-5.6 SolCodex6.57.010.531.0
SWE Refactor Bench, Claude Opus 5Claude Code28.527.529.536.3
Terminal-Bench 4.0, GPT-5.6 SolCodex33.032.436.451.8
Terminal-Bench 4.0, Claude Opus 5Claude Code47.047.347.648.5
DevOps-Gym, GPT-5.6 SolCodex49.8949.6551.3355.41
DevOps-Gym, Claude Opus 5Claude Code52.0652.3654.1257.10

At the native budget, the re-runs were within 1.0 point of the baselines in Tables 1–4 in all 9 pairs. At the matched budget, HERMES remained ahead in all 9 pairs, and the paired 95% CI excluded zero in 6 of them. We report these re-runs in Appendix K.1, Table 18.

W4/Q3. Novelty relative to existing file-level multi-agent approaches is not fully isolated. Attaching an LLM to a file and exchanging natural-language dependency messages resembles prior file-assigned multi-agent editors. The direct MAGIS/CodeTeam comparison (Table 5) uses reported results from different model/evaluation configurations and is not a controlled method comparison. A same-backbone, same-compute file-agent baseline without dynamic activation/diagnosis would sharpen the novelty claim. How much of the gain remains after comparing HERMES against a file-agent baseline with identical activation-selected files, per-file contexts, communication rounds, and model-call budget, differing only in the proposed mechanisms?
R4/A3. We would like to clarify that the novelty of HERMES lies in what defines each reasoning unit and in how requirements and failures travel between the units. Each Dev-Primitive pairs one repository artifact with a resident LLM, so the repository induces the set of primitives. File-assigned multi-agent editors, such as MAGIS and CodeTeam, assign file-level tasks to developer agents under a manager, a shared plan, or a design contract. In these systems, the designer fixes in advance which developers exist and along which paths they exchange information.

This creates two difficulties on long repository tasks. First, an edit to one file often has to be matched in its callers, configuration, and tests, and these requirements appear while the files are being edited. Second, every edit changes the repository again, so a central agent has to keep rebuilding and relaying what each change requires of the other files. We address these difficulties with three mechanisms. Each primitive sends what its change requires directly to the files that have to meet it, so messages follow dependency edges within a round. Dynamic activation selects the artifacts that take part. Bug diagnosis names the components behind an execution failure, so that only their primitives are revised. The differences are also stated in Sec. 2.

Following your suggestion, we built a file-agent baseline that differs from HERMES only in the proposed mechanisms. This baseline (F0) follows the description of MAGIS and CodeTeam in Appendix A. It has one agent per file and a manager that relays every message, keeps the file set fixed, and reflects on the raw execution output. We then added the mechanisms one at a time. F1 replaces the relay with inter-primitive communication, F2 adds on-demand activation, and F3 replaces the manager's reflection with bug diagnosis, which gives HERMES. All variants started from the same activation-selected files and used the same per-file contexts and communication rounds. Their call budgets were within 3% of HERMES's (GPT-5.6 Sol, B = 3). The table reports the scores of the four variants.

VariantSWE-bench VerifiedSWE RefactorTerminal-Bench 4.0
F0: manager relay93.214.039.1
F1: + inter-primitive communication94.020.043.9
F2: + on-demand activation94.422.545.8
F3: HERMES97.031.051.8

F0 reached 14.0 on SWE Refactor Bench and 39.1 on Terminal-Bench 4.0, compared with 31.0 and 51.8 for HERMES. Each mechanism raised the score on all three benchmarks. On each benchmark, the largest step came from replacing the manager's reflection with bug diagnosis. We also re-ran MAGIS and CodeTeam (PE) with our backbones. On SWE-bench Verified with GPT-5.6 Luna, they reached 91.6 and 92.8, compared with 95.6 for HERMES. On SWE Refactor Bench with GPT-5.6 Sol, they reached 14.5 and 17.5, compared with 31.0. We report the ladder in Appendix K.5, Table 22, and have replaced Table 5 with the same-backbone re-runs.

Comments on the paper

1 comment
Area_Chair_DMjGOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.