acceptodds
Under review as a conference paper at ICLR 2027

OriGent: Post-Training Hardware Engineering Agents through Scalable Environment Synthesis

Abstract

LLM-based coding agents have made substantial progress in solving software engineering tasks. Applying these agents to hardware engineering requires them to reason about circuit behavior, use simulation and electronic design automation (EDA) tools, and coordinate changes across register-transfer-level (RTL) designs, configuration files, and verification code. Learning these workflows requires hardware-specific interaction experience, but much of the relevant development data is proprietary, limiting public training resources. We introduce OriGent, a framework that constructs executable training environments from the commit history of public hardware repositories. Starting from merged pull requests, OriGent reconstructs repository states, builds target tests validated to fail before the reference repair and pass afterward, and generates problem statements aligned with the tested behavior. Separating reusable environment setup from task-specific construction yields over 18,000 executable tasks from 90 repositories spanning RTL design, verification, EDA software, hardware domain-specific languages, and firmware. We post-train Qwen3.6-35B-A3B through supervised fine-tuning on successful teacher trajectories collected in these environments, followed by online reinforcement learning. On HWE-Bench, a repository-level hardware engineering benchmark, the resolve rate increases from 39.3% for the base model to 55.9% after supervised fine-tuning and 64.3% after reinforcement learning. OriGent outperforms KAT-Coder-V2.5-Dev (51.3%), which is post-trained for agentic coding from the same base model. On a 200-task SWE-bench Pro subset, OriGent improves resolve rate from 44.0% to 58.0%, showing transfer from hardware-focused training to software engineering.

Primary area:
datasets and benchmarks
Keywords:
Hardware Engineering, LLM Agents, Environment Synthesis, Post-Training, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jf3XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OriGent turns about 257K merged pull requests from 90 public hardware-related repositories into 18,257 executable tasks, each with agent-built fail-before/pass-after tests and a problem statement written to match the tested behavior. The authors post-train Qwen3.6-35B-A3B with SFT on GLM-5.3 teacher trajectories followed by online GSPO RL. HWE-Bench resolve rate rises from 39.3% to 64.3%, and SWE-bench Pro from 44.0% to 58.0%.

Critical strengths

Executable training data for hardware repositories is scarce, and the dataset's scale and coverage of different toolchain eras are a real engineering achievement. The empirical gains over a same-base baseline (KAT-Coder) are large.

Critical weaknesses
  1. PR mining, agent-built fail-to-pass tests, problem statements written after test validation, and reusable environment profiles per repository closely follow R2E-Gym, Scale-SWE, SWE-smith, and SWE-rebench V2. The contribution is mainly adapting these to hardware toolchains. What is methodologically new compared with these software pipelines, and how many tasks fail without the hardware-specific steps?

  2. The scope is not really hardware engineering. All tasks are functional bug fixes with binary tests. Nearly 70% are EDA-software or firmware code, mostly C/C++, and only 11% are RTL. No task covers PPA optimization, timing closure, or physical design. Can the pipeline build tasks with PPA or QoR rewards? If not, should the claim be narrowed to functional repair in hardware-related repositories?

  3. Is the gain from hardware data or from the teacher? SFT on GLM-5.3 trajectories lifts SWE-bench Pro by 12 points, which suggests much of the gain is general agentic distillation rather than hardware-specific learning. How does SFT on the same number of GLM-5.3 software-task trajectories perform on HWE-Bench?

  4. The training set includes core-v-verif (the verification environment for CVA6), caliptra-mcu-sw (related to Caliptra), and chipyard (which bundles Rocket). The largest gains are on Caliptra (37.5% to 87.5%) and Rocket (25.0% to 59.4%). What are HWE-Bench results after removing all training repositories that share code, submodules, or IP with the evaluation repositories?

  5. The tests and problem statements are LLM-written, and the only check is fail-before/pass-after. What fraction of tasks pass a manual audit of test adequacy, and how often do alternative correct fixes fail the tests?

  6. The RL stage is small and its cost is unreported. Online GSPO runs only about 16 iterations (about 6.3K trajectories) yet is credited with an 8.4-point gain. What are the GPU-hours and container executions, are rollouts strictly on-policy, and does performance keep improving with more iterations?

  7. The dataset is the main contribution, but the environments, pipeline, and checkpoints are only promised, and no anonymous sample is provided. Can you share a representative subset and confirm that the teacher trajectories will be released?

  8. Terminal-Bench drops by 3.4 points, and KAT outperforms OriGent there. Does mixing in general terminal tasks remove the regression?

Questions for authors
  1. PR mining, agent-built fail-to-pass tests, problem statements written after test validation, and reusable environment profiles per repository closely follow R2E-Gym, Scale-SWE, SWE-smith, and SWE-rebench V2. The contribution is mainly adapting these to hardware toolchains. What is methodologically new compared with these software pipelines, and how many tasks fail without the hardware-specific steps?

  2. The scope is not really hardware engineering. All tasks are functional bug fixes with binary tests. Nearly 70% are EDA-software or firmware code, mostly C/C++, and only 11% are RTL. No task covers PPA optimization, timing closure, or physical design. Can the pipeline build tasks with PPA or QoR rewards? If not, should the claim be narrowed to functional repair in hardware-related repositories?

  3. Is the gain from hardware data or from the teacher? SFT on GLM-5.3 trajectories lifts SWE-bench Pro by 12 points, which suggests much of the gain is general agentic distillation rather than hardware-specific learning. How does SFT on the same number of GLM-5.3 software-task trajectories perform on HWE-Bench?

  4. The training set includes core-v-verif (the verification environment for CVA6), caliptra-mcu-sw (related to Caliptra), and chipyard (which bundles Rocket). The largest gains are on Caliptra (37.5% to 87.5%) and Rocket (25.0% to 59.4%). What are HWE-Bench results after removing all training repositories that share code, submodules, or IP with the evaluation repositories?

  5. The tests and problem statements are LLM-written, and the only check is fail-before/pass-after. What fraction of tasks pass a manual audit of test adequacy, and how often do alternative correct fixes fail the tests?

  6. The RL stage is small and its cost is unreported. Online GSPO runs only about 16 iterations (about 6.3K trajectories) yet is credited with an 8.4-point gain. What are the GPU-hours and container executions, are rollouts strictly on-policy, and does performance keep improving with more iterations?

  7. The dataset is the main contribution, but the environments, pipeline, and checkpoints are only promised, and no anonymous sample is provided. Can you share a representative subset and confirm that the teacher trajectories will be released?

  8. Terminal-Bench drops by 3.4 points, and KAT outperforms OriGent there. Does mixing in general terminal tasks remove the regression?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Jf3X (1/2)

Q1 — Methodological novelty; tasks that fail without hardware-specific steps

Please see General Response G3 for what is methodologically new. Without the hardware-specific steps, task quality and synthesis efficiency degrade together. 54% of our reference pull requests change no test or verification file, so a pipeline that relies on the pull request's own tests would lose these tasks or fall back on long regression suites that do not target the change. 37 of the 90 repositories need more than one toolchain profile, and 9.2% of the tasks require a toolchain different from the main one of their repository, which one environment per repository would not cover; building a separate image for every pull request instead multiplies storage and I/O. In early pilots, tests sometimes required the reference's exact error text or generated names, which led to our rule that tests must not fit the reference patch. With this rule and the independent statement review, which revises 51.7% of the generated statements, passing teacher patches reuse a median of only 11.7% of the reference's added lines (G4).

Q2 — Scope: PPA/QoR rewards and narrowing the claim

Thank you for this suggestion. Yes, the pipeline can construct tasks with PPA or QoR rewards. Such tasks need synthesis or implementation tools in the toolchain environment and a QoR metric in place of the binary test reward, while environment reconstruction, task construction, and the RL loop stay the same. Functional correctness is the first step, since a design must work before it can be optimized; our next step is to extend the rewards to design-quality objectives, such as PPA, QoR, and timing.

Q3 — Does the gain come from hardware data or from the teacher?

Please see General Response G2.

Q4 — Results after removing related repositories

Please see General Response G5.

Q5 — Manual audit of tests and alternative correct fixes

Please see General Response G4.

AuthorsOpenReview ↗

Response to Reviewer Jf3X (2/2)

Q6 — RL cost, on-policy execution, and additional iterations

Although the RL stage uses only 18 updates, the number of optimizer updates understates the compute required for environment-in-the-loop RL. Each update involves long-horizon agent rollouts with executable verification, followed by a model update on 64 H100 GPUs. A completed iteration takes approximately 6 hours, with rollout collection accounting for 86.6% of the framework-reported step time and training for 9.6%. This corresponds to approximately 384 H100 GPU-hours per update and 6,912 H100 GPU-hours for 18 updates. By comparison, the SFT stage completed 843 updates on 32 H100 GPUs in 18.65 hours, approximately 597 H100 GPU-hours, excluding data construction and teacher generation.

Across the 18 rollout rounds, we started 9,082 executions, counting one trial per agent attempt. Of these, 7,728 yielded binary verifier outcomes; 1,354 were canceled, aborted, or otherwise lacked a binary outcome. To reduce waiting time, we used oversampling and aborted the remaining trials once 32 mixed-outcome groups were available for an update, which accounts for many of the aborted trials. The 788 complete groups contain 6,304 executions: 77 all-success groups (616 executions), 127 all-failure groups (1,016 executions), and 584 mixed groups (4,672 executions). Zero-variance filtering removed the all-success and all-failure groups. The 18 updates used 576 mixed groups.

For each update, rollouts are collected under a fixed policy and followed by one synchronous update, so the schedule introduces no asynchronous policy staleness. TITO and rollout routing replay preserve the generated tokens and expert selections during training.

Most of the RL gains occurred within the first 15–20 iterations. After submission, we extended training to approximately 40 iterations, but HWE-Bench performance varied within about two percentage points without a clear upward trend. We stopped there due to compute constraints. We have added this compute and execution accounting to Appendix D.

Q7 — Anonymous sample; release of teacher trajectories

Thank you for this request. Please see General Response G1. The anonymous links already provide all 8,013 SFT tasks with their tests, the construction pipeline including the environment templates, and the GLM-5.3 teacher trajectories of all these tasks.

Q8 — Terminal-Bench regression and mixing terminal tasks

Thank you for this suggestion. Mixing in general terminal tasks would certainly add some terminal capability. However, our further investigation indicates that the main cause of the decline is the verbose output of the GLM-5.3 teacher, which the student inherits and which hurts performance outside the training domain. With the same teacher and recipe, the software-trained model in G2 also falls below the base model on Terminal-Bench 2.1 (37.1% vs. 39.3%; Table R1). In early experiments, we replaced GLM-5.3 with DeepSeek V4.1 Flash 1017, which solves our SFT tasks with 57% fewer output tokens, to generate the SFT trajectories for the same hardware tasks. The resulting model no longer declines on Terminal-Bench 2.1 and instead improves over the base model, with a similar HWE-Bench resolve rate. Our next step is therefore to mix in general terminal tasks and use a teacher with better output properties.

Reviewer_MuwuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OriGent, a pipeline that constructs executable hardware-engineering tasks from historical pull requests. It builds reusable toolchain environments, generates tests validated before and after reference repairs, and produces aligned problem statements. The resulting dataset contains 18,257 tasks from 90 repositories. Using these tasks for supervised fine-tuning and reinforcement learning improves Qwen3.6-35B-A3B from 39.3% to 64.3% on HWE-Bench. Performance also improves on a 200-task SWE-bench Pro subset, although it declines on Terminal-Bench.

Critical strengths
  1. The paper provides a potentially valuable training resource for repository-level hardware engineering. Preparing executable tasks across 90 repositories and heterogeneous toolchains is substantial engineering work and addresses a practical bottleneck in training hardware agents.

  2. The reported improvements are substantial and occur across all six HWE-Bench repositories, which are excluded from training. The comparison with the same-base KAT model is useful, and the authors transparently report negative terminal-task transfer and increased interaction length.

Critical weaknesses
  1. The reliability and incremental value of the environment-construction pipeline need stronger validation. Closely related workflows already exist in R2E-Gym [1] and Scale-SWE [2], making the quality and practical value of the hardware resource central to this contribution. However, the paper does not report an independent task-quality audit or a focused comparison establishing the benefits of its hardware-specific construction choices. Tests that distinguish the original code from the reference repair do not necessarily reject other incorrect implementations or cover the full problem specification.

  2. The performance gains come with substantially greater inference consumption. Mean output length increases from 22.8k tokens for the base model to 331.7k for OriGent, approximately 14.5 times as much. The shared six-hour limit makes the end-to-end comparison meaningful, and longer productive interaction can itself be a learned capability. Nevertheless, without budget-matched evaluation, the accuracy–cost trade-off remains unclear.

[1] Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., & Stoica, I. (2025). R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07164.

[2] Zhao, J., Chen, G., Meng, F., Li, M., Chen, J., Xu, H., ... & Jia, K. (2026). Immersion in the github universe: Scaling coding agents to mastery. arXiv preprint arXiv:2602.09892.

Questions for authors
  1. How do the base, KAT, SFT, and RL models compare under matched cumulative output budgets or measured inference costs?

  2. Is there an independent audit of generated task descriptions and tests, particularly their ability to reject incorrect alternative repairs? What evidence demonstrates the benefit of the hardware-specific construction choices over closely related workflows?

  3. Section 5.3 reports 66 tasks newly solved by RL and 129 tasks failing under both SFT and RL. However, Table 2 implies only 184 SFT failures, while 66 + 129 = 195. Please reconcile these counts and confirm whether the related token-consumption analysis changes.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Muwu

W1 / Q2 — Independent audit; benefit of hardware-specific construction

Please see General Response G4 for task and test quality, and G3 for the hardware-specific construction choices.

W2 / Q1 — Matched output budgets and inference cost

Thank you for raising this. We agree that OriGent is not more token-efficient than the base model or KAT. Table R7 compares the models under cumulative output budgets on the trajectories analyzed in Table 5: an attempt counts as solved within a budget if it succeeds using at most that many output tokens.

Table R7. HWE-Bench tasks solved within a cumulative output budget (% of 417 tasks).

Model≤50K≤100K≤200K≤400KNo limit
Qwen3.6-35B-A3B35.336.736.936.939.3
KAT-Coder-V2.5-Dev48.050.150.450.451.3
OriGent SFT8.421.135.343.655.9
OriGent SFT+RL6.719.735.547.064.3

We attribute the long outputs mainly to the GLM-5.3 demonstrations. GLM-5.3 trajectories are long, with verbose reasoning, and only about half of their turns contain reasoning. Under the Qwen chat template, a turn without reasoning is tokenized differently in a training sequence than at generation time, so we exclude such turns from the SFT loss (Section 4.1). The student is thus supervised only on turns with reasoning and produces more output per turn than its teacher (1,454 versus 880 tokens; Table 5). We are therefore exploring teachers with shorter trajectories and more consistent reasoning. DeepSeek V4.1 Flash 1017 solves our SFT tasks with 57% fewer output tokens than GLM-5.3 and includes reasoning in 92% of its turns, compared with 53%. In early experiments, an SFT model trained on its trajectories uses about one third of the output tokens on HWE-Bench and reaches a similar resolve rate. We plan to continue in this direction.

Q3 — Paired counts in Section 5.3

The discrepancy comes from reruns following infrastructure issues, such as container startup failures. Table 2 includes the rerun results needed for a fair comparison, whereas Section 5.3 inadvertently retains the pre-rerun numbers. We have updated Section 5.3 in the revised paper.

Reviewer_odb6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces OriGent, a pipeline that turns merged pull requests from public hardware-related GitHub repositories into executable training environments for coding agents. LLM agents restore each repository’s pre-change state and set up reusable toolchain profiles. They write target tests that must fail before the reference fix and pass after it, plus regression tests that must pass in both states. A separate agent then writes and reviews a problem statement aligned with the tested behavior. The result is 18,257 tasks from 90 repositories covering EDA software, firmware, RTL design, simulation/verification, and hardware DSLs.

The authors use these environments to post-train Qwen3.6-35B-A3B in two stages. The first is SFT on 8,013 successful GLM-5.3 trajectories. The second is online GSPO reinforcement learning on 2,163 filtered tasks, with context-compaction summaries trained jointly under the task reward. On HWE-Bench (417 tasks, repositories excluded from training), the resolve rate rises from 39.3% to 55.9% after SFT and 64.3% after RL, beating KAT-Coder-V2.5-Dev (51.3%) from the same base model. On a 200-task SWE-bench Pro subset, the score rises from 44.0% to 58.0%. Terminal-Bench 2.1 drops slightly (39.3% to 36.0%), and the trained model produces far longer trajectories than the base model.

Critical strengths
  1. A large, valuable resource for an area with little public training data. Executable, test-validated tasks from real hardware projects are hard to build: toolchains differ, histories are messy, and tests often don’t exist. The pipeline handles this carefully with reusable toolchain profiles, fail-before/pass-after checks in fresh containers, and a separate statement review step. The promised release of environments, pipeline, and checkpoints would be useful to the community.

  2. Strong, carefully decontaminated results. A 25-point gain on HWE-Bench, with clear gains over a same-base agentic coding model, is substantial. The authors take sensible precautions against leakage: they exclude the benchmark repositories, strip git history, and drop teacher trajectories that accessed the network. They also report results that don’t flatter the method, such as the Terminal-Bench drop and the large increase in output length.

Critical weaknesses
  1. It’s unclear whether the hardware environments themselves are what helps. The model is distilled from GLM-5.3, which already scores 83.5% on HWE-Bench, and the trained model copies its long, exploratory style. There is no baseline that applies the same teacher and the same SFT+RL recipe to general software environments (e.g., SWE-smith or SWE-rebench tasks). Without it, we can’t tell whether the gains come from hardware-specific data or simply from agentic distillation of a much stronger teacher. The SWE-bench Pro improvement points the same way. In addition, most of the training data isn’t really hardware design: EDA software (36%) and firmware (33%) dominate, mostly in C/C++, and only 11% is RTL design. The paper doesn’t break down which parts of the data drive the gains on an RTL-heavy benchmark.

  2. Test quality is never measured. The tests are written by an LLM that sees the reference patch, so they may be too tied to the original fix and reject valid alternatives, or too weak and accept wrong fixes. Since these tests also serve as the RL reward, this matters. There is no human audit or false-positive analysis.

Questions for authors
  1. Can you train the same base model with the same teacher and recipe on a comparable number of general software tasks, and compare on HWE-Bench? This is the key test of whether hardware environments matter.
  2. Which repository areas drive the HWE-Bench gains? For example, what happens if you train on only the RTL and simulation tasks, or remove the EDA and firmware tasks?
  3. How much code or task overlap exists between training repositories (e.g., caliptra-mcu-sw, Chipyard, BOOM, openhwgroup cores) and the HWE-Bench repositories? Can you report results excluding related repositories?
  4. How does the base model perform at matched compute, e.g., pass@k with similar total tokens?
  5. Has anyone manually checked a sample of the constructed tests for validity, both whether they accept alternative correct fixes and whether they reject wrong ones?
3 comments
AuthorsOpenReview ↗

Response to Reviewer odb6

W1 / Q1 — Same teacher and recipe on general software tasks

Please see General Response G2.

W1 / Q2 — Which data areas drive the HWE-Bench gains

Thank you for this question. Two experiments in the general response bear on it. The de-correlation retraining (G5) removes 666 SFT tasks, all from the RTL design, hardware DSL, and simulation and verification areas, about one third of the SFT tasks in these areas, and none from EDA or firmware. HWE-Bench performance remains at 56.8%, compared with 55.9% before removal, so the gain does not depend on the RTL-area tasks closest to the benchmark. Conversely, training on general software tasks with the same teacher and recipe gains only 4.8 points, compared with 16.6 points for our hardware data (G2). Together, these results indicate that the gain comes from the hardware data as a whole.

Q3 — Overlap with evaluation repositories

Please see General Response G5.

Q4 — Base model at matched compute

Thank you for this question. Table R6 compares base-model pass@k at similar total tokens, estimated from eight independent runs on all 417 tasks, with single attempts of our models.

Table R6. Base-model pass@k versus single attempts of our models on HWE-Bench. Output tokens for pass@k are k times the base model's mean output per attempt.

Base pass@2Base pass@4Base pass@8SFTSFT+RL
Output tokens per task~46K~91K~182K249K332K
Resolve rate (%)47.157.565.555.964.3

We attribute the long outputs of our models mainly to the GLM-5.3 demonstrations. Its trajectories are long, with verbose reasoning, and only about half of its turns contain reasoning. Under the Qwen chat template, a turn without reasoning is tokenized differently in a training sequence than at generation time, so we exclude such turns from the SFT loss (Section 4.1). The student is thus supervised only on turns with reasoning and produces more output per turn than its teacher (1,454 versus 880 tokens; Table 5). We are therefore exploring SFT teachers with better properties in both respects. In recent experiments, DeepSeek V4.1 Flash 1017 solves our SFT tasks with 57% fewer output tokens than GLM-5.3 and includes reasoning in 92% of its turns, compared with 53%. In early experiments, an SFT model trained on DeepSeek trajectories inherits the shorter outputs: on HWE-Bench, it uses about one third of the output tokens of a model trained on GLM-5.3 trajectories and reaches a similar resolve rate. We plan to continue in this direction.

W2 / Q5 — Manual check of test validity

Please see General Response G4.

Reviewer_odb6OpenReview ↗

Post-rebuttal assessment and follow-up questions

Thank you for the additional experiments and released materials. The software-data control and retraining after removing related repositories strengthen the evidence for the value of the hardware environments. I am maintaining my rating of 3 (weak acceptance), with a few remaining questions:

  1. The software-data control uses fewer training tokens and updates, and covers SFT rather than the full SFT+RL pipeline. Could you clarify how these differences limit the attribution of gains to hardware-specific data?

  2. Have you manually audited your own generated tests for accepting incorrect fixes or rejecting alternative correct fixes? The patch-diversity results are useful, but do not directly measure these error rates.

  3. Table R6 shows base-model pass@8 reaching 65.5% at approximately 182K output tokens, versus 64.3% at 332K tokens for SFT+RL. Could you distinguish the single-attempt improvement from compute efficiency, accounting for pass@k’s oracle selection and the cost of selecting and verifying candidates in practice?

AuthorsOpenReview ↗

Response to Reviewer odb6 (follow-up)

Thank you for taking the time to review our detailed response and for clarifying your assessment. We address your three remaining questions below.

1. Limits of the software-data control (fewer tokens/updates; SFT rather than SFT+RL)

Due to the limited rebuttal time, we could build only a smaller software set and run it through SFT, not a full SFT+RL pipeline: constructing software tasks, generating teacher trajectories, and training are each costly, and a single RL stage alone is about 6,900 H100-hours (Appendix D). We kept everything else matched to the hardware run, so the comparison stays like-for-like at the SFT stage: same base model, same recipe, same teacher, differing only in the source of the SFT data.

Given that, the two differences do not undercut the attribution. The token difference is small: the software run uses 71% of the hardware run's tokens per epoch (856M vs. 1,203M) and 659 versus 843 updates, and a 29% difference cannot explain a more-than-threefold difference in the HWE-Bench gain (4.8 vs. 16.6 points). The software data is also not weak: with the same teacher and recipe it gains 11.0 points on SWE-bench Pro, close to the 12.0 from the hardware data (Table R1), so the teacher and the recipe transfer; what does not transfer to HWE-Bench is the content of general software tasks.

2. Manual audit of our tests for false accepts and false rejects

We manually audited a random sample of 20 SFT tasks. For each we read the problem statement, the target and regression tests, the reference pull request, and a teacher patch that passed, and judged two things.

Rejecting an alternative correct fix (overfitting to the reference): 0 of 20. The tests check behavior by rebuilding the repository and running a stimulus or probe against runtime values or outputs. 13 of the 20 passing patches differ substantially from the reference: different files, different operation names, a different mechanism, an exception raised in a different translation unit, in one case an Input rather than Output HID descriptor byte. All still pass. Where a test pins a name or message, it is one the problem statement itself specifies.

Accepting an incorrect fix: none of the 20 passing patches passed by gaming the test (no edit to the test, no hardcoding of the checked value, no no-op change); all 20 are genuine implementations of the stated behavior. What we did see, in the opposite direction from overfitting, is that a few tests do not exercise every corner case of the stated behavior and could therefore miss an edge case.

On this sample, our tests do not reject correct alternatives. Exhaustive edge-case coverage is hard to guarantee for any behavioral test suite, and where ours falls short the error is under-coverage rather than fit-to-reference, which is the less harmful direction for a training signal: a loose accept still contributes a largely correct trajectory, whereas a false reject would discard a correct one.

3. Single-attempt improvement versus compute efficiency in Table R6

Table R6 places single attempts of our models next to base-model pass@k, and the two should not be read on one axis. pass@k is an oracle upper bound: it counts a task as solved if any of k independent attempts succeeds, which assumes a perfect selector that picks the correct attempt for free. In deployment there is no such oracle: choosing among k candidate hardware patches means running the testbenches and simulation that make up verification, k times, and then still selecting. So base pass@8 = 65.5% is the ceiling of eight verified attempts, not a single-submission result. The deployable comparison is a single attempt, on which SFT+RL resolves 64.3% against the base model's 39.3%.

Compute efficiency is a separate and real concern, and we agree the per-attempt cost is high (332K output tokens). We attribute it to the GLM-5.3 teacher, whose trajectories are long and carry reasoning in only about half their turns (Section 4.1), and we are addressing it directly: in early experiments, an SFT model trained on DeepSeek-V4.1-Flash trajectories reaches a similar resolve rate with about one third of the output tokens(90K output tokens), which reduces the per-attempt cost behind Table R6.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3): Overview, release, and hardware data versus teacher (G0–G2)

G0. Overview

We thank all reviewers for their thoughtful and constructive reviews. We are glad that the reviewers find our environments a valuable resource for an area where executable training data is scarce (odb6, Muwu, Jf3X), recognize their construction across heterogeneous hardware toolchains as substantial engineering work (Muwu, Jf3X), and find the gains substantial, including over the same-base KAT-Coder model (odb6, Muwu, Jf3X).

We have released the construction pipeline, all 8,013 SFT tasks, and their teacher trajectories at anonymous links (G1). During the rebuttal period, we also ran new experiments and analyses for the shared concerns:

  • Hardware data versus teacher (G2). With the same GLM-5.3 teacher and recipe, training on general software tasks gains 4.8 points on HWE-Bench, compared with 16.6 points for our hardware data.
  • Construction choices (G3). With the same teacher, our tasks need more than twice as many turns and tokens as SWE-rebench V2.1 and Scale-SWE tasks, and an audit of Scale-SWE tasks shows the failures that our checks prevent.
  • Task and test quality (G4). Passing teacher patches reuse a median of only 11.7% of the reference's added lines, and the independent review revised 51.7% of the generated statements.
  • Repository correlation (G5). Removing every training repository that shares code, submodules, or IP with an evaluation repository leaves the SFT resolve rate essentially unchanged (56.8% vs. 55.9%).

The revised paper incorporates these results, with changes marked in blue (G6).

G1. Release of code and data

To help the reviewers examine our work, we have released anonymized versions of our code and SFT data:

  • Construction pipeline and tasks (https://anonymous.4open.science/r/origent/): the toolchain profile templates of all 90 repositories, which render the Dockerfiles of the task environments; the agent prompts and skill for test construction and for statement writing and review; and the execution-check scripts. It also contains all 8,013 SFT tasks, each with its problem statement, target and regression tests, and verifier script.
  • SFT trajectories (https://huggingface.co/datasets/ORIGENT/origent-sft-trajectories): the GLM-5.3 teacher trajectories of all 8,013 SFT tasks.

We will further release the complete pipeline, all 18,257 environments with their images, the trained models, and a complete recipe for reproducing our training, so that the community can build on this work.

G2. Do the gains come from the hardware training data or from the teacher? (odb6 W1/Q1; Jf3X Q3)

To answer this, we kept the teacher fixed and changed only the training tasks: we trained the same base model on GLM-5.3 OpenCode trajectories from public software-repository tasks, built with the general components of our pipeline but without its hardware-specific steps. Training follows our hardware SFT recipe (256K context, three epochs, batch size 32, turn-level masks, and compaction supervision).

Table R1. Software-data control. Resolve rates are in %.

Base modelSoftware-trained SFTHardware-trained SFT (ours)
Training tasks—4,7088,013
Training tokens per epoch—856M1,203M
Optimizer updates—659843
HWE-Bench (417 tasks)39.344.155.9
SWE-bench Pro (200 tasks)44.055.056.0
Terminal-Bench 2.1 (89 tasks)39.337.134.8

The HWE-Bench gap is far out of proportion to the difference in data size (Table R1): with over 70% of the training tokens, the software-trained model gains only 4.8 points on HWE-Bench, less than a third of the 16.6-point gain of the hardware-trained model. On SWE-bench Pro, the same software data still yields a comparable gain (11.0 vs. 12.0 points). This indicates that the HWE-Bench gain comes mainly from our hardware training data rather than from the teacher alone.

AuthorsOpenReview ↗

General Response (2/3): Methodological contribution and hardware-specific construction (G3)

G3. Methodological contribution and hardware-specific construction (Muwu W1/Q2; Jf3X Q1)

Like R2E-Gym, Scale-SWE, SWE-smith, and SWE-rebench, OriGent builds executable tasks from repository histories. It differs in how the tasks are constructed and how they are used: these works stop at SFT or data release, whereas we further train with online RL and jointly learn context compaction for long-horizon tasks.

First, three construction choices make our training data more demanding and of higher quality.

Shared toolchain environments. Prior pipelines reuse one environment per repository snapshot (SWE-smith, SWE-rebench V2) or the environment of the nearest pull request (Scale-SWE). This does not carry over to hardware repositories, which combine HDL generators, simulators, and software build tools whose versions are coupled and change across a repository's history, so one environment often cannot build older revisions. A separate image per pull request is impractical for such heavy toolchains. We therefore group historical revisions by the toolchain they require and share one validated environment image within each group, which substantially reduces disk usage and I/O.

Per-task tests for checkable behavior. Hardware pull requests often change design behavior without adding a test for it, since verification usually lives in separate testbenches or long regression suites. Requiring pull requests with tests, as SWE-rebench does, would therefore discard many substantive hardware changes, and reusing existing suites, as SWE-smith and Scale-SWE partly do, would rely on slow simulations that are not aimed at the changed behavior. We instead keep such changes, and our agent writes a target test and a focused regression test for each task, including the testbenches or stimuli they need. The tests must not fit the reference patch, such as by requiring its exact error text or generated names, so alternative correct implementations pass.

Rewritten statements with independent review. Original issue text may leave out behavior the tests check or even reveal the solution; SWE-rebench V2, which keeps such text, drops 22% of its executable tasks at its issue-filtering stage. We instead have an agent rewrite each statement to cover all behavior and public interfaces the tests require while describing only the failure and the expected behavior, not the fix. Since this writer sees the reference patch, a separate review agent re-derives the verified behavior on its own and revises any statement that omits a tested requirement, admits a fix for a different bug, or leaks the solution.

What these checks prevent. Scale-SWE also writes its statements with an agent and generates tests for pull requests that lack them. We ran the same GLM-5.3 teacher on 759 of its tasks; on 7.0% of them the reference patch itself fails the tests. We then inspected all 278 tasks that the teacher failed while the reference passes: 234 of them (84%) failed because of the task, 40 because of the teacher, 3 on network access, and 1 remains unclear. In 170 of the 234, the tests check details the statement does not specify, such as module paths, exact error messages, or a version bump made in the same pull request; in the other 64, the tests themselves are defective, mostly because they depend on fixture files that only the reference patch adds or match the reference's source text. Most of the teacher's failures on Scale-SWE thus come from tasks that cannot be solved as stated.

Our tasks are also more demanding: their reference pull requests are larger, and the same teacher needs more than twice as many turns and tokens to solve them (Table R2).

Table R2. Our hardware tasks vs. SWE-rebench V2.1 and Scale-SWE, with the same GLM-5.3 teacher in OpenCode.

Our hardware tasksSWE-rebench V2.1Scale-SWE
Files changed per reference PR, median433
Lines changed per reference PR, median (IQR)132 (54–325)75 (37–153)66 (27–176)
Solved teacher trajectories8,013968†415†
Assistant turns per trajectory, median (IQR)113 (69–166)41 (27–69)33 (19–49)
Generated tokens per trajectory, median (IQR)56.8K (32.1K–88.9K)25.8K (10.3K–48.2K)17.7K (5.6K–37.6K)

†From uniform random samples of the SWE-rebench and Scale-SWE tasks.

Second, we provide a complete post-training recipe, SFT followed by online RL at 256K context with jointly trained compaction, that turns the resulting environments into a stronger hardware agent. The recipe deliberately uses simple, general components: execution-filtered teacher trajectories for SFT; turn-level masks for invalid or excessively long responses; standard GSPO with binary target-and-regression test rewards; zero-variance group filtering with oversampling; synchronous updates; and joint training of compaction summaries under the same task reward.

AuthorsOpenReview ↗

General Response (3/3): Task quality, repository correlation, and revisions (G4–G6)

G4. Task and test quality (odb6 W2/Q5; Muwu W1/Q2; Jf3X Q5)

Fail-before/pass-after is not the only check on our tasks.

Alternative correct fixes pass. As described in G3, the per-task tests are written not to fit the reference patch. GLM-5.3's passing patches mostly edit the same files as the reference but reuse only a small fraction of its added lines (Table R3), so the tests accept implementations that differ substantially from the reference.

Table R3. GLM-5.3 passing patches vs. the reference, on our SFT tasks.

Reference files also edited, mean72.9%
Reference added lines reused, median (IQR)11.7% (2.8%–28.6%)
Tasks reusing less than half of the reference's added lines88.5%

Wrong fixes fail. The target test exercises the repaired behavior and must fail on the original code through its behavioral assertion, so a patch that leaves this behavior unchanged cannot pass; the regression test rejects patches that break related behavior. The RL reward is positive only when both pass.

Rewritten statements with independent review. The independent review agent described in G3 revised 51.7% of the 18,080 statements it reviewed.

G5. Training/evaluation repository correlation (odb6 Q3; Jf3X Q4)

The reviewers name training repositories that may be related to the HWE-Bench repositories: caliptra-mcu-sw, Chipyard, BOOM, and OpenHW Group repositories such as core-v-verif. Jf3X further links them to the largest gains, on Caliptra and Rocket. Table R4 lists the training tasks each of them contributes.

Table R4. Training tasks from the repositories named by the reviewers (Table 8).

Named repositorySFT tasksRL tasksRelation to an eval repo
chipsalliance/caliptra-mcu-sw041Caliptra MCU firmware (Rust); same project as caliptra-rtl, no shared code or submodule
openhwgroup/core-v-verif710CVA6 verification environment
ucb-bar/chipyard260SoC generator that can bundle Rocket Chip
riscv-boom/riscv-boom (BOOM)80Same CPU family as Rocket Chip
openhwgroup/cv-hpdcache200Data cache IP included in CVA6 as a submodule; vendors OpenTitan ECC primitives

Following Jf3X's criterion, we removed every training repository that shares code, submodules, or IP with an evaluation repository. This removes 666 SFT tasks (8.3% of 8,013), covering the named repositories and the general hardware libraries that evaluation repositories vendor as submodules (common_cells, axi, chisel, riscv-isa-sim, riscv-arch-test). We retrained the SFT stage on the reduced data with the same base model and recipe (Table R5).

Table R5. HWE-Bench resolve rates (%) after removing related training repositories.

IbexCVA6CaliptraRocketXiangShanOpenTitanOverall
Base62.968.637.525.027.836.339.3
Original SFT65.774.387.559.451.950.255.9
De-correlated SFT68.688.675.053.155.650.256.8

Removing these repositories leaves the overall resolve rate essentially unchanged and preserves most of the gain on Caliptra and Rocket, which suggests that the related repositories are not the main source of the improvement.

G6. Summary of revisions

The revised paper includes the following changes.

SectionChange
§5.2Add the software-data control (Table R1, G2).
§5.2, Appendix AAdd the de-correlation retraining (Table R5) and the list of removed repositories (G5).
Appendix AAdd the comparison with SWE-rebench V2.1 and Scale-SWE (Table R2, G3).
Appendix BAdd the patch comparison (Table R3) and the statement-review statistics (G4).
§5.3Reconcile the counts of tasks newly solved by RL and tasks failing under both SFT and RL (Muwu Q3).
Appendix DAdd the RL compute and container-execution accounting (Jf3X Q6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.