- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OriGent turns about 257K merged pull requests from 90 public hardware-related repositories into 18,257 executable tasks, each with agent-built fail-before/pass-after tests and a problem statement written to match the tested behavior. The authors post-train Qwen3.6-35B-A3B with SFT on GLM-5.3 teacher trajectories followed by online GSPO RL. HWE-Bench resolve rate rises from 39.3% to 64.3%, and SWE-bench Pro from 44.0% to 58.0%.
Critical strengths
Executable training data for hardware repositories is scarce, and the dataset's scale and coverage of different toolchain eras are a real engineering achievement. The empirical gains over a same-base baseline (KAT-Coder) are large.
Critical weaknesses
-
PR mining, agent-built fail-to-pass tests, problem statements written after test validation, and reusable environment profiles per repository closely follow R2E-Gym, Scale-SWE, SWE-smith, and SWE-rebench V2. The contribution is mainly adapting these to hardware toolchains. What is methodologically new compared with these software pipelines, and how many tasks fail without the hardware-specific steps?
-
The scope is not really hardware engineering. All tasks are functional bug fixes with binary tests. Nearly 70% are EDA-software or firmware code, mostly C/C++, and only 11% are RTL. No task covers PPA optimization, timing closure, or physical design. Can the pipeline build tasks with PPA or QoR rewards? If not, should the claim be narrowed to functional repair in hardware-related repositories?
-
Is the gain from hardware data or from the teacher? SFT on GLM-5.3 trajectories lifts SWE-bench Pro by 12 points, which suggests much of the gain is general agentic distillation rather than hardware-specific learning. How does SFT on the same number of GLM-5.3 software-task trajectories perform on HWE-Bench?
-
The training set includes core-v-verif (the verification environment for CVA6), caliptra-mcu-sw (related to Caliptra), and chipyard (which bundles Rocket). The largest gains are on Caliptra (37.5% to 87.5%) and Rocket (25.0% to 59.4%). What are HWE-Bench results after removing all training repositories that share code, submodules, or IP with the evaluation repositories?
-
The tests and problem statements are LLM-written, and the only check is fail-before/pass-after. What fraction of tasks pass a manual audit of test adequacy, and how often do alternative correct fixes fail the tests?
-
The RL stage is small and its cost is unreported. Online GSPO runs only about 16 iterations (about 6.3K trajectories) yet is credited with an 8.4-point gain. What are the GPU-hours and container executions, are rollouts strictly on-policy, and does performance keep improving with more iterations?
-
The dataset is the main contribution, but the environments, pipeline, and checkpoints are only promised, and no anonymous sample is provided. Can you share a representative subset and confirm that the teacher trajectories will be released?
-
Terminal-Bench drops by 3.4 points, and KAT outperforms OriGent there. Does mixing in general terminal tasks remove the regression?
Questions for authors
-
PR mining, agent-built fail-to-pass tests, problem statements written after test validation, and reusable environment profiles per repository closely follow R2E-Gym, Scale-SWE, SWE-smith, and SWE-rebench V2. The contribution is mainly adapting these to hardware toolchains. What is methodologically new compared with these software pipelines, and how many tasks fail without the hardware-specific steps?
-
The scope is not really hardware engineering. All tasks are functional bug fixes with binary tests. Nearly 70% are EDA-software or firmware code, mostly C/C++, and only 11% are RTL. No task covers PPA optimization, timing closure, or physical design. Can the pipeline build tasks with PPA or QoR rewards? If not, should the claim be narrowed to functional repair in hardware-related repositories?
-
Is the gain from hardware data or from the teacher? SFT on GLM-5.3 trajectories lifts SWE-bench Pro by 12 points, which suggests much of the gain is general agentic distillation rather than hardware-specific learning. How does SFT on the same number of GLM-5.3 software-task trajectories perform on HWE-Bench?
-
The training set includes core-v-verif (the verification environment for CVA6), caliptra-mcu-sw (related to Caliptra), and chipyard (which bundles Rocket). The largest gains are on Caliptra (37.5% to 87.5%) and Rocket (25.0% to 59.4%). What are HWE-Bench results after removing all training repositories that share code, submodules, or IP with the evaluation repositories?
-
The tests and problem statements are LLM-written, and the only check is fail-before/pass-after. What fraction of tasks pass a manual audit of test adequacy, and how often do alternative correct fixes fail the tests?
-
The RL stage is small and its cost is unreported. Online GSPO runs only about 16 iterations (about 6.3K trajectories) yet is credited with an 8.4-point gain. What are the GPU-hours and container executions, are rollouts strictly on-policy, and does performance keep improving with more iterations?
-
The dataset is the main contribution, but the environments, pipeline, and checkpoints are only promised, and no anonymous sample is provided. Can you share a representative subset and confirm that the teacher trajectories will be released?
-
Terminal-Bench drops by 3.4 points, and KAT outperforms OriGent there. Does mixing in general terminal tasks remove the regression?
2 comments
Response to Reviewer Jf3X (2/2)
Q6 — RL cost, on-policy execution, and additional iterations
Although the RL stage uses only 18 updates, the number of optimizer updates understates the compute required for environment-in-the-loop RL. Each update involves long-horizon agent rollouts with executable verification, followed by a model update on 64 H100 GPUs. A completed iteration takes approximately 6 hours, with rollout collection accounting for 86.6% of the framework-reported step time and training for 9.6%. This corresponds to approximately 384 H100 GPU-hours per update and 6,912 H100 GPU-hours for 18 updates. By comparison, the SFT stage completed 843 updates on 32 H100 GPUs in 18.65 hours, approximately 597 H100 GPU-hours, excluding data construction and teacher generation.
Across the 18 rollout rounds, we started 9,082 executions, counting one trial per agent attempt. Of these, 7,728 yielded binary verifier outcomes; 1,354 were canceled, aborted, or otherwise lacked a binary outcome. To reduce waiting time, we used oversampling and aborted the remaining trials once 32 mixed-outcome groups were available for an update, which accounts for many of the aborted trials. The 788 complete groups contain 6,304 executions: 77 all-success groups (616 executions), 127 all-failure groups (1,016 executions), and 584 mixed groups (4,672 executions). Zero-variance filtering removed the all-success and all-failure groups. The 18 updates used 576 mixed groups.
For each update, rollouts are collected under a fixed policy and followed by one synchronous update, so the schedule introduces no asynchronous policy staleness. TITO and rollout routing replay preserve the generated tokens and expert selections during training.
Most of the RL gains occurred within the first 15–20 iterations. After submission, we extended training to approximately 40 iterations, but HWE-Bench performance varied within about two percentage points without a clear upward trend. We stopped there due to compute constraints. We have added this compute and execution accounting to Appendix D.
Q7 — Anonymous sample; release of teacher trajectories
Thank you for this request. Please see General Response G1. The anonymous links already provide all 8,013 SFT tasks with their tests, the construction pipeline including the environment templates, and the GLM-5.3 teacher trajectories of all these tasks.
Q8 — Terminal-Bench regression and mixing terminal tasks
Thank you for this suggestion. Mixing in general terminal tasks would certainly add some terminal capability. However, our further investigation indicates that the main cause of the decline is the verbose output of the GLM-5.3 teacher, which the student inherits and which hurts performance outside the training domain. With the same teacher and recipe, the software-trained model in G2 also falls below the base model on Terminal-Bench 2.1 (37.1% vs. 39.3%; Table R1). In early experiments, we replaced GLM-5.3 with DeepSeek V4.1 Flash 1017, which solves our SFT tasks with 57% fewer output tokens, to generate the SFT trajectories for the same hardware tasks. The resulting model no longer declines on Terminal-Bench 2.1 and instead improves over the base model, with a similar HWE-Bench resolve rate. Our next step is therefore to mix in general terminal tasks and use a teacher with better output properties.
Response to Reviewer Jf3X (1/2)
Q1 — Methodological novelty; tasks that fail without hardware-specific steps
Please see General Response G3 for what is methodologically new. Without the hardware-specific steps, task quality and synthesis efficiency degrade together. 54% of our reference pull requests change no test or verification file, so a pipeline that relies on the pull request's own tests would lose these tasks or fall back on long regression suites that do not target the change. 37 of the 90 repositories need more than one toolchain profile, and 9.2% of the tasks require a toolchain different from the main one of their repository, which one environment per repository would not cover; building a separate image for every pull request instead multiplies storage and I/O. In early pilots, tests sometimes required the reference's exact error text or generated names, which led to our rule that tests must not fit the reference patch. With this rule and the independent statement review, which revises 51.7% of the generated statements, passing teacher patches reuse a median of only 11.7% of the reference's added lines (G4).
Q2 — Scope: PPA/QoR rewards and narrowing the claim
Thank you for this suggestion. Yes, the pipeline can construct tasks with PPA or QoR rewards. Such tasks need synthesis or implementation tools in the toolchain environment and a QoR metric in place of the binary test reward, while environment reconstruction, task construction, and the RL loop stay the same. Functional correctness is the first step, since a design must work before it can be optimized; our next step is to extend the rewards to design-quality objectives, such as PPA, QoR, and timing.
Q3 — Does the gain come from hardware data or from the teacher?
Please see General Response G2.
Q4 — Results after removing related repositories
Please see General Response G5.
Q5 — Manual audit of tests and alternative correct fixes
Please see General Response G4.