acceptodds
Under review as a conference paper at ICLR 2027

ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generation

Abstract

Existing API-based agentic systems for RTL code generation are fundamentally misaligned with industrial practice: they assume a golden testbench is available at generation time, rely on closed-source APIs incompatible with chip vendors' air-gapped security requirements, and cannot be trained on vendors' proprietary RTL codebases, leaving valuable internal data unused. Recent self-trained models address the deployment constraint but remain single-turn generators that overlook the critical role of verification in real industrial flows. To bridge these gaps, we present ChipMATE, the first self-trained multi-agent framework for RTL generation. Inspired by industrial practice where correctness emerges from cross-comparison between independently written RTL modules and reference models, ChipMATE pairs a Verilog agent with a Python reference-model agent that mutually verify each other's outputs without any golden oracle. We design a backtrack-based inference workflow to prevent error propagation across turns, and a two-stage training pipeline that first trains each agent individually to saturate its code-generation capability, then trains the team jointly to collaborate effectively. To support the training, we further build a hybrid data-generation framework that produces 64.4K high-quality reference model training samples. ChipMATE achieves 75.0% and 80.1% pass@1 on VerilogEval V2 with 4B and 9B base models, outperforming all existing self-trained models and even DeepSeek V4 with 1600B parameters. We will open-source our model weights, inference workflow, and training dataset to facilitate future research.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AI4Chip, RTL, Multi-agent RL

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_3AeyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ChipMATE pairs two LLM agents for RTL generation. One writes Verilog, the other writes a Python reference model from the same spec, and neither sees the other's code. They compare simulation traces and keep a revision only when agreement improves. After a separate stage for each, the pair is trained jointly with a multi-agent RL method, X-GRPO. The 9B-per-agent system beats single-sample open RTL models on five benchmarks, significantly on VerilogEval. The system-level result holds up and is carefully measured. What the paper does not show is the title's premise, that joint multi-agent RL is what produces the improvement.

Critical strengths
  • The diagnostics are the most informative part of the paper. Without backtracking, mutual correction makes the pair worse than a single agent. The authors also measure where agreement breaks down: false consensus roughly doubles on the harder ChipBench-SC, mostly because both agents misread the spec the same way.
  • The role-swap experiment makes the system-level result convincing. Substituting the ChipMATE agents for CodeV-R1 one at a time raises accuracy at each step, and the lead over CodeV-R1 survives on the VerilogEval tasks that pass the overlap audit. Taken on its own terms, this is a solid incremental paper: the supported contribution is small, but the evidence behind it is broad and honestly reported.
Critical weaknesses
  • Joint training, the thing the title is about, may do little beyond run-to-run noise. Replacing the separately trained agents with jointly trained ones in the same workflow adds 1.4 points at 9B and 3.2 at 4B. Each comes from a single unpaired run on VerilogEval, and both sit inside the paper's own bootstrap intervals. At 4B, the X-GRPO run in Table 4 lands barely above the workflow without it and nearly three points below the 4B headline, and the paper doesn't explain why. Sixteen cells are also marked as pending reconciliation, including some of the method's own rows that Sections 3.3 and 3.4 build on.
  • The method adds one more knob to an existing recipe, which is fine, but then the pair's edge over plain test-time sampling has to carry the paper, and that edge is small. Checking Verilog against a Python reference model isn't new, as the authors themselves say; what's new is giving that role its own fine-tuned model, plus the X-GRPO credit. The Verilog agent alone, with self-consistency at about the same sampling budget, reaches 77.4 versus the pair's 80.1. That gap isn't tested for significance, and there's no 4B counterpart at all.
Questions for authors
  1. Can you compare the separately trained and jointly trained agents in the identical workflow, paired, on at least two benchmarks and more than one seed? A difference that survives that design is what the title needs.
  2. Do the Table 4 X-GRPO run and the 4B headline run share checkpoints and settings, or is 75.0 one of several runs?
  3. Is the 80.1 vs. 77.4 gap significant under a paired test on your existing outputs, and how does the 4B system compare with a similar self-consistency baseline?
  4. Can you resolve the sixteen pending cells and restate Sections 3.3 and 3.4 from final values?
3 comments

Thank you for the feedback and suggestions. New results are marked in blue in the revised PDF.


Q1, W1 Multi-seed, multi-benchmark training comparison

To isolate the gain from X-GRPO, we compare three versions of the same workflow: the Stage-1 agents without multi-agent RL, the same agents after AT-GRPO, and after X-GRPO. AT-GRPO and X-GRPO use the same reward and data, three candidates per agent per turn, and three seeds.

Pass@1 (%)Stage-1 agents, no multi-agent RL+ AT-GRPO (3-seed mean)+ X-GRPO (3-seed mean)X-GRPO gain over no multi-agent RL, pp [95% CI], paired p
4B, VerilogEval v271.870.475.4+3.6 [0.6, 6.7], p=.017
4B, RTLLM v272.071.375.2+3.2 [−1.5, 8.0], p=.18
4B, ChipBench-SC30.729.634.0+3.3 [−2.7, 9.4], p=.27
4B, CVDP cid00340.139.443.1+3.0 [−1.7, 7.8], p=.19
4B, all four task sets (314 tasks)+3.4 [1.2, 5.5], p=.001
9B, VerilogEval v278.778.081.2+2.5 [−0.2, 5.3], p=.064
9B, RTLLM v274.674.377.4+2.8 [−1.6, 7.4], p=.21
9B, ChipBench-SC35.034.037.1+2.1 [−3.6, 7.9], p=.44
9B, CVDP cid00347.346.851.0+3.7 [−0.7, 8.2], p=.086
9B, all four task sets (314 tasks)+2.8 [0.8, 4.8], p=.004

X-GRPO improves pass@1 on every task set at both model sizes, and the pooled paired test over all 314 tasks confirms the gain (p=.001 for 4B, p=.004 for 9B).


Q2, W1 Do the Table 4 run and the 4B headline share checkpoints?

No. Due to GPU-hour limits, the optimizer ablation in Fig. 7(a) and Table 4 was trained for 100 Stage-2 steps instead of 200. Its 72.1 is therefore not comparable to the fully trained 75.0. We have clarified this in the paper, and Table R2 (Appendix B.1) now lists the details of every checkpoint.


Q3, W2 SC@9 paired tests and the 4B counterpart

We compare ChipMATE with a self-consistency baseline, in which the Verilog agent samples nine times and takes a majority vote (SC@9), at both the 9B and 4B sizes. ChipMATE beats SC@9 on every task set, and the paired test over all tasks gives p=.001 at both sizes.

Model, task set (number of tasks)Verilog agent alone, 9-sample majority vote (SC@9), pass@1 (%)ChipMATE (Verilog + Python agents), pass@1 (%)ChipMATE gain, pp [95% CI]Paired p-value
9B, VerilogEval v2 (156)77.480.1+2.7 [−0.6, 6.0].11
9B, RTLLM v2 (50)73.875.8+2.0 [−3.5, 7.5].47
9B, ChipBench-SC (30)32.736.7+4.0 [−2.3, 10.3].23
9B, CVDP cid003 (78)44.250.1+5.9 [0.6, 11.3].03
9B, CVDP cid004 (52)42.746.9+4.2 [−2.0, 10.5].19
9B, all five task sets (366)61.264.8+3.6 [1.4, 5.8].001
4B, VerilogEval v2 (156)70.375.0+4.7 [1.0, 8.4].013
4B, RTLLM v2 (50)71.474.6+3.2 [−2.6, 9.0].28
4B, ChipBench-SC (30)30.033.3+3.3 [−3.5, 10.2].35
4B, CVDP cid003 (78)38.543.0+4.5 [−1.2, 10.2].12
4B, all four task sets (314)58.763.0+4.3 [1.7, 6.9].001

Q4 Pending cells in Sections 3.3 and 3.4

We re-ran the tests and completed the author-side checks for all ‡ entries in Tables 1–2. The revised paper reports the final values.

Dear Reviewers,

As the discussion period is nearing its end, we wanted to briefly follow up on our response to your review. We would greatly appreciate any feedback you may have, particularly on whether our response addresses your main concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.

Best regards,

The Authors

Thank you for the thorough rebuttal. Most of my points were answered with new experiments and a revised PDF. The numbers in the reply match the revised tables exactly. I appreciate the effort.

Resolved:

  • Joint RL is now compared against the Stage-1 agents in the same workflow across task sets and seeds. X-GRPO is ahead in every cell, and the pooled paired tests are significant at both 4B and 9B.
  • Table 4 used 100 Stage-2 steps, so the results I compared came from runs that weren't comparable. That's on me.
  • SC@9 now has a paired test and a 4B counterpart.
  • X-GRPO beats AT-GRPO in every seed comparison with the same reward and data.
  • The ‡ cells are finalized. The OpenReview title and abstract now match the PDF.

Still open:

  • By task set, the joint RL gain is significant only for 4B on VerilogEval.
  • The lead over SC@9 is significant only for 4B on VerilogEval and 9B on CVDP cid003. For 9B on VerilogEval, the gain is positive but not significant.
  • SC@9 uses the Stage-1 Verilog agent. So the gap mixes the Python agent, the correction workflow, and joint RL. With the same external model in both roles, the workflow alone gives a small gain over SC@9 on VerilogEval.

My main reservation is novelty. Checking Verilog against a Python reference model is already in AutoVeriFix, and X-GRPO is a modest change to AT-GRPO's credit assignment. The rebuttal takes a position on this but adds no new evidence. I'm not sure the idea is new enough.

The supported contribution is a well-engineered RTL pipeline with small but consistent gains over sampling baselines. I am keeping my score.

Reviewer_ydsXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChipMATE, a multi-agent framework for RTL generation that combines a Verilog generation agent with a separately trained Python reference-model agent. The two agents independently generate implementations from the same specification and iteratively check their outputs through cross-language simulation. To reduce error propagation, the authors introduce a backtracking mechanism that retains revisions only when inter-agent agreement improves. Experiments are conducted on VerilogEval v2, RTLLM v2, ChipBench-SC, and CVDP benchmarks. The reported results show improvements over several specialized RTL generation baselines, with the 9B-per-agent system achieving 80.1% pass@1 on VerilogEval v2. Additional ablations examine the effects of backtracking, joint training, and reference-model data construction.

Critical strengths
  1. I appreciate that the paper focuses on RTL generation without requiring a golden testbench or external LLM APIs at inference time. This is a realistic constraint in hardware development, and addressing it makes the overall framework potentially useful beyond a purely academic setting.

  2. Having separate Verilog and Python agents is a reasonable design choice. The two implementations provide different views of the same specification, while the mismatch feedback gives each agent an opportunity to identify and correct errors. Although this idea is not entirely new, I think it fits the application well.

  3. Encouraging end-to-end results. The performance improvements are worth acknowledging. On VerilogEval v2, the 9B system reaches 80.1% pass@1, compared with 75.3% for its Verilog-only counterpart. The framework also improves performance across several other benchmarks, suggesting that the interaction between the agents provides some practical benefit.

Critical weaknesses
  1. My biggest concern is that the central ideas are fairly close to existing work on agent-based RTL verification, iterative code repair, and multi-agent reinforcement learning. The authors acknowledge that Python-assisted checking has been explored in AutoVeriFix, while X-GRPO builds on AT-GRPO. Averaging rewards over cross-agent candidate pairs seems like a reasonable extension, but I am not fully convinced that it represents a sufficiently substantial methodological advance for ICLR.

  2. It is difficult to tell where most of the improvement comes from. Looking at Figure 5, supervised fine-tuning already produces a large performance gain, while the additional improvement from joint multi-agent RL is considerably smaller. Similarly, Figure 7(a) shows X-GRPO at 72.1%, compared with 70.4% for AT-GRPO. These results are positive, but they leave me wondering whether the main contribution is a new learning algorithm or the effective combination of existing training and inference techniques. A clearer separation of these effects would make the contribution more convincing.

  3. Another issue is that one ChipMATE output can involve up to 18 model completions and nine pair simulations, whereas several baselines in Table 1 use single-turn generation. The authors do provide self-consistency and same-framework controls, which I appreciate, but these do not completely account for differences in model size, training cost, and inference computation. I would like to see stronger compute-matched comparisons before drawing broader conclusions about the framework's efficiency.

  4. The reliability analysis in Section 3.7 raises an important concern. On ChipBench-SC, agreement precision drops to 62.0%, with false consensus occurring in 20.7% of runs. This suggests that the agents can reinforce the same misunderstanding rather than correct each other. The fresh-stimulus filtering experiment helps, but it also reduces automatic acceptance coverage to 37% on this benchmark. I think this limitation deserves more attention, given that cross-verification is central to the proposed method.

  5. Although the evaluation includes several benchmarks, most experiments are still based on public RTL tasks and two Qwen3.5 model sizes. The reference-model conversion also targets a restricted subset of Verilog behavior, and the category-specific augmentation was motivated by observed benchmark failures. The authors provide an overlap audit, but 33 of the 156 VerilogEval tasks are still flagged for potential overlap. The clean-subset result is encouraging; nevertheless, additional evidence on genuinely unseen or more complex hardware designs would better support the claimed deployment motivation.

Questions for authors
  1. What is the main methodological advantage of X-GRPO over existing multi-agent RL methods? In Figure 7(a), X-GRPO improves over AT-GRPO by 1.7 percentage points. Could the authors clarify which part of the proposed credit assignment is responsible for this improvement? In particular, how much of the gain remains when both methods are evaluated under the same pair-simulation budget?

  2. How much improvement remains under a strictly matched inference budget? The current experiments show that the multi-agent workflow outperforms single-agent generation, but the number of model completions differs substantially. It would be helpful to compare ChipMATE against the trained Verilog agent with comparable total inference computation. Would the advantage remain similar?

  3. How often does higher inter-agent agreement actually lead to better RTL correctness? Section 3.7 reports a relatively high false-consensus rate on ChipBench-SC. Could the authors provide a more direct analysis of the relationship between changes in agreement scores and actual correctness across refinement turns? This would help determine when the backtracking mechanism is genuinely beneficial.

3 comments

Response, part 1 of 2

Thank you for the feedback and suggestions. New results are marked in blue in the revised PDF.


Q1 Why does X-GRPO outperform AT-GRPO?

X-GRPO extracts a richer training signal from the same sampling budget. Because it checks each candidate against several independent partners rather than one, it forms more Verilog–Python pairs and exposes more errors per training step. Averaging the reward across partners also improves credit assignment: a correct candidate is less likely to be penalized just because its partner is faulty.


Q2, W3 Inference budget and the single-agent baseline

To compare under a strictly matched inference budget, we evaluate two Verilog-only baselines with the same budget as ChipMATE:

  1. Verilog (SC@9), which generates nine samples and selects the output by majority vote.
  2. Verilog–Verilog, which replaces ChipMATE's Python agent with a second Verilog agent, so that the two agents cross-verify and refine each other's outputs.

ChipMATE outperforms both baselines at both model sizes.

Method, pass@1 (%)Inference budget4B, VerilogEval v29B, VerilogEval v24B, RTLLM v29B, RTLLM v2
Verilog (single sample)Low67.475.368.071.9
Verilog (SC@9)Matched70.377.471.073.8
Verilog–VerilogMatched71.077.971.674.0
ChipMATE (Verilog–Python)Matched75.080.174.675.8

Response, part 2 of 2


Q3 How often does higher agreement lead to better RTL correctness?

To test whether stronger agreement between the agents corresponds to better RTL, we compare consecutive turns of ChipMATE's generation process. When the RTL–Python match rate rises, RTL correctness is much more likely to improve than to worsen, which supports the effectiveness of cross-verification. More often, RTL correctness stays unchanged as agreement rises. These turns likely correct the Python code rather than the RTL, which is also what cross-verification is designed to do.

Benchmark, change in RTL–Python match rate from one turn to the nextRTL correctness improvedRTL correctness unchangedRTL correctness worsened
VerilogEval v2, match rate increased21%71%8%
VerilogEval v2, match rate decreased7%32%61%
ChipBench-SC, match rate increased28%54%18%
ChipBench-SC, match rate decreased14%43%43%

W1 The contribution of the paper

Beyond the technical contributions, the main purpose of the paper is to remind the community of three things:

  1. Stop using golden information for RTL generation tasks. Golden Verilog code and golden verification environments are not available in industrial deployment. Our workflow uses no golden testbench, output, or implementation to select, repair, or stop. (Only the port list in the prompt is extracted from the reference design.)
  2. Self-trained models and multi-agent flows are more practical for industry than API-based flows. Chip vendors often cannot send RTL to cloud APIs, but they can train local models. Even with small models and limited public data, our two agents beat CodeV-R1 on all five task sets. Vendors that train larger models on internal data should do much better.
  3. Verification is even more important than RTL generation. The Verilog agent improves only through the mismatches reported against the Python reference model, so the accuracy of the reference model bounds the repair loop. We hope to inspire more work on verification: reference models, stimulus generation, formal methods, and Lean-based methods.

W2 Where the improvement comes from

We agree that SFT brings the largest gain: our base models lack domain knowledge of RTL generation, and that is exactly what SFT teaches. However, once the models have learned this knowledge, it is hard for them to keep improving and to generalize to different RTL modules without our proposed training pipeline. The table lists the VerilogEval v2 gain of each stage (Fig. 5; the last two rows are alternatives to each other):

Stage added to the pipeline4B: VerilogEval v2 pass@1 gain (pp)9B: VerilogEval v2 pass@1 gain (pp)
SFT of the Verilog agent+19.2+22.0
Single-agent GRPO (Stage 1)+6.5+4.8
Two-agent workflow with our Python agent+4.4+3.4
Joint RL with X-GRPO (3-seed mean)+3.6+2.5
Joint RL with AT-GRPO instead (3-seed mean)−1.4−0.7

W4 Agents can make the same mistake

Good point. This is true, and it is also why chip design is hard for both humans and AI: there is no golden result to refer to. We cannot guarantee that the two agents never make the same mistake. Verilog–Python cross-correction is the solution we propose, and it reduces common errors in three ways:

  • Python and Verilog follow fundamentally different programming models (sequential code vs. concurrent hardware), so a language-specific slip, such as a one-cycle timing or bit-width error, shows up as a mismatch between the two simulations.
  • The two agents never see each other's code, so neither can copy the other's mistake.
  • We first strengthen each agent individually with SFT and RL.

Although the agents still sometimes make the same mistake, cross-verification does improve results. The 9B two-agent system beats its own Verilog agent on all five task sets, by 3.9–10.8 pp. It also beats a nine-sample majority vote of that Verilog agent on all five task sets, by 3.6 pp on average (p=.001). We believe this is a promising direction for chip design without golden testbenches, and we expect more work to build on it.

Dear Reviewers,

As the discussion period is nearing its end, we wanted to briefly follow up on our response to your review. We would greatly appreciate any feedback you may have, particularly on whether our response addresses your main concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_h28yOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ChipMATE, an offline self-trained multi-agent framework for Register-Transfer Level (RTL) code generation designed to operate without external LLM APIs or golden testbenches during inference. Inspired by hardware design and verification division of labor, ChipMATE pairs a Verilog agent with a separately parameterized Python reference-model agent that iteratively cross-check execution traces over 1,000 random stimuli and repair their implementations using natural-language mismatch diagnostics. To prevent error cascades across turns, a joint backtracking mechanism retains updated code pairs only when measured trace agreement strictly improves. The agents are trained via a two-stage pipeline—combining role-specific SFT and GRPO with joint multi-agent reinforcement learning (X-GRPO) using partner-averaged candidate credit—supported by a hybrid data-generation framework that yields 64.4K simulation-verified Python reference models. Evaluated on Qwen3.5 4B and 9B backbones, ChipMATE achieves 75.0% and 80.1% pass@1 on VerilogEval v2, substantially outperforming single-turn open-source baselines.

Critical strengths
  • ChipMATE establishes a practical, fully offline cross-verification workflow that eliminates reliance on LLM APIs and golden testbenches at inference, using joint backtracking to prevent multi-turn performance degradation (improving 9B VerilogEval v2 pass@1 from 60.1% without backtracking to 78.7%).
  • The paper introduces X-GRPO for joint multi-agent reinforcement learning with partner-averaged candidate credit, effectively training the two code-blind roles to collaborate and repair faults from waveform mismatch diagnostics.
  • In Section 2.3, the authors develop a scalable hybrid data-generation pipeline combining LLM distillation, AST-level IR conversion, and targeted category augmentation to construct 64.4K cycle-accurate Python reference models verified by finite differential simulation.
  • Empirical evaluations across five benchmarks demonstrate strong system-level gains (reaching 80.1% pass@1 on VerilogEval v2 and 75.8% on RTLLM v2 with 9B parameters per agent), supported by a rigorous AST-level decontamination audit (Appendix C.5) and synthesis PPA analysis (Appendix C.8).
Critical weaknesses
  • As analyzed in Section 3.7 and Table 6, cross-agent trace agreement suffers from correlated errors when both agents misinterpret the natural-language specification. On harder benchmarks like ChipBench-SC, agreement precision drops to 62.0% and false consensus (agreed but incorrect) reaches 20.7% of all runs, while applying fresh-suite abstention lowers auto-accept coverage to just 37%.
  • The deterministic IR-based conversion pipeline (which supplies 36K of the 64.4K training samples) is restricted to a single-clock, two-state Verilog subset and relies on post-hoc synthetic rationales ("unreal CoT"). This limits the Python agent's ability to model multi-clock domains, latches, asynchronous resets, or X/Z propagation behaviors common in production RTL.
  • In Section 3.7 and Appendix C.5, problem-bootstrap confidence intervals indicate that while the 9B pair's gain over CodeV-R1 is statistically significant on VerilogEval v2 (p=0.003p = 0.003), the improvements on RTLLM v2 and ChipBench-SC are not individually statistically significant, and the 3.4K category-specific data augmentation was directly motivated by development-set failure modes.
Questions for authors

Since Stage-1 GRPO and Stage-2 X-GRPO rely on the same local correctness reward (RlocalR_{\text{local}}) and X-GRPO yields only a +1.4 pp gain at 9B (Figure 5), have the authors evaluated applying X-GRPO directly after SFT under a matched total RL step and simulation budget to verify whether Stage-1 single-agent RL is truly necessary?

3 comments

Thank you for the feedback and suggestions. New results are marked in blue in the revised PDF.


Q1 Direct X-GRPO after SFT

We compare our two-stage pipeline (GRPO followed by X-GRPO) with applying X-GRPO directly after SFT. Our pipeline outperforms direct X-GRPO even when the latter gets twice the RL steps, which shows that the single-agent RL stage is necessary. We attribute this to the instability of multi-agent RL: training is sensitive to the starting capability of each agent, so each agent must first be well trained on its own.

Training after SFT (9B), pass@1 (%, 3-seed mean)Training stepsVerilogEval v2RTLLM v2ChipBench-SCCVDP cid003
GRPO, then X-GRPO (ours)300 + 20081.277.437.151.0
X-GRPO directly50077.073.633.745.4
X-GRPO directly, more steps1,05078.074.034.646.4

W1 Both agents can make the same mistake

Good point. This is true, and it is also why chip design is hard for both humans and AI: there is no golden result to refer to. We cannot guarantee that the two agents never make the same mistake. Verilog–Python cross-correction is the solution we propose, and it reduces common errors in three ways:

  • Python and Verilog follow fundamentally different programming models (sequential code vs. concurrent hardware), so a language-specific slip, such as a one-cycle timing or bit-width error, shows up as a mismatch between the two simulations.
  • The two agents never see each other's code, so neither can copy the other's mistake.
  • We first strengthen each agent individually with SFT and RL.

Although the agents still sometimes make the same mistake, cross-verification does improve results. The 9B two-agent system beats its own Verilog agent on all five task sets, by 3.9–10.8 pp. It also beats a nine-sample majority vote of that Verilog agent on all five task sets, by 3.6 pp on average (p=.001). We believe this is a promising direction for chip design without golden testbenches, and we expect more work to build on it.


W2 IR-based conversion is restricted to single-clock, two-state Verilog

We agree. We support only single-clock, two-state logic because most current benchmarks consist of such designs. The converter rejects latches, multi-clock logic, and delays, and it does not model X/Z values (Section 2.3, Appendix A). With more engineering effort, the converter can be extended to more general cases, such as multi-clock-domain modules, which are essential in SoC design, and the X/Z propagation used in reset and initialization.

Thank you for addressing my comments, especially the direct X-GRPO ablation (Q1) and the multi-seed, budget-matched comparisons. After reading the other reviews and responses as well, I would like to maintain my current score and stay with a positive recommendation.

Dear Reviewer h28y, Thank you for your follow-up and positive assessment of our work. We greatly appreciate your time and thoughtful feedback throughout the review process. Please let us know if there is anything else we can clarify. Best regards, The Authors

Reviewer_91QzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ChipMATE pairs a Verilog agent with a separate Python reference-model agent for RTL generation. With no golden testbench, the agents cross-check their outputs on random stimuli; joint backtracking keeps a new pair only when agreement improves. Each agent first gets SFT and RL ; X-GRPO then trains both jointly with partner-averaged rewards. A hybrid pipeline supplies 64.4K reference-model samples; the 4B and 9B pairs reach 75.0% and 80.1% VerilogEval v2 pass@1.

Critical strengths
  • The 9B pair beats CodeV-R1 by 6.7–11.3 pp of pass@1 on five task sets. On VerilogEval, the gain is significant and holds on the 123 tasks without detected overlap (79.3 vs. 67.6). Backend, role-swap, and false-consensus analyses make these claims credible.
  • A useful corpus and a cheap converter. The IR converter adds about 36K simulation-checked samples in under two hours, versus over $2,000 for API distillation. Without the IR subset, the pass@1 of the 4B Python agent after SFT falls from 71.2 to 61.1. This is below the result of a size-matched random subset. A release of the corpus would fill a real data gap.
Critical weaknesses
  • Table 1 marks entries "‡: reconciliation pending", and Appendix A says "Some reported aggregates still require count/protocol reconciliation". Four ‡ entries in Table 1 are ChipMATE results, for example ChipMATE-Verilog-9B on RTLLM v2 (71.9) and CVDP cid003 (39.3). These two entries set both ends of the 3.9–10.8 pp gain of the 9B workflow over its Verilog agent. I cannot tell which numbers are final.
  • Thin evidence for the trained multi-agent parts. AutoVeriFix already checks Verilog against Python reference models, so the novelty lies in the trained Python agent and X-GRPO. The 9B pair beats its own Verilog agent with SC@9 by 2.7 pp, on VerilogEval only, without a paired test. Multi-agent RL adds 1.4 pp at 9B , and X-GRPO beats AT-GRPO by a mean 1.8 pp over three seeds, with unequal pair-simulation budgets.
Questions for authors
  • Does the golden RTL supply the pre-declared state variables of the benchmark Python skeletons?
  • Which model size does Figure 7(a) use? X-GRPO reaches 72.1 there, but the 4B pair without multi-agent RL reaches 71.8 (Fig. 5)
2 comments

Thank you for the feedback and suggestions. New results are marked in blue in the revised PDF.


W1 Final values and pending entries

We re-ran the tests and completed the author-side checks for all ‡ entries in Tables 1–2. The final values are close to the original ones, and the revised paper reports them.


W2 The improvement from X-GRPO

To isolate the gain from X-GRPO, we compare three versions of the same two-agent workflow: without multi-agent RL (No MA-RL), after AT-GRPO, and after X-GRPO. All three start from the same Stage-1 agents. AT-GRPO and X-GRPO use the same reward and data, three candidates per agent per turn, and three seeds.

Pass@1 (%)Stage-1 agents, no multi-agent RL (No MA-RL)+ AT-GRPO (3-seed mean)+ X-GRPO (3-seed mean)X-GRPO − AT-GRPO per seed, pp
4B, VerilogEval v271.870.475.45.2 / 4.6 / 5.1
4B, RTLLM v272.071.375.23.0 / 4.6 / 4.2
4B, ChipBench-SC30.729.634.06.0 / 2.3 / 5.0
4B, CVDP cid00340.139.443.14.2 / 3.1 / 3.8
9B, VerilogEval v278.778.081.23.3 / 2.7 / 3.7
9B, RTLLM v274.674.377.43.2 / 2.4 / 3.6
9B, ChipBench-SC35.034.037.10.7 / 4.0 / 4.7
9B, CVDP cid00347.346.851.03.5 / 5.1 / 4.0

X-GRPO beats AT-GRPO in all 24 seed comparisons and improves on the No MA-RL agents by 2.1–3.7 pp on every task set.


W2 The improvement from the multi-agent workflow

We compare ChipMATE with a budget-matched single-agent baseline, in which the Verilog agent samples nine times and takes a majority vote (SC@9). ChipMATE beats SC@9 on every task set, and the paired test over all tasks gives p=.001 at both model sizes.

Model, task set (number of tasks)Verilog agent alone, 9-sample majority vote (SC@9), pass@1 (%)ChipMATE (Verilog + Python agents), pass@1 (%)ChipMATE gain, pp [95% CI]Paired p-value
9B, VerilogEval v2 (156)77.480.1+2.7 [−0.6, 6.0].11
9B, RTLLM v2 (50)73.875.8+2.0 [−3.5, 7.5].47
9B, ChipBench-SC (30)32.736.7+4.0 [−2.3, 10.3].23
9B, CVDP cid003 (78)44.250.1+5.9 [0.6, 11.3].03
9B, CVDP cid004 (52)42.746.9+4.2 [−2.0, 10.5].19
9B, all five task sets (366)61.264.8+3.6 [1.4, 5.8].001
4B, VerilogEval v2 (156)70.375.0+4.7 [1.0, 8.4].013
4B, RTLLM v2 (50)71.474.6+3.2 [−2.6, 9.0].28
4B, ChipBench-SC (30)30.033.3+3.3 [−3.5, 10.2].35
4B, CVDP cid003 (78)38.543.0+4.5 [−1.2, 10.2].12
4B, all four task sets (314)58.763.0+4.3 [1.7, 6.9].001

W2 The novelty of the paper

The main purpose of the paper is to remind the community of three things:

  1. Stop using golden information for debugging and self-evolution. Golden Verilog code and golden verification environments are not available in industrial deployment. Our workflow uses no golden testbench, output, or implementation to select, repair, or stop. (Only the port list in the prompt is extracted from the reference design; see Q1.)
  2. Self-trained models and multi-agent flows are more practical for industry than API-based flows. Chip vendors often cannot send RTL to cloud APIs, but they can train local models. Even with small models and limited public data, our two agents beat CodeV-R1 on all five task sets. Vendors that train larger models on internal data should do much better.
  3. Verification is even more important than RTL generation. The Verilog agent improves only through the mismatches reported against the Python reference model, so the accuracy of the reference model bounds the repair loop. We hope to inspire more work on verification: reference models, stimulus generation, formal methods, and Lean-based methods.

Q1 Pre-declared state variables of the Python skeletons

Yes. Our tool extracts the port list from the golden Verilog and passes it to both the Python agent and the Verilog agent. It copies no logic. This matches industry practice, where the specification fixes all I/O before engineers write RTL. With the interface fixed, the LLM spends no effort on aligning I/O and can focus entirely on the logic.


Q2 The model size in Fig. 7 and Fig. 5

Both Fig. 7(a) and Fig. 5 use the 4B model, but with different checkpoints. Due to GPU-hour limits, we trained the Fig. 7(a) ablation for 100 Stage-2 steps, whereas the Fig. 5 model was trained for 200. Its 72.1 is therefore not comparable to the fully trained 75.0. Table R2 of the revised paper now lists the details of every checkpoint.

Dear Reviewers,

As the discussion period is nearing its end, we wanted to briefly follow up on our response to your review. We would greatly appreciate any feedback you may have, particularly on whether our response addresses your main concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.