- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ChipMATE pairs two LLM agents for RTL generation. One writes Verilog, the other writes a Python reference model from the same spec, and neither sees the other's code. They compare simulation traces and keep a revision only when agreement improves. After a separate stage for each, the pair is trained jointly with a multi-agent RL method, X-GRPO. The 9B-per-agent system beats single-sample open RTL models on five benchmarks, significantly on VerilogEval. The system-level result holds up and is carefully measured. What the paper does not show is the title's premise, that joint multi-agent RL is what produces the improvement.
Critical strengths
- The diagnostics are the most informative part of the paper. Without backtracking, mutual correction makes the pair worse than a single agent. The authors also measure where agreement breaks down: false consensus roughly doubles on the harder ChipBench-SC, mostly because both agents misread the spec the same way.
- The role-swap experiment makes the system-level result convincing. Substituting the ChipMATE agents for CodeV-R1 one at a time raises accuracy at each step, and the lead over CodeV-R1 survives on the VerilogEval tasks that pass the overlap audit. Taken on its own terms, this is a solid incremental paper: the supported contribution is small, but the evidence behind it is broad and honestly reported.
Critical weaknesses
- Joint training, the thing the title is about, may do little beyond run-to-run noise. Replacing the separately trained agents with jointly trained ones in the same workflow adds 1.4 points at 9B and 3.2 at 4B. Each comes from a single unpaired run on VerilogEval, and both sit inside the paper's own bootstrap intervals. At 4B, the X-GRPO run in Table 4 lands barely above the workflow without it and nearly three points below the 4B headline, and the paper doesn't explain why. Sixteen cells are also marked as pending reconciliation, including some of the method's own rows that Sections 3.3 and 3.4 build on.
- The method adds one more knob to an existing recipe, which is fine, but then the pair's edge over plain test-time sampling has to carry the paper, and that edge is small. Checking Verilog against a Python reference model isn't new, as the authors themselves say; what's new is giving that role its own fine-tuned model, plus the X-GRPO credit. The Verilog agent alone, with self-consistency at about the same sampling budget, reaches 77.4 versus the pair's 80.1. That gap isn't tested for significance, and there's no 4B counterpart at all.
Questions for authors
- Can you compare the separately trained and jointly trained agents in the identical workflow, paired, on at least two benchmarks and more than one seed? A difference that survives that design is what the title needs.
- Do the Table 4 X-GRPO run and the 4B headline run share checkpoints and settings, or is 75.0 one of several runs?
- Is the 80.1 vs. 77.4 gap significant under a paired test on your existing outputs, and how does the 4B system compare with a similar self-consistency baseline?
- Can you resolve the sixteen pending cells and restate Sections 3.3 and 3.4 from final values?
3 comments
Dear Reviewers,
As the discussion period is nearing its end, we wanted to briefly follow up on our response to your review. We would greatly appreciate any feedback you may have, particularly on whether our response addresses your main concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.
Best regards,
The Authors
Thank you for the thorough rebuttal. Most of my points were answered with new experiments and a revised PDF. The numbers in the reply match the revised tables exactly. I appreciate the effort.
Resolved:
- Joint RL is now compared against the Stage-1 agents in the same workflow across task sets and seeds. X-GRPO is ahead in every cell, and the pooled paired tests are significant at both 4B and 9B.
- Table 4 used 100 Stage-2 steps, so the results I compared came from runs that weren't comparable. That's on me.
- SC@9 now has a paired test and a 4B counterpart.
- X-GRPO beats AT-GRPO in every seed comparison with the same reward and data.
- The ‡ cells are finalized. The OpenReview title and abstract now match the PDF.
Still open:
- By task set, the joint RL gain is significant only for 4B on VerilogEval.
- The lead over SC@9 is significant only for 4B on VerilogEval and 9B on CVDP cid003. For 9B on VerilogEval, the gain is positive but not significant.
- SC@9 uses the Stage-1 Verilog agent. So the gap mixes the Python agent, the correction workflow, and joint RL. With the same external model in both roles, the workflow alone gives a small gain over SC@9 on VerilogEval.
My main reservation is novelty. Checking Verilog against a Python reference model is already in AutoVeriFix, and X-GRPO is a modest change to AT-GRPO's credit assignment. The rebuttal takes a position on this but adds no new evidence. I'm not sure the idea is new enough.
The supported contribution is a well-engineered RTL pipeline with small but consistent gains over sampling baselines. I am keeping my score.
Thank you for the feedback and suggestions. New results are marked in blue in the revised PDF.
Q1, W1Multi-seed, multi-benchmark training comparisonTo isolate the gain from X-GRPO, we compare three versions of the same workflow: the Stage-1 agents without multi-agent RL, the same agents after AT-GRPO, and after X-GRPO. AT-GRPO and X-GRPO use the same reward and data, three candidates per agent per turn, and three seeds.
X-GRPO improves pass@1 on every task set at both model sizes, and the pooled paired test over all 314 tasks confirms the gain (p=.001 for 4B, p=.004 for 9B).
Q2, W1Do the Table 4 run and the 4B headline share checkpoints?No. Due to GPU-hour limits, the optimizer ablation in Fig. 7(a) and Table 4 was trained for 100 Stage-2 steps instead of 200. Its 72.1 is therefore not comparable to the fully trained 75.0. We have clarified this in the paper, and Table R2 (Appendix B.1) now lists the details of every checkpoint.
Q3, W2SC@9 paired tests and the 4B counterpartWe compare ChipMATE with a self-consistency baseline, in which the Verilog agent samples nine times and takes a majority vote (SC@9), at both the 9B and 4B sizes. ChipMATE beats SC@9 on every task set, and the paired test over all tasks gives p=.001 at both sizes.
Q4Pending cells in Sections 3.3 and 3.4We re-ran the tests and completed the author-side checks for all ‡ entries in Tables 1–2. The revised paper reports the final values.