- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether passing an RTL verifier really means that an LLM has produced a correct design. It compares generated testbenches, differential simulation, and bounded model checking against a separate formal-equivalence oracle. The experiments cover candidate selection, iterative repair, and RL training. The main finding is that a higher test pass rate does not always translate into more correct designs. The paper also studies how much additional verification helps and when the gains start to level off.
Critical strengths
-
The paper shows why a better RTL generator still needs a reliable checker. I liked the comparison on the same candidate pool. Table 1 (p. 6) shows that a proved solution exists for 82% of tasks, but the generated testbenches select one for only 61–66%. BMC and differential simulation do considerably better. This makes the problem concrete: the model may already produce a correct answer, yet the system fails to identify it. I find this useful for understanding where improvements in generation can get lost.
-
The verification-budget experiments offer practical guidance. Section 5.3 (p. 6) shows both the benefit of stronger checking and where that benefit starts to flatten. This helps readers think about how much verification effort is worthwhile, rather than simply choosing the strongest available checker. I also appreciated that Section 3.1 (p. 3) distinguishes an actual counterexample from an unfinished proof, since these outcomes support different conclusions.
Critical weaknesses
-
The RL findings are promising, but their consistency is still unclear. Section 5.5 (p. 8) shows that testbench reward improves while the oracle proof rate changes little. However, the early and late windows were chosen after looking at the trajectory. Section 7 (p. 9) also notes that there is only one run per model configuration, with different rollout counts across configurations. I would be more convinced by another seed or a matched run using a stronger reward verifier. The authors acknowledge these limits, but the RL evidence remains less convincing than the candidate-selection results.
-
Missing experimental details make the results harder to reproduce. Figure 3’s caption (p. 8) says that the run tag does not establish the exact model checkpoint, while Appendix L (p. 26) says that historical tool versions were not preserved. Appendix P (p. 30) also notes that some cost measurements came from different machines or load conditions. Since the paper’s contribution depends heavily on these measurements, I would feel more comfortable with clearly identified checkpoints, a fixed tool environment, and cost comparisons collected under the same conditions.
Questions for authors
My background is in RL training, and I am relatively new to RTL/EDA. I understand the evaluation setup in Section 4.1 (p. 4), but I would appreciate a concrete example showing how it fits into an actual design workflow. In particular, it would help to explain where the golden reference comes from, why a new implementation is needed when that reference already exists, and what the user hopes to gain. Connecting this example to the results would also help me understand what becomes easier or more reliable once the verification problem is addressed.
I would be interested in your perspective on how this study fits alongside recent methods such as CodeV-R1, ChipSeek, and STG [1–3]. A short discussion of the main bottlenecks in these workflows would help me place the contribution. For someone building a system today, I would especially like to understand whether these findings should change how they construct training rewards, select candidate designs, or provide tool feedback. Since Section 4.4 (pp. 4–5) evaluates repair and RL separately, it would also be useful to discuss which lessons might carry over to multi-turn tool-using agents, while keeping those expectations separate from the findings directly supported here.
The disclosures in Figure 3 and Appendix L leave me unsure how closely someone could reproduce the reported runs. It would help to clarify which exact model checkpoints and tool versions can still be recovered. If some information is no longer available, a confirmation run in a pinned environment would make the main findings easier to trust and build on.
[1] Yaoyu Zhu et al. QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation. NeurIPS 2025.
[2] Zhirong Chen et al. ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning. arXiv:2507.04736, 2025; revised 2026.
[3] En-Ming Huang et al. Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation. arXiv:2606.12983, 2026.
3 comments
How are the model, evaluation settings, and cost claims clarified?
We agree that reproducibility requires clearly identified model checkpoints and software environments. The statements in the original Figure 3 caption and Appendix L were misleading and have been corrected. The run-tag caveat meant that a label alone does not distinguish repeated runs using the same model and verifier under different configurations; it did not indicate uncertainty about the model checkpoint. Similarly, the statement about historical tool versions should not have implied that the software environment was undocumented. All experimental data, results, model checkpoints, and associated configuration and environment records are available, and we will release them publicly.
The main RL evidence now comes from six Qwen3-32B non-thinking runs: three rewards, two seeds per reward, and 200 updates under a shared training recipe. This directly addresses the request for repeated seeds and stronger-reward controls. The RL configuration appendix specifies sampling parameters, optimizer settings, rollout counts, validation frequency, and reward definitions. The earlier 4B/8B runs remain separately identified supporting experiments with their own configurations.
All DeepSeek/L5 training and validation outputs receive the same offline formal evaluation, with a 600-second limit per check and common memory and concurrency settings specified in the appendix. The shared protocol fixes the checker code, configuration, and tool identities, in addition to the resource limits. Oracle-control curves use their online judgments, as stated explicitly. This makes the evaluation conditions and the interpretation of each comparison clear.
The cost concern in the original Appendix P is separate: the revised paper distinguishes checker construction, candidate checking, and formal evaluation. It reports CPU-based verification costs while recognizing that measurements across different hosts or concurrency settings do not establish a controlled end-to-end efficiency comparison. The new reward controls support conclusions about learning and reward reliability; they do not establish an equal-cost advantage.
Where does the golden reference come from, and why generate a new implementation?
Consider training on an existing trusted RAM and its specification. The model receives only the specification and proposes an implementation; the reference provides verification feedback. Repeating this over many known blocks teaches RTL generation. The purpose is to learn a generator, rather than to reconstruct one known block for its own sake. The new Oracle-reward runs provide a concrete example of learning under this reference-available setup.
The same reference can also define the behavior of alternative implementations, for example when exploring area or timing trade-offs. We measure functional reliability, not PPA improvements. The RAM witness illustrates the checking problem: a generated design can pass clock-edge tests yet violate required behavior between edges. Selection or training can then reward the wrong behavior despite having a trusted target available.
The practical lesson is to distinguish whether a proved solution exists, whether the checker accepts it, and whether selection recovers it. A test pass rate alone does not answer these questions.
How should these findings inform related methods and tool-using agents?
Our audit complements the methods highlighted by the reviewer. CodeV-R1 motivates validating reference-based verification used for data construction and RL. ChipSeek motivates auditing functional acceptance before interpreting downstream hardware-efficiency rewards. STG motivates extending evaluation to structured test generation rather than treating generic benches as representative of all verifiers.
For system builders, the supported recommendations are to measure selection losses against available proved candidates, audit rewarded training outputs independently, distinguish counterexamples from unresolved proofs, and report checking effort and coverage. The new reward controls strengthen this message: learning under a reward does not by itself establish the reliability of the outputs it accepts.
Independent outcome evaluation is also relevant to tool-using agents. Our repair experiments, however, use assigned feedback and restricted history; unrestricted tool choice, self-revised tests, and autonomous budget allocation are extensions, not measured results here.
Follow-up on Author Rebuttal
Dear Reviewer,
A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.
Your feedback would be appreciated before the discussion period closes.
Thank you!
We thank the reviewer for requesting stronger RL controls, clearer evaluation settings, and a concrete account of the intended workflow. The revised manuscript makes the six-run Qwen3-32B study the main RL evidence.
Are the RL results consistent across seeds and stronger rewards?
The revised manuscript now centers the requested repeated seeds and stronger-reward controls. Section 5.5 includes all six Qwen3-32B runs, a three-reward validation figure, and this late held-out comparison. Rates average checkpoints 185, 190, 195, and 200, each with 500 nonpadding outputs:
Oracle rewards support held-out learning in both seeds: proof success rises from 13.80% to 25.65% and from 12.80% to 21.95%, with gains of +11.85 pp [4.35, 19.45] and +9.15 pp [1.90, 16.50]. This empirical learning result is distinct from zero reward disagreement: Oracle-control reward and evaluation use the same online judgment, so zero is an identity.
L5 more closely tracks formal outcomes on its learned policies' outputs. Its late rewarded-counterexample rates are much lower than DeepSeek's, and its signed gaps are −2.10/−3.60 pp versus +16.80/+17.05 pp. Negative gaps reflect rejected proofs as well as residual accepted counterexamples; L5 is not perfect. This is consistent with the earlier selection and effort results. L5's late proof advantage over DeepSeek has task intervals including zero in each seed, so we do not infer a universal final proof-rate ranking.
All six runs improve proof success from their own baseline. DeepSeek training disagreement grows in both seeds; held-out gap growth is clear only in seed 1. Proxy-policy proof labels use a common offline formal protocol, while Oracle-control curves use their online judgments. The runs share an update recipe, not equal total verification compute.
The main figures show both DeepSeek training seeds and all three rewards in validation. Full three-reward training curves and paired-effect tables are in the appendix. The earlier 4B/8B experiments are now supplementary historical evidence.