- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VeriRefine is an agentic flow for spec-to-RTL generation. An LLM first converts the natural-language specification into a JSON intermediate representation built from per-signal Abstract Signal Transition Functions. Each ASTF records the signal declaration, hardware type, ordered guarded commands, and verbatim citations from the specification. The IR is checked by schema validation and a set of programmatic audits before being translated into Verilog. An LLM-generated internal testbench is then used for up to two debug iterations. A failure analyzer decides whether to revise the IR or repair the RTL.
Critical strengths
-
Strong RTLLM results without reference RTL. RTLLM does not provide a reference implementation to the ITB, so the verification signal is derived from the specification. VeriRefine still passes 48/50 designs compared with 45/50 for the same-model VerilogCoder reproduction, while using only 13.2% of the tokens. This is the strongest part of the paper.
-
The intermediate representation is useful and inspectable. VeriRefine-Direct already improves over direct generation on both models and both benchmarks. For example, Sonnet 4.6 improves by 20 points on RTLLM before simulation-guided debugging. The ASTF representation also gives designers something concrete to inspect before RTL is generated.
Critical weaknesses
The paper has good end-to-end results, but I have two main concerns. First, it is not clear which part of the pipeline is responsible for the gain. Second, the VerilogEval result is not directly comparable with the other settings.
-
The VerilogEval result uses reference RTL during verification. In the reference-assisted setting, the ITB compares DUT outputs with the provided reference module. This makes the 143-to-153 improvement from debugging hard to interpret. It also makes the comparison with VerilogCoder unclear unless the same reference information is available to that baseline.
-
Spec grounding appears mostly lexical. The grounding check verifies that a cited text span appears in the specification, but it does not seem to check whether the citation supports the generated action. In Appendix A, clk_div1 uses a thresholded assignment while the cited sentence says the divider is “toggled”. The IR still passes with zero warnings.
-
The main components are not separated. The paper does not isolate ASTF construction, auditing, and diagnostic routing. It also lacks a code-only repair baseline and a compute-matched direct-generation baseline.
-
Related verification work is missing from the discussion. Structured IRs already appear in AoT, QiMeng-CRUX, and Spec2RTL-Agent. On the verification side, testbench self-correction [1] and failure attribution between RTL and verification infrastructure [2] are closely related. This is especially relevant because Appendix A contains an incorrect expected period in the generated verification plan, while the final run still passes.
-
The evaluation is still limited. The reported results use one run per task. VerilogEval-Human v2 is also close to saturation for the strongest methods. MAGE is cited but not compared. Results on a harder benchmark such as CVDP [3] would make the evaluation more convincing.
[1] R. Qiu et al. CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design. arXiv:2411.08510, 2024.
[2] Y. Zhao et al. PRO-V: An Efficient Program Generation Multi-Agent System for Automatic RTL Verification. arXiv:2506.12200, 2025.
[3] N. Pinckney, C. Deng, C. Ho, Y. Tsai, M. Liu, W. Zhou, B. Khailany, and H. Ren. Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification. arXiv:2506.14074, 2025.
Questions for authors
Q1. Which ITB configuration was used for the VerilogEval numbers in Table 2? How many of the 153 passing designs use comparison with the reference RTL? What is the result if the ITB uses only the specification as the oracle? Does the VerilogCoder reproduction have access to the reference module or benchmark testbench?
Q2. How often does each audit layer trigger, and what kinds of errors does it fix? How does the grounding check handle cases where the cited text appears in the specification but does not support the generated action, such as the clk_div1 example in Appendix A?
Q3. How does VeriRefine-Direct compare with direct generation under a similar token budget? For example, what happens with self-refine or best-of-n plus compile checks? Also, how does diagnostic routing compare with repairing the RTL directly using the same ITB feedback?
Q4. How accurate are the ITB and the failure analyzer? How often does the ITB reject correct RTL? How often does the analyzer send a failure to the wrong level? What happens when the ITB itself is wrong?
Q5. Are the results stable across runs? Please report variance over several runs, at least for the full VeriRefine flow.