acceptodds
Under review as a conference paper at ICLR 2027

VeriRefine: An LLM-Enabled Abstraction Refinement Flow for RTL Design Generation

Abstract

Large language models (LLMs) can generate register-transfer-level (RTL) designs from natural-language specifications, yet many generation errors stem from misinterpreting the specification. Intermediate representations make specification interpretations explicit, but these interpretations still require checks for missing requirements and contradictions with the source specification. This paper presents VeriRefine, an LLM-enabled abstraction-refinement framework that refines informal specifications into schema-constrained Abstract Signal Transition Functions (ASTFs), capturing per-signal logic, clocking, reset, and transition semantics before RTL generation. The resulting ASTF-based representation is audited for specification grounding, coverage, consistency, state transition integrity, and RTL design-rule compliance. Post-generation simulation failures are classified to guide ASTF refinement or RTL repair. Using Claude Sonnet 4.6, VeriRefine achieves 96.0% functional correctness on RTLLM v2.0 and 98.1% on VerilogEval-Human v2, exceeding direct generation by 32.0 and 12.8 percentage points, respectively. Iterative debugging improves upon the audited generation flow on benchmarks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language models, RTL code generation, abstraction refinement, intermediate representation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LQ9LOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

VeriRefine is an agentic flow for spec-to-RTL generation. An LLM first converts the natural-language specification into a JSON intermediate representation built from per-signal Abstract Signal Transition Functions. Each ASTF records the signal declaration, hardware type, ordered guarded commands, and verbatim citations from the specification. The IR is checked by schema validation and a set of programmatic audits before being translated into Verilog. An LLM-generated internal testbench is then used for up to two debug iterations. A failure analyzer decides whether to revise the IR or repair the RTL.

Critical strengths
  • Strong RTLLM results without reference RTL. RTLLM does not provide a reference implementation to the ITB, so the verification signal is derived from the specification. VeriRefine still passes 48/50 designs compared with 45/50 for the same-model VerilogCoder reproduction, while using only 13.2% of the tokens. This is the strongest part of the paper.

  • The intermediate representation is useful and inspectable. VeriRefine-Direct already improves over direct generation on both models and both benchmarks. For example, Sonnet 4.6 improves by 20 points on RTLLM before simulation-guided debugging. The ASTF representation also gives designers something concrete to inspect before RTL is generated.

Critical weaknesses

The paper has good end-to-end results, but I have two main concerns. First, it is not clear which part of the pipeline is responsible for the gain. Second, the VerilogEval result is not directly comparable with the other settings.

  • The VerilogEval result uses reference RTL during verification. In the reference-assisted setting, the ITB compares DUT outputs with the provided reference module. This makes the 143-to-153 improvement from debugging hard to interpret. It also makes the comparison with VerilogCoder unclear unless the same reference information is available to that baseline.

  • Spec grounding appears mostly lexical. The grounding check verifies that a cited text span appears in the specification, but it does not seem to check whether the citation supports the generated action. In Appendix A, clk_div1 uses a thresholded assignment while the cited sentence says the divider is “toggled”. The IR still passes with zero warnings.

  • The main components are not separated. The paper does not isolate ASTF construction, auditing, and diagnostic routing. It also lacks a code-only repair baseline and a compute-matched direct-generation baseline.

  • Related verification work is missing from the discussion. Structured IRs already appear in AoT, QiMeng-CRUX, and Spec2RTL-Agent. On the verification side, testbench self-correction [1] and failure attribution between RTL and verification infrastructure [2] are closely related. This is especially relevant because Appendix A contains an incorrect expected period in the generated verification plan, while the final run still passes.

  • The evaluation is still limited. The reported results use one run per task. VerilogEval-Human v2 is also close to saturation for the strongest methods. MAGE is cited but not compared. Results on a harder benchmark such as CVDP [3] would make the evaluation more convincing.


[1] R. Qiu et al. CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design. arXiv:2411.08510, 2024.

[2] Y. Zhao et al. PRO-V: An Efficient Program Generation Multi-Agent System for Automatic RTL Verification. arXiv:2506.12200, 2025.

[3] N. Pinckney, C. Deng, C. Ho, Y. Tsai, M. Liu, W. Zhou, B. Khailany, and H. Ren. Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification. arXiv:2506.14074, 2025.

Questions for authors

Q1. Which ITB configuration was used for the VerilogEval numbers in Table 2? How many of the 153 passing designs use comparison with the reference RTL? What is the result if the ITB uses only the specification as the oracle? Does the VerilogCoder reproduction have access to the reference module or benchmark testbench?

Q2. How often does each audit layer trigger, and what kinds of errors does it fix? How does the grounding check handle cases where the cited text appears in the specification but does not support the generated action, such as the clk_div1 example in Appendix A?

Q3. How does VeriRefine-Direct compare with direct generation under a similar token budget? For example, what happens with self-refine or best-of-n plus compile checks? Also, how does diagnostic routing compare with repairing the RTL directly using the same ITB feedback?

Q4. How accurate are the ITB and the failure analyzer? How often does the ITB reject correct RTL? How often does the analyzer send a failure to the wrong level? What happens when the ITB itself is wrong?

Q5. Are the results stable across runs? Please report variance over several runs, at least for the full VeriRefine flow.

Reviewer_P4R3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VeriRefine, an LLM-enabled abstraction-refinement framework for generating RTL. To bridge the semantic gap between informal natural-language specifications and precise RTL implementations, the framework introduces Abstract Signal Transition Functions (ASTFs) as an intermediate representation (IR) to explicitly capture clocking, reset, and per-signal transition semantics before generating Verilog code. The framework strictly audits the generated ASTF IR across dimensions such as specification grounding, coverage, consistency, and state-transition integrity. The framework then uses an internally generated testbench (ITB) for simulation-guided iterative debugging. The method achieves impressive functional correctness rates of 96.0% on RTLLM v2.0 and 98.1% on VerilogEval-Human v2.

Critical strengths

Structured Semantic Decoupling: Introducing ASTF as a strongly constrained intermediate representation is highly valuable. It effectively decouples complex underlying hardware semantics (e.g., non-blocking assignments, clock domain priorities) from pure Verilog syntax generation. This design establishes an explicit intent-verification barrier before RTL generation, effectively intercepting LLM syntax hallucinations and interpretation errors.

Exceptional Token Efficiency: In multi-agent collaborative code generation frameworks struggling with high inference costs, VeriRefine offers strong lightweight advantages. Experimental data show that while achieving functional correctness comparable to or exceeding the SOTA baseline, it uses only 13.2% and 26.6% of the latter's tokens.

Critical weaknesses

Lack of Anonymous Repository Hinders Reproducibility. Although the authors state in the conclusion that they plan to release the code and prompts on GitHub, they provide no anonymous repository during the current double-blind review phase. Because this framework relies heavily on extensive prompt engineering (e.g., ASTF rule extraction, ITB test plan generation) and complex JSON parsing/auditing logic, the lack of source code severely limits reviewers' ability to objectively evaluate the framework's efficacy.

Logical Paradox and Verification Blind Spots in ITB Generation. The Internal Testbench (ITB) generation process relies entirely on a two-stage LLM pipeline that parses the same natural-language specification. If the LLM misses implicit concurrent read/write conflicts or critical corner cases during requirement parsing, the generated ITB will inherit that exact cognitive bias. This "self-confirming" validation, without constraints from an independent external oracle, can easily produce a spuriously high pass rate while hiding deep-seated sequential-logic defects.

Weak Hardware Vulnerability Defense and Abnormal State Handling. Although the ASTF audit layer checks State-Transition Integrity (STI) and RTL Design-Rule Compliance (RDRC), these schema-based static constraints struggle against realistic hardware vulnerability threats. If the input natural language specification contains security blind spots (e.g., potential buffer overflows from ambiguous boundary conditions or illegal jumps from undefined FSM states), the current ASTF mechanism appears to blindly instantiate these risks in the underlying hardware, lacking intent-driven vulnerability scrutiny and anomaly-interception capabilities.

Module-Level Benchmarks Fail to Validate System-Level Scalability. This work achieved excellent results on RTLLM v2.0 (50 designs) and VerilogEval-Human v2. However, these datasets are heavily skewed towards single-module designs. When facing real SoC-level architectures, deep pipelines, or complex cross-module information-flow scheduling, the ASTF, which focuses heavily on per-signal update logic, will face severe state-space explosion and dependency-management issues.

Questions for authors

Can you provide a link to an anonymous repository during the rebuttal phase? Reviewers need to see the framework's core source code, complete prompt templates, and execution scripts capable of reproducing the benchmark results to verify the authenticity and reproducibility of the experiments.

Given that the ITB and target code are driven by the same or an isomorphic LLM pipeline, how do you quantify or demonstrate the effectiveness of the generated ITB stimuli? When dealing with complex state-machine interactions, how does the framework avoid blind spots caused by the LLM's own cognitive biases?

When refining natural language into ASTFs, if the original input lacks definitions for certain boundary conditions or abnormal states, which is very common in human-written specifications, can VeriRefine's audit mechanism automatically complete or warn about these potential hardware vulnerabilities (e.g., illegal control flows, memory corruption, or register-level data leakage risks)?

The ASTF construction logic applies extremely fine-grained constraints to individual driven signals. In subsystems with thousands of interdependent signals and complex data flows, will this bottom-up representation cause the LLM to hit severe context truncation or lose dependencies when generating the IR? Are there preliminary experimental data or strategies for larger-scale designs?

The current evaluation in the paper stops at pure software logic simulation (i.e., Icarus Verilog) and logic synthesis (i.e., Yosys). Considering that the ultimate deployment target for RTL code is physical hardware, has the generated code undergone actual FPGA board testing? Under real physical timing constraints and board-level environments, will these LLM-generated hardware codes expose timing violations or post-synthesis functional failures that the software simulation phase did not catch?

Reviewer_jgBzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed VeriRefine, an LLM based framework for generating Verilog RTL from natural-language specifications. Usually practices directly generate RTL, but this paper proposed that we first create a structured intermediate representation using Abstract Signal Transition Functions (ASTFs) to describe signal behaviour, clocking, reset and updating conditions. The representation is then audited to extract design-rule violations before being translated into Verilog. These automatically generated test cases identify errors and aids in repairs or in generating RTL code. Using Claude Sonnenr 4.6m VeriRefine achieves 96% and 98.1% functional correctness on RTLLM v2.0 and VerilogEval-Human v2.

Critical strengths

(1) The paper addresses an important challenge in LLM-based RTL generation where incorrect understanding of natural language specifications has led to incorrect production of Verilog code.

(2) The representation of ASTF is very useful and clearly describes each signal’s properties, clocking,reset behavior and condition under which a signal changes.

(3) The paper shows strong experimental results, showing that VeriRefine improves over direct generation even without simulation guided debugging.

(4) The use of programmatic checks rather than relying on LLM self-evaluation provided a more deterministic and consistent way to identify errors before the verilog is generated.

Critical weaknesses

The ablation evaluates only complete stages but doesn’t show whether the audit or each of the failure routing improves results. Therefore, it is unclear if the gain you achieve can’t be attributed to your paper’s proposed components rather than the structured prompting in general.

Questions for authors

(1) Could you state that ASTF is checked using programmatic rules than an LLM but it is unclear whether these checks can detect when the LLM misunderstands the original specification?

(2) How does your auditing process identify when such errors highlighted in question1 occur or is it something your paper doesn’t cover ?

(3) How do the programmatic audit checks determine whether the ASTF correctly represents the meaning of the original specification, rather than simply checking whether the cited text exists?

(4) It is not clear how your framework handles incorrect expected output such as the example you showed in Appendix A.4

Reviewer_oGefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VeriRefine, an LLM-based IR generation and validation workflow, to improve the completeness of human-written hardware specifications used for RTL code generation. The authors give a clear and formal definition of the contents of an ASTF-based IR, and provide metrics for semantic auditing. Based on the IR, this paper also designed LLM-generawted ITBs and stimulus, that help LLMs to debug the generated RTL code. According to the experiment results, the refined specifications bring markedly functional correctness improvement on RTLLM and VerilogEval tasks.

Critical strengths
  • The specification refining workflow proposed by is paper solves the low-quality-description issue of the two widely used benchmarks. The ITB generation workflow may also solve the false-negative (incorrect but passed) issue of their original TBs. The pass rate increment and token cost decrement on these benchmarks are remarkable.
  • This paper gives a formal definition of the contents of an LLM-friendly specification, which is also a good paradigm for human engineers.
Critical weaknesses
  • If IR Auditing is conducted using the same LLM as IR Generation, the model may fail to notice coverage or consistency issues ignored by itself during IR Generation. It is not guaranteed that the correct rate improvement is caused by the refined inputs. Consider the extreme case that the generated Verilog code only stores and emits the expected output signal step-by-step, which may also pass the test. Evaluation (based on the five check objectives) on the refined specification is essential.
  • The ASTF-IRs shown in the Appendix contain code segments from the sample code, and thus ground-truth leakage is possible.
  • The proposed workflow is not evaluated on new benckmarks (e.g., ArchXBench, CVDP) with original description higher quality and harder tasks. So, if no further proof of scalability is provided, this work may be regarded as an improvement on engineering practice.
Questions for authors
  • How does the simulation-guided debugger identify whether an error is caused by the ASTF-IR or the RTL implementation? The ITB can find mismatches between the IR and the RTL code, but the error can be caused by both sides.

  • Where does the IR extract the information from if the original materials are incomplete? For example, the task descriptions of RTLLM and VerilogEval benchmarks are too brief. However, the original TBs of RTLLM only cover a small portion of the corner cases, and the stimuli of VerilogEval is lack of semantic interpretability.

  • Can this workflow be applied to fine-tuning dataset generation? Current datasets (e.g., which used by CodeV-R1) are usually constructed with descriptions extracted from the RTL implementation, which may have a different distribution compared to human-written ones used in the real-world scenario.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.