acceptodds
Under review as a conference paper at ICLR 2027

CktFormalizer: Autoformalization of Natural Language into Circuit Representations

Abstract

Hardware infrastructure is a critical bottleneck for LLM-driven circuit design, limiting what agents can express, compile, and iteratively refine within an agentic loop. To address this bottleneck, we introduce CktLean, a typed hardware infrastructure embedded in Lean. It supports hardware description, compilation to SystemVerilog, and interactive type-checking and proof feedback through a persistent read-eval-print loop (REPL). On this foundation, we build CktFormalizer, an agent framework for hardware generation, repair, optimization, and source-level equivalence proving. We evaluate structural correctness through compilation, functional correctness through RTL and gate-level simulation, and formal correctness through proofs relative to stated specifications. Across VerilogEval, RTLLM, ResBench, and CVDP, CktFormalizer with CktLean achieves compilation rates of 91.1%–99.4%. Among designs that pass RTL simulation, 95.4%–100.0% jointly complete synthesis and place-and-route and pass design-rule and layout-versus-schematic checks. In a separate evaluation on 30 VerilogEval problems, interactive proof-state feedback raises kernel-accepted equivalence proof completion from 53.3% to 63.3%. Hardware evaluation feedback also guides architecture exploration and iterative power, performance, and area (PPA) optimization, with synthesis-area reductions of up to 58.6% in the optimization loop. These results suggest that typed representations and explicit proof-state feedback help structure the agentic loop: compiler diagnostics guide targeted repairs, while proof-state feedback helps agents identify what remains to be proved and determine the next proof step. Our code is available anonymously at https://anonymous.4open.science/r/CKT-D51D/.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Hardware Generation, Electronic Design Automation, Autoformalization, Formal Verification, Theorem Proving, Agentic Workflows

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KHhEOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CktFormalizer, an agent framework that translates natural-language hardware descriptions into typed circuit representations in Lean and exports them to SystemVerilog. The framework combines compiler-guided generation and repair with separate experiments on equivalence proof generation and hardware optimization. Evaluation spans VerilogEval, RTLLM, ResBench, and a CVDP subset, using RTL and gate-level simulation, synthesis, and physical-design checks. Reported compilation rates range from 91.1% to 99.4%. A 30-problem proof pilot improves kernel-accepted proof generation from 16 to 19 cases through stepwise feedback, while optimization experiments report synthesis-area reductions of up to 58.6%.

Critical strengths
  1. Clear engineering contributions. These include CKTLean, a typed DSL for characterizing RTL semantics, as well as a complete pipeline for hardware design, verification, and PPA optimization.
  2. Solid experiments. They cover 4 mainstream RTL generation benchmarks, and beyond compilation and simulation, they also evaluate metrics required in real-world hardware design scenarios, such as synthesis, place-and-route, DRC/LVS.
Critical weaknesses
  1. There is a mismatch between the "Lean-to-RTL" paradigm and real-world hardware design scenarios. The method proposed in this paper uses Lean to describe the behavior of a circuit and then converts it into RTL. For a hardware designer without a background in functional programming and theorem proving, the cost of learning Lean and its toolchain is prohibitively high. A more common technical route in the field is to convert the RTL code written by hardware designers into Lean ("RTL-to-Lean") [1-2] and prove properties over it, thereby verifying the correctness of an existing design so that design engineers need not concern themselves with the specific details of formal verification. The authors should provide additional evidence to demonstrate that the proposed method can indeed be used in real-world IC design scenarios.

  2. Missing comparison with other verification methods. In the PPA Optimization Loop, this paper guarantees the correctness of the implementation during optimization by generating Lean proofs of the spec. However, in the traditional hardware design flow, one could also use model checking (i.e., having the model generate SystemVerilog assertions [3]) or fuzzing testing [4-5] to verify the correctness of the optimization. The authors need to explain the advantages of theorem proving over these methods. Moreover, compared with assertion or test-case generation, generating formal proofs may incur a larger token overhead.

[1] Rtl2lean: Automated RTL-to-Lean Translation with Hierarchical Theorem Generation and Lemma Reuse. https://arxiv.org/abs/2607.16855

[2] CircuitProver: Agentic Lean 4 Theorem Proving with Reusable Circuit Proof Library for Hardware Verification. https://arxiv.org/abs/2607.27259

[3] QiMeng-CodeV-SVA: Training Specialized LLMs for Hardware Assertion Generation via RTL-Grounded Bidirectional Data Synthesis. https://arxiv.org/abs/2603.14239

[4] AutoPPA: Automated Circuit PPA Optimization via Contrastive Code-based Rule Library Learning. https://arxiv.org/abs/2604.18445

[5] QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation. https://arxiv.org/abs/2505.24183

Questions for authors
  1. Can the conversion from CKTLean to Verilog guarantee semantic consistency? If not, is there a human-evaluated accuracy rate?
  2. Can CKTLean support all Verilog features? The semantics of the same Verilog code may differ when synthesized by different EDA tools (e.g., Yosys)—how does CKTLean handle this?
6 comments
AuthorsOpenReview ↗

Response to the Reviewer KHhE Part I

Q1: There is a mismatch between the "Lean-to-RTL" paradigm and real-world hardware design scenarios. The method proposed in this paper uses Lean to describe the behavior of a circuit and then converts it into RTL. For a hardware designer without a background in functional programming and theorem proving, the cost of learning Lean and its toolchain is prohibitively high. A more common technical route in the field is to convert the RTL code written by hardware designers into Lean ("RTL-to-Lean") [1-2] and prove properties over it, thereby verifying the correctness of an existing design so that design engineers need not concern themselves with the specific details of formal verification. The authors should provide additional evidence to demonstrate that the proposed method can indeed be used in real-world IC design scenarios.

A1: We thank the reviewer for highlighting this usability concern. CktLean serves as an intermediate representation (IR) within our agentic workflow. Designers provide natural-language specifications and interface requirements, while the agent generates and manipulates the Lean representation. This IR makes the workflow more general and flexible by decoupling circuit generation and formal reasoning from any particular backend. The same representation can support RTL generation, C++ simulation, and JIT execution, extending its use beyond RTL- based implementation.

AuthorsOpenReview ↗

Response to the Reviewer KHhE Part II

Q2: Missing comparison with other verification methods. In the PPA Optimization Loop, this paper guarantees the correctness of the implementation during optimization by generating Lean proofs of the spec. However, in the traditional hardware design flow, one could also use model checking (i.e., having the model generate SystemVerilog assertions [3]) or fuzzing testing [4-5] to verify the correctness of the optimization. The authors need to explain the advantages of theorem proving over these methods. Moreover, compared with assertion or test-case generation, generating formal proofs may incur a larger token overhead.

A2: Lean supports reusable libraries of machine-checked circuit lemmas, allowing component properties and optimization rules to be proved once and reused under their stated assumptions. Effective reuse requires semantic retrieval: identifying lemmas whose properties and assumptions match the current verification goal. Lean lemmas can preserve meaningful names, typed interfaces, and component-level abstractions that support such retrieval. Semantically relevant results are often harder to retrieve from low-level SMT formulas, where the connection to circuit components and design intent is less explicit.

Such reuse is critical for industrial-scale verification, where recurring IP blocks and frequent revisions make repeated verification costly. Existing proofs can help focus effort on modified logic and component interactions. We acknowledge that proof generation may require more tokens than generating assertions or tests. Reusable lemmas could amortize this cost across designs and optimization iterations.

AuthorsOpenReview ↗

Response to the Reviewer KHhE Part III

Q3: Can the conversion from CKTLean to Verilog guarantee semantic consistency? If not, is there a human-evaluated accuracy rate?

A3: Thanks for your valuable suggestion!

We formulate equivalence between CktLean and a semantic model of the emitted SystemVerilog as a proof obligation in Lean 4. We performed per-design translation validation on all 152 compiled single-clock synchronous VerilogEval designs:

Design setEvaluatedEquivalence provedInequivalence provedEquivalence proof rate
All eligible designs152150298.7%
RTL-simulation-passing subset1291290100%

Each equivalence proof establishes agreement on every output port at every cycle, for every input sequence under the modeled semantics. Failed equivalence proofs can also help reveal semantic modeling issues. In the two cases reported here, inequivalence was additionally proved in Lean. Further analysis exposed a registerWithEnable inconsistency: its Lean simulation semantics failed to preserve state when the enable signal was low, while the emitted RTL behaved correctly. These cases demonstrate the diagnostic value of translation validation for identifying inconsistencies between the source semantics and generated RTL.

We additionally cross-checked the SystemVerilog-side model against Icarus Verilog and tested the validation procedure using injected mutations:

Additional validationResult
Cycles cross-checked against Icarus Verilog28,000
Combinational designs exhaustively checked63
Injected SystemVerilog mutations rejected8/8 (100%)

We resolve the source-to-RTL verification gap for all 152 evaluated designs under the modeled semantics through per-design translation validation in Lean 4. Each design receives a machine-checked proof of either equivalence or inequivalence, supported by additional validation through Icarus Verilog cross-checks and mutation testing.

When proof reveals a behavioral mismatch, we can feed the diagnostic information back into the design loop, correct the responsible modeling or translation component, and repeat validation until equivalence between the Lean and emitted SystemVerilog semantics is proved. A Lean-checked equivalence proof thus serves as the acceptance criterion for the revised design.

Reviewer_ebWBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CktFormalizer, an LLM-based framework, based on the SPARKLE HDL platform, to generate, optimize, and verify hardware designs from natural language specifications. The underlying platform, CktLEAN, is built on Lean 4, the interactive theorem prover. The framework is evaluated by using the generated RTL through various phases like Synthesis, P&R, etc. The authors evaluate the platform on several benchmarks. Although the results produce high compilation and physical design success rates, the poor RTL simulation results reduce confidence in the approach's efficacy.

Critical strengths
  1. Integration of formal verification, specially Lean4 theorem proving, in hardware generation is a novel approach, since it helps in utilizing machine-checked proofs for automatic hardware generation.
  2. There is a thorough evaluation across several benchmark suites, including industry-standard benchmarks like VerilogEval.
Critical weaknesses
  1. A major concern for this paper is the low functional correctness of the generated design. As can be seen in Table 3, the RTL simulation rates for CktLean are lower than Verilog Agent's. Even if the synthesis and P&R are successful, a lower RTL simulation rate undermines the results. The reviewer fails to understand how beneficial it is to synthesize a functionally incorrect design.
  2. The verification protocol described in the paper is also insufficient. The proposed Lean 4-based framework verifies the source-level circuit representations. However, whether the SystemVerilog preserves the same behavior is not clear. Furthermore, as seen in Fig. 3(c), the formal proof evaluation is performed only on 30 problems. This also results in improvement of "kernel-accepted proof completion from 53.3% (16/30) to 63.3% (19/30)", which is not at all significant. These results undermines the paper's claims.
Questions for authors
  1. Can you explain how the lower RTL simulation rates in Table 3 are justified to carry out the subsequent steps of synthesis, P&R, etc.?
  2. How can you verify that the generated SystemVerilog design is functionally correct, retaining all the components of the verified CktLean representation?
  3. Why does Figure 3c only verify 30 instances?
11 comments
AuthorsOpenReview ↗

Response to the Reviewer ebWB Part I

Q1: A major concern for this paper is the low functional correctness of the generated design. As can be seen in Table 3, the RTL simulation rates for CktLean are lower than Verilog Agent's. Even if the synthesis and P&R are successful, a lower RTL simulation rate undermines the results. The reviewer fails to understand how beneficial it is to synthesize a functionally incorrect design.

A1: We thank the reviewer for raising this important concern.

  • We agree that a design that fails RTL simulation should not be considered functionally correct.

  • The existing Table 3 caption in paper explicitly states this denominator distinction: the Compile and RTL Simulation columns use all benchmark problems as the denominator, whereas synthesis, P&R, DRC/LVS, GLS, and PD Pass are reported only for designs that already pass RTL simulation. Designs that fail RTL simulation are not sent to the backend flow. Therefore, the backend results do not treat functionally incorrect designs as successful, nor do they compensate for the lower RTL simulation pass rate. They measure the conditional probability

P(physical implementation∣RTL simulation pass),P(\text{physical implementation} \mid \text{RTL simulation pass}),

rather than functional correctness or overall task success. We will make this already-stated evaluation protocol even more prominent in the revised paper to avoid ambiguity.

The appropriate end-to-end metric is the joint RTL-and-physical-design success rate,

P(RTL pass)×P(physical pass∣RTL pass).P(\text{RTL pass}) \times P(\text{physical pass}\mid\text{RTL pass}).
  • For example, on ResBench, CktLean has a lower RTL simulation rate than Verilog Agent (78.6% versus 83.9%), but its conditional PD Pass is higher (100.0% versus 57.4%), yielding an overall joint pass rate of 78.6% versus 48.2%. This does not eliminate the RTL simulation gap, which remains a limitation of the current generation results. It shows instead that, once a candidate passes functional testing, CktLean is more likely to remain physically implementable.

  • The same distinction applies to PPA optimization. Optimization starts from RTL-simulation-passing designs, and candidate implementations are retained only after recompilation and simulation checks. Source-level equivalence proofs provide an additional behavior-preservation check under the stated assumptions. Table 1. Results with GPT-6.1-sol on VerilogEval. Each result is reported as the number of passing designs followed by the pass rate in parentheses. Compilation and RTL simulation percentages use all 156 tasks as the denominator. Backend percentages use only the RTL-simulation-passing designs, which are the only designs sent to the backend flow. P&R denotes place-and-route, DRC design-rule checking, LVS layout-versus-schematic checking, and GLS gate-level simulation. P&R, DRC, and LVS have identical pass counts within each cohort and are reported in a shared column. Post-P&R GLS measures gate-level simulation success after place-and-route.

BenchmarkMethodCompilationRTL simulationSynthesisP&R / DRC / LVSPost-P&R GLS
VerilogEval (156)Verilog Agent153 (98.1%)145 (92.9%)145 (100.0%)139 (95.9%)137 (94.5%)
VerilogEval (156)CktFormalizer156 (100.0%)144 (92.3%)144 (100.0%)139 (96.5%)138 (95.8%)
  • With GPT-6.1-sol, CktFormalizer has a slightly lower RTL simulation pass rate than Verilog Agent (144/156 versus 145/156), but achieves higher conditional P&R/DRC/LVS and post-P&R GLS pass rates (96.5% versus 95.9% and 95.8% versus 94.5%, respectively). It completes the full flow on 138/156 tasks versus 137/156 for Verilog Agent, yielding a higher end-to-end pass rate (88.5% versus 87.8%).
  • The results above also show that the coding gains of the state-of-the-art model GPT-6.1 over the older model GPT-5.6 translate into an advantage in chip design with Lean. Latest model significantly improves RTL simulation accuracy for chips designed in Lean. It matches the Verilog Agent on simulation and outperforms it on end-to-end physical implementation.
AuthorsOpenReview ↗

Response to the Reviewer ebWB Part II

Q2: Can you explain how the lower RTL simulation rates in Table 3 are justified to carry out the subsequent steps of synthesis, P&R, etc.?

A2: As explicitly stated in the caption of Table 3, all downstream metrics are evaluated only on designs that pass RTL simulation. These checks assess whether simulation-passing designs can also be physically implemented and retain their tested behavior after synthesis and routing.

On ResBench, 78.6% of CktLean designs pass both RTL simulation and physical-design checks, compared with approximately 48.2% for Verilog Agent, using all tasks as the denominator. CktLean therefore achieves a substantially higher overall success rate on this benchmark despite its lower initial simulation pass rate.

The lower simulation rate highlights a current limitation of the model’s ability to generate correct circuits using CktLean. Both configurations use the same underlying model without task-specific fine-tuning. The model’s likely greater exposure to SystemVerilog during pretraining, compared with newly introduced CktLean abstractions, may partly explain this gap.

AuthorsOpenReview ↗

Response to the Reviewer ebWB Part III

Q3: How can you verify that the generated SystemVerilog design is functionally correct, retaining all the components of the verified CktLean representation?

A3: Thanks for your review.

  • We formulate equivalence between CktLean and a semantic model of the emitted SystemVerilog as a proof object in Lean 4. We performed per-design translation validation on all 152 compiled single-clock synchronous VerilogEval designs:

Table 2. Per-design translation-validation results between CktLean and the emitted SystemVerilog semantic model on compiled single-clock synchronous VerilogEval designs. “Equivalence proved” counts designs for which Lean proves agreement under the modeled semantics; “Inequivalence proved” counts designs for which Lean proves a behavioral mismatch.

Design setEvaluatedEquivalence provedInequivalence provedEquivalence proof rate
All eligible designs152150298.7%
RTL-simulation-passing subset1291290100%
  • Each equivalence proof establishes agreement on every output port at every cycle, for every input sequence under the modeled semantics. Translation validation therefore serves as an acceptance gate: when the emitted RTL disagrees with the CktLean semantics, the equivalence claim is rejected rather than silently accepting the candidate. In the two inequivalent cases, it intercepted a registerWithEnable inconsistency: the Lean simulation semantics failed to preserve state when enable was low, whereas the emitted RTL behaved correctly. This diagnosis prevented the mismatched source-to-RTL pair from being accepted as equivalent and identified the semantic issue for correction.

  • We additionally cross-checked the SystemVerilog-side model against Icarus Verilog and tested the validation procedure using injected mutations:

Table 3. Additional validation of the SystemVerilog semantic model and translation-validation procedure. The first row reports cycle-level cross-checks against Icarus Verilog, the second reports exhaustive checks for combinational designs, and the third reports rejection of intentionally mutated SystemVerilog implementations.

Additional validationResult
Cycles cross-checked against Icarus Verilog28,000
Combinational designs exhaustively checked63
Injected SystemVerilog mutations rejected8/8 (100%)
  • These results address the source-to-RTL verification gap across a substantially larger set of designs.
Reviewer_m4TFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework built on Lean for generating and verifying digital circuits from natural language specifications. It first proposes CktLean, which extends SPARKLE's hardware representation and compilation pipeline to support parameterized state, memories, bit operations, signed arithmetic, and multiple clock domains. Built on CktLean, CktFormalizer uses an LLM agent to generate and repair circuit descriptions, supports interactive equivalence proving, and optimizes designs using feedback from hardware evaluation tools. The authors compare CktFormalizer with Verilog Agent and other competitive methods on four diverse hardware design benchmarks and conduct additional experiments on equivalence proving, architecture exploration, and optimization. The results show high physical implementation success among designs that pass RTL simulation, higher proof completion rates with interactive feedback than with a single attempt, and PPA improvements.

Critical strengths
  1. CktLean makes useful improvements to the underlying hardware infrastructure. Its extensions address practical limitations of SPARKLE, supporting the practical value of the implementation.

  2. The evaluation covers 430 tasks across four benchmarks and includes both functional verification and physical implementation. The results demonstrate that designs passing RTL simulation can largely proceed through physical implementation in the tested flow, supporting the practical usefulness of the generated circuits.

Critical weaknesses
  1. CktLean provides useful hardware infrastructure, but it is unclear what is new about the agent workflow. RTLFixer already uses compiler feedback to repair RTL code, and LeanDojo supports interactive proof construction. The paper does not clearly explain what new ideas the proposed workflow introduces beyond these approaches, which weakens the case for the agent workflow as a major contribution.

  2. The results do not consistently support improved circuit generation. The RTL simulation pass rate is lower than that of Verilog Agent on all four benchmarks. In the reported ablation, removing interactive REPL feedback also leaves compilation and RTL simulation results unchanged. These results limit the evidence that the workflow improves functional correctness during circuit generation.

Questions for authors

Have you evaluated CktFormalizer with other LLMs, and if so, are the benefits over direct SystemVerilog generation consistent across models?

3 comments
AuthorsOpenReview ↗

Response to the Reviewer m4TF Part I

Q1: CktLean provides useful hardware infrastructure, but it is unclear what is new about the agent workflow. RTLFixer already uses compiler feedback to repair RTL code, and LeanDojo supports interactive proof construction. The paper does not clearly explain what new ideas the proposed workflow introduces beyond these approaches, which weakens the case for the agent workflow as a major contribution.

A1: We thank the reviewer for highlighting the need to clarify the workflow contribution. We acknowledge that compiler-guided repair and interactive proof construction are established techniques, as demonstrated by RTLFixer and LeanDojo. Our contribution is the joint design of a hardware-specific agent workflow and CktLean, a typed circuit representation that connects circuit generation, formal reasoning, and implementation evaluation. Compared with direct SystemVerilog generation, the key difference is the information available in the agent loop’s feedback. SystemVerilog permits implicit width conversions and truncation, so some unintended width combinations can compile successfully. In CktLean, signal widths are encoded in types, making incompatible combinations explicit type errors. Figure 1 in Section 2 illustrates this mechanism: adding a Signal dom (BitVec 8) to a Signal dom (BitVec 16) is rejected by Lean’s type checker. Such diagnostics expose the incompatible types to the agent and guide an explicit correction. Table 1 summarizes the differences in representation constraints and diagnostic feedback. These foundational checks build on Cktlean; CktLean extends the infrastructure to support parameterized state, memories, bit operations, signed arithmetic, and multiple clock domains, and provides a persistent REPL for interaction with the agent.

Type checking addresses representation-level consistency; establishing that a circuit satisfies its specification requires additional reasoning. CktFormalizer supports source-level equivalence proving over the same CktLean representation, using interactive proof states to expose remaining obligations and guide subsequent proof steps. The representation also compiles to SystemVerilog, allowing synthesis and physical-design results to guide architecture exploration and PPA optimization. Appendix B.8 provides a practical application through AXI4-Lite target-side and initiator-side modules, where typed channel interfaces and exhaustive case handling support circuit generation.

Relative to RTLFixer’s compiler-feedback repair, our workflow incorporates feedback from explicitly encoded hardware constraints and source-level proof obligations. Relative to LeanDojo’s interactive proof construction, it connects formal reasoning to executable circuit descriptions, RTL compilation, and measurable implementation outcomes. The workflow contribution lies in organizing these feedback channels in agentic loop around a common hardware representation.

We will revise the paper to distinguish our infrastructure and workflow contributions from the established feedback techniques they use.

AuthorsOpenReview ↗

Response to the Reviewer m4TF Part II

Q2: The results do not consistently support improved circuit generation. The RTL simulation pass rate is lower than that of Verilog Agent on all four benchmarks. In the reported ablation, removing interactive REPL feedback also leaves compilation and RTL simulation results unchanged. These results limit the evidence that the workflow improves functional correctness during circuit generation.

A2: We thank for your review.

If we consider both RTL simulation success and downstream physical implementation, CktLean achieves a higher proportion of designs passing both stages on ResBench.

Table1. Comparison with Verilog Agent on ResBench:

MetricDenominatorCktLeanVerilog Agent
Physical-design pass rate (PD Pass)RTL-simulation-passing designs100%57.4%
Post-route gate-level simulation pass rateRTL-simulation-passing designs100%55.3%
Joint RTL simulation and physical-design pass rateAll benchmark tasks78.6%≈48.2%

Thus, on ResBench, CktLean produces a higher proportion of designs that pass both RTL simulation and physical-design checks. Conditional PD Pass is also higher on RTLLM and CVDP.

Table2. Comparison with Sparkle under the same agentic workflow on CVDP:

MetricDenominatorCktLeanSparkle
RTL simulation pass rateAll benchmark tasks40.5%25.6%
Synthesis success rateRTL-simulation-passing designs100%67.4%
Physical-design pass rate (PD Pass)RTL-simulation-passing designs97.1%67.4%

These results support CktLean’s contribution to improving both functional simulation success and downstream implementability relative to Sparkle. Without task-specific fine-tuning, CktFormalizer also achieves an 83.3% RTL simulation pass rate on VerilogEval. We will clarify that the demonstrated benefits concern improved infrastructure capability and downstream implementation success.

Regarding the REPL ablation, we would like to clarify that interactive proof-state feedback supports equivalence checking during iterative PPA optimization, rather than initial circuit generation and repair. As described in Appendix B.3, removing Lean REPL feedback retains the generation and repair loop, with batch compiler diagnostics still available to the agent. The interactive proof component therefore does not affect the compilation and RTL simulation results reported for initial generation. Its role is to help verify that optimization candidates preserve the original circuit’s source-level behavior.

AuthorsOpenReview ↗

Response to the Reviewer m4TF Part III

Q3: Have you evaluated CktFormalizer with other LLMs, and if so, are the benefits over direct SystemVerilog generation consistent across models?

A3: Thank for your suggestion. We have conducted additional evaluations with GLM-5.3-Flash and GPT-6.1-sol on VerilogEval (156 tasks).

With GPT-6.1-sol, CktFormalizer achieves a higher compilation rate (100.0% versus 98.1%) and slightly more post-route gate-level simulation passes (138 versus 137). Despite this lower RTL-simulation rate, CktFormalizer achieves higher conditional P&R/DRC/LVS and post-P&R GLS pass rates (96.5% versus 95.9% and 95.8% versus 94.5%, respectively), and completes the full flow on more tasks overall (138/156 versus 137/156; 88.5% versus 87.8%). With GLM-5.3-Flash, CktFormalizer achieves a higher compilation rate and higher conditional backend pass rates, but a lower RTL simulation pass rate.

Table 3. Comparison of CktFormalizer and direct SystemVerilog generation (Verilog Agent) using two LLMs on VerilogEval (156 tasks). Each result is reported as the number of passing designs followed by the pass rate in parentheses. Compilation and RTL simulation percentages use all 156 tasks as the denominator. Backend percentages use only the RTL-simulation-passing designs for each method, which are the only designs sent to the backend flow. P&R denotes place-and-route, DRC design-rule checking, LVS layout-versus-schematic checking, and GLS gate-level simulation. P&R, DRC, and LVS have identical pass counts within each cohort and are reported in a shared column. Post-P&R GLS measures gate-level simulation success after place-and-route.

ModelBenchmarkMethodCompilationRTL simulationSynthesisP&R / DRC / LVSPost-P&R GLS
GLM-5.3-FlashVerilogEval (156)Verilog Agent152 (97.4%)118 (75.6%)117 (99.2%)112 (94.9%)111 (94.1%)
GLM-5.3-FlashVerilogEval (156)CktFormalizer154 (98.7%)96 (61.5%)96 (100.0%)93 (96.9%)92 (95.8%)
GPT-6.1-solVerilogEval (156)Verilog Agent153 (98.1%)145 (92.9%)145 (100.0%)139 (95.9%)137 (94.5%)
GPT-6.1-solVerilogEval (156)CktFormalizer156 (100.0%)144 (92.3%)144 (100.0%)139 (96.5%)138 (95.8%)

With GLM-5.3-Flash, CktFormalizer improves compilation from 152/156 to 154/156 and achieves higher conditional backend pass rates at every reported backend stage: synthesis (100.0% versus 99.2%), P&R/DRC/LVS (96.9% versus 94.9%), and post-P&R GLS (95.8% versus 94.1%). Although its RTL simulation pass rate is lower than Verilog Agent's (96/156 versus 118/156), the higher backend pass rates are a key advantage: conditional on reaching the backend flow, CktFormalizer preserves physical implementability more reliably. With GPT-6.1-sol, CktFormalizer has a slightly lower RTL simulation pass rate (144/156 versus 145/156), but it achieves higher conditional P&R/DRC/LVS (96.5% versus 95.9%) and post-P&R GLS (95.8% versus 94.5%). More importantly, it completes the full flow on 138/156 tasks versus 137/156 for Verilog Agent, yielding a higher end-to-end pass rate (88.5% versus 87.8%) despite the lower RTL simulation rate.

This experiment supports our main conclusion: CktFormalizer can better preserve physical implementability through the backend flow, even when its RTL simulation pass rate is slightly lower.

Reviewer_BiuFOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CktFormalizer, an approach to translate between natural language specifications and a formal specification for a hardware design. To this end the authors propose a lean-4 based specification with the ability to lower the specification from a purely functional one to an implementation and demonstrates its use in several cases. for instance Architectural exploration with respect to an abstract functional specification and PPA optimization. Overall the paper is well written, comprehensively evaluated from an empirical standpoint, although the evaluations are still on relatively simple benchmarks which is a limitation of the community, not of this paper.

Critical strengths

Overall this paper brings the power of auto formalization closer to the hardware domain. This is going to become critical as more and more code is generated by language models and therefore formal correctness of these generated codes is of crucial importance.

Critical weaknesses

No substantial weakness is found.

Questions for authors

How would you expect these solutions to scale to complex industry-scale designs?

1 comment
AuthorsOpenReview ↗

Response to the Reviewer BiuF Part I

Q1: How would you expect these solutions to scale to complex industry-scale designs?

A1: We address this question through the AXI4-Lite case study in Appendix B.8. Building on these interfaces, a possible future extension is a DMA subsystem comprising configuration registers, address generators, transfer controllers, and buffers. Each component could be generated and formally verified against its specification, followed by formal verification that the integrated subsystem satisfies system-level requirements, such as correct data transfer and completion signaling. This illustrates a path toward industrial designs through hierarchical generation, component reuse, and compositional verification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.