acceptodds
Withdrawn from ICLR 2027

Reassessing Software Intermediates in Spec-to-RTL Generation: Accuracy, Cost, and Failure Boundaries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Writing correct register-transfer-level (RTL) designs in hardware description languages (HDLs) such as Verilog requires substantial engineering effort. High-level synthesis (HLS) offers a higher-level alternative by compiling behavioral descriptions in languages such as C and C++ into RTL. As modern models become substantially better at direct RTL generation, we ask whether these software relays still justify their added calls and translation boundaries. We study this question by separating the end-to-end pipeline effect of using a software relay from the incremental artifact effect of adding a software view while holding the final hardware rules and authoritative contract fixed. Across three repeats on 156 VerilogEval v2 and 50 RTLLM tasks, direct RTL achieves the highest observed pass rates (88.2% and 58.0%) and lowest token cost among eight evaluated pipelines. In a separate 100-task, three-repeat control, adding frozen Python/C++ views alongside an intact hardware contract yields no observed accuracy gain while increasing final-lowering token use by 18.8%/35.6%, excluding view construction. Controlled ablations and selected failure traces examine intermediate generation, execution, and RTL lowering, complemented by an analytical framework explaining why intermediate test success does not guarantee final RTL correctness. Together, these findings support direct RTL as the reference correctness-cost baseline in the evaluated module-level settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, spec-to-RTL generation, software intermediate representations, hardware design automation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3EAmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares direct RTL generation from a shared, LLM-generated hardware contract with seven pipelines using pseudocode, Python, or C++ intermediates. On 156 VerilogEval v2 and 50 RTLLM tasks with three repeats, direct generation achieves the highest observed pass rates, 88.2% and 58.0%, and the lowest downstream token cost. A separate 100-task experiment finds no observed accuracy improvement from adding frozen Python or C++ code to the same hardware contract, while increasing final-generation token use by 18.8% and 35.6%. Additional experiments examine prompt differences, intermediate testing, and RTL translation failures.

Critical strengths
  1. The paper addresses a relevant question and provides useful experimental evidence. In particular, the persistent-contract experiment keeps the final hardware requirements and instructions fixed while varying the additional software code. The paired accuracy and token-cost measurements make this comparison informative.
Critical weaknesses
  1. All pipelines start from StructuredInfo, which already extracts interfaces, behavior, state, and hardware constraints from the original specification. However, the paper does not adequately explain how StructuredInfo is generated, what its complete structure looks like, or how its accuracy is verified. Understanding and extracting these requirements is a major part of spec-to-RTL generation. As a result, the experiments evaluate additional software stages after substantial hardware-oriented preprocessing, rather than whether software intermediates help interpret the original specification. Without understanding the contribution and accuracy of this preprocessing step, it is difficult to assess the significance of the results.

  2. The main comparison does not preserve the same information across pipelines. Direct RTL generation receives the complete hardware contract, whereas the Python and C++ lowering stages do not. The paper also reports translation failures involving missing expected ports, which are consistent with this concern. Although the authors provide additional controls, these either continue to omit the hardware contract or test frozen software code added to an otherwise complete RTL-generation prompt. Neither evaluates a complete software relay that preserves the same hardware requirements throughout generation and translation. Therefore, the reported performance gaps cannot establish whether software intermediates themselves are responsible for the lower accuracy.

Questions for authors
  1. How is StructuredInfo generated and validated? Please provide the generation prompt, input context, output structure, and representative examples. How does RTL generation from the original specification compare with generation from StructuredInfo?
  2. How does a complete Python or C++ relay perform when the same hardware requirements are preserved through the final RTL translation? Please compare its functional accuracy and total generation, testing, and repair costs with direct RTL generation.
  3. How many generated software artifacts fail local tests before and after repair? If an artifact still fails after repair, does the pipeline continue to RTL generation? How does final RTL correctness vary with the intermediate test results?
  4. How are the expected outputs in the generated software tests checked for correctness? Are these tests independently validated, and how are incorrect or incomplete tests handled?
Reviewer_UGGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates different LLM-based methods that utilize software code as an intermediate relay for specification-to-RTL generation. The conclusion is that Direct RTL generation outperforms the evaluated software relays. Based on this finding, the paper analyzes the failure reasons and argues that intermediate success does not guarantee final RTL correctness. Overall, this work seems to be an analytic paper that tries to discuss different relays for the LLM-based methodology. I have big concerns about the motivation and value of such analytic efforts. Specifically, I can not identify the meaning of utilizing LLMs to translate high-level or software language into RTL. This work also lacks comparison with methods that directly utilize LLMs to generate HLS C and use the tool to convert the HLS into RTL. I think it is only reasonable to utilize strict tools that support conversion from LLM-generated high-level languages such as HLS to RTL code.

Critical strengths

It is interesting to evaluate a new set of methodologies where LLMs generate intermediate code first to support the final RTL generation. The manuscript is manually written and understandable.

Critical weaknesses
  1. The motivation of the work is questionable. I can not identify a strong motivation for why we have to generate the intermediate language for RTL generation. The introduction of the related work is also vague. I don't think it is a big trend or common practice to utilize the intermediate code for RTL generation. This work seems to be based on the assumption that utilizing intermediate language for RTL generation is well recognized as a promising direction, which is not demonstrated by any related work.

  2. This work contributes no knowledge. The introduction summarizes the contributions with three important points: (1) Direct RTL generation outperforms the evaluated software relays. This finding seems to be common sense and makes the motivation weaker. Given that direct RTL generation is the best, what is the meaning of this work to explore a large set of related works utilizing the intermediate code? (2) "Detailed analysis of intermediate-stage failures." does not seem to contribute effective knowledge because the motivation itself is questionable. (3) "Intermediate success does not guarantee final RTL correctness." sounds reasonable and seems to convey a similar idea to (1).

In summary, this is a strange paper that is based on a vague assumption and seems to conduct meaningless explorations. My suggestion is that if the authors think utilizing intermediate code for RTL generation is meaningful, they should first provide evidence and demonstrate such directions worth exploring. It is common sense that direct specification-to-RTL generation is the most effective and reasonable way. However, this paper draws this common sense as a conclusion, arguing that traditional direct specification-to-RTL generation is still most effective, by exploring a set of less-recognized methods.

Questions for authors

(1) I have a big concern about your motivation to explore intermediate software codes for RTL generation. From your introduction of the related works, I can not identify the evidence that intermediate code is a promising direction for specification-to-RTL generation. Your conclusion that "Direct RTL generation outperforms the evaluated software relays" makes the motivation harder to identify. Do you think intermediate software code is promising for RTL generation? If the answer is yes, how do you explain your conclusion that "Direct RTL generation is the most effective"?

(2) Since you conducted a detailed analysis of the failures, do you think it is still meaningful to utilize intermediate software code for RTL generation? If it is meaningful, what should we do to improve the related methodology?

Reviewer_kJxhOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work inspects LLM-based generation of register-transfer-level (RTL) designs in hardware description languages (HDLs) such as Verilog. Previous work generated the intermediates in C++ or Python since the LLMs were not yet good at directly generating good RTL designs. In this work, the authors explore if the added cost for those steps is still justified, given the new models that might be improved at directly generating RTL designs. The authors separate the pipeline effect and incremental artifact effect, and test across different benchmark tasks. The results indicate that the lowest token cost and highest pass rate are achieved with the direct RTL generation with the new LLM models, specifically GPT-5.4.

Critical strengths
  • Previously, the intermediates were added due to poor direct generation of RTL with those older LLM models. Exploring if this is still a better approach is a timely question to address due to increased abilities of the newer LLM models.
  • The pipeline effect and the incremental artifact effect are different questions, and the paper nicely differentiates them.
  • Experimental design includes repeated runs with different seeds (reported in the paper), provided prompts, statistical analysis and reporting.
Critical weaknesses
  • The paper is missing comparison with existing work.
    • Previous works showed opposite results, where direct RTL generation was not good, so the relays are added (Swaroopa et al., 2024). The paper is testing the newer models against this. Therefore, direct comparison with those existing works should be provided.
    • If I understand correctly, the Abstractions-of-Thought (AoT) work (DeLorenzo et al., 2025) showed the same result as this paper (direct LLM generation of RTL works best). The authors state that the differences from AoT are executable intermediates and the persistent-contract control. Given that the conclusion matches, please clarify what new insights this work provides.
  • If there are compilers for lowering, why would we use a non-deterministic process with LLMs for that? Also, using HLS lowering would be a nice additional pipeline to test, and that one might be related to the prior work and possibly be a good baseline to compare to. HLS is also discussed quite a bit initially in the abstract and introduction, but later, every lowering step is done by the LLM and is therefore only comparable to those setups. That should be clearer in the abstract and introduction.
  • The relay prompts don't receive all the information as direct generation. For example, they receive no port list, hardware rules, or clock/reset configuration, and five C++ to RTL rules refer to a specification that is never supplied. The authors acknowledge this, and find that some generations fail because of a missing port name. However, the relay worse performance could be because of these prompt design choices rather than the cost of the intermediate representations themselves. Eliminating this type of failure, by providing full specifications in all stages, would make a fairer comparison.
  • Use of single model family (GPT) limits generalizability of the observed results and their implications.
  • The contribution is an empirical study of prompting pipelines for the RTL design generation with LLMs, a single application, and done using a single model family. Therefore, I find that the paper does not have a good fit for ICLR, but would be better for venues with RTL design topics. The terminology is heavy on the topic, and there is lack of supporting explanations and background for ICLR community to fully understand the paper, in my opinion.

Minor comments

  • lines 60-68: are the tasks still the same 206 tasks but sampled for the pre-specified control evaluations, and robustness? Overall, this is a very unclear section and hard to follow.
  • A lot of unknown terms, like "lowering", "StructuredInfo", and "relay". Possibly these terms are common and understood by the RTL design community, but the ICLR community might not know this so it needs more explanations, or change of submission venue.
  • "with the aggregate extension results broadly supporting this pattern": This statement is very strange, and at this point in reading, it is unclear what are the aggregate extension results, so it just sounds strange and vague.
  • AoT, IR, pass@1: not defined
  • line 95: P2 and P6 mean nothing to the reader at this point, only defined later at Section 3.3
  • It is strange to see research questions in the section with background and related work
Questions for authors

Please address the stated weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.