- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper compares direct RTL generation from a shared, LLM-generated hardware contract with seven pipelines using pseudocode, Python, or C++ intermediates. On 156 VerilogEval v2 and 50 RTLLM tasks with three repeats, direct generation achieves the highest observed pass rates, 88.2% and 58.0%, and the lowest downstream token cost. A separate 100-task experiment finds no observed accuracy improvement from adding frozen Python or C++ code to the same hardware contract, while increasing final-generation token use by 18.8% and 35.6%. Additional experiments examine prompt differences, intermediate testing, and RTL translation failures.
Critical strengths
- The paper addresses a relevant question and provides useful experimental evidence. In particular, the persistent-contract experiment keeps the final hardware requirements and instructions fixed while varying the additional software code. The paired accuracy and token-cost measurements make this comparison informative.
Critical weaknesses
-
All pipelines start from StructuredInfo, which already extracts interfaces, behavior, state, and hardware constraints from the original specification. However, the paper does not adequately explain how StructuredInfo is generated, what its complete structure looks like, or how its accuracy is verified. Understanding and extracting these requirements is a major part of spec-to-RTL generation. As a result, the experiments evaluate additional software stages after substantial hardware-oriented preprocessing, rather than whether software intermediates help interpret the original specification. Without understanding the contribution and accuracy of this preprocessing step, it is difficult to assess the significance of the results.
-
The main comparison does not preserve the same information across pipelines. Direct RTL generation receives the complete hardware contract, whereas the Python and C++ lowering stages do not. The paper also reports translation failures involving missing expected ports, which are consistent with this concern. Although the authors provide additional controls, these either continue to omit the hardware contract or test frozen software code added to an otherwise complete RTL-generation prompt. Neither evaluates a complete software relay that preserves the same hardware requirements throughout generation and translation. Therefore, the reported performance gaps cannot establish whether software intermediates themselves are responsible for the lower accuracy.
Questions for authors
- How is StructuredInfo generated and validated? Please provide the generation prompt, input context, output structure, and representative examples. How does RTL generation from the original specification compare with generation from StructuredInfo?
- How does a complete Python or C++ relay perform when the same hardware requirements are preserved through the final RTL translation? Please compare its functional accuracy and total generation, testing, and repair costs with direct RTL generation.
- How many generated software artifacts fail local tests before and after repair? If an artifact still fails after repair, does the pipeline continue to RTL generation? How does final RTL correctness vary with the intermediate test results?
- How are the expected outputs in the generated software tests checked for correctness? Are these tests independently validated, and how are incorrect or incomplete tests handled?