- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RTL-BenchLS, a benchmark designed to evaluate the capabilities of large language models (LLMs) and agents in the generation, completion, and debugging of hardware designs at the Register-Transfer Level (RTL). The benchmark is organized into 12 task categories—structured across dimensions of task scope, type, and protocol—comprising over 17,000 task instances that span both module-level and project-level designs. In addition to standard tasks based on fixed specifications, the paper proposes a "round-trip" protocol: a model first extracts an intermediate representation—constrained by specific format and length requirements—from a correct RTL design; subsequently, a model or agent, operating within an isolated context, uses this representation to reconstruct, complete, or repair the RTL. The study evaluates eight prominent models and their corresponding agent frameworks, analyzing the impact of factors such as design scale, failure types, token consumption, and the nature of the intermediate representation. The results indicate that project-level tasks remain significantly challenging, and the round-trip protocol generally leads to lower performance in generation and completion tasks. The primary contributions of this work include the task dataset, the evaluation protocol, and the associated diagnostic analysis.
Critical strengths
-
This work organizes code generation, completion, and debugging tasks into module-level and project-level categories. It introduces two types of protocols—direct and round-trip—enabling the evaluation of diverse engineering capabilities within a unified framework. Project-level tasks, in particular, involve the coordination of multiple modules and interfaces, thereby extending the scope beyond simple, small-scale module generation problems. The primary contribution of this work lies in the expanded scale of tasks, the breadth of their sources, and the diverse combinations of evaluation protocols.
-
The authors performed deduplication and near-duplicate filtering, verified consistency between specifications and reference RTL, conducted independent reviews, and carried out manual validation. Furthermore, they employed functional fault variants to ensure the evaluation pipeline is capable of detecting errors. For genuine functional issues, the authors verified that the proposed fixes are behaviorally equivalent to the developers' original patches; the results demonstrate that alternative implementations—distinct from the reference patches—are also accepted as valid. These design choices enhance the credibility of the benchmark.
Critical weaknesses
- There are unexplained inconsistencies in the experimental results. In Table 2, the score for DeepSeek V3.2 on "direct module generation" is 46.4%, whereas Table 3—which references the same benchmark, protocol, and model—reports a score of 42.9%. Furthermore, Appendix C.1 states that Llama failed to pass any test cases across all difficulty levels, yet Table 2 shows a score of 3.1% for "direct item generation." These discrepancies may stem from differences in samples, configurations, or software versions, but the paper provides no explanation.
- Additionally, the item-level experiments utilize Codex CLI, Claude Code, and OpenHands, yet the paper fails to systematically report the iterations, time and token budgets, number of repetitions, tool versions, or termination criteria for each configuration. Consequently, it is difficult to disentangle the respective contributions of the base model’s capabilities, the agent framework, and the allocated resources. For a paper whose primary contribution is a benchmark, the traceability of configurations and data for each result is essential for substantiating its conclusions.
- The category of "non-functional issue fixes" encompasses objectives such as removing redundant logic and optimizing performance. However, Task 3.2a evaluates performance solely via "CommitMatch" against developer-submitted commits. The paper does not specify the matching criteria, how various valid fixes are accepted, or whether the target attributes—and the preservation of existing functionality—are actually verified. Moreover, this score is averaged with results from actual functional bug fixes and human-introduced fault variants before being presented in the main table; therefore, it cannot be interpreted as a direct measure of the success rate for fixing functional issues in real-world projects.
Questions for authors
-
Please explain the discrepancies regarding DeepSeek (46.4% in Table 2 vs. 42.9% in Table 3) and Llama (0% vs. 3.1% in Appendix C.1 and Table 2, respectively). Could you provide the specific task sets, raw pass counts, number of repetitions, model/agent versions, and computational budgets associated with these results? Furthermore, please clarify which results were derived from the same set of "frozen" experiments.
-
What intermediate representation is used in the main table for the round-trip protocol? Is the length ratio () calculated based on characters, words, or tokens? What are the criteria for the code leakage check? Since Appendix Table 7 employs different length ratios for various representations, would it be possible to provide a comparison based on a consistent design and information budget? Additionally, could you report the compliance rate for the intermediate representation separately from the reconstruction success rate (achieved after ensuring compliance)?
-
Regarding the scoring for non-functional repairs: What is the operational definition of "CommitMatch"? If an alternative patch preserves the original functionality and achieves the same optimization goals—but differs from the patch submitted by the developer—is it still awarded points? Could you present these scores separately from those based on behavioral verification to avoid ambiguity in the aggregated metrics?
6 comments
Response (2/3) to Reviewer uKvs
Q2. Intermediate representation
In Question 2, the reviewer asks four questions, which we will answer one by one.
Q2.1 Intermediate representation format for main results:
In Table 2, the results are mainly in NL format, as denoted by A_NL in the appendix.
Because the results in Appendix C.3 are mainly an exploratory analysis and introduce many notations that may confuse readers, we do not include the A_xx notations in the main paper.
Q2.2 Evaluation method
To meet the length limit, we calculate all lengths in tokens, using the same tokenizer for each model. We use both a script-based judge to check the code snippet and a GPT-6.1 Sol-based judge to determine whether the intermediate representation leaks code.
Q2.3 Comparison based on the consistent ratio
For code and table formats, we design this to primarily explore the influence of intermediate representation style on recovery effects. Usually, the generated code and diagram will be around r=1. So we think it is reasonable to compare it with A_NL or A_Free at ratio r= 1. We think it is hard to specify a ratio for the diagram or code because it may lead to malformed diagrams or code.
Q3. CommitMatch-based evaluation method
For CommitMatch, we implement the CommitMatch evaluator using the standard Python library: “difflib”.
We acknowledge the limitations of the CommitMatch methods. To alleviate this limitation, we add a new Checklist-based judgment. Specifically, we list a set of checkpoints that the commit should pass. We then use the strongest model, such as GPT-6.1 Sol, to conduct a judgment based on the checklist. We provide additional results in Table 9 of Appendix Section C.3. We manually verified and found that around 5% to 8% of alternative solutions are rejected by the strict CommitMatch method. We will integrate this additional evaluation method into our final public benchmark pipeline.
Response (3/3) to Reviewer uKvs
W2. Agentic task configuration
We appreciate the reviewer’s comments that the current experimental setup lacks a detailed introduction of agentic experiments. Following the reviewer’s suggestion, we introduce the detailed configuration for agentic methodologies. Specifically, we add configuration details in Section 5.1 on page 7: “For project-level tasks, agents are given a running budget of 32 agent iterations under a 30-minute time limit for easy and medium cases and 64 agent iterations under a 60-minute time limit for difficult cases. We monitor runtime and confirm that these budgets are sufficient for agents to complete the project design tasks and submit for the final evaluation.”
W3. Detailed results on debugging tasks
In Weakness 3, the reviewer mentions that the results are averaged across three debugging types: mutant-based debugging, functional issue-fixing, and nonfunctional issue-fixing. We apologize for not mentioning where the detailed experiments are located in the main pages of the paper. Specifically, in Table 9 of Appendix Section C.4 on page 25, we report the separate results for debugging tasks. We add the reminder in Section 5.2 on page 8: “Appendix C.4 provides separate results on debugging tasks.”
Response (1/3) to Reviewer uKvs
Thank you for your thorough review of the experimental results, which significantly helps us to improve the experimental sections. We apologize for the unclear experimental setup and inconsistent results for DeepSeek. Based on your comments and suggestions, we have added the necessary clarifications to our experimental setup and fixed the inconsistency. We will list the explanations and updates to the manuscript below.
Q1. Inconsistent Results
In Question 1. The reviewer points out two inconsistent results (DeepSeek in Table 2 v.s. Table 3 and Llama in Table 2 v.s. Appendix). Specifically, the results are from three repetitions (n=3) in Table 2. The DeepSeek result in Table 3 is from an intermediate stage (under n=2). We apologize for this mistake and have updated this result. The analysis in Appendix C.1 for all models is based on a single run (n=1). In that single run, the LLaMA fails all test points. We apologize for this confusion and have added clarification on the repetition time in Appendix C.1.
The authors are also confused about the model versions. We apologize for the unclear notation. Specifically, all models follow the version introduced in the experimental setup (Section 5.1): “ For commercial models, we select GPT-5.6 Sol, Claude Sonnet 4.5, and Claude Haiku 4.5. For open-source models, we select DeepSeek V3.2, Mistral Large, Qwen3.5-397B, and Llama 3.3 70B.” In the following part, some models are denoted in a simplified form due to space limitations. For example, DeepSeek still refers to DeepSeek V3.2, and Qwen still refers to Qwen3.5-397B.
Regarding the raw results, we show the raw counts below for your reference: