acceptodds
Under review as a conference paper at ICLR 2027

RTL-BenchLS: A Large-Scale Benchmark for Agentic RTL Reasoning and Implementation

Abstract

Agentic frameworks are advancing rapidly, driving growing interest in agents for automated hardware design. Implementing hardware in RTL (i.e., RTL generation) is a key task in VLSI circuit design. However, existing RTL benchmarks are too easy for cutting-edge agentic frameworks. Specifically, we observe three gaps in the RTL benchmarks. (1) Limited task scope. Benchmarks are limited in module-level scope. Such module-level scope is insufficient to evaluate agentic frameworks. The EDA community urgently requires a qualified project-level RTL benchmark. (2) Narrow task type. Most benchmarks only focus on a narrow task scope. A single narrow task type is not enough to thoroughly evaluate agents on complicated RTL tasks. (3) Small benchmark scale and design size. Existing benchmarks are usually in small scale with small design size. To address these limitations, we introduce RTL-BenchLS, a large-scale benchmark of 17K+ cases with 12 tasks. We contribute in three aspects. (1) New task scope. We expand the scope from module level to project level with realistic project-level designs. Experiments show that our module-level cases are sufficiently challenging, and project-level tasks expose obvious limitations of current agentic frameworks. (2) Expanded task type. We cover three important types to evaluate agents' abilities from different perspectives, including generation, completion, and debugging. (3) Enlarged benchmark scale and design size. We collect a significantly larger scale of cases covering larger design size. To further increase the benchmark scale, we define a new task protocol named round-trip protocol, which eliminates the need to manually craft qualified specifications and testbenches. As a result, we can significantly expand the benchmark scale using collected RTL designs. The round-trip tasks are also significantly challenging for the current LLMs. This new benchmark, at unprecedented scale, will help evaluate academic work and industrial solutions and improve LLM/agent solutions for hardware RTL design. Our benchmark will be released at https://anonymous.4open.science/r/RTL-BenchLS-A0AB.

Primary area:
datasets and benchmarks
Keywords:
Electronic Design Automation (EDA), Benchmarking, RTL generation, Agentic frameworks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uKvsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RTL-BenchLS, a benchmark designed to evaluate the capabilities of large language models (LLMs) and agents in the generation, completion, and debugging of hardware designs at the Register-Transfer Level (RTL). The benchmark is organized into 12 task categories—structured across dimensions of task scope, type, and protocol—comprising over 17,000 task instances that span both module-level and project-level designs. In addition to standard tasks based on fixed specifications, the paper proposes a "round-trip" protocol: a model first extracts an intermediate representation—constrained by specific format and length requirements—from a correct RTL design; subsequently, a model or agent, operating within an isolated context, uses this representation to reconstruct, complete, or repair the RTL. The study evaluates eight prominent models and their corresponding agent frameworks, analyzing the impact of factors such as design scale, failure types, token consumption, and the nature of the intermediate representation. The results indicate that project-level tasks remain significantly challenging, and the round-trip protocol generally leads to lower performance in generation and completion tasks. The primary contributions of this work include the task dataset, the evaluation protocol, and the associated diagnostic analysis.

Critical strengths
  1. This work organizes code generation, completion, and debugging tasks into module-level and project-level categories. It introduces two types of protocols—direct and round-trip—enabling the evaluation of diverse engineering capabilities within a unified framework. Project-level tasks, in particular, involve the coordination of multiple modules and interfaces, thereby extending the scope beyond simple, small-scale module generation problems. The primary contribution of this work lies in the expanded scale of tasks, the breadth of their sources, and the diverse combinations of evaluation protocols.

  2. The authors performed deduplication and near-duplicate filtering, verified consistency between specifications and reference RTL, conducted independent reviews, and carried out manual validation. Furthermore, they employed functional fault variants to ensure the evaluation pipeline is capable of detecting errors. For genuine functional issues, the authors verified that the proposed fixes are behaviorally equivalent to the developers' original patches; the results demonstrate that alternative implementations—distinct from the reference patches—are also accepted as valid. These design choices enhance the credibility of the benchmark.

Critical weaknesses
  1. There are unexplained inconsistencies in the experimental results. In Table 2, the score for DeepSeek V3.2 on "direct module generation" is 46.4%, whereas Table 3—which references the same benchmark, protocol, and model—reports a score of 42.9%. Furthermore, Appendix C.1 states that Llama failed to pass any test cases across all difficulty levels, yet Table 2 shows a score of 3.1% for "direct item generation." These discrepancies may stem from differences in samples, configurations, or software versions, but the paper provides no explanation.
  2. Additionally, the item-level experiments utilize Codex CLI, Claude Code, and OpenHands, yet the paper fails to systematically report the iterations, time and token budgets, number of repetitions, tool versions, or termination criteria for each configuration. Consequently, it is difficult to disentangle the respective contributions of the base model’s capabilities, the agent framework, and the allocated resources. For a paper whose primary contribution is a benchmark, the traceability of configurations and data for each result is essential for substantiating its conclusions.
  3. The category of "non-functional issue fixes" encompasses objectives such as removing redundant logic and optimizing performance. However, Task 3.2a evaluates performance solely via "CommitMatch" against developer-submitted commits. The paper does not specify the matching criteria, how various valid fixes are accepted, or whether the target attributes—and the preservation of existing functionality—are actually verified. Moreover, this score is averaged with results from actual functional bug fixes and human-introduced fault variants before being presented in the main table; therefore, it cannot be interpreted as a direct measure of the success rate for fixing functional issues in real-world projects.
Questions for authors
  1. Please explain the discrepancies regarding DeepSeek (46.4% in Table 2 vs. 42.9% in Table 3) and Llama (0% vs. 3.1% in Appendix C.1 and Table 2, respectively). Could you provide the specific task sets, raw pass counts, number of repetitions, model/agent versions, and computational budgets associated with these results? Furthermore, please clarify which results were derived from the same set of "frozen" experiments.

  2. What intermediate representation is used in the main table for the round-trip protocol? Is the length ratio (rr) calculated based on characters, words, or tokens? What are the criteria for the code leakage check? Since Appendix Table 7 employs different length ratios for various representations, would it be possible to provide a comparison based on a consistent design and information budget? Additionally, could you report the compliance rate for the intermediate representation separately from the reconstruction success rate (achieved after ensuring compliance)?

  3. Regarding the scoring for non-functional repairs: What is the operational definition of "CommitMatch"? If an alternative patch preserves the original functionality and achieves the same optimization goals—but differs from the patch submitted by the developer—is it still awarded points? Could you present these scores separately from those based on behavioral verification to avoid ambiguity in the aggregated metrics?

6 comments
AuthorsOpenReview ↗

Response (1/3) to Reviewer uKvs

Thank you for your thorough review of the experimental results, which significantly helps us to improve the experimental sections. We apologize for the unclear experimental setup and inconsistent results for DeepSeek. Based on your comments and suggestions, we have added the necessary clarifications to our experimental setup and fixed the inconsistency. We will list the explanations and updates to the manuscript below.

Q1. Inconsistent Results

In Question 1. The reviewer points out two inconsistent results (DeepSeek in Table 2 v.s. Table 3 and Llama in Table 2 v.s. Appendix). Specifically, the results are from three repetitions (n=3) in Table 2. The DeepSeek result in Table 3 is from an intermediate stage (under n=2). We apologize for this mistake and have updated this result. The analysis in Appendix C.1 for all models is based on a single run (n=1). In that single run, the LLaMA fails all test points. We apologize for this confusion and have added clarification on the repetition time in Appendix C.1.

The authors are also confused about the model versions. We apologize for the unclear notation. Specifically, all models follow the version introduced in the experimental setup (Section 5.1): “ For commercial models, we select GPT-5.6 Sol, Claude Sonnet 4.5, and Claude Haiku 4.5. For open-source models, we select DeepSeek V3.2, Mistral Large, Qwen3.5-397B, and Llama 3.3 70B.” In the following part, some models are denoted in a simplified form due to space limitations. For example, DeepSeek still refers to DeepSeek V3.2, and Qwen still refers to Qwen3.5-397B.

Regarding the raw results, we show the raw counts below for your reference:

TaskTotal Runs/TestpointsGPT-5.6 Sol PassGPT-5.4 PassSonnet 4.5 PassHaiku 4.5 PassDeepSeek V3.2 PassMistral Large PassQwen 3.5 PassLlama 3.3 Pass
1.1 Direct module generation2,115156911451154817981551828416
2.1 Direct module completion3,888207315562613118911916481233432
3.1 Direct module debugging4,69229652454281432832660107784647
1.2 Direct project generation1,002208130217156144658331
2.2 Direct project completion1,77966744559244464329730031
3.2 Direct project debugging2,481856851932806699570414248
1.3 Round-trip module generation7,27231982546290225972401178424131892
2.3 Round-trip module completion1,251364280269285284225287293
3.3 Round-trip module debugging1,236916786872784714415682511
1.4 Round-trip project generation1,110305691112714184126
2.4 Round-trip project completion15,087395822662843172028556531696252
3.4 Round-trip project debugging2,310750549836635530238374276
AuthorsOpenReview ↗

Response (2/3) to Reviewer uKvs

Q2. Intermediate representation

In Question 2, the reviewer asks four questions, which we will answer one by one.

Q2.1 Intermediate representation format for main results:

In Table 2, the results are mainly in NL format, as denoted by A_NL in the appendix.

Because the results in Appendix C.3 are mainly an exploratory analysis and introduce many notations that may confuse readers, we do not include the A_xx notations in the main paper.

Q2.2 Evaluation method

To meet the length limit, we calculate all lengths in tokens, using the same tokenizer for each model. We use both a script-based judge to check the code snippet and a GPT-6.1 Sol-based judge to determine whether the intermediate representation leaks code.

Q2.3 Comparison based on the consistent ratio

For code and table formats, we design this to primarily explore the influence of intermediate representation style on recovery effects. Usually, the generated code and diagram will be around r=1. So we think it is reasonable to compare it with A_NL or A_Free at ratio r= 1. We think it is hard to specify a ratio for the diagram or code because it may lead to malformed diagrams or code.

Q3. CommitMatch-based evaluation method

For CommitMatch, we implement the CommitMatch evaluator using the standard Python library: “difflib”.

We acknowledge the limitations of the CommitMatch methods. To alleviate this limitation, we add a new Checklist-based judgment. Specifically, we list a set of checkpoints that the commit should pass. We then use the strongest model, such as GPT-6.1 Sol, to conduct a judgment based on the checklist. We provide additional results in Table 9 of Appendix Section C.3. We manually verified and found that around 5% to 8% of alternative solutions are rejected by the strict CommitMatch method. We will integrate this additional evaluation method into our final public benchmark pipeline.

AuthorsOpenReview ↗

Response (3/3) to Reviewer uKvs

W2. Agentic task configuration

We appreciate the reviewer’s comments that the current experimental setup lacks a detailed introduction of agentic experiments. Following the reviewer’s suggestion, we introduce the detailed configuration for agentic methodologies. Specifically, we add configuration details in Section 5.1 on page 7: “For project-level tasks, agents are given a running budget of 32 agent iterations under a 30-minute time limit for easy and medium cases and 64 agent iterations under a 60-minute time limit for difficult cases. We monitor runtime and confirm that these budgets are sufficient for agents to complete the project design tasks and submit for the final evaluation.”

W3. Detailed results on debugging tasks

In Weakness 3, the reviewer mentions that the results are averaged across three debugging types: mutant-based debugging, functional issue-fixing, and nonfunctional issue-fixing. We apologize for not mentioning where the detailed experiments are located in the main pages of the paper. Specifically, in Table 9 of Appendix Section C.4 on page 25, we report the separate results for debugging tasks. We add the reminder in Section 5.2 on page 8: “Appendix C.4 provides separate results on debugging tasks.”

Reviewer_zzsoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RTL-BenchLS, which evaluates LLMs and agents on RTL hardware design through 12 task combinations. Its central scaling mechanism is the round-trip protocol: a model reads reference RTL, produces a constrained intermediate description, and then reconstructs or repairs RTL in a fresh context. Correctness is checked against the hidden reference using formal equivalence or simulation. This allows evaluation without manually writing specifications for every design. Results show substantial remaining headroom for direct project-generation.

Critical strengths
  1. The paper provides useful coverage of hardware development tasks.

  2. The paper proposes a promising mechanism for scaling evaluation, providing significant engineering contribution.

  3. The proposed benchmark is good-quality and diagnostic-oriented.

Critical weaknesses
  1. Some scoring rules measure agreement with a reference patch rather than successful engineering. For instance, nonfunctional issue fixing uses commit matching (Appendix B.3), which risks rejecting alternative valid improvements. A functionally equivalent patch also does not establish that an area or performance objective was achieved. Hard debugging additionally gates success on explicit localization before repair. These choices need validation against independently judged correct solutions. Project testpoint scores are useful for partial progress, but complete-project success should also be reported systematically.

  2. The headline scale overstates the number of independent evaluation problems. Completion and debugging cases are derived from shared parent designs, and much of the module corpus comes from existing public datasets. These remain useful tasks, but their errors and successes are correlated. The paper needs clearer accounting of unique designs, overlap between tasks, and potential training contamination.

  3. The direct difficulty comparison focuses on VerilogEval and RTLLM, although CVDP and HWE-Bench are closer to the paper’s agentic claims. The claimed verification advantage over HWE-Bench also needs a concrete audit: HWE-Bench already documents fail-to-pass validation and manual review. Formal checking could provide additional assurance, but its incremental benefit should be demonstrated.

Questions for authors
  1. Can the round-trip performance cleanly isolate RTL reasoning? The final score combines understanding the reference, compressing its behavior into a length-limited representation, and reconstructing it. However, it seems that failure can originate at any stage. Can lower scores by themselves reliably establish weaker hardware reasoning or greater practical task difficulty?

  2. DeepSeek’s direct module-generation score is 46.4% in Table 2 but 42.9% in Table 3. It seems that there is no explanation on the difference in setting that lead to the different scores.

5 comments
AuthorsOpenReview ↗

Response (1/4) to Reviewer zzso

Thank you very much for acknowledging our key contributions. Following your instructions and comments, we added a new experiment and further refined our manuscript.

In the following response sections, (1) discussion of the round-trip tasks and (2) inconsistent results for DeepSeek, we answer the reviewer’s question by updating the experimental results.

In response Sections (3), (4), and (5), we present our refined manuscript and methodology based on the reviewer’s comments.

(1) Discussion of the round-trip tasks

Thank you for the reviewer’s insightful question. We carefully considered your question and designed a new experimental methodology to evaluate the quality of intermediate representations.

To further quantify the quality of intermediate representations, we introduce the “Requirement Fidelity Score” (RFS) in the current Section 6.3 of the updated manuscript on Page 10.

Specifically, we first extract a checklist of requirements for a qualified intermediate representation, such as necessary clock/reset behavior, state conditions, etc. Then we use the strongest commercial model, such as GPT-6.1 Sol, to check the intermediate representation against this checklist. Specifically, each requirement is labeled as preserved (scored 1), partial(scored 0.5), omitted (scored 0), or contradicted (scored -1). We calculate the Coverage score by summing the weighted scores from preserved and partial labels, and the Contradiction score by summing the contradicted labels.

Finally, the Requirement Fidelity Score (RFS) is calculated from Coverage and Contradiction: RFS = Coverage -Contradiction, ranging in [-100,100], with a higher score denoting better quality.

Figure 6 shows the analysis score for the intermediate evaluation score based on slice 01 at the Round-trip Module-level RTL Generation task. The figure indicates that higher-quality intermediate representations (higher RFS score) contribute to higher final RTL pass rates. Additionally, we calculate the RFS score based on PASS and Fail groups (final RTL pass results). The results demonstrate that higher-quality representations generally lead to higher RTL recovery success rates.

Appendix Section E.1 on Page 28 also contains several auxiliary case studies to help illustrate our quantitative findings. Weaker models can generate many redundant words and fail to capture important semantics. The stronger models can capture the key logic more accurately.

AuthorsOpenReview ↗

Response (2/4) to Reviewer zzso

(2) Inconsistent results for DeepSeek

We apologize for the mistake in the results.

Specifically, we report our results based on three repetitions (n=3) in Table 2. The DeepSeek result in Table 3 comes from an intermediate stage (n=2). We have updated this result to the final stage.

(3) Improvement of scoring rules for nonfunctional issue-fixing

In Weakness 1. The reviewer points out the weakness of our current CommitMatch-based scoring rules. We thank the reviewers for pointing out this weakness and helping us improve our evaluation pipeline. To alleviate this limitation, we add a new Checklist-based judgment. Specifically, we list a set of checkpoints that the commit should pass. We then use the strongest model, such as GPT-6.1 Sol, to conduct a judgment based on the checklist. We provide additional results in Table 9 of Appendix Section C.3. We manually verify and found that around 5% to 8% of alternative solutions are rejected by the strict CommitMatch method. We will integrate this additional evaluation method into our final public benchmark pipeline.

AuthorsOpenReview ↗

Response (3/4) to Reviewer zzso

(4) Benchmark case independence

In weakness 2. The reviewer points out the unclear introduction of the independence of benchmark cases. We apologize for the unclear introduction of the task cases, especially regarding the independence of RTL designs.

We apologize for the ambiguity and overstatement of the task cases, especially regarding the independence of RTL designs.

We have added clarification to the paper in the Introduction Section on Page 2: “Specifically, the 17K+ cases originate from 9000+ RTL deisgns. We construct 16K+ module-level task cases based on 9K+ RTL modules, and construct 2000+ project-level task cases from 70+ RTL projects.” We have three main task types: Type 1: generation, Type 2: completion, and Type 3: debugging. For Type 1 (generation), we strictly control the RTL design for each task case to ensure independence. For the following Type 2: completion and Type 3: debugging tasks, one original RTL design contributes only 2-3 task cases.

During data collection and construction (as introduced in Step 1: Collect and extract designs in Section 4. RTL-BenchLS Generation Process), we strictly control source diversity and design size by removing duplicate designs and balancing lines of code (LoC) in the dataset. We also remove very similar designs based on the similarity scores, as illustrated by Figure 8 in Appendix Section A.3 Module-Level Design Filtering.

(4.1) Paper revision

We add an introduction section on page 2:

“ Specifically, the 17K+ cases originate from 9000+ RTL deisgns. We construct 16K+ module-level task cases based on 9K+ RTL modules, and construct 2000+ project-level task cases from 70+ RTL projects.”

We also add details in Step 3. Construct task cases in Section 4. RTL-BenchLS Generation Process: “Specifically, one RTL design contributes only one task case for Type 1: design generation, and 2-3 cases for Type 2 and Type 3.”

Reviewer_L2hnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RTL-BenchLS, a benchmark for evaluating LLMs and agents on RTL generation, completion, and debugging. The benchmark contains over 17,000 tasks organized across 12 categories, spanning module-level and project-level designs under direct and round-trip protocols. The proposed round-trip protocol requires models to translate RTL into an intermediate representation and subsequently reconstruct or repair the design. The authors evaluate 8 models using formal equivalence checking, simulation, and task-specific evaluation metrics.

The paper addresses an important limitation of existing RTL benchmarks, particularly their focus on relatively small, module-level tasks. However, because of the paper derives from previously built datasets, the benchmark task diversity, the construct validity of the round-trip protocol, and the consistency of evaluation metrics become naturally problematic. While the dataset could be a useful community resource, the paper does not yet sufficiently establish that its scores reliably reflect real-world hardware engineering capabilities. That is, if concerns about data contamination, benchmark diversity, and the validity of formal equivalence as a correctness oracle are convincingly addressed during rebuttal, I would be willing to raise to acceptance.

Critical strengths
  1. The proposed taxonomy covers a broad range of tasks and presents an useful verification infrastructure: 3.5K module cases from 705 designs, 50 projects, and around 800 project debugging cases. These are real project-level agentic tasks, real GitHub issues separated into functional vs. nonfunctional by formal checks, and an anonymous release link is included.

  2. The benchmark construction pipeline and multiple model baselines as well as their analysis across design sizes and task types represent meaningful work. In addition, the project level experiments reveal substantial limitations in existing agents.

Critical weaknesses
  1. The benchmark advertises 17K+ cases, but 9,057 of its 10,028 module designs come from four existing RTL datasets used for LLM development and training (Table 4). These designs average only 63 LoC, and multiple tasks are derived from the same designs. The paper provides no convincing evaluation of data contamination, particularly for the RTL-specialized models in Table 3. Consequently, the benchmark's effective diversity, independence, and claimed increase in design complexity are insufficiently established.

  2. The paper treats equivalence to a single reference implementation as functional correctness. However, a specification may admit multiple valid microarchitectures, including different state encodings, register boundaries, or permitted latencies. Whether such implementations pass LEC/SEC depends on the equivalence setup. The paper provides insufficient detail about reset assumptions, state correspondence, latency alignment, and interface mapping.

Questions for authors
  1. How many genuinely independent RTL designs underlie the 17K+ task instances? Can the authors report performance grouped by source design and repository rather than individual task instances?

  2. What evidence demonstrates that the round-trip protocol measures hardware reasoning ability rather than sensitivity to artificial information bottlenecks?

  3. How were the correctness and coverage of specifications, testbenches, and formal verification environments independently validated? What measures prevent contamination from existing public RTL datasets?

5 comments
AuthorsOpenReview ↗

Response (1/4) to Reviewer L2hn

Thank you for acknowledging our core contributions. In the following, we would like to address your concerns and answer your questions in four main sections. Specifically, sections (1) benchmark case independence, (2) data contamination, and (3) specification validation process are intended to address your concerns. And (4) discussion about the reasoning ability aims to demonstrate the reasoning steps during round-trip tasks.

(1) Benchmark case independence

We apologize for the unclear introduction of the task cases, especially regarding the independence of RTL designs.

We have added clarification to the paper in the Introduction Section on Page 2: “Specifically, the 17K+ cases originate from 9000+ RTL deisgns. We construct 16K+ module-level task cases based on 9K+ RTL modules, and construct 2000+ project-level task cases from 70+ RTL projects.” We have three main task types: Type 1: generation, Type 2: completion, and Type 3: debugging. For Type 1 (generation), we strictly control the RTL design for each task case to ensure independence. For the following Type 2: completion and Type 3: debugging tasks, one original RTL design contributes only 2-3 task cases.

During data collection and construction (as introduced in Step 1: Collect and extract designs in Section 4. RTL-BenchLS Generation Process), we strictly control source diversity and design size by removing duplicate designs and balancing lines of code (LoC) in the dataset. We also remove very similar designs based on the similarity scores, as illustrated by Figure 8 in Appendix Section A.3 Module-Level Design Filtering.

(1.1) LoC discussion

In Weakness 1, the reviewer mentioned that the designs average only 63 LoC. We acknowledge the low LoC. This is mainly because we include considerably more relatively small designs, which lowers the average LoC. But we still have a lot more large designs than existing benchmarks, as shown in Figure 2 on page 4.

(1.2) Performance grouped by source design

In {Question 1.}, the reviewer asks for performance grouped by source design. Because the task cases are generally independent, in Type: generation, all cases are from independent RTL designs. In Type 2 and Type 3, only 2-3 cases may share the same source design. So it may be hard to report performance by source design. For this reason, we present performance analysis mainly grouped by LoC (the content is in the original Section 6.1 and has been moved to Appendix Section C.1 on page 20 in the updated manuscript).

(1.3) Paper revision

We add an introduction section on page 2:

“ Specifically, the 17K+ cases originate from 9000+ RTL deisgns. We construct 16K+ module-level task cases based on 9K+ RTL modules, and construct 2000+ project-level task cases from 70+ RTL projects.”

We also add details in Step 3. Construct task cases in Section 4. RTL-BenchLS Generation Process: “Specifically, one RTL design contributes only one task case for Type 1: design generation, and 2-3 cases for Type 2 and Type 3.”

AuthorsOpenReview ↗

Response (2/4) to Reviewer L2hn

(2) Data contamination

We appreciate the reviewers’ insightful concerns because our benchmark is derived from the open-source dataset and GitHub/OpenCores projects. We apologize for not providing enough data contamination analysis in the paper.

We address the reviewers’ concerns as follows: (2.1) data contamination analysis and (2.2) discussion on data contamination.

(2.1) Data Contamination Analysis

We add a detailed data contamination analysis and compare the contamination risk of our cases against existing benchmarks like RTLLM and VerilogEval. The additional results are shown in Table 4 and Figure 5 in the additional Section 6.2 Data Contamination Analysis

Specifically, following the overfitting detection methodology introduced by the work RTL-BenchMT, we further conduct data contamination analysis on Direct Module-level Generation task cases. Specifically, we rewrite one variant for each specification. Each specification variant uses a different description style but preserves all semantics of the original. Figure 5 shows an illustration based on the QiMeng-SALV in VerilogEval. In this matrix, FN means the model passes the original specification but fails the variant. We take FN as an indicator of overfitting due to data contamination in the benchmark cases.

Table 4 on page 9 summarizes the measured performance changes and FN rates. If overall performance decreases, the model may also be at risk of data contamination. From the table, our current benchmark cases have relatively low contamination risk.

(2.2) Discussion on Data Contamination

We acknowledge the reviewers’ concern about data contamination. This is a long-standing challenge for all benchmarks that are public. RTL code is relatively special because the common types of RTL deisgns are limited [1, 2], so current cutting-edge LLMs should have mostly seen all design types and similar RTL designs.

But exposure to similar RTL designs does not ensure promising performance. For example, pretrained LLMs are usually limited in downstream tasks, such as specification-to-RTL generation, even though they have seen RTL designs during pretraining. LLMs still need supervised fine-tuning on {specification, RTL} (input, output) pairs to achieve promising performance on existing benchmarks like RTLLM and VerilogEval. As demonstrated by works like RTLCoder, CraftRTL, CodeV-R1, QiMeng-SALV, etc.

So we think the contamination risk may mainly come from the leakage of the task cases in the benchmark, where both the input and output are leaked. LLMs can overfit the benchmark by simply memorizing the inputs and outputs.

Our experiments in Table 3: Comparison with module-level benchmarks on cases and task design also provide meaningful evidence. The LLMs, including both general and downstream LLMs, show promising results on the specification-to-RTL tasks but much lower performance on the round-trip tasks, even with identical designs. The main reason may be that these models have not yet been well trained on similar tasks.

In our benchmark, the Direct Module-Level RTL Generation task may be at the highest risk, because the task cases are derived from the existing dataset, on which many models might already be trained. However, during our strict collection and validation process, we refined and modified all specifications to remove ambiguity. After this refinement, the cases differ from the original dataset cases. Our (2.1) contamination analysis also further demonstrates a low contamination risk.

[1] RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model” ASP-DAC 2024.

[2] A Survey of Research in Large Language Models for Electronic Design Automation.

AuthorsOpenReview ↗

Response (3/4) to Reviewer L2hn

(3) Specification validation process

The reviewer raises a concern, especially about the equivalence of the specification and the reference RTL code. We appreciate the reviewer’s professional question and apologize for the unclear explanation in our paper. An unclear specification may correspond to multiple valid implementations. To avoid this situation, during our case collection and validation process, we will strictly check the quality of the specifications to eliminate ambiguities.

Specifically, for ``module-level`` cases, the specifications clearly specify a module-level RTL module. For ``project-level`` cases, we actually further fix the configuration before the evaluation process so that for each task case, there is only an explicit golden reference implementation.

We further add clarifications in Step 2 of Section 4. RTL-BenchLS Generation Process: “Specifically, for module-level designs, the detailed behavior, like reset conditions and module interfaces, is clearly specified, so that each specification clearly refers to one RTL module. For project-level designs, one specification may correspond to several valid RTL implementations, with different configurations. During project-level task evaluation, the specification and one specific configuration will be explicitly given to the agent so that there is only one specific reference RTL implementation for verification.”

We add two additional examples in Appendix Section A.4: Figures 9 and 10 show examples of module-level and project-level specifications. The detailed operation should be clearly described in the specification so that there is no ambiguity.

For testbenches and formal verifications. We will verify against the same modules and mutants to confirm that the evaluation pipeline can detect both correct and functionally incorrect RTL implementations.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/4)

We appreciate the reviewers' professional comments and insightful questions. We are very encouraged by your kind review.

Specifically, all reviewers acknowledged our core contributions to the benchmark, including task coverage and benchmark scale. The reviewers also acknowledged the design quality and benchmark expansion methodology. The reviewers’ concerns mainly focus on RTL design independence and data contamination risks. The reviewers also raised further questions about the experimental results, mainly related to our proposed new round-trip reasoning tasks.

In this general response, we briefly answer the common questions and outline the updates to the paper. We will answer the detailed questions in the comments to each reviewer.

(1) Benchmark case independence

Reviewers L2hn and zzso ask for clarification on the independence of benchmark cases. We apologize for the ambiguity and overstatement. We have added clarification to the paper in the Introduction Section on Page 2: “Specifically, the 17K+ cases originate from 9000+ RTL deisgns. We construct 16K+ module-level task cases based on 9K+ RTL modules, and construct 2000+ project-level task cases from 70+ RTL projects.” We have three main task types: Type 1: generation, Type 2: completion, and Type 3: debugging. For Type 1 (generation), we strictly control the RTL design for each task case to ensure independence. For the following Type 2: completion and Type 3: debugging tasks, one original RTL design contributes only 2-3 task cases.

During data collection and construction (as introduced in Step 1: Collect and extract designs in Section 4. RTL-BenchLS Generation Process), we strictly control source diversity and design size by removing duplicate designs and balancing lines of code (LoC) in the dataset. We also remove very similar designs based on the similarity scores, as illustrated by Figure 8 in Appendix Section A.3 Module-Level Design Filtering.

AuthorsOpenReview ↗

General Response (2/4)

(2) Data contamination

Reviewers L2hn and zzso raise concerns about data contamination because our benchmark is derived from open-source datasets and GitHub/OpenCores projects. We appreciate the reviewers' comments and apologize for not providing enough data contamination analysis in the paper. We address the reviewers’ concerns as follows: (2.1) data contamination analysis and (2.2) discussion on data contamination.

(2.1) Data Contamination Analysis

We add a detailed data contamination analysis and compare the contamination risk of our cases against existing benchmarks like RTLLM and VerilogEval. The additional results are shown in Table 4 and Figure 5 in the additional Section 6.2, Data Contamination Analysis

Specifically, following the overfitting detection methodology introduced by the work RTL-BenchMT, we further conduct data contamination analysis on Direct Module-level Generation task cases. Specifically, we rewrite one variant for each specification. Each specification variant uses a different description style but preserves all semantics of the original. Figure 5 shows an illustration based on the QiMeng-SALV in VerilogEval. In this matrix, FN means the model passes the original specification but fails the variant. We take FN as an indicator of overfitting due to data contamination in the benchmark cases.

Table 4 on page 9 summarizes the measured performance changes and FN rates. If overall performance decreases, the model may also be at risk of data contamination. From the table, our current benchmark cases have relatively low contamination risk.

(2.2) Discussion on Data Contamination

We acknowledge the reviewers’ concern about data contamination. This is a long-standing challenge for all benchmarks that are public. RTL code is relatively special because the common types of RTL deisgns are limited [1, 2], so current cutting-edge LLMs should have mostly seen all design types and similar RTL designs.

But exposure to similar RTL designs does not ensure promising performance. For example, pretrained LLMs are usually limited in downstream tasks, such as specification-to-RTL generation, even though they have seen RTL designs during pretraining. LLMs still need supervised fine-tuning on {specification, RTL} (input, output) pairs to achieve promising performance on existing benchmarks like RTLLM and VerilogEval. As demonstrated by works like RTLCoder, CraftRTL, CodeV-R1, QiMeng-SALV, etc.

So we think the contamination risk may mainly come from the leakage of the task cases in the benchmark, where both the input and output are leaked. LLMs can overfit the benchmark by simply memorizing the inputs and outputs.

Our experiments in Table 3: Comparison with module-level benchmarks on cases and task design also provide meaningful evidence. The LLMs, including both general and downstream LLMs, show promising results on the specification-to-RTL tasks but much lower performance on the round-trip tasks, even with identical designs. The main reason may be that these models have not yet been well trained on similar tasks.

In our benchmark, the Direct Module-Level RTL Generation task may be at the highest risk, because the task cases are derived from the existing dataset, on which many models might already be trained. However, during our strict collection and validation process, we refined and modified all specifications to remove ambiguity. After this refinement, the cases differ from the original dataset cases. Our (2.1) contamination analysis also further demonstrates a low contamination risk.

[1] RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model,” ASP-DAC 2024.

[2] A Survey of Research in Large Language Models for Electronic Design Automation.

AuthorsOpenReview ↗

General Response (3/4)

(3) Round-Trip Reasoning Analysis

All of the reviewers have insightful questions about the round-trip reasoning analysis. Specifically, Reviewer L2hn asks how the round-trip process effectively evaluates LLMs' reasoning ability, Reviewer zzso asks for a more detailed analysis that isolates the reasoning steps, and Reviewer uKvs asks for more detailed technique configurations for additional analysis in Appendix Table 8.

We will address each reviewer’s concern specifically in each reviewer’s comments.

(4) Improvement of scoring rules for non-functional issue fixing.

Reviewers zzso and uKvs point out that the evaluation scoring based on ``CommitMatch ’’ for non-functional issue fixing is insufficient, especially as it risks rejecting alternative valid solutions. We thank the reviewers for pointing out this weakness and helping us improve our evaluation pipeline. To alleviate this limitation, we add a new Checklist-based judgment. Specifically, we list a set of checkpoints that the commit should pass. We then use the strongest model, such as GPT-6.1 Sol, to conduct a judgment based on the checklist. We provide additional results in Table 9 of Appendix Section C.3. We manually verified and found that around 5% to 8% of alternative solutions are rejected by the strict CommitMatch method. We will integrate this additional evaluation method into our final public benchmark pipeline.

(5) Clarified Experimental Setups

Reviewers zzso and uKvs point out inconsistencies in the experimental results for DeepSeek in our paper. We appreciate the reviewers’ reminders and apologize for the inconsistent results. Specifically, the main results in Table 2 for each model are at n=3 repetitions. The results for DeepSeek in Table 3 are the intermediate results at n=2. We have updated the DeepSeek results in Table 3 to the final stage.

Reviewer uKvs also requests additional experimental details, which we will provide in our response.

AuthorsOpenReview ↗

General Response (4/4)

List of modifications

Thank you again for your time and valuable suggestions again. We list the revisions to the paper below.

(1) We add clarification of independent RTL designs and add a fair comparison with existing benchmark agentic RTL tasks in the introduction section on page 2.

(2) We revise the inaccurate introduction of HWE-Bench and add a fair comparison in Section 3.1 on page 4.

(3) We add the introduction of the new evaluation method {checklist-based} judgment in Section 3.1 on page 5.

(4) We add clarification on how to validate the design specification, especially for the specifications that may have multiple valid implementations, in Steps 2 and 3 of Section 4 on page 6. We also introduce the data contamination analysis post-step after step 4 in Section 4 on page 6.

In the experimental part:

(1) We add detailed experimental settings in Section 5.1 on page 7.

(2) We fix the result inconsistency of DeepSeek in Table 3 on page 8.

(3) We add a data contamination analysis as a new Section 6.3 on page 9.

(4) We add a diagnostic experiment to measure the quality of the intermediate representation in Section 6.3 on page 10.

(5) Due to space limitations, we move the original Section 6.1: benchmarking results across design size on page 8 to Appendix C as a new Appendix subsection C.1 on page 20.

(6) We add a new appendix section {A.4: examples of specifications} with Figures 9 and 10 to demonstrate examples of specifications.

(7) We add a clarification that the token analysis is with one repetition (n=1) on page 22.

(8) We add detailed results for Figure 17 in Appendix Section C.3 for Figure 17.

(9) We add new results on non-functional issue-fixing based on checklist-based judgment in Table 9 in Appendix Section C.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.