acceptodds
Withdrawn from ICLR 2027

ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Large Language Models (LLMs) show significant potential in hardware engineering, cur- rent benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs’ performance in real industrial workflows. To address this gap, we propose a comprehensive bench- mark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the- art Claude-4.5-opus achieving only 30.74% on Verilog generation and 13.33% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at

Primary area:
datasets and benchmarks
Keywords:
AI4Chip, RTL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gKjFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ChipBench evaluates LLMs on three hardware tasks: 44 Verilog generation tasks (self-contained modules, 6 hierarchical modules and 9 CPU components), 89 debugging tasks with injected bugs, and 132 reference model tasks in Python, SystemC and CXXRTL. It also releases a toolbox that turns Verilog datasets into verified reference models. The best pass@1 is 30.74% on Verilog generation and lower on Python reference models.

Critical strengths
  • Useful task coverage. Debugging and reference model generation are practical tasks that most Verilog benchmarks miss, and differential testing against Verilated golden RTL is a sensible way to check reference models.
  • Broad evaluation. 13 models plus MAGE are tested, with pass@k and API cost reported.
Critical weaknesses

My two main concerns are that the benchmark is too small for stable measurements, and that some claims about realism and novelty go beyond the evidence.

  • Too few tasks per category. CPU IP has 9 tasks and non-self-contained has 6, so one task moves the score by 11.1 or 16.7 points. State-machine bugs have only 6 cases. Many differences in Tables 2-4 come down to one or two examples, and no uncertainty is reported.
  • Realism is overstated. Most self-contained tasks are OJ-style counters, sequence detectors and FSMs, and the CPU parts come from public projects. The mean cell count is also driven by a few large designs (LCM with 7,409 cells, regfile with 3,122). Medians would be more informative.
  • Some results need diagnosis. Claude Sonnet 4.5 gets 2.71% and Gemini 2.5 Pro 0.83% on debugging, below Llama 3.1 8B (7.77%). Gemini 2.5 Pro also gets 1.11% on generation despite very high output token usage. This could be truncation or extraction failure.
  • SystemC and CXXRTL results are missing.
  • Debugging vs. generation is not a fair comparison. Debugging starts from nearly correct code, so higher scores are expected. The one-shot setting pastes raw waveforms into the prompt, which mainly shows that models do not benefit from this format.
  • The toolbox is not validated as training data. 2,206 Python models are generated, but there is no experiment showing that training on them helps.
Questions for authors

Q1. What causes the very low scores of Claude Sonnet 4.5 and Gemini 2.5 Pro? Did you check for truncation, formatting or extraction failures?

Q2. How are the interface and cycle-level semantics of a Python reference model specified to the LLM? How many failures come from interface or timing mismatches rather than wrong functionality?

Q3. Is there any SystemC and CXXRTL results? (If the benchmark contribution includes all three reference-model languages)

Reviewer_EahHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChipBench, a benchmark for evaluating LLMs on three chip-design tasks: (1) Verilog generation from 44 modules (29 self-contained, 6 non-self-contained, 9 CPU-IP); (2) Verilog debugging over 89 cases built by manually injecting one of four bug types; (3) reference-model generation across Python, SystemC, and CXXRTL, verified by differential testing against Verilator-compiled golden RTL over 1,000+ random stimuli.

Critical strengths
  1. the critique of existing Verilog benchmarks (saturation, self-contained-only, contest-style sources, generation-only focus) is accurate; debugging and reference-model evaluation are genuine gaps. 2, VerilogEval-compatible harness, directed + constrained-random testbenches, documented model versions/APIs, released code; the reset-handling design (Fig. 3) is thoughtful.
  2. This paper has broad model coverage (13 models + MAGE) with cost accounting; honest reporting of low scores.
Critical weaknesses
  1. VerilogEval harness with harder problems; single-fault manual injection for debugging; textbook differential testing (used for decades, and already in AutoBench/CorrectBench/PRO-V); the toolbox is plain rejection sampling. Contribution is essentially curation effort, below the ICLR bar.
  2. 6 and 9 modules in the two most novel categories make per-category rankings meaningless (one module ≈ 11 points of pass@1); no confidence intervals or significance tests. The "3.8× / 13.9×" complexity claims are driven by a few large modules.
  3. Modules come from public online-judge platforms (recognizably HDLBits-style exercises) and open-source GitHub CPU IPs; no n-gram overlap or decontamination study anywhere — a basic requirement for a benchmark whose value is a trustworthy measure.
Questions for authors

Why are no SystemC/CXXRTL results reported despite heterogeneity being a core claim? What decontamination/overlap analysis was done for OJ and open-source-IP modules? What statistical justification supports rankings on 6–9 modules?

Reviewer_oopfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChipBench, a benchmark for LLMs in AI-aided chip design with three task families: Verilog generation (44 cases), Verilog debugging (89 manually injected bug cases), and reference- model generation (132 samples spanning Python, SystemC, and CXXRTL). The authors also release a heterogeneous test engine and a toolbox for generating verified reference-model training data. The benchmark uses compilation and functional comparison against golden Verilog implementations, with directed and random stimuli. Experiments cover 13 language models and report pass@1/5/10, debugging modes with and without waveform files, cost, and a small-scale validation of the data-generation pipeline.

Critical strengths
  1. Broader task coverage. Bringing debugging and reference-model generation into the same benchmark is consequential because these tasks are closer to verification workflows than isolated code completion.

  2. Useful evaluation infrastructure. The compilation, heterogeneous comparison, reset handling, and training-data generation toolbox could become a practical foundation for future work if the released artifacts are complete and maintainable.

Critical weaknesses
  1. Insufficient evidence for representativeness and robustness. The cases are selected from OJ and open-source CPU projects, but the paper gives limited information about inclusion/exclusion criteria, circuit-type and difficulty distributions, project and author overlap, licensing, or train/test contamination. Claims of industrial realism therefore remain largely asserted, and the small case count makes aggregate percentages sensitive to selection.

  2. The analysis does not support the strongest conclusions. The paper mainly reports aggregate pass rates and uses low scores to argue that prior benchmarks are saturated and current models are weak at hardware modeling. It does not provide enough case-level failure analysis to distinguish task difficulty from prompt ambiguity, test-harness artifacts, or model limitations. Directed/random testing and high-quality prompts are described but not quantitatively characterized

Questions for authors
  1. Please provide a case-level manifest for all 44/89/132 cases: source project and license, circuit/module type, size and difficulty, selection and exclusion criteria, test count, and train/test contamination checks. What fraction of the benchmark belongs to each circuit type, and why should this distribution be considered representative of industrial work?

  2. You describe the prompts as high-quality and the testbenches as high-fidelity, with directed tests plus more than 1,000 random iterations. What quantitative evidence supports these claims (functional/branch coverage, corner-case coverage, mutation score, independent review, or false-pass analysis)? How many cases fail when evaluated with an independently designed testbench?

  3. The low pass rates show that models struggle on this selected suite, but what evidence shows that ChipBench is a stronger benchmark rather than simply a harder one? Please give case-level failure analysis: which modules and behaviors fail, what error types occur, and how do these failures differ from prior benchmarks?

  4. For the 10,000 CodeV-R1 samples, please report the circuit-type, size, complexity, and duplicate distributions for the 2,206 verified Python models, together with the corresponding distributions for all inputs and failed samples. How was “high-quality” assessed beyond passing the generated testbench, and were any samples manually reviewed?

  5. When a generated reference model fails, what exact error information is fed back, how is the repair prompt constructed, and how do you prevent repeated or superficial fixes? Please report pass rates by iteration and failure category, not only the final total.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.