- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
ChipBench evaluates LLMs on three hardware tasks: 44 Verilog generation tasks (self-contained modules, 6 hierarchical modules and 9 CPU components), 89 debugging tasks with injected bugs, and 132 reference model tasks in Python, SystemC and CXXRTL. It also releases a toolbox that turns Verilog datasets into verified reference models. The best pass@1 is 30.74% on Verilog generation and lower on Python reference models.
Critical strengths
- Useful task coverage. Debugging and reference model generation are practical tasks that most Verilog benchmarks miss, and differential testing against Verilated golden RTL is a sensible way to check reference models.
- Broad evaluation. 13 models plus MAGE are tested, with pass@k and API cost reported.
Critical weaknesses
My two main concerns are that the benchmark is too small for stable measurements, and that some claims about realism and novelty go beyond the evidence.
- Too few tasks per category. CPU IP has 9 tasks and non-self-contained has 6, so one task moves the score by 11.1 or 16.7 points. State-machine bugs have only 6 cases. Many differences in Tables 2-4 come down to one or two examples, and no uncertainty is reported.
- Realism is overstated. Most self-contained tasks are OJ-style counters, sequence detectors and FSMs, and the CPU parts come from public projects. The mean cell count is also driven by a few large designs (LCM with 7,409 cells, regfile with 3,122). Medians would be more informative.
- Some results need diagnosis. Claude Sonnet 4.5 gets 2.71% and Gemini 2.5 Pro 0.83% on debugging, below Llama 3.1 8B (7.77%). Gemini 2.5 Pro also gets 1.11% on generation despite very high output token usage. This could be truncation or extraction failure.
- SystemC and CXXRTL results are missing.
- Debugging vs. generation is not a fair comparison. Debugging starts from nearly correct code, so higher scores are expected. The one-shot setting pastes raw waveforms into the prompt, which mainly shows that models do not benefit from this format.
- The toolbox is not validated as training data. 2,206 Python models are generated, but there is no experiment showing that training on them helps.
Questions for authors
Q1. What causes the very low scores of Claude Sonnet 4.5 and Gemini 2.5 Pro? Did you check for truncation, formatting or extraction failures?
Q2. How are the interface and cycle-level semantics of a Python reference model specified to the LLM? How many failures come from interface or timing mismatches rather than wrong functionality?
Q3. Is there any SystemC and CXXRTL results? (If the benchmark contribution includes all three reference-model languages)