acceptodds
Under review as a conference paper at ICLR 2027

TurboRTL: Rigorous Evaluation of LLMs and Agents for RTL Optimization

Abstract

Although large language models (LLMs) have demonstrated growing capabilities in Verilog code generation, existing research largely prioritizes syntactical and functional correctness, overlooking the critical hardware design goal of optimizing circuit efficiency. Consequently, the field lacks a well-established benchmark to rigorously assess the ability of LLMs to optimize register-transfer level (RTL) designs. To this end, we introduce **TurboRTL**, a comprehensive Verilog code optimization benchmark constructed via an automated pipeline that utilizes equivalence graph (e-graph) rewriting to systematically identify designs with significant optimization potential. Furthermore, reflecting the reality that hardware engineers optimize designs through iterative refinement rather than single-shot attempts, our evaluation framework extends beyond individual models to incorporate two tool-augmented interaction paradigms, specifically population-based test-time search and experience-driven agentic workflows. We demonstrate that while individual models show initial promise, these advanced agentic strategies are essential for tackling the complexities of this domain. By providing both a high-quality benchmark and a diverse evaluation pipeline, this work establishes a critical foundation for advancing automated RTL optimization.

Primary area:
datasets and benchmarks
Keywords:
Verilog Code Optimization, Benchmark Construction and Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_az5ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TurboRTL is a 288-design Verilog benchmark that keeps only designs where e-graph rewriting finds an equivalent variant with lower ADP after DC compile_ultra. It evaluates 19 LLMs single-shot and compares evolutionary search with an agentic workflow.

Critical strengths

The paper targets an important gap, optimization that holds up after real synthesis rather than just functional correctness, and the automated pipeline scales beyond hand-curated benchmarks.

Critical weaknesses
  1. E-graph RTL optimization (ROVER, SEER, E-Syn) and LLM-guided e-graph rewriting with real PPA feedback (ASPEN, MLCAD 2025) already exist, so using a fixed-rule e-graph as a filter is a small step. How does your approach differ from ASPEN, and how does d_egg compare with ASPEN-optimized designs on the same inputs?

  2. The rewrite rules are a black box. The 76 rules come from an unpublished submission and are never listed, yet they determine every target in the benchmark. Will you release and categorize the rule set?

  3. The cost model is undefined. ADP is a post-synthesis metric, and delay is a max-over-paths quantity, so neither can be an additive per-e-node cost before extraction. How is each e-node's cost, especially delay, assigned during extraction, and how well does that estimate agree with post-synthesis ADP?

  4. Extraction is heuristic. You use egg's greedy extraction plus DC selection among a few iteration snapshots, not ILP, and report no e-graph sizes or saturation limits. How much does d_egg improve with ILP or PPA-guided extraction, and how many designs actually reach saturation?

  5. Two toolchains, one metric. Targets are validated with DC, but evaluation uses Yosys+ABC, and eff@k shifts by up to 15 points between the two. Which flow defines ADP(d_egg) in eff@k, and is eff clipped when a model beats d_egg?

  6. "Complex" designs are mostly simple. The average complexity is 0.91 out of 5 and designs average 129 lines, so most are trivial or simple sequential. How many designs fall at each complexity level, and are any at C≥4? Could you try NVDLA or some OpenTitan block level designs?

  7. The designs come from RTLCoder, MG-Verilog, and DeepRTL, the same corpora used to train several of the evaluated models. Could memorization explain the identity-copying behavior of the Verilog-specialized models? Could the author provide the results on an unseen testing set designs?

Questions for authors
  1. E-graph RTL optimization (ROVER, SEER, E-Syn) and LLM-guided e-graph rewriting with real PPA feedback (ASPEN, MLCAD 2025) already exist, so using a fixed-rule e-graph as a filter is a small step. How does your approach differ from ASPEN, and how does d_egg compare with ASPEN-optimized designs on the same inputs?

  2. The rewrite rules are a black box. The 76 rules come from an unpublished submission and are never listed, yet they determine every target in the benchmark. Will you release and categorize the rule set?

  3. The cost model is undefined. ADP is a post-synthesis metric, and delay is a max-over-paths quantity, so neither can be an additive per-e-node cost before extraction. How is each e-node's cost, especially delay, assigned during extraction, and how well does that estimate agree with post-synthesis ADP?

  4. Extraction is heuristic. You use egg's greedy extraction plus DC selection among a few iteration snapshots, not ILP, and report no e-graph sizes or saturation limits. How much does d_egg improve with ILP or PPA-guided extraction, and how many designs actually reach saturation?

  5. Two toolchains, one metric. Targets are validated with DC, but evaluation uses Yosys+ABC, and eff@k shifts by up to 15 points between the two. Which flow defines ADP(d_egg) in eff@k, and is eff clipped when a model beats d_egg?

  6. "Complex" designs are mostly simple. The average complexity is 0.91 out of 5 and designs average 129 lines, so most are trivial or simple sequential. How many designs fall at each complexity level, and are any at C≥4? Could you try NVDLA or some OpenTitan block level designs?

  7. The designs come from RTLCoder, MG-Verilog, and DeepRTL, the same corpora used to train several of the evaluated models. Could memorization explain the identity-copying behavior of the Verilog-specialized models? Could the author provide the results on an unseen testing set designs?

Reviewer_NuxAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TurboRTL, a benchmark for LLM-based RTL optimization, i.e., transforming a Verilog design into a functionally equivalent variant with lower area-delay product. The benchmark is built by an automated pipeline: mining ~119k Verilog files from existing datasets, deduplicating and synthesis-filtering to ~60k, then applying e-graph equality saturation to retain only designs with a formally verified, strictly-lower-ADP variant that survives Synopsys DC compile_ultra — 3,414 validated candidates, clustered and sampled down to 288 designs. Evaluation uses progressive metrics across three paradigms under a 20-query budget.

Critical strengths
  1. Using e-graph equality saturation as a validator — retaining only designs where a formally verified, strictly lower-ADP variant survives Synopsys DC compile_ultra — guarantees that every sample has machine-checked, non-trivial optimization potential. This directly fixes the core flaw of RTLRewriter and the scale limits of manually curated Pluto/RTL-OPT, and it is a genuinely reusable construction methodology.
  2. The progressive syn@k → fun@k → opt@k → eff@k metrics cleanly separate syntax, functionality, and optimization magnitude; the paradigm comparison controls query budget (with a token-budget check in Appendix O); the Spec-to-RTL control (Appendix N) isolates optimization ability from generation ability; and cross-toolchain validation (Appendix M) plus per-problem standard errors (Appendix J) are more rigorous than the norm for benchmark papers.
  3. The evaluation spans 17–19 models across proprietary, open-weight, and hardware-tuned families, and surfaces useful observations: reasoning variants improve dramatically (Grok-4-1-Fast-Reasoning 63.65% fun@1 vs. 15.17% base), hardware-tuned models degenerate into identity mappings, and evolutionary search trades synthesizability for optimization depth.
Critical weaknesses

First, every component in this paper is off-the-shelf: equality saturation, MinHash, embeddings + HDBSCAN, AlphaEvolve adaptation, generic LLM-agent loop. No new model, objective, algorithm, or representation. The findings are expected confirmations rather than field-shaping insights. Second, "Agentic strategies are essential" rests on subset evaluations (token-normalized experiment: 51 tasks, 2 models), with no comparison to existing agentic RTL systems (VerilogCoder, MAGE) and no human-expert baseline to calibrate the residual gap. Third, the metrics are bias. eff@k measures the fraction of the gap between d and d_egg closed — but d_egg comes from exactly the local algebraic rewriting LLMs reproduce from textbooks; the metric rewards rewrite-style similarity, not optimization quality. ADP is a single scalar with no area/delay Pareto analysis.

Questions for authors
  1. Any decontamination/overlap analysis between the corpus and evaluated models' training data?
  2. What fraction of models' successful improvements are rewrites the 76-rule e-graph can itself derive?
  3. Full 288-design results for the search/agentic paradigms — which subset was used and how selected?
  4. Human-expert baseline on even 20–30 designs to calibrate the d_egg gap?
Reviewer_rUvCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TurboRTL, a benchmark for evaluating LLMs and agents on RTL (Verilog) optimization.

Existing Verilog benchmarks like VerilogEval and RTLLM mainly test whether generated code is syntactically valid and functionally correct, while prior RTL optimization benchmarks either inject artificial redundancies that commercial synthesis tools can easily optimize away, or rely on small hand-crafted datasets.

To build a larger benchmark, the authors use equality saturation (e-graph rewriting) to filter roughly 119k raw Verilog modules down to 288 designs, and evaluate 19+ models under single-shot generation, evolutionary search, and an agentic workflow.

Critical strengths
  • Optimizing RTL for post-synthesis Area-Delay Product (ADP) with EDA tool feedback in the loop is a practical and relevant problem in hardware design.

  • The experimental evaluation is broad, covering 19+ open-weight, proprietary, code-specialized, and Verilog-finetuned models, with detailed breakdowns by complexity, category, and synthesis tool in the appendix.

Critical weaknesses

W1. Bugs in the equivalence checking scripts in the anonymous repository invalidate both the benchmark construction and model evaluation.

Code bugs:

  1. Treating "Networks are UNDECIDED", "timeout", and empty outputs as True (functionally equivalent): run/single_shot_eval/benchmark_sample_yosys_eval.py, run/agentic-workflow/ppa.py, run/openevolve-optimization/evaluator.py
  2. Combining combinational (cec) and sequential (dsec) equivalence checks with logical OR (meaning that even when cec returns False, an UNDECIDED result from dsec still marks the circuit as equivalent): run/single_shot_eval/model_outputs_yosys_eval.py, run/single_shot_eval/stats.py, run/agentic-workflow/ppa.py, run/openevolve-optimization/evaluator.py

Consequences:

  1. 66 out of the 288 benchmark samples in final_benchmark_samples/final_benchmark_samples.json were admitted into the benchmark with cec=False and dsec=True.
  2. Model fun@k and opt@k scores are inflated: since LLM outputs are evaluated using the same scripts, any broken design that causes dsec to time out or return UNDECIDED is counted as functionally equivalent.

W2. Inconsistent definition of the eff@k metric.

The definition of the efficiency score in the main text contradicts the appendix and the code. Section 4.1 (Eq. 3) defines e_i = (ADP(d) - ADP(s_i)) / (ADP(d) - ADP(d_egg)) without an upper bound, and line 305 states that 1.0 corresponds to matching d_egg. However, Appendix P (Eq. 7) and stats.py clip this value at 1.0 using min(1, ...). Meanwhile, the paper never reports the actual percentage reduction in ADP relative to the original design d.

W3. Mismatch between the synthesis tools used for data filtering and evaluation.

Section 3.1 stresses that benchmark candidates are filtered using commercial Synopsys Design Compiler (compile_ultra) so that trivial redundancies are already removed. Yet the main results in Tables 1 and 2 are evaluated using open-source Yosys + ABC. When the authors switch back to Design Compiler in Table 7 (Appendix M), eff@10 drops substantially across the top models, suggesting that much of the improvement reported in Table 1 comes from redundancies that a commercial synthesizer would eliminate automatically.

W4. Impossible values and copy-paste errors in the result tables.

  1. opt@10 > fun@10 in Table 19 (Page 25): For Llama-4-maverick at Complexity = 3, fun@10 is 8.33% while opt@10 is 8.35%. By definition, a design must be functionally correct to be counted as optimized, so opt@k cannot exceed fun@k.
  2. opt@k = 0.00% but eff@k > 0 in Table 25 (Page 27): For CodeV-DS-6.7B on the processor category, opt@1, opt@5, and opt@10 are all 0.00%, but eff@1, eff@5, and eff@10 are 0.04%, 0.19%, and 0.37% (which appear to be copied from Grok-4-1-Fast in Table 13; by definition, eff@k must be 0 when opt@k is 0).
Questions for authors
  1. Why does _equiv_from_text treat "Networks are UNDECIDED", "timeout", and empty outputs as True under cec OR dsec? If you strictly reject UNDECIDED and timeout cases (and require cec=True), how many valid benchmark designs remain, and how do the results in Tables 1 and 2 change?
  2. Could you clarify the discrepancy between Eq. (3) in Section 4.1 and Eq. (7) in Appendix P, and report the actual average percentage reduction in ADP relative to d for each model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.