acceptodds
Withdrawn from ICLR 2027

Bridging the Last Mile of Circuit Design: PostEDA-BENCH, a Hierarchical Benchmark for PPA Convergence and DRC Fixing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based agents are increasingly applied to the "last mile" of Electronic Design Automation (EDA): repairing residual sign-off Design Rule Check (DRC) violations and converging Power–Performance–Area (PPA) targets after tool runs. Existing EDA-LLM benchmarks, however, omit DRC fixing entirely and rely on flat hierarchies tied to a single toolchain. We introduce PostEDA-Bench, a hierarchical benchmark with tasks across DRC-Essential, DRC-Reasoning, PPA-Mono, and PPA-Multi, supported by EDA toolchains with machine-checkable evaluation. Across eight commercial and open-source LLMs under multiple agent scaffolds, we find that agents handle synthetic DRC-Essential and single-objective PPA-Mono reasonably well but degrade sharply on the more practical DRC-Reasoning (best SR ) and PPA-Multi (best SR ); vision augmentation consistently enhances DRC-Bench; and trade-off reasoning—rather than knob knowledge—is the dominant PPA-Multi bottleneck.

Primary area:
datasets and benchmarks
Keywords:
benchmark, EDA(Electronic Design Automation), agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3VtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PostEDA-Bench, a benchmark for evaluating LLM agents on DRC repair and PPA convergence after EDA tool runs. It contains 145 tasks covering basic and practical DRC repair, single-objective PPA optimization, and multi-objective trade-offs. The paper evaluates several models and agent frameworks, but I think some more experiments about the latest model are necessary.

Critical strengths
  1. The benchmark addresses an important and practically relevant problem in the EDA domain: resolving DRC violations and unmet PPA targets after traditional EDA flows.

  2. The paper makes good use of visual information and includes useful ablation experiments demonstrating its benefits for DRC repair.

  3. It also explicitly discusses whether the information used in the benchmark evaluation could have been included in the LLMs’ training data, and provides a reasonable explanation of why memorizing that information would not directly solve the tasks.

Critical weaknesses
  1. The interpretation of performance differences is limited by incomplete control and reporting of inference resources (for example, different token consumption, different inference time and API fees should be taken into consideration). Section E.2 specifies tool-action budgets, but the frameworks use different numbers of internal LLM calls and different optimization procedures. The reasoning effort or thinking budget for each model in Table 3 is also unclear. Although Section E.1 reports some costs and runtimes, these do not establish comparisons under matched resources.

  2. The evaluation does not sufficiently establish how challenging the benchmark remains for representative strong models and complete the available coding agents. In particular, it lacks evaluations of systems such as Codex, Claude Code or DeepSeek Harness. This limits the evidence supporting conclusions about current capability gaps, since low success rates may partly reflect limitations of the selected agent implementations.

Questions for authors

I appreciate the motivation and practical value of this benchmark. Evidence addressing these concerns would make me reconsider my current weak-rejection recommendation.

  1. Could the authors specify the reasoning settings and token limits used for each model, and provide a targeted resource-matched comparison or success-versus-cost analysis for representative configurations? Do the main comparative conclusions remain consistent under these conditions?

  2. Could the authors evaluate a representative strong model and a complete coding agent, or explain the limitations preventing such evaluations? A clearly defined subset covering the four task families would be useful.

Reviewer_xVxLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes PostEDA-Bench, a 145-task benchmark for evaluating LLM agents on post-EDA chip-design closure. It consists of DRC-Bench (70 tasks), which asks agents to fix design-rule violations in ASAP7 GDS layouts, and PPA-Bench (75 tasks), which asks agents to tune OpenROAD knobs (and RTL) to meet power, performance and area targets. Each part has a difficulty hierarchy (L1–L3), and the paper introduces a new partial-credit metric (NIS) for PPA. Nine LLMs are evaluated under several agent scaffolds (ReAct, Proposer–Critic, ORFS-Agent, etc.). Agents perform well on synthetic DRC and single-objective PPA tasks but degrade sharply on realistic DRC and multi-objective PPA tasks. However, the paper and released code alone do not sufficiently establish the benchmark's reliability or its practical value.

Critical strengths
  • The paper targets an interesting problem. If LLM agents can take over post-EDA closure work such as DRC repair and PPA tuning, it would be a meaningful help to the EDA domain.
  • It is one of the first benchmarks [1,2] to evaluate LLM agents on residual DRC repair after place-and-route, a task not covered by the LLM-EDA benchmarks compared in Table 1.

[1] EvoDRC: A Self-Evolving Agentic Framework for Automated DRC Violation Repair, arXiv:2607.20019, 2026.

[2] SCALE: Self-Supervised Constraint-Aware Layout GEneration for Local P&R DRV Fixing at Advanced Nodes, arXiv:2607.21850, 2026.

Critical weaknesses

W1. The paper's description of the benchmark is incomplete, and the released code does not contain the information needed to verify it.

  • Core modules fail with import errors, so neither the repository's results nor the paper's key claims can be reproduced or verified.
  • The paper and code do not provide enough information to reproduce task generation: the per-knob sweep ranges and generation scripts (stated as released in App. D.3) are missing, and design filtering is described only at a high level (Sec. 3, App. B).
  • Some released artifacts contradict the paper.
    • The paper says each DRC-Reasoning task is isolated to one violation (Sec. 3.1.3), yet drc_reasoning/L2/q5/top_cell.gds still contains 341 violations under the repository's own deck.
    • The prompt lists only one, while the agent's DRC tool returns all 341. The agent thus receives contradictory information, which may sharply degrade performance.
    • Five of the seven tasks built this way are in L2, exactly where performance drops sharply (2–18% SR).

W2. The experimental design does not adequately measure practical engineering ability, and the results may be dominated by noise.

  • PPA-Mono does not reflect practical closure.
    • Each task starts from an optimal configuration with only 1–3 knobs deliberately broken (Sec. 3.2.2), whereas real closure starts from scratch and must search for a good combination of all knobs.
  • DRC success includes no circuit verification.
    • Simply deleting the shape that causes a violation also reduces DRC errors, since the edit tool (change_shape) allows deletion.
    • Preventing this requires combining DRC with the authors' own LVS checker (App. C.1, C.7), but LVS is used only partially, as a post-hoc check on three models that excludes the top DRC model.
  • PPA-Bench does not sufficiently control for the noise inherent in hyper-parameter search.
    • There is no comparison against a naive search baseline (e.g., random search over a fixed knob space with the same run budget), so gains from LLM reasoning cannot be separated from search noise.
    • Uncertainty is reported only to a limited extent (only in App. H). Even there, the CIs are wide (often >20 pp) because they rest on few tasks. This suggests that many of the smaller differences in the PPA results, and the conclusions drawn from them, could be reversed.
  • The core metric comparison is circular.
    • SR and NIS are the core metrics in Tables 3 and 5. For PPA-Mono, however, NIS is clipped to [0, 1] and successes score 1, so NIS ≥ SR holds for any method.
    • The claim that "NIS exceeds SR, so agents move the right metrics" (Sec. 4.2) is therefore unsupported.
Questions for authors
  • Can the authors release complete, runnable code that reproduces the main numbers in the paper (including the missing modules and both scorers), and revise the paper to fully describe the task-generation process?
  • Can the authors address the weaknesses above while preserving the paper's core claims? For example:
    • consistent DRC-Reasoning isolation
    • LVS as part of DRC success
    • random-search/BO baselines with variance reporting for PPA
    • a metric analysis that does not rely on NIS ≥ SR
  • How does PostEDA-Bench differ from recent work on LLM-agent DRC repair (e.g., [1], [2] and the DAC26 DRC Benchmark used in [1]) in task scope, layout scale, and evaluation protocol?
Reviewer_YmSuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

  • The paper proposes PostEDA-Bench, which includes 145 tasks across four tracks: DRC Essential and Reasoning, and PPA Mono and Multi. DRC tracks measure the LLM-based agents’ capability of repairing the GDS layouts to match the DRC requirements, and the PPA tracks measure the capability of editing GDS layouts to meet the PPA requirements.
Critical strengths

Strength

  • The paper adds a useful contribution to the field by adding an open-source benchmark with meaningful tasks needed by real-world workloads.
  • The problems were reviewed by physical design engineers and consist of high-difficulty problems that even frontier models struggle with.
  • They measure diverse agentic harnesses to measure the performance across different harness, and also measure the effect of vision information as well.
Critical weaknesses

Weakness

  • I think the authors should report the wall-clock runtime, tokens, and number of tool calls for each model and agentic harness and draw the Pareto curve to identify the Pareto-efficient solutions, not only reporting the success rate.
  • In Appendix C.7, regarding the DRC-bench, the authors did an audit on DRC runs of three models to check that connectivity stays consistent before and after the DRC repair. I believe they should change the success rate definition to be gated on this connectivity check, and run the audit for this for models and agentic harnesses reported in Table 3 to make sure that DRC repairs keep the functionality of the GDS layout.
  • In Section 4.5, it is hard to understand why adding reasoning effort hurts for PPA-Multi, which is arguably harder reasoning tasks than other tracks. Even with a fixed iteration cap, I believe scaling reasoning should help in such challenging tasks.
Questions for authors

See the weakness section please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.