acceptodds
Withdrawn from ICLR 2027

PDAgent-Bench: Characterizing, Grounding, and Architecting LLM/VLM Agents for VLSI Physical Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) and vision-language models (VLMs) have shown remarkable success in the front-end design of Very Large-Scale Integrated Circuits (VLSI), yet their capabilities for VLSI physical design (PD) remain significantly underexplored. One primary cause is the lack of standardized benchmarks for evaluating agentic PD workflows that require high-dimensional, multi-stage optimization under strict design constraints, coordinated interaction with diverse Electronic Design Automation (EDA) tools, and iterative refinement. This work introduces PDAgent-Bench, a comprehensive and multi-dimensional benchmark for evaluating LLM/VLM-based agents across the physical design stack. PDAgent-Bench integrates both task-level assessment and workflow-level execution. The benchmark suite contains 353 curated problems that combine conceptual questions with real-world industrial artifacts, with expert-validated references and executable solutions. These tasks cover five key capability dimensions: foundational knowledge, report comprehension, root-cause analysis, script generation, and full-flow implementation. In addition, the benchmark provides a unified, human-aligned agentic PD workflow framework that enables closed-loop evaluation of holistic PD in realistic EDA environments. Experiments on state-of-the-art models reveal that while modern LLMs/VLMs perform competitively on conceptual tasks (e.g., Opus5 achieves 74.8% on root-cause analysis), they remain substantially limited in tool-centric execution and long-horizon (Opus5 achieves 39.8% on Script-ICC2), multi-stage reasoning. Our studies further show that human-skill-enhanced agentic workflows significantly improve end-to-end PD performance. PDAgent-Bench establishes a standardized, reproducible, and realistic evaluation framework for advancing LLM/VLM-driven holistic physical design automation.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Placement and routing, Agentic, VLSI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_92b7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PDAgent-Bench, a benchmark of 353 tasks designed to evaluate physical‑design understanding, script generation, and full‑flow implementation capabilities. We also introduce PDAgent, a five‑agent framework leveraging expert Skill Packs, and benchmark its performance with 16 LLMs/VLMs across ten design flows in TSMC's 28nm process.

Critical strengths
  1. The benchmark covers a wider range of physical design capabilities than prior work. Most existing EDA benchmarks focus on a single subtask -- macro placement for ChiPBench, script generation for ChatEDA-Bench and iScript-Bench. This work attempts to span from conceptual knowledge through report interpretation, script generation, and end-to-end RTL-to-GDSII execution on commercial tools. The error breakdown is also informative: approximately 92% of script-generation failures are near-misses where the model picks the correct command but gets flags or arguments wrong. This points toward a syntax grounding problem rather than a procedural knowledge gap, which is a useful signal for model developers.

  2. The skill library ablation helps clarify what the LLM actually contributes. Running the same full-flow evaluation three ways -- bare GPT-5-mini API (fails at initialization on every design), RAG with documentation retrieval (stalls within the first two stages), and the full skill-augmented agent (reaches timing closure on smaller designs) -- gives a reasonable picture of where each component matters. The bare-LLM errors are mostly hallucinated commands (e.g., read_lef, pdnInit, createPowerRing), while the skill library supplies the correct Innovus command vocabulary. This separation is useful for understanding the division of labor.

Critical weaknesses
  1. I am not fully convinced by the LLM-as-judge setup. The judge-scored tasks (Foundational, RootOpt, Report) use different judges in the two evaluation rounds (Claude Sonnet 4.6 vs. GPT-5.4-mini), and the paper explicitly states these scores should only be compared within a block. Three senior engineers provide human calibration scores (in parentheses in Table 6), but the human-LLM agreement is just shown as side-by-side numbers. Looking at them, the gaps are notable: GPT-5-mini Foundational scores 0.800 (LLM judge) vs. 0.701 (human), a 10-point difference, while Claude Opus 4.7 is 0.824 vs. 0.784, only 4 points. Without any correlation analysis, it's unclear whether the LLM judge is systematically inflating scores for certain model families. Another issue is that the rubric granularity varies a lot across tasks -- mean 37.1 items for Foundational, 10.1 for RootOpt, 9.2 for Report -- which makes it hard to compare scores across categories. A score of 0.7 on Foundational does not mean the same thing as 0.7 on RootOpt.

  2. The full-flow results are presented without a meaningful baseline. The paper shows that PDAGENT can drive designs through placement, CTS, and routing with positive WNS at RouteOpt on the smaller designs. But there's no comparison against an automated flow that does the same thing -- e.g., OpenROAD or a vendor default flow on the same ten designs. Without this, it's hard to tell whether the agent's PPA outcomes are actually competitive. The skill-free and RAG baselines are useful for showing that the skill library is necessary, but they don't tell us whether the full system is better than what's already available. I also note that the paper admits LVS is excluded, and on larger designs like NVDLA-large, intermediate stages show substantial DRC violations (242 at the Route stage, resolved only at RouteOpt). The full-flow suite uses only TSMC 28nm with open-source designs that are much simpler than what you'd see at an foundry, which limits how far these results should be generalized.

Questions for authors
  1. Could you compare PDAgent with a single agent or a fixed scripted flow using the same Skill Packs, design constraints, and tool-call budget?
  2. Which full-flow results are authoritative? Please clarify the discrepancy between the description in Appendix A.6 and the results in Table 10.
  3. Can the authors report some form of correlation between the LLM-judge and human-engineer scores across all models? Spearman or Kendall would be fine. I'm particularly interested in whether the judge is biased toward certain model families -- the varying human-LLM gaps I noted above make me somewhat suspicious.
  4. The leakage resistance criterion (C4) is mentioned but not elaborated on. The script-generation tasks involve specific Innovus and ICC2 commands that may appear in publicly available documentation or open-source repositories. What concrete steps were taken to ensure the evaluated models haven't seen these tasks or their solutions during training? Was any held-out evaluation attempted?
  5. For the full-flow designs, could the authors provide at least a rough comparison against a default automated flow on the same designs? This doesn't need to be an optimally tuned human flow -- even a baseline would help calibrate the results.
Reviewer_PGWpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PDAgent-Bench, a set of 353 tasks for testing how well AI models handle chip physical design. These include understanding design concepts and reports, diagnosing problems, writing tool commands, and completing the design process. It also introduces PDAgent, a system of five specialized AI agents that uses expert-written instructions and scripts to work with chip-design tools. The authors evaluate 16 models, have experts review the questions and reference answers, and compare automated scoring with human scoring. They also test the agent system on 10 chip designs using commercial tools and a 28-nm manufacturing process. The results show that models perform better on knowledge questions than on writing correct tool commands. Adding expert-written guidance helps the agents complete design workflows that simpler baselines struggle to execute.

Critical strengths

It has the following strengths: 1- The benchmark covers several connected parts of physical design. It includes report interpretation, problem diagnosis, script generation, and full-flow execution. This breadth is valuable because it helps assess whether models can connect domain knowledge to practical tool use, rather than succeed only on isolated questions. 2- The evaluation includes actual execution with commercial design tools. The full-flow experiments use a 28-nm process and report timing and design-rule violations across design stages. This provides stronger evidence of practical usefulness than judging generated text alone and makes the benchmark relevant to real engineering workflows.

Critical weaknesses

1-The experiments do not establish what the multi-agent system adds beyond the expert-written guidance. The baselines lack the same skill library, so their failure mainly shows that tool-specific guidance is important. A single agent with the same skills, or a scripted workflow using the supplied scripts, would provide a stronger comparison. Without these controls, the claimed benefit of the agent architecture and its reasoning remains unclear. 2- Problems in evaluation and reporting weaken confidence in the conclusions. Section 4.1 defines “pass@5” as an average score across five responses, which does not measure success within five attempts. Different model groups also use different judges and sampling settings, limiting direct comparisons. Additionally, the appendix describes remaining violations for Open910 and NVDLA-large, while Table 10 reports zero final DRC violations and positive timing slack for both. Since reliable evaluation is the benchmark’s central contribution, these issues materially weaken the case for acceptance.

Questions for authors

I have the following questions: 1- Do you have comparisons against a single agent given the same skill library, or a scripted workflow using the supplied scripts, under comparable execution budgets? What evidence shows that the five-agent architecture adds value beyond the expert-written guidance? 2- Is pass@5 calculated as the average score across five responses or as success in at least one of five attempts? Please clarify this separately for rubric-scored and script-generation tasks. 3- How sensitive are the model rankings to the different judges and sampling settings used across evaluation rounds? Were any shared model outputs scored by both judges to check consistency? 4- Appendix A.6 states that Open910 and NVDLA-large retain violations, but Table 10 reports zero final DRC violations and positive timing slack for both. Which results are correct, and how many designs achieved both timing closure and zero DRC violations?

Reviewer_EZDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PDAgent-Bench, a benchmark for LLM/VLM agents in VLSI physical design. It has 353 tasks in three suites. Physical Design Understanding (133 tasks) covers foundational knowledge, report comprehension, root-cause analysis, and STA. Script Generation (210 tasks) asks for Tcl scripts for Innovus, ICC2, ECO, and Formality. Full Flow (10 designs) runs a complete implementation flow. The paper also proposes PDAgent, a multi-agent framework (Planner, Worker, Analyzer, Debugger, Optimizer) that uses expert skill packs and closed-loop optimization with commercial EDA tools on TSMC 28 nm. The evaluation of 16 LLMs/VLMs shows strong results on conceptual tasks but weak results on tool-specific script generation. With skills, PDAgent completes the full flow on all ten designs, but without skills the agent fails at initialization.

Critical strengths

Strengths

S1. Well-motivated benchmark with diverse task types. Physical design is underexplored for LLM/VLM agents, and the paper gives a reasonable motivation for a benchmark in this domain (Sec. 1–2, Table 1). The benchmark covers many task types: conceptual knowledge, report comprehension, root-cause analysis, STA, tool-specific script generation, and full-flow implementation (Table 2).

S2. Practical full-flow approach. PDAgent follows the way real engineers work. It plans stages, runs tool commands, debugs, and tunes parameters with expert knowledge (Fig. 1, Sec. 3.2). It completes the full flow on ten designs (Table 10), so it is a promising direction for agentic physical design.

Critical weaknesses

Weaknesses

W1. The main contribution is an open-source benchmark, but the submission does not give enough material to verify it.

  • A.11 says the complete benchmark is "included in the supplementary material." But no supplementary material, code (checkers, PDAgent, skills), or anonymized link is provided. The appendix has only one example task (Figs. 11–12) and some statistics (Figs. 4–7).
  • The paper does not say whether the parts that need commercial tools and the TSMC 28 nm PDK will be released. It also does not say how others can run them as an open-source benchmark (Sec. 4.1, Table 10).
  • Without the skills, the flow fails at initialization (Tables 11–12), but the paper does not give the skill content or how the skills were made. The Worker can also run reference scripts directly (A.9.1). So it is unclear whether the skills hold general knowledge or scripts written for these ten designs. No baseline runs the skill scripts without the LLM, so the LLM's contribution is also unclear.

W2. The reliability of the results is not shown in a transparent way.

  • In Table 13, many values repeat the same digits (e.g., 327,648 / 227,648 / 127,648; 4,013 twice). If these are real measurements, they need an explanation. Runtime, cost, and iteration counts are also not reported.
  • All full-flow runs use one weak backbone, GPT-5-mini (Innovus pass@1 13.3%, Table 6), and no repeated runs are reported. So the suite does not compare LLMs, although Sec. 3.2 says PDAgent is for "evaluating" agents.
  • Reference scripts are checked by execution (Sec. 3.1, L265–267). Model scripts are only compared with a set of acceptable solutions (L250–252; Table 4 caption, L393–394). So the scores depend on how complete this set is and how good the checker is. The paper reports neither.
Questions for authors
  1. Full-flow evaluation scope. Table 2 says that the full flow is evaluated on PPA and DRC/LVS. But LVS is excluded (A.2), and Table 10 does not report power, area, or signoff timing. Can the authors report these metrics for at least some designs?

  2. pass@k definition. For judge-scored tasks, "pass@5" is the mean of five samples. For rule-graded tasks, it means that at least one of five samples is correct (Table 4 caption). So one column name has two meanings. The usual convention is to report the mean over n samples as pass@1 (with n stated), and best-of-k or any-correct as pass@k. The authors could consider this convention for clarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.