- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PDAgent-Bench, a benchmark of 353 tasks designed to evaluate physical‑design understanding, script generation, and full‑flow implementation capabilities. We also introduce PDAgent, a five‑agent framework leveraging expert Skill Packs, and benchmark its performance with 16 LLMs/VLMs across ten design flows in TSMC's 28nm process.
Critical strengths
-
The benchmark covers a wider range of physical design capabilities than prior work. Most existing EDA benchmarks focus on a single subtask -- macro placement for ChiPBench, script generation for ChatEDA-Bench and iScript-Bench. This work attempts to span from conceptual knowledge through report interpretation, script generation, and end-to-end RTL-to-GDSII execution on commercial tools. The error breakdown is also informative: approximately 92% of script-generation failures are near-misses where the model picks the correct command but gets flags or arguments wrong. This points toward a syntax grounding problem rather than a procedural knowledge gap, which is a useful signal for model developers.
-
The skill library ablation helps clarify what the LLM actually contributes. Running the same full-flow evaluation three ways -- bare GPT-5-mini API (fails at initialization on every design), RAG with documentation retrieval (stalls within the first two stages), and the full skill-augmented agent (reaches timing closure on smaller designs) -- gives a reasonable picture of where each component matters. The bare-LLM errors are mostly hallucinated commands (e.g.,
read_lef,pdnInit,createPowerRing), while the skill library supplies the correct Innovus command vocabulary. This separation is useful for understanding the division of labor.
Critical weaknesses
-
I am not fully convinced by the LLM-as-judge setup. The judge-scored tasks (Foundational, RootOpt, Report) use different judges in the two evaluation rounds (Claude Sonnet 4.6 vs. GPT-5.4-mini), and the paper explicitly states these scores should only be compared within a block. Three senior engineers provide human calibration scores (in parentheses in Table 6), but the human-LLM agreement is just shown as side-by-side numbers. Looking at them, the gaps are notable: GPT-5-mini Foundational scores 0.800 (LLM judge) vs. 0.701 (human), a 10-point difference, while Claude Opus 4.7 is 0.824 vs. 0.784, only 4 points. Without any correlation analysis, it's unclear whether the LLM judge is systematically inflating scores for certain model families. Another issue is that the rubric granularity varies a lot across tasks -- mean 37.1 items for Foundational, 10.1 for RootOpt, 9.2 for Report -- which makes it hard to compare scores across categories. A score of 0.7 on Foundational does not mean the same thing as 0.7 on RootOpt.
-
The full-flow results are presented without a meaningful baseline. The paper shows that PDAGENT can drive designs through placement, CTS, and routing with positive WNS at RouteOpt on the smaller designs. But there's no comparison against an automated flow that does the same thing -- e.g., OpenROAD or a vendor default flow on the same ten designs. Without this, it's hard to tell whether the agent's PPA outcomes are actually competitive. The skill-free and RAG baselines are useful for showing that the skill library is necessary, but they don't tell us whether the full system is better than what's already available. I also note that the paper admits LVS is excluded, and on larger designs like NVDLA-large, intermediate stages show substantial DRC violations (242 at the Route stage, resolved only at RouteOpt). The full-flow suite uses only TSMC 28nm with open-source designs that are much simpler than what you'd see at an foundry, which limits how far these results should be generalized.
Questions for authors
- Could you compare PDAgent with a single agent or a fixed scripted flow using the same Skill Packs, design constraints, and tool-call budget?
- Which full-flow results are authoritative? Please clarify the discrepancy between the description in Appendix A.6 and the results in Table 10.
- Can the authors report some form of correlation between the LLM-judge and human-engineer scores across all models? Spearman or Kendall would be fine. I'm particularly interested in whether the judge is biased toward certain model families -- the varying human-LLM gaps I noted above make me somewhat suspicious.
- The leakage resistance criterion (C4) is mentioned but not elaborated on. The script-generation tasks involve specific Innovus and ICC2 commands that may appear in publicly available documentation or open-source repositories. What concrete steps were taken to ensure the evaluated models haven't seen these tasks or their solutions during training? Was any held-out evaluation attempted?
- For the full-flow designs, could the authors provide at least a rough comparison against a default automated flow on the same designs? This doesn't need to be an optimally tuned human flow -- even a baseline would help calibrate the results.