acceptodds
Withdrawn from ICLR 2027

PV-Debugger: Agentic Debugging for VLSI Physical Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Electronic design automation (EDA) has automated much of the very large-scale integrated circuit (VLSI) design flow, yet physical verification (PV) remains manual. Debugging design rule checking (DRC) violations and layout-versus-schematic (LVS) mismatches in PV requires interpreting reports, localizing faults, repairing layouts, and iteratively re-running sign-off, a challenging workflow largely unexplored by existing agents. We present PV-Debugger, a framework for agentic PV debugging with three components: PV-Bench, 99 DRC and LVS debugging tasks across 5 real designs; PV-World, a reproducible GUI-based environment built entirely on the benchmark, open-source semiconductor technologies, and EDA tools; and PV-Agent, an expert-aligned multi-agent framework that grounds diagnosis and repair in design rules, layout geometry, and sign-off feedback. Experiments show that general-purpose agents struggle on PV-Bench in PV-World, with the strongest baseline solving only 23.2% of tasks. PV-Agent raises the pass rate to 54.5% on the same backbone and improves LVS debugging from 11.1% to 44.4%. PV-Debugger establishes PV as a challenging, automatically verifiable testbed for multimodal, tool-using agents and offers a foundation for developing specialized agents for real-world chip-design workflows.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Physical Verification, VLSI, Benchmark, DRC, LVS, Agentic

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hFz2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents PV-Debugger, a framework for automating VLSI physical verification debugging, consisting of PV-Bench, PV-World, and PV-Agent. It provides a reproducible benchmark and a multi-agent system for end-to-end DRC and LVS debugging. Experiments show significant improvements over general-purpose GUI agents, increasing the success rate from 23.2% to 54.5%. However, concerns remain regarding the benchmark's representativeness, the unique contribution of PV-World, error propagation among agents, and the lack of comparisons with specialized EDA methods and fine-grained ablation studies.

Critical strengths
  1. The paper addresses an important and underexplored problem in VLSI physical verification. The proposed PV-Bench and PV-World provide a reproducible framework for evaluating end-to-end DRC and LVS debugging, extending beyond the scope of existing benchmarks.

  2. The proposed PV-Agent effectively integrates domain-specific knowledge, multi-agent collaboration, and verification feedback. Extensive experiments demonstrate significant improvements over general-purpose GUI agents, particularly in LVS debugging.

Critical weaknesses
  1. The proposed PV-Bench is an innovative contribution that covers end-to-end DRC and LVS debugging, enabling a more comprehensive evaluation of agents' physical verification capabilities. However, its representativeness of real-world scenarios remains unclear. Most errors are artificially injected, and only 14 tasks involve complex faults. Further justification is needed to demonstrate whether the error types, complexity, and distribution adequately reflect industrial scenarios.

  2. PV-World provides a reproducible environment by integrating open-source EDA tools, PDKs, and standardized evaluation mechanisms. However, its advantages appear to mainly stem from engineering integration of existing tools. The authors should clarify its unique contributions compared to directly providing agents with EDA tool access and a standardized evaluation setup.

  3. PV-Agent employs multiple specialized agents with strong interdependencies. However, the framework primarily relies on sequential execution and final verification feedback, without sufficiently addressing joint optimization and error propagation across agents. The authors should clarify how upstream errors affect downstream decisions and how verification feedback can help identify error sources and improve collaboration.

  4. Experiments show that PV-Agent significantly outperforms general-purpose GUI agents, but direct comparisons with specialized EDA repair methods are lacking. Moreover, the ablation studies mainly focus on domain-specific skills, without separately assessing the contributions of multi-agent collaboration, rule retrieval, and verification feedback.

Questions for authors
  1. Further justification is needed regarding the representativeness of PV-Bench, particularly its artificially injected faults and limited number of complex tasks.

  2. The authors should clarify the unique contributions of PV-World beyond integrating existing EDA tools and providing a standardized evaluation environment.

  3. Further analysis is needed on error propagation and joint optimization across specialized agents, particularly how verification feedback can improve their coordination.

  4. Additional comparisons with specialized EDA repair methods and more fine-grained ablation studies are needed to better establish the contributions of individual components.

Reviewer_UccsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PV-Debugger addresses the task of having agents correct VLSI layouts that fail Design Rule Checking (DRC) or Layout Versus Schematic (LVS) checks. It combines five SKY130 circuits, 99 tasks with errors manually injected by humans, an execution and verification environment with fixed tool versions, and four types of agents equipped with work instructions and verified repair tools. Under the proposed evaluation metrics, PV-Agent achieves overall success rates of 54.5% and 46.5%, which are higher than the 23.2% and 22.2% achieved by Agent S3, which uses the same backbone. We examine the system’s effectiveness and limitations through ablation studies and result analysis.

Critical strengths
  1. It transforms tasks with practical engineering value into concrete and actionable evaluations. The agent must diagnose and correct actual layouts, and the final results are verified externally using DRC/LVS tools with fixed versions. It provides evaluations that go beyond simply generating checker code or making local DRC corrections, including 45 easy and intermediate LVS tasks and 9 hard mixed tasks. Since it uses publicly available designs and tools, the proposed evaluation environment is relatively accessible. Provided that the promised artifact supports the described protocol, it serves as a useful foundation for researching agents that link reports to component locations and perform verifiable corrections. [§§3.1–3.2; Table 1; Appendix A.3]

  2. In the two comparisons using the same backbone, the overall improvement across the entire system is significant. The overall success rates increase by 31.3 and 24.3 percentage points, respectively, and LVS improves from 11.1% to 44.4% and from 15.6% to 40.0%. Appendix A.5 also evaluates the screenshot removal performance of Agent S3 and PV-Agent, which use only shell/Tcl, so the results go beyond a comparison with systems that rely solely on a GUI. These results support the overall value of the specialized system for the tested tasks. However, they do not prove the effectiveness of individual components or demonstrate superiority under the same budget. [Table 3; Table 6]

Critical weaknesses

1.There is limited evidence to support PV-Agent’s unique methodological contribution. The approach of repeatedly calling the Checker to repair local layouts is also present in SCALE, which the paper cites; the more distinct differentiating factors of PV-Debugger are the benchmark and runtime environment, which include tasks where LVS fails from the outset. [§2; SCALE §3.2] The “skills” in Appendix A.4 consist of a bundle of tool code, repair examples, and procedural experience, and the ablation removes this entire bundle. Therefore, while the results demonstrate the usefulness of a domain-specific system, they do not isolate the contributions of specialized repair tools from those of prompts and workflows. However, since even the system without “skills” outperforms Agent S3, the results should not be attributed solely to the effect of a single “skill” document. A clearer explanation of the methodological advances beyond domain engineering would help distinguish and evaluate the contributions of the agent versus those of the benchmark. [§§4, 5.3; Appendix A.4]

  1. The scope of transferability beyond the development dataset, as verified through experiments, is limited. While the 99 tasks encompass 5 designs and various error types, they all involve the SKY130 and manual injection errors; furthermore, both proposed backbones solve only 2 out of 14 hard tasks. [Tables 1, 3; Appendix A.3] Appendix A.4 explains that skills are not dependent on the design and remain consistent throughout the evaluation; however, it does not disclose the sources of the mathematical examples or the cases where prior execution experience was obtained. Therefore, while the performance on the evaluated tasks is demonstrated, the basis for applicability beyond the development cases is unclear. This is not evidence of overfitting, nor do the limitations acknowledged by the authors negate the value of the benchmark. The key uncertainty lies not in the number of tasks itself, but in the scope within which the agent’s usefulness has been verified. [§5.5; Appendix A.4]
Questions for authors
  1. Beyond existing “checker-in-the-loop” repair methods and the domain-specific tools provided, what are the key methodological advancements of PV-Agent? Appendix A.4 removes both code and procedural instructions. Are there any comparisons that retain the repair tools and rule-based approach while changing only the workflow or guidelines—for example, a comparison with a simple agent using the same tools? If not, please distinguish between what the current ablation demonstrates for the entire system and the claims regarding individual components. §§4, 5.3; Appendix A.4; SCALE §3.2]
  2. Which cases were used to develop the skills, and when were they finalized prior to evaluation? Please clarify any overlap with the 99 tasks and provide existing results categorized by design or error type, as well as evaluations of cases not used in development. The objective is to verify whether improvements occur even outside the cases used to develop the skills; this is not a problem that can be solved simply by increasing the number of variations of the same case. If there is no relevant evidence, please clearly specify the scope of the generalization supported by your findings. [Tables 1, 3; Appendix A.4]
  3. What are the termination thresholds, inference settings, number of independent attempts per task, and costs recorded by the baseline agent for each agent? Additionally, how does each agent determine the number of baseline DRCs required to interpret the current report? §4 explicitly uses baseline comparisons, and Appendix A.3 describes prompts that provide only the goal. Presenting the existing execution settings and information access paths helps in interpreting reproducibility and the success rate–cost comparison. [§§3.2, 4, 5.1; Table 4]
  4. §3.2 determines success based on the number of baseline DRCs per rule and LVS judgments. For the four designs where original DRC errors remain (Table 5), do you verify whether the injected errors have been eliminated in addition to this aggregation criterion? Please describe any location, shape, or sample checks that have already been performed. If no such checks exist, please clarify the scope of the claim that the baseline restoration verified the removal of the target errors without degrading the design. This is not a criticism that actual misgrading cases were found, but rather a question regarding the scope of the guarantee; its significance depends on whether it affects the repairs that were actually processed as successful.
Reviewer_RxPFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PV-Debugger, which includes PV-Bench, PV-World, and PV-Agent for agentic debugging of DRC and LVS issues. The benchmark is useful and the results are promising, but I have several concerns about benchmark realism, evaluation fairness, and whether the reported experimental gains are sufficient to support the main claims.

Critical strengths
  1. The paper studies an important and underexplored problem. End-to-end DRC and LVS debugging is useful and much less explored than RTL generation or EDA scripting.
  2. PV-Bench and PV-World are valuable contributions. The tasks are executable and verified using real DRC/LVS tools, which makes the evaluation more meaningful.
  3. The paper reports a broad set of experiments. It includes multiple backbones, several agent baselines, cost analysis, ablations, and failure analysis.
Critical weaknesses
  1. The benchmark size is limited for the strong general conclusions. PV-Bench contains has 99 tasks from 5 designs. In particular, the hard subset has only 14 tasks, and PV-Agent solves only 2 of them. This makes it difficult to draw strong conclusions about performance on complex debugging cases.
  2. The performance gain will be affected by specialized tools and skills. With GPT-5.6-sol, PV-Agent improves the overall pass rate from 23.2% to 54.5%, but removing the specialized skills already reduces it to 37.4%. Since these skills include tool scripts and prior debugging experience, it is unclear how much of the gain comes from the multi-agent design itself.
  3. The improvement on hard tasks remains very small. Although the overall pass rate improves substantially, the hard-task pass rate is only 14.3% for both tested PV-Agent backbones. Since hard tasks are closer to realistic multi-fault debugging, this limitation should be discussed more carefully.
  4. The reported results do not show enough robustness across repeated runs. The results appear to be based on one run per task. Since LLM agents are stochastic, repeated runs or confidence intervals would help show whether differences such as 54.5% versus 46.5% are stable.
  5. The current evaluation is limited to SKY130 and a small set of designs. All results are obtained on one technology node and five open-source designs. Testing unseen designs, held-out fault types, or another technology setting would better support the generality of the benchmark and agent framework. I do understand the difficulty of access to commercial technologies and PDKs.
Questions for authors
  1. Can you study the robustness more thoroughly (see weakness 4). (clarification)
  2. Are there other tech nodes on which this can be applied to backup the claims? (future work)
  3. Can you provide the insights into why the hard tasks fails (clarification)
  4. How can the benchmark size be expanded? (future work)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.