- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PostEDA-Bench, a benchmark for evaluating LLM agents on DRC repair and PPA convergence after EDA tool runs. It contains 145 tasks covering basic and practical DRC repair, single-objective PPA optimization, and multi-objective trade-offs. The paper evaluates several models and agent frameworks, but I think some more experiments about the latest model are necessary.
Critical strengths
-
The benchmark addresses an important and practically relevant problem in the EDA domain: resolving DRC violations and unmet PPA targets after traditional EDA flows.
-
The paper makes good use of visual information and includes useful ablation experiments demonstrating its benefits for DRC repair.
-
It also explicitly discusses whether the information used in the benchmark evaluation could have been included in the LLMs’ training data, and provides a reasonable explanation of why memorizing that information would not directly solve the tasks.
Critical weaknesses
-
The interpretation of performance differences is limited by incomplete control and reporting of inference resources (for example, different token consumption, different inference time and API fees should be taken into consideration). Section E.2 specifies tool-action budgets, but the frameworks use different numbers of internal LLM calls and different optimization procedures. The reasoning effort or thinking budget for each model in Table 3 is also unclear. Although Section E.1 reports some costs and runtimes, these do not establish comparisons under matched resources.
-
The evaluation does not sufficiently establish how challenging the benchmark remains for representative strong models and complete the available coding agents. In particular, it lacks evaluations of systems such as Codex, Claude Code or DeepSeek Harness. This limits the evidence supporting conclusions about current capability gaps, since low success rates may partly reflect limitations of the selected agent implementations.
Questions for authors
I appreciate the motivation and practical value of this benchmark. Evidence addressing these concerns would make me reconsider my current weak-rejection recommendation.
-
Could the authors specify the reasoning settings and token limits used for each model, and provide a targeted resource-matched comparison or success-versus-cost analysis for representative configurations? Do the main comparative conclusions remain consistent under these conditions?
-
Could the authors evaluate a representative strong model and a complete coding agent, or explain the limitations preventing such evaluations? A clearly defined subset covering the four task families would be useful.