- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Analog Design Bench introduces 50 expert-contributed analog/mixed-signal circuit-design tasks with an isolated electrical verifier. Across 2,250 attempts from 15 agent configurations, pass rates range from 8% to 78%. Additional experiments examine time budgets, reasoning effort, harnesses, and design guidance, finding substantial benefits from longer runs and task-matched reference topologies.
Critical strengths
-
The benchmark combines six circuit families, expert-verified reference solutions, and protected specification-based evaluation. This provides a valuable test of autonomous electrical design beyond netlist generation.
-
The extensive evaluation measures reliability, resource use, and performance over time. Reference-topology interventions and concrete failure trajectories provide useful evidence about where agents struggle.
Critical weaknesses
-
Lack of layout evaluation, which is critical for analog design.
-
Evaluation can be more comprehensive to include more advanced models.
Questions for authors
-
How does frontier model such as GPT6 Astra and Claude Opus 5.5 perform in this benchmark?
-
What does the analog design bench result represent? Can I interpret the results as meaning that frontier LLMs can already solve front-end analog design? Or is it simply the benchmark's limitation? Is there still a need for more front-end analog design benchmarks for LLMs?
-
Is the current benchmark approach sustainable?
-
How can human analog designers benefit from this benchmark?
2 comments
Thanks for your response. Layout, to me, at least for RF circuits, is an important step that can't be ignored, and front-end design itself is rather limited in determining functionality and sign-off. Maybe for certain types of circuits, these two stages can be decoupled and solved separately. But for most analog circuits, I think it's only safe to say that circuits pass sign-off after layout or even after the silicon measurement. Overall, I think this benchmark has demonstrated frontier models' capability to solve part of the analog circuit design task, but there are still more problems open to be solved. It should be noted, especially for non-expert people, to interpret these benchmark results. In the end, I would like to maintain my score based on the current paper quality.
Frontier models added; scope and value of the benchmark
We thank the reviewer. Changes are marked in blue.
W1 (layout). Layout is out of scope by design: the benchmark targets front-end design.
W2/Q1 (frontier models). We added both models. GPT-6 Astra passes 98.7% of attempts and Claude Opus 5.5 passes 97.3%, and both solve every task at least once (Table 2), confirming that every task is solvable. The evaluation now covers 24 configurations and 3,600 attempts.
Q2 (what the results mean). The strongest frontier models can now close well-specified front-end tasks of this size. This does not make the benchmark less useful: it now measures how far other models, especially open-weight and smaller models, are from the frontier.
Q3 (sustainability). Yes, the format is sustainable. The difficulty is set by the specifications, the verifiers, and the time budget. We are already building harder tasks with longer design chains and larger decision spaces in the same format.
Q4 (value for designers). The benchmark indicates how well each model performs on which kinds of design tasks, both absolutely, by pass rates on tasks of known complexity, and relatively, by ranking models on the same tasks. Its contract-and-verifier format also transfers to commercial PDKs and simulators, letting design teams evaluate agents on their own tasks and in their own environments.