acceptodds
Under review as a conference paper at ICLR 2027

Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks

Abstract

Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry analog and mixed-signal circuits to electrical specification remains unclear. We introduce Analog Design Bench, a long-horizon agentic benchmark of 50 transistor-level design tasks contributed by 17 chip designers. Agents work with an open-source simulator, while an isolated verifier evaluates the submitted circuit using specification-based electrical tests. We evaluate 15 agent configurations across 2,250 two-hour attempts and observe full-specification pass rates from 8.0% to 78.0%. Coding-benchmark performance correlates with analog results but leaves much of the performance spread unexplained. Our failure analysis shows that most unsuccessful submissions have no recorded legality rejection but fail electrical acceptance, identifying electrical closure as the dominant endpoint challenge. We test time, reasoning effort, agent harness, and supplied design knowledge as interventions. Longer budgets and higher reasoning effort improve performance, while general skill documents provide little benefit and sometimes reduce performance. Supplying a task-matched reference topology, an idealized form of circuit-IP retrieval, raises DeepSeek V4 Pro by 18.7 percentage points and mainly accelerates GPT-5.6 Sol. The released benchmark, trajectories, and analysis tools provide a testbed for developing long-horizon agents for physics-grounded analog and mixed-signal integrated circuit design.

Primary area:
datasets and benchmarks
Keywords:
Agentic benchmark, long-horizon, agent evaluation, analog circuit design

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_gewVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Analog Design Bench introduces 50 expert-contributed analog/mixed-signal circuit-design tasks with an isolated electrical verifier. Across 2,250 attempts from 15 agent configurations, pass rates range from 8% to 78%. Additional experiments examine time budgets, reasoning effort, harnesses, and design guidance, finding substantial benefits from longer runs and task-matched reference topologies.

Critical strengths
  1. The benchmark combines six circuit families, expert-verified reference solutions, and protected specification-based evaluation. This provides a valuable test of autonomous electrical design beyond netlist generation.

  2. The extensive evaluation measures reliability, resource use, and performance over time. Reference-topology interventions and concrete failure trajectories provide useful evidence about where agents struggle.

Critical weaknesses
  1. Lack of layout evaluation, which is critical for analog design.

  2. Evaluation can be more comprehensive to include more advanced models.

Questions for authors
  1. How does frontier model such as GPT6 Astra and Claude Opus 5.5 perform in this benchmark?

  2. What does the analog design bench result represent? Can I interpret the results as meaning that frontier LLMs can already solve front-end analog design? Or is it simply the benchmark's limitation? Is there still a need for more front-end analog design benchmarks for LLMs?

  3. Is the current benchmark approach sustainable?

  4. How can human analog designers benefit from this benchmark?

2 comments
AuthorsOpenReview ↗

Frontier models added; scope and value of the benchmark

We thank the reviewer. Changes are marked in blue.

W1 (layout). Layout is out of scope by design: the benchmark targets front-end design.

W2/Q1 (frontier models). We added both models. GPT-6 Astra passes 98.7% of attempts and Claude Opus 5.5 passes 97.3%, and both solve every task at least once (Table 2), confirming that every task is solvable. The evaluation now covers 24 configurations and 3,600 attempts.

Q2 (what the results mean). The strongest frontier models can now close well-specified front-end tasks of this size. This does not make the benchmark less useful: it now measures how far other models, especially open-weight and smaller models, are from the frontier.

Q3 (sustainability). Yes, the format is sustainable. The difficulty is set by the specifications, the verifiers, and the time budget. We are already building harder tasks with longer design chains and larger decision spaces in the same format.

Q4 (value for designers). The benchmark indicates how well each model performs on which kinds of design tasks, both absolutely, by pass rates on tasks of known complexity, and relatively, by ranking models on the same tasks. Its contract-and-verifier format also transfers to commercial PDKs and simulators, letting design teams evaluate agents on their own tasks and in their own environments.

Thanks for your response. Layout, to me, at least for RF circuits, is an important step that can't be ignored, and front-end design itself is rather limited in determining functionality and sign-off. Maybe for certain types of circuits, these two stages can be decoupled and solved separately. But for most analog circuits, I think it's only safe to say that circuits pass sign-off after layout or even after the silicon measurement. Overall, I think this benchmark has demonstrated frontier models' capability to solve part of the analog circuit design task, but there are still more problems open to be solved. It should be noted, especially for non-expert people, to interpret these benchmark results. In the end, I would like to maintain my score based on the current paper quality.

Reviewer_oxZ3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce Analog Design Bench, comprising 50 transistor-level design tasks contributed by 17 chip-design experts. The benchmark evaluates whether LLM agents can meet electrical specifications for analog and mixed-signal circuits through topology selection, sizing, simulation, and iterative revision. The paper evaluates 15 agent configurations and investigates the effects of execution time, reasoning effort, agent harnesses, and supplied design knowledge.

Critical strengths
  • The benchmark covers six circuit families through expert-contributed tasks and verified reference results, combining explicit electrical specifications with isolated verification. Constructing these tasks and their verification environments requires substantial domain expertise and provides a useful benchmark contribution that is distinct from existing evaluations (§3, Table 1).
  • The paper clearly describes its evaluation scope and protocol, and demonstrates the benchmark’s utility through interventions on execution time and design knowledge. The analysis goes beyond model rankings to examine how iterative design and structural knowledge affect performance (§5, Fig. 4).
Critical weaknesses

  • The long-horizon characterization remains insufficiently analyzed from an LLM evaluation perspective. Multi-hour execution and improvements with additional time do not clearly distinguish challenges involving sustained state management and dependent decisions from additional search and simulation time. The interpretation in terms of long-horizon agent capabilities therefore needs further clarification (Table 1, §5–6, Appendix D).
  • The paper describes excluding easily solved tasks based on trial outcomes, but does not sufficiently specify the agent configurations and exclusion criteria used for filtering. Since some of these configurations also appear in the final evaluation, the selection procedure and its potential effects on comparisons require clearer documentation (§3.1, Appendix C).
Questions for authors
  1. Can the existing trajectories demonstrate challenges that require retaining and integrating earlier measurements and design decisions across multiple steps, beyond long elapsed runtime? How should the long-horizon characterization be interpreted after accounting for simulation and other tool wait?
  2. Which six configurations and specific exclusion criteria were used for task filtering? Can the authors clarify the potential effects on comparisons between configurations that participated in filtering and those that did not?
2 comments
AuthorsOpenReview ↗

Long-horizon evidence and task-screening details

We thank the reviewer. Changes are marked in blue.

Q1 (long horizon). Tool wait is a median 12% of runtime (§4), so runs are not mainly simulator wait. We classify every round of six configurations as topology change, sizing, or other work, labelled by GPT-6 Luna and manually checked on a sample by the authors, as shown in the new Figure 5. In the four strongest configurations, 30–38% of rounds change topology, 37–57% are sizing and 8–25% are other work. With each run normalized to its own length, topology changes still take 24–32% of rounds at 80–90% of the run, so agents keep revising the architecture rather than only tuning parameters late in the run. Some runs also lose an earlier pass (Figure 6b). Separating memory from search is future work.

Q2 (task filtering). The six screening configurations were Claude Opus 5, Claude Fable 5, and GPT-5.6 Sol, Terra and Luna [max], plus MiniMax M3, which is not evaluated. Each ran one attempt per task, and the tasks most of them solved were removed (Appendix C). This may disadvantage the five that are evaluated relative to the other 19.

Thank you for addressing my questions. My initial assessment was positive, and I maintain my weak accept recommendation.

Reviewer_nsYFOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work introduce a Analog Design Bench, which is a long-horizon agentic benchmark with 50 tasks. It evaluates 15 agent configurations on the proposed benchmark. It also studies factors that may impact the performance, including reasoning effort, agent harness, general skill documents etc.

Critical strengths
  • This work focuses a very practical domain, the analog design. Besides, instead of targeting on achieving higher score on existing benchmark, it proposes a new benchmark, which will produce large impact on LLM's development.
  • It gives a clear comparison between the proposed benchmark and the existing benchmark.
  • It includes many advanced LLMs during evaluation, including both open-sourced and close-sourced, providing a comprehensive report.
  • The performance factor section is insightful and valuable, offering lots of interesting observations.
  • It also writes a very good limitation section, which presents where can be further improved in the future.
Critical weaknesses
  • It would be better to also include some small models, e.g., Qwen 27B/35B, in the evaluation to provide a even comprehensive comparison, and also provide a study on whether small models have a potential to work well on analog design tasks.
  • I expect there are more tasks included in the benchmark, while I acknowledge that it not easy and expensive to collect more tasks.
  • It would be better if it includes a correlation analysis with terminal bench (like what has been done with DeepSWE).
Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Small models and Terminal-Bench added

We thank the reviewer for the positive assessment. The revision expands the main experiment to 24 configurations, with changes marked in blue.

W1 (small models). We added Qwen 3.8 27B: it passes 11.3% of attempts, versus 30.0% for Qwen 3.8 Max, and its runs keep changing topology without converging (Figure 5).

W2 (more tasks). We agree; each expert-written task is costly, and future versions will add held-out tasks. The current 50 already separate configurations from 5.3% to 98.7%.

W3 (Terminal-Bench). Added in Figure 3(c) and Appendix G: Spearman ρ = 0.85 (n = 12), with large analog gaps at similar terminal scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.