acceptodds
Under review as a conference paper at ICLR 2027

SGAnalog: An End-to-End Circuit Benchmark from Open-Source Silicon Tapeouts

Abstract

Existing analog integrated circuit design benchmarks make two questions hard to answer: whether a model has learned transferable circuit skills rather than recalled familiar examples, and whether its output works under defined process and test conditions. We introduce SGAnalog, a benchmark built from human-designed, open-source circuits associated with Tiny Tapeout manufacturing shuttles. The collection holds 425 top-level designs spanning 273 distinct circuit topologies; the fixed evaluation sets hold 66 transcription tasks and 17 sizing tasks. Every source is retrieved at the revision recorded for its shuttle submission and processed in a fixed containerized environment. The pipeline exports each schematic image and its SPICE netlist from the same source file, giving transcription an exact structural reference. Commit dates support model-specific trainingcutoff analysis, while author testbenches provide the simulation context for sizing. The benchmark evaluates schematic-to-netlist transcription and device sizing. Across seven models on the fixed transcription set, the strongest model reaches 56.1% exact graph isomorphism, and six of seven models drop sharply from the small to the medium tier. Removing author-chosen labels lowers exact matches for every model above floor, with larger drops for stronger models, while parameter accuracy holds, indicating that labels aid connectivity tracing. On the sizing set a different model leads. It converges on every proposal and reaches 91.2 out of 100 against the human reference (85.1 on the ten tasks with a scalar metric), while the two newest Claude models refuse 4 and 11 of the same prompts they transcribe without objection; a proposal without sizes scores zero. The two tasks produce different model rankings, exposing distinct visual and design capabilities and, in one family, a policy rather than capability limit.

Primary area:
datasets and benchmarks
Keywords:
analog circuit design, open-source tapeouts, reproducible benchmarks, large language models, SPICE simulation, device sizing, data contamination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dS4dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SGAnalog, an end‑to‑end analog‑circuit benchmark built from open‑source Tiny Tapeout silicon tapEOUT submissions. It consists of 425 top‑level designs covering 273 distinct circuit topologies, with 66 schematic‑to‑netlist transcription tasks and 17 device‑sizing tasks. Using a containerized pipeline, it generates paired schematic images and SPICE netlists from identical source files and preserves original author‑written testbenches. It leverages commit timestamps and label‑stripped blinded schematics to probe whether large models generalize circuit reasoning or merely memorize examples. Evaluated across seven large‑language models, the benchmark reveals divergent model rankings between transcription and sizing tasks, sharp performance degradation for most models as circuit complexity grows, and safety‑policy‑induced refusal behaviors in some newer models. SGAnalog delivers a reproducible, audit‑ready, simulation‑driven evaluation suite for assessing LLMs’ analog IC design capabilities.

Critical strengths
  1. Authentic tape-out source, complete simulation conditions: All benchmarks are taken from Tiny Tapeout's actual open-source tape-out project, retaining the original author's test platform and PDK process environment. Unlike textbooks and synthetic datasets, the evaluation results reflect model performance in real simulated circuit design scenarios.
  2. Provides memory discrimination mechanism: Supports submission of time-truncated comparisons, blinded schematics with label removal, and other audit methods, enabling the distinction between models that truly have circuit inference capabilities and memory of circuit samples seen in the training set.
  3. Dual-task end-to-end reproducible evaluation: Set up two complementary tasks—schematic conversion to netlist and device size optimization—encapsulate the entire process in a fixed Docker container, ensure reproachable scoring logic, and reveal differences in model rankings across capabilities and expose evaluation interference caused by security policies and other non-capability factors.
Critical weaknesses
  1. Single circuit source, limited process coverage: The vast majority of circuits in the dataset are based on Sky130 processes, with insufficient support for other mainstream processes such as GF180, IHP SG13G2, and SiGe devices, making it difficult to directly transfer evaluation conclusions to other process nodes.
  2. Task scope is limited, topology generation tasks are not supported: Currently, only schematic transcription and device size adjustment are supported, and more challenging circuit topology generation tasks have not yet been realized; Due to the lack of standardized design requirements descriptions and automatic test bench matching solutions, there is no innovative topology design capability covering the core of simulation design.
  3. Dimensioning task risk of answer leakage: Tiny Tapeout's public warehouse comes with human-marked device dimensions, allowing models to directly memorize reference dimensions and achieve high scores; Although anti-plagiarism diagnostic tools are provided, they can only be marked and cannot completely eliminate inflated scores caused by copying answers.
  4. Some tasks lack unified standardized metric templates: The author's built-in Testbench test metrics are heterogeneous; out of 17 dimensional tasks, only 10 have usable scalar metrics, while the rest can only determine whether simulation converges. Unified metrics such as gain and phase margin are lacking, which is not conducive to fair cross-circuit model performance comparison.
  5. Insufficient support for layered circuits: Currently, the fixed evaluation set only uses leaf-level subcircuits, and the layered dimensioning tasks have not yet been completed; A large number of real analog circuits with complex hierarchical structures cannot directly participate in evaluation, which is a gap compared to real-world engineering scenarios that heavily use hierarchical design.
Questions for authors
  1. In commit-date truncation analysis experiments, some medium-difficulty tasks fall entirely after the model training cutoff time, leaving no model to solve. How do you plan to expand the dataset in the future to obtain enough post-cutoff medium/high-difficulty samples to more rigorously quantify the model's memory and generalization capabilities?
  2. The Sizing task carries the risk of reference answer leakage. Currently, only plagiarism detection marks are provided, which cannot completely eliminate the inflated scores caused by replicating reference sizes. Is there consideration of designing a variant task that modifies target specifications while keeping the topology unchanged, thereby avoiding the problem of directly copying open-source reference dimensions from the source?
  3. The current benchmark only supports leaf-level planar circuits; the sizing task for hierarchical circuits has not yet been achieved. When dealing with hierarchical analog circuits, do you think the biggest engineering challenge is whether the test bench automatically matches and the layered netlist scores and aligns them? Which part do you plan to prioritize?
  4. Experiments observed that the new generation Claude model randomly rejects sizing prompts due to security policies, but the transcription of the same circuit outputs normally. Have you identified the key factors triggering rejection? How will future benchmarks handle the fluctuations caused by such safety strategies to restore the model's true circuit capabilities?
  5. Currently, the dataset is mainly based on the Sky130 process. When introducing circuits from other processes such as GF180 and IHP SG13G2, besides collecting design source code, what other evaluation issues need to be addressed to ensure model scores are comparable across processes?
Reviewer_bb4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The SGAnalog dataset is a collection of analog circuits obtained by converting analog circuits that exist based on tapeouts and have matching schematics and netlists from publicly available designs with links to their commits in source code. The design is targeted at achieving two things, namely transcription of netlists from the circuit schematic images and sizing of unsized circuits. Seven models have been evaluated for 66 transcription tasks and 17 sizing problems.

Critical strengths

S1. Verifiable analog-circuit evaluation.
By connecting the design to the commit, and producing schematic images and reference netlists using the same commit, the authors ensure that the connection between the benchmark input and reference output is verifiable. The pinned simulation environment also makes it possible to evaluate the proposed sizes under the same conditions as the reference designs. Reliable data and simulation conditions are key to comparing different designs, and obtaining them is an important achievement in this field.

S2. Analyses of results aid in establishing confidence in the results.
The authors investigate whether differences found are consistent under iterations of different runs and modifications in the data being analyzed or in the scoring process used, instead of solely depending on rankings of aggregate results. These analyses show that the leading transcription models are not statistically distinguishable in the reported comparisons and that sizing comparisons depend on refusals and scoring choices.

Critical weaknesses

W1. Sizing scores do not establish functional design success.
Out of 17 sizing tasks, 7 tasks give full marks based on convergence, while the other 10 tasks use proximity to the reference swings or low frequency gains. Both criteria do not guarantee that the circuit satisfies the specifications required. Convergence does not mean successful operation, while proximity to the reference may punish the improvement or not check for a failure on some other criteria. The score composition materially affects model comparisons. Table 9 reports Gemini at 91.2 overall but 85.1 on metric-bearing tasks, below Opus at 86.4 on that subset. The Appendix decomposition exposes this sensitivity, but it does not establish whether score component measures functional design success.

W2. Lack of calibration to assess significant and generalizable sizing competence.
Sizing is done based on 17 tasks from a much larger pool of testbench runs, without determining how difficult the selected tasks are compared to simple sizing using PDK rules. This is particularly important given that there are seven convergence only tasks, as well as the repeated use of default values, as explained in Section 5: good performance may be due to either easy tasks or just the use of defaults instead of any reasoning about circuits. Moreover, since the results are from answered tasks only, which is based on a model-based subset, capability on refused tasks cannot be determined and an objective comparison cannot be made.

Questions for authors

Q1. Clarification. Validity of sizing.
What are the functional requirements for the ten metric-bearing tasks, and how does one know from the evidence that a proposal with high marks fulfills those requirements? With respect to the seven convergence tasks, what is the difference between a full-credit outcome and a circuit that converges but is dysfunctional?

Q2. Clarification. Baseline calibration.
What is the behavior of simple heuristics such as PDK-valid default-size or circuit-template heuristics on the same set of 17 tasks, where the convergence-only and metric-bearing performance are evaluated independently? How does the lack of this baseline eliminate the possibility that the performance was attainable without much LLM circuit reasoning?

Q3. Clarification. Task selection and refusal.
What is different about the structure of the 17 sizing problems versus the 190 valid testbench experiments, and what are the criteria that determine whether they can be included? Are the supposed differences between the models still there when one only considers common questions?

Q4. Clarification. Transcription scoring.
Just what does the denominator for W/L accuracy consist of, and how do non-isomorphic or incomplete parses get handled? Table 3 shows that the parameter accuracy is flawless despite very poor exact-topology accuracy. What does this mean in terms of guarantees for the reconstructed circuit? Is structural F1=1 an absolute guarantee of isomorphism, or is it merely observed in practice?

Q5. Clarification. Exposure claims.
What is the reasoning behind referring to the post-cutoff sizing activities as “provably-unseen” in Appendix D, when according to Section 5 there could have been other design iterations prior to the pinned one? Also, what evidence supports the distinction between forgetting cues for memorization versus forgetting visual markers due to label deletion?

Reviewer_nDebOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SGAnalog, a benchmark built from open-source analog circuits in Tiny Tapeout shuttle submissions. A containerized pipeline fetches each design at its recorded commit, verifies licenses, and exports a schematic image and a SPICE netlist from the same Xschem source. The collection has 425 top-level designs (273 distinct device–net graph hashes). Two fixed evaluation sets are drawn from it.

Task 1, schematic-to-netlist transcription (66 tasks). Scoring uses device–net graph isomorphism, plus a structural F1 and W/L parameter accuracy.

Task 2, device sizing (17 tasks). The model sizes an unsized topology, and its proposal is simulated in the author’s testbench. The score is the ratio of a simulated output metric to that of the human-sized design. The metric exists for 10 of the 17 tasks, and the other 7 score 1 if the simulation converges.

The paper also proposes exposure controls: commit-date splits against vendor-reported training cutoffs, label-blinded schematic images, a BIG-bench canary, and a “regurgitation” diagnostic for sizing.

Critical strengths
  1. A well-engineered, reproducible, license-aware resource. Designs are commit-pinned and license-verified, the pipeline runs in a digest-pinned container, and image and reference netlist come from one source file. This gives an exact structural reference without a separate annotation step, which is a real advantage over scan-based datasets (AMSnet, Masala-CHAI). The released collection (425 top-levels, 1,448 schematics, author testbenches, PDK-based simulation flow) is potentially valuable beyond the two tasks evaluated.
  2. The paper reports repeat runs, Wilson/bootstrap intervals, paired McNemar tests, rescoring on duplicate-free subsets, copy-robust sizing, and its own scorer corrections. Several of its own caveats (frontier orderings are not separated, the 2026-cutoff exposure test returns “no verdict”) are stated honestly. It treats training-data exposure as a measurable problem rather than ignoring it, pairing commit-date cutoff splits with blinded images
Critical weaknesses
  1. The title and Fig. 2 describe the benchmark as “end-to-end,” but the paper’s own definition is only “source file to scored simulation.” It does not cover topology generation, layout or post-layout verification, or fabricated-silicon measurement, as commonly used in analog design. The two tasks run on separate task sets and are not together, and the sizing task covers the leaf subcircuits, mostly scored by convergence or a single scalar ratio without spec compliance. “End-to-end” and “silicon-grade” are over-claimed than is delivered.
  2. The paper’s central motivation is separating capability from recall, but the evidence is too underpowered to deliver it: for the three 2026-cutoff models the commit-date test “returns no verdict”, only 3 post-cutoff trivial/small tasks, and the blinding diagnostic lacks a control showing the blinded image still determines the netlist. On sizing,12 of 103 answered proposals reproduce every reference size and the leader (Gemini) falls from 91.2 to 73.6. So Table 3, Gemini's highest sizing score depends on how much of its score is copying.
Questions for authors
  1. What is the expected growth of the sizing and transcription sets per shuttle, and why not evaluate on all eligible transcription designs rather than 66? Restricting to human-authored tapeout designs gives provenance, but it caps the pool and the strict filters cut it further to 66 and 17 tasks. The paper does not quantify how the benchmark would grow with future shuttles, or a different way, that could raise statistical power while keeping the provenance.
  2. How is it computed for non-isomorphic outputs, and what is the denominator (e.g., Haiku’s 1.000 at 1.5% exact)? How are nf/m multipliers and merged or split parallel devices treated by the isomorphism check?
  3. Which “target metrics” does the model see, and are they derived from the reference? Does the reference meet the project’s published specs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.