- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SGAnalog, an end‑to‑end analog‑circuit benchmark built from open‑source Tiny Tapeout silicon tapEOUT submissions. It consists of 425 top‑level designs covering 273 distinct circuit topologies, with 66 schematic‑to‑netlist transcription tasks and 17 device‑sizing tasks. Using a containerized pipeline, it generates paired schematic images and SPICE netlists from identical source files and preserves original author‑written testbenches. It leverages commit timestamps and label‑stripped blinded schematics to probe whether large models generalize circuit reasoning or merely memorize examples. Evaluated across seven large‑language models, the benchmark reveals divergent model rankings between transcription and sizing tasks, sharp performance degradation for most models as circuit complexity grows, and safety‑policy‑induced refusal behaviors in some newer models. SGAnalog delivers a reproducible, audit‑ready, simulation‑driven evaluation suite for assessing LLMs’ analog IC design capabilities.
Critical strengths
- Authentic tape-out source, complete simulation conditions: All benchmarks are taken from Tiny Tapeout's actual open-source tape-out project, retaining the original author's test platform and PDK process environment. Unlike textbooks and synthetic datasets, the evaluation results reflect model performance in real simulated circuit design scenarios.
- Provides memory discrimination mechanism: Supports submission of time-truncated comparisons, blinded schematics with label removal, and other audit methods, enabling the distinction between models that truly have circuit inference capabilities and memory of circuit samples seen in the training set.
- Dual-task end-to-end reproducible evaluation: Set up two complementary tasks—schematic conversion to netlist and device size optimization—encapsulate the entire process in a fixed Docker container, ensure reproachable scoring logic, and reveal differences in model rankings across capabilities and expose evaluation interference caused by security policies and other non-capability factors.
Critical weaknesses
- Single circuit source, limited process coverage: The vast majority of circuits in the dataset are based on Sky130 processes, with insufficient support for other mainstream processes such as GF180, IHP SG13G2, and SiGe devices, making it difficult to directly transfer evaluation conclusions to other process nodes.
- Task scope is limited, topology generation tasks are not supported: Currently, only schematic transcription and device size adjustment are supported, and more challenging circuit topology generation tasks have not yet been realized; Due to the lack of standardized design requirements descriptions and automatic test bench matching solutions, there is no innovative topology design capability covering the core of simulation design.
- Dimensioning task risk of answer leakage: Tiny Tapeout's public warehouse comes with human-marked device dimensions, allowing models to directly memorize reference dimensions and achieve high scores; Although anti-plagiarism diagnostic tools are provided, they can only be marked and cannot completely eliminate inflated scores caused by copying answers.
- Some tasks lack unified standardized metric templates: The author's built-in Testbench test metrics are heterogeneous; out of 17 dimensional tasks, only 10 have usable scalar metrics, while the rest can only determine whether simulation converges. Unified metrics such as gain and phase margin are lacking, which is not conducive to fair cross-circuit model performance comparison.
- Insufficient support for layered circuits: Currently, the fixed evaluation set only uses leaf-level subcircuits, and the layered dimensioning tasks have not yet been completed; A large number of real analog circuits with complex hierarchical structures cannot directly participate in evaluation, which is a gap compared to real-world engineering scenarios that heavily use hierarchical design.
Questions for authors
- In commit-date truncation analysis experiments, some medium-difficulty tasks fall entirely after the model training cutoff time, leaving no model to solve. How do you plan to expand the dataset in the future to obtain enough post-cutoff medium/high-difficulty samples to more rigorously quantify the model's memory and generalization capabilities?
- The Sizing task carries the risk of reference answer leakage. Currently, only plagiarism detection marks are provided, which cannot completely eliminate the inflated scores caused by replicating reference sizes. Is there consideration of designing a variant task that modifies target specifications while keeping the topology unchanged, thereby avoiding the problem of directly copying open-source reference dimensions from the source?
- The current benchmark only supports leaf-level planar circuits; the sizing task for hierarchical circuits has not yet been achieved. When dealing with hierarchical analog circuits, do you think the biggest engineering challenge is whether the test bench automatically matches and the layered netlist scores and aligns them? Which part do you plan to prioritize?
- Experiments observed that the new generation Claude model randomly rejects sizing prompts due to security policies, but the transcription of the same circuit outputs normally. Have you identified the key factors triggering rejection? How will future benchmarks handle the fluctuations caused by such safety strategies to restore the model's true circuit capabilities?
- Currently, the dataset is mainly based on the Sky130 process. When introducing circuits from other processes such as GF180 and IHP SG13G2, besides collecting design source code, what other evaluation issues need to be addressed to ensure model scores are comparable across processes?