acceptodds
Withdrawn from ICLR 2027

CircuitTrace: Tracing Circuit Intent from Prediction to Editable Schematics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Graph accuracy alone can overstate the advantage of a circuit-generation method. CircuitTrace traces natural-language requests through model prediction, executable validation, and saved KiCad artifacts to reveal where correctness is lost. It compares two contracts through a shared deterministic backend: SemanticIntent predicts circuit intent and delegates symbol and pin resolution to a binder; FullSpec predicts executable graphs directly. Layered checks separate graph correctness, contract validity, artifact fidelity, reference topology, and native electrical rule checking. On 59 project-disjoint passive-circuit tasks with one or two components across three training seeds, post-hoc value and net-name diagnostics give FullSpec 141 correct graphs against SemanticIntent's 131, reversing the literal-value ordering. This 10-graph lead shrinks to 3 joint reference-topology successes after realization: 133/177 versus 130/177 (75.1% versus 73.4%). FullSpec loses 4 successes at executable validation, 2 at artifact saving, and 2 at topology checking; SemanticIntent loses one artifact. Native electrical checks further distinguish topology agreement from error-free artifacts. CircuitTrace exposes implementation failures hidden by graph accuracy and provides a reproducible protocol for evaluating circuit generation through the saved design.

Primary area:
datasets and benchmarks
Keywords:
large language models, electronic design automation, circuit generation, structured prediction, tool use, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_w1uYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an evaluation for natural-language-to-KiCad schematic generation that tracks each prediction through executable validation, artifact saving, reference-topology checks and native ERC. It compares two output formats for a LoRA-tuned Qwen3-8B: SemanticIntent, where a deterministic binder fills in pins and references, and FullSpec, which predicts them directly. On 59 tasks with one or two passive components, FullSpec’s 10-case lead in graph accuracy shrinks to 3 at final topology (133 vs. 130 of 177).

Critical strengths
  • The motivation is reasonable. Checking the saved artifact rather than the intermediate text is the right direction for circuit generation, and separating graph, contract, artifact and topology failures is a useful way to think about the problem.
  • Transparent reporting. Failed and interrupted attempts stay in the denominator, post-hoc steps are disclosed, and per-case traces are provided.
Critical weaknesses
  • The main result is not significant. As Section 5.2 and the conclusion acknowledge, the paired 95% intervals span zero. Neither the 10-case lead nor the 3-case lead can be separated from zero.
  • Key results depend on post-hoc choices. The graph ordering reverses only under ValueSemantics v1, which was designed after inspecting the results. NetName v2, the source corrections and the REF006 exclusion are also post-hoc.
  • The losses mostly reflect the authors’ own toolchain. Artifact failures come from GUI menu failures and process errors, and contract failures include a runtime policy that rejects ADC_AVDD as a power name. These measure the executor rather than the generation method.
  • The native ERC metric does not discriminate. Even the gold-input Oracle60 run has only 19/60 error-free and 0/60 violation-free artifacts, so the ERC results for the two contracts say little.

The work is mainly an evaluation protocol on a tiny benchmark, and much of the text is provenance bookkeeping, which makes the findings hard to extract.

Questions for authors

Q1. Since no paired comparison is significant, what does the paper claim beyond “the two formats perform similarly on this benchmark”?

Q2. How do the graph-to-final losses split between prediction errors, backend limitations and GUI or environment failures? What does the comparison look like if pure infrastructure failures are excluded?

Q3. What is the main conclusion under the original literal scoring alone? Why should the post-hoc value-aware metric be the basis of the headline result?

Reviewer_iTJ5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses circuit generation, a problem commonly handled by LLMs nowadays. Previous benchmarks for circuit-generation methods often use graph accuracy for evaluation. However, this paper argues that graph-accuracy-based evaluation can overstate the advantages of a circuit-generation method. Therefore, the paper introduces a new benchmark with 59 project-disjoint passive-circuit tasks. Using this benchmark with three seeds for each task, resulting in a total of 117 task instances, the paper directly compares two different types of approaches: (1) FullSpec, which directly predicts executable graphs, and (2) SemanticIntent, which predicts component classes, roles, values, and logical nets, while a deterministic binder supplies implementation identities and pin endpoints. Through experiments with these two methods, the paper reveals a limitation of graph-level scores: they cannot analyze prediction failures such as contract failures, artifact failures, and ERC failures.

Critical strengths

This reviewer thinks that the problem tackled in this paper, the limitation of graph-level score-based evaluation (graph accuracy) in analyzing the various issues that arise in circuit generation, is valid and clearly defined.

Critical weaknesses
  1. This reviewer thinks that one of the biggest problems with this paper is that it is difficult to follow. This is a dataset and evaluation contribution paper. Therefore, this reviewer thinks that a paper of this type should clearly explain what problem the proposed benchmark addresses that existing benchmarks do not, how the dataset was designed to address that problem, how its scale compares with those of other datasets, and what benefits it provides. However, these points are not clearly presented.

  2. This reviewer particularly thinks that 59 tasks is too small a scale. The paper uses three seeds for each task and thus considers there to be 117 task instances. However, there are actually only 59 tasks, with three seeds for each. Since these are not 117 independent tasks, this reviewer thinks that the scale is too small for a benchmark contribution. This reviewer wonders whether the main contribution of the paper is not so much a benchmark, but rather the argument that evaluation metrics other than graph accuracy should be used to better analyze failures.

Questions for authors
  1. With 59 tasks and three seeds for each, there are ultimately 59 tasks. Should the variation across seeds be examined? Can the results for each seed be interpreted as separate, independent tasks?

  2. It seems that the evaluation uses only Qwen3-8B. For a benchmark and evaluation paper, this reviewer thinks that it would be better the authors to evaluate and compare the performance of a wider range of LLMs.

Reviewer_NVuVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CircuitTrace compares semantic-intent and executable-graph prediction through a shared KiCad backend, separating graph correctness, executable validity, saved topology, and ERC outcomes. On 59 passive-circuit tasks across three seeds, FullSpec’s 141-versus-131 graph successes become 133-versus-130 joint topology successes, demonstrating failures hidden by graph accuracy.

Critical strengths
  • Saved-artifact verification distinguishes prediction, implementation, and execution failures, providing actionable diagnostics.
Critical weaknesses
  • Limited scope: tasks explicitly specify values and wiring for only one or two passive components. This is far from the complexity one would expect of real-world PCB designs. Evaluation covers one model family without prior-system baselines.
  • Robustness remains uncertain: scoring/reference revisions are post hoc, and a complete pre-inference backend freeze is not established. Final paired intervals include zero, while role supervision differs between contracts. The findings remain exploratory.
  • The quality of writing is subpar, with heavy use of AI for editing as judged by the writing style and word choices.
Questions for authors
  1. Were backend dependencies changed after predictions were scored? If so, could this affect contract-specific losses?
  2. Where can reviewers access the runnable benchmark, code, and full per-case evidence beyond the hash inventory? Is the dataset and framework going to be open-sourced?
  3. Justify the use of "59 project-disjoint passive-circuit tasks with one or two components", whether the scale, complexity, and practicality are sufficient for a rigorous study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.