- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an evaluation for natural-language-to-KiCad schematic generation that tracks each prediction through executable validation, artifact saving, reference-topology checks and native ERC. It compares two output formats for a LoRA-tuned Qwen3-8B: SemanticIntent, where a deterministic binder fills in pins and references, and FullSpec, which predicts them directly. On 59 tasks with one or two passive components, FullSpec’s 10-case lead in graph accuracy shrinks to 3 at final topology (133 vs. 130 of 177).
Critical strengths
- The motivation is reasonable. Checking the saved artifact rather than the intermediate text is the right direction for circuit generation, and separating graph, contract, artifact and topology failures is a useful way to think about the problem.
- Transparent reporting. Failed and interrupted attempts stay in the denominator, post-hoc steps are disclosed, and per-case traces are provided.
Critical weaknesses
- The main result is not significant. As Section 5.2 and the conclusion acknowledge, the paired 95% intervals span zero. Neither the 10-case lead nor the 3-case lead can be separated from zero.
- Key results depend on post-hoc choices. The graph ordering reverses only under ValueSemantics v1, which was designed after inspecting the results. NetName v2, the source corrections and the REF006 exclusion are also post-hoc.
- The losses mostly reflect the authors’ own toolchain. Artifact failures come from GUI menu failures and process errors, and contract failures include a runtime policy that rejects ADC_AVDD as a power name. These measure the executor rather than the generation method.
- The native ERC metric does not discriminate. Even the gold-input Oracle60 run has only 19/60 error-free and 0/60 violation-free artifacts, so the ERC results for the two contracts say little.
The work is mainly an evaluation protocol on a tiny benchmark, and much of the text is provenance bookkeeping, which makes the findings hard to extract.
Questions for authors
Q1. Since no paired comparison is significant, what does the paper claim beyond “the two formats perform similarly on this benchmark”?
Q2. How do the graph-to-final losses split between prediction errors, backend limitations and GUI or environment failures? What does the comparison look like if pure infrastructure failures are excluded?
Q3. What is the main conclusion under the original literal scoring alone? Why should the post-hoc value-aware metric be the basis of the headline result?