- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces 2D-FET-Bench V2, a benchmark of 128 tasks for constructing FET layouts from textual specifications and microscopy-derived flake contours. Agents generate geometry that is rendered to GDSII and evaluated using deterministic geometric, structural, and input-preservation checks. The evaluation compares six model configurations and several workflow and scaffold variants, with five attempts per task. Validation includes scripted reference layouts and an expert audit of sampled verifier-passing outputs.
Critical strengths
- The paper provides a concrete test bed for generating FET layouts from textual specifications and microscopy-derived contours. Its 128 tasks cover irregular geometries, multiple flakes, and different device structures, connecting spatial reasoning to a relevant scientific design problem.
- The task-level analysis also relates performance to transistor demand and the distribution of devices across flakes, showing lower success when individual flakes require multiple transistors. These findings highlight a concrete challenge in geometry-adaptive layout construction and provide a useful starting point for improving gate and contact placement in multi-device layouts.
Critical weaknesses
- Verifier success needs clearer validation against the intended device structure. Figure 6 reports expert acceptance of 260 out of 436 sampled passing layouts (59.6%). The audit combines structural correctness and experimental usability, leaving the frequency of structural failures unclear. The verifier also allows a wrap-around contact to satisfy both contact-side conditions and checks some auxiliary layers only for presence. These permissive checks and the unresolved audit gap weaken confidence in the benchmark as a measure of structural correctness.
- The workflow comparison changes prompt examples, verification timing, layer locking, and revision opportunities together. In 296 of 333 ReAct-4 via-task attempts with contacts, the saved contact geometry was incompatible with the required 0.5 μm enclosure margin. The relevant checks ran after contacts were locked. These implementation constraints offer a plausible explanation for part of the performance gap and limit the broader conclusions about planning and feedback strategies.
Questions for authors
- How do the expert-rejected layouts break down across the six audit criteria, including overlaps between criteria? How many violate the requested device structure, and how many fail solely on additional experimental-usability requirements? Which representative examples illustrate this distinction?
- How does the ReAct-3 versus ReAct-4 performance gap change when ReAct-4 can revise contacts after a via-enclosure failure, with the remaining settings held fixed? What are the resulting success rates on the via-task subset?
2 comments
Response to Authors
Dear Authors,
Thank you for your response. It seems the replies to Reviewer Fr7X and to me may have been posted under each other’s reviews. The reply under Fr7X's review answers my two questions, so my assessment is based on that reply.
I appreciate the clarifications and the added analyses. My two questions remain open: the criterion-level breakdown of the audit is unavailable, and the unlocked ReAct-4 condition was not run. I will therefore keep my original rating.
Thank you for your time and feedback. We have substantially revised the manuscript to clarify which placement decisions the agent makes and what the verifier checks. The earlier V1 used synthetic polygons and was an unpublished predecessor. This paper evaluates the 128 microscopy-derived tasks.
What the agent draws. The prompt supplies one suggested gate crossing per flake and source and drain zones. The agent draws every gate and contact. Among tasks requiring two to four transistors, GPT5.6-Luna ReAct-3/C0 passed 114/130 attempts when each flake needed one device and 70/130 when a flake needed several. Across five attempts per task, Luna found at least one pass on 26/26 and 23/26 tasks in those groups, respectively. It passed all five times on 18/26 and 3/26. Flake count also differed between groups, and all runs used hints. We therefore describe this as construction with supplied guidance (Sections 3 and 6.3).
What a pass means. Across 14 configurations, 128 of 980 CMOS attempts passed the original device checks. One also passed the wiring check on the saved layout. The other 127 had no extracted path between n- and p-type terminals on the modeled layers. GPT5.6-Luna lost all 26 of its CMOS passes, while its overall score fell from 399/640 to 373/640. An independent device researcher accepted 260 of 436 selected verifier-passing layouts. The expert was outside our research group and was not an author. Only passing agent layouts were sampled. Reference layouts and verifier-rejected layouts received no human review. For the other 176 layouts, the available counts combine FAIL and UNSURE. The expert’s reasons for individual judgments are unknown, so we cannot tell which judgments reflected the missing connections (Sections 5 and 6.2).
Why these comparisons. Our workflow comparison asks whether to replan the whole layout or build and repair one layer at a time. ReAct-3 permits three whole-layout plans. The suffixes count historical roles: planner, executor, and critic, with a fixer added in the “4” workflows. Those workflows lock finished layers. Reflexion-4 adds reflection after successful checks, and ExpeL-4 also saves experience across tasks. Table 2 gives the planning and repair budgets.
Panel selection. The earlier plan used ReAct-4/C0 on 74 tasks. We selected ReAct-3/C0 after the runs because only it had five attempts on all 128 tasks for all six available configurations. We chose models available through our serving routes to measure the range of performance under a common workflow. Section 5 now reports each model’s reasoning setting and the shared configured output cap. The supplementary Opus 5.5 evaluation used the same research workflow. Performance with production agent harnesses remains untested.