acceptodds
Under review as a conference paper at ICLR 2027

2D-FET-BENCH: FROM SPATIAL REASONING TO FET DESIGN ON FLAKES

Abstract

Field-effect transistor (FET) layouts on exfoliated two-dimensional flakes are typ- ically drawn by hand for each flake, placing contacts and gates to match its posi- tion and outline in optical micrographs. To our knowledge, no executable bench- mark tests whether language-model agents can perform this flake-specific con- struction reliably. We introduce 2D-FET-Bench V2, a benchmark of 128 lay- out tasks built from microscopy-derived flake contours, including hole-containing flakes and multi-flake tasks. Each task supplies a textual device specification and contour coordinates. An agent generates typed polygon and path operations ren- dered to GDSII. A deterministic verifier checks geometric and structural require- ments, and a separate integrity check verifies that the supplied contours remain unchanged. Scripted reference layouts pass all 128 tasks, showing that every task is solvable. We evaluate six models and seven workflow and scaffold variants of GPT5.6-Luna, with five attempts per task. The best-performing configuration in the six-model panel, GPT5.6-Luna with ReAct-3, passes 62.3% of attempts and solves 80.5% of tasks at least once (coverage) and 43.8% in all five attempts (con- sistency). ReAct-3 exceeds the one-pass Plan-and-Execute by 27.0 pass@1 points at 2.46× the tokens. An expert audit of one sampled verifier-passing layout per covered task, across five ReAct-3 configurations, accepts 56.4%–63.5% of them. The benchmark evaluates geometric and structural FET layout construction.

Primary area:
datasets and benchmarks
Keywords:
language model agents, benchmarking, geometric reasoning, constrained geometric construction, layout generation, electronic design automation, 2D materials, reliability evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PWgiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces 2D-FET-Bench V2, a benchmark of 128 tasks for constructing FET layouts from textual specifications and microscopy-derived flake contours. Agents generate geometry that is rendered to GDSII and evaluated using deterministic geometric, structural, and input-preservation checks. The evaluation compares six model configurations and several workflow and scaffold variants, with five attempts per task. Validation includes scripted reference layouts and an expert audit of sampled verifier-passing outputs.

Critical strengths
  • The paper provides a concrete test bed for generating FET layouts from textual specifications and microscopy-derived contours. Its 128 tasks cover irregular geometries, multiple flakes, and different device structures, connecting spatial reasoning to a relevant scientific design problem.
  • The task-level analysis also relates performance to transistor demand and the distribution of devices across flakes, showing lower success when individual flakes require multiple transistors. These findings highlight a concrete challenge in geometry-adaptive layout construction and provide a useful starting point for improving gate and contact placement in multi-device layouts.
Critical weaknesses
  • Verifier success needs clearer validation against the intended device structure. Figure 6 reports expert acceptance of 260 out of 436 sampled passing layouts (59.6%). The audit combines structural correctness and experimental usability, leaving the frequency of structural failures unclear. The verifier also allows a wrap-around contact to satisfy both contact-side conditions and checks some auxiliary layers only for presence. These permissive checks and the unresolved audit gap weaken confidence in the benchmark as a measure of structural correctness.
  • The workflow comparison changes prompt examples, verification timing, layer locking, and revision opportunities together. In 296 of 333 ReAct-4 via-task attempts with contacts, the saved contact geometry was incompatible with the required 0.5 μm enclosure margin. The relevant checks ran after contacts were locked. These implementation constraints offer a plausible explanation for part of the performance gap and limit the broader conclusions about planning and feedback strategies.
Questions for authors
  • How do the expert-rejected layouts break down across the six audit criteria, including overlaps between criteria? How many violate the requested device structure, and how many fail solely on additional experimental-usability requirements? Which representative examples illustrate this distinction?
  • How does the ReAct-3 versus ReAct-4 performance gap change when ReAct-4 can revise contacts after a via-enclosure failure, with the remaining settings held fixed? What are the resulting success rates on the via-task subset?
2 comments

Thank you for your time and feedback. We have substantially revised the manuscript to clarify which placement decisions the agent makes and what the verifier checks. The earlier V1 used synthetic polygons and was an unpublished predecessor. This paper evaluates the 128 microscopy-derived tasks.

What the agent draws. The prompt supplies one suggested gate crossing per flake and source and drain zones. The agent draws every gate and contact. Among tasks requiring two to four transistors, GPT5.6-Luna ReAct-3/C0 passed 114/130 attempts when each flake needed one device and 70/130 when a flake needed several. Across five attempts per task, Luna found at least one pass on 26/26 and 23/26 tasks in those groups, respectively. It passed all five times on 18/26 and 3/26. Flake count also differed between groups, and all runs used hints. We therefore describe this as construction with supplied guidance (Sections 3 and 6.3).

What a pass means. Across 14 configurations, 128 of 980 CMOS attempts passed the original device checks. One also passed the wiring check on the saved layout. The other 127 had no extracted path between n- and p-type terminals on the modeled layers. GPT5.6-Luna lost all 26 of its CMOS passes, while its overall score fell from 399/640 to 373/640. An independent device researcher accepted 260 of 436 selected verifier-passing layouts. The expert was outside our research group and was not an author. Only passing agent layouts were sampled. Reference layouts and verifier-rejected layouts received no human review. For the other 176 layouts, the available counts combine FAIL and UNSURE. The expert’s reasons for individual judgments are unknown, so we cannot tell which judgments reflected the missing connections (Sections 5 and 6.2).

Why these comparisons. Our workflow comparison asks whether to replan the whole layout or build and repair one layer at a time. ReAct-3 permits three whole-layout plans. The suffixes count historical roles: planner, executor, and critic, with a fixer added in the “4” workflows. Those workflows lock finished layers. Reflexion-4 adds reflection after successful checks, and ExpeL-4 also saves experience across tasks. Table 2 gives the planning and repair budgets.

Panel selection. The earlier plan used ReAct-4/C0 on 74 tasks. We selected ReAct-3/C0 after the runs because only it had five attempts on all 128 tasks for all six available configurations. We chose models available through our serving routes to measure the range of performance under a common workflow. Section 5 now reports each model’s reasoning setting and the shared configured output cap. The supplementary Opus 5.5 evaluation used the same research workflow. Performance with production agent harnesses remains untested.

Reviewer_PWgiOpenReview ↗

Response to Authors

Dear Authors,

Thank you for your response. It seems the replies to Reviewer Fr7X and to me may have been posted under each other’s reviews. The reply under Fr7X's review answers my two questions, so my assessment is based on that reply.

I appreciate the clarifications and the added analyses. My two questions remain open: the criterion-level breakdown of the audit is unavailable, and the unlocked ReAct-4 condition was not run. I will therefore keep my original rating.

Reviewer_Fr7XOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a FET layout benchmark for LLM agents. It contains 128 tasks, each giving a textual device specification and the contours of 2D material flakes taken from micrographs. The agent returns the layout as geometry, and a deterministic verifier checks it against geometric and structural requirements. The paper evaluates six models and several workflow and scaffold variants of GPT5.6-Luna with five attempts per task, reports the authors' own ReAct-3 workflow as the best configuration, and has one expert audit a sample of verifier-passing layouts.

Critical strengths

The paper tests agentic workflows on a FET layout task, with the benchmark constructed from real, released microscopy masks and a deterministic verifier.

Critical weaknesses
  1. The paper is extremely hard to follow. It reads like an experiment report without a backbone. Across the paper it leans on naming that is not self-contained, such as C0, ReAct-3, hole probes and the residual set, which all appear before they are defined, and the numbers in the workflow names are explained late or not at all. The 3 in ReAct-3 turns out to be a cap of three planning rounds, said only in Section 5. One can guess that the 4 in ReAct-4, Reflexion-4 and ExpeL-4 is four rounds, but why those workflows are capped at four is never explained. V1, which seems to be either a prior publication or an internal working version, gets one line of explanation and no citation. Taking V2 as a standalone benchmark, why the authors separate V1 and V2 is unclear, and no claim or experiment in the paper seems to need two variants. The paper moves between the benchmark and ReAct-3 without settling on which one it is about, and mostly what I learned is that the authors built an agentic FET task with a verifier and that their own ReAct-3 workflow performs well, for no stated reason.

  2. The reported numbers are not well enough supported by the evidence provided. The only check of what the verifier accepts is an expert audit of one verifier-passing layout per covered task, and there the expert accepted between 56.4% and 63.5% of them depending on the configuration, so roughly four in ten layouts the benchmark counts as successes are not acceptable devices. Layouts the verifier rejected and the reference layouts were not audited. Every success rate in the paper rests on this verifier.

  3. The main results are about the authors' own workflows, which are presented without a design rationale. Instead of directly applying the original methods, ReAct-3, ReAct-4, Reflexion-4 and ExpeL-4 are the authors' variants "inspired by" the originals. The paper gives no reason for their design and describes them in a few sentences. The model panel mixes models with reasoning enabled and models with reasoning disabled. The same model with and without reasoning would be fine, but here GPT5.6-Luna, GLM-5 and Kimi K2.5 run with high reasoning while Qwen3-235B, Qwen3-32B and Ministral-3-14B run without it, with no design reason given. The paper compares scaffolds and workflows for one model, but it never ablates the harness's own hints (the prompt that suggests a gate-crossing point and source/drain segments for each flake), and it draws no insight from any of this.

Questions for authors

Clarification requests

  1. For the verifier-passing layouts the expert rejected, what was wrong, and does the verifier miss it by design? Were the reference layouts and the layouts the verifier rejected audited? Who performed the audit, and is the expert an author?
  2. The only reason given in the paper for making ReAct-3 the primary comparison is that it alone had complete, comparable records for all six models. Why is it the only pick, and what decided it? Were other baselines such as ReAct-4 planned with full records?
  3. ReAct-3 appears to be a plan, verify and replan loop with at most three rounds, and ReAct-4, Reflexion-4 and ExpeL-4 differ mainly in when a reflector is called. Is that right, what in them is ReAct, Reflexion or ExpeL, why were they designed this way, and what does the 4 stand for?
  4. Why these six models, and why do three of them run with reasoning and the other three without? What specific reason is there to pick these models, and are they representative open-source models for spatial reasoning? What question about agents is the model panel meant to answer? Please also report the reasoning effort and output limit for every model in the appendix, not only for GPT5.6-Luna.
  5. How much of the spatial reasoning is guided by the harness? The scaffold prompts already hint where the gate should cross each flake, and the data card in Appendix L says success may partly reflect discovering hidden requirements through feedback. Is there any run without these hints, and what is left for the agent to reason about?

Suggestions further

  1. Rewrite the abstract and introduction top down, covering what the task is, why it is hard for an agent, and what was found. Make the short names and version numbers self-contained.
  2. State the expert's relation to the authors in the paper.
  3. Add a separate section in the appendix introducing the tested models, and give the appendix a table of contents.
  4. If possible, also test production harnesses such as Codex or Hermes Agent with recent models (like GPT5.6-Luna in the paper, or Opus 4.8 or newer). Otherwise any insights about spatial reasoning found here might not reflect current progress. In my experience no one runs a task this complex with a ReAct loop on Ministral-3-14B, or on whatever model is meant to represent low-resource or local deployment, which the paper should state explicitly. In either case a ReAct loop is not what production environments use.
2 comments

Thank you for your time and feedback. We have substantially revised the manuscript to explain what the expert labels establish and to show which circuit connections were missing from the saved layouts. Below we answer your questions about structural errors and contact locking, with section numbers from the revised manuscript.

Expert labels. The reviewer of the layouts assigned one overall label to each selected verifier pass. Of 436 layouts, 260 received PASS. The available counts combine FAIL and UNSURE for the other 176 and give no breakdown by criterion. The individual labels and criterion-level reasons are unavailable. We cannot recover overlaps among the six criteria, count usability-only failures, or identify an expert-labeled example of that distinction (Section 6.2).

A structural error we can verify. On inverter L3/case_25, GPT5.6-Luna placed a transistor on each flake but left the drains on separate nets. The original verifier accepted it. Across 14 configurations, 127 of 128 originally passing CMOS attempts lacked a joined output under the added check. These findings come from the saved layouts. The expert’s reasons for individual labels remain unknown. None of the 14 released CMOS references passed the added wiring check. All 14 revised references passed it. Removing the inter-flake contact metal from each revised reference triggers a connectivity error in all 14 (Section 6.2).

Contact locking. On 69 via tasks, GPT5.6-Luna passed 152/345 attempts with whole-layout revision and 4/345 with ReAct-4, which locks contacts before checking via enclosure. In 296 of 333 ReAct-4 attempts with contacts on these tasks, the contacts lacked room to enclose a via. We did not run the requested condition in which ReAct-4 can unlock contacts after a via-enclosure failure, so there is no unlocked success rate to report. The two observed workflows also differed in prompts and check timing. The 296 cases identify an obstruction in the saved geometry. They do not tell us how many the agent would repair (Section 6.4).

Dear Authors,

Thank you for your response. With the discussion period drawing to a close, I have two formatting requests so I can review your updates:

  • Please reply point by point so I can quickly locate where each concern is addressed.
  • Please highlight all revisions in blue in the revised manuscript, as it is currently very difficult to identify what has been changed

Also, there is a major mismatch between your answers and what I asked for (for instance, I never requested an experiment on unlocking contacts, sorry if I misunderstand anything, and still all of my concerns remain unaddressed) therefore, I will maintain my original score.

Reviewer_t4vNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces 2D-FET-Bench V2, a set of 128 FET layout tasks built from graphene flake contours segmented from microscopy images, including holes, multiple flakes, and circuit templates. An agent receives a textual device specification, contour coordinates, layer definitions, and some geometric hints, and outputs polygon or path operations. An interpreter generates GDSII; a KLayout verifier checks design rules, geometric structure, and required layers, while a separate check tests whether the input contours were altered. Multi-flake scenes use synthetic placement, and some graphene contours are assigned MoSe₂/WSe₂ material roles. The authors construct verifier-passing reference layouts for all 128 tasks and independently re-score 7,680 recorded attempts. Six model configurations are each run five times per task. GPT5.6-Luna with ReAct-3/C0 achieves 62.3% pass@1, 80.5% pass@5, and 43.8% all-five. The authors also compare workflow and prompt configurations for that model. One device expert reviews one randomly sampled verifier-passing layout for each covered task in five configurations: 436 layouts in total, of which 260 (59.6%) are accepted.

Critical strengths
  1. A clearly defined, executable evaluation target. The tasks start from irregular real contours and cover multiple flakes, holes, different structures, and circuit templates; the integrity of fixed inputs is checked separately. All 128 reference layouts pass the current verifier, and an independent re-scoring reproduces the verdicts for 7,680 recorded attempts. The paper thus provides at least a well-defined test environment for studying errors in geometric generation. Passing the reference layouts establishes solvability only under the current rules, which the authors largely make clear.
  2. Repeated runs and a human audit offer valuable diagnostics. The authors report pass@1, five-attempt coverage, and all-five consistency, and perform sensitivity analyses for task dependence and multiple comparisons. They also proactively report that many verifier-passing layouts were not accepted in the expert audit. These results let readers see the gap between automated scoring and device judgment, rather than only a high automated pass rate.
Critical weaknesses
  1. Insufficient construct validity of the primary metric. In Section 6.5, the expert accepts only 260 of 436 sampled verifier-passing layouts; acceptance rates across configurations are 56.4%–63.5%. This is not a false-positive rate for all successful attempts, because only one passing layout was sampled per covered task. It nevertheless shows a substantial gap between “passes the verifier” and “forms the intended, experimentally usable device structure.” Appendices G and L further explain that ordinary transistor_count is a lower bound, a single wraparound contact can satisfy the two-sided-contact criterion, circuits receive no electrical netlist extraction, and some pad-cover/dielectric layers are checked only for nonempty geometry. Consequently, the current pass@1, pass@5, and model rankings cannot directly establish practical FET design capability or support the efficiency extrapolation in Section 7. The paper candidly states the limits of geometric scoring, but, for a benchmark that is central to the paper, the effect of these rule gaps on the main results still needs to be quantified.
  2. Weak causal interpretation of configuration comparisons. Section 5 and Appendix B explain that the six-model panel mixes serving interfaces, reasoning settings, and output-validation settings, so it compares model-plus-serving configurations. The GPT5.6-Luna workflows also differ in source-code hashes and concurrency settings; ReAct-3/C0 uses 2.46 times as many tokens as PaE; and the layer-by-layer approaches change prompting, verification timing, and layer locking together. Appendix B also states that the primary panel changed from the previously planned ReAct-4/74-task condition to ReAct-3/128 tasks after results became available. The authors disclose these circumstances, so the 27.0-percentage-point difference can be treated as a descriptive result for the recorded configuration packages. It does not, by itself, establish that the number of feedback rounds or any single workflow mechanism caused the gain. If these comparisons are a central contribution, more controlled ablations or more restrained conclusions are needed.
Questions for authors
  1. Metric validity (essential additional evidence): For the 176 layouts the expert did not accept, can you report the main reasons under the six audit criteria, their task types and configuration distribution, and results of a re-review or agreement with a second expert? The current design of sampling one passing layout per covered task cannot estimate expert acceptance across all attempts. Could you use probability sampling of attempts to report expert-judgment-adjusted pass@1/coverage and its uncertainty?
  2. Verifier (essential additional evidence): For the wraparound contacts, extra transistors, incorrect short circuits, and functional layers that satisfy only a nonempty check described in Appendices G and L, can you provide targeted counterexample tests, their frequency among passing layouts, and the effect on the main results after adding corresponding checks? This evidence would directly affect the assessment of benchmark validity.
  3. Interpretation of comparisons (essential clarification/additional experiment): Can you rerun a paired comparison of ReAct-3/C0 and PaE with the same fixed harness version, model settings, and concurrency settings, and report results under similar token budgets? If not, which conclusions apply only to the existing configuration packages, and which are supported as effects of verifier feedback itself?
2 comments

Thank you for your time and feedback. We have substantially revised the manuscript to report how often passing layouts missed the required output connection and to compare revision with repeated sampling at similar token cost. Below we answer your questions, with section numbers from the revised manuscript.

Circuit connections. We ran the original verifier again on all 8,960 saved layouts from 14 configurations. For the 980 layouts from the 14 CMOS tasks, we also checked the wiring. In the 12-configuration panel, none of the 107 CMOS layouts that passed the device checks also passed the wiring check. Across its two workflows, Gemma-4-31B generated 140 CMOS layouts. Of the 21 layouts that passed the device checks, only one also passed the wiring check. It was an inverter generated with ReAct-3/C0. The extractor found no path between n- and p-type terminals on the modeled layers in the other 127 original passes. For GPT5.6-Luna ReAct-3/C0, overall passes fell from 399/640 to 373/640, the number of tasks with at least one pass fell from 103/128 to 96/128, and the number passing all five times fell from 56/128 to 52/128. In a separate Opus 5.5 run with five attempts per CMOS task, 54 of 70 layouts passed the device checks. Six of those 54 also passed the wiring check. Wiring was checked after generation in both studies (Section 6.2).

Other rule gaps. Recounting the archived predicates for seven additional geometric checks removed 98 of 2,325 passes in the 7,680-attempt primary panel. The six primary models kept the same ordering, and Luna fell from 62.3% to 59.5%. Among applicable original passes, dielectric coverage removed 15/23. Exact-count and distinct-side-contact checks each removed 0/2,268. Isolated tests confirmed that a wraparound contact, an extra counted device, and a remote pad cover can each satisfy the corresponding original check. The observed label changes are specific to the added checks. We do not have an archive-wide frequency for all short circuits or spacing errors, or a joint count of removals by geometry and wiring (Sections 3 and 6.2).

Expert audit. An independent device researcher accepted 260 of 436 selected verifier-passing layouts. The other 176 received FAIL or UNSURE: 41, 48, and 44 under Luna C0, C1, and C2, then 23 under GLM-5/C0 and 20 under Kimi/C0. We have no individual reasons or task-type breakdown and no second rating of these layouts. One passing layout was sampled per covered task and configuration. That design gives a lower bound of 62/128 tasks with an expert-accepted Luna C0 layout. It cannot supply the requested attempt-weighted pass rate or its uncertainty (Section 6.2). We removed the estimated speedup over manual design.

I thank the authors for the clarifications. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.