acceptodds
Withdrawn from ICLR 2027

PICAgentBench: Benchmarking Multimodal Agents for Photonic Integrated Circuit Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agents for photonic integrated circuit (PIC) design must interpret visual and structural evidence, reason about constraints, and produce valid design artifacts. Final-artifact success alone provides limited insight into which engineering tasks agents perform reliably or where errors arise. We introduce PICAgentBench, a multimodal benchmark with a selected bank of 982 cases across six separately scored PIC engineering tasks. These tasks comprise requirement-to-design, device recognition, circuit recognition, layout-to-netlist extraction, fault localization, and fault repair. They place overlapping demands on perception, reasoning, and action through task-specific combinations of specifications, visual layouts, and structured circuit representations. Fault repair evaluates constrained geometric editing given a local crop and known fault type. Expert-developed checking scripts and manual review support reference construction, with explicit observation and verification boundaries defining what each score measures. A shared agent environment supports model comparisons under common interaction conditions, while task-specific acceptance criteria define six-task performance profiles. Separate native metrics distinguish device classification from localization and circuit identity from potential function. Together with task-level breakdowns and resource reporting, these profiles provide a structured basis for identifying model strengths and limitations in PIC design.

Primary area:
datasets and benchmarks
Keywords:
photonic integrated circuit, multimodal benchmark, agent, design automation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QnkmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PICAgentBench, a benchmark for evaluating multimodal agents across photonic integrated circuit engineering tasks, including design reasoning, recognition, connectivity extraction, fault localization, and constrained repair. Its main contribution is a collection of task-specific observation and output contracts, aligned design artifacts, and analytical, structural, geometric, and semantic evaluation procedures. Validation combines expert-developed checks and manual review with multi-model experiments and analyses of task performance, errors, and resource use.

Critical strengths
  1. Broader coverage of PIC engineering operations. The benchmark complements prior generation-focused resources by separately evaluating recognition, connectivity extraction, diagnosis, and artifact editing. Native metrics distinguish errors in device labels, localization, ports, circuit identity, and function, providing more useful detail than a single end-to-end success rate.
  2. Explicit artifact and verification boundaries. The protocol separates participant inputs from private references and post-submission grading. Graph equivalence permits alternative encodings, while repair checks admit non-nominal solutions and enforce preservation constraints. The paper also carefully distinguishes geometric validity from physical recovery and supplied diagnosis from autonomous fault discovery.
Critical weaknesses
  1. The incremental diagnostic value is insufficiently demonstrated. Existing work already provides PIC agents and design benchmarks, including PICBench, PhIDO, PICopilot, and PIC-Set. The experiments primarily compare underlying models within one shared agent environment, without establishing whether the new tasks reveal actionable limitations beyond model rankings. Modality ablations, task-appropriate domain baselines, or controlled downstream interventions would help establish what the added task decomposition measures. In particular, structured graph/context inputs for circuit recognition and supplied fault diagnoses for repair substantially constrain the interpretation of these scores.
  2. Incomplete coverage and evaluator qualification weaken the comparisons. The main results use model-dependent resolved populations, while the appendix reports substantial missingness and different denominators for native metrics and complete-task success. Their implications for rankings are not sufficiently clear in the main presentation. TASK-3 additionally relies on a participating model as semantic judge without reported human calibration, and some TASK-4 annotations await independent review. Together with source-dependent variants and the absence of source-held-out evaluation, these issues limit confidence in comparative and generalization claims (Appendices E–F).
Questions for authors
  1. Clarification affecting assessment: Establish the exact eligible and scored populations underlying the main tables, explain denominator changes between native metrics and complete-task success, and provide common-case comparisons or sensitivity bounds for unresolved outcomes. Confirm how pending TASK-4 annotation reviews affect the reported results. Validation evidence affecting assessment: Provide any existing human calibration/adjudication evidence for 2. TASK-3 and any completed controls showing that the added tasks measure useful PIC engineering capabilities beyond structured-input interpretation and submission compliance. Additional agent architectures or broader physical validation would be valuable future work; the immediate concern is the evidence supporting the current scores and their interpretation.
Reviewer_DqVbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PICAgentBench, a multimodal benchmark covering six representative tasks across the PIC design workflow, with 982 test cases spanning diverse devices, circuits, and applications. The evaluation of 13 models demonstrates the benchmark's feasibility and provides useful insights into their capabilities and limitations. However, several aspects could be further strengthened, including the connection between benchmark errors and real-world engineering failures, the role of functional simulation in fault detection and repair, and the dependencies and error propagation across tasks. Additionally, ablation studies on input modalities and tool usage would help clarify their respective contributions to agent performance.

Critical strengths
  1. The paper provides comprehensive coverage of the PIC design workflow by decomposing it into six representative engineering tasks. The relatively large dataset of 982 cases, spanning diverse devices, circuits, and applications, offers a valuable benchmark for systematically evaluating multimodal agents.

  2. The paper evaluates a diverse set of 13 models across the proposed tasks, demonstrating the feasibility and practical utility of the benchmark.

Critical weaknesses
  1. While fine-grained error diagnosis is valuable for understanding agent limitations, real-world PIC engineering failures are often more complex than task-level failures or predefined layout faults. It would be helpful for the authors to clarify how the benchmark's error categories relate to practical engineering failures and discuss whether the current diagnostic scope is sufficient to reflect real-world challenges.

  2. The fault localization and repair tasks are valuable, but their evaluation primarily focuses on geometric and structural correctness. In practical PIC design, a geometrically valid repair does not necessarily restore the intended circuit functionality. It would strengthen the benchmark to incorporate functional simulation for representative cases, assessing whether identified faults affect circuit behavior and whether repairs actually restore the desired functionality.

  3. While the decomposition of the PIC design workflow into six tasks is valuable, these tasks are primarily evaluated independently. It would be interesting to explore the dependencies between upstream and downstream tasks and whether joint optimization could improve overall performance, such as incorporating fault prevention into the initial design stage. Furthermore, it remains unclear how errors propagate across tasks and affect downstream outcomes.

  4. The benchmark emphasizes multimodal agent capabilities, but the contributions of different input modalities and tools remain unclear. Ablation studies comparing image-only, structured-data-only, and combined inputs, where applicable, would help clarify whether performance gains arise from visual understanding, structured representations, or tool-assisted processing.

Questions for authors
  1. The authors are encouraged to clarify the definition and categorization of errors in the benchmark and their correspondence to real-world PIC engineering failures.

  2. It would be helpful to discuss or evaluate the necessity of incorporating functional simulation, particularly whether image-based analysis alone is sufficient to identify relevant errors and verify that repairs restore the intended circuit functionality.

  3. Further analysis of error propagation across upstream and downstream tasks, as well as the potential benefits of joint optimization, would strengthen the evaluation of end-to-end PIC design capabilities.

  4. Additional ablation studies on input modalities and tool usage would help clarify their respective contributions to agent performance.

Reviewer_vswwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multimodal benchmark named PICAgentBench, containing 982 test cases across six photonic integrated circuit (PIC) tasks. The authors evaluate 13 Large Language Models (LLMs) using the OpenHands agent framework to compare their performance across different tasks, and the results are reported as per-task scores rather than a single aggregate score.

Critical strengths

Instead of concentrating on a single task, the authors present a benchmark that contains six PIC tasks, and each task has its own input, output, and metric. The authors compare the performance of 13 LLMs using a shared agent framework, which gives good insight into their strengths and limitations.

Critical weaknesses

One of the paper's arguments is that native metrics reveal failure modes through a success rate. However, the main text only reports complete-task success rates, and the per-task acceptance criteria are not defined in the main text. Multiple analyses are discussed in Section 4, such as failure and recovery analysis, but they are not supported by corresponding examples and experimental results. The fault-repair evaluation concentrates on geometric correctness rather than investigating the functionality of the repaired circuit. In Table 4, the scores for compact models on Task 4 are zero and also nearly zero on Task 5. It remains unclear which types of failures contribute more to these low scores. Several of the 982 examples in the benchmark share the same source material, and it remains unclear whether the diversity of source designs is sufficient to evaluate generalization.

Questions for authors

What explains the 100% success rate of top models on the fault-repair task? What are the complete-task acceptance criteria for each task, and how do the authors decide if a task is completed successfully? Could the authors also evaluate the functionality of the repaired circuits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.