- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PICAgentBench, a benchmark for evaluating multimodal agents across photonic integrated circuit engineering tasks, including design reasoning, recognition, connectivity extraction, fault localization, and constrained repair. Its main contribution is a collection of task-specific observation and output contracts, aligned design artifacts, and analytical, structural, geometric, and semantic evaluation procedures. Validation combines expert-developed checks and manual review with multi-model experiments and analyses of task performance, errors, and resource use.
Critical strengths
- Broader coverage of PIC engineering operations. The benchmark complements prior generation-focused resources by separately evaluating recognition, connectivity extraction, diagnosis, and artifact editing. Native metrics distinguish errors in device labels, localization, ports, circuit identity, and function, providing more useful detail than a single end-to-end success rate.
- Explicit artifact and verification boundaries. The protocol separates participant inputs from private references and post-submission grading. Graph equivalence permits alternative encodings, while repair checks admit non-nominal solutions and enforce preservation constraints. The paper also carefully distinguishes geometric validity from physical recovery and supplied diagnosis from autonomous fault discovery.
Critical weaknesses
- The incremental diagnostic value is insufficiently demonstrated. Existing work already provides PIC agents and design benchmarks, including PICBench, PhIDO, PICopilot, and PIC-Set. The experiments primarily compare underlying models within one shared agent environment, without establishing whether the new tasks reveal actionable limitations beyond model rankings. Modality ablations, task-appropriate domain baselines, or controlled downstream interventions would help establish what the added task decomposition measures. In particular, structured graph/context inputs for circuit recognition and supplied fault diagnoses for repair substantially constrain the interpretation of these scores.
- Incomplete coverage and evaluator qualification weaken the comparisons. The main results use model-dependent resolved populations, while the appendix reports substantial missingness and different denominators for native metrics and complete-task success. Their implications for rankings are not sufficiently clear in the main presentation. TASK-3 additionally relies on a participating model as semantic judge without reported human calibration, and some TASK-4 annotations await independent review. Together with source-dependent variants and the absence of source-held-out evaluation, these issues limit confidence in comparative and generalization claims (Appendices E–F).
Questions for authors
- Clarification affecting assessment: Establish the exact eligible and scored populations underlying the main tables, explain denominator changes between native metrics and complete-task success, and provide common-case comparisons or sensitivity bounds for unresolved outcomes. Confirm how pending TASK-4 annotation reviews affect the reported results. Validation evidence affecting assessment: Provide any existing human calibration/adjudication evidence for 2. TASK-3 and any completed controls showing that the added tasks measure useful PIC engineering capabilities beyond structured-input interpretation and submission compliance. Additional agent architectures or broader physical validation would be valuable future work; the immediate concern is the evidence supporting the current scores and their interpretation.