- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether VLMs can control real optical experiments in closed loop. It also provides lightweight simulations of the three systems. The agent receives a system prompt(setup, goal, guidance). Each round, it outputs an action(executed on the hardware). Each task targets a different capability:
-
Michelson interferometer (visual state interpretation and actuator-to-image mapping): GPT-5.6 Sol, Gemini 3.5 Flash, and Claude Sonnet 5 all succeed.
-
Two-mirror cavity (resolving ambiguous feedback via interaction history): GPT-5.6 Sol 9/10, Gemini 6/10, Qwen3-VL-32B 0/10. All models always declare completion.
-
Four-mirror relay(choosing what to measure when feedback is missing, by moving the camera upstream to align segment by segment): GPT-5.6 Sol 9/10 , Gemini 10/10.
Critical strengths
Strength 1: A new and somewhat surprising capability result. VLMs can control real optical experiments in closed loop, one action at a time, using only raw camera images. Many researchers in ICLR community may be interested in these results.
Strength 2: The paper explains when a VLM agent is more useful than numerical optimization, and the accompanying simulations make it possible to study this question at scale.
Critical weaknesses
Weakness 1: Missing controls make it hard to attribute the results to the VLM itself. For example, there is no ablation that weakens the strategy guidance in the prompt, and no text-only control to show how much each parts actually contributes. The paper also does not test whether a simple pipeline (e.g. spot detection with a standard alignment algorithm) could achieve the same results.
Weakness 2: The evaluation is small in scale and coarse in granularity, the results cannot explain when and why the agent succeeds or fails.
Questions for authors
- How much of the success comes from the procedural guidance in the prompt?
- Given the same information, does the VLM outperform simple baselines?
- Where exactly do failures occur? Maybe categorize failures using the logs, and describe what actually happens
- Does the simulation hold up as a testbed beyond a single model? Can the simulation reproduce the differences between models, as well as the failing configuration?
- It is unclear how well this approach scales, since four mirrors is a rather simple setup. The hardest part of real experiments is likely the coupling between elements: fixing one throws another out of alignment. How to coordinate dozens of elements so that everything suddenly snaps into place currently depends entirely on human intuition and feel. I am curious whether this approach has any hope of replacing that.
We thank the reviewer for highlighting the value of testing VLMs on real optical experiments as well as the interest to the ICLR community. We have added prompt and interaction-history ablations, a classical baseline with access to intermediate camera measurements, and new laboratory results for Claude Sonnet 5 on the four-mirror relay experiment to explore failure modes. We address the reviewer's questions below. All changes to the revised manuscript are highlighted in blue.
We evaluated GPT-5.6 Sol on the same ten simulated starting configurations per task using three prompts: a basic task description, a prompt with general control guidance, and the full laboratory prompt with task-specific verification procedures and failure cases (Sec. 3.6 and Appendix F). We found that the three prompts yielded similar alignment errors on the Michelson interferometer and similar iteration counts on the four-mirror relay whereas the two-mirror cavity was more sensitive to guidance. The full prompt achieved 10/10 successful alignments of the cavity, compared with 6/10 for the simpler prompts. We observed that the full prompt led the agent to gather more evidence before settling on an apparent overlap. We also varied access to interaction history while holding the basic prompt and visual input fixed. In the four-mirror relay, success rose from 0/10 with the current observation alone to 10/10 with one prior observation also available (Appendix C). This shows that a basic task description is often sufficient for alignment, but that detailed guidance can improve reliability.
As the reviewer notes, a baseline with comparable access to measurements is important. We added a classical controller for the simulated relay that uses the same camera stations, estimates the beam center from pixel intensities, and uses motor perturbations to estimate how the beam moves (Sec. 3.5 and Appendix A). Both methods aligned the relay, with GPT-5.6 Sol using fewer iterations on average than this implementation. In the Michelson and cavity simulations, the VLM and Bayesian optimization reached low error on similar iteration scales, and Bayesian optimization ultimately achieved lower error in the cavity. We emphasize that we see these capabilities as complementary since a VLM can help recover informative feedback by deciding where to measure, and a conventional optimizer can refine the alignment once a reliable objective is available.
Sections 3.2 to 3.4 describe the observed failure modes for each experiment. In the Michelson interferometer, the less precise runs ended with an elongated spot, indicating incomplete overlap. In the cavity, some runs misidentified the faint secondary beam, leading to inaccurate actuator-response estimates and premature completion. In the relay, mistaking diffuse speckle for the beam could lead the agent to proceed downstream before the first segment was aligned. The revised manuscript adds ten Claude Sonnet 5 laboratory runs on the relay where the agent only aligned 3/10 configurations. In five of its seven failures, it declared success based on speckle, fringes, or diffuse illumination. In the other two, it moved downstream before confirming alignment of the first segment and exhausted the iteration budget.
We selected GPT-5.6 Sol for the simulations because it performed best overall across the laboratory tasks. The simulations reproduce broad trends for this model, including similar convergence behavior on the Michelson and cavity tasks and sequential alignment on the relay (Appendix A and Fig. 5). The simulated relay succeeded on all ten nominal starting configurations, including the one associated with the laboratory failure in which GPT-5.6 Sol mistook diffuse speckle for the beam. The simulations capture the main features of the observations and actuator responses while simplifying real-world optical effects. Since the non-idealities of the real-world laboratory environment are not included, we do not expect the performance trends to match exactly between simulation and experiment.
As the reviewer points out, coupling between optical elements is a central challenge. In practice, experimentalists often align large optical systems by working through smaller optical sections in sequence, checking intermediate beam positions and revisiting earlier sections when a downstream measurement reveals a problem. Our four-mirror relay tests this approach on eight motorized degrees of freedom. The agents recovered upstream feedback, aligned successive sections, and in some runs went back to an earlier measurement point to refine alignment before proceeding. This suggests a path to larger setups whose subsections are comparable in complexity to those tested here. Setups with strong coupling across sections would test whether the agent can tell when a local correction is sufficient and when it needs to revisit several elements together. Testing at this scale remains an important next step for future work.