acceptodds
Under review as a conference paper at ICLR 2027

Vision-language agents for active perception in optics laboratories

Abstract

Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Vision-language agents, embodied AI, active perception, laboratory automation, optical alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RE8TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether VLMs can control real optical experiments in closed loop. It also provides lightweight simulations of the three systems. The agent receives a system prompt(setup, goal, guidance). Each round, it outputs an action(executed on the hardware). Each task targets a different capability:

  1. Michelson interferometer (visual state interpretation and actuator-to-image mapping): GPT-5.6 Sol, Gemini 3.5 Flash, and Claude Sonnet 5 all succeed.

  2. Two-mirror cavity (resolving ambiguous feedback via interaction history): GPT-5.6 Sol 9/10, Gemini 6/10, Qwen3-VL-32B 0/10. All models always declare completion.

  3. Four-mirror relay(choosing what to measure when feedback is missing, by moving the camera upstream to align segment by segment): GPT-5.6 Sol 9/10 , Gemini 10/10.

Critical strengths

Strength 1: A new and somewhat surprising capability result. VLMs can control real optical experiments in closed loop, one action at a time, using only raw camera images. Many researchers in ICLR community may be interested in these results.

Strength 2: The paper explains when a VLM agent is more useful than numerical optimization, and the accompanying simulations make it possible to study this question at scale.

Critical weaknesses

Weakness 1: Missing controls make it hard to attribute the results to the VLM itself. For example, there is no ablation that weakens the strategy guidance in the prompt, and no text-only control to show how much each parts actually contributes. The paper also does not test whether a simple pipeline (e.g. spot detection with a standard alignment algorithm) could achieve the same results.

Weakness 2: The evaluation is small in scale and coarse in granularity, the results cannot explain when and why the agent succeeds or fails.

Questions for authors
  1. How much of the success comes from the procedural guidance in the prompt?
  2. Given the same information, does the VLM outperform simple baselines?
  3. Where exactly do failures occur? Maybe categorize failures using the logs, and describe what actually happens
  4. Does the simulation hold up as a testbed beyond a single model? Can the simulation reproduce the differences between models, as well as the failing configuration?
  5. It is unclear how well this approach scales, since four mirrors is a rather simple setup. The hardest part of real experiments is likely the coupling between elements: fixing one throws another out of alignment. How to coordinate dozens of elements so that everything suddenly snaps into place currently depends entirely on human intuition and feel. I am curious whether this approach has any hope of replacing that.
1 comment

We thank the reviewer for highlighting the value of testing VLMs on real optical experiments as well as the interest to the ICLR community. We have added prompt and interaction-history ablations, a classical baseline with access to intermediate camera measurements, and new laboratory results for Claude Sonnet 5 on the four-mirror relay experiment to explore failure modes. We address the reviewer's questions below. All changes to the revised manuscript are highlighted in blue.

  1. We evaluated GPT-5.6 Sol on the same ten simulated starting configurations per task using three prompts: a basic task description, a prompt with general control guidance, and the full laboratory prompt with task-specific verification procedures and failure cases (Sec. 3.6 and Appendix F). We found that the three prompts yielded similar alignment errors on the Michelson interferometer and similar iteration counts on the four-mirror relay whereas the two-mirror cavity was more sensitive to guidance. The full prompt achieved 10/10 successful alignments of the cavity, compared with 6/10 for the simpler prompts. We observed that the full prompt led the agent to gather more evidence before settling on an apparent overlap. We also varied access to interaction history while holding the basic prompt and visual input fixed. In the four-mirror relay, success rose from 0/10 with the current observation alone to 10/10 with one prior observation also available (Appendix C). This shows that a basic task description is often sufficient for alignment, but that detailed guidance can improve reliability.

  2. As the reviewer notes, a baseline with comparable access to measurements is important. We added a classical controller for the simulated relay that uses the same camera stations, estimates the beam center from pixel intensities, and uses motor perturbations to estimate how the beam moves (Sec. 3.5 and Appendix A). Both methods aligned the relay, with GPT-5.6 Sol using fewer iterations on average than this implementation. In the Michelson and cavity simulations, the VLM and Bayesian optimization reached low error on similar iteration scales, and Bayesian optimization ultimately achieved lower error in the cavity. We emphasize that we see these capabilities as complementary since a VLM can help recover informative feedback by deciding where to measure, and a conventional optimizer can refine the alignment once a reliable objective is available.

  3. Sections 3.2 to 3.4 describe the observed failure modes for each experiment. In the Michelson interferometer, the less precise runs ended with an elongated spot, indicating incomplete overlap. In the cavity, some runs misidentified the faint secondary beam, leading to inaccurate actuator-response estimates and premature completion. In the relay, mistaking diffuse speckle for the beam could lead the agent to proceed downstream before the first segment was aligned. The revised manuscript adds ten Claude Sonnet 5 laboratory runs on the relay where the agent only aligned 3/10 configurations. In five of its seven failures, it declared success based on speckle, fringes, or diffuse illumination. In the other two, it moved downstream before confirming alignment of the first segment and exhausted the iteration budget.

  4. We selected GPT-5.6 Sol for the simulations because it performed best overall across the laboratory tasks. The simulations reproduce broad trends for this model, including similar convergence behavior on the Michelson and cavity tasks and sequential alignment on the relay (Appendix A and Fig. 5). The simulated relay succeeded on all ten nominal starting configurations, including the one associated with the laboratory failure in which GPT-5.6 Sol mistook diffuse speckle for the beam. The simulations capture the main features of the observations and actuator responses while simplifying real-world optical effects. Since the non-idealities of the real-world laboratory environment are not included, we do not expect the performance trends to match exactly between simulation and experiment.

  5. As the reviewer points out, coupling between optical elements is a central challenge. In practice, experimentalists often align large optical systems by working through smaller optical sections in sequence, checking intermediate beam positions and revisiting earlier sections when a downstream measurement reveals a problem. Our four-mirror relay tests this approach on eight motorized degrees of freedom. The agents recovered upstream feedback, aligned successive sections, and in some runs went back to an earlier measurement point to refine alignment before proceeding. This suggests a path to larger setups whose subsections are comparable in complexity to those tested here. Setups with strong coupling across sections would test whether the agent can tell when a local correction is sufficient and when it needs to revisit several elements together. Testing at this scale remains an important next step for future work.

Reviewer_7uuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses VLM models to directly control and optimize optics laboratory tasks. They focus on three real world optical tasks: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. They build an agentic harness which gives the LLMs access to a set of instructions, sensor images and the ability to directly control the actuators which determine the alignment of these systems. Overall they find that for these tasks today’s frontier VLMs can serve as useful decision making agents.

Critical strengths

Overall the writing and figures are fairly easy to follow and communicate the method clearly.

The use of real optical hardware with agents taking actions in the real world is a strength. Furthermore the development of a simulation which is consistent with this physical hardware is an important contribution.

Critical weaknesses

Unclear if this is really a good fit for ICLR. While this paper is an application paper of frontier models to optics experiments, there is no direct examination of any form of learning or representations. I feel like this may be a better fit for an optics or physics conference. I think it is difficult to make claims about general scientific reasoning or problem solving with only experiments done in a fairly limited optics setup.

Overall I feel the contribution is somewhat limited relative to prior work. Choi et al. (2026) [A Framework for Closed-Loop Robotic Assembly, Alignment and Self-Recovery of Precision Optical Systems] uses a similar set of optical tasks. Moreover in this paper they use an automated system for what appears to be a similar cavity task, which achieves similar results to the LLM results outlined in this paper. That is an important missing baseline. To be clear I am definitely not an expert in optics, so I am not sure how much these tasks correspond to each other.

Questions for authors

A more thorough explanation of the differences between the experiments and contributions of Choi et al. (2026) and this paper (for a non optics expert).

To what extent are the instructions or prompt engineering provided to the model responsible for the performance ? Are they models able to do this task without detailed context on the correct approach ?

2 comments

We thank the reviewer for highlighting the real-hardware experiments and matched simulations. A central challenge for physical AI in science is enabling models to use observations from a live experiment to decide what to do next. This capability remains underexplored in part because relatively few laboratory instruments can be controlled programmatically and provide feedback after each action. Our results from three real optical setups are a promising step towards automating low-level laboratory tasks that require reasoning based on visual feedback. We address the reviewer's comments and questions below. All changes to the revised manuscript are highlighted in blue.

  1. Fit with ICLR: the 2027 Call for Papers explicitly includes "applications in robotics" and "to other fields" and ICLR has a history of pubilshing many AI-for-science papers, including work on autonomous scientific discovery, materials discovery, and protein design. Our contribution fits within this category and represents a behavioral evaluation of pretrained VLMs as closed-loop controllers in optics. We believe that our work will be of interest to the broader ICLR community.

  2. Comparison with Choi et al. (2026). We thank the reviewer for highlighting this related work. Choi et al. automate optical assembly, alignment, and recovery using task-specific computer vision and optimization routines. Their workflow defines an explicit objective for each alignment step, such as a score based on beam intensity and quality for cavity recovery. In contrast, our experiments focus on situations in which the available measurement is ambiguous or provides too little information to guide the next adjustment. We test whether a pretrained vision-language agent can draw on domain knowledge and the observed effects of its previous actions to interpret ambiguous images and decide where to measure next when feedback is missing. We see these capabilities as necessary when informative feedback is unavailable and complementary to the alignment methods of Choi et al. In an automated laboratory, an agent could first recover informative feedback and then hand off to a conventional optimizer once a reliable objective is available. We have clarified this distinction in the related works section of our paper.

  3. Dependence on prompt guidance. To address the reviewer's question, we added a three-level prompt ablation study using our matched simulations (Sec. 3.6 and Appendix F). For each task, we evaluated GPT-5.6 Sol on the same ten simulated starting configurations with a basic prompt describing the apparatus, goal, and available actions, a strategy prompt adding general control guidance, and the full laboratory prompt, which includes task-specific verification procedures and failure cases. The three prompts yielded similar alignment errors on the Michelson interferometer and similar iteration counts on the four-mirror relay. With the basic relay prompt, the agent moved the camera upstream to recover a visible beam and revisited an earlier station in four of ten runs. The two-mirror cavity was more sensitive to procedural guidance. The full prompt achieved 10/10 successful alignments, compared with 6/10 for each reduced prompt. Thus, we find that a basic task description was enough for the agent to align all three systems in our matched simulation, and explicit guidance made the two-mirror cavity alignment more reliable.

Thank you for the detailed response. I have read and considered it. The prompt ablations helped clarify that the LLM performance was not a function of explicit guidance. I choose to keep my score and confidence unchanged.

Reviewer_BAajOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether pretrained VLMs can function as closed-loop scientific agents that directly control physical experiments through visual feedback rather than relying on predefined numerical objectives. The authors develop a VLM-based control framework and evaluate it on three optical alignment tasks designed to test complementary capabilities including learning actuator-response relationships through intervention, resolving ambiguous visual observations using interaction history, and actively selecting informative measurements. Laboratory experiments and matched simulations, including comparisons with Bayesian optimization and random search, demonstrate promising VLM capabilities under sparse or missing visual feedback.

Critical strengths
  1. The paper demonstrates that general-purpose VLMs can perform closed-loop control across three real optical systems using visual feedback without engineered scalar objectives, providing meaningful evidence of their potential for scientific automation.
  2. The three optical setups examine distinct capabilities, including actuator-response estimation, resolving ambiguous visual observations, and active measurement selection, offering a broader evaluation than a single experimental demonstration.
  3. The four-mirror relay experiment demonstrates that VLM agents can actively reposition a camera to recover informative observations when downstream feedback is unavailable, highlighting a potentially valuable capability for scientific agents.
  4. The authors construct matched simulation environments and compare VLMs with conventional optimization baselines, providing additional quantitative evidence for evaluating their behavior in optical alignment tasks.
Critical weaknesses
  1. Concerns about the prompt design

My main concern is that the prompts provide extensive task-specific guidance, making it difficult to determine how much of the reported performance can actually be attributed to the VLM's own reasoning and active perception capabilities.

(1) Overly prescriptive control strategies. The authors emphasize that VLMs can learn actuator-response relationships, resolve visual ambiguity, and actively seek informative observations. However, in the second task, the prompt in Appendix C.3 already explicitly describes the visual ambiguity, prescribes how to resolve it, and specifies how to verify successful alignment, making it unclear whether the model independently develops effective strategies or primarily follows expert-designed procedures with visual feedback.

(2) Task-specific optimization guidance. Although the agents do not receive engineered scalar objectives, the prompts explicitly instruct them to estimate beam coordinates, infer quantitative actuator-response relationships, and predict target motor settings. This means that substantial optimization knowledge is already encoded in the prompts, which I think qualifies the paper's framing of control without engineered scalar objectives.

(3) Predefined failure cases. The prompts also anticipate many potential failure cases, such as stationary optical artifacts in the second task and speckle clouds in the third. This makes me wonder whether the models can genuinely recognize and resolve unexpected ambiguities, or whether their robustness largely depends on having potential failure modes described in advance.

  1. Lack of ablation studies

The paper repeatedly attributes the agents' success to their ability to leverage interaction history and perform informative interventions, yet provides no ablation studies to isolate the contributions of these mechanisms.

  1. Concerns about the BO baseline

I find the BO comparison somewhat problematic, particularly in the four-mirror relay task, where BO receives uninformative feedback while the VLM can actively reposition the camera. This comparison highlights the value of active sensing but does not necessarily demonstrate an advantage specific to VLM reasoning. A non-VLM baseline with comparable measurement capabilities would be more convincing.

Questions for authors
  1. Prompt ablation. Could the authors compare VLM performance under three levels of prompt complexity: (i) only basic optical principles and experimental setup, including the task goal and available actions; (ii) additional general control strategies, but without explicit guidance on handling ambiguities or specific failure cases; and (iii) the full prompts used in the paper? I would be interested in how the performance changes across these settings.
  2. Autonomous ambiguity resolution. I am particularly curious whether, without explicit guidance, VLMs can independently recognize and resolve visual ambiguities, such as distinguishing genuine beam overlap from a beam disappearing outside the camera's field of view. Can the models identify this ambiguity and actively design interventions to resolve it? Together with ablations on interaction history and interventions, these results would help clarify how much of the reported capability comes from the VLM's own reasoning versus detailed expert-designed prompts.
2 comments

We thank the reviewer for their thoughtful feedback. We appreciate their positive assessment of our work's potential for laboratory automation and their recognition that the matched simulations enable quantitative comparisons. As programmable laboratory hardware opens up new opportunities for automation, integrating AI agents directly into the control loop lets us test their ability to adjust physical instruments under real laboratory conditions. This capability remains underexplored and is central to laboratory automation. Below, we address the reviewer's concerns in detail. We have revised the manuscript accordingly and highlighted all changes in blue.

  1. How much does performance depend on prompt guidance? We added the suggested ablation in Sec. 3.6 and Appendix B (Fig. 6). In simulation, we evaluated the agent on the same ten initial configurations per task under three prompts. The basic prompt describes the setup, goal, available actions, and basic optics. The strategy prompt adds general guidance on using observations to guide adjustments. The full laboratory prompt includes explicit procedures for alignment, verification, and handling known failure modes. Under all three prompts, the agent reached similar final best-so-far errors on the Michelson interferometer and completed the relay in similar numbers of iterations. Cavity alignment was more sensitive to guidance, with 6/10 successes under each of the simpler prompts and 10/10 under the full prompt. These results show that the agent can use detector images to guide alignment with basic task instructions. By "without an engineered scalar objective," we mean that the agent receives camera images and derives its own estimates of progress from them. The full prompts still provide task-specific control guidance.

  2. Can the agent resolve cavity ambiguity on its own, and what role does history play? With either of the two simpler prompts, the agent used observed beam motion to estimate the actuator response and predict an overlap setting. It sampled opposite sides of the primary beam in 4/10 runs under each simpler prompt, compared with 10/10 under the full prompt. The full prompt elicited a median of five probe images before the first apparent overlap, compared with two under either reduced prompt. With the basic prompt, the agent continued probing after the first apparent overlap in three runs, and one of those runs ended with lower alignment error. The logs therefore show some self-directed probing and provide evidence that the agent reliably distinguishes overlap from a beam leaving the field of view without explicit guidance.

As suggested by the reviewer, we also varied access to interaction history while holding the basic prompt and action interface fixed (Appendix C, Fig. 7). Removing prior interactions degraded performance on all three tasks. In the relay, success rose from 0/10 with no history to 10/10 with one prior interaction. This ablation measures the benefit of retaining previous interactions. The effect of disabling interventions remains unmeasured.

  1. How does the relay compare with a classical controller that can also move the camera? The reviewer raised a concern about the original Bayesian optimization comparison, where feedback from the final camera can be uninformative until the beam reaches it. We added a classical controller with access to the same camera stations and image feedback as the VLM in simulation (Sec. 3.5 and Appendix A, Fig. 5). The controller visits stations in a programmed sequence and uses image-based beam localization and motor perturbations to guide corrections. Both methods recovered useful intermediate feedback and aligned the relay. GPT-5.6 Sol averaged 28.2 iterations, while the classical controller averaged 46.0. The iteration difference is specific to the controller we tested.

Our ablation studies showed that the agent could draw on its own domain knowledge to perform alignment tasks without explicit guidance and that access to interaction history improved alignment across all three tasks. Explicit guidance in the prompt improved performance on more difficult alignment tasks, particularly in the two-mirror cavity task.

Thank you for the additional ablations and baseline comparison. These address several concerns, but reliable autonomous ambiguity resolution without explicit guidance remains insufficiently demonstrated, particularly in the cavity task. I therefore maintain my original score of 2 (weak rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.