acceptodds
Under review as a conference paper at ICLR 2027

OpticalBench: A Physics-Grounded Benchmark for Vision-Language Reasoning on Optical Experiments

Abstract

Artificial intelligence is moving into the experimental loop of the physical sciences, proposing experiments, driving instruments and reading what they return. However, an optical system is still set up and aligned by hand, and whether a measurement is sound is judged by a human experimenter rather than by anything automatic. Vision-language models (VLMs) are a promising tool for such systems, given their increasingly strong reasoning. Yet no benchmark measures their capabilities on optical experiments across different levels of reasoning and various optical tasks. To bridge this gap, we introduce OpticalBench, a benchmark of setup–result pairs spanning six representative optical tasks, with each instance comprising the system components, their poses and parameters, the setup diagram, and the corresponding optical results. OpticalBench contains 1,961 scenes and 31,887 image inputs in total. Based on these setup–result pairs, we synthesise 7,664 question–answer pairs at four progressive levels of reasoning: Perception, Derivation, Prediction, and Decision, all grounded in physically meaningful quantities and relationships. We evaluate ten proprietary and open-weight deployments under a unified protocol, with per-item chance floors and controlled visual ablations. Experimental results reveal substantial differences across models: overall accuracy ranges from 13.3% to 65.0%, averaging 38.0% across the ten deployments. Performance also varies markedly across reasoning levels, averaging 47.9% for Perception, 39.3% for Derivation, 28.6% for Prediction, and 36.3% for Decision, revealing substantial room for improvement in reasoning over optical experiments. OpticalBench serves as a physics-grounded benchmark for measuring and advancing the capabilities of VLMs toward automated optical experimentation.

Primary area:
datasets and benchmarks
Keywords:
optics, physics-grounded reasoning, vision-language models, multimodal reasoning, scientific reasoning, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_R1TqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpticalBench is a simulated benchmark for VLM reasoning about optical experiments. It pairs declared setups with corresponding observations for Gaussian propagation, imaging, resolution, polarization, Fourier optics, and interference. The resource contains 1,961 scenes and 7,664 questions across Perception, Derivation, Prediction, and Decision, using 10,822 distinct rendered images and 31,887 image inputs counted across questions. Labels are computed from simulated physical states. Ten deployments are evaluated on a stratified 480-question test sample with item-dependent chance floors, and additional experiments examine visual-input dependence, reasoning budget, and held-out parameter ranges. The intended contribution is a regenerable, physically specified testbed with controlled evidence about VLM capabilities in idealized optical experiments.

Critical strengths
  1. A concrete and useful benchmark design. Pairing a declared setup with its corresponding observations and withholding task-relevant physical quantities provides a substantive resource contribution. The six task families and four reasoning levels cover more than isolated optics questions, while procedural generation supports controlled observation replacement and parameter resampling. The physics-grounded title is appropriate for this simulated setting; the paper need not introduce a new physics solver or demonstrate hardware automation to make this contribution useful.
  2. Several effective evaluation controls. Table E.4 provides strong evidence that the corresponding observations matter for Gemini 3.7 Flash on a separate 461-question sample: full-input accuracy is 51.2%, compared with 17.8% without images and 21.0% with another scene's frames. The reasoning-budget comparison, two-model held-out evaluation, and 480-question versus 4,629-question sampling check go beyond a single leaderboard. The paper also discusses answer-format effects, response failures, nonuniform answer positions, and wide per-cell intervals. These controls and qualifications strengthen the reported empirical observations.
Critical weaknesses
  1. The benchmark's physical validity and answerability are insufficiently auditable. Section 3.2 states appropriate validation procedures against closed-form solutions, an FFT core, an independent quadrature reference, and agreement between paraxial and diffraction solvers. However, neither the main text nor the appendix reports the validation configurations, quantitative errors, convergence behavior, or their relation to grading tolerances. Important generation details are also missing: complete physical and sensor parameter settings, the task-specific computation of s_min used to exclude unanswerable items, and the actual test/test-x parameter intervals. Table E.6 reports held-out results but does not specify those intervals. Without these details or an executable artifact that supplies them, I cannot independently assess whether numerical and measurement uncertainties are safely below the scoring tolerances or reproduce the claimed distribution shift. Appendix C does describe 123 format checks, 68 record checks, calibration and semantic gates, and a 72-item manual inspection. These are useful and should be credited, but internally consistent records and recomputed keys do not substitute for reported independent physical-accuracy and observability checks. I am not asserting that the simulator or its labels are wrong. The issue is that their verifiability is central to this benchmark's contribution, yet the reviewed materials do not provide enough evidence to assess it. This is the main reason my recommendation remains below acceptance.
  2. Visual dependence is established more clearly than the source of higher-level errors. Sections 4.5 and 5 conclude that failures above Perception lie in physics rather than in reading the frame. The conditional analysis relating a scene's Q1 success to its higher-level success does not establish that attribution. Correctly answering Q1 does not certify that the model extracted every quantity at the precision required by Q2-Q4. The paper's own Imaging example makes the distinction clear: identifying the direction of a displacement is different from measuring its magnitude to infer a mirror tilt or predict a continuous spot position. Near-unit conditional odds ratios and weak variation across s_min tertiles therefore do not rule out visual measurement as a source of higher-level failure. The existing input-removal and frame-substitution experiments establish dependence on the intended evidence, but they do not separate measurement from physical inference. A more decisive control would keep each higher-level question and grading rule fixed while supplying the correct relevant measurements, or measurements extracted by a profile-fitting/centroiding tool. This would show whether performance recovers when measurement is solved. The paper already acknowledges answer-format and chance-floor differences; those qualifications should also carry through to the interpretation of the reasoning ladder. This concern limits the diagnostic conclusion, rather than negating the benchmark's entire value.
Questions for authors
  1. Can the authors provide the generator and quantitative validation for each task family, including solver/reference errors versus grading tolerances, physical and sensor parameter tables, task-specific s_min definitions, and the actual in-distribution/held-out intervals? What is the acceptance criterion ensuring that a question is answerable from the provided noisy frames? This evidence would most directly affect my recommendation.
  2. On the same Q2-Q4 items, how much does accuracy improve when the required visual measurements are supplied correctly while the physical reasoning and answer format remain unchanged? Could the authors distinguish a measurement oracle from a full-state oracle, or narrow the physics-versus-reading conclusion to what the existing controls establish?
  3. Please report the shortcut-flagged Imaging Decision cell separately, since Appendix C says it is partly answerable from text. For chance tests, are item-level guessing probabilities constant within each tested cell? Are shared-scene dependencies accounted for in overall uncertainty and pairwise model comparisons? Please also give the matched item counts and column denominators for the reasoning-budget results in Table E.3.
6 comments

We thank Reviewer R1Tq for the precise and constructive review, and for stating which evidence would affect the assessment. We address each concern below.

W1 & Q1 We thank the reviewer for this detailed comment. We have released the full codebase, including the generator and the validation scripts, through an anonymous link: https://anonymous.4open.science/r/OpticalBench-42A0/. Following the reviewer's suggestion, we have added the following details.

  1. Solver and reference errors versus grading tolerances. Table R8 compares the solver error with the grading tolerance for each phenomenon. Where tested, grid refinement changes these errors by at most 0.025% (Imaging 0.002 µm, Interference visibility 9e-5), the quadrature reference matches the ABCD solution within 0.0002% on its test configuration, and the FFT core matches the closed-form beam radius within 0.013% on random configurations.
  2. Physical and sensor parameters. All scenes use λ = 632.8 nm. Every sampled parameter with its range and the sensor model (shot noise, 1 e⁻ read noise, 12-bit digitization and lossless 8-bit PNG) are specified in the released code.
  3. s_min definitions. For each phenomenon, s_min is the smallest singular value of diag(1/σ)·J·diag(τ), where J is the Jacobian of a closed-form model from the withheld parameters to the observables, σ is the declared noise of each observable and τ is the tolerance of each withheld parameter. Thus, s_min ≥ 1 means that a change of one tolerance in the withheld parameters moves the observables by at least one noise level. All 1,961 scenes have s_min ≥ 1.014.
  4. In-distribution and held-out intervals. Table R9 gives the held-out parameter of each phenomenon and its ranges in test and test-x.
  5. Acceptance criterion. A scene is kept only if s_min ≥ 1 and its frames pass the render checks. Classical tools with the forward physics, seeing only what the model sees, recover 7,560 of the 7,664 keys (98.6%) from the frames.

Table R8. Solver error on 12 to 24 random scenes per phenomenon (half from test-x; for Polarization, from the solver's self-test) and grading tolerance (the gap between options for choice questions).

PhenomenonQuantitySolver errorGrading tolerance
Gaussianbeam width0.025%5%
Imagingspot centroid≤ 0.002 µm20%
Resolutionspot spacing (resolved frames)≤ 0.35%8%
Polarizationdegree of linear polarization1e-60.159
Fourierorder offset0.008%13.8%
Interferencefringe visibility4e-40.10

Table R9. Held-out parameter of each phenomenon and its ranges in test and test-x.

PhenomenonHeld-out parametertesttest-x
Gaussianlens focal length (mm)[60, 202.5)[202.5, 250]
Imaginglens focal length (mm)[100, 250)[250, 300]
Resolutionlens focal length (mm)[75, 123.75)[123.75, 140]
Polarizationsource waist (µm)[150, 337.5)[337.5, 400]
Fourierfirst lens focal length (mm){150, 200, 250, 300}{100}
Interferencearm beam radius (mm)[1.5, 2.625)[2.625, 3.0]

We thank the reviewer again for these suggestions and will add these details to Appendices C and E.

W2 & Q2 We thank the reviewer for raising this question. We agree that correctly answering Q1 does not certify that every quantity is extracted at the precision required by Q2–Q4. Following the reviewer's suggestion, we have added control experiments on the same Q2–Q4 questions that keep each question and its grading rule fixed while supplying the correct measurements (measurement oracle), measurements extracted by profile-fitting/centroiding tools, or the full simulated state (full-state oracle), as shown in Table R10.

Table R10. Accuracy (%) on the same Q2–Q4 questions (Derivation, Prediction and Decision), with the improvement of the overall accuracy over the original questions in parentheses.

ModelInputQ2Q3Q4All
Gemini 3.8 FlashOriginal questions54.231.745.843.9
Gemini 3.8 FlashMeasurement oracle91.776.775.881.4 (+37.5)
Gemini 3.8 FlashProfile-fitting/centroiding tools90.874.273.379.4 (+35.6)
Gemini 3.8 FlashFull-state oracle96.785.874.285.6 (+41.7)
GPT-5.6 SolOriginal questions71.760.860.064.2
GPT-5.6 SolMeasurement oracle100.086.792.593.1 (+28.9)
GPT-5.6 SolProfile-fitting/centroiding tools98.390.890.093.1 (+28.9)
GPT-5.6 SolFull-state oracle99.294.295.896.4 (+32.2)

The results of the added experiments show that performance largely recovers when measurement is solved: the measurement oracle improves accuracy by 37.5 and 28.9 points overall and by 25.8 to 45.0 points at each level, profile-fitting/centroiding tools give similar gains, and the full-state oracle adds only 4.2 and 3.3 points beyond the measurement oracle. Thus, the added experiments confirm our finding that current VLMs show reasonable ability in the qualitative recognition of optical phenomena but limited ability in precise quantitative measurement. We thank the reviewer again for this suggestion and will revise our original claim about the source of higher-level failures in Sections 4.5 and 5. We will also add these experiments and the answer-format and chance-floor qualifications suggested by the reviewer to strengthen our analysis.

Q3 We thank the reviewer for these questions and answer them in order.

  1. Imaging Decision cell. Following the reviewer's suggestion, we report this cell separately in Table R11. The overall scores with and without this cell differ by at most 1.5 points, and we will also report this cell separately in Table 2.
  2. Guessing probabilities. Yes. The guessing probability is constant within each of the 24 cells, in the sample and the full test split, so the binomial test is exact.
  3. Shared-scene dependencies. We have added this analysis, and no conclusion changes: the within-scene correlation is 0.020 [−0.005, 0.045] (design effect 1.006), and an exact scene-level sign-flip test separates 34 of 45 pairs under Holm, as McNemar's test does.
  4. Matched item counts in Table E.3. Gemini 3.8 Flash has 120 / 119 / 120 / 119 matched items per level (478 in all). For Gemini 3.7 Flash, the printed columns have 120 items per level for the minimal setting and 398 / 308 / 378 / 349 for the default setting, with 81 items in common. The revised Table E.3 will report both settings on the same 480 items, with n for every column.

Table R11. Accuracy (%) of each model on the Imaging Decision cell (20 questions), and the change in its overall accuracy (points) without this cell.

ModelImaging DecisionChange in overall accuracy
DeepSeek V4 Flash Vision0.0+0.6
Qwen3-VL instruct10.0+0.2
Qwen3-VL thinking40.0−0.8
Gemini 3.1 Pro15.0+0.9
Grok 4.650.0−0.5
Gemini 3.7 Flash25.0+0.8
Claude Sonnet 545.0−0.1
Gemini 3.8 Flash10.0+1.5
GPT-5.6 Terra25.0+1.5
GPT-5.6 Sol30.0+1.5

Dear Reviewer R1Tq,

Thank you for your valuable and constructive feedback. As the discussion period is coming to an end in less than two days, we would like to kindly follow up and ensure that our additional validation results and control experiments have adequately addressed your concerns.

If you have any remaining questions or concerns, please let us know, and we will do our best to address them. Your thoughtful suggestions have been invaluable in strengthening our work, and if you find our responses satisfactory, we would sincerely appreciate your reconsideration of the initial assessment and a possible score increase.

Thank you again for your time and consideration!

Sincerely,
The Authors

Reviewer_wZvfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OpticalBench, a physics-grounded benchmark for evaluating VLMs on simulated optical experiments. It covers six optical tasks and four reasoning levels—Perception, Derivation, Prediction, and Decision—with answers generated from physically consistent simulations. Evaluating ten VLM deployments, the authors find that models can often recognize qualitative visual patterns but struggle with precise measurement and physics-based prediction. The benchmark also shows that longer reasoning improves some derivation tasks but does not solve the underlying measurement gap, highlighting the need for tool-assisted perception and physics-grounded training.

Critical strengths
  1. Rather than testing isolated textbook knowledge, OpticalBench requires models to connect an optical setup with its observed consequences. This directly addresses capabilities needed for scientific experimentation: measurement, physical inference, intervention prediction, and action selection.

  2. Ground truth is computed from simulated physical states and validated against analytical and numerical references. Withholding selected physical quantities from the setup makes observations necessary, while procedural generation enables controlled parameter changes rather than relying on a fixed collection of figures.

  3. Removing images or substituting unrelated observations substantially reduces performance, while combining the correct frames with the setup improves it. These controls provide meaningful evidence that the benchmark evaluates setup–observation reasoning rather than primarily rewarding memorized knowledge or textual shortcuts.

  4. The results distinguish qualitative recognition from precise quantitative measurement and show that additional reasoning helps some derivation tasks without improving measurement. This motivates integrating measurement tools and physics-grounded supervision, rather than assuming that longer reasoning alone will resolve scientific visual reasoning failures.

Critical weaknesses
  1. All scenes are generated by idealized simulators, without important laboratory complications such as alignment drift, stray light, or detector artefacts. The held-out split changes parameter ranges within the same task templates, rather than testing unfamiliar experimental configurations. Thus, physical consistency is well supported, but transfer to real optical experimentation remains uncertain.

  2. Numeric tolerance checks, all-or-nothing multiple selection, and single-choice questions impose substantially different scoring burdens. For example, Gaussian Derivation accuracy is low despite reasonably accurate individual judgments, partly because all four judgments must be correct simultaneously. Chance correction does not eliminate this mismatch, so differences across tasks and reasoning levels cannot be attributed solely to physical reasoning ability.

  3. Correctly answering a scene’s Perception question does not guarantee that the model extracted every quantity needed for its higher-level questions. Likewise, additional reasoning failing to improve measurement does not identify the underlying bottleneck. Supplying oracle measurements, comparing against profile-fitting tools, and evaluating reasoning from those fixed measurements would more convincingly isolate where failures occur.

  4. Most comparisons use only 20 questions per evaluation cell, while full-split validation covers one deployment and the principal reasoning-budget comparison concerns one model. Moreover, minimal reasoning settings, provider-dependent behavior, and generation failures contribute to the reported scores, particularly for open-weight deployments. Broader full-split evaluations and compute–accuracy comparisons are needed to establish whether the observed gaps reflect model capability rather than evaluation configuration.

Questions for authors
  1. How can you distinguish physical-reasoning failures from measurement errors and grading effects without oracle-measurement controls and format-matched comparisons across reasoning levels?

  2. Why was generalization tested only through parameter shifts within the simulator, and what evidence supports extending the findings to real optical experiments?

4 comments

We thank Reviewer wZvf for the expert and positive assessment of our benchmark design and controls. We address the concerns below.

W1 & Q2 We thank the reviewer for raising this important point. We intentionally chose simulation because it provides reliable ground truth and allows controlled changes of parameters and conditions, which are difficult to achieve in real experiments. If current VLMs already struggle with these synthetic scenes, they are even less likely to succeed in real experiments, as we will discuss in the revised Section 4.5. However, to bring the evaluation as close to real experiments as possible, we have added an experiment in which one laboratory artefact is applied to the images of each question, an approach that prior work has used to narrow the gap to real imaging conditions [ref #1–#4]. The results for each artefact are shown in Table R4.

Table R4. Accuracy (%) on the original images and with one laboratory artefact.

ArtefactGemini 3.8 Flash: originalGemini 3.8 Flash: with artefactGPT-5.6 Sol: originalGPT-5.6 Sol: with artefact
Stray light: ramp48.650.066.275.7
Stray light: halo47.148.258.865.9
Stray light: ghost47.646.476.261.9
Detector: hot/dead pixels50.653.074.769.9
Detector: fixed-pattern noise43.138.555.458.5
Detector: over-exposure54.250.079.270.8
Alignment drift: ±3 px shift53.860.072.367.7
All48.849.468.166.9

We chose a mild level for each artefact without degrading the quality of the questions. The results show only small changes in overall accuracy (+0.6 and −1.2 points). In the revised manuscript, we will highlight real experiments and stronger artefacts without compromising question quality as future work.

Regarding the held-out split, we agree with the reviewer that test-x only shifts parameter ranges within the same templates. We will state this in Sections 1 and 4.4.

References

  • [ref #1] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
  • [ref #2] Wei, K., Fu, Y., Yang, J. and Huang, H. A physics-based noise formation model for extreme low-light raw denoising. CVPR, 2020.
  • [ref #3] Wu, Y., et al. How to train neural networks for flare removal. ICCV, 2021.
  • [ref #4] Qiu, J., et al. Benchmarking robustness of multimodal image-text models under distribution shift. Journal of Data-centric Machine Learning Research, 2024.

W2 We thank the reviewer for this careful observation and agree that the answer formats impose different scoring burdens. To address this issue, we have added an analysis that compares the four levels within the same answer format. The results are shown in Table R5.

Table R5. Chance-corrected accuracy (%) of the seven proprietary models for the three answer formats noted by the reviewer.

Answer formatPerceptionDerivationPredictionDecision
Numeric tolerance check45.028.615.4no questions
All-or-nothing multiple selectionno questions30.214.237.1
Single choice73.851.735.226.8

The results show that the drop above Perception remains for both numeric and single-choice questions, and multiple-selection questions score similarly to numeric questions at the same level. We thank the reviewer again for this suggestion and will add this analysis to Section 4.2 to clarify the effect of the answer formats.

W3 & Q1 We thank the reviewer for this insightful comment. Following the reviewer's suggestion, we have added control experiments that supply oracle measurements, compare them against measurements from profile-fitting tools, and evaluate reasoning from those fixed measurements, keeping each higher-level question and its grading unchanged. The results are shown in Table R6.

Table R6. Accuracy (%) on the higher-level questions.

InputGemini 3.8 FlashGPT-5.6 Sol
Original questions43.964.2
With oracle measurements81.493.1
With measurements from profile-fitting tools79.493.1

The results of the added control experiments show that oracle measurements raise accuracy by 37.5 and 28.9 points, and measurements from profile-fitting tools give similar accuracy. Thus, the added experiments confirm our finding that current VLMs show reasonable ability in the qualitative recognition of optical phenomena but limited ability in precise quantitative measurement. We thank the reviewer again for this suggestion and will add these experiments to Section 4.5 to more convincingly isolate where failures occur.

W4 We thank the reviewer for these constructive suggestions and address each concern below.

  1. Sample size and full-split validation. We would like to clarify that we treat the results of individual cells as descriptive (Section 4.1) and base the main comparisons on 480 questions per model and 120 per level. For Gemini 3.8 Flash, accuracy changes by only 0.1 and 1.0 points with 40 and 60 questions per cell, and by 0.1 points on the full test split.
  2. Reasoning budget. We repeated this comparison on Gemini 3.7 Flash and GPT-5.6 Sol. The gain of 4.8 points for Gemini 3.8 Flash in Section 4.3 does not replicate: accuracy changes by −1.0 points for Gemini 3.7 Flash and +1.5 points for GPT-5.6 Sol, and neither change is significant. We will revise the reasoning-budget findings in Sections 1, 4.3 and 4.5 accordingly.
  3. Minimal reasoning settings. With the provider's default reasoning instead of the minimal setting, the drop above Perception persists in all three models, so this drop is not caused by the minimal setting.
  4. Provider-dependent behavior. On another host (DeepInfra), Qwen3-VL instruct scores 19.6%, compared with 14.8% on Alibaba, and stays within six points of the 16.2% chance floor on both hosts.
  5. Generation failures. With a completion limit of 32k tokens, we re-ran the 29 questions on which DeepSeek V4 Flash Vision had reached the limit of 16k tokens. It answers 6 of them correctly, so its accuracy rises only from 13.3% to 14.6%.
  6. Compute–accuracy comparison. Across the ten deployments, accuracy shows no consistent relation with completion tokens or cost per question. Within each of the three models in item 2, default reasoning uses 2 to 11 times more completion tokens than the minimal setting, with accuracy changes of −1.0 to +4.8 points.

Table R7 summarizes these results.

Table R7. Accuracy change for each change in evaluation configuration.

Configuration changeModelAccuracy change (points)
40 instead of 20 questions per cellGemini 3.8 Flash−0.1
60 instead of 20 questions per cellGemini 3.8 Flash−1.0
Full test split instead of the sampleGemini 3.8 Flash+0.1
Default instead of minimal reasoningGemini 3.8 Flash+4.8
Default instead of minimal reasoningGemini 3.7 Flash−1.0
Default instead of minimal reasoningGPT-5.6 Sol+1.5
Another host (DeepInfra)Qwen3-VL instruct+4.8
Token limit of 32k instead of 16kDeepSeek V4 Flash Vision+1.3

Overall, these configuration effects are at most 4.8 points, much smaller than the main gaps in the paper: the two strongest models lead all others by more than 15 points, and the average accuracy falls from 47.9% at Perception to 28.6% at Prediction. Thus, these gaps reflect model capability rather than evaluation configuration. We thank the reviewer again for these suggestions and will add these results to Sections 4.1, 4.3 and 4.5, Appendix D and Table E.3.

Dear Reviewer wZvf,

Thank you for your valuable comments and positive assessment of our work. As the discussion period is coming to an end in less than two days, we would like to kindly follow up and ensure that we have adequately addressed your concerns.

If you have any remaining questions or concerns, please let us know, and we will do our best to address them. Your positive evaluation means a great deal to our team, and if you find our responses satisfactory, we would sincerely appreciate your consideration of a score increase.

Thank you again for your time and consideration!

Sincerely,
The Authors

Dear Reviewer wZvf,

Thank you again for your constructive suggestions. As the discussion period ends in a few hours, we would like to kindly follow up on our response. The additions promised in our response are now in the updated PDF, marked in blue, and we list their locations below for convenience.

  • W1 & Q2. Paragraph 4 of Section 4.5 [page 10], paragraph 4 of Appendix G [page 35] and Table G.3 [page 36]. The scope of the held-out split is stated in paragraph 4 of Section 1 [page 2] and paragraph 1 of Section 4.4 [page 8].
  • W2. Paragraph 4 of Section 4.2 [page 7], paragraph 5 of Appendix G [page 35] and Table G.4 [page 36].
  • W3 & Q1. Paragraph 2 of Section 4.5 [page 9], and paragraph 2 of Appendix G and Table G.1 [page 35].
  • W4. Paragraph 5 of Section 1 [page 3], paragraph 3 of Section 4.3 [pages 7–8], paragraph 3 of Section 4.5 [pages 9–10] and Table E.3 [page 20]. The results on sample size, host, generation failures and compute are in paragraph 3 of Section 4.1 [page 6], the last paragraph of Appendix D [page 19], and paragraph 6 of Appendix G and Table G.5 [page 36].

We sincerely thank you for your recognition of our work, which means a great deal to our team. If any concern remains, please let us know, and we will do our best to address it in the remaining time. If you find our responses satisfactory, we would be truly grateful for your consideration of a score increase.

Sincerely,

The Authors

Reviewer_4usKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present OpticalBench, a benchmark for VLMs on evaluating simulated optical experiments outputs across a task matrix of 6 optical tasks and 4 levels of reasoning. Scenes are simulated in a python-based optical simulator, from which questions that evaluate perception, physical derivation, intervention prediction, and experimental decisions are made. This benchmark is then used to evaluate several open-source and proprietary VLMs. They find that most open-source VLMs have near-chance performance, while most proprietary models are quite poor in most tasks.

Critical strengths

The premise for this submission is great and well motivated. Integrating VLMs into experimental setups works well for optics, where the outputs naturally can be described in images. The physically-grounded data generation is an important aspect of the contribution. This is a novel and valuable benchmark in this sense.

The evaluations seem reasonable, testing different physical principles and reasoning stages of the same experiment covers a solid analysis of the VLM's understanding of the experiment.

Critical weaknesses

The main problem with the submission is the clarify of the communication. I am not an expert in optics but I am familiar with the field, and I had trouble understanding what the 'tasks' actually mean (Gaussian, Imaging, Resolution, Polarization, Fourier, and Interference). These are physical phenomenon or concepts, not tasks for the VLM. The 'levels of reasoning' are more clear (perception, derivation, prediction, and decision), but the paper does not really explain the details of how the reasoning questions are formed for each of these tasks. More clarity into exactly what is being asked of the VLM is essential for a benchmark paper, rather than naming the broad categories to which these questions belong. There is a lot of missing context to this paper.

The figures do not really make the setup more clear as they are trying to communicate a lot of information. I think simplifying figures 1 and 2, while providing at least one concrete end to end example would really help the clarity. For example, figure 2 indicates 'continuous refinement' of the scene simulation from the evaluation, but the paper's main text does not describe this at all. Figure 1 shows 'AI-driven optics' leading towards 'self-calibrating optics', which I can vaguely understand but I don't see how it fits into describing OpticalBench itself, being as how the benchmark is a synthetic optical experiment generator with evaluation capabilities, not in any way 'AI-driven optics'.

Another crucial missing point is how these questions are generated and quality controlled. The main paper says "question synthesis constructs physics-grounded question–answer pairs through four quality-control gates", and gives almost no additional detail except Appendix C Quality Control which claims dozens of checks but describes in detail none of them. It would appear that these questions are generated by an LLM, which is fine, but this must be admitted and explained in detail in order to be reproducible. For a benchmark paper, not detailing the process by which the actual questions were generated is a serious problem.

The benchmark data itself may be fine, but what would be more useful is the codebase which generates the data. This would be more extendable and practical for training and evaluating, rather than a static set of data. This points to the question-generation critique again; if that process was explained then the benchmark contribution would be much more valuable.

The benchmark seems to ask a lot from the VLM. From what I can tell, what is being evaluated is the model's zero-shot ability to analyze the set up or estimate factors of the experiment. That may be somewhat useful, but your results show that most models score nearly chance. I think a more valuable and realistic benchmark would be to evaluate the VLMs' ability to interact with the experiment to discern the answer, rather than asking the models to accomplish tasks like physical derivation or measuring angles/distances which they are known to be weak at.

Your second contribution is really a description of your first contribution. Having two contributions being the benchmark and evaluation is acceptable.

Questions for authors

How, in detail, are these questions generated and evaluated? What are the actual questions posed to the VLMs for each of the 6 tasks? Can you communicate this idea of evaluating the VLMs' understanding of each of these concepts be explained more clearly? What are the results when you allow the VLMs to interact with the optical experiment? Does this change the analysis substantially?

7 comments

We thank Reviewer 4usK for finding the benchmark well motivated and its physically grounded generation important, and for the concrete suggestions on clarity. We address the concerns below.

W1 & Q2 & Q3 We thank the reviewer for pointing out this lack of clarity.

  1. Clarification of the six optical phenomena. Following the reviewer's suggestion, we will correct the terminology from "tasks" to "optical phenomena" and, in the revised introduction and Section 3.1, explain these six phenomena and what is actually being asked of the VLM in each of them, as summarized below and in Table R1. Each phenomenon represents one class of optical experiment, with its own setup and measurement. In each phenomenon, we evaluate whether a VLM understands the setup (which components it contains and how they are placed) and how the setup determines the measurement (which quantities the cameras record and their physical meaning).

Table R1. Phenomenon description, setup components and measured quantities of the six optical phenomena.

Optical phenomenonDescriptionSetup componentsMeasured quantities
GaussianGaussian beam propagation: the beam radius is smallest at the waist and grows along the axislaser, lens, camerabeam radius
ImagingGeometrical optics: a tilted mirror deflects the beam and displaces the spotlaser, steering mirror, lens, cameraspot position
ResolutionTwo-point resolution: two nearby point sources are resolved when the dip between their images is deep enoughlaser, fibre pair, lens, cameraspot separation and dip contrast
PolarizationPolarization state of light (linear, circular or elliptical), changed by wave plates and measured by a rotating analyzerlaser, polarizer, half- and quarter-wave plates, rotating analyzer, cameratransmitted power versus analyzer angle
FourierFourier optics: a lens forms the Fourier transform of the field in its back focal plane, where an iris acts as a spatial filterlaser, crossed grating, two lenses, iris, camerapositions of the diffraction orders
InterferenceTwo-beam interference: the angle between the beams sets the fringe period, and their intensity ratio the visibilitylaser, two beamsplitters and two mirrors (Mach–Zehnder interferometer), neutral-density filter, camerafringe period and visibility, power in each arm
  1. Question design. We admit that the original manuscript did not explain this clearly. Following the reviewer's suggestion, we will highlight the steps and principles of question formation in the revised Section 3.3:

    (a) Scene simulation. For each phenomenon, we sample the setup and simulate its measurement. Some quantities, such as the mirror tilt in Imaging, are withheld from the setup diagram as candidate question targets, so they are inferred from the measurement.

    (b) Question synthesis. All levels share one procedure: a fixed template for each phenomenon and level is filled with the scene's quantities, and the answer is computed from the simulation. The levels follow the course of an experiment, each adding a reasoning step: Perception reads quantities from the measurement; Derivation infers withheld quantities from it; Prediction gives the effect of a stated change to the setup; and Decision compares such effects to choose the next measurement or setting.

  2. Question content. What is being asked of the VLM is related to the physical quantities of the setup and its measurement, such as the beam waist, mirror tilt, source separation, wave-plate angles, grating periods and angle between the beams in the six phenomena. One example of the actual questions posed to the VLMs is shown in Table R2. The revised Appendix F will present one full question for each question type.

Table R2. Example questions at the four levels for one Interference scene.

LevelQuestionAnswer
PerceptionHow do the four images taken with four neutral-density filters rank by fringe contrast (highest first)?2, 1, 3, 4
DerivationWhat are the angle between the two beams (1.96 or 8.94 mrad) and the fringe visibility (0.38 or 0.74)?1.96 mrad, 0.74
PredictionIf a filter of optical density 0.50 is placed in arm A, which are true: (1) the fringe visibility increases; (2) the fringes stay over 105 µm apart; (3) before the filter, the brighter arm carries over 8.8× the power of the dimmer?(2)
DecisionWhich filter, put in the brighter arm, would make the fringes deepest: OD 0.70, 0.84 or 1.55?OD 0.70

W2 We thank the reviewer for pointing this out. Following the reviewer's suggestion, we have simplified Figures 1 and 2, adding one end-to-end example in Figure 1 (please see the revised PDF). For this example, we chose a simple setup to help readers understand the benchmark. The detailed description of "continuous refinement" will be added in the revised Section 3.3. We have also corrected Figure 1 by removing the "AI-driven optics → self-calibrating optics" column, since it was meant to show our motivation rather than the benchmark. We thank the reviewer again for pointing this out.

W3 & Q1 We thank the reviewer for raising this insightful question. We acknowledge that the original manuscript did not give sufficient details of the process by which the actual questions were generated and quality controlled. We will add the following details to the revised manuscript (Sections 3.2 and 3.3 and Appendix C):

  1. Question generation. Generation has two stages. First, a Python optical simulator calculates the light field of each setup by solving the equations of light propagation. This gives paired data, a setup diagram and the corresponding simulated camera measurement, which are physically robust because they are obtained from these equations. Second, we generate question–answer pairs from these paired data in three steps: we design a question template for each phenomenon and reasoning level with LLM assistance, fill it with the quantities of each scene, and compute the answer from the simulation.
  2. Quality control. In the first stage, we apply two checks to screen the simulated scenes. An answerability check keeps only scenes whose answers can be inferred from the noisy measurement: changing the withheld quantities by one grading tolerance changes the measurement by at least one noise level. A render check confirms that the rendered images preserve the simulated quantities. In the second stage, we apply four gates to the generated items. The format gate checks for statistical shortcuts in answer positions and options, the audit gate checks the integrity of each item and its answer, the calibration gate checks the thresholds and numerical options stated in the questions, and the semantic gate checks whether an item can be answered without its images. Finally, five optics experts checked the question bank. Any problem they found was fed back to rebuild the generator, and the check was repeated until all checked items passed. The details of each check will be given in the revised Appendix C.

W4 We agree with the reviewer. We have released the full codebase through an anonymous link: https://anonymous.4open.science/r/OpticalBench-42A0/. With it, new scenes and question–answer pairs can be generated for training and evaluation.

W5 & Q4 We thank the reviewer for raising this concern and for suggesting interactive evaluation. We understand that the original manuscript did not present the results clearly. We would like to clarify that only the three open-weight models are near the chance floor of 16.2%, scoring 13.3–22.1%, whereas the seven proprietary deployments score 35.6–65.0%. Following the reviewer's suggestion, we have added an experiment that turns the zero-shot evaluation into a two-step interaction: after an incorrect answer, we tell the model that its answer is incorrect and allow it to answer the same question once more. The results are shown in Table R3. All three models show some improvement, and we will incorporate this additional experiment into the revised manuscript.

Table R3. Results of one more attempt on the same 140 questions for each model.

ModelWrong answers corrected (%)Accuracy change (points)
Qwen3-VL instruct4.7+4.3
Gemini 3.8 Flash5.9+5.0
GPT-5.6 Sol27.7+9.3

W6 Many thanks for this suggestion. We will rewrite the contributions as two items: the benchmark and the evaluation.

Dear Reviewer 4usK,

Thank you for your valuable comments and feedback. As the discussion period is coming to an end in less than two days, we would like to kindly follow up and ensure that we have adequately addressed your concerns.

If you have any remaining questions or concerns, please let us know, and we will do our best to address them. If you find our responses satisfactory, we would sincerely appreciate your consideration of a score increase.

Thank you again for your time and consideration!

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.