- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OpticalBench is a simulated benchmark for VLM reasoning about optical experiments. It pairs declared setups with corresponding observations for Gaussian propagation, imaging, resolution, polarization, Fourier optics, and interference. The resource contains 1,961 scenes and 7,664 questions across Perception, Derivation, Prediction, and Decision, using 10,822 distinct rendered images and 31,887 image inputs counted across questions. Labels are computed from simulated physical states. Ten deployments are evaluated on a stratified 480-question test sample with item-dependent chance floors, and additional experiments examine visual-input dependence, reasoning budget, and held-out parameter ranges. The intended contribution is a regenerable, physically specified testbed with controlled evidence about VLM capabilities in idealized optical experiments.
Critical strengths
- A concrete and useful benchmark design. Pairing a declared setup with its corresponding observations and withholding task-relevant physical quantities provides a substantive resource contribution. The six task families and four reasoning levels cover more than isolated optics questions, while procedural generation supports controlled observation replacement and parameter resampling. The physics-grounded title is appropriate for this simulated setting; the paper need not introduce a new physics solver or demonstrate hardware automation to make this contribution useful.
- Several effective evaluation controls. Table E.4 provides strong evidence that the corresponding observations matter for Gemini 3.7 Flash on a separate 461-question sample: full-input accuracy is 51.2%, compared with 17.8% without images and 21.0% with another scene's frames. The reasoning-budget comparison, two-model held-out evaluation, and 480-question versus 4,629-question sampling check go beyond a single leaderboard. The paper also discusses answer-format effects, response failures, nonuniform answer positions, and wide per-cell intervals. These controls and qualifications strengthen the reported empirical observations.
Critical weaknesses
- The benchmark's physical validity and answerability are insufficiently auditable. Section 3.2 states appropriate validation procedures against closed-form solutions, an FFT core, an independent quadrature reference, and agreement between paraxial and diffraction solvers. However, neither the main text nor the appendix reports the validation configurations, quantitative errors, convergence behavior, or their relation to grading tolerances. Important generation details are also missing: complete physical and sensor parameter settings, the task-specific computation of s_min used to exclude unanswerable items, and the actual test/test-x parameter intervals. Table E.6 reports held-out results but does not specify those intervals. Without these details or an executable artifact that supplies them, I cannot independently assess whether numerical and measurement uncertainties are safely below the scoring tolerances or reproduce the claimed distribution shift. Appendix C does describe 123 format checks, 68 record checks, calibration and semantic gates, and a 72-item manual inspection. These are useful and should be credited, but internally consistent records and recomputed keys do not substitute for reported independent physical-accuracy and observability checks. I am not asserting that the simulator or its labels are wrong. The issue is that their verifiability is central to this benchmark's contribution, yet the reviewed materials do not provide enough evidence to assess it. This is the main reason my recommendation remains below acceptance.
- Visual dependence is established more clearly than the source of higher-level errors. Sections 4.5 and 5 conclude that failures above Perception lie in physics rather than in reading the frame. The conditional analysis relating a scene's Q1 success to its higher-level success does not establish that attribution. Correctly answering Q1 does not certify that the model extracted every quantity at the precision required by Q2-Q4. The paper's own Imaging example makes the distinction clear: identifying the direction of a displacement is different from measuring its magnitude to infer a mirror tilt or predict a continuous spot position. Near-unit conditional odds ratios and weak variation across s_min tertiles therefore do not rule out visual measurement as a source of higher-level failure. The existing input-removal and frame-substitution experiments establish dependence on the intended evidence, but they do not separate measurement from physical inference. A more decisive control would keep each higher-level question and grading rule fixed while supplying the correct relevant measurements, or measurements extracted by a profile-fitting/centroiding tool. This would show whether performance recovers when measurement is solved. The paper already acknowledges answer-format and chance-floor differences; those qualifications should also carry through to the interpretation of the reasoning ladder. This concern limits the diagnostic conclusion, rather than negating the benchmark's entire value.
Questions for authors
- Can the authors provide the generator and quantitative validation for each task family, including solver/reference errors versus grading tolerances, physical and sensor parameter tables, task-specific s_min definitions, and the actual in-distribution/held-out intervals? What is the acceptance criterion ensuring that a question is answerable from the provided noisy frames? This evidence would most directly affect my recommendation.
- On the same Q2-Q4 items, how much does accuracy improve when the required visual measurements are supplied correctly while the physical reasoning and answer format remain unchanged? Could the authors distinguish a measurement oracle from a full-state oracle, or narrow the physics-versus-reading conclusion to what the existing controls establish?
- Please report the shortcut-flagged Imaging Decision cell separately, since Appendix C says it is partly answerable from text. For chance tests, are item-level guessing probabilities constant within each tested cell? Are shared-scene dependencies accounted for in overall uncertainty and pairwise model comparisons? Please also give the matched item counts and column denominators for the reasoning-budget results in Table E.3.
6 comments
W2 & Q2 We thank the reviewer for raising this question. We agree that correctly answering Q1 does not certify that every quantity is extracted at the precision required by Q2–Q4. Following the reviewer's suggestion, we have added control experiments on the same Q2–Q4 questions that keep each question and its grading rule fixed while supplying the correct measurements (measurement oracle), measurements extracted by profile-fitting/centroiding tools, or the full simulated state (full-state oracle), as shown in Table R10.
Table R10. Accuracy (%) on the same Q2–Q4 questions (Derivation, Prediction and Decision), with the improvement of the overall accuracy over the original questions in parentheses.
| Model | Input | Q2 | Q3 | Q4 | All |
|---|---|---|---|---|---|
| Gemini 3.8 Flash | Original questions | 54.2 | 31.7 | 45.8 | 43.9 |
| Gemini 3.8 Flash | Measurement oracle | 91.7 | 76.7 | 75.8 | 81.4 (+37.5) |
| Gemini 3.8 Flash | Profile-fitting/centroiding tools | 90.8 | 74.2 | 73.3 | 79.4 (+35.6) |
| Gemini 3.8 Flash | Full-state oracle | 96.7 | 85.8 | 74.2 | 85.6 (+41.7) |
| GPT-5.6 Sol | Original questions | 71.7 | 60.8 | 60.0 | 64.2 |
| GPT-5.6 Sol | Measurement oracle | 100.0 | 86.7 | 92.5 | 93.1 (+28.9) |
| GPT-5.6 Sol | Profile-fitting/centroiding tools | 98.3 | 90.8 | 90.0 | 93.1 (+28.9) |
| GPT-5.6 Sol | Full-state oracle | 99.2 | 94.2 | 95.8 | 96.4 (+32.2) |
The results of the added experiments show that performance largely recovers when measurement is solved: the measurement oracle improves accuracy by 37.5 and 28.9 points overall and by 25.8 to 45.0 points at each level, profile-fitting/centroiding tools give similar gains, and the full-state oracle adds only 4.2 and 3.3 points beyond the measurement oracle. Thus, the added experiments confirm our finding that current VLMs show reasonable ability in the qualitative recognition of optical phenomena but limited ability in precise quantitative measurement. We thank the reviewer again for this suggestion and will revise our original claim about the source of higher-level failures in Sections 4.5 and 5. We will also add these experiments and the answer-format and chance-floor qualifications suggested by the reviewer to strengthen our analysis.
Q3 We thank the reviewer for these questions and answer them in order.
- Imaging Decision cell. Following the reviewer's suggestion, we report this cell separately in Table R11. The overall scores with and without this cell differ by at most 1.5 points, and we will also report this cell separately in Table 2.
- Guessing probabilities. Yes. The guessing probability is constant within each of the 24 cells, in the sample and the full test split, so the binomial test is exact.
- Shared-scene dependencies. We have added this analysis, and no conclusion changes: the within-scene correlation is 0.020 [−0.005, 0.045] (design effect 1.006), and an exact scene-level sign-flip test separates 34 of 45 pairs under Holm, as McNemar's test does.
- Matched item counts in Table E.3. Gemini 3.8 Flash has 120 / 119 / 120 / 119 matched items per level (478 in all). For Gemini 3.7 Flash, the printed columns have 120 items per level for the minimal setting and 398 / 308 / 378 / 349 for the default setting, with 81 items in common. The revised Table E.3 will report both settings on the same 480 items, with n for every column.
Table R11. Accuracy (%) of each model on the Imaging Decision cell (20 questions), and the change in its overall accuracy (points) without this cell.
| Model | Imaging Decision | Change in overall accuracy |
|---|---|---|
| DeepSeek V4 Flash Vision | 0.0 | +0.6 |
| Qwen3-VL instruct | 10.0 | +0.2 |
| Qwen3-VL thinking | 40.0 | −0.8 |
| Gemini 3.1 Pro | 15.0 | +0.9 |
| Grok 4.6 | 50.0 | −0.5 |
| Gemini 3.7 Flash | 25.0 | +0.8 |
| Claude Sonnet 5 | 45.0 | −0.1 |
| Gemini 3.8 Flash | 10.0 | +1.5 |
| GPT-5.6 Terra | 25.0 | +1.5 |
| GPT-5.6 Sol | 30.0 | +1.5 |
Dear Reviewer R1Tq,
Thank you for your valuable and constructive feedback. As the discussion period is coming to an end in less than two days, we would like to kindly follow up and ensure that our additional validation results and control experiments have adequately addressed your concerns.
If you have any remaining questions or concerns, please let us know, and we will do our best to address them. Your thoughtful suggestions have been invaluable in strengthening our work, and if you find our responses satisfactory, we would sincerely appreciate your reconsideration of the initial assessment and a possible score increase.
Thank you again for your time and consideration!
Sincerely,
The Authors
We thank Reviewer R1Tq for the precise and constructive review, and for stating which evidence would affect the assessment. We address each concern below.
W1 & Q1 We thank the reviewer for this detailed comment. We have released the full codebase, including the generator and the validation scripts, through an anonymous link: https://anonymous.4open.science/r/OpticalBench-42A0/. Following the reviewer's suggestion, we have added the following details.
Table R8. Solver error on 12 to 24 random scenes per phenomenon (half from test-x; for Polarization, from the solver's self-test) and grading tolerance (the gap between options for choice questions).
Table R9. Held-out parameter of each phenomenon and its ranges in test and test-x.
We thank the reviewer again for these suggestions and will add these details to Appendices C and E.