- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for evaluating VLMs under visually challenging conditions in embodied scenes. The benchmark organizes visual stress into four dimensions (Material, Viewpoint, Lighting, Geometry) and 16 fine-grained categories. It contains 7k examples collected from naturally occurring cases, synthetic stress generation, and additional real-world data. The benchmark covers target grounding, placement grounding, spatial reasoning, state understanding, and planning-style action selection. It also introduces matched nominal-versus-stressed scene pairs and a test-time stress-conditioned rectification baseline called StressDART.
The main contribution is a structured evaluation framework for studying how different visual stress factors affect model performance, together with an extensive evaluation across 16 VLMs.
Critical strengths
-
The organization of visual stress into Material, Viewpoint, Lighting, and Geometry is intuitive and grounded in image formation principles. This provides a useful structure for robustness analysis and makes benchmark results more interpretable than aggregate evaluation alone.
-
The paired evaluation is the strongest part for me. Comparing the same scene and question before and after editing reduces important confounders in ordinary benchmark comparisons. This is useful, although the interpretation still depends on the edits preserving the task-relevant content.
Critical weaknesses
-
My main concern is the evidence behind the matched evaluation. Section 3.2 describes a verified subset, but I could not find a clear account of how many pairs are used in Table 1, which tasks and stress categories they cover, or how the model-family scores are calculated. The paper describes manual checks, which is good, but it is still unclear how often editing changes other task-relevant content or makes a question unanswerable. Since these pairs support the stronger claims about stress-induced degradation, I would like this part to be more explicit.
-
StressDART gives a useful improvement, but the current comparison does not tell us how much comes from predicting the stress category. There is no comparison with generic image enhancement or editing without stress conditioning. Also, the aggregate gain does not show how often editing turns a correct prediction into a wrong one. The authors acknowledge both points, but they still limit what we can conclude about the method. Also the benchmark mixes naturally stressed images, synthetic edits, and additionally collected real images. I would like to see the main results separated by source, especially whether the stress-wise patterns are similar on natural and synthetic examples. This would help establish how broadly the findings hold within the paper's stated scope of embodied visual understanding.
Questions for authors
-
How many verified pairs are used in Table 1, and how are they distributed across tasks and stress categories? Please also clarify how the family-level scores are aggregated and provide results for the individual models.
-
Appendix A.5 gives the number of multiple-choice examples. Have you checked sensitivity to option order or performance without the image? The 99.8% planning accuracy of Qwen3-VL-30B-A3B in Table 2 stands out. Does it remain this high after shuffling the options?
-
Do the main stress-wise findings hold when naturally occurring and synthetically generated examples are evaluated separately?
-
How does StressDART compare with the same editor using a generic instruction without stress prediction? How often does rectification change a correct prediction to an incorrect one, and vice versa?
8 comments
[Q2] Option-Order Sensitivity, Text-Only Performance, and Planning Accuracy
We evaluate Qwen3-VL-30B-A3B on all 821 planning questions using text-only inputs and five random option permutations. Text-only evaluation removes the image. Option shuffling preserves the option text and updates the answer labels to match the new order.
| Planning subset | N | Original | Shuffled, mean ± SD | Text-only |
|---|---|---|---|---|
| All questions | 821 | 99.76% | 99.83% ± 0.11 pp | 77.10% |
| Questions without order-referencing options | 670 | 99.70% | 99.79% ± 0.13 pp | 79.25% |
The 670-question subset provides the primary option-order comparison because its options retain their meaning after permutation. Accuracy remains essentially unchanged across the tested orders, directly addressing whether the near-perfect result persists after shuffling.
Images provide a 22.66 pp gain over text-only input across all planning questions, with a paired 95% bootstrap interval of [19.85, 25.46]. The text-only results also reveal differences between question types:
| Question type | Text-only accuracy |
|---|---|
| Next interaction | 96.69% |
| Obstacle-related | 95.58% |
| Arrow direction | 56.04% |
These results explain the high aggregate score through the task composition. Next-interaction and obstacle-related questions are often solvable from text, while arrow-direction questions have substantially lower text-only accuracy. The action-selection task is close to saturation for this model, and images still contribute a measurable aggregate benefit.
We also checked option mapping, answer-label synchronization, and output parsing. Across all 2,823 multiple-choice examples, modifying the ground-truth field leaves the model prompt unchanged, verifying that this field does not directly enter the inspected prompt-construction path.
The text-only and permutation experiments specifically target the planning result highlighted by the reviewer. We will report their scope, question-type breakdown, and evaluation protocol in the revised paper.
[W2, Q3] Source-Specific Results and the Consistency of Stress-Wise Findings
We separate the benchmark by its recorded construction sources and compare performance within the same task, answer format, and stress dimension. The source distribution is:
| Source | Total | Target grounding | Placement | Spatial reasoning | State understanding | Planning |
|---|---|---|---|---|---|---|
| Selection from existing datasets | 2,927 | 283 | 193 | 1,194 | 604 | 653 |
| Controlled synthesis | 2,596 | 1,828 | 571 | 0 | 29 | 168 |
| Additional real-world sources | 1,660 | 1,300 | 185 | 175 | 0 | 0 |
| Total | 7,183 | 3,411 | 949 | 1,369 | 633 | 821 |
Controlled synthesis includes 1,828 scene-synthesis examples and 768 lighting edits. Additional real-world sources include web collection and our own collection. “Selection from existing datasets” identifies a construction path; the originating datasets determine the image provenance within that group.
The sources have different task compositions, so we compare results within task and answer-format groups. Using archived predictions from Qwen3-VL-30B-A3B, we obtain:
| Task / Source | Material | Viewpoint | Lighting | Geometry |
|---|---|---|---|---|
| Target grounding, bbox / Controlled synthesis | 48.33% (840) | — | — | 15.08% (988) |
| Target grounding, bbox / Additional real-world sources | 26.96% (345) | 24.12% (199) | 28.25% (177) | 31.59% (345) |
| Spatial reasoning, MCQ / Existing-dataset selection | 76.69% (532) | 72.80% (375) | 73.50% (434) | 68.87% (485) |
| Spatial reasoning, MCQ / Additional real-world sources | 58.95% (95) | 57.14% (49) | 72.73% (55) | 57.14% (105) |
Each cell reports accuracy and sample count. For this source analysis, bbox success uses IoU ≥ 0.95, and MCQ success uses exact answer correctness. Stress groups may overlap; “—” indicates an empty group.
The comparison reveals both shared difficulties and source-dependent stress profiles.
Target grounding. Precise localization remains difficult across synthetic and additionally collected examples: all reported bbox groups score below 50%. In the synthetic subset, Geometry scores 15.08%, below Material at 48.33%. In the additionally collected subset, Geometry scores 31.59%, above Material at 26.96%. Thus, localization difficulty appears across both sources, while the relative stress ranking changes.
Spatial reasoning. Geometry is among the lower-performing groups in both sources: 68.87% in existing-dataset selection and 57.14% in additional real-world sources. The highest-performing dimension changes from Material in the former to Lighting in the latter.
These results support the benchmark’s emphasis on task- and stress-specific diagnosis. They also show why source composition should be reported: a single aggregate ranking would conceal meaningful differences between source groups. We will include the source distributions and these within-task comparisons in the revised paper.
We sincerely thank the reviewer for the thoughtful and encouraging feedback, especially for recognizing the interpretability of the taxonomy and the value of the paired evaluation. We address the matched-evaluation protocol, multiple-choice diagnostics, source-specific results, and StressDART controls below.
[W1, Q1] Matched-Pair Composition, Verification, and Model-Level Results
The matched evaluation contains 2,591 verified nominal–stressed scene–question pairs, covering three stress dimensions and eight fine-grained categories. The distribution and corresponding performance changes are:
The task breakdown is:
These results average over 14 models. We compute two-sided 95% confidence intervals using 10,000 paired bootstrap resamples clustered by source image, keeping all questions and edited variants sharing the same nominal image together. Changes are computed from unrounded scores.
Pair verification. Each retained pair preserves the question and, for multiple-choice items, the answer options. The referred target remains the same. Spatial coordinates are updated where necessary to identify that target in the edited image; lighting edits preserve the original target annotations. Every retained pair was manually inspected for question answerability.
We quantify the verification outcomes as follows:
The excluded total counts unique pairs because the two failure criteria can overlap. We will include these statistics and representative rejected examples in the revision.
Scoring and family aggregation. Box grounding uses mAcc over IoU thresholds from 0.50 to 0.95 in steps of 0.05. Point affordance uses point-in-mask accuracy, and the multiple-choice tasks use exact-match accuracy. For each model, Table 1 micro-averages over questions. Each family score is then the arithmetic mean of its included model scores, computed separately for nominal and stressed inputs. The family-level change is the stressed mean minus the nominal mean.
The paired analysis includes 14 models. We provide the individual-model results and family assignments below:
The grouped results identify distinct responses to different stresses: stacked layout produces a 19.2 pp decline, specular confusion a 10.0 pp decline, and lighting a 4.3 pp decline. Complex texture has a smaller estimated change whose interval includes zero, while the planning estimate is close to zero. We will incorporate the complete pair distributions, verification statistics, aggregation rules, and individual-model results into the revised paper.