acceptodds
Under review as a conference paper at ICLR 2027

RoboStressBench: A Diagnostic Benchmark for Visual Stress in Embodied Scene Understanding

Abstract

Vision-language models (VLMs) used in embodied systems must interpret scenes under challenging visual conditions, yet it remains difficult to distinguish failures caused by visual stress from those caused by the underlying task or scene. We introduce RoboStressBench, a diagnostic benchmark for evaluating VLM robustness to visually challenging conditions in robot-centric scenes. An image-formation-inspired taxonomy organizes visual stress into four dimensions—Material, Viewpoint, Lighting, and Geometry—and 16 fine-grained categories, without assuming that these factors are independent in natural images. RoboStressBench contains more than 7K examples collected through naturally occurring stress cases, targeted stress synthesis, and additional real-world collection, and evaluates complementary capabilities including target and placement grounding, spatial reasoning, state understanding, and action-selection-style question answering. To complement broad stress coverage, we further evaluate matched nominal–stressed scene-question pairs, enabling direct measurement of performance degradation under controlled visual modifications. Evaluating 16 VLMs reveals substantial variation across both stress types and task formats, with different visual conditions exposing distinct weaknesses that are obscured by aggregate scores. We additionally study StressDART, a test-time rectification baseline that processes visually stressed inputs before reasoning, and analyze both its performance gains and the risk of altering task-relevant evidence. RoboStressBench targets the visual components relevant to embodied systems, rather than end-to-end closed-loop robot reliability, and provides a structured testbed for studying and improving multimodal robustness under challenging visual observations.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Multimodal Robustness, Robustness Benchmarking, Visual Stress, Embodied Scene Understanding, Visual Grounding, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qT5NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for evaluating VLMs under visually challenging conditions in embodied scenes. The benchmark organizes visual stress into four dimensions (Material, Viewpoint, Lighting, Geometry) and 16 fine-grained categories. It contains 7k examples collected from naturally occurring cases, synthetic stress generation, and additional real-world data. The benchmark covers target grounding, placement grounding, spatial reasoning, state understanding, and planning-style action selection. It also introduces matched nominal-versus-stressed scene pairs and a test-time stress-conditioned rectification baseline called StressDART.

The main contribution is a structured evaluation framework for studying how different visual stress factors affect model performance, together with an extensive evaluation across 16 VLMs.

Critical strengths
  1. The organization of visual stress into Material, Viewpoint, Lighting, and Geometry is intuitive and grounded in image formation principles. This provides a useful structure for robustness analysis and makes benchmark results more interpretable than aggregate evaluation alone.

  2. The paired evaluation is the strongest part for me. Comparing the same scene and question before and after editing reduces important confounders in ordinary benchmark comparisons. This is useful, although the interpretation still depends on the edits preserving the task-relevant content.

Critical weaknesses
  1. My main concern is the evidence behind the matched evaluation. Section 3.2 describes a verified subset, but I could not find a clear account of how many pairs are used in Table 1, which tasks and stress categories they cover, or how the model-family scores are calculated. The paper describes manual checks, which is good, but it is still unclear how often editing changes other task-relevant content or makes a question unanswerable. Since these pairs support the stronger claims about stress-induced degradation, I would like this part to be more explicit.

  2. StressDART gives a useful improvement, but the current comparison does not tell us how much comes from predicting the stress category. There is no comparison with generic image enhancement or editing without stress conditioning. Also, the aggregate gain does not show how often editing turns a correct prediction into a wrong one. The authors acknowledge both points, but they still limit what we can conclude about the method. Also the benchmark mixes naturally stressed images, synthetic edits, and additionally collected real images. I would like to see the main results separated by source, especially whether the stress-wise patterns are similar on natural and synthetic examples. This would help establish how broadly the findings hold within the paper's stated scope of embodied visual understanding.

Questions for authors
  1. How many verified pairs are used in Table 1, and how are they distributed across tasks and stress categories? Please also clarify how the family-level scores are aggregated and provide results for the individual models.

  2. Appendix A.5 gives the number of multiple-choice examples. Have you checked sensitivity to option order or performance without the image? The 99.8% planning accuracy of Qwen3-VL-30B-A3B in Table 2 stands out. Does it remain this high after shuffling the options?

  3. Do the main stress-wise findings hold when naturally occurring and synthetically generated examples are evaluated separately?

  4. How does StressDART compare with the same editor using a generic instruction without stress prediction? How often does rectification change a correct prediction to an incorrect one, and vice versa?

8 comments

We sincerely thank the reviewer for the thoughtful and encouraging feedback, especially for recognizing the interpretability of the taxonomy and the value of the paired evaluation. We address the matched-evaluation protocol, multiple-choice diagnostics, source-specific results, and StressDART controls below.

[W1, Q1] Matched-Pair Composition, Verification, and Model-Level Results

The matched evaluation contains 2,591 verified nominal–stressed scene–question pairs, covering three stress dimensions and eight fine-grained categories. The distribution and corresponding performance changes are:

GroupPairsNominalStressedΔ in pp [95% CI]
Material84058.253.8−4.4 [−6.0, −2.7]
Geometry98854.742.8−11.8 [−13.6, −10.0]
Lighting76348.043.7−4.3 [−5.4, −3.2]
Complex texture25661.560.0−1.5 [−3.4, 0.3]
Transparency36557.754.7−3.0 [−4.8, −1.2]
Specular confusion21955.145.1−10.0 [−13.5, −6.5]
Non-rigid deformation44955.552.5−3.0 [−4.8, −1.2]
Stacked layout53954.034.8−19.2 [−21.5, −16.9]
Global overexposure34246.243.8−2.4 [−3.7, −1.1]
Local overexposure14451.246.1−5.1 [−7.7, −2.6]
Global underexposure27748.542.3−6.2 [−8.2, −4.2]

The task breakdown is:

TaskPairsNominalStressedΔ in pp [95% CI]
Grounding, box1,82856.347.9−8.4 [−9.8, −7.0]
Affordance, point56642.136.6−5.5 [−6.9, −4.1]
State understanding2967.762.3−5.4 [−10.3, −0.7]
Planning16864.564.5+0.1 [−1.2, +1.4]

These results average over 14 models. We compute two-sided 95% confidence intervals using 10,000 paired bootstrap resamples clustered by source image, keeping all questions and edited variants sharing the same nominal image together. Changes are computed from unrounded scores.

Pair verification. Each retained pair preserves the question and, for multiple-choice items, the answer options. The referred target remains the same. Spatial coordinates are updated where necessary to identify that target in the edited image; lighting edits preserve the original target annotations. Every retained pair was manually inspected for question answerability.

We quantify the verification outcomes as follows:

Verification outcomeNumber
Candidate pairs inspected3000
Pairs with task-invalid content changes193
Pairs with questions made unanswerable by editing164
Pairs failing both checks39
Unique pairs excluded13
Final verified pairs2,591

The excluded total counts unique pairs because the two failure criteria can overlap. We will include these statistics and representative rejected examples in the revision.

Scoring and family aggregation. Box grounding uses mAcc over IoU thresholds from 0.50 to 0.95 in steps of 0.05. Point affordance uses point-in-mask accuracy, and the multiple-choice tasks use exact-match accuracy. For each model, Table 1 micro-averages over questions. Each family score is then the arithmetic mean of its included model scores, computed separately for nominal and stressed inputs. The family-level change is the stressed mean minus the nominal mean.

The paired analysis includes 14 models. We provide the individual-model results and family assignments below:

ModelFamilyPairsNominalStressedΔ in pp
Qwen3-VL-4B-InstructQwen3-VL259157.2935.79−21.50
Qwen3-VL-8B-InstructQwen3-VL259141.5532.30−9.25
Qwen3-VL-30B-A3B-InstructQwen3-VL259154.2538.37−15.88
Qwen3.5-4BQwen3.5259152.6331.24−21.39
Qwen3.5-9BQwen3.5259138.1631.10−7.06
Qwen3.5-27BQwen3.5259161.4544.30−17.15
Qwen3.5-35B-A3BQwen3.5259161.8140.38−21.43
Qwen3.6-27BQwen3.6259166.7143.13−23.58
Qwen3.6-35B-A3BQwen3.6259161.8137.13−24.67
InternVL3_5-4B-HFInternVL3.5259110.069.88−0.18
InternVL3_5-8B-HFInternVL3.5259110.7311.12+0.39
InternVL3_5-14B-HFInternVL3.525919.188.79−0.39
Molmo2-4BMolmo2259112.1111.30−0.81
Molmo2-8BMolmo2259112.3511.79−0.56

The grouped results identify distinct responses to different stresses: stacked layout produces a 19.2 pp decline, specular confusion a 10.0 pp decline, and lighting a 4.3 pp decline. Complex texture has a smaller estimated change whose interval includes zero, while the planning estimate is close to zero. We will incorporate the complete pair distributions, verification statistics, aggregation rules, and individual-model results into the revised paper.

[Q2] Option-Order Sensitivity, Text-Only Performance, and Planning Accuracy

We evaluate Qwen3-VL-30B-A3B on all 821 planning questions using text-only inputs and five random option permutations. Text-only evaluation removes the image. Option shuffling preserves the option text and updates the answer labels to match the new order.

Planning subsetNOriginalShuffled, mean ± SDText-only
All questions82199.76%99.83% ± 0.11 pp77.10%
Questions without order-referencing options67099.70%99.79% ± 0.13 pp79.25%

The 670-question subset provides the primary option-order comparison because its options retain their meaning after permutation. Accuracy remains essentially unchanged across the tested orders, directly addressing whether the near-perfect result persists after shuffling.

Images provide a 22.66 pp gain over text-only input across all planning questions, with a paired 95% bootstrap interval of [19.85, 25.46]. The text-only results also reveal differences between question types:

Question typeText-only accuracy
Next interaction96.69%
Obstacle-related95.58%
Arrow direction56.04%

These results explain the high aggregate score through the task composition. Next-interaction and obstacle-related questions are often solvable from text, while arrow-direction questions have substantially lower text-only accuracy. The action-selection task is close to saturation for this model, and images still contribute a measurable aggregate benefit.

We also checked option mapping, answer-label synchronization, and output parsing. Across all 2,823 multiple-choice examples, modifying the ground-truth field leaves the model prompt unchanged, verifying that this field does not directly enter the inspected prompt-construction path.

The text-only and permutation experiments specifically target the planning result highlighted by the reviewer. We will report their scope, question-type breakdown, and evaluation protocol in the revised paper.

[W2, Q3] Source-Specific Results and the Consistency of Stress-Wise Findings

We separate the benchmark by its recorded construction sources and compare performance within the same task, answer format, and stress dimension. The source distribution is:

SourceTotalTarget groundingPlacementSpatial reasoningState understandingPlanning
Selection from existing datasets2,9272831931,194604653
Controlled synthesis2,5961,828571029168
Additional real-world sources1,6601,30018517500
Total7,1833,4119491,369633821

Controlled synthesis includes 1,828 scene-synthesis examples and 768 lighting edits. Additional real-world sources include web collection and our own collection. “Selection from existing datasets” identifies a construction path; the originating datasets determine the image provenance within that group.

The sources have different task compositions, so we compare results within task and answer-format groups. Using archived predictions from Qwen3-VL-30B-A3B, we obtain:

Task / SourceMaterialViewpointLightingGeometry
Target grounding, bbox / Controlled synthesis48.33% (840)——15.08% (988)
Target grounding, bbox / Additional real-world sources26.96% (345)24.12% (199)28.25% (177)31.59% (345)
Spatial reasoning, MCQ / Existing-dataset selection76.69% (532)72.80% (375)73.50% (434)68.87% (485)
Spatial reasoning, MCQ / Additional real-world sources58.95% (95)57.14% (49)72.73% (55)57.14% (105)

Each cell reports accuracy and sample count. For this source analysis, bbox success uses IoU ≥ 0.95, and MCQ success uses exact answer correctness. Stress groups may overlap; “—” indicates an empty group.

The comparison reveals both shared difficulties and source-dependent stress profiles.

Target grounding. Precise localization remains difficult across synthetic and additionally collected examples: all reported bbox groups score below 50%. In the synthetic subset, Geometry scores 15.08%, below Material at 48.33%. In the additionally collected subset, Geometry scores 31.59%, above Material at 26.96%. Thus, localization difficulty appears across both sources, while the relative stress ranking changes.

Spatial reasoning. Geometry is among the lower-performing groups in both sources: 68.87% in existing-dataset selection and 57.14% in additional real-world sources. The highest-performing dimension changes from Material in the former to Lighting in the latter.

These results support the benchmark’s emphasis on task- and stress-specific diagnosis. They also show why source composition should be reported: a single aggregate ranking would conceal meaningful differences between source groups. We will include the source distributions and these within-task comparisons in the revised paper.

Reviewer_zxSmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RoboStressBench, a benchmark designed to diagnose whether VLM failures in embodied scenes are caused by visual stress or by the underlying task. Motivated by the physical image-formation process, the authors organize visual stress into four dimensions: Material, Viewpoint, Lighting, and Geometry. Evaluating 16 VLMs on the benchmark reveals task-dependent stress profiles that are obscured by aggregate scores. Beyond benchmarking, the paper also proposes StressDART, a test-time baseline that uses predicted stress categories to guide image rectification before reasoning.

Critical strengths

1.This paper focuses on task-relevant physical visual stress in embodied scenes, which makes it highly relevant to embodied applications.

2.The image-formation-inspired taxonomy provides an interpretable framework for annotating and organizing visual stress.

Critical weaknesses
  1. The matched nominal–stressed comparison in Table 1 is a strength, as it examines the effect of visual stress while holding the scene and question fixed. However, the paper does not appear to report the size of this subset or its distribution over tasks and stress categories. InternVL3.5 and Molmo2 score only 10.0% and 12.2% on its nominal images, compared with 39.2–67.9% on the stressed multiple-choice tasks in Table 2. This suggests that the subset’s task composition may differ considerably from the full benchmark, which could limit the scope of the conclusions drawn from Table 1.

  2. The reliability of generative rectification in StressDART is not quantified. All stress categories are handled by the same generative editor, but the paper does not describe how categories such as occlusion or truncation are rectified. In these cases, as well as when overexposure saturates pixels, the task-relevant evidence may no longer be present in the image, so the rectified output may contain content inferred by the editor. The authors acknowledge this risk but do not analyze it. Moreover, rectified-only and original + rectified inputs perform almost identically in Table 3, so it remains unclear whether keeping the original image, which the paper presents as a safeguard, has a measurable effect.

Questions for authors
  1. Could the authors report the number of matched nominal–stressed scene–question pairs, their breakdown by task and stress category, and the Δ for each task type?

  2. Could the authors report how StressDART performs on categories where task evidence may be missing, such as occlusion, truncation, and overexposure? For example, accuracy before and after rectification, and whether the rectified images introduce content absent from the original. In addition, could the authors compare rectified-only and original + rectified inputs on examples where rectification goes wrong, to show whether keeping the original image actually helps correct such errors?

6 comments

We sincerely thank the reviewer for the thoughtful feedback, especially for recognizing the relevance of task-related visual stress in embodied scenes and the interpretability of the taxonomy. We address the matched-subset composition and the reliability of rectification below with additional statistics and analyses.

[W1, Q1] Matched-Subset Composition, Task-Level Changes, and Low Nominal Scores

The matched evaluation contains 2,591 verified nominal–stressed scene–question pairs, covering three stress dimensions and eight fine-grained categories. The distribution is:

Dimension / CategoryPairs
Material840
Complex texture256
Transparency365
Specular confusion219
Geometry988
Non-rigid deformation449
Stacked layout539
Lighting763
Global overexposure342
Local overexposure144
Global underexposure277
Total unique pairs2,591

The task-level results are:

TaskPairsNominalStressedΔ in pp [95% CI]
Grounding, box1,82856.347.9−8.4 [−9.8, −7.0]
Affordance, point56642.136.6−5.5 [−6.9, −4.1]
State understanding2967.762.3−5.4 [−10.3, −0.7]
Planning16864.564.5+0.1 [−1.2, +1.4]

These results average over 14 models. We compute two-sided 95% confidence intervals using 10,000 paired bootstrap resamples clustered by source image, keeping questions and edited variants sharing the same nominal image together. Changes are calculated from unrounded scores.

Why the nominal scores differ from the multiple-choice results in Table 2. Localization accounts for 2,394 of the 2,591 matched pairs, or 92.4%: 1,828 box-grounding questions and 566 point-affordance questions. State understanding and planning together account for the remaining 197 pairs, or 7.6%.

For each model, Table 1 micro-averages over questions, so each task contributes in proportion to its sample count. Family scores then average the included model scores. Consequently, localization performance strongly influences Table 1. InternVL3.5 and Molmo2 perform relatively poorly on these localization tasks, contributing to their nominal scores of 10.0% and 12.2%.

The 39.2–67.9% figures highlighted from Table 2 concern multiple-choice tasks, which use a different answer format and scoring criterion. Box grounding uses mAcc over IoU thresholds from 0.50 to 0.95, point affordance uses point-in-mask accuracy, and multiple-choice tasks use exact-match accuracy. The task composition and scoring protocol therefore explain why the nominal localization-dominated scores can be lower than the stressed multiple-choice scores.

Pair construction and interpretation. Each pair preserves the question and, for multiple-choice items, the answer options. The referred target remains the same. Spatial coordinates are updated where necessary to identify that target in the edited image; lighting edits preserve the original target annotations. Every retained pair was manually inspected for question answerability.

The paired results quantify changes within the covered tasks and stress categories. Grounding and affordance show declines of 8.4 pp and 5.5 pp, respectively, while the planning estimate is +0.1 pp with an interval spanning zero. The full benchmark provides broader task and stress coverage, and the matched subset supplies controlled comparisons within its verified coverage. We will include the complete distributions, task-level changes, and aggregation protocol in the revised paper.

[W2, Q2] Rectification Under Missing Evidence and the Contribution of the Original Image

We evaluate Qwen3-VL-4B with Qwen-Image-Edit and separately examine category-level performance, image-content changes, and the effect of retaining the original image.

Stress-conditioned editing procedure. StressDART uses the predicted stress category to select the editing instruction. Although the editor is shared across categories, the requested operation changes with the detected condition. For the categories highlighted by the reviewer, the instructions are:

  • Occlusion: Remove the occluding objects and reconstruct the previously hidden regions of the target object. Preserve the target's original geometry, appearance, position, and scale, while keeping the surrounding scene unchanged.
  • Truncation: Restore the missing portions of the target object by extending the image boundaries or adjusting the framing to ensure the entire object is visible. Preserve the object's original geometry, appearance, and spatial relationships without altering unrelated scene content.
  • Overexposure: Correct excessive exposure by reducing brightness and recovering details in overexposed regions. Restore natural illumination and color consistency while preserving object appearances, scene geometry, and spatial relationships.

We will include these instructions and representative original–rectified image pairs in the revision.

Performance on occlusion, truncation, and overexposure. The results are:

CategoryNOriginalRectified OnlyOriginal + RectifiedΔ over Rectified Only
Occlusion1,20550.29%56.10%58.09%+1.99 pp
Truncation66547.37%53.53%55.64%+2.11 pp
Global overexposure36448.08%54.12%57.42%+3.30 pp
Local overexposure57553.57%61.22%63.48%+2.26 pp
Overall7,18357.00%63.90%64.00%+0.10 pp

Relative to the unedited input, StressDART with original-plus-rectified input improves accuracy by 7.80 pp on occlusion, 8.27 pp on truncation, 9.34 pp on global overexposure, and 9.91 pp on local overexposure.

Across the full benchmark, it changes 749 predictions from wrong to correct and 246 from correct to wrong, yielding 503 additional correct predictions and a 7.00 pp overall gain. This reports both the improvement and the frequency of harmful prediction changes.

Whether rectified images introduce unsupported or incorrect content. We manually inspected 500 rectified images and evaluated two aspects of task-relevant content:

Audit criterionCount / 500Proportion
Task-relevant additions unsupported by the original image316.2%
Incorrect changes to task-relevant content479.4%

The first criterion records content added without sufficient support in the original image. The second records changes judged incorrect for the task. We report these criteria separately. The audit directly quantifies the image-content issues raised by the reviewer and will be accompanied by the annotation criteria and representative cases.

Whether retaining the original image corrects rectified-only errors. We track prediction transitions between rectified-only and original-plus-rectified inputs:

CategoryRescue: Wrong → CorrectHarm: Correct → WrongNet additional correct predictions
Occlusion7551+24
Truncation4228+14
Global overexposure3018+12
Local overexposure4532+13
Overall225218+7

“Rescue” counts examples answered incorrectly using the rectified image alone and correctly after adding the original image. “Harm” counts the reverse transition. Categories can overlap; the overall row is computed independently over unique examples.

This analysis explains the similar aggregate scores in Table 3: adding the original image rescues 225 predictions and changes 218 correct predictions to incorrect ones, leaving a net increase of seven. The four examined high-risk categories show larger net gains of 1.99–3.30 pp, identifying where retaining the original image is most helpful among these evaluated groups. We further connect this prediction analysis to the manually audited editing errors. Among 500 examples with a verified problematic edit and an incorrect rectified-only prediction, adding the original image corrects 36, corresponding to 7.2%. We will include examples showing how the original image supplies evidence that corrects the rectified-only answer. Together, these results quantify performance under evidence loss, the frequency of unsupported or incorrect image changes, and the category-dependent contribution of retaining the original image. We will incorporate the editing instructions, audit results, and prediction-transition analysis into the revised paper.

Thank you for the detailed response. I have only one small question.

My understanding is that Table 1 and the new task-level table are computed from the same matched pairs and the same models, just aggregated differently. If so, their overall averages should be the same. Taking the stressed scores as an example, the highest family-level stressed score in Table 1 is 40.1. So the overall stressed average from Table 1 should be at most 40.1, regardless of how many models each family contains. However, a rough calculation from the task-level table gives about 46.7: (1828×47.9 + 566×36.6 + 29×62.3 + 168×64.5) / 2591.

I may have misunderstood this. If the two tables use different evaluation settings, such as a different box-grounding metric, simply stating this in the revised paper would be enough. In any case, it does not affect the paper’s main contributions. If you can explain it, I will raise my score to weak accept.

Reviewer_2j9yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboStressBench, a benchmark for evaluating VLMs under visual stress in embodied-scene understanding. It contains 7,183 examples organized into four stress dimensions and 16 categories, covering grounding, spatial reasoning, state understanding, and planning-style action selection. The authors evaluate 16 VLMs, conduct matched nominal–stressed comparisons, and introduce StressDART to explore test-time image rectification.

Critical strengths
  1. The benchmark addresses a relevant problem and covers complementary capabilities, providing a useful perspective on how visual conditions affect embodied-scene understanding.
  2. The combination of naturally stressed examples and synthetic variants is well motivated. Natural examples provide practical coverage, while matched pairs support comparisons on the same scene–question instances.
  3. The interpretable stress taxonomy supports detailed performance analysis.
Critical weaknesses

While the benchmark is well motivated, several aspects of the evaluation and interpretation of the results would benefit from further clarification:

  1. Could matched nominal–stressed changes be reported by stress category to show how the physically motivated taxonomy supports diagnosis beyond organizing performance by visual condition?
  2. Despite the overall improvement shown in Table 3, which tasks or stress conditions benefit most from rectification, and under what circumstances does it introduce errors?
  3. What characteristics of the planning questions might explain the near-perfect accuracy of Qwen3VL-30B-A3B, and what does this result suggest about their difficulty under visual stress?
Questions for authors
  1. Can you report matched nominal–stressed deltas per dimension, per fine-grained category, and per task, with Ns and CIs? If the matched subset is too small, what is the largest defensible breakdown?
  2. Can you quantify M/V/L/G co-occurrence and, ideally, run a small crossed stress grid (N scenes × {none, M, V, L, G}) to estimate main effects and interactions?
  3. For StressDART: please add (a) a random/fixed-edit control, (b) detector accuracy by dimension, (c) flip-rate analysis, (d) a second, stronger reasoner, and (e) matched-subset results. Does stress-conditioned editing beat generic editing? 4.Are motion blur, defocus, mirrors, and weather excluded by design? How would the taxonomy extend, and can a small subset be added?
6 comments

We sincerely thank the reviewer for the thoughtful and encouraging feedback, especially for recognizing the relevance of the benchmark, the complementary roles of natural and synthetic examples, and the interpretability of the stress taxonomy. We address each concern below with additional analyses and results.

[W1, Q1] Matched Evaluation: Taxonomy-Specific Degradation, Sample Sizes, and Confidence Intervals

We report matched performance changes at the dimension, category, and task levels. The verified subset contains 2,591 nominal–stressed scene–question pairs, covering three dimensions and eight fine-grained categories. Each pair preserves the question and, for multiple-choice items, the answer options.

The following results average over 14 models. We compute two-sided 95% confidence intervals using 10,000 paired bootstrap resamples clustered by source image, keeping questions and edited variants from the same nominal image together. Changes are computed from unrounded scores.

GroupPairsNominalStressedΔ in pp [95% CI]
Material84058.253.8−4.4 [−6.0, −2.7]
Geometry98854.742.8−11.8 [−13.6, −10.0]
Lighting76348.043.7−4.3 [−5.4, −3.2]
Complex texture25661.560.0−1.5 [−3.4, 0.3]
Transparency36557.754.7−3.0 [−4.8, −1.2]
Specular confusion21955.145.1−10.0 [−13.5, −6.5]
Non-rigid deformation44955.552.5−3.0 [−4.8, −1.2]
Stacked layout53954.034.8−19.2 [−21.5, −16.9]
Global overexposure34246.243.8−2.4 [−3.7, −1.1]
Local overexposure14451.246.1−5.1 [−7.7, −2.6]
Global underexposure27748.542.3−6.2 [−8.2, −4.2]

The task breakdown is:

TaskPairsNominalStressedΔ in pp [95% CI]
Grounding, box1,82856.347.9−8.4 [−9.8, −7.0]
Affordance, point56642.136.6−5.5 [−6.9, −4.1]
State understanding2967.762.3−5.4 [−10.3, −0.7]
Planning16864.564.5+0.1 [−1.2, +1.4]

These comparisons establish the taxonomy’s diagnostic value through category-specific changes on matched scene–question pairs. Stacked layout produces a substantial localization decline of 19.2 pp, while specular confusion produces a 10.0 pp decline. Complex texture has a smaller estimated change whose interval includes zero. The lighting comparison further isolates performance changes under edits that preserve both the question and target annotations.

For the requested breakdown, we report dimension, category, and task marginals. The paired evidence covers Material, Geometry, and Lighting and the four tasks listed above. State understanding contains 29 pairs, so its results are reported at the task level. Viewpoint and spatial reasoning are evaluated in the full benchmark. We will include these distributions, scoring details, and confidence intervals in the revised paper.

[Q2] Stress Co-occurrence and Exploratory Analysis of Main Effects and Interactions

Based on the existing multi-label annotations, 1,669 of 7,183 examples (23.24%) contain at least two stress dimensions. Among multiple-choice examples, the corresponding proportion is 942 of 2,823 (33.37%).

MaterialViewpointLightingGeometry
Material2,785248473594
Viewpoint2481,292185377
Lighting4731851,753408
Geometry5943774083,327

Diagonal entries count examples carrying each dimension; off-diagonal entries count examples carrying both dimensions, including examples with additional labels. Material–Geometry is the most frequent pair, with 594 examples, followed by Material–Lighting with 473 and Lighting–Geometry with 408. The taxonomy therefore represents multi-factor scenes through overlapping labels.

We also conducted a pilot covering all 16 on/off combinations of M/V/L/G on the same 100 source image–question units: 50 spatial-reasoning, 30 planning, and 20 state-understanding questions. We evaluated Qwen3-VL-30B-A3B and InternVL3.5-4B. Selection was stratified by Qwen’s original-image performance, with 70 correct and 30 incorrect predictions, before evaluating the edited conditions.

Averaging each on/off difference over the other three factors gives the following descriptive main effects:

OperationQwen3-VL-30B-A3BInternVL3.5-4B
M−6.25 pp−1.25 pp
V−8.75 pp−6.25 pp
L-1.25 pp−1.25 pp
G−11.25 pp-6.25 pp

We will include the complete 16-condition results alongside the co-occurrence analysis.

[W2, Q3(a–e)] StressDART: Editing Controls, Task-Specific Gains, Failure Analysis, and Additional Validation

We extend the StressDART evaluation with the requested editing controls, detector analysis, prediction-flip statistics, stronger reasoner, and matched-subset results.

[Q3(a)] Generic, Fixed-Stress, and Random-Stress Editing Controls

We compare the following settings on 7,183 examples, using Qwen3-VL-4B as the reasoner and Qwen-Image-Edit as the image editor. The controls differ in how the editing instruction is selected:

  • Generic Editing: We use the same prompt, “please refine the image”, for every image, without specifying a stress type.
  • Fixed Stress: In each run, we fix one stress type and instruct the editor to rectify that type for every image, regardless of the stress present. We repeat this evaluation with each stress type in turn and report the average performance across these fixed-stress runs.
  • Random Stress: We randomly select a stress type to rectify for each image, without using the stress detector’s prediction.
  • StressDART: We predict the stress category for each image and use that prediction to guide the editing instruction.
SettingAccuracy
Original57.00%
Generic Editing57.30%
Fixed Stress, averaged across stress types58.21%
Random Stress58.40%
StressDART, Rectified Only63.90%
StressDART, Original + Rectified64.00%

StressDART achieves 63.90% with rectified-only input and 64.00% with original-plus-rectified input, exceeding generic editing by 6.60 pp and 6.70 pp, respectively. The rectified-only variant also exceeds the averaged fixed-stress control by 5.69 pp and the random-stress control by 5.50 pp. These results support the value of selecting an image-specific stress condition to guide rectification. We will include the complete editing instructions and input configurations in the revised paper.

[W2, Q3(c)] Task- and Stress-Specific Gains, Prediction Flips, and Editing Failures

The improvements extend across all five task groups:

TaskNOriginalStressDARTΔ in ppWrong → CorrectCorrect → Wrong
Placement Grounding94955.01%61.85%+6.8510843
Target Grounding3,41156.00%64.06%+8.06388113
Spatial Reasoning1,36953.54%61.94%+8.4016449
State Understanding63360.98%64.93%+3.954419
Planning82166.14%68.94%+2.804522
Overall7,18357.00%64.00%+7.00749246

The largest task-level gains occur in spatial reasoning (+8.40 pp) and target grounding (+8.06 pp). Across stress dimensions, the gains are +6.10 pp for Material, +8.13 for Viewpoint, +9.18 for Lighting, and +5.50 for Geometry. Fine-grained gains range from +1.55 pp for non-rigid deformation to +10.27 pp for dark absorptive surfaces.

Across the full evaluation set, 749 predictions change from wrong to correct, while 246 change from correct to wrong, corresponding to 10.43% and 3.42% of all examples. Their difference gives 503 additional correct predictions, accounting for the +7.00 pp overall improvement.

To characterize editing failures, we manually inspected 500 rectified images. We identified 31 with task-relevant additions unsupported by the original image and 47 with incorrect changes to task-relevant content. We will include these findings and representative examples with the task- and stress-level results.

[Q3(b)] Stress Detector Performance and Oracle-Label Comparison

On a stress-detection evaluation set (N = 1,000), the detector achieves the following dimension-wise F1 scores:

DimensionF1
Material80.48%
Viewpoint79.35%
Lighting85.51%
Geometry78.34%
Macro average80.92%

Replacing predicted stress labels with oracle labels improves downstream performance by 1.80 pp, from 72.50% to 74.30%. This comparison quantifies the effect of label quality on downstream rectification performance.

[Q3(d)] Evaluation with a Stronger Reasoner

With Qwen3.8-27B, performance increases from 70.03% to 74.22%, a gain of 4.19 pp. The additional reasoner therefore also benefits from StressDART.

[Q3(e)] StressDART Performance on the Matched Subset

ConditionScore
Nominal71.42%
Stressed56.00%
Stressed + StressDART63.96%

StressDART recovers 7.96 pp of the 15.42 pp nominal-to-stressed performance gap, yielding a 51.62% recovery ratio. This directly connects the rectification evaluation to the degradation measured on matched scene–question pairs.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading, constructive comments, and the time devoted to evaluating our work. Your feedback has helped us strengthen the experimental evidence and clarify the scope and limitations of the paper.

In response to the reviews, we have completed the following additional work:

  • Matched evaluation: Expanded the reporting of paired results, task and stress distributions, uncertainty estimates, verification procedures, and model-family aggregation, and clarified the screening counts.
  • Stress taxonomy: Quantified stress co-occurrence, conducted an exploratory crossed-stress pilot, and clarified the taxonomy’s coverage and possible extensions.
  • Planning diagnostics: Conducted text-only and option-shuffling evaluations, examined differences across question types, and checked prompt construction and answer mapping.
  • Source-specific analysis: Reported construction-source distributions and compared stress-wise performance within task and answer-format groups.
  • StressDART validation: Compared stress-conditioned editing with generic, fixed-stress, and random-stress controls, and added detector evaluation, oracle-label comparison, an additional reasoner, and matched-subset rectification results.
  • Editing reliability: Analyzed beneficial and harmful prediction changes, manually audited rectified images, and examined when retaining the original image helps correct rectified-only errors.

We have also updated the revised paper, clarified the intended meaning of “diagnostic,” and expanded the discussion of language priors, matched-subset coverage, source-dependent findings, and the risks of generative editing.

We hope this additional work addresses the concerns raised by the reviewers. We welcome any further questions and would be happy to provide clarification. Thank you again for your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.