- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
SABRE constructs VLM stress tests from task specifications using image generation/editing, model-based difficulty filtering, human verification, and localized repair. The benchmark contains 600 images and 1,000 questions across context, texture, attribute, and language-elicitation subsets. Six SOTA VLMs (Claude, Kimi, Qwen, Gemini, GPT, Grok) are evaluated. Additionally it contains more difficult stress-test instantiations with 20 samples for counting and spatial where the models answer at most 1/20 counting samples and 0/20 spatial samples correctly. Comparison against baselines such as HallusionBench and VLMBias is done.
Critical strengths
- Produces challenging visual tests with human verification. Also includes comparisons with existing benchmarks, alternate filtering models, and a repair-preference study. The counting and spatial extensions also show that the workflow can support different tasks.
Critical weaknesses
- Contribution of the data pipeline is not sufficiently isolated where it doesnt establish how much filtering and repair improve benchmark quality.
- Additional counting and spatial tests contain only 20 samples each and produce almost uniformly zero accuracy. Without independent human performance on these samples (and a more detailed analysis of model errors), it is hard to tell what these results mean. Generally, the paper provides not enough quantitative analysis of where and why models fail.
Questions for authors
- How accurate do human evaluators answer the 20-sample counting and spatial tests?
- What is the total construction cost and measured human time per accepted sample?
8 comments
(2) How well do humans perform on Counting and Spatial, and what do the near-zero model scores tell us?
Independent human evaluators achieved 90% on Counting and 100% on Spatial on the two 20-example pilots. The two evaluators had no role in constructing the data and saw only the images and questions, following the protocol used for SABRE-Prior in Appendix E.1. We will add these scores and the evaluation procedure to the revision.
The two pilots show that SABRE's generation, screening, verification, and repair workflow can support tasks beyond reliance on world priors. Appendix F also reports larger procedural versions that vary scene complexity. The programs render scenes and compute reference answers from the scene specifications without paid image-generation API calls.
| Procedural task | Dataset size | Complexity levels | Qwen 3.5 27B accuracy at those levels |
|---|---|---|---|
| Counting (Table 12) | 200 images, 200 questions | 20, 50, 100, 150 objects | 36%, 0%, 0%, 0% |
| Spatial (Table 13) | 200 images, 400 questions | 2×2×2, 4×4×4, 7×7×7, 10×10×10 grids | 65%, 24%, 3%, 1% |
The procedural scenes and questions were fixed before model evaluation. These results show where performance drops as scenes become denser or grids become larger. We will point to these results more clearly in the main text.
(3) What are the construction costs and human time per accepted example?
For Context, Appendix D.4 and Table 7 report approximately $0.44 in initial image-output costs per accepted case. The appendix also reports typical times for individual review and repair operations.
| Context construction metric | Reported value |
|---|---|
| Initial image outputs | 440, including a base image and an initial edited image for each of 220 candidates |
| Estimated cost of initial image outputs | $44.44 |
| Estimated initial image-output cost per accepted case | $0.44, across 100 accepted cases |
| Inspect a candidate and record an initial decision | Typically under 5 seconds |
| Mark a region and enter a repair instruction | Typically under 15 seconds |
| Wait for the repair API to return a result | About 5 seconds under normal network conditions |
Image generation and editing are the main paid costs in our workflow. At fixed model and output settings, the cost of initial image generation scales with the number of outputs. The reported subtotal includes initial outputs for candidates that were later discarded. It covers initial image outputs only.
The review and repair durations are experience-based estimates for individual operations. Total active human time per accepted case was not systematically measured. We will make these distinctions clearer in the revision.
(1) How do filtering and repair contribute to benchmark quality?
Our appendix reports quantitative evidence for both stages. In Context, difficulty filtering reduced the number of candidates requiring human review by 33.2%. About 21 of the 100 retained edited images underwent localized repair. In a separate blind study, 374 of 400 selections (93.5%) favored SABRE's repair results.
Difficulty filtering. A stress-test example needs to satisfy two criteria. It must support a clear question and a correct answer, and it must expose a model failure. Even a fully valid image can be easy for a VLM. SABRE screens for model failures before human review, so reviewers can focus on candidates that are more likely to meet the difficulty criterion. In Context, this reduced the pool requiring manual review from 220 to 147 cases (Appendix D.4). Human reviewers then checked the remaining images, questions, and reference answers. The 33.2% figure measures the reduction in the review pool. The independent human evaluation in Appendix E.1 and Table 9 provides evidence that the final SABRE-Prior examples are solvable by humans.
Localized repair. Image generators can miss precise visual requirements. For example, a prompt asking for a zebra with ten legs may produce the wrong number of visible legs. If the defect is local, reviewers can mark and repair the affected region, preserve the rest of the image, and verify the result before accepting it. About 21 retained Context images followed this repair-and-recheck process.
Repair quality. Figure 11 shows the instructions and interface used in the blind study. Participants saw an image with an unwanted object marked and selected the result that best removed the object while preserving unrelated content and looking natural. The four methods were shown anonymously and in randomized order. The hard-paste and soft-mask methods used the same generated patch, allowing participants to compare the compositing methods directly. Twenty participants each evaluated 20 cases, and 374 of 400 selections (93.5%) favored SABRE's result (Appendix D.3, Table 6).
Figures 9 and 10 show how the review platform supports this process. Reviewers can inspect the image, question, reference answer, and model prediction, correct annotations, and review a localized repair in the same interface.