- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Hob-VL evaluates visually grounded Boolean reasoning through two tasks, Boolean condition evaluation and object identification. It contains 6,000 Yes/No questions and 1,000 identification questions over 1,000 synthetic images and 46 photographs. Each question has symbolic and structured natural-language versions. The Boolean questions also include verified De Morgan pairs.
- The paper evaluates eight VLMs in nine configurations and reports accuracy, equivalent-pair correctness, disagreement, and output validity. Most configurations disable or minimize thinking. One GLM configuration enables thinking and achieves substantially higher scores in several conditions.
Critical strengths
- The benchmark has a clear and useful motivation. Correct answers to individual visual questions do not guarantee correct composition or consistent answers across equivalent expressions. The paired design makes these distinctions explicit.
- Executable validation is a valuable feature. Exhaustive truth-table checks provide strong guarantees about the Boolean expressions and their equivalence. The reported human review of visual annotations is also welcome, since logical validation alone cannot establish that the visual facts are correct.
- The paper provides useful logical annotations and some detailed analysis. Construction families, certificate sizes, and input influence support analyses beyond aggregate accuracy. Reporting label bias and invalid outputs alongside consistency also helps avoid misleading interpretations.
Critical weaknesses
-
The scale and diversity of the visual collection are not sufficiently justified for a benchmark contribution.
- The synthetic component contains only 1,000 scenes, each with ten objects drawn from three shapes and six colors. The real-image component has 46 photographs from 45 scene families reused across 3,000 questions, and the 1,000 identification questions cover only 241 distinct photograph and target combinations (Section 4.4).
- For scale context, CLEVR's validation split alone contains 15,000 images and 149,991 questions. The natural-image portion of MM-CondChain contains 398 images and 796 instances, and ConVBench reports 686 images and 1,372 questions. These sizes are not minimum requirements, but they show the difference between question count and coverage of visual evidence.
- Table 10 offers partial evidence of stability, since results on synthetic scenes and photographs are broadly similar across configurations. It does not yet show stability across fresh synthetic samples or a broader photographic collection.
-
Near-chance scores mainly reflect the evaluation protocol rather than model capability.
- Every configuration except GPT-5.6 Luna receives the instruction "Respond with exactly one word" (Appendix A.1), and any answer with explanatory text is scored as invalid. When thinking is disabled, models must evaluate expressions of 54 to 370 AST nodes in one word without intermediate computation. The paper itself cites Merrill and Sabharwal (2024) and Li et al. (2024), who show that intermediate generation extends what transformers can compute on serial problems. Results near 50% are therefore largely expected under this protocol.
- The near-chance accuracy also reflects collapse onto a single label rather than attempted reasoning. Table 11 shows Yes rates of 98.60% for GLM-off in natural language and 96.25% for Llama in symbolic form, but only 1.35% and 1.78% for Claude Haiku and DeepSeek in symbolic form. Both-correct accuracy is below the 50% constant-label baseline for all nine configurations in both formats.
- Most evaluated models are small or fast tiers, such as Claude Haiku 4.5, Gemini 3.5 Flash-Lite, DeepSeek Flash, GLM-4.6V-FlashX, Llama 4 Scout, and Qwen3-VL-8B. The GLM comparison shows how much the configuration matters. Symbolic Boolean accuracy rises from 48.58% to 63.50% and symbolic identification from 31.30% to 73.70% with thinking and a larger budget.
- The evaluation should allow visible chain-of-thought and extract only the final answer from a marked field. Representative frontier models with reasoning enabled and adequate budgets should also be included. Otherwise the benchmark mainly exposes restricted inference settings rather than its remaining difficulty.
-
Capability categories should focus where the design leaves skills mixed.
- The synthetic Boolean track queries only color and shape, and the identification track uses only spatial predicates. Table 10 already reports results by image source. A per-skill breakdown would therefore add most value for the photographic Boolean statements, which mix attributes and spatial relations.
- For that track, each category should have definitions, image and question counts, and per-model scores. The logical-family breakdown could then be crossed with these categories where sample size permits.
-
The current evaluation does not sufficiently separate visual grounding from Boolean execution.
- A wrong final answer may come from an incorrect visual fact, a parsing error, or an incorrect logical calculation. Logical-property annotations characterize the questions but do not identify which stage caused a failure. The paper itself acknowledges this limitation.
- Two controls would substantially improve the diagnostic value. One should evaluate the atomic visual statements separately. The other should provide the correct truth values and ask the model to evaluate the same expression.
- Output-budget failures should also remain distinct from reasoning errors. Thinking-enabled GLM reaches the token limit on 536 natural-language Boolean questions, which makes an adequate-budget evaluation particularly important.
Questions for authors
-
Visual coverage and scale
- What evidence supports the adequacy of 1,000 synthetic scenes and 46 photographs?
- Could the authors show that the findings are stable across fresh synthetic samples and scene-level resampling?
- Please distinguish unique images, scene families, base expressions, equivalent variants, and presentation formats in any comparison with existing benchmarks.
-
Evaluation protocol and strong reasoning models
- Could the authors rerun a representative subset with visible chain-of-thought allowed and the final answer extracted from a marked field?
- Could they evaluate at least two representative frontier multimodal models with reasoning enabled and adequate budgets, and report token use and truncation rates?
- Do substantial errors and equivalent-expression conflicts remain under these settings? A prespecified stratified subset would suffice if full evaluation is too costly.
-
Capability categories
- Can the authors report performance by visual skill for the photographic Boolean statements, such as attribute grounding, object-reference binding, and spatial relations?
- Please include sample counts and uncertainty estimates so that small categories are not overinterpreted.
-
Diagnostic validity and annotation quality
- How much of the error remains when the correct atomic truth values are supplied?
- Please describe the human audit in sufficient detail, with its coverage, disagreement resolution, and any corrections made. For statistical analysis, questions that share an image or expression pair should remain grouped.
Limitations
- The current collection supports a narrow diagnostic setting. Its logical diversity is more extensive than its visual diversity. General claims about real-world visual reasoning should reflect this distinction.
- Observed disagreement is not necessarily caused entirely by the rewrite. Since sampling settings are not fixed, repeated identical prompts would provide a useful reference for the instability observed across equivalent formulations.
- My main concerns are an evaluation protocol that prevents explicit reasoning and the limited image coverage. For a benchmark contribution, these are central to assessing both its remaining difficulty and its coverage. The formal validation and human review are valuable, but they do not establish these properties on their own.
- I would consider raising my score if the authors show meaningful difficulty under protocols that allow reasoning, broader or convincingly stable visual coverage, and clearer capability-level diagnostics. I do not require the dataset to match the largest existing benchmarks. I need stronger evidence that its size, composition, and evaluation protocol support a reliable and useful benchmark.
3 comments
Response to Reviewer iJfm (II)
- Capability categories.
We thank you for this good suggestion, as it needs no new inference! For the photographic Boolean track, we will certainly tag each atomic statement as attribute, object-reference binding, or spatial relation, and report definitions, image and question counts, and per-configuration scores with scene-grouped intervals. We will cross this with logical family only where cell sizes allow. Since each question mixes ten statements, we will report by a question's statement composition and note that this is a property of the question, not an isolation of the failing skill.
- Separating grounding from Boolean execution
We agree the current evaluation is end-to-end. However, our work provides some evidence in that accuracy is the same on generated scenes, where each statement is one colour or shape of a rendered labelled object, as on photographs (Table 10), and that near-constant labels (Table 11) show little dependence on the image.
We now answer your remaining questions.
- Human audit
To elaborate on your questions about the human auditing, we would like to clarify that we, the authors, audited every AI-proposed photographic annotation: all 46 photographs, every atomic statement used in the 2,000 photographic Yes/No questions (2,000 distinct statements), and every object reference point. The reviewers judged each statement from the image without access to the questions' reference answers, then compared against the AI-assistant label, and any disagreement was to go to discussion with the third author. Here, no annotation needed correction. All authors also inspected a randomly sampled subset of 600 generated scenes, and the authors jointly solved 600 complete questions by hand, all matching the reference answers.
- Limitations
We will add a limitations section covering the fact that our benchmark was not focused on a large scope of visual diversity, and did not evaluate the latest frontier models, and while we recognise that your conditions for raising the score include showing difficulty under reasoning-enabled protocols on stronger models, we are unfortunately unable to provide this due to a restricted academic compute budget. That being said, we sincerely hope the stability analyses, capability breakdown, scoped claims and the evidence we have provided above address your remaining conditions.
We hope these answer your concerns, and, of course, we would be happy to elaborate further and answer any further questions.
Thank you,
Authors
Response to Rebuttals
I understand the design intent behind keeping the visual inputs simple, along with the detailed explanation of the human review process, and my concerns regarding this aspect have been alleviated. I also acknowledge the evidence showing that errors persist even with short problems and in the 'thinking-on' setting. However, based on the current results, it remains difficult to determine the task difficulty under sufficient reasoning conditions and to distinguish between failures at the perceptual versus logical stages. Please notify me via reply once the promised additions have been incorporated!
Response to Reviewer iJfm (I)
Dear Reviewer iJfm,
We thank you for your detailed review. Please find our response to each of your concerns below:
We agree the collection is logically much more diverse than it is visually, and you are right that we should state this in the limitations. However, we wish to point out that the design is somewhat deliberate: our benchmark's variable of interest is the Boolean function, so the visual facts are kept simple in order to stay exactly verifiable.
The 3,000 base expressions realise 2,746 distinct truth tables over ten inputs, each paired with a verified rewrite.
On generated scenes, all 1,024 assignments are realisable, and labels are balanced within each scene.
In the revision's comparison table, we will separate the quantities you list: 1,046 unique images (1,000 generated, 46 photographs), 45 photographic scene families, 3,000 base expressions, 3,000 equivalent variants, 2 presentation formats, and, for identification, 1,000 questions over 241 photograph–target combinations and nine templates.
On stability, we will add scene-grouped resampling for both sources (1,000 generated scenes; 45 scene families), with questions that share an image or expression pair kept together, as you requested, which should address your concerns of stability under resampling of the released scenes.
We agree in part, and will rewrite Section 5 to reflect the following:
Where we agree: For 54–370-node expressions answered in one word, near-chance accuracy is consistent with the expressivity results we cite, and we agree that the paper should present it as the measured behaviour of a restricted, though widely deployed, inference regime and not as a general capability ceiling. We also agree that label collapse, not attempted reasoning, explains the ~50% figures. We report this in Tables 7 and 11 and will move it into the main text.
Where the results are less expected:
The same configurations fail on items that are not long. Identification uses 2–5 predicates, and two predicate values suffice for 870 of 1,000 items, yet accuracy is 16.5-43.0% against 16.9% for uniform guessing. On the sparse-CNF control, three facts can certify the answer, and GLM-off scores 40.8%,
The one-word instruction is not the cause. GPT-5.6 Luna received the original prompts without that suffix and scored 49.05% / 48.67% with 2 invalid responses in 12,000.
Reasoning does not uniformly remove the difficulty in the one matched pair we have. GLM-on reaches 82.62% when 3–4 facts settle the answer but 51.60% when 7–10 do; four families covering half the Boolean questions stay at or below 56.8%, and both-correct accuracy (45.13%) stays under the constant-label baseline, and conflicts on equivalent pairs rise to 36.47%.