acceptodds
Under review as a conference paper at ICLR 2027

Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

Abstract

Reliable visual reasoning requires composing multiple visual observations and preserving their answers across logically equivalent expressions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises of two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves substantial but uneven gains, but still demonstrates notable failure modes. Hob-VL exposes these failures through executable reference answers and matched evaluations.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, visual question answering, compositional reasoning, logical reasoning, visual grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iJfmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Hob-VL evaluates visually grounded Boolean reasoning through two tasks, Boolean condition evaluation and object identification. It contains 6,000 Yes/No questions and 1,000 identification questions over 1,000 synthetic images and 46 photographs. Each question has symbolic and structured natural-language versions. The Boolean questions also include verified De Morgan pairs.
  • The paper evaluates eight VLMs in nine configurations and reports accuracy, equivalent-pair correctness, disagreement, and output validity. Most configurations disable or minimize thinking. One GLM configuration enables thinking and achieves substantially higher scores in several conditions.
Critical strengths
  • The benchmark has a clear and useful motivation. Correct answers to individual visual questions do not guarantee correct composition or consistent answers across equivalent expressions. The paired design makes these distinctions explicit.
  • Executable validation is a valuable feature. Exhaustive truth-table checks provide strong guarantees about the Boolean expressions and their equivalence. The reported human review of visual annotations is also welcome, since logical validation alone cannot establish that the visual facts are correct.
  • The paper provides useful logical annotations and some detailed analysis. Construction families, certificate sizes, and input influence support analyses beyond aggregate accuracy. Reporting label bias and invalid outputs alongside consistency also helps avoid misleading interpretations.
Critical weaknesses
  1. The scale and diversity of the visual collection are not sufficiently justified for a benchmark contribution.

    • The synthetic component contains only 1,000 scenes, each with ten objects drawn from three shapes and six colors. The real-image component has 46 photographs from 45 scene families reused across 3,000 questions, and the 1,000 identification questions cover only 241 distinct photograph and target combinations (Section 4.4).
    • For scale context, CLEVR's validation split alone contains 15,000 images and 149,991 questions. The natural-image portion of MM-CondChain contains 398 images and 796 instances, and ConVBench reports 686 images and 1,372 questions. These sizes are not minimum requirements, but they show the difference between question count and coverage of visual evidence.
    • Table 10 offers partial evidence of stability, since results on synthetic scenes and photographs are broadly similar across configurations. It does not yet show stability across fresh synthetic samples or a broader photographic collection.
  2. Near-chance scores mainly reflect the evaluation protocol rather than model capability.

    • Every configuration except GPT-5.6 Luna receives the instruction "Respond with exactly one word" (Appendix A.1), and any answer with explanatory text is scored as invalid. When thinking is disabled, models must evaluate expressions of 54 to 370 AST nodes in one word without intermediate computation. The paper itself cites Merrill and Sabharwal (2024) and Li et al. (2024), who show that intermediate generation extends what transformers can compute on serial problems. Results near 50% are therefore largely expected under this protocol.
    • The near-chance accuracy also reflects collapse onto a single label rather than attempted reasoning. Table 11 shows Yes rates of 98.60% for GLM-off in natural language and 96.25% for Llama in symbolic form, but only 1.35% and 1.78% for Claude Haiku and DeepSeek in symbolic form. Both-correct accuracy is below the 50% constant-label baseline for all nine configurations in both formats.
    • Most evaluated models are small or fast tiers, such as Claude Haiku 4.5, Gemini 3.5 Flash-Lite, DeepSeek Flash, GLM-4.6V-FlashX, Llama 4 Scout, and Qwen3-VL-8B. The GLM comparison shows how much the configuration matters. Symbolic Boolean accuracy rises from 48.58% to 63.50% and symbolic identification from 31.30% to 73.70% with thinking and a larger budget.
    • The evaluation should allow visible chain-of-thought and extract only the final answer from a marked field. Representative frontier models with reasoning enabled and adequate budgets should also be included. Otherwise the benchmark mainly exposes restricted inference settings rather than its remaining difficulty.
  3. Capability categories should focus where the design leaves skills mixed.

    • The synthetic Boolean track queries only color and shape, and the identification track uses only spatial predicates. Table 10 already reports results by image source. A per-skill breakdown would therefore add most value for the photographic Boolean statements, which mix attributes and spatial relations.
    • For that track, each category should have definitions, image and question counts, and per-model scores. The logical-family breakdown could then be crossed with these categories where sample size permits.
  4. The current evaluation does not sufficiently separate visual grounding from Boolean execution.

    • A wrong final answer may come from an incorrect visual fact, a parsing error, or an incorrect logical calculation. Logical-property annotations characterize the questions but do not identify which stage caused a failure. The paper itself acknowledges this limitation.
    • Two controls would substantially improve the diagnostic value. One should evaluate the atomic visual statements separately. The other should provide the correct truth values and ask the model to evaluate the same expression.
    • Output-budget failures should also remain distinct from reasoning errors. Thinking-enabled GLM reaches the token limit on 536 natural-language Boolean questions, which makes an adequate-budget evaluation particularly important.
Questions for authors
  1. Visual coverage and scale

    • What evidence supports the adequacy of 1,000 synthetic scenes and 46 photographs?
    • Could the authors show that the findings are stable across fresh synthetic samples and scene-level resampling?
    • Please distinguish unique images, scene families, base expressions, equivalent variants, and presentation formats in any comparison with existing benchmarks.
  2. Evaluation protocol and strong reasoning models

    • Could the authors rerun a representative subset with visible chain-of-thought allowed and the final answer extracted from a marked field?
    • Could they evaluate at least two representative frontier multimodal models with reasoning enabled and adequate budgets, and report token use and truncation rates?
    • Do substantial errors and equivalent-expression conflicts remain under these settings? A prespecified stratified subset would suffice if full evaluation is too costly.
  3. Capability categories

    • Can the authors report performance by visual skill for the photographic Boolean statements, such as attribute grounding, object-reference binding, and spatial relations?
    • Please include sample counts and uncertainty estimates so that small categories are not overinterpreted.
  4. Diagnostic validity and annotation quality

    • How much of the error remains when the correct atomic truth values are supplied?
    • Please describe the human audit in sufficient detail, with its coverage, disagreement resolution, and any corrections made. For statistical analysis, questions that share an image or expression pair should remain grouped.

Limitations

  • The current collection supports a narrow diagnostic setting. Its logical diversity is more extensive than its visual diversity. General claims about real-world visual reasoning should reflect this distinction.
  • Observed disagreement is not necessarily caused entirely by the rewrite. Since sampling settings are not fixed, repeated identical prompts would provide a useful reference for the instability observed across equivalent formulations.

  • My main concerns are an evaluation protocol that prevents explicit reasoning and the limited image coverage. For a benchmark contribution, these are central to assessing both its remaining difficulty and its coverage. The formal validation and human review are valuable, but they do not establish these properties on their own.
  • I would consider raising my score if the authors show meaningful difficulty under protocols that allow reasoning, broader or convincingly stable visual coverage, and clearer capability-level diagnostics. I do not require the dataset to match the largest existing benchmarks. I need stronger evidence that its size, composition, and evaluation protocol support a reliable and useful benchmark.
3 comments
AuthorsOpenReview ↗

Response to Reviewer iJfm (I)

Dear Reviewer iJfm,

We thank you for your detailed review. Please find our response to each of your concerns below:

  • Scale and diversity of the visual collection.

We agree the collection is logically much more diverse than it is visually, and you are right that we should state this in the limitations. However, we wish to point out that the design is somewhat deliberate: our benchmark's variable of interest is the Boolean function, so the visual facts are kept simple in order to stay exactly verifiable.

  1. The 3,000 base expressions realise 2,746 distinct truth tables over ten inputs, each paired with a verified rewrite.

  2. On generated scenes, all 1,024 assignments are realisable, and labels are balanced within each scene.

In the revision's comparison table, we will separate the quantities you list: 1,046 unique images (1,000 generated, 46 photographs), 45 photographic scene families, 3,000 base expressions, 3,000 equivalent variants, 2 presentation formats, and, for identification, 1,000 questions over 241 photograph–target combinations and nine templates.

On stability, we will add scene-grouped resampling for both sources (1,000 generated scenes; 45 scene families), with questions that share an image or expression pair kept together, as you requested, which should address your concerns of stability under resampling of the released scenes.

  • Near-chance scores reflect the protocol

We agree in part, and will rewrite Section 5 to reflect the following:

Where we agree: For 54–370-node expressions answered in one word, near-chance accuracy is consistent with the expressivity results we cite, and we agree that the paper should present it as the measured behaviour of a restricted, though widely deployed, inference regime and not as a general capability ceiling. We also agree that label collapse, not attempted reasoning, explains the ~50% figures. We report this in Tables 7 and 11 and will move it into the main text.

Where the results are less expected:

  1. The same configurations fail on items that are not long. Identification uses 2–5 predicates, and two predicate values suffice for 870 of 1,000 items, yet accuracy is 16.5-43.0% against 16.9% for uniform guessing. On the sparse-CNF control, three facts can certify the answer, and GLM-off scores 40.8%,

  2. The one-word instruction is not the cause. GPT-5.6 Luna received the original prompts without that suffix and scored 49.05% / 48.67% with 2 invalid responses in 12,000.

  3. Reasoning does not uniformly remove the difficulty in the one matched pair we have. GLM-on reaches 82.62% when 3–4 facts settle the answer but 51.60% when 7–10 do; four families covering half the Boolean questions stay at or below 56.8%, and both-correct accuracy (45.13%) stays under the constant-label baseline, and conflicts on equivalent pairs rise to 36.47%.

AuthorsOpenReview ↗

Response to Reviewer iJfm (II)

  • Capability categories.

We thank you for this good suggestion, as it needs no new inference! For the photographic Boolean track, we will certainly tag each atomic statement as attribute, object-reference binding, or spatial relation, and report definitions, image and question counts, and per-configuration scores with scene-grouped intervals. We will cross this with logical family only where cell sizes allow. Since each question mixes ten statements, we will report by a question's statement composition and note that this is a property of the question, not an isolation of the failing skill.

  • Separating grounding from Boolean execution

We agree the current evaluation is end-to-end. However, our work provides some evidence in that accuracy is the same on generated scenes, where each statement is one colour or shape of a rendered labelled object, as on photographs (Table 10), and that near-constant labels (Table 11) show little dependence on the image.

We now answer your remaining questions.

  • Human audit

To elaborate on your questions about the human auditing, we would like to clarify that we, the authors, audited every AI-proposed photographic annotation: all 46 photographs, every atomic statement used in the 2,000 photographic Yes/No questions (2,000 distinct statements), and every object reference point. The reviewers judged each statement from the image without access to the questions' reference answers, then compared against the AI-assistant label, and any disagreement was to go to discussion with the third author. Here, no annotation needed correction. All authors also inspected a randomly sampled subset of 600 generated scenes, and the authors jointly solved 600 complete questions by hand, all matching the reference answers.

  • Limitations

We will add a limitations section covering the fact that our benchmark was not focused on a large scope of visual diversity, and did not evaluate the latest frontier models, and while we recognise that your conditions for raising the score include showing difficulty under reasoning-enabled protocols on stronger models, we are unfortunately unable to provide this due to a restricted academic compute budget. That being said, we sincerely hope the stability analyses, capability breakdown, scoped claims and the evidence we have provided above address your remaining conditions.

We hope these answer your concerns, and, of course, we would be happy to elaborate further and answer any further questions.

Thank you,

Authors

Reviewer_iJfmOpenReview ↗

Response to Rebuttals

I understand the design intent behind keeping the visual inputs simple, along with the detailed explanation of the human review process, and my concerns regarding this aspect have been alleviated. I also acknowledge the evidence showing that errors persist even with short problems and in the 'thinking-on' setting. However, based on the current results, it remains difficult to determine the task difficulty under sufficient reasoning conditions and to distinguish between failures at the perceptual versus logical stages. Please notify me via reply once the promised additions have been incorporated!

Reviewer_RHdWOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Hob-VL, a benchmark for evaluating Boolean reasoning over visual facts and consistency across logically equivalent expressions. It contains 6,000 balanced Yes/No questions organized into 3,000 base/De Morgan pairs, along with 1,000 object-identification questions, presented in symbolic and structured natural-language formats. The benchmark uses exhaustively validated logical expressions and reports accuracy, pairwise correctness, disagreement, and output validity. Experiments across eight model configurations show that Boolean accuracy remains around 50% when thinking is disabled or minimized. Enabling thinking substantially improves GLM's symbolic reasoning and object identification, although consistency and completion errors remain.

Critical strengths
  1. The benchmark provides a controlled setting for evaluating visual Boolean reasoning. Its logical expressions and equivalence transformations can be exhaustively verified, while annotations such as influence, certificate size, and two-fact predictability support more detailed analysis beyond expression length.

  2. The paired evaluation design is useful for analyzing consistency. By comparing equivalent Boolean expressions and different presentation formats, the benchmark distinguishes logical consistency from format sensitivity. Reporting pairwise correctness alongside disagreement also helps avoid treating consistent but incorrect answers as successful reasoning.

  3. The paper provides useful implementation details and discusses important evaluation limitations. The appendices document inference settings, model-family differences, and constraints on realizable photographic scenes. The requirement that object-identification answers remain unique under uncertain spatial relations is also a reasonable design choice.

Critical weaknesses
  1. The experiments do not separate visual perception errors from Boolean reasoning failures. The reported results are based on end-to-end evaluation, without measuring the accuracy of individual visual facts or testing Boolean reasoning with ground-truth fact assignments. As a result, it is unclear how much of the observed failure comes from incorrect visual perception rather than logical reasoning.

  2. The effects of presentation format and thinking are not fully isolated. Natural-language prompts are substantially longer than symbolic ones (1911.5 vs. 371 median words), and GLM-on reaches the output limit on 536 of 6000 natural-language Boolean questions (Table 8). This makes it difficult to attribute the 51.20% versus 63.50% accuracy gap to presentation format alone. The different output budgets for GLM-on and GLM-off (4096 vs. 2048 tokens) also complicate the comparison of thinking configurations.

  3. The reliability of the photographic annotations needs further validation. The paper does not clearly report the extent of independent human review or how annotation disagreements and corrections were handled. Moreover, the photographic questions come from only 46 images across 45 scene families, limiting the diversity of independent visual evidence. Without scene-level analysis and clearer annotation validation, it is difficult to assess how representative and reliable the photographic results are.

Questions for authors
  1. Can the authors evaluate atomic visual-fact accuracy and Boolean reasoning using ground-truth fact assignments on a representative subset? This would help distinguish perception errors from logical reasoning failures.

  2. Does the GLM performance gap between symbolic and natural-language formats remain when both receive sufficient output budgets? A shorter natural-language representation preserving the same logical structure would also help isolate the effect of format.

  3. How were the photographic annotations validated, and how were any disagreements or errors handled? Could the authors also report whether the results remain stable across different photographic scenes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RHdW

Dear Reviewer RHdW,

We thank you for your close reading and the detailed review of our paper. We do agree with the weaknesses you listed, and respond about what can be established now. Please find our response below:

  • The evaluation remains end-to-end

We agree, and we acknowledge this limitation in Section 2. However, our existing results offer some indirect evidence about where the error arises in the thinking-disabled configurations.

  1. On generated scenes, every statement is a single colour or shape of a large, labelled, rendered object, so the perception demands are minimal. And here, accuracy is the same as it is on photographs (48.08-50.10% vs 48.85-52.20%, Table 10).

  2. The predicted-label frequencies of 1-2% or 96-99% Yes (Table 11) mean that the answer barely depends on the image at all.

Both points suggest the bottleneck is not photographic perception, and while we are not able to run additional model evaluations due to a restricted academic compute budget, we cannot report atomic-fact or oracle-assignment results. However, we will certainly include their absence as a limitation in the main text of the manuscript, and we will release, for the stratified subset you describe (by family and image source, retaining equivalent pairs), the per-statement atomic prompts and the oracle-assignment prompts with reference truth values supplied in text. Generating these is deterministic and requires no model calls, so the controls can become a ready-made part of the benchmark for anyone who runs it.

  • Format and thinking comparisons are entangled with length and completion budget.

We agree with you on this point, and will reword the 51.20% vs 63.50% comparison as a difference between configurations and not as a format effect. We also provide the following clarifications:

  1. Truncation alone does not close the gap. If all 536 truncated NL responses are counted as correct, GLM reasoning on NL accuracy is 60.13% (derived), still below the 63.50% symbolic figure, which itself counts 47 invalid responses as wrong. Restricted to valid outputs, the accuracy is 56.2% NL and 64.0% symbolic (derived), and so the length confounder remains.

  2. The differing caps do not affect the reasoning off/on comparison. GLM without reasoning produced zero invalid responses and zero token-limit terminations (Table 8), so its 2,048-token cap was never reached.

We will certainly release a compact NL rendering of the same AST (statements defined once and referred to by name), which is deterministic and needs no model calls, so that the format comparison can be made at matched length.

  • Photographic annotation assurance

Thanks for catching this, and we will add the following paragraph to Appendix B.6 on the details of the human audit:

Human audit. The audit was carried out by the authors. Initial photographic facts and object reference points were proposed by an AI assistant; the audit checked every one of them. Coverage was complete: all 46 photographs, every atomic statement used in the 2,000 photographic Yes/No questions (2,000 distinct statements), and every object reference point used by the identification track. Reviewers were shown the image and the statement and judged it from the image, without access to the reference answers of the questions built on those statements; their judgement was then compared with the AI-assistant label. Any disagreement was to be resolved by discussion with the third author, in the event that no annotation required correction. All authors also inspected the 1,000 generated scenes, in addition to the automated geometric and pixel checks. Finally, the authors jointly solved 600 randomly sampled complete questions by hand, end to end, and all 600 matched the reference answers.

We will also correct the framing of scale: indeed, the photographic questions share 46 images from 45 scene families, and the question counts should not be read as independent visual coverage. Concretely, from our saved responses, we are happy to add the per-photograph accuracy for every configuration, the leave-one-scene-family-out accuracy, and intervals from resampling scene families while keeping each base/rewrite pair together.

We hope these answer your concerns, and, of course, we would be happy to elaborate further and answer any further questions.

Thank you,

Authors

AuthorsOpenReview ↗

Follow-up to Reviewer RHdW

Dear Reviewer RHdW,

We wish to follow up in case you have had a chance to review our rebuttal.

Of course, we greatly appreciate the time you have taken to review our work and would be grateful for any remaining questions or concerns that we can clarify before the discussion period closes in one day.

If you find that we have answered all your concerns, we would be grateful if you could increase your score to reflect this. We would also be very happy to provide any further detail that would help address any of your remaining concerns.

Thank you again for your consideration.

Best,

Authors

Reviewer_4RRoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new visual reasoning benchmark called Hob-VL which asks complex logical questions about synthetic and naturally generated images. Each question is presented in two formats (natural language and symbolic) and in two logically equivalent forms (by rewriting sub-expressions using De Morgan’s law).

Several LLMs are evaluated using the benchmark. Among the evaluated LLMs, non-reasoning models perform roughly at chance; and enabling reasoning improves accuracy.

Critical strengths

The paper introduces a new visual reasoning benchmark that current LLMs cannot solve. The benchmark seems generally well-designed to eliminate biases and stress-test complex visual reasoning tasks.

Critical weaknesses

The overall motivation of this benchmark is unclear; there isn’t enough discussion and contextualization of the goals of the project, or the significance of the results. Is it a problem that LLMs cannot do this task? Is this a bug to be fixed? Or does it tell us something interesting about LLMs? Should this type of evaluation be part of the standard set of benchmarks we use to evaluate LLMs?

A new benchmark is mostly interesting when current methods cannot solve it. But the paper does not evaluate state-of-the-art LLMs. It evaluates previous-generation budget models such as Haiku 4.5 (released October 2025), GPT 5.6 Luna (released July 2026) and Gemini 3.5 Flash-Lite (released July 2026). All of these models are the cheapest and least capable in their family, and all of them have since been superceded by more recent models. In order to demonstrate that the benchmark is useful, it would be much more convincing to evaluate the latest set of models and the most capable models.

The decision to evaluate models with reasoning disabled is odd and not discussed. I could imagine some intellectual value in understanding a model’s implicit vs explicit reasoning skills, but this is not discussed. Suppose that a state-of-the-art model (e.g. GPT 6 Astra, Opus 5.5, Fable 5.1) with full reasoning and tool use enabled is able to solve this task perfectly. Would this benchmark still have value? Why or why not?

Questions for authors

How do the current state-of-the-art models perform on this benchmark?

Why did you choose to disable reasoning mode in the main experiments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4RRo (I)

Dear Reviewer 4RRo,

We thank you for your review. Please find our responses below.

  • The overall motivation of this benchmark is unclear; there isn’t enough discussion and contextualization of the goals of the project, or the significance of the results. Is it a problem that LLMs cannot do this task? Is this a bug to be fixed? Or does it tell us something interesting about LLMs?

Thanks for the question! In the second paragraph of the introduction in our manuscript, we listed a few applications of this benchmark. To elaborate, we really believe that it is a problem for a concrete class of use cases. For instance, the examples in our introduction (paragraph 2) are Boolean filters over visual facts: a package that is damaged, beside a loading bay, not covered, and not inside a container; plants not under irrigation that either have yellow leaves and are wilting, show pest damage; record-linked X-ray queries (EHRXQA); video-event queries (EQUI-VOCAL), or perception-to-planning questions (DriveLM). And while each ``atomic fact’’ can be easy for a VLM to verify, the prediction risk is in the Boolean combination of these facts, since a wrong combination can produce a plausible but wrong result set with no visible error.

In particular, this does say some interesting things about the failure modes of LLMs (and we certainly point to it as an issue that should be fixed to make it compatible with the aforementioned downstream applications):

  1. Failures take the form of label collapse: with balanced labels, Claude Haiku 4.5 and DeepSeek answer Yes on 1.35% and 1.78% of symbolic questions, while Llama 4 Scout and GLM (off, NL) answer Yes on 96.25% and 98.60% (Table 11). Specifically, the models do not hedge, refuse, or signal difficulty, and so the user sees a confident one-word answer that is ultimately incorrect,

  2. It is not only very long expressions. Our identification tasks combine just 2-5 spatial predicates, and for 870 of 1,000 items, two predicate values suffice to single out the target (Appendix B.6). So, accuracy is 16.5-43.0% against a 16.9% uniform baseline. In the sparse-CNF control, where three facts can certify the answer, GLM without thinking scores 40.8% (Table 9).

  3. Reasoning seems to help selectively. Enabling thinking on GLM raises symbolic accuracy to 82.62% when 3-4 facts settle the answer, but to only 51.60% when 7-10 are needed (Table 5). In fact, four of seven families, which make up half the Boolean questions, stay at or below 56.8% (Table 9).

  4. Accuracy and consistency come apart. With thinking, GLM's accuracy rises while its conflict rate on logically equivalent pairs roughly doubles, from 18.90% to 36.47% (Table 7).

  • A new benchmark is mostly interesting when current methods cannot solve it. But the paper does not evaluate state-of-the-art LLMs. Why did you choose to disable reasoning mode in the main experiments? The decision to evaluate models with reasoning disabled is odd and not discussed. I could imagine some intellectual value in understanding a model’s implicit vs. explicit reasoning skills, but this is not discussed.

This is correct, and we will certainly bring this up as a limitation. To clarify, since this work was done under academic compute constraints, it would be far too expensive to run full-reasoning frontier evaluation at 14,000 multimodal requests on frontier LLMs, and our paper makes no claim about those more frontier models, though we hope that models such as Gemini-2.5 Flash, GPT 5.6 Luna, and the GLM reasoning model (which saturated multimodal benchmarks at their time of release) are reliable proxies for the more up-to-date frontier models (as they differ by only < 3 months between release dates).

Now, we would like to point out that the models we evaluated are also representative of the kind of models that are deployed in extremely large-scale, high-volume applications, where reasoning is typically turned off or is kept minimal for cost and latency: we believe that results for this tier of models are informative in their own right. Additionally, the contribution of a benchmark paper is the instrument as well as a snapshot of scores. Our independent contributions, which consist of the construction, validation suite, and metrics, are independent of which models we could afford, and the harness is released so that future work can add rows.

Moreover, our experiments also isolate whether composition happens without externalised intermediate computation, which is exactly the distinction drawn by the expressivity results we cite in our manuscript, due to Merrill and Sabharwal (2024) and Li et al. (2024). The matched GLM off/on pair we included was to show the contrast on fixed weights and fixed prompts.

AuthorsOpenReview ↗

Response to Reviewer 4RRo (II)

  • Suppose a state-of-the-art model with full reasoning and tool use solves this perfectly. Would the benchmark still have value?

We thank you for this question, and we argue that the benchmark would still have value even if a SOTA model with full reasoning and tool use saturated it.

Firstly, our benchmark could then measure cost. If full reasoning is required, the benchmark quantifies how much reasoning budget a model needs before visually grounded composition becomes reliable, and which function properties drive that budget (Table 5 is a first such curve). This is a valuable measurement since a system (such as the ones corresponding to the applications in the introduction of our manuscript) that processes thousands of images cares about that price. Moreover, as our GLM experiments demonstrate, consistency can remain a separate target. Namely, a model can be accurate on average but still contradict itself on equivalent expressions. Finally, with code execution, the logical step should be easy: once the ten facts are known, evaluation should be linear in expression size (Section 2). In this case, a good tool-using model that succeeds must reduce the task to “perception plus a call to an interpreter”, which, while is a legitimate system, answers a different question from whether the model itself composes. In this setting, our benchmark, Hob-VL, studies the setting without tools. We will also add a short "what would saturate this benchmark, and what then" paragraph to our manuscript.

We hope these answer your concerns, and, of course, we would be happy to elaborate further and answer any further questions.

Thank you,

Authors

AuthorsOpenReview ↗

Follow up to Reviewer 4RRo

Dear Reviewer 4RRo,

We wish to follow up in case you have had a chance to review our rebuttal.

Of course, we greatly appreciate the time you have taken to review our work and would be grateful for any remaining questions or concerns that we can clarify before the discussion period closes in one day.

If you find that we have answered all your concerns, we would be grateful if you could increase your score to reflect this. We would also be very happy to provide any further detail that would help address any of your remaining concerns.

Thank you again for your consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.