acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Reasoning Over Visual Evidence for Compositional VQA

Abstract

Training-free compositional methods answer visual questions by chaining pretrained vision and language models to collect intermediate evidence and compose answers. Methods that execute a program to answer a question make hard decisions on each perception, discarding its uncertainty, while methods that keep perceptions uncertain follow a fixed evidence collection strategy. We introduce PROVE (Probabilistic Reasoning Over Visual Evidence), a training-free neuro-symbolic framework for compositional visual question answering that adaptively collects uncertain visual evidence. PROVE preserves perception uncertainty with calibrated probabilistic logic facts that define a Bayesian network, then answers the query by exact inference with ProbLog. Under matched components on four benchmarks, PROVE outperforms five training-free compositional methods by up to points and improves over every VLM it is built on by up to points. A deterministic ablation that rounds each perception to true or false shows that preserving uncertainty adds up to points, with the gain rising to points on questions with the most borderline perceptions. PROVE also inherits gains in LLM reasoning, as thinking mode alone adds up to points.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
visual question answering, compositional visual reasoning, neuro-symbolic reasoning, probabilistic logic programming, ProbLog, Bayesian networks, uncertainty propagation, training-free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k6RSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce PROVE, a training-free neurosymbolic approach for compositional VQA that incorporates perception uncertainty into the answer generation process. The authors compare this approach to existing program synthesis and neurosymbolic methods with matched tools and find large increases to performance across a suite of compositional VQA benchmarks.

Critical strengths

Premise of the paper is strong: The idea of keeping the probability associated with each stage of a solution is a good idea - and I particularly like the use of ProbLog to compute the probability of each hypothesis. The paper presents a rigorous and principled approach for simple VQA over attributes and spatial relations.

Comparison to DePROVE: The comparison ablating the preservation of uncertainty was necessary, and gives confidence to the validity of the approach on the shown benchmarks.

Writing: The writing is clear and easy to follow, the figures present in the paper are valuable and complimentary to the text.

Critical weaknesses

Strict framework is limiting: While the proposed framework is effective at the shown benchmarks, the authors have limited their experiments to VQA queries that are generated as a composition of attributes and spatial relation. In a sense, this method is tailor-made for these queries - the method decomposes the scene into entities, their relations & attributes and the simple hypotheses that these benchmarks take as answers. The task of visual question answering is much broader than this (e.g. medical queries, 3D spatial reasoning queries with open-ended numeric answers, questions about intent and causality) - all of these are not easily decomposable in the way this method operates. The authors should either explore the generalization of their method to a broader scope of visual question answering or describe how this method can augment existing, more general systems.

Additional qualitative examples: I would have liked to see more qualitative examples. The example of the corgi is particularly illustrative, but further examples (e.g. highlighting cases where PROVE and DePROVE yield different answers) would have been valuable. I recommend the authors to add more, even if they are put in the supplementary.

Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer k6RS (1/1)

We thank the reviewer for the thoughtful feedback. We address each concern in turn and attempt to implement the reviewer's suggestions. Our changes to the paper in response to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

1. "Strict framework is limiting: ...the authors have limited their experiments to VQA queries that are generated as a composition of attributes and spatial relation."

1.1) "In a sense, this method is tailor-made for these queries... The task of visual question answering is much broader than this (e.g. medical queries, 3D spatial reasoning queries with open-ended numeric answers, questions about intent and causality)..."

1.2) "The authors should either explore the generalization of their method to a broader scope of visual question answering or describe how this method can augment existing, more general systems."

We agree that VQA is a much broader field than compositional VQA, the setting we evaluate. We aim to study how to logically compose multiple uncertain perceptions to derive an answer. To isolate this problem, we evaluate on standard compositional VQA benchmarks that require no domain-specific knowledge (Section 4, L352-353), where a scene is represented by its entities, attributes, and relationships. Tasks such as medical queries or 3D spatial reasoning also require capabilities such as domain knowledge or depth perception. These depend on the underlying models rather than on how their perceptions are composed, so results on them would not isolate PROVE’s contribution.

The core design of PROVE can be adapted to other domains that share the problem of logically composing multiple uncertain perceptions. Adapting it may require a different symbolic representation and other foundation models as tools. For example, in 3D spatial reasoning, a depth estimation model could provide 3D relationships as probabilistic facts. A more general system could also use PROVE as one component, supplying confidence scores for compositional claims about an image. We believe these are interesting directions for future work and have added this discussion to the conclusion (Section 5, L536-539).

In particular in lines 536-539 (Section 5), we added:

Future work could adapt PROVE to other domains that require logically composing uncertain perceptions, using other foundation models as tools (e.g., depth estimation for 3D scenes) and extending the symbolic representation beyond entities, attributes, and relationships where needed. PROVE could also serve as one component of a more general system, supplying confidence scores for compositional visual claims.

2. "Additional qualitative examples: I would have liked to see more qualitative examples."

2.1) "...further examples (e.g. highlighting cases where PROVE and DePROVE yield different answers) would have been valuable. I recommend the authors to add more, even if they are put in the supplementary."

We agree that more qualitative examples make the benefit of preserving uncertainty more concrete, and we thank the reviewer for the suggestion. We have added three examples that highlight cases where PROVE and DePROVE yield different answers, each shown with its collected facts, probabilities, and rule, in a new appendix (Appendix G). The main text now references them (Section 4.2, L432).

I thank the authors for their rebuttal. I appreciate the inclusion of the discussion on extending this framework for general applications and the additional qualitative examples. I maintain my score of 3.

Reviewer_Dq1tOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional methods for compositional visual question answering typically resolve a query by executing a generated program to make a hard decision on the result of each perception, discarding its uncertainty. This paper, instead, considers such perception uncertainty and proposes a probabilistic reasoning process based on the uncertainty. Empirical results show that the proposed reasoning can effectively improve performance compared with five baseline models on four compositional visual question answering benchmarks.

Critical strengths

The idea of utilizing the perception uncertainty and turning deterministic VQA reasoning into probabilistic reasoning is good and makes sense.

Critical weaknesses

Most descriptions are in plain text, making it hard for readers to accurately understand what the authors refer to.

Some claims can be overly certain.

Questions for authors
  1. Assumptions: The two assumptions (L159-160) are provided without description and justification. First, can you explain these assumptions more explicitly? For example, for A2, I wonder whether it means that attributes are independent of relationships, or all attributes are independent of each other, and the same for relationships. A mathematical formulation here would be helpful. Also, can you justify these assumptions, like why they are practical assumptions?

  2. Method description: I do not follow the sentences in L183-185 about the goal of scene entities. How is the scene entity used in the proposed paradigm?

  3. Detector description:

3.1 L257 mentions that the detector takes an image and an object class. I wonder if these object classes are predefined. Moreover, a broader question is: What needs to be predefined in this paper?

3.2 I don’t think one can treat the detector scores as the probability of the existence of an entity. There is a gap between the two. What the detection model’s score measures is the model's confidence that a predicted bounding box corresponds to a particular object category or text description. This score does not correspond to the (unnormalized) probability of the existence of an object.

  1. Verifier description: L266-268 states that the returned probabilities from VLMs are treated as conditional probabilities because when asking VLMs about the entity attribute, you have assumed that the entity exists. But we can always ask VLMs this question about an entity attribute even if the entity does not exist. Hence, we should interpret the returned probabilities as Pr(entity exists AND entity color is orange), rather than Pr(entity color is orange | entity exists). Therefore, the returned probabilities are not conditional ones.

  2. Logic part: L148 states that “a rule-generation LLM generates the logic rule equivalent to the query”. This sentence is overly certain, because there is no such guarantee in the LLM’s generation. In other words, the LLM-generated rules might not be the ground-truth ones. There can still be errors within the LLM-generated rules, which would affect the downstream performance.

  3. A typo in L43: “decisions” -> “decision”.

  4. I am just curious if it is possible to extend this framework to generation tasks rather than only MQA tasks.

  5. According to L42, this paper targets datasets with ambiguous visual inputs. I wonder if there is any dataset used in this paper that covers such ambiguous visual inputs. There is a VQA dataset named VizWiz-VQA, which is designed for people with visual impairments. The images in this dataset may be blurry, objects may be occluded, and photos may be taken from poor angles. It can be a testbed for the proposed method to show its effectiveness on ambiguous visual inputs.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Dq1t (1/5)

We thank the reviewer for the thoughtful feedback. We address each concern in turn and attempt to implement the reviewer's suggestions. Our changes to the paper in response to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

1. "Most descriptions are in plain text, making it hard for readers to accurately understand what the authors refer to."

We agree, and we have added formal statements where the paper relied on plain text, including the two assumptions and the joint probability of an entity and its attribute (Section 3.1, L200-213 and L192-193). We describe these changes in our responses to Q1 and Q4.

Q1 (part 1). "Assumptions: ...can you explain these assumptions more explicitly? For example, for A2, I wonder whether it means that attributes are independent of relationships, or all attributes are independent of each other... A mathematical formulation here would be helpful..."

We agree that A1 and A2 were stated too briefly, and we now explain each one in words and state both formally in a new Assumptions paragraph (Section 3.1, L200-213):

Assumptions. Each perception is a separate observation with its own probability, and the network composes them under two assumptions. First, we assume that entities are mutually independent, so the presence of one entity does not change the probability of another (e.g., in Figure 2, knowing that the mat is present does not change the probability that the corgi is present). Second, we assume that attributes and relationships are conditionally independent given the entities. When their entities are present, the probability of one attribute or relationship does not depend on whether any other attribute or relationship holds (e.g., knowing that the corgi is orange does not change the probability that it is on the mat). Formally, for the entity nodes EE, attribute nodes AA, and relationship nodes RR, the assumptions are

A1: P(E)=∏iP(Ei)P(E) = \prod_i P(E_i)

A2: P(A,R∣E)=∏jP(Aj∣Eπ(j))∏kP(Rk∣Eπ1(k),Eπ2(k))P(A, R \mid E) = \prod_j P(A_j \mid E_{\pi(j)}) \prod_k P(R_k \mid E_{\pi_1(k)}, E_{\pi_2(k)})

Since perceptions may be correlated (e.g., multiple perceptions of the same image by the same model), we measure the extent to which these assumptions hold in practice in Section 4.4.

A2 therefore covers all three cases the reviewer mentions: two attributes, two relationships, and an attribute and a relationship.

Q1 (part 2). "...Also, can you justify these assumptions, like why they are practical assumptions?"

PROVE's design aligns with both assumptions. For A1, each entity is a separate detection with its own score, and overlapping boxes are merged into one entity. For A2, each attribute and relationship is verified with its own call, on a single statement and crop (Section 3.2). We also test both assumptions on held-out labeled facts (Section 4.4, Table 7). Under the independence assumption, the probability that two facts in the same image are both true is the product of their calibrated probabilities. Table 7 reports the difference Δ\Delta between how often both facts are actually true and this product. In response to the reviewer's question, we added attribute–relationship pairs as a new row of Table 7, where Δ\Delta is +0.18 ± 0.18 points. We discuss this row and the results for every pair type in Lines 521-524 (Section 4.4):

Attribute–relationship pairs are measured on the synthetic set only, since the natural set takes its attribute and relationship labels from different sets of photographs (Section F.1). In every case, Δ\Delta is within 1.2 points of zero, so the pairwise products of Equation (4) are within about a point of the joint frequencies they stand for.

Since Δ\Delta is small for every pair type, these results suggest that both assumptions hold closely in practice.

AuthorsOpenReview ↗

Response to Reviewer Dq1t (2/5)

Q2. "Method description: I do not follow the sentences in L183-185 about the goal of scene entities. How is the scene entity used in the proposed paradigm?"

We agree that more clarification on the scene entity is needed. Sometimes a query needs a property of the entire scene (e.g., dark, indoors) or relates an entity to the scene itself (e.g., in the foreground, near the center). To keep the framework consistent, we make the scene itself an entity. An attribute can then be placed on the scene entity, and another entity can be related to it, just like any other attribute or relationship. Since the scene always exists, the scene entity has probability 1. Consider the following two examples:

“Is it dark outside?” The attribute dark is placed on the scene entity named img.

q(true) :- ent(img, scene), attr(img, dark).

“Is the dog in the foreground?” The dog is related to the scene entity.

q(true) :- ent(D, dog), ent(img, scene), rel(D, img, in_foreground).

To make this clearer in the paper, the passage the reviewer cites now references further examples of how the scene entity is used in Appendix B (Section 3.1, L180-182).

Q3. "Detector description:"

Q3.1) "...I wonder if these object classes are predefined. Moreover, a broader question is: What needs to be predefined in this paper?"

We use an open-vocabulary object detector, SAM3 (Carion et al., 2026), so the coordinator can pass any object class as free text. We now state this in the description of the detector (Section 3.2, L275-276):

A detect call passes an image and an object class, written as free text, to an open-vocabulary object detector, requiring no set of predefined classes.

We use a similar strategy for the verifier, where the coordinator writes a free-text statement about an attribute or relationship and the verifier returns the probability that it is true.

We fix the set of tools and their models (Section 4), the prompts, which are the same for every benchmark and question type (Appendix C), and the hyperparameters, such as the action budget (Appendix A). The detector settings (Appendix A.1) and the calibration maps (Section 4.4) are set once for synthetic scenes and once for photographs.

Q3.2) "I don’t think one can treat the detector scores as the probability of the existence of an entity. There is a gap between the two..."

We thank the reviewer for pointing out this gap. We agree that a detector’s score is not a probability, and the paper now follows the reviewer’s description of the score (Section 3.2, L276-279):

The detector returns every instance as a bounding box with a score. The score reflects the detector’s confidence that the box localizes an object of the class, and we calibrate it on a held-out set to approximate the probability of this event (Section 4.4).

An entity node represents this same event, the presence of an object of a given class at a specified location (Section 3.1, L176-177). We are motivated by prior closely related work that uses a detector’s confidence as the probability of a detection, such as Vieira (Li et al., 2024), NAVER (Cai et al., 2025), and EVPG (Wan et al., 2025). We extend this idea by calibrating the score to better approximate the probability.

The approximation also works well in practice. On held-out images, the detector’s calibration error drops from 15.15% to 0.67% on synthetic scenes and from 2.47% to 1.49% on photographs (Table 6). Although the calibrated scores are approximations and not exact probabilities, preserving them improves accuracy by up to 6.8 points over DePROVE, which rounds each perception to true or false (Table 2).

AuthorsOpenReview ↗

Response to Reviewer Dq1t (3/5)

Q4. "Verifier description: L266-268 states that the returned probabilities from VLMs are treated as conditional probabilities... we should interpret the returned probabilities as Pr(entity exists AND entity color is orange), rather than Pr(entity color is orange | entity exists)..."

The reviewer is right that the wording at L266-268 of the original submission does not justify treating the verifier’s value as a conditional probability. Although we phrase each statement as if the entity exists to encourage a conditional answer, this phrasing does not automatically make the VLM’s answer conditional. We also recognize that the VLM can be asked about an entity that is not there. There is no inherent guarantee of how it will respond in this case, so we cannot claim that its raw answer is either Pr(entity exists AND entity color is orange) or Pr(entity color is orange | entity exists). We have rewritten the passage to reflect this (Section 3.2, L287-294):

Since the relevant entities must already have been detected to make the call, the statement and crop assume the entity is present (e.g., “This corgi is orange”, “This person is touching this mat”). To obtain the confidence, we read the VLM’s token-level logits and take the maximum, ztruez_{\text{true}}, over the tokens {True, true, TRUE}, and zfalsez_{\text{false}} over {False, false, FALSE}. The raw score is the sigmoid of their margin, σ(ztrue−zfalse)\sigma(z_{\text{true}} - z_{\text{false}}), and reflects the VLM’s confidence in the statement. We calibrate it to approximate the conditional probabilities αj\alpha_j and ρk\rho_k of Equations (2) and (3) (Section 4.4).

Ideally, PROVE captures the uncertainty of each individual perception, not of several perceptions at once. Consider the corgi example in Figure 3. The confidence that the corgi is at its bounding box comes from the detector call (0.98). The query then depends on two other verified facts, that it is orange (0.94) and that it is on the mat (0.95), which come from separate calls to the verifier. If these values were the joint, each would also include the chance that the corgi is at that box. The corgi’s presence at that box would then be counted multiple times instead of once, giving it too much weight in the final answer. This is why we want the verifier’s value to be closer to the conditional, so that each verification only adds the uncertainty of the attribute or relationship and not of the underlying entity.

We use calibration to bring the verifier’s value closer to the conditional, without relying on how the VLM’s raw answer behaves. We fit the verifier’s calibration map only on labeled statements about objects that are present (Appendix F.1), and only the calibrated value enters the network. For a given raw score, the map estimates how often such statements are true, which is Pr(entity color is orange | entity exists). Since the map is fitted to the outcomes of these statements and not to an interpretation of the raw answer, it estimates this conditional whether the raw answer behaves more like the joint or more like the conditional. We now state this in the calibration paragraph (Section 4.4, L484-485):

The verifier maps are fitted only on statements about entities that are present, so they are calibrated toward the conditional probabilities αj\alpha_j and ρk\rho_k.

On held-out statements about present objects, the calibrated value closely matches how often the statement is true. The attribute verifier’s calibration error drops from 5.20% to 0.49% on synthetic scenes and from 5.12% to 2.46% on photographs (Table 6).

The chance that the entity is absent is carried by the detector’s probability, so the joint the reviewer describes, Pr(entity exists AND entity color is orange), follows from the product rule. We have added this joint to the paper (Section 3.1, L192-193):

An attribute therefore holds with probability P(Aj=1)=pπ(j)αjP(A_j = 1) = p_{\pi(j)} \alpha_j, the joint probability that its entity is present and has the attribute.

Here pπ(j)p_{\pi(j)} is the detector’s probability that the entity is there and αj\alpha_j is the verifier’s calibrated value. For the corgi, the joint is 0.98×0.94≈0.920.98 \times 0.94 \approx 0.92, with its presence counted once.

Reviewer_WfbsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PROVE, a training-free neuro-symbolic framework for reasoning over uncertain perception. It uses an LLM coordinator that collects evidence using three tools: a detector, a verifier, and an observer. At the end, it reasons over the facts with a ProbLog program to answer the question with exact inference. They evaluate their method on VQA datasets such as GQA, CLEVR, NLVR2, and COVR, showing improvement over other neuro-symbolic methods and directly querying the verifier VLM.

Critical strengths
  1. The paper provides a detailed analysis of their own method, comparing PROVE and DePROVE with the same evidence and rules. It also breaks down the results by uncertainty and number of facts, making the comparison easier to assess.
  2. While they calibrate their atomic grounders, they evaluate calibration on held-out data and also check the independence assumptions used in their reasoning.
  3. These kinds of methods make facts and generated rules explicit, providing inspectability and observability into how the answer is obtained, and separating perception and reasoning.
Critical weaknesses
  1. The unique novelty of this work is not clear to me. NePTune and NAVER already use symbolic reasoning over uncertain perception, with NAVER using ProbLog and HYDRA collecting evidence for symbolic reasoning iteratively. The paper's main contribution beyond these doesn't seem clear to me. The paper should clarify what is new beyond combining these existing ideas and show which part accounts for the reported gains.

  2. Questions in the benchmarks seem too simple to benefit from symbolic reasoning. On GQA, the gain over direct Qwen3.5-2B is only 0.8 points with no statistical significance. Can you also compare with appropriate baselines such as Qwen3.5-2B and Qwen3-32B (the used coordinator) with CoT or thinking enabled?

  3. Calibration can make a huge difference when comparing with baselines. The backbone models are calibrated for PROVE. In comparison with baselines where their backbones match calibration, its contribution is difficult to separate from the reasoning method.

  4. Runtime, tokens, and tool calls are not reported. Table 9 measures compilation failures, but not whether compiled rules represent the question correctly. The paper should provide more details on the logical correctness of the compiled rules and the sufficiency of grounding evidence.

  5. NAVER is the closest related work similar to PROVE but focuses on grounding rather than VQA. Can you compare them on a shared task to show what PROVE adds? Do the gains carry over to grounding tasks such as RefCOCO or human-written VQA questions such as CLEVR-Humans?

Questions for authors
  1. What is the main contribution beyond NAVER, NePTune, and HYDRA, and which experiment isolates its effect? Why didn't you include HYDRA as a baseline?
  2. Can you compare PROVE with NAVER on a shared task?
  3. Do the gains remain when Vieira, NePTune, and PROVE use the same calibrated scores?
  4. What happens if Qwen3-32B reasons and answers from the same collected facts, with and without uncertainty scores, instead of using ProbLog?
  5. How does PROVE compare with the same coordinator querying the VLM and answering directly? What if you enable thinking on Qwen3.5-2B?
  6. What are the runtime, token usage, and tool calls per question?
11 comments
AuthorsOpenReview ↗

Response to Reviewer Wfbs (1/7)

We thank the reviewer for the thoughtful feedback. We address each concern in turn and attempt to implement the reviewer's suggestions. Our changes to the paper in response to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

AuthorsOpenReview ↗

Response to Reviewer Wfbs (2/7)

1. "The unique novelty of this work is not clear to me."

1.1) "NePTune and NAVER already use symbolic reasoning..."

Q1 (part 1). "What is the main contribution beyond NAVER, NePTune, and HYDRA..."

We agree that PROVE shares ProbLog with NAVER and iterative evidence collection with HYDRA. Our novelty is simultaneously collecting evidence adaptively while keeping every perception uncertain. Prior methods do one or the other, each at a cost, and the two strategies are conflicting. In Lines 44-48 (Section 1), the introduction now states both approaches and their downsides:

Methods that resolve a query by executing a program (Gupta & Kembhavi, 2023; Surís et al., 2023; Ge et al., 2024; Ke et al., 2024) make a hard decision on the result of each perception, discarding its uncertainty, while methods that keep each perception uncertain (Li et al., 2024; Cai et al., 2025; Kamali & Kordjamshidi, 2026) have no such decision to act on, so they fix their evidence collection strategy in advance, missing any evidence it did not anticipate.

Prior methods that keep perceptions uncertain take what to perceive from the query or from a hand-written program. NAVER extracts the object categories from the query and an image caption, detects them, and scores the attributes and relations named in its rule over the detected entities. This is suitable for grounding, where the objects to perceive are named in the query. NePTune writes its program and the object names to detect from the question alone, and scores each predicate over every detected object. Vieira relies on a query language that is hand-written for each dataset, and exhaustively builds a scene graph on CLEVR.

A fixed strategy is costly in VQA, where the answer is often something the question never names (e.g., the mat in Figure 3). This is the case for at least 239 of our 500 GQA questions (Appendix A.6). For example, NePTune’s programs operate only on the entities read off the question, and it only scores 31.8% on GQA (Section 4.1). Collecting everything is an option on CLEVR, whose vocabulary is closed, but is not feasible on real photographs, where the objects, attributes, and relationships are open-ended.

Prior methods that collect evidence adaptively make hard decisions and discard perception uncertainty. In Lines 118-120 (Section 2), we now name HYDRA among these methods:

A second line has an LLM coordinator choose each step from the text of previous tool outputs (You et al., 2023; Hu et al., 2023), or trains one to do so, as in HYDRA (Ke et al., 2024) and MATA (Cai et al., 2026). All resolve the query by executing the program, making a hard decision on each perception.

Borderline perceptions are rounded to true or false, and a wrong rounding can change an answer. Of the 2,460 questions in Table 3, 460 depend on at least one fact within 0.1 of the threshold, and on these questions rounding costs 8.3 points.

Doing both is difficult, because choosing the next step from a result means conditioning on the result. PROVE resolves this by separating collection from the decision. The coordinator sees which entities were detected and which statements were verified, but never a probability or the truth of a statement (Section 3.2). It can therefore choose its next step from its history, but it has no result on which to discard a candidate. Instead, it collects the evidence for each candidate, and each perception enters exact inference with its probability (Section 3.3). In Figure 3, a method that acted on each result would commit to the mat or the pool after one check, while PROVE verifies both and inference returns 0.73 and 0.18.

AuthorsOpenReview ↗

Response to Reviewer Wfbs (3/7)

1.2) "...and show which part accounts for the reported gains."

Q1 (part 2). "...and which experiment isolates its effect?..."

Every method in Table 1 uses the same detector, VLM, LLM, and calibrated scores, so the differences come from the methods and not their components. This lets us measure our contribution directly. We compare against prior methods that preserve uncertainty but fix their evidence collection, and against prior methods that collect evidence adaptively but make hard decisions. In Lines 393-398 (Section 4.1), the discussion of Table 1 now reads:

The baselines cover both sides of the trade-off in Section 1. VisProg, ViperGPT, RVP, and HYDRA make a hard decision on each perception. Like PROVE, HYDRA collects evidence adaptively, choosing each step from the results of the previous ones. PROVE outperforms HYDRA by 22.6 points on GQA, 47.4 on CLEVR, and 24.6 on CLEVR-Humans (Section E). Vieira and NePTune keep perceptions uncertain but fix what evidence to collect when their program is written. PROVE outperforms Vieira and NePTune by at least 13.4 points on GQA, 21.2 on CLEVR, and 23.0 on CLEVR-Humans.

To isolate the effect of preserving uncertainty, we also compare against DePROVE, which uses the same evidence, rules, and calibration as PROVE and only rounds each perception to true or false. PROVE gains 6.8 points on GQA, 2.6 on CLEVR, 2.8 on CLEVR-Humans, and 6.8 on COVR (Table 2).

Q1 (part 3). "...Why didn't you include HYDRA as a baseline?"

We thank the reviewer for suggesting HYDRA as a baseline. We cite HYDRA in Section 2 but did not run it, because its controller is a trained RL agent and Table 1 compares training-free methods. HYDRA also releases a variant without the RL agent, which is training-free, so we now include this variant in our main results (Table 1; Appendix E). In Line 378 (Section 4), we added:

For HYDRA, we use its training-free variant, which runs without the RL agent.

MethodGQACLEVRCLEVR-Humans
HYDRA39.648.657.0
PROVE62.296.081.6

HYDRA takes a single image, so it is evaluated on GQA, CLEVR, and CLEVR-Humans. With the same detector, VLM, and LLM, PROVE is ahead by 22.6 points on GQA, 47.4 on CLEVR, and 24.6 on CLEVR-Humans.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.