acceptodds
Under review as a conference paper at ICLR 2027

Truth in Focus: Scene Graph Grounding for Faithful Visual Question Answering

Abstract

Recent advances in large vision–language models (VLMs) have substantially improved Visual Question Answering. Nevertheless, reliable answer generation remains a persistent challenge, as VLMs may produce plausible responses that are insufficiently grounded in the visual content. Scene graphs provide a structured representation of visual content by organizing object entities and their relationships. However, a complete scene graph often contains information beyond what a given question requires, which can distract the model from relevant evidence and lead to factually unsupported answers. We therefore shift from global scene modeling to question-conditioned local grounding and propose a scene graph grounding framework that extracts a compact subgraph containing the visual information needed to answer each question. Specifically, we introduce a dual-differential causal attention module that jointly models node–node dependencies and node–query associations, enabling the relative contribution of graph elements to be assessed for subgraph prediction. To mitigate the scarcity of fine-grained grounding supervision, we further construct a factual–counterfactual dataset by leveraging the generative capabilities of large models. Extensive experiments on multiple benchmarks demonstrate that our method achieves superior performance against state-of-the-art approaches.

Primary area:
causal reasoning
Keywords:
Scene graph, visual question answering, causal modeling, visual grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9QqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a scene graph grounding framework for improving the faithfulness and accuracy of visual question answering. The method introduces Dual-Differential Causal Attention (DDCA) to identify question-relevant evidence by comparing node–node associations and question–node relevance against reference scores, aiming to mitigate structural and linguistic biases. It also employs factual–counterfactual supervision, where an LLM generates incorrect alternative answers and extracts corresponding evidence subgraphs to guide node selection. The selected subgraph is then provided to a frozen VLM alongside the original image and question. Experiments across multiple VQA benchmarks and VLMs demonstrate improvements in answer accuracy and evidence selection quality over existing baselines.

Critical strengths

1.The paper addresses an important problem in visual question answering by selecting relevant evidence from scene graphs to reduce distracting information. The proposed framework is well motivated and can be integrated with different VLMs without modifying their parameters.

  1. The experimental evaluation is relatively comprehensive, covering multiple datasets and VLMs. The improvements in answer accuracy and evidence selection, together with the ablation and robustness studies, provide encouraging evidence for the effectiveness of the overall approach.
Critical weaknesses
  1. Counterfactual supervision. The construction of counterfactual evidence is not sufficiently clear. Since the question and scene graph remain unchanged, it is unclear how a subgraph can genuinely support an incorrect answer. This raises concerns about the validity of the counterfactual evidence and the resulting supervision.

  2. Causal interpretation. The theoretical analysis of DDCA assumes that structural and linguistic biases remain unchanged under interventions, but this assumption is not adequately justified. Moreover, the differential scores may reflect sensitivity to contextual changes rather than the causal importance of evidence.

3.The training objective focuses on selecting nodes according to LLM-generated supervision rather than directly optimizing answer faithfulness. Since the original image remains available during inference, the current experiments do not clearly establish whether the VLM actually relies on the selected subgraph.

4.The paper lacks comparisons with simpler supervised node selectors trained under the same conditions. In addition, the robustness experiments do not sufficiently cover challenging distractors involving similar objects and spatial relationships, and the reported results lack systematic statistical uncertainty estimates.

Questions for authors

1.How can the counterfactual subgraph provide valid supporting evidence for an incorrect answer when the original question and scene graph remain unchanged? What proportion of the generated counterfactual subgraphs are empty or invalid?

2.What justifies the assumption that structural and linguistic biases remain invariant under the proposed interventions? How can the authors distinguish genuine causal contributions from general sensitivity to contextual changes?

3.Have the authors considered evidence removal or replacement experiments to verify whether the downstream VLM actually relies on the selected subgraph when generating its answers?

4.Could the authors provide comparisons with simpler supervised selectors, such as MLPs or standard graph attention models using the same supervision? Additionally, how does the method perform with more challenging same-category distractors, and how stable are the results across random seeds?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 9Qqz (Part 1/2: W1/Q1 - W3/Q3)

W1 / Q1. Construction and validity of the counterfactual subgraphs, and proportion of empty cases.

As shown in Appendix A.4, we construct the counterfactual subgraph S−S^- by extracting the nodes and relations of an existing distractor object A−A^- of the same semantic type, which directly serves as local evidence for the incorrect answer.

Table 10 confirms that S−S^- is valid in 87.0% of cases. Furthermore, Table A demonstrates that the proportion of empty S−S^- remains consistently low across all evaluated datasets.

Table A: Proportion of empty counterfactual subgraphs (S−S^-) across evaluated datasets.

GQACLEVRVCRCOCO-QAVQA-Abstract
Empty S−S^- (%)2.230.036.172.149.40

Crucially, our counterfactual supervision remains robust even in rare cases where S−S^- is noisy or empty. As shown in Appendix B.6, the factual subgraph S+S^+ determines the gradient direction by assigning positive weights to its nodes and negative weights to all others. S−S^- only scales the weight magnitudes. Consequently, our strategy avoids providing a misleading supervision signal whether S−S^- is noisy or empty.


W2 / Q2. Invariance of the biases under intervention and distinction from contextual sensitivity.

1. Invariance of Biases under Intervention

Structural bias (BstructB_{\text{struct}}) is the higher score assignment to high-degree nodes, depending solely on the target node's degree. Linguistic bias (BlingB_{\text{ling}}) is the higher score assignment to common answers, driven by the prior co-occurrence between the query and target entity. We hold viv_i fixed in Eq. (3) and (Q,vi)(Q, v_i) fixed in Eq. (4) during interventions. Therefore, both biases remain invariant.

2. Distinction from Contextual Sensitivity

We distinguish causal effect from general contextual sensitivity through a reference-shuffling experiment in Table 12 of Appendix B.7. Specifically, we randomly shuffle the reference expectations across nodes, substituting each node's correct reference with that of another node. This mismatch lowers the F1 score to 63.2%, falling below the 64.2% baseline obtained without using any reference terms. This degradation precludes general contextual sensitivity, demonstrating that our differential mechanism preserves the causal effect.


W3 / Q3. Training objective and reliance on the subgraph.

1. Rationale for Offline Training Supervision

Directly optimizing answer faithfulness requires querying the VLM at every training step, which leads to a high computational cost. In contrast, our offline supervision generates the target subgraphs only once, allowing the selector to be trained without any VLM calls and work with any frozen VLM.

2. Evidence Removal and Replacement

We conducted evidence removal experiments to verify that the VLM relies on the selected subgraph GS\mathcal{G}_S rather than the original image alone. As shown in Table B, removing GS\mathcal{G}_S from the full scene graph G\mathcal{G} drops GQA accuracy from 85.9% to 52.2%, falling below the 61.2% image-only baseline. Moreover, providing GS\mathcal{G}_S without the original image still achieves 80.2% accuracy, surpassing the image-only baseline by 19.0%.

We also replaced internal relations and attributes within GS\mathcal{G}_S. Reversing spatial relations within GS\mathcal{G}_S lowers accuracy from 85.9% to 81.6%, and modifying answer-relevant attributes within GS\mathcal{G}_S drops accuracy to 57.9%. These removal and replacement results confirm that the VLM relies on the selected subgraph when generating answers.

Table B: Evidence removal and replacement experiments on the selected subgraph.

Image InputScene Graph InputAccuracy (%)
✓✗61.2
✓GS\mathcal{G}_S85.9
✓G∖GS\mathcal{G} \setminus \mathcal{G}_S52.2
✗GS\mathcal{G}_S80.2
✓GS\mathcal{G}_S w/ reversed spatial relations81.6
✓GS\mathcal{G}_S w/ modified attributes57.9
AuthorsOpenReview ↗

Reply to Reviewer 9Qqz (Part 2/2: W4/Q4)

W4 / Q4. Simpler selectors, difficult distractors, and random seeds.

1. Comparison with Simpler Selectors

In Table 2, removing DDCA leaves a simple 3-layer GAT selector, which drops GQA accuracy to 84.4%. We also evaluate a 3-layer MLP selector without DDCA, which achieves 83.2%. Under the same factual-counterfactual supervision, our full selector reaches 85.9%, confirming the effectiveness of DDCA.

2. Robustness to Difficult Distractors

On GQA, our method remains robust to same-category and spatial distractors. To evaluate same-category distractors, we insert up to three duplicates of an evidence object into the scene graph. Each duplicates keeps the original object name but modifies an answer-relevant attribute. As shown in Table C, baseline accuracies drop noticeably under these distractors, while ours drops by less than 0.7%. For spatial distractors, Appendix B.3 shows that reversing all spatial relations leaves our method at 80.0%, while baselines fall to 72.4%–76.7%.

Table C: Performance comparison (%) under same-category distractors on GQA.

MethodOriginal+1 distractor+2 distractors+3 distractors
Full SG81.979.279.780.4
EMRAG83.079.276.977.2
PCST78.462.259.559.5
SubgraphRAG83.781.681.480.1
Ours85.985.685.285.4

3. Stability Across Random Seeds

To verify stability against randomness, we conducted experiments on COCO-QA using Qwen3.5 4B, which yields the smallest gain in Table 1. Across two additional random seeds, our model achieves 66.17% and 66.12%, consistently outperforming all baselines.

The authors have provided helpful explanations and additional experiments. However, some concerns about the causal interpretation remain unresolved. In particular, the justification for bias invariance is still insufficient, and it remains unclear whether the differential scores capture genuine causal contributions or simply reflect sensitivity to contextual changes. Therefore, I maintain my original score.

Thank you for your thoughtful follow-up. We address your remaining concerns regarding bias invariance and causal interpretation in detail below.

1. How Linguistic and Structural Biases Arise

During training, scene-graph-based VQA models frequently learn dataset-level statistical priors, including linguistic and structural biases.

Linguistic bias stems from the frequent co-occurrence of specific questions and answers in the training data. For instance, if "yellow" frequently answers "What color is the banana on the plate?", the model learns to rely on this statistical shortcut from the question alone. Consider a specific scene with a yellow banana on the table and a green banana on the plate. Driven by this learned prior, the model still assigns an inflated score to the "yellow" node and ignores the spatial constraint "on the plate".

Structural bias arises from imbalanced node connectivity in scene graphs. In a visual scene, central nodes like "dining table" or "person" often connect to many surrounding objects, resulting in high node degrees. Because these high degree nodes frequently appear along reasoning paths during training, the model relies on node degree as a shortcut. Consequently, it assigns inflated scores to these high degree nodes even when they act as distractors rather than true evidence.

Ultimately, both biases reflect scene-independent statistical correlations rather than causality grounded in the specific visual context.

2. Why Statistical Biases Remain Unchanged Under Interventions

Once trained, the model stores these dataset-level priors in its fixed parameters. Because these priors reflect general statistics across diverse training scenes, they act as direct shortcuts driven by local elements rather than specific scene context. Under the principle of independent causal mechanisms in causal inference [1], intervening on the surrounding context leaves independent local mechanisms invariant. When we preserve these local elements and modify only the surrounding context during intervention, the resulting bias contributions remain unchanged:

  • In the Query-Node branch (Eq. 4): We hold the question QQ and candidate node viv_i fixed while removing other background nodes to construct reference graphs G~\tilde{\mathcal{G}}. Although removing background nodes changes the surrounding scene context of viv_i, the linguistic prior is driven by (Q,vi)(Q, v_i) rather than the removed background nodes, and therefore remains unchanged between G\mathcal{G} and G~\tilde{\mathcal{G}}.
  • In the Node-Node branch (Eq. 3): We hold the target node vjv_j fixed and replace the source node viv_i with another node vkv_k, without modifying the graph structure G\mathcal{G}. Because neither vjv_j nor its node degree in G\mathcal{G} changes, its degree-based structural bias remains unchanged.

3. Bias Cancellation via Differential Scoring

Raw scores on the intact scene graph capture statistical correlation rather than causation, combining the causal effect with the biases above. To remove these biases, we introduce the differential formulation in Eq. 2: Δ(x;pref)=E[Y∣do(X=x)]−Ex~∼pref[E[Y∣do(X=x~)]].\Delta(x; p_{\text{ref}}) = \mathbb{E}[Y \mid do(X=x)] - \mathbb{E}_{\tilde{x} \sim p_{\text{ref}}}[\mathbb{E}[Y \mid do(X=\tilde{x})]]. Here, the first term evaluates the intact scene context, capturing both the causal effect and the bias. The second term holds the local elements fixed and marginalizes out surrounding context by taking the expectation over interventions, isolating the invariant bias. Under the additivity assumption analyzed in Appendix D, this subtraction directly cancels out the shared bias component.

4. Empirical Verification

In Appendix B.9 (Table 16), we construct biased datasets from GQA and CLEVR to evaluate whether DDCA mitigates structural and linguistic biases. With DDCA, the F1 drop decreases from 18.7% to 13.7% on higher degree distractors and from 11.9% to 6.4% on rare answers. This confirms that our differential formulation effectively mitigates both biases.

Furthermore, Appendix B.8 (Table 15) validates the role of the intervention expectation terms on GQA. Removing these terms or replacing them with global graph averages yields 64.2% and 64.1% F1, respectively. In contrast, DDCA reaches 65.0% F1. Crucially, randomly permuting the expectation terms across nodes within the same graph drops F1 to 63.2%. This indicates that the performance gains do not arise simply from sensitivity to contextual changes, but depend on contrasting each node against its own intervention baseline.

We have revised Section 3.2 in the updated manuscript to present exact cancellation under additive and invariant bias assumptions as the analytical basis for our design, while clarifying that in practical nonlinear networks DDCA acts as an empirical approximation to mitigate confounding biases.

References

[1] Schölkopf et al., Toward causal representation learning, Proceedings of the IEEE, 2021.

Reviewer_5ByhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the issue in Visual Question Answering (VQA) where full scene graphs contain substantial irrelevant information that may distract the answering model, this paper proposes a question-conditioned subgraph selection method to extract key visual evidence. Specifically, the authors design a Dual-Differential Causal Attention (DDCA) module, combined with iterative node selection and LLM-generated factual-counterfactual supervision, to extract a question-relevant subgraph before feeding it into a Vision-Language Model (VLM) for final answer generation. Experiments demonstrate that the proposed method achieves strong performance across multiple benchmarks and VLM backbones.

Critical strengths

1.The research problem is practical and well-motivated. Improving VQA faithfulness by filtering out distracting context from scene graphs is a sensible and effective direction.

2.The empirical evaluation is comprehensive, covering multiple datasets and VLM backbones, supplemented by ablation studies, noise robustness tests, and subgraph quality analyses.

Critical weaknesses

W1. Limited methodological novelty and missing discussion of prior work.Both subgraph retrieval for LLMs and causal differential debiasing have been extensively studied in prior literature, such as G-Retriever, SubgraphRAG, CF-VQA (CVPR 2021), and Unbiased Scene Graph Generation (CVPR 2020). This paper primarily combines these established ideas for scene graph selection in VQA. While the combination yields solid empirical results, the authors do not sufficiently clarify how DDCA fundamentally differs from existing causal debiasing formulations (e.g., Total Direct Effect), and the Related Work section omits discussions of these foundational visual/graph causal inference studies.

W2. Insufficient implementation details for core scoring functions and overly strong theoretical assumptions. The exact network architectures of the two scoring functions in Eqs. (3) and (4) are not specified. In particular, for Eq. (4), it remains unclear how the global scene graph G\mathcal{G} conditions the node scoring function gθ(Q,G,vi)g_\theta(Q, \mathcal{G}, v_i) and why randomly deleting other nodes alters the alignment score of target node viv_i (especially since node and query features are encoded independently by a frozen Sentence-BERT). Furthermore, the theoretical justification in Appendix D relies heavily on a strict additive decomposition assumption of the bias terms. This only shows algebraic cancellation under idealized linear separability and is insufficient to prove that causal confounding is genuinely eliminated in nonlinear neural networks.

Questions for authors

1.Could the authors elaborate on the fundamental theoretical and technical differences between DDCA and prior causal debiasing methods in VQA and scene graphs, such as CF-VQA and Unbiased Scene Graph Generation?

2.How are the scoring functions fθf_\theta and gθg_\theta in Eqs. (3) and (4) specifically parameterized and implemented? After randomly deleting nodes to form G~\tilde{\mathcal{G}}, how does gθg_\theta incorporate the modified graph structure to compute a different score for node viv_i?

3.Where do the ground-truth evidence subgraphs used in Table 6 (and other analyses) come from? Are they derived from GQA's native functional programs/annotations, or are they generated by the same LLM pipeline used for training supervision?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 5Byh

W1 / Q1. Methodological novelty and comparison with prior work.

Existing subgraph retrievers such as G-Retriever and SubgraphRAG capture statistical correlations rather than causality, often retrieving distracting nodes driven by spurious associations. Meanwhile, prior causal debiasing methods such as CF-VQA and Unbiased SGG face two fundamental limitations when directly applied to subgraph retrieval. First, they apply a single global counterfactual to debias final outputs, yielding debiased answers or relations rather than node-level scores. Second, they require access to internal features or output logits, thus cannot be applied to black-box API models. To this end, we design DDCA to perform node-level causal interventions to compute debiased scores. Based on these scores, we select a compact evidence subgraph for the VLM to ensure faithful reasoning, while reducing token usage and accelerating inference.


W2 / Q2. Parameterization of scoring functions and the additive assumption.

1. Implementation details of scoring functions

The scoring functions are implemented as gθ(Q,G,vi)=MLP([hi;hˉ;q])g_\theta(Q,\mathcal{G},v_i)=\mathrm{MLP}([h_i;\bar{h};q]) and fθ(vi,vj,eij)=MLP([hi;hj;eij])f_\theta(v_i,v_j,e_{ij})=\mathrm{MLP}([h_i;h_j;e_{ij}]), where both MLPs have two layers with a hidden dimension of 512. Here, hˉ\bar{h} is the mean of all node representations in the global scene graph G\mathcal{G}. This design ensures that node deletions change the global context hˉ\bar{h}, dynamically propagating the intervention effect to the final score gθg_\theta of any target node viv_i. We have clarified these details in Appendix A.3 of the revised manuscript.

2. Validity of the additive assumption

The additive assumption does not require the underlying MLPs to be linear. It applies solely to their final scalar outputs. We decompose this final score into a causal effect, a systematic bias, and residual noise. This decomposition is justified because structural and linguistic shortcuts typically act as systematic shifts on the prediction, which naturally aligns with classical measurement models (y=h(x)+b+vy = h(x) + b + v).

To verify this formulation, we conducted experiments under skewed training distributions in Appendix B.9. Table 16 demonstrates that this differential mechanism effectively mitigates both structural and linguistic biases, confirming that this formulation effectively mitigates both structural and linguistic biases


Q3. Source of the ground-truth evidence subgraphs.

The ground-truth evidence in Table 6 and all other evidence-level analyses comes from the official GQA and CLEVR functional programs that record the objects and relations required for the answer.

Reviewer_5ByhOpenReview ↗

Final Assessment After Rebuttal

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns.

The clarification of the scoring functions and their implementation addresses my concerns about reproducibility. The explanation that ground-truth evidence subgraphs are derived from the official functional programs of GQA and CLEVR also resolves my concern about the evaluation references. Furthermore, the additional comparisons with SubgraphRAG under matched counterfactual supervision provide stronger evidence that the improvements come from the proposed method rather than additional supervision alone.

I still have two remaining concerns:

  1. Causal interpretation. The theoretical justification relies on the assumption that structural and linguistic biases are additive and remain unchanged under interventions. While the additional experiments support the effectiveness of DDCA in mitigating biases, they do not necessarily establish that the differential scores isolate genuine causal effects. I appreciate the authors' clarification that the mechanism should be understood as an empirical approximation, and I encourage them to maintain this distinction throughout the paper.

  2. Methodological novelty. The authors have clarified how DDCA differs from previous methods through node-level scoring and compatibility with black-box VLMs. These are meaningful distinctions. However, the underlying differential scoring principle remains closely related to existing causal debiasing approaches. I encourage the authors to provide a more balanced discussion of these connections and avoid overstating the novelty of the causal formulation.

Overall, I believe the rebuttal has substantially strengthened the paper. Although some concerns remain, the additional evidence and clarifications have addressed my main questions about the method and evaluation.

I am therefore willing to raise my rating from 2 (Weak Reject) to 3 (Weak Accept).

Thank you again for the thoughtful responses and constructive discussion.

Reviewer_oyXyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a question-conditioned scene graph grounding method for visual question answering. Instead of providing the answering model with the full scene graph, it iteratively selects a compact subgraph containing relevant objects and relations. The proposed dual-differential causal attention module combines node-node association scores and query-node relevance scores, contrasting them with reference scores obtained through node replacement and graph perturbations. The selector is trained using LLM-generated factual and incorrect-answer subgraphs. Experiments cover five VQA benchmarks and four answering models, with both ground-truth and predicted scene graphs. The paper reports consistent accuracy improvements, including an increase from 81.9% to 85.9% on GQA with ground-truth graphs and Gemini 2.5 Flash Lite, alongside evidence-selection and object-hallucination evaluations.

Critical strengths
  • The paper is well-written and easy to follow. The motivation is clear: identifying the evidence needed for a question is a reasonable way to reduce interference from irrelevant scene information.

  • The evaluation across five VQA benchmarks and four answering models is a strong point. Including predicted scene graphs alongside annotated graphs helps establish that the method can provide benefits when the graph itself is imperfect.

  • The component ablations help establish the usefulness of the proposed design. The additional evaluation on non-visual graph benchmarks also broadens the empirical scope beyond scene graphs.

Critical weaknesses
  1. One of my main concerns is that the interpretation of the differential scores as removing structural and linguistic biases is stronger than what the construction itself establishes. Equations (3) and (4) measure changes in learned scores relative to selected reference distributions. This is a reasonable way to study the effect of perturbations on the model, but it does not by itself establish that the subtracted component represents bias rather than useful information. In particular, deleting other nodes can remove genuine relational evidence needed to establish the target node's relevance. The ablations support the usefulness of the differential operations, but the main text should more clearly explain the assumptions connecting these operations to the claimed debiasing mechanism.

  2. My second main concern is about the novelty of the work: the novelty and attribution of the gains should be more clearly established. Question-guided graph reasoning is already represented in the related work, including GraphVQA, so the distinction cannot simply be global graph modeling versus question-conditioned reasoning. The more specific contributions appear to be the differential scoring mechanism and the supervision strategy. The ablations in Tables 2 and 3 provide useful evidence under matched supervision: removing DDCA or its differential operations while retaining counterfactual supervision reduces accuracy from 85.9% to 84.4%. However, it remains unclear whether the external learned baselines receive equivalent factual-counterfactual supervision and training budgets.

  3. The meaning of the counterfactual evidence subgraph is not sufficiently clear. Section 3.3 keeps the question and scene graph fixed, chooses an incorrect answer, and asks an LLM to extract nodes supporting that answer. If the answer is genuinely incorrect, there should not be valid evidence supporting it under all of the question's constraints. The intended output may instead be evidence for a plausible distractor or an incorrect reasoning path, but these are different notions. This distinction matters because these node sets determine which objects receive stronger negative weights, and some objects involved in an incorrect reasoning path may also be necessary for correctly answering the question.

  4. The evidence for faithful answer generation is narrower than the framing suggests. Higher VQA accuracy and better overlap with reference evidence nodes are useful results, but they do not establish that the answering model actually relies on the selected evidence. The model still receives the original image, and there is no explicit constraint requiring its answer to follow from the subgraph. The POPE evaluation provides additional evidence about object-presence errors, but does not directly test faithfulness to attributes, relations, or the reasoning chain. I think the paper should distinguish improved answer accuracy and evidence retrieval from demonstrated use of that evidence.

  5. The practical contribution with predicted graphs needs stronger interpretation. The large gain from adding ground-truth graphs on GQA (from 61.2% to 81.9%) shows how much the pipeline benefits from structured information before the proposed selection step. Predicted-graph results are therefore particularly important, but several improvements over Full SG are small, such as 0.5 percentage points on GQA for GPT-4o and Qwen3.5 4B, and 0.1 points on COCO-QA for Qwen3.5 4B. Table 1 does not include image-and-question-only results for these settings or uncertainty estimates. This makes it harder to assess how reliably the complete pipeline improves visual answering when accurate scene graphs are not already available.

Questions for authors

I would appreciate it if the authors could elaborate on the weaknesses above.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer oyXy (Part 1/2: W1 - W2)

W1. Theoretical interpretation of differential scores and preservation of relational evidence.

As shown in Appendix D and Appendix B, we introduce differential scoring to eliminate both structural and linguistic biases.

In our framework, we model each matching score S as the sum of three components, including a true causal effect S∗S^*, and additive bias term BB, and zero-mean residual noise ϵ\epsilon:

S=S∗+B+ϵS = S^* + B + \epsilon

The additive bias BB can further decomposes into structural bias and linguistic bias:

  • Structural bias (BstructB_{\text{struct}}): Higher score assignment to high-degree nodes, depending solely on the target node's degree.
  • Linguistic bias (BlingB_{\text{ling}}): Higher score assignment to common answers, driven by the prior co-occurrence between the query and target entity.

Interestingly, we observe that this decomposition aligns with classical measurement models in estimation and control (y=h(x)+b+vy = h(x) + b + v). In these models, an observed signal is corrupted by a systematic bias bb and zero-mean noise vv. This alignment provides a principled basis for our additive assumption, treating structural and linguistic shortcuts as systematic shifts on the raw matching scores.

1. Bias Cancellation under Intervention

We fix the elements that the bias depends on in our interventions, so that the reference baseline Sˉ\bar{S} shares the exact same systematic bias BB as the observed score SS, and subtracting Sˉ\bar{S} cancels out BB directly: ΔS=S−Sˉ=(S∗−Sˉ∗)+(ϵ−ϵˉ)\Delta S = S - \bar{S} = (S^* - \bar{S}^*) + (\epsilon - \bar{\epsilon}) Specifically, we hold vjv_j fixed in Eq. (3) and hold (Q,vi)(Q, v_i) fixed in Eq. (4), which eliminates BstructB_{\text{struct}} and BlingB_{\text{ling}} in Eqs. (13) and (17) and isolates the causal effect. As shown in Table 13 of Appendix B.8, this differential mechanism effectively mitigates both structural and linguistic biases under skewed training distributions. We will add this analysis to the main text in the revision.

2. Preserving Relational Evidence during Node Deletion

We note that deleting other nodes does not remove the relational evidence needed to evaluate viv_i, because node deletion is used only to construct the reference graphs G~\tilde{\mathcal{G}} for bias estimation. The observed score SS is directly computed on the intact full scene graph, retaining all relational connections from other nodes. By subtracting the baseline Sˉ\bar{S} from the full-graph score SS, the final differential score preserves the causal effect. In Table 3, we incorporate this query-node differential term to improve GQA accuracy from 84.7% to 85.9%, confirming that useful relational evidence is preserved.


W2. Novelty and attribution of the gains.

1. Distinction from Existing Question-Guided Graph Reasoning

Existing question-guided graph reasoning methods capture statistical correlation rather than causality, often retrieving distracting nodes driven by spurious associations. In contrast, our method advances beyond this paradigm through two key designs. First, DDCA formulates subgraph selection from a causal intervention perspective, explicitly disentangling causal effects from structural and linguistic shortcuts via differential scoring. Second, our factual-counterfactual pipeline provides fine-grained supervision to train the selector to identify the key nodes that determine the answer. We will clarify these two contributions in the revision.

2. Analysis of Performance Attribution

Moreover, the performance gains stem from our selector design rather than extra supervision alone. In Table 1, SubgraphRAG is trained with the same data and budget as our selector, but without counterfactual supervision. For a fair comparison, we further train SubgraphRAG with our counterfactual supervision on GQA using Gemini 2.5 Flash Lite. As shown in Table A, our method consistently outperforms SubgraphRAG on ground-truth and predicted scene graphs both with and without counterfactual supervision, demonstrating the effectiveness of our selector.

Table A: Performance comparison (%) with and without counterfactual supervision on GQA.

MethodGT SGPredicted SG
Full SG81.972.4
SubgraphRAG83.772.7
SubgraphRAG w/ counterfactual supervision84.173.3
Ours w/o counterfactual supervision85.275.2
Ours85.976.1
AuthorsOpenReview ↗

Reply to Reviewer oyXy (Part 2/2: W3 - W5)

W3. Meaning of the counterfactual subgraph.

The counterfactual subgraph S−\mathcal{S}^- represents the evidence supporting a plausible distractor already present in the scene graph. As detailed in Appendix B.6, the factual subgraph S+\mathcal{S}^+ determines the sign of supervision, and the counterfactual subgraph S−\mathcal{S}^- only modulates the weight magnitude. Therefore, any node in S+\mathcal{S}^+ still receives a positive weight of +1+1 even if it also appears in S−\mathcal{S}^-, ensuring that supporting evidence is never penalized.


W4. Evidence for faithful answer generation.

We verified that the VLM relies on the selected subgraph GS\mathcal{G}_S rather than the original image alone. In Appendix B.4, removing GS\mathcal{G}_S from the full scene graph G\mathcal{G} drops GQA accuracy from 85.9% to 54.5%, falling below the 61.2% image-only baseline. Moreover, providing GS\mathcal{G}_S without the original image still achieves 80.2% accuracy, surpassing the image-only baseline by 19.0% and confirming that GS\mathcal{G}_S alone preserves sufficient reasoning evidence.

We further evaluated faithfulness to relations and attributes within GS\mathcal{G}_S. In Appendix B.3, reversing spatial relations reduced accuracy from 85.9% to 80.0%, and modifying answer-relevant attributes within GS\mathcal{G}_S drops accuracy to 57.9%.

Table B: Ablation study on the selected subgraph.

Image InputScene Graph InputAccuracy (%)
✓✗61.2
✓GS\mathcal{G}_S85.9
✓G∖GS\mathcal{G} \setminus \mathcal{G}_S54.5
✗GS\mathcal{G}_S80.2
✓GS\mathcal{G}_S w/ reversed spatial relations80.0
✓GS\mathcal{G}_S w/ modified attributes57.9

W5. Practical contribution with predicted scene graphs.

As shown in Table 1, our method outperforms the Full SG baseline across all 28 settings on predicted graphs, confirming its effectiveness with noisy inputs.

We conducted McNemar's test to provide uncertainty estimates for these small improvements, confirming statistically significant improvements on GQA with p-values of 0.0024 for GPT-4o and 0.0073 for Qwen3.5 4B. The gains on COCO-QA and VQA-Abstract are smaller because their simpler scenes rarely require complex reasoning. For COCO-QA using Qwen3.5 4B with a p-value of 0.12, we further evaluated additional random seeds, yielding 66.17% and 66.12% to consistently outperform the 66.0% baseline. As shown in Appendix B.5, our method achieves these stable improvements while requiring 53.1% fewer average input tokens and 25.0% lower end-to-end latency, validating the practical contribution of subgraph grounding.

Table C evaluates the Gemini 2.5 Flash Lite baseline using only the image and question. Predicted scene graphs strictly outperform this baseline on all datasets, confirming that structured information remains beneficial despite inherent noise. Moreover, our subgraph grounding further enhances this advantage.

Table C: Performance comparison (%) between the Gemini 2.5 Flash Lite image-only baseline and predicted scene graphs.

InputGQACLEVRVCR Q-AVCR QA-RVCR Q-ARCOCO-QAVQA-Abstract
Image and question only61.255.454.957.832.567.679.1
Full SG72.460.856.158.533.369.383.1
Ours76.161.957.659.134.270.684.1

Thank you for the responses and additional analyses. The comparison with SubgraphRAG under the same counterfactual supervision, the clarification of the supervision labels, and the image-only comparisons address several of my concerns.

My main remaining concern is the causal interpretation. The cancellation argument assumes that bias is additive and unchanged under the reference interventions. The experiments under biased training distributions are useful, but do not establish these assumptions or show that the differential scores isolate causal effects. Could the authors qualify these claims and make the limitations explicit?

I would also appreciate clarification of the evidence-removal experiment. The rebuttal describes removing the selected subgraph, whereas Appendix B.4 describes deleting ground-truth evidence nodes from the input graph. These are different tests. Was Table B obtained from new experiments that modify the selected subgraph after selection, or does it summarize the existing input-graph perturbation experiments? This distinction matters for interpreting the results as evidence that answers rely on the selected subgraph.

Comments on the paper

2 comments

Dear Reviewers,

Please kindly review the authors' rebuttal and engage in discussions if you have further questions.

Thanks, AC

AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. We have updated the manuscript to address all main concerns raised during the review process.

In this paper, we formulate question-conditioned subgraph grounding for VQA from a causal perspective. To this end, we introduce dual-differential causal attention to mitigate structural and linguistic biases via interventional contrast, and leverage factual-counterfactual subgraph pairing for fine-grained supervision. Reviewers recognized our clear presentation and comprehensive experiments. Across all evaluated benchmarks, our method consistently achieves the best results while reducing average input tokens by 53.1% and end-to-end latency by 25.0% on GQA. During the rebuttal, we addressed the key concerns as follows.

1. Causal Formulation and Debiasing Verification.

We decompose each raw score into a causal effect, an additive bias, and residual noise. Under the principle of independent causal mechanisms, these bias terms are driven by fixed local elements and remain unchanged when we intervene on the surrounding context. Under this additive assumption, we subtract the expected score across context interventions to cancel out the bias, establishing our theoretical foundation in Section 3.2 and Appendix D. In nonlinear networks, DDCA operates as an empirical approximation to mitigate confounding biases. Experiments on biased datasets in Appendix B.9 and shuffled expectation terms in Appendix B.8 further confirm effective debiasing and rule out general contextual sensitivity.

2. Methodological Novelty and Gain Attribution.

Existing subgraph retrievers rely on statistical correlations and often retrieve distracting nodes, while prior causal debiasing methods apply a single global counterfactual to final outputs rather than scoring individual graph elements. In contrast, we design DDCA to perform node-level causal interventions and extract a compact subgraph prior to VLM answering. We further conducted controlled ablations across selector architectures and counterfactual supervision, confirming that the gains stem from DDCA itself rather than the model architecture or supervision alone.

3. Subgraph Faithfulness and Evidence Reliance.

To verify that the VLM relies on the selected subgraph rather than defaulting to the original image alone, we conducted subgraph perturbation experiments in Appendix B.5. Providing only the selected subgraph without the image still outperforms the image-only baseline by 19.0%, whereas removing selected nodes or perturbing attributes and spatial relations within the selected subgraph noticeably degrades accuracy. These results confirm that the answering model faithfully grounds its predictions in the selected subgraph.

4. Supervision Quality and Robustness.

We clarified that counterfactual subgraphs represent existing distractor evidence and are rarely empty. As verified in Appendix B.7, the factual subgraph dictates the gradient direction while the counterfactual subgraph only scales the magnitude, preserving optimization stability even when counterfactual subgraphs are noisy or empty. Furthermore, experiments in Appendix B.3 confirm that our method remains robust in cluttered scenes with distractors and effective on noisy predicted scene graphs.

We sincerely thank Reviewers oyXy and 5Byh for engaging in the discussion and raising their scores to 3. We also thank Reviewer 9Qqz for the follow-up discussion and kindly invite Reviewer 9Qqz to review our latest responses.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.