- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a scene graph grounding framework for improving the faithfulness and accuracy of visual question answering. The method introduces Dual-Differential Causal Attention (DDCA) to identify question-relevant evidence by comparing node–node associations and question–node relevance against reference scores, aiming to mitigate structural and linguistic biases. It also employs factual–counterfactual supervision, where an LLM generates incorrect alternative answers and extracts corresponding evidence subgraphs to guide node selection. The selected subgraph is then provided to a frozen VLM alongside the original image and question. Experiments across multiple VQA benchmarks and VLMs demonstrate improvements in answer accuracy and evidence selection quality over existing baselines.
Critical strengths
1.The paper addresses an important problem in visual question answering by selecting relevant evidence from scene graphs to reduce distracting information. The proposed framework is well motivated and can be integrated with different VLMs without modifying their parameters.
- The experimental evaluation is relatively comprehensive, covering multiple datasets and VLMs. The improvements in answer accuracy and evidence selection, together with the ablation and robustness studies, provide encouraging evidence for the effectiveness of the overall approach.
Critical weaknesses
-
Counterfactual supervision. The construction of counterfactual evidence is not sufficiently clear. Since the question and scene graph remain unchanged, it is unclear how a subgraph can genuinely support an incorrect answer. This raises concerns about the validity of the counterfactual evidence and the resulting supervision.
-
Causal interpretation. The theoretical analysis of DDCA assumes that structural and linguistic biases remain unchanged under interventions, but this assumption is not adequately justified. Moreover, the differential scores may reflect sensitivity to contextual changes rather than the causal importance of evidence.
3.The training objective focuses on selecting nodes according to LLM-generated supervision rather than directly optimizing answer faithfulness. Since the original image remains available during inference, the current experiments do not clearly establish whether the VLM actually relies on the selected subgraph.
4.The paper lacks comparisons with simpler supervised node selectors trained under the same conditions. In addition, the robustness experiments do not sufficiently cover challenging distractors involving similar objects and spatial relationships, and the reported results lack systematic statistical uncertainty estimates.
Questions for authors
1.How can the counterfactual subgraph provide valid supporting evidence for an incorrect answer when the original question and scene graph remain unchanged? What proportion of the generated counterfactual subgraphs are empty or invalid?
2.What justifies the assumption that structural and linguistic biases remain invariant under the proposed interventions? How can the authors distinguish genuine causal contributions from general sensitivity to contextual changes?
3.Have the authors considered evidence removal or replacement experiments to verify whether the downstream VLM actually relies on the selected subgraph when generating its answers?
4.Could the authors provide comparisons with simpler supervised selectors, such as MLPs or standard graph attention models using the same supervision? Additionally, how does the method perform with more challenging same-category distractors, and how stable are the results across random seeds?
4 comments
Reply to Reviewer 9Qqz (Part 2/2: W4/Q4)
W4 / Q4. Simpler selectors, difficult distractors, and random seeds.
1. Comparison with Simpler Selectors
In Table 2, removing DDCA leaves a simple 3-layer GAT selector, which drops GQA accuracy to 84.4%. We also evaluate a 3-layer MLP selector without DDCA, which achieves 83.2%. Under the same factual-counterfactual supervision, our full selector reaches 85.9%, confirming the effectiveness of DDCA.
2. Robustness to Difficult Distractors
On GQA, our method remains robust to same-category and spatial distractors. To evaluate same-category distractors, we insert up to three duplicates of an evidence object into the scene graph. Each duplicates keeps the original object name but modifies an answer-relevant attribute. As shown in Table C, baseline accuracies drop noticeably under these distractors, while ours drops by less than 0.7%. For spatial distractors, Appendix B.3 shows that reversing all spatial relations leaves our method at 80.0%, while baselines fall to 72.4%–76.7%.
Table C: Performance comparison (%) under same-category distractors on GQA.
| Method | Original | +1 distractor | +2 distractors | +3 distractors |
|---|---|---|---|---|
| Full SG | 81.9 | 79.2 | 79.7 | 80.4 |
| EMRAG | 83.0 | 79.2 | 76.9 | 77.2 |
| PCST | 78.4 | 62.2 | 59.5 | 59.5 |
| SubgraphRAG | 83.7 | 81.6 | 81.4 | 80.1 |
| Ours | 85.9 | 85.6 | 85.2 | 85.4 |
3. Stability Across Random Seeds
To verify stability against randomness, we conducted experiments on COCO-QA using Qwen3.5 4B, which yields the smallest gain in Table 1. Across two additional random seeds, our model achieves 66.17% and 66.12%, consistently outperforming all baselines.
The authors have provided helpful explanations and additional experiments. However, some concerns about the causal interpretation remain unresolved. In particular, the justification for bias invariance is still insufficient, and it remains unclear whether the differential scores capture genuine causal contributions or simply reflect sensitivity to contextual changes. Therefore, I maintain my original score.
Thank you for your thoughtful follow-up. We address your remaining concerns regarding bias invariance and causal interpretation in detail below.
1. How Linguistic and Structural Biases Arise
During training, scene-graph-based VQA models frequently learn dataset-level statistical priors, including linguistic and structural biases.
Linguistic bias stems from the frequent co-occurrence of specific questions and answers in the training data. For instance, if "yellow" frequently answers "What color is the banana on the plate?", the model learns to rely on this statistical shortcut from the question alone. Consider a specific scene with a yellow banana on the table and a green banana on the plate. Driven by this learned prior, the model still assigns an inflated score to the "yellow" node and ignores the spatial constraint "on the plate".
Structural bias arises from imbalanced node connectivity in scene graphs. In a visual scene, central nodes like "dining table" or "person" often connect to many surrounding objects, resulting in high node degrees. Because these high degree nodes frequently appear along reasoning paths during training, the model relies on node degree as a shortcut. Consequently, it assigns inflated scores to these high degree nodes even when they act as distractors rather than true evidence.
Ultimately, both biases reflect scene-independent statistical correlations rather than causality grounded in the specific visual context.
2. Why Statistical Biases Remain Unchanged Under Interventions
Once trained, the model stores these dataset-level priors in its fixed parameters. Because these priors reflect general statistics across diverse training scenes, they act as direct shortcuts driven by local elements rather than specific scene context. Under the principle of independent causal mechanisms in causal inference [1], intervening on the surrounding context leaves independent local mechanisms invariant. When we preserve these local elements and modify only the surrounding context during intervention, the resulting bias contributions remain unchanged:
- In the Query-Node branch (Eq. 4): We hold the question and candidate node fixed while removing other background nodes to construct reference graphs . Although removing background nodes changes the surrounding scene context of , the linguistic prior is driven by rather than the removed background nodes, and therefore remains unchanged between and .
- In the Node-Node branch (Eq. 3): We hold the target node fixed and replace the source node with another node , without modifying the graph structure . Because neither nor its node degree in changes, its degree-based structural bias remains unchanged.
3. Bias Cancellation via Differential Scoring
Raw scores on the intact scene graph capture statistical correlation rather than causation, combining the causal effect with the biases above. To remove these biases, we introduce the differential formulation in Eq. 2: Here, the first term evaluates the intact scene context, capturing both the causal effect and the bias. The second term holds the local elements fixed and marginalizes out surrounding context by taking the expectation over interventions, isolating the invariant bias. Under the additivity assumption analyzed in Appendix D, this subtraction directly cancels out the shared bias component.
4. Empirical Verification
In Appendix B.9 (Table 16), we construct biased datasets from GQA and CLEVR to evaluate whether DDCA mitigates structural and linguistic biases. With DDCA, the F1 drop decreases from 18.7% to 13.7% on higher degree distractors and from 11.9% to 6.4% on rare answers. This confirms that our differential formulation effectively mitigates both biases.
Furthermore, Appendix B.8 (Table 15) validates the role of the intervention expectation terms on GQA. Removing these terms or replacing them with global graph averages yields 64.2% and 64.1% F1, respectively. In contrast, DDCA reaches 65.0% F1. Crucially, randomly permuting the expectation terms across nodes within the same graph drops F1 to 63.2%. This indicates that the performance gains do not arise simply from sensitivity to contextual changes, but depend on contrasting each node against its own intervention baseline.
We have revised Section 3.2 in the updated manuscript to present exact cancellation under additive and invariant bias assumptions as the analytical basis for our design, while clarifying that in practical nonlinear networks DDCA acts as an empirical approximation to mitigate confounding biases.
References
[1] Schölkopf et al., Toward causal representation learning, Proceedings of the IEEE, 2021.
Reply to Reviewer 9Qqz (Part 1/2: W1/Q1 - W3/Q3)
W1 / Q1. Construction and validity of the counterfactual subgraphs, and proportion of empty cases.
As shown in Appendix A.4, we construct the counterfactual subgraph S− by extracting the nodes and relations of an existing distractor object A− of the same semantic type, which directly serves as local evidence for the incorrect answer.
Table 10 confirms that S− is valid in 87.0% of cases. Furthermore, Table A demonstrates that the proportion of empty S− remains consistently low across all evaluated datasets.
Table A: Proportion of empty counterfactual subgraphs (S−) across evaluated datasets.
Crucially, our counterfactual supervision remains robust even in rare cases where S− is noisy or empty. As shown in Appendix B.6, the factual subgraph S+ determines the gradient direction by assigning positive weights to its nodes and negative weights to all others. S− only scales the weight magnitudes. Consequently, our strategy avoids providing a misleading supervision signal whether S− is noisy or empty.
W2 / Q2. Invariance of the biases under intervention and distinction from contextual sensitivity.
1. Invariance of Biases under Intervention
Structural bias (Bstruct) is the higher score assignment to high-degree nodes, depending solely on the target node's degree. Linguistic bias (Bling) is the higher score assignment to common answers, driven by the prior co-occurrence between the query and target entity. We hold vi fixed in Eq. (3) and (Q,vi) fixed in Eq. (4) during interventions. Therefore, both biases remain invariant.
2. Distinction from Contextual Sensitivity
We distinguish causal effect from general contextual sensitivity through a reference-shuffling experiment in Table 12 of Appendix B.7. Specifically, we randomly shuffle the reference expectations across nodes, substituting each node's correct reference with that of another node. This mismatch lowers the F1 score to 63.2%, falling below the 64.2% baseline obtained without using any reference terms. This degradation precludes general contextual sensitivity, demonstrating that our differential mechanism preserves the causal effect.
W3 / Q3. Training objective and reliance on the subgraph.
1. Rationale for Offline Training Supervision
Directly optimizing answer faithfulness requires querying the VLM at every training step, which leads to a high computational cost. In contrast, our offline supervision generates the target subgraphs only once, allowing the selector to be trained without any VLM calls and work with any frozen VLM.
2. Evidence Removal and Replacement
We conducted evidence removal experiments to verify that the VLM relies on the selected subgraph GS rather than the original image alone. As shown in Table B, removing GS from the full scene graph G drops GQA accuracy from 85.9% to 52.2%, falling below the 61.2% image-only baseline. Moreover, providing GS without the original image still achieves 80.2% accuracy, surpassing the image-only baseline by 19.0%.
We also replaced internal relations and attributes within GS. Reversing spatial relations within GS lowers accuracy from 85.9% to 81.6%, and modifying answer-relevant attributes within GS drops accuracy to 57.9%. These removal and replacement results confirm that the VLM relies on the selected subgraph when generating answers.
Table B: Evidence removal and replacement experiments on the selected subgraph.