- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce PROVE, a training-free neurosymbolic approach for compositional VQA that incorporates perception uncertainty into the answer generation process. The authors compare this approach to existing program synthesis and neurosymbolic methods with matched tools and find large increases to performance across a suite of compositional VQA benchmarks.
Critical strengths
Premise of the paper is strong: The idea of keeping the probability associated with each stage of a solution is a good idea - and I particularly like the use of ProbLog to compute the probability of each hypothesis. The paper presents a rigorous and principled approach for simple VQA over attributes and spatial relations.
Comparison to DePROVE: The comparison ablating the preservation of uncertainty was necessary, and gives confidence to the validity of the approach on the shown benchmarks.
Writing: The writing is clear and easy to follow, the figures present in the paper are valuable and complimentary to the text.
Critical weaknesses
Strict framework is limiting: While the proposed framework is effective at the shown benchmarks, the authors have limited their experiments to VQA queries that are generated as a composition of attributes and spatial relation. In a sense, this method is tailor-made for these queries - the method decomposes the scene into entities, their relations & attributes and the simple hypotheses that these benchmarks take as answers. The task of visual question answering is much broader than this (e.g. medical queries, 3D spatial reasoning queries with open-ended numeric answers, questions about intent and causality) - all of these are not easily decomposable in the way this method operates. The authors should either explore the generalization of their method to a broader scope of visual question answering or describe how this method can augment existing, more general systems.
Additional qualitative examples: I would have liked to see more qualitative examples. The example of the corgi is particularly illustrative, but further examples (e.g. highlighting cases where PROVE and DePROVE yield different answers) would have been valuable. I recommend the authors to add more, even if they are put in the supplementary.
Questions for authors
Please see weaknesses.
2 comments
I thank the authors for their rebuttal. I appreciate the inclusion of the discussion on extending this framework for general applications and the additional qualitative examples. I maintain my score of 3.
Response to Reviewer k6RS (1/1)
We thank the reviewer for the thoughtful feedback. We address each concern in turn and attempt to implement the reviewer's suggestions. Our changes to the paper in response to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.
1. "Strict framework is limiting: ...the authors have limited their experiments to VQA queries that are generated as a composition of attributes and spatial relation."
1.1) "In a sense, this method is tailor-made for these queries... The task of visual question answering is much broader than this (e.g. medical queries, 3D spatial reasoning queries with open-ended numeric answers, questions about intent and causality)..."
1.2) "The authors should either explore the generalization of their method to a broader scope of visual question answering or describe how this method can augment existing, more general systems."
We agree that VQA is a much broader field than compositional VQA, the setting we evaluate. We aim to study how to logically compose multiple uncertain perceptions to derive an answer. To isolate this problem, we evaluate on standard compositional VQA benchmarks that require no domain-specific knowledge (Section 4, L352-353), where a scene is represented by its entities, attributes, and relationships. Tasks such as medical queries or 3D spatial reasoning also require capabilities such as domain knowledge or depth perception. These depend on the underlying models rather than on how their perceptions are composed, so results on them would not isolate PROVE’s contribution.
The core design of PROVE can be adapted to other domains that share the problem of logically composing multiple uncertain perceptions. Adapting it may require a different symbolic representation and other foundation models as tools. For example, in 3D spatial reasoning, a depth estimation model could provide 3D relationships as probabilistic facts. A more general system could also use PROVE as one component, supplying confidence scores for compositional claims about an image. We believe these are interesting directions for future work and have added this discussion to the conclusion (Section 5, L536-539).
In particular in lines 536-539 (Section 5), we added:
2. "Additional qualitative examples: I would have liked to see more qualitative examples."
2.1) "...further examples (e.g. highlighting cases where PROVE and DePROVE yield different answers) would have been valuable. I recommend the authors to add more, even if they are put in the supplementary."
We agree that more qualitative examples make the benefit of preserving uncertainty more concrete, and we thank the reviewer for the suggestion. We have added three examples that highlight cases where PROVE and DePROVE yield different answers, each shown with its collected facts, probabilities, and rule, in a new appendix (Appendix G). The main text now references them (Section 4.2, L432).