- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NeSyCAPE, a neurosymbolic framework that integrates global consistency constraints into differentiable program execution for compositional question answering. The authors identify a key limitation in existing neurosymbolic methods: atomic concepts are predicted locally. NeSyCAPE addresses this by representing each task as a graph of entities, relations, and atomic concepts, defining query-specific executable programs over this graph, and encoding domain knowledge as logical constraints. The framework is evaluated on three reasoning tasks with increasing complexity: Temporal QA (MATRES), Spatial VQA (3D-FORCE), and KB-VQA (VQAR). Incorporating global consistency yields additional improvements in several settings.
Critical strengths
1: The paper introduces a novel and elegant framework that unifies differentiable program execution with global logical constraint satisfaction over a shared concept graph. The key insight that program execution and constraints can operate over the same grounded concept predictions, allowing both training signals and inference to benefit from global consistency addresses a genuine gap in the neurosymbolic literature. This integration is well motivated and yields encouraging results, particularly on compositional visual reasoning tasks, although the benefits vary across tasks and model configurations.
2: Table 4 shows that LLM-generated constraints achieve performance comparable to GT-KB constraints across VQAR C2–C6 under the 20% training-data setting. This suggests that schema-restricted LLM generation, combined with compilation and execution validation, may help reduce reliance on manually specified domain constraints. Although the evidence is limited to one benchmark and does not establish cross-domain scalability, it provides a useful initial demonstration of automated constraint acquisition.
Critical weaknesses
1: Limited analysis of ILP scalability. Although Tables 1–3 report inference latency, the paper does not systematically examine how ILP cost scales with grounded concepts, instantiated constraints, or candidate answers. On 3D-FORCE SAG, adding ILP increases inference time from 0.65 to 10.15 seconds per question without improving accuracy (Table 2), raising concerns about practical scalability.
2: Task-dependent benefits of global constraints. GC and ILP do not consistently improve performance across tasks and model configurations. For example, GC reduces Qwen3-8B performance on MATRES, while ILP decreases accuracy in several settings. Although the authors acknowledge these outcomes, the causes of the negative effects and their sensitivity to constraint weights and training variability remain insufficiently investigated, limiting evidence of robust improvements.
Questions for authors
1: Tables 1–3 report inference latency, and Appendix B describes the task-level graph and constraint inventory. Could the authors characterize the actual per-instance ILP size, including the number of grounded decision variables, instantiated constraints, and solver calls, and analyze how inference time scales with these quantities? In particular, how does the per-object optimization procedure in Appendix A.5 affect inference cost as the number of candidate objects increases? Are approximate or relaxed inference alternatives practical in larger settings?
2: Appendix A.3 describes constraint construction using GPT-5.6-Terra with DomiKnowS-based validation, while acknowledging that syntactic and execution checks do not guarantee semantic correctness. How is the semantic validity of accepted constraints assessed? Additionally, how sensitive are the generated constraint sets and downstream performance to changes in the LLM, domain descriptions, or prompt formulations?
3: Section 3.6 and Appendix A.6 specify the combined training objective and task-dependent GC weights. How sensitive is performance to the relative weighting of execution supervision and global-consistency loss? Could the authors provide a sensitivity analysis and discuss why GC decreases Qwen3-8B performance on MATRES while improving several visual reasoning results?
7 comments
[Q2] Semantic Evaluation
For the rules/constraints generated by LLMs, for MATRES and 3D-FORCE, we evaluated them manually before using them in the model; therefore, they were all correct.
For VQAR, the dataset provided ground-truth rules, but we also added more auto-generated rules. To review the semantics of generated rules, we first used an LLM to decide whether to accept or reject the rules. We accepted an auto-generated rule if its conclusion followed from its premises given the intended meanings of the predicates and the task’s commonsense knowledge. If the rule is rejected, we then perform human evaluation to ensure that the rejection is valid. We will provide the details and prompt for this evaluation in the Appendix of the updated revision.
Out of 128 generated rules, AI-assist accepted 114 rules. The remaining 14 rules are marked either as rules invalid (6 rules) or possibly invalid (8 rules). The author reviewed all 14 rules, accepted 12, and rejected the two. For LLM-assisted program generation, we checked the generated program's quality and semantics by comparing the generated program with the ground-truth program in 3D-FORCE and VQAR. For tasks without a ground-truth program, we manually evaluated 100 generated programs against the natural language question. All of them are correct and represent the correct translation. This means that, since our datasets are all synthetic, the translation is not a problem, and the challenge is in the perception and reasoning components.
We appreciate your thoughtful feedback and hope the additional results address your concerns. We would be grateful if you would take them into account when reconsidering your assessment. We will share the revised manuscript soon.
Thank you for the additional experiments and clarifications. The new results have largely resolved most of my concerns about ILP cost and the roles of GC and ILP. Some limitations remain regarding the semantic validation of the generated VQAR rules, as well as the scalability of ILP. Overall, I appreciate the authors’ efforts and maintain my original score.
We appreciate the reviewer’s thoughtful feedback on our work. We’ve addressed each concern and question below.
[W1/Q1] ILP Scalability
We agree that latency alone does not explain ILP scalability. Enforcing constraints adds computation and does not guarantee better answer accuracy, as our SAG results illustrate. To investigate ILP scaling, we added a timing study on the 3D-FORCE REF test questions. Variable and constraint counts vary across questions. Times are measured per question and include ILP construction and processing after atomic prediction. As the number of candidates doubled from 6 to 12, the ILP grew from about 4.9K to 19.2K variables. At 12 candidates, building the ILP accounted for 3.79 of the 5.50 seconds for Exec. Future work may investigate reusable formulations, shared computation across candidates, and more compact grounding to reduce this overhead.
To compare inference efficiency, we measured the time NeSyCAPE and DOLPHIN[1] used to find an answer under hard constraints on all 1,822 3D-FORCE REF test questions. NeSyCAPE’s ILP ranking averaged 4.17 seconds per question with execution-only and 4.10 seconds with execution + GC. DOLPHIN checked up to 64 candidate worlds (top-k proof) and averaged 4.36 seconds per question with DTKP-AM scoring and 19.69 seconds with DAMP scoring. Thus, NeSyCAPE Exec + GC + ILP took 0.26 fewer seconds per question than DTKP-AM and 15.59 fewer seconds per question than DAMP on average. All timings exclude neural prediction and prior graph preparation. These results show that we are faster than DOLPHIN in 3D-FORCE.
[1]Naik et al. DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. ICML, 2025.
[W2/Q3] Benefit of GC and ILP
To better understand how GC training and ILP inference affect performance, we examined task score and constraint violation rates on MATRES and 3D-FORCE REF. For each sample, we evaluated the model's atomic predictions against the applicable domain constraints to compute the rule violation rate. This metric measures consistency among predictions, which is distinct from accuracy against ground truth. On MATRES, we used Qwen3-8B as the initial checkpoint, following the setting described in Section 4.2, and compared predictions with and without ILP at various GC weights. On 3D-FORCE REF, we focused on training with GC. We will report the full details of the study in the revision.
MATRES results with Qwen3-8B under full-data training. All models were trained on the full training set.
3D-FORCE REF results with the geometric predictor under 2K training. All models were trained on 2,000 questions and evaluated on the same 1,822 REF test questions.
On MATRES, increasing the GC weight from 0.001 to 0.05 reduced the constraint violation rate from 4.72% to 3.85%, while execution accuracy slightly decreased from 72.16% to 72.04%. ILP exhibited a similar trade-off: although it eliminated all constraint violations, final-answer accuracy either decreased or remained unchanged. These results suggest that models trained without global constraints may learn to produce correct final answers despite inconsistencies in their intermediate predictions. Enforcing global consistency through GC training or ILP inference may discourage such predictions, reducing final-answer accuracy.
In contrast, on 3D-FORCE REF, increasing the GC weight from 0.0001 to 0.01 reduced the constraint violation rate from 17.53% to 16.22%, while improving final-answer accuracy from 68.60% to 69.50%. These results suggest that tasks such as 3D-FORCE REF, that require the model to compose several atomic predictions, have more gain from supervision with global constraints..