acceptodds
Under review as a conference paper at ICLR 2027

NeSyCAPE: Neuro Symbolic Constraint Aware Program Execution

Abstract

One effective strategy for handling complex compositional Question Answering is to decompose questions into atomic-concept questions and compose the answers through executable programs, a so-called neurosymbolic methodology. Although effective, these methods often predict atomic concepts locally, which can produce individually plausible but globally inconsistent predictions and lead to incorrect program outputs. To address this limitation, we propose NeSyCAPE, a framework that integrates global consistency constraints into differentiable program execution. We represent each task as a graph of entities, relations, and atomic concepts. A query-specific executable program is then defined over this graph to compose the concepts required to answer the question, while domain knowledge is encoded as logical constraints over the involved concepts. This shared representation allows query-specific execution and global consistency to jointly influence the final answers. During training, executable supervision and differentiable global constraints jointly update the atomic concept predictors. At inference time, constrained optimization combines the neural prediction scores while satisfying the global constraints to produce a feasible concept assignment from which program execution determines the answer. We evaluate NeSyCAPE across three reasoning problems with increasing execution and global-consistency requirements: Temporal QA, Spatial VQA, and KB-VQA. Our results show that executable supervision substantially improves over local prediction baselines, while incorporating global consistency provides further gains in program execution.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-symbolic, Constraint Learning, Optimization, Question-Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C8RROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NeSyCAPE, a neurosymbolic framework that integrates global consistency constraints into differentiable program execution for compositional question answering. The authors identify a key limitation in existing neurosymbolic methods: atomic concepts are predicted locally. NeSyCAPE addresses this by representing each task as a graph of entities, relations, and atomic concepts, defining query-specific executable programs over this graph, and encoding domain knowledge as logical constraints. The framework is evaluated on three reasoning tasks with increasing complexity: Temporal QA (MATRES), Spatial VQA (3D-FORCE), and KB-VQA (VQAR). Incorporating global consistency yields additional improvements in several settings.

Critical strengths

1: The paper introduces a novel and elegant framework that unifies differentiable program execution with global logical constraint satisfaction over a shared concept graph. The key insight that program execution and constraints can operate over the same grounded concept predictions, allowing both training signals and inference to benefit from global consistency addresses a genuine gap in the neurosymbolic literature. This integration is well motivated and yields encouraging results, particularly on compositional visual reasoning tasks, although the benefits vary across tasks and model configurations.

2: Table 4 shows that LLM-generated constraints achieve performance comparable to GT-KB constraints across VQAR C2–C6 under the 20% training-data setting. This suggests that schema-restricted LLM generation, combined with compilation and execution validation, may help reduce reliance on manually specified domain constraints. Although the evidence is limited to one benchmark and does not establish cross-domain scalability, it provides a useful initial demonstration of automated constraint acquisition.

Critical weaknesses

1: Limited analysis of ILP scalability. Although Tables 1–3 report inference latency, the paper does not systematically examine how ILP cost scales with grounded concepts, instantiated constraints, or candidate answers. On 3D-FORCE SAG, adding ILP increases inference time from 0.65 to 10.15 seconds per question without improving accuracy (Table 2), raising concerns about practical scalability.

2: Task-dependent benefits of global constraints. GC and ILP do not consistently improve performance across tasks and model configurations. For example, GC reduces Qwen3-8B performance on MATRES, while ILP decreases accuracy in several settings. Although the authors acknowledge these outcomes, the causes of the negative effects and their sensitivity to constraint weights and training variability remain insufficiently investigated, limiting evidence of robust improvements.

Questions for authors

1: Tables 1–3 report inference latency, and Appendix B describes the task-level graph and constraint inventory. Could the authors characterize the actual per-instance ILP size, including the number of grounded decision variables, instantiated constraints, and solver calls, and analyze how inference time scales with these quantities? In particular, how does the per-object optimization procedure in Appendix A.5 affect inference cost as the number of candidate objects increases? Are approximate or relaxed inference alternatives practical in larger settings?

2: Appendix A.3 describes constraint construction using GPT-5.6-Terra with DomiKnowS-based validation, while acknowledging that syntactic and execution checks do not guarantee semantic correctness. How is the semantic validity of accepted constraints assessed? Additionally, how sensitive are the generated constraint sets and downstream performance to changes in the LLM, domain descriptions, or prompt formulations?

3: Section 3.6 and Appendix A.6 specify the combined training objective and task-dependent GC weights. How sensitive is performance to the relative weighting of execution supervision and global-consistency loss? Could the authors provide a sensitivity analysis and discuss why GC decreases Qwen3-8B performance on MATRES while improving several visual reasoning results?

7 comments

We appreciate the reviewer’s thoughtful feedback on our work. We’ve addressed each concern and question below.

[W1/Q1] ILP Scalability

We agree that latency alone does not explain ILP scalability. Enforcing constraints adds computation and does not guarantee better answer accuracy, as our SAG results illustrate. To investigate ILP scaling, we added a timing study on the 3D-FORCE REF test questions. Variable and constraint counts vary across questions. Times are measured per question and include ILP construction and processing after atomic prediction. As the number of candidates doubled from 6 to 12, the ILP grew from about 4.9K to 19.2K variables. At 12 candidates, building the ILP accounted for 3.79 of the 5.50 seconds for Exec. Future work may investigate reusable formulations, shared computation across candidates, and more compact grounding to reduce this overhead.

CandidatesQuestionsILP variables (mean)Linear constraints (mean)Exec (s/q)
654.9K8.9K0.93
756.7K12.0K1.33
868.7K15.9K1.87
9710.9K20.3K2.78
10713.4K25.1K3.40
11716.2K30.3K4.53
12719.2K36.1K5.50

To compare inference efficiency, we measured the time NeSyCAPE and DOLPHIN[1] used to find an answer under hard constraints on all 1,822 3D-FORCE REF test questions. NeSyCAPE’s ILP ranking averaged 4.17 seconds per question with execution-only and 4.10 seconds with execution + GC. DOLPHIN checked up to 64 candidate worlds (top-k proof) and averaged 4.36 seconds per question with DTKP-AM scoring and 19.69 seconds with DAMP scoring. Thus, NeSyCAPE Exec + GC + ILP took 0.26 fewer seconds per question than DTKP-AM and 15.59 fewer seconds per question than DAMP on average. All timings exclude neural prediction and prior graph preparation. These results show that we are faster than DOLPHIN in 3D-FORCE.

MethodMean answer-finding time ± SD (s/question)
NeSyCAPE Exec + ILP4.17 ± 2.02
NeSyCAPE Exec + GC + ILP4.10 ± 1.94
DOLPHIN + hard constraints (DTKP-AM, k=64)4.36 ± 2.12
DOLPHIN + hard constraints (DAMP, k=64)19.69 ± 45.61

[1]Naik et al. DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. ICML, 2025.


[W2/Q3] Benefit of GC and ILP

To better understand how GC training and ILP inference affect performance, we examined task score and constraint violation rates on MATRES and 3D-FORCE REF. For each sample, we evaluated the model's atomic predictions against the applicable domain constraints to compute the rule violation rate. This metric measures consistency among predictions, which is distinct from accuracy against ground truth. On MATRES, we used Qwen3-8B as the initial checkpoint, following the setting described in Section 4.2, and compared predictions with and without ILP at various GC weights. On 3D-FORCE REF, we focused on training with GC. We will report the full details of the study in the revision.

MATRES results with Qwen3-8B under full-data training. All models were trained on the full training set.

GC weightExec accuracy (%) ↑+ ILP accuracy (%) ↑Exec violation rate (%) ↓+ ILP violation rate (%) ↓
072.4072.163.980.00
0.00172.1671.924.720.00
0.00571.8071.574.350.00
0.0572.0472.043.850.00

3D-FORCE REF results with the geometric predictor under 2K training. All models were trained on 2,000 questions and evaluated on the same 1,822 REF test questions.

GC weightExec top-1 accuracy (%) ↑Constraint violations (%) ↓
0 (Exec only)67.2317.87
0.000168.6017.53
0.00166.7416.17
0.0169.5016.22

On MATRES, increasing the GC weight from 0.001 to 0.05 reduced the constraint violation rate from 4.72% to 3.85%, while execution accuracy slightly decreased from 72.16% to 72.04%. ILP exhibited a similar trade-off: although it eliminated all constraint violations, final-answer accuracy either decreased or remained unchanged. These results suggest that models trained without global constraints may learn to produce correct final answers despite inconsistencies in their intermediate predictions. Enforcing global consistency through GC training or ILP inference may discourage such predictions, reducing final-answer accuracy.

In contrast, on 3D-FORCE REF, increasing the GC weight from 0.0001 to 0.01 reduced the constraint violation rate from 17.53% to 16.22%, while improving final-answer accuracy from 68.60% to 69.50%. These results suggest that tasks such as 3D-FORCE REF, that require the model to compose several atomic predictions, have more gain from supervision with global constraints..

[Q2] Semantic Evaluation

For the rules/constraints generated by LLMs, for MATRES and 3D-FORCE, we evaluated them manually before using them in the model; therefore, they were all correct.

For VQAR, the dataset provided ground-truth rules, but we also added more auto-generated rules. To review the semantics of generated rules, we first used an LLM to decide whether to accept or reject the rules. We accepted an auto-generated rule if its conclusion followed from its premises given the intended meanings of the predicates and the task’s commonsense knowledge. If the rule is rejected, we then perform human evaluation to ensure that the rejection is valid. We will provide the details and prompt for this evaluation in the Appendix of the updated revision.

Out of 128 generated rules, AI-assist accepted 114 rules. The remaining 14 rules are marked either as rules invalid (6 rules) or possibly invalid (8 rules). The author reviewed all 14 rules, accepted 12, and rejected the two. For LLM-assisted program generation, we checked the generated program's quality and semantics by comparing the generated program with the ground-truth program in 3D-FORCE and VQAR. For tasks without a ground-truth program, we manually evaluated 100 generated programs against the natural language question. All of them are correct and represent the correct translation. This means that, since our datasets are all synthetic, the translation is not a problem, and the challenge is in the perception and reasoning components.


We appreciate your thoughtful feedback and hope the additional results address your concerns. We would be grateful if you would take them into account when reconsidering your assessment. We will share the revised manuscript soon.

Thank you for the additional experiments and clarifications. The new results have largely resolved most of my concerns about ILP cost and the roles of GC and ILP. Some limitations remain regarding the semantic validation of the generated VQAR rules, as well as the scalability of ILP. Overall, I appreciate the authors’ efforts and maintain my original score.

Reviewer_WzxuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a neurosymbolic framework which combines elements of program execution, logical auxillary loss, and integer linear programming in order to enable question answering for complex tasks. For each task, the authors leverage logical rules that must always be true, sourced either from the task or from an LLM, and use these rules to construct an auxiliary loss during training and to provide a more logically-constrained answer during inference. These rules need not be directly relevant to the target question of each sample. The authors provide an honest, well-written study with their approach, highlighting and explaining settings of strength and weakness of their approach, which is scientifically refreshing. However, questions remain about the novelty, performance, and efficiency of the proposed approach, particularly around baseline evaluation and an incremental quantitative empirical advance.

Critical strengths
  • The proposed method uses neurosymbolic AI to address failure modes of LLMs for multimodal reasoning tasks
  • the formulation of program execution inference in neurosymbolic AI as ILP seems like an interesting direction
  • elements of the proposed approach can have empirical advantages in certain settings
  • the experimental discussion is objective and honest in its analysis about the advantages of the approach
Critical weaknesses
  • My biggest concern with this work is that it is hard to discern the novel contribution. As I understand it, the individual component of Exec, GC, or ILP are already present individually in frameworks established by prior works. NeSyCAPE combines ideas from these three domains in a way that necessarily providing a fundamentally new capability along these dimensions. This may be okay, but then it rests on the empirical results to not only demonstrate strong results, but to present them against baselines from each individual domain (where applicable), in order to present an advantage and a clear insight that combining them is more than the sum of parts. It is my understanding that this has not occurred in this revision.

  • The comparison to prior work does not seem fully fleshed out to me. Frameworks like Scallop [1] are quite expressive and have support for task-specific constraints as rules, so what exactly can NeSyCAPE encode beyond "what the underlying logical languages support"? Furthermore, as in [2] just as a simple example, LLMs have already been shown to mine logic rules, so is NeSyCAPE doing this better in any way than the comparable SOTA in this space, or if a simpler approach was used, why? Similarly, for ILP -- what is the backbone DomiKnowS incapable of doing that is novel in NeSyCAPE, and how do similar approaches stack up [3]? As far as writing goes, more effort needs to go into delineating these distinctions; experimentally, I encourage the authors to inspect each of these domains, find the relevant baselines, and compare to them.

  • Benchmarking is quite limited. As a neurosymbolic method, I would have expected to see comparisons to some more recent works in the field, such as NePTune or [4]. For table 1, I see a reference to baseline GDLLM, but I do not see an explanation of what this method is anywhere in the paper. Table 2 is just comparing to LLM SFT. Table 3 has some neurosymbolic baselines, but NeSyCAPE is not a clear winner against Scallop, and there are no error bars which can help understand if there really is a clear advantage in the settings where is does win.

  • Performance is incremental while efficiency is worse. As I read the paper, it is a challenge to see a clear empirical win from the proposed method. For example, in 3D-FORCE table 2, it is not clear that Exec/GC/ILP is most performant on REF, and when it does win it is by 1.1 points, and there are no error bars to make this result significant. Similarly for C6 on VQAR. These results, without a sound justification for importance, undermine novelty. Similarly, the formulation to ILP comes with computational overhead. Training and testing speeds are multiple times worse than Scallop despite unclear empirical improvement, raising the question of whether the extra compute is needed. It also comes at a time when frameworks such as [4] are GPU-accelerated.

  • From a higher level, a stronger motivation for this work is needed. It is coming at a time where the strongest frontier LLMs can solve Millennium Prize Problems, so the writing should reflect what advantages this approach poses in lieu of this new reality. This is really a writing issue and not a major problem in my eyes.

Overall I enjoyed reading this work and believe it has a lot of promise -- I encourage the authors to use this feedback for a much stronger revision moving forward!

[1] Ziyang Li, Jiani Huang, and Mayur Naik. 2023. Scallop: A Language for Neurosymbolic Programming. Proc. ACM Program. Lang. 7, PLDI, Article 166 (June 2023), 25 pages. https://doi.org/10.1145/3591280

[2] Linhao Luo, Jiaxin Ju, Bo Xiong, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2025. ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning. In Advances in Knowledge Discovery and Data Mining: 29th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2025, Sydney, NSW, Australia, June 10–13, 2025, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg, 314–325. https://doi.org/10.1007/978-981-96-8173-0_25

[3] Weichun Shi, Minghao Liu, Wanting Zhang, Langchen Shi, Fuqi Jia, Feifei Ma, and Jian Zhang. 2025. ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15999–16019, Suzhou, China. Association for Computational Linguistics.

[4] Naik, A., Liu, J., Wang, C., Sethi, A., Dutta, S., Naik, M. & Wong, E.. (2025). DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. <i>Proceedings of the 42nd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 267:45461-45483 Available from https://proceedings.mlr.press/v267/naik25a.html.

Questions for authors
  • Can the author provide more experimental results for other baselines that are either 1) general neurosymbolic baselines or 2) specialized baselines relating to the subdomains of EXEC/GC/ILP, to the extent applicable?
  • Can the authors more clearly define any new capabilities in EXEC/GC/ILP individually?
  • Can the authors provide error bars in their reported results?
6 comments

We thank the reviewer for the thoughtful feedback on our work. Below, we address each concern and suggestion.

[W1/Q2] Novelty

We would argue that NeSyCAPE’s contribution is not simply the combination of program execution (Exec), global-constraint training (GC), and ILP inference. To combine these paradigms, we propose a more involved algorithm. Our algorithm treats each program execution as an objective function for an ILP optimization problem. In other words, the program’s execution trace defines the ILP objective. We then sample execution traces and, for each trace, use ILP optimizer to find the highest-scoring assignment that satisfies the global constraints. We select the answer associated with the highest objective value across these assignments. In this way, we exploit the explicit probabilistic composition of variables while ensuring that the selected assignment respects the global constraints.


[W1/W3/Q1/Q3] Additional Baselines with error bar results

To provide a more robust comparison, we added four baselines that use program execution and domain constraints in different ways. DOLPHIN [1], like Scallop[2], is a general neuro-symbolic framework that can execute probabilistic programs. Our NePTune [3] baseline uses a frozen model to execute task-specific programs. The two DomiKnowS baselines use domain constraints without program execution: one applies Semantic Loss [4] during training (GC), while the other enforces constraints through ILP at inference, following GLUECons [5]. We also added supervised baselines (SFT) that predict answers directly without executing a program. For the baselines trained across three seeds, we report the mean and sample standard deviation. NePTune is reported as a single evaluation because it uses no task-specific training. We discuss the results below and will include the full baseline configurations in the revised manuscript.

3D-FORCE Results on Top-1 Acc. for REF and Exact-Match Acc. for SAG on all of the evaluation datasets.

MethodREFSAG
NePTune — frozen Qwen43.0355.83
SFT17.82 ± 1.1550.26 ± 2.09
DomiKnowS + Semantic Loss18.08 ± 1.8150.96 ± 0.45
DomiKnowS + ILP17.82 ± 1.1550.26 ± 2.88
DOLPHIN73.31 ± 1.3072.35 ± 0.71
DOLPHIN + Constraints68.28 ± 0.9472.29 ± 0.72
NeSyCAPE (EXEC )71.08 ± 2.5769.51 ± 2.35
NeSyCAPE (EXEC + ILP )75.28 ± 1.7869.51 ± 2.35
NeSyCAPE (EXEC + GC )70.11 ± 1.9468.46 ± 1.86
NeSyCAPE (EXEC + GC + ILP)75.10 ± 1.0868.46 ± 1.86

VQAR Results on Recall@5 on the entire evaluation dataset. All of these models are trained with 20% of the training data.

MethodRecall@5 (%)
NMN67.821 ± 0.012
DOLPHIN47.74 ± 2.310
DOLPHIN + constraints47.74 ± 2.310
NeSyCAPE (EXEC)80.24 ± 0.086
NeSyCAPE  (EXEC + GC)80.70 ± 0.176
NeSyCAPE (EXEC + ILP)82.22 ± 0.14
NeSyCAPE (EXEC + GC + ILP)82.60 ± 0.10

On 3D-FORCE, adding ILP to the DomiKnowS made REF accuracy unchanged at 17.82%, while Semantic Loss raised it only to 18.08%. These results indicate that domain constraints alone did not close the gap to the program-execution baselines on 3D-FORCE. NePTune achieved 43.03% on REF and 55.83% on SAG, showing that a frozen model can use program execution without task-specific training. However, it scored below both trained execution models, NeSyCAPE and DOLPHIN. NeSyCAPE (Exec) are competitive with DOLPHIN on both subset: REF ( 71.08% vs 73.31%) and SAG (69.51% vs 72.29%). On VQAR, NeSyCAPE shows more effectiveness: with 10K training questions, NeSyCAPE Exec achieved 80.24% Recall@5, compared with 47.74% for DOLPHIN with or without constraints. Adding GC raised NeSyCAPE’s score to 80.70%. ILP further improved in both settings of NeSyCAPE, reaching 82.22% without GC and 82.60% with GC.

[1] Naik et al. DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. ICML, 2025.

[2] Huang et al. Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning. NeurIPS, 2021.

[3] Kamali and Kordjamshidi. NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language. ICLR, 2026.

[4] Xu et al. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. ICML, 2018.

[5] Rajaby Faghihi et al. GLUECons: A Generic Benchmark for Learning under Constraints. AAAI, 2023.

[W2] Related Work

We agree that the distinctions from prior work need to be clearer.

Scallop: Expressivity is not an issue in Scallop[1] vs. NeSyCAPE, but the way hard constraints and soft compositions are handled is fundamentally different in our algorithm from Scallop’s underlying reasoning. In Scallop, reasoning is done using a forward-chaining algorithm (underlying DataLog) that tries to entail the best solution satisfying all logical rules in proofs (top-k-proof). The algorithm becomes very inefficient as the number of hard constraints increases, especially when we have numerical and interval comparisons. However, in our setting, we do not do logical entailment through resolution; instead, we perform soft composition through procedural execution of the program. Then, we apply the constraint using a separate mechanism: ILP applied to the program trace, which is well-suited to efficiently and flexibly handle various constraint types, including numerical ones. In response to the reviewer, we added a new timing comparison. In the evaluated setting of 3D-FORCE, NeSyCAPE’s ILP step was faster than DOLPHIN[2]’s top-(k) proof, similar to Scallop's underlying method.

LLM-generated rules: ChatRule[3] develops rule generation and ranking for knowledge-graph reasoning. We use schema-restricted LLM generation to construct symbolic graphs and constraints that are specifically used inside our NeSYCAPE, rather than proposing a competing rule-mining method. Our focus is on their interaction with executable supervision, not superior rule-generation quality.

DomiKnowS[4] and ILP: NeSyCAPE contributes an algorithm to treat each program as an objective in an ILP optimization problem instance for combining program execution with constraint learning and ILP inference. More details on this objective are discussed in the Novelty clarification. Note that DomiKnowS already supports constrained learning (GC) and ILP, which we use as our underlying method.

[1] Huang et al. Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning. NeurIPS, 2021.

[2] Naik et al. DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. ICML, 2025.

[3] Luo et al. ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning. PAKDD, 2025.

[4] Rajaby Faghihi et al. DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning. EMNLP, 2021.


[W4] Time Efficiency Analysis of ILP

We agree that enforcing hard constraints at inference time, such as with ILP, can add computational cost. We believe our method still offers advantages over other approaches for determining answers with hard constraints. Tables 1–3 report runtimes for our main experiments, but they do not separate the time spent on neural prediction from the time spent finding an answer under constraints (ILP). To complement those results, we compared the time needed to find an answer under hard constraints on the 3D-FORCE REF test set. NeSyCAPE used ILP to determine the answer. We evaluated DOLPHIN with hard constraints under two scoring schemes: DAMP and DTKP-AM. Both retained up to 64 candidate worlds (top-k proof) that satisfied the constraints and evaluated the query. Timings cover all 1,822 test questions. SD measures variation across questions. These timings include constraint construction, verification, and answer decoding but exclude neural prediction and prior graph preparation to focus on constraint-finding algorithms.

MethodMean answer-finding time ± SD (s/question)
NeSyCAPE (Exec + ILP)4.17 ± 2.02
NeSyCAPE (Exec + GC + ILP)4.10 ± 1.94
DOLPHIN + hard constraints (DTKP-AM)4.36 ± 2.12
DOLPHIN + hard constraints (DAMP)19.69 ± 45.61

NeSyCAPE Exec + ILP took 15.52 seconds less per question on average than DOLPHIN with DAMP (4.17 vs. 19.69 s) and was slightly faster than DOLPHIN with DTKP-AM (4.17 vs. 4.36 s). This shows that ILP was competitive and even better with both DOLPHIN variants.

The key distinction between these results and VQAR (Table 3) is between returning an answer set and ranking candidate objects. NeSyCAPE uses one ILP solve to produce an answer set. For Recall@5, however, it uses additional ILP solves to score each object. Scallop can rank objects directly using the answer probabilities estimated from its top-k proofs. Avoiding these additional solves may help explain Scallop’s lower runtime on VQAR. We will add this analysis and additional results in the appendix and, if appropriate, in the main paper.

[W5] Motivation

Unlike a mathematical problem with explicit premises and a defined structure, a visual reasoning problem requires the model to infer the objects and relations it will reason about. This remains difficult even for strong vision-language models. Several recent benchmarks illustrate where off-the-shelf VLMs still fail on visual reasoning tasks [1,2,3]. In particular, GPT-5.1 and Gemini-3.1-Pro achieved only 6.90% and 47.89% accuracy on 3D-FORCE REF [3]. These results motivate the use of neuro-symbolic methods to complement frontier models [4,5,6]. We therefore believe that our framework remains useful for tasks that require consistency among the atomic predictions used in program execution. In the revision, we will highlight these failures and explain why this consistency matters for visual reasoning.

[1]Ma et al. 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark. ICCV, 2025.

[2]Yang et al. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. CVPR, 2025.

[3]Kamali et al. SATURN: Symbolic Spatial Reasoning for Multi-Perspective Grounding. EMNLP, 2026.

[4]Surís et al. ViperGPT: Visual Inference via Python Execution for Reasoning. ICCV, 2023.

[5]Kamali and Kordjamshidi. NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language. ICLR, 2026.

[6]Hsu et al. What’s Left? Concept Grounding with Logic-Enhanced Foundation Models. NeurIPS, 2023.


Thank you again for your thoughtful feedback. We hope our responses and additional results address your concerns, and we would be grateful if you would reconsider your assessment in light of the new evidence. We will share the revised manuscript soon.

Reviewer_15BnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NeSyCAPE, a neuro-symbolic framework that combines neural concept prediction, question-specific executable programs, and global logical constraints. For each input, the method builds an instance-specific graph containing entities, relations, and atomic concepts. Neural predictors assign probabilities to these concepts. A question is translated into an executable program that composes the relevant predictions to produce an answer. During training, the model is optimized using both answer supervision from program execution and a loss that penalizes violations of global logical constraints. At inference time, an ILP solver selects a high-scoring concept assignment that satisfies both the program conditions and the global constraints. The method is evaluated on Temporal QA with MATRES, spatial visual reasoning with 3D-FORCE, and knowledge-based VQA with VQAR. The paper studies the contributions of program-execution supervision, global-consistency training, and constrained inference. The reported results suggest that execution supervision is most useful in more compositional settings, while the effects of global constraints and ILP inference vary across tasks and predictors.

Critical strengths

S1. The paper provides a clear unified view of program execution, global consistency, and constrained inference. NeSyCAPE does more than apply post-hoc rule filtering. It lets answer-level program supervision and global constraints update the same atomic concept predictors during training, then enforces both program conditions and logical constraints at inference time. This is a meaningful and technically coherent direction for neuro-symbolic reasoning.

S2. The evaluation spans multiple reasoning settings and reports component-level behavior. The paper evaluates temporal reasoning, spatial VQA, and knowledge-based VQA, and separately studies execution supervision, consistency training, and ILP inference. It also reports negative or mixed effects in some settings, which makes the empirical analysis more credible.

Critical weaknesses

W1. The paper does not cleanly isolate the effect of executable supervision from differences in model architecture and training setup. The central claim is that executable supervision improves over local or direct prediction. However, in 3D-FORCE, the direct SFT baseline and NeSyCAPE do not use the same visual backbone, predictors, or architecture. In VQAR, the comparison relies largely on published results obtained under different protocols. A matched direct-supervision baseline using the same features, predictors, parameter budget, and training data is needed to establish the contribution of execution supervision.

W2. The semantic validity and reproducibility of the LLM-generated symbolic artifacts are insufficiently demonstrated. The framework relies on LLM-assisted construction of task graphs, executable programs, and logical constraints. Compilation and execution checks ensure that the artifacts are formally valid, but do not establish that they are semantically correct for the task domain. The paper does not provide the full final manifests, rule-level provenance, or an independent semantic audit. This weakens the claim that LLM-generated constraints can reliably improve reasoning.

Questions for authors

Q1. Could the authors provide the complete final task graphs, program mappings, and constraint manifests for each dataset? For every constraint, please specify whether it comes from manual design, ground-truth KB knowledge, or LLM generation, as well as the configuration in which it is enabled.

Q2. Beyond syntax, type checking, and executability, was there any independent semantic validation of LLM-generated programs and rules? Please report the validation protocol, sample size, error rate, and how the process avoids introducing task- or benchmark-specific answer information.

Q3. Can the authors add architecture-matched direct-supervision baselines for 3D-FORCE and VQAR? Ideally, these should use the same features, atomic predictors, parameter budget, training data, and compute budget as NeSyCAPE, while removing program execution.

Q4. Can the authors report multi-seed results with standard deviations or confidence intervals for the main comparisons, especially Execution, Execution plus GC, Execution plus ILP, and Execution plus GC plus ILP? This is important for the relatively small GC gains on VQAR and for the mixed results on MATRES.

Q5. For VQAR, candidate objects are ranked using separate ILP solves, and the paper notes that the resulting top-five set may not be jointly feasible. How does this relate to the claim that constrained program execution produces a coherent answer set? Could the authors also provide an evaluation based on jointly feasible answer sets?

3 comments

We appreciate your thoughtful feedback and valuable insights on our studies. Below, we respond to the weaknesses, comments, and suggestions.


[W1/Q3] Missing SFT baselines

We agree that the original comparison does not fully isolate the effect of executable supervision from differences in model architecture. To address this concern, we added an SFT baseline for 3D-FORCE and VQAR using the same visual features, object representations, and training data as the corresponding NeSyCAPE model. In particular, we use the SigLIP (the underlying architecture of Qwen3-VL) visual representation for 3D-FORCE, while we use Masked R-CNN and ResNet for VQAR. We chose these models because they have light computational requirements for training and execution. SFT with Qwen3-VL is already in Table 2 in the current paper, but training Qwen3-VL itself with execution was not computationally affordable for us. This comparison provides a more direct assessment of the contribution of executable supervision while controlling for differences in input representations and training data. We will describe both baseline configurations in the revision. The results below use the following settings: 20% and 100% of the training data; ± denotes SD across three random seeds.

TaskSFTNeSyCAPE (Program Execution)
3D-FORCE REF accuracy (20%)13.14 ± 3.7371.08 ± 2.57
3D-FORCE SAG accuracy (20%)50.70 ± 1.7369.51 ± 2.35
VQAR Recall@5 (20%)66.67 ± 1.2580.245 ± 0.086
3D-FORCE REF accuracy (100%)34.83 ± 1.8779.80
3D-FORCE SAG accuracy (100%)49.01 ± 1.0181.30

NeSyCAPE (Exec) models achieved substantially higher accuracy than the direct-prediction baselines in these comparisons. This shows that our NeSyCAPE is more data-efficient and effective due to the major focus on learning the building-block atomic concepts rather than learning shortcuts from complex questions. Please note that this table also emphasizes how a weak underlying neural model can be boosted dramatically with our training approach that it can exceed the very large VLMs like Qwen3-VL 8B zero-shot (45.60%) and also the fine-tuned Qwen3-VL 8B with SFT (56.80%), already presented in the paper.


[W2/Q1/Q2] Complete Rules Set and Semantic Validation of LLM-Generated Symbolic Artifacts

We agree that compilation and execution checks do not indicate semantic correctness. To improve reproducibility, we will provide the complete rule sets and available rule-level provenance in the appendix.

For the rules/constraint generated by LLMs, for MATRES and 3D-FORCE, we evaluated them manually before using them in the model; therefore, those were all correct.

For VQAR, we had the ground-truth rules given in the dataset, but we added more auto-generated rules. To review the semantics of generated rules, we first used an LLM to decide whether to accept or reject the rules. We accepted an auto-generated rule if its conclusion followed from its premises given the intended meanings of the predicates and the task’s commonsense knowledge. If the rule is rejected, we then perform human evaluation to ensure that the rejection is valid. We will provide the details and prompt for this evaluation in the Appendix of the updated revision.

Out of 128 generated rules, AI-assist accepted 114 rules. The remaining 14 rules are marked either as rules invalid (6 rules) or possibly invalid (8 rules). The author reviewed all 14 rules, accepted 12, and rejected the two.

For LLM-assisted program generation, we checked the generated program's quality and semantics by comparing the generated program with the ground-truth program in 3D-FORCE and VQAR. For the task without the ground truth program, we manually evaluated 100 generated programs against the natural language of the question. All of them are correct and represent the correct translation. This means that, since our datasets are all synthetic, the translation is not a problem, and the challenge is in the perception and reasoning components.

[Q4] Results with error bar

To measure the consistency across the run, we added three-seed experiments on 3D-FORCE and VQAR with 20% of the training data. Based on the results with the error bars, ILP improved mean accuracy on 3D-FORCE REF by 4.20 percentage points without GC and 4.99 points with GC. On VQAR, ILP improved mean Recall@5 by about 2 points in both settings. GC also improved VQAR Recall@5 by 0.46 points, but slightly reduced mean accuracy on 3D-FORCE. Overall, both GC and ILP can consistently provide improvement.

3D-FORCE test accuracy (%) with 20% training data

MethodREF accuracy (%)SAG accuracy (%)
SFT17.82 ± 1.1550.26 ± 2.09
NeSyCAPE (Exec)71.08 ± 2.5769.51 ± 2.35
NeSyCAPE (Exec + ILP)75.28 ± 1.7869.51 ± 2.35
NeSyCAPE (Exec + GC)70.11 ± 1.9468.46 ± 1.86
NeSyCAPE (Exec + GC + ILP)75.10 ± 1.0868.46 ± 1.86

VQAR test Recall@5 (%) with 20% training data

MethodRecall@5 (%)
SFT66.67 ± 1.25
NeSyCAPE (Exec)80.245 ± 0.086
NeSyCAPE (Exec + GC)80.703 ± 0.176
NeSyCAPE (Exec + ILP)82.22 ± 0.14
NeSyCAPE (Exec + GC + ILP)82.60 ± 0.10

Recall@5 is computed on the 4,898 test questions with non-empty gold answer sets. Values are means ± sample SD across three training seeds.


[Q5] ILP with Recall@5 and Exact-Match Results

ILP Claim: We apologize for the confusion. Our coherence claim applies to the answer set produced from a single feasible ILP assignment. For Recall@5 on VQAR, we instead score each candidate object separately by their ILP assignment score, as explained in Appendix A.5. Then, we rank the candidate objects by score to form the top-five ranking. Since different objects receive their scores from different feasible ILP assignments, the top-five ranking should not be treated as a single answer set. We will clarify this in the revision.

Exact-Match Results: To address the reviewer’s request, we evaluated one predicted answer set per question on the same 5,000 VQAR test questions. For NeSyCAPE with ILP, this set comes from a single assignment that satisfies the constraints. The table below reports exact-set match and mean set-F1 for NeSyCAPE and the available baselines, including DOLPHIN [1], a neuro-symbolic framework suggested by another reviewer. Exact-set match requires the predicted set to contain all and only the gold objects, while set-F1 gives partial credit for overlap.

Training percentages indicate the proportion of VQAR training questions used. For DOLPHIN and the Neural Module Network, we report the mean ± sample standard deviation across three training seeds. Within each NeSyCAPE training setting, local and ILP decoding use the same checkpoint.

MethodVQAR training Exact-set match (%)Set-F1 (%)
DOLPHIN20% (10K)2.21 ± 0.159.51 ± 0.49
Neural Module Network (our evaluation)20% (10K)9.91 ± 0.0924.10 ± 0.43
NeSyCAPE (Exec)20% (10K)27.09 ± 0.2639.10 ± 0.40
NeSyCAPE (Exec + ILP)20% (10K)15.33 ± 0.0333.78 ± 0.67
NeSyCAPE (Exec + GC)20% (10K)27.49 ± 0.0939.69 ± 0.06
NeSyCAPE (Exec + GC + ILP)20% (10K)15.55 ± 0.1334.03 ± 0.29
Scallop (our evaluation)100% (50K)14.5432.74
NeSyCAPE Exec100% (50K)27.8040.83
NeSyCAPE  (Exec + ILP)100% (50K)17.1837.02

On VQAR with 10K training questions, NeSyCAPE with execution achieved higher exact-set match and Set-F1 than both Scallop and DOLPHIN. Adding global constraints during training further improved both metrics: exact-set match rose from 26.84% to 27.42%, and Set-F1 from 38.68% to 39.73%. However, ILP reduced exact-set match, even though it improved Recall@5 in Table 3. These metrics measure different outcomes: Recall@5 rewards ranking gold objects highly, whereas exact match requires returning all and only the gold objects. To understand why ILP reduces exact-set accuracy on VQAR, we performed an additional analysis with NeSyCAPE. On 629 VQAR questions, program execution selected the correct object, but ILP returned an empty set. In 627 of these cases, the model’s atomic predictions gave a higher score to a rule-consistent assignment in which no object matched the query than to any assignment in which an object matched. Therefore, this makes the ILP select incorrect predictions. We will provide the details of this study in the appendix.

[1] Naik et al. DOLPHIN: A Programmable Framework for Scalable Neurosymbolic Learning. Proceedings of the 42nd International Conference on Machine Learning, 2025.

We hope our responses and additional results address your concerns, and we would appreciate it if you would consider revising your assessment of our paper based on this new evidence.

Comments on the paper

2 comments

Dear Reviewers, as the rebuttal deadline is approaching, please actively engage in the discussion based on the authors' responses. Thank You!

We appreciate the reviewers’ time and constructive feedback. Below, we summarize the main strengths recognized by the reviewers and the additional analyses we shared during the discussion. The revised manuscript highlights changes and new results in blue. Given the limited rebuttal time, some additional seed runs, timing analyses, and baseline evaluations could not be completed. We report the results that we were able to complete and verify.


Strengths mentioned by the reviewers

  • A unified neuro-symbolic framework. All three reviewers recognized the value of integrating program execution and global constraints during both training and inference.

  • Evaluation across three tasks. Reviewer 15Bn highlighted the evaluation across temporal reasoning, spatial VQA, and knowledge-based VQA.

  • Balanced and honest reporting. All reviewers noted that the paper reports both gains and regressions, showing where the framework helps and where it falls short.

  • Promising results with LLM-generated constraints. Reviewer C8RR highlighted that generated constraints performed comparably to knowledge-base rules on VQAR, suggesting that they could reduce reliance on manually specified rules.

Additional analyses provided during discussion

  • Additional baselines. We added NePTune and DOLPHIN on 3D-FORCE, DOLPHIN on VQAR, and Scallop baselines on MATRES. We also added SFT baselines for 3D-FORCE and VQAR that use the same visual inputs and training questions as NeSyCAPE. (Tables 1–3; Sections 3–4; Appendix A.6)

  • Training variability. We report means and sample standard deviations where three training seeds were available. The repeated runs show where gains hold across seeds and where the effects remain mixed, consistent with the original finding. (Tables 1–3)

  • Graph and constraint examples. We added a task-graph example and representative constraints for MATRES, 3D-FORCE, and VQAR, distinguishing KB-derived rules from LLM-generated rules. (Appendices A.2 and D)

  • Motivation and related work. We revised the introduction and related work to clarify the motivation and explain how NeSyCAPE differs from existing neuro-symbolic frameworks in its use of constraints during training and inference. (Sections 1–2)

  • ILP scalability. We measured how formulation size and ranking time change with the number of candidate answers. In a preliminary 3D-FORCE REF study to help scaling of ILP, one-hop pruning more than halved answer-finding time with nearly unchanged accuracy, though some constraints violations remained. (Section 4; Appendices C.4–C.5)

  • Effects of GC and ILP. We added GC-weight sensitivity and constraint-violation analyses on MATRES and 3D-FORCE REF. The results suggest that the effects of GC and ILP depend on how consistency relates to correctness and consistency in the model’s predictions: reducing violations can improve answer accuracy, but it can also leave accuracy unchanged or lower it. (Section 4; Appendix C.6)

  • Semantic validation. We documented how we checked generated programs and constraints. We provide the sample size, and error rate of the current author review. (Section 3; Appendices A.2–A.3). We are also planning to provide an independent audit in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.