- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The main contribution of this paper is to improve the robustness of combining LLMs with formal reasoning systems in neuro-symbolic systems, by having the LLM judge whether an entailment relation (including commonsense relations) holds between semantically related predicates after formalization. The method works backward: starting from the goal to be proved, it traces back to where the proof gets stuck and has the LLM judge and resolve those points. The authors report performance on five logical reasoning benchmarks with four LLMs, and ablate the effectiveness of the goal-directed backward search, as well as of the the grounding table (type information plus natural-language templates).
Critical strengths
- The problem the paper addresses is real and clearly described. It is a problem the neuro-symbolic field genuinely needs to solve, and the proposed approach is easy to follow.
- The ablation studies demonstrate the effectiveness of the grounding table and the goal-directed backward search, with clear accuracy gains.
Critical weaknesses
-
The abstract and introduction frame the contribution around the observation that semantically related expressions may be mapped to symbolically unrelated predicates. This problem, however, is already the target of several existing works and does not by itself distinguish this paper. SymBa (Lee & Hwang, NAACL 2025) likewise uses a solver-controlled backward proof and invokes the LLM only when the proof stalls. LLM-TRes (Toroghi et al., EMNLP 2024) and LLM-TH (Toroghi et al., ICLR 2025) use an entailment model during proof search to connect symbolically mismatched predicates, and LLM-TH further uses type information to prune the search space. ARGOS (Cotnareanu et al., ICLR 2026) augments the solver with commonsense supplied by the LLM. SymBa and LLM-TRes are cited, but the paper does not explain how it differs from them; LLM-TH and ARGOS are not cited at all. Consequently, it is unclear to readers what the actual novelty of this paper is, and the contribution needs to be articulated more precisely. Moreover, the baselines are all fairly dated; could more recent ones be included?
-
The cost and reliability of the LLM Gate are not sufficiently analyzed. Regarding cost, I suggest reporting the average number of Gate calls and the number of bridges used per problem. Regarding reliability, additional evidence would be valuable, such as the precision of accepted bridges and the proportion of gold-UNKNOWN instances that are incorrectly proved.
-
Certain figures are difficult to follow. In Figure 2, for example, symbols such as and cannot be understood from the figure alone. I encourage the authors to check and improve these figures.
Questions for authors
- Could the authors include some more recent baselines? The baselines in the main table seem fairly dated to me.
- Could the authors provide further analysis of the cost and of the reliability of the LLM Gate?
- Could the authors state more clearly how this work differs from related works such as SymBa, LLM-TRes/LLM-TH, and ARGOS, so as to make their own contributions clearer?
General Response from the Authors
We sincerely thank the reviewers for their valuable feedback and constructive suggestions. In response to their insightful comments, we have carefully revised the manuscript, with particular emphasis on technical details, experimental evaluations, and the corresponding discussions and analyses. The main revisions are summarized as follows:
We look forward to further discussion with the reviewers and would be happy to address any additional questions or comments to further improve the manuscript.