- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper propose RAVEN to address two problem in logical reasoning: premise order and source-faithfulness. Experiments show that it can outperform baselines.
Critical strengths
- The targeted problem is important, especially source faithfulness.
- Experiments show that it improves over baselines.
Critical weaknesses
-
The writing could be substantially improved in terms of clarity, logical flow, and terminology. For example, the introduction first describes the complementary capabilities of LLMs and symbolic solvers in neuro-symbolic reasoning, but the first reliability boundary does not clearly follow from this discussion. Specifically, the paper states that solvers execute formal theories, but then discusses how premise ordering affects the model's direct predictions, making it unclear whether the order sensitivity arises from the LLM or the solver. To my understanding, the solver itself should not be sensitive to premise ordering; rather, the LLM-generated formalization may change. This distinction should be made explicit. In addition, key terms such as "source," "source-preserving," and "source support" are introduced without sufficient explanation, making the methodology difficult to follow. More importantly, the central motivation and framing need stronger justification. The paper appears to address two relatively distinct problems: premise-order sensitivity and translation faithfulness. However, the introduction frames them together as two reliability boundaries of neuro-symbolic reasoning without clearly explaining their conceptual connection or why they should be addressed within a unified framework. Clarifying this relationship would help establish a more coherent motivation for the proposed method.
-
Methodology needs further clarification, especially the source-faithfulness part. The source-faithfulness certification relies on a trusted compiler or registered typed oracle, yet the paper does not sufficiently specify how these components are constructed or how typed semantic equivalence is actually checked. Since this Translation obligation is central to the paper's certification guarantee, its implementation and trust assumptions need to be made explicit. In particular, it remains unclear whether the checker relies on benchmark-specific reference formalizations and how the proposed certification can generalize to natural-language inputs without such trusted semantic evidence.
Questions for authors
Could the authors further clarify the connection between the two research questions, premise-order sensitivity and source faithfulness, and explain what motivates the design of your source-faithfulness method as well as the implementation?
4 comments
W2.3. Generalization and guarantee scope
The guarantee concerns an exact source-to-response chain under admitted semantic evidence and sound replay. Without such evidence, neither solver execution nor agreement across orderings can establish source faithfulness. We therefore make no claim of unrestricted natural-language assurance: native certification is not available for FOLIO or ProverQA under the current profiles, and the revised Limitations paragraph says so. Conditional on faithful admitted Translation evidence, sound execution, and correct enforcement of all release obligations, the checker is designed to prevent incorrect accepted responses. Empirically, native release authorizes 10,901 of 19,790 finite-Horn Core responses (55.08%) with no observed accepted error. Coverage therefore measures how often this conditional certification can be established. A source-only compiler plus solver answers the supported banks without model calls (Table 36). The certificate therefore audits a supplied model response and its bindings; it does not replace a solver.
Q1. Shared motivation and checker design
Reordering and certification address different consequences of the same translation uncertainty. Reordering provides additional evidence when the LLM's interpretation depends on premise order, but neither agreement nor solver execution proves that the generated theory is faithful to the input. This is why RAVEN separates selecting an inference label from authorizing its release. The certifier checks the frozen chain from source to formalization, solver witness, displayed reasoning, and final answer. The check follows translation validation and proof-carrying code: it does not trust the model's translation and compares it with independent evidence. On ProofWriter-full, solver replay alone accepts 137 incorrect responses that the full checker withholds (Section 5.3.1), which shows that local proof validity is insufficient. The revised Introduction, Figure 1, and Appendix E make the common motivation and conditional scope of the two contributions clearer, and we hope this addresses the reviewer's concerns about methodological coherence.
Dear Reviewer xt1Q,
Thank you again for your constructive review. We would appreciate your feedback on our response and revised manuscript.
The revised Introduction and Figure 1 clarify that premise-order sensitivity arises in the LLM-generated formalization, while logical consequence for a fixed theory remains order invariant. They also explain the connection between the two contributions: reordering supplies additional formalization evidence, whereas certification checks whether the selected response is bound to the source. Appendix E.2 now specifies the benchmark-specific Translation evidence, typed comparison procedure, and trust assumptions. We explicitly state that full native certification is unavailable on FOLIO and ProverQA without admitted semantic evidence.
We would welcome your assessment of whether these changes resolve the concerns about motivation, implementation, and scope, and are happy to clarify any remaining questions.
Dear Area Chair and the Reviewer, We sincerely thank the reviewer for their constructive comments, which have significantly improved the readability of our paper. We have clarified the sections that may have caused misunderstanding in the previous version, ensuring our ideas and results are clearly presented. In the revised manuscript and our responses below, we have addressed all of the reviewer's comments and provided additional results. We look forward to your feedback and a reconsideration of the rating. Thank you very much!
We thank the reviewer for the constructive suggestions on motivation, terminology, and source-faithfulness checking. The original paper separates LLM-generated theories from solver execution and states a conditional certification guarantee (Sections 3-4). The revision brings this distinction into the Introduction and Figure 1, and specifies the dataset-dependent Translation evidence in Appendix E.2.
W1. Clarity and motivation
We clarified that premise-order sensitivity arises in the model's translation and not in the solver, and we explained why this and translation faithfulness concern the same object, the theory the model generates from the text. We moved the definitions of "source", "source-equivalent ordering" and "source support" and a concrete ProofWriter example (Figure 1) to the start of the paper, and added a glossary. Empirically, disagreement across orderings is a weak error signal (AUROC 0.50-0.66), and reversal and random ordering have similar point estimates, so verification is needed beyond agreement. On ProofWriter-full, solver replay alone accepts 137 wrong answers that the full source-bound check withholds.
W1.1. LLM sensitivity versus solver invariance
Logical consequence for a fixed formal theory is invariant to premise order, as assumed by RAVEN. The order-sensitive component is the LLM's generated theory Z_p, which the solver then executes. The revised Introduction states this distinction explicitly, and Figure 1 (p. 2) illustrates it on a recorded ProofWriter item: the observed order replaces "Anne is quiet" with ¬nice(anne), yielding Unknown, while the reversed order preserves quiet(anne) and allows the solver to prove True.
W1.2. Linking ordering and faithfulness
Both concern the same failure boundary: the mapping from natural-language premises to a formal theory. If two source-equivalent orderings are translated faithfully, complete execution under the same registered semantics must return the same semantic label. A disagreement between such completed solver labels therefore shows that at least one translation is unfaithful, although it does not identify which view is wrong. Agreement does not establish faithfulness because both views may share an error. This motivates RAVEN's two stages:
Disagreement alone is a weak error signal (AUROC 0.50-0.66, Appendix D.5), and reversal and random ordering have similar point estimates (Table 16). This is why verification, not agreement alone, is necessary.
W1.3. Terminology
We now define "source" and "source-equivalent ordering" explicitly in the opening paragraph of the Introduction and clarify "source support" when introducing the inference workflow. Figure 1 explains its notation directly, and Appendix I (Table 46) consolidates the terminology and symbols. We have also made these terms more consistent throughout the paper.
W2. Translation checks and trust assumptions
We specified the trusted semantic evidence used for each benchmark (Appendix E.2, Tables 35-36), how typed comparison is performed, and the scope of the guarantee. The evidence is benchmark-specific: compilers or the benchmark's own annotations for ProofWriter and PrOntoQA, and none for FOLIO and ProverQA, where native certification is therefore unavailable. Under admitted evidence, native release authorizes 10,901 of 19,790 finite-Horn responses (55.08%) with no observed accepted error.
W2.1. Construction of Translation evidence
The original Section 4 and Appendix E described compiler- or oracle-backed Translation checks. Revised Appendix E.2 now specifies the evidence construction and access conditions for each benchmark:
The model that generates the formalization receives neither reference formalizations nor scorer labels. The benchmark-supplied annotations are privileged semantic evidence at the certification stage, and we state this explicitly.
W2.2. Typed semantic comparison
The checker verifies source-ID and query alignment, parses each aligned formula into a typed abstract syntax tree, and compares normalized signatures premise by premise. Appendix E.2 describes the supported grammar, the normalization, and the known limitations. This is a restricted validation criterion under registered semantics, not a procedure for proving equivalence between arbitrary English and logic.