acceptodds
Under review as a conference paper at ICLR 2027

RAVEN: Reasoning Across Premise Orders with Source-to-Response Certification

10 views

Abstract

Logical consequence is invariant to premise order, yet language models are not: reordering the same facts can change both the predicted answer and the symbolic program executed by a solver. Consequently, a locally valid proof may still certify a mistranslated input. We introduce RAVEN, a neuro-symbolic reasoner that turns premise-order sensitivity into testable evidence. It produces direct labels and formal results from source-equivalent premise orderings, validates each formalization, repairs syntax or type failures, executes each theory, and checks proof and source support. When the orderings disagree, a fixed decision rule uses their checked evidence to select a label; a separate reasoning-release certifier subsequently binds the selected formalization, proof, displayed reasoning, and answer to the source. With Qwen2.5-32B, RAVEN's accuracy-first controller reaches 88.76% macro-average accuracy across four benchmarks. Full reversal improves the fixed two-ordering study by 2.60 points over its same-order control. Under a common source-bound certification protocol on the same 600 ProofWriter test items, 46.33% (278/600) of RAVEN-Core outputs and 9.83% (59/600) of SymbCoT outputs are accepted, with no observed error in either accepted subset. Finally, on 19,790 audited finite-Horn items under registered translation evidence, the Core-attached certifier authorizes 10,901 responses with no observed accepted error and a 0.035% item-level 95% Wilson upper bound.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neuro-symbolic reasoning, logical reasoning, premise-order sensitivity, formal verification, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xt1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose RAVEN to address two problem in logical reasoning: premise order and source-faithfulness. Experiments show that it can outperform baselines.

Critical strengths
  1. The targeted problem is important, especially source faithfulness.
  2. Experiments show that it improves over baselines.
Critical weaknesses
  1. The writing could be substantially improved in terms of clarity, logical flow, and terminology. For example, the introduction first describes the complementary capabilities of LLMs and symbolic solvers in neuro-symbolic reasoning, but the first reliability boundary does not clearly follow from this discussion. Specifically, the paper states that solvers execute formal theories, but then discusses how premise ordering affects the model's direct predictions, making it unclear whether the order sensitivity arises from the LLM or the solver. To my understanding, the solver itself should not be sensitive to premise ordering; rather, the LLM-generated formalization may change. This distinction should be made explicit. In addition, key terms such as "source," "source-preserving," and "source support" are introduced without sufficient explanation, making the methodology difficult to follow. More importantly, the central motivation and framing need stronger justification. The paper appears to address two relatively distinct problems: premise-order sensitivity and translation faithfulness. However, the introduction frames them together as two reliability boundaries of neuro-symbolic reasoning without clearly explaining their conceptual connection or why they should be addressed within a unified framework. Clarifying this relationship would help establish a more coherent motivation for the proposed method.

  2. Methodology needs further clarification, especially the source-faithfulness part. The source-faithfulness certification relies on a trusted compiler or registered typed oracle, yet the paper does not sufficiently specify how these components are constructed or how typed semantic equivalence is actually checked. Since this Translation obligation is central to the paper's certification guarantee, its implementation and trust assumptions need to be made explicit. In particular, it remains unclear whether the checker relies on benchmark-specific reference formalizations and how the proposed certification can generalize to natural-language inputs without such trusted semantic evidence.

Questions for authors

Could the authors further clarify the connection between the two research questions, premise-order sensitivity and source faithfulness, and explain what motivates the design of your source-faithfulness method as well as the implementation?

4 comments

We thank the reviewer for the constructive suggestions on motivation, terminology, and source-faithfulness checking. The original paper separates LLM-generated theories from solver execution and states a conditional certification guarantee (Sections 3-4). The revision brings this distinction into the Introduction and Figure 1, and specifies the dataset-dependent Translation evidence in Appendix E.2.

W1. Clarity and motivation

We clarified that premise-order sensitivity arises in the model's translation and not in the solver, and we explained why this and translation faithfulness concern the same object, the theory the model generates from the text. We moved the definitions of "source", "source-equivalent ordering" and "source support" and a concrete ProofWriter example (Figure 1) to the start of the paper, and added a glossary. Empirically, disagreement across orderings is a weak error signal (AUROC 0.50-0.66), and reversal and random ordering have similar point estimates, so verification is needed beyond agreement. On ProofWriter-full, solver replay alone accepts 137 wrong answers that the full source-bound check withholds.

W1.1. LLM sensitivity versus solver invariance

Logical consequence for a fixed formal theory is invariant to premise order, as assumed by RAVEN. The order-sensitive component is the LLM's generated theory Z_p, which the solver then executes. The revised Introduction states this distinction explicitly, and Figure 1 (p. 2) illustrates it on a recorded ProofWriter item: the observed order replaces "Anne is quiet" with ¬nice(anne), yielding Unknown, while the reversed order preserves quiet(anne) and allows the solver to prove True.

W1.2. Linking ordering and faithfulness

Both concern the same failure boundary: the mapping from natural-language premises to a formal theory. If two source-equivalent orderings are translated faithfully, complete execution under the same registered semantics must return the same semantic label. A disagreement between such completed solver labels therefore shows that at least one translation is unfaithful, although it does not identify which view is wrong. Agreement does not establish faithfulness because both views may share an error. This motivates RAVEN's two stages:

  • Inference: reordering provides another formalization candidate, evaluated through validation, solver replay, and source-support checks.
  • Certification: the selected response is checked against registered semantic evidence, so that the formalization, witness, displayed derivation, and final answer are bound to the source.

Disagreement alone is a weak error signal (AUROC 0.50-0.66, Appendix D.5), and reversal and random ordering have similar point estimates (Table 16). This is why verification, not agreement alone, is necessary.

W1.3. Terminology

We now define "source" and "source-equivalent ordering" explicitly in the opening paragraph of the Introduction and clarify "source support" when introducing the inference workflow. Figure 1 explains its notation directly, and Appendix I (Table 46) consolidates the terminology and symbols. We have also made these terms more consistent throughout the paper.

W2. Translation checks and trust assumptions

We specified the trusted semantic evidence used for each benchmark (Appendix E.2, Tables 35-36), how typed comparison is performed, and the scope of the guarantee. The evidence is benchmark-specific: compilers or the benchmark's own annotations for ProofWriter and PrOntoQA, and none for FOLIO and ProverQA, where native certification is therefore unavailable. Under admitted evidence, native release authorizes 10,901 of 19,790 finite-Horn responses (55.08%) with no observed accepted error.

W2.1. Construction of Translation evidence

The original Section 4 and Appendix E described compiler- or oracle-backed Translation checks. Revised Appendix E.2 now specifies the evidence construction and access conditions for each benchmark:

  • ProofWriter-test: a deterministic controlled-English compiler.
  • PrOntoQA: a unary controlled-language compiler.
  • ProofWriter-full: the benchmark's upstream logical annotations, aligned to stable source IDs and exact text.
  • FOLIO and ProverQA: no admitted evidence sufficient for full native Translation certification.

The model that generates the formalization receives neither reference formalizations nor scorer labels. The benchmark-supplied annotations are privileged semantic evidence at the certification stage, and we state this explicitly.

W2.2. Typed semantic comparison

The checker verifies source-ID and query alignment, parses each aligned formula into a typed abstract syntax tree, and compares normalized signatures premise by premise. Appendix E.2 describes the supported grammar, the normalization, and the known limitations. This is a restricted validation criterion under registered semantics, not a procedure for proving equivalence between arbitrary English and logic.

W2.3. Generalization and guarantee scope

The guarantee concerns an exact source-to-response chain under admitted semantic evidence and sound replay. Without such evidence, neither solver execution nor agreement across orderings can establish source faithfulness. We therefore make no claim of unrestricted natural-language assurance: native certification is not available for FOLIO or ProverQA under the current profiles, and the revised Limitations paragraph says so. Conditional on faithful admitted Translation evidence, sound execution, and correct enforcement of all release obligations, the checker is designed to prevent incorrect accepted responses. Empirically, native release authorizes 10,901 of 19,790 finite-Horn Core responses (55.08%) with no observed accepted error. Coverage therefore measures how often this conditional certification can be established. A source-only compiler plus solver answers the supported banks without model calls (Table 36). The certificate therefore audits a supplied model response and its bindings; it does not replace a solver.

Q1. Shared motivation and checker design

Reordering and certification address different consequences of the same translation uncertainty. Reordering provides additional evidence when the LLM's interpretation depends on premise order, but neither agreement nor solver execution proves that the generated theory is faithful to the input. This is why RAVEN separates selecting an inference label from authorizing its release. The certifier checks the frozen chain from source to formalization, solver witness, displayed reasoning, and final answer. The check follows translation validation and proof-carrying code: it does not trust the model's translation and compares it with independent evidence. On ProofWriter-full, solver replay alone accepts 137 incorrect responses that the full checker withholds (Section 5.3.1), which shows that local proof validity is insufficient. The revised Introduction, Figure 1, and Appendix E make the common motivation and conditional scope of the two contributions clearer, and we hope this addresses the reviewer's concerns about methodological coherence.

Dear Reviewer xt1Q,

Thank you again for your constructive review. We would appreciate your feedback on our response and revised manuscript.

The revised Introduction and Figure 1 clarify that premise-order sensitivity arises in the LLM-generated formalization, while logical consequence for a fixed theory remains order invariant. They also explain the connection between the two contributions: reordering supplies additional formalization evidence, whereas certification checks whether the selected response is bound to the source. Appendix E.2 now specifies the benchmark-specific Translation evidence, typed comparison procedure, and trust assumptions. We explicitly state that full native certification is unavailable on FOLIO and ProverQA without admitted semantic evidence.

We would welcome your assessment of whether these changes resolve the concerns about motivation, implementation, and scope, and are happy to clarify any remaining questions.

Dear Area Chair and the Reviewer, We sincerely thank the reviewer for their constructive comments, which have significantly improved the readability of our paper. We have clarified the sections that may have caused misunderstanding in the previous version, ensuring our ideas and results are clearly presented. In the revised manuscript and our responses below, we have addressed all of the reviewer's comments and provided additional results. We look forward to your feedback and a reconsideration of the rating. Thank you very much!

Reviewer_ZMGUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper integrates direct predictions with verified formal evidence across various premise sequences. It employs a distinct checker to determine if a frozen response may be released. The system distinguishes between semantic unknowns, unavailable evidence, and withholding, and it verifies the entire chain from the source text through formalization, proof, presentation, and the final output. Final achieves 88.76 macro accuracy on four tasks. Under a separate restricted finite-Horn release protocol, Core accepts 55.08% of items with no observed errors among accepted items. Evaluation encompasses paired statistics, order and budget controls, a common certification protocol, and injected faults.

Critical strengths
  1. The paper makes explicit checks on the difference between an executable proof and a faithful answer to the original question. It demonstrates through a natural-failure analysis why local proof validity alone is insufficient, and highlights the importance of translation and response binding (pp. 18-23).
  2. The evaluation encompasses paired comparisons, multiplicity correction, content deduplication, and a shared certification protocol. It details coverage losses, nonsignificant differences, and rejected correct answers to clarify the scope and cost of the assurance claims (pp. 13-19).
Critical weaknesses
  1. The strongest assurance relies on the trusted Translation interface. The document should provide more information about how the controlled compiler or typed oracle is implemented, its origin, supported datasets, and costs. Additionally, a direct trusted-compiler-plus-solver baseline is required to determine the value added by the language model when source translation is already accessible (pp. 20-22).
  2. The numerous controller versions and certification levels complicate tracking the conclusions. Final's headline accuracy and Core's native release coverage answer different evaluation questions. The order sweep uses one seed, with zero native release coverage on FOLIO and ProverQA. The current evidence more strongly supports a bounded certification system and useful diagnosis over broad natural-language reasoning assurance.
Questions for authors
  1. How is Translation implemented for each dataset, and does it use supplied logical forms or labels? And how does direct trusted compilation followed by solving fare in terms of coverage, accuracy, and costs?
  2. Which development items were used to choose Final's U rule, and how do they relate to each reported evaluation pool? Please show when the rule was frozen and how its development data relate to the evaluation splits, including how they are kept separate.
  3. Do the improvements over SingleView and formal sampling continue to hold across multiple seeds and randomized premise orders at matched realized compute?
6 comments

We thank the reviewer for the detailed assessment and for recognizing the distinction between an executable proof and a source-faithful response, together with the paired and multiplicity-corrected evaluation. The original guarantee is conditional on registered semantic evidence and sound checking; the revision makes the evidence sources and supported scope more explicit. We answer the weaknesses and questions in order, combining W1 and Q1 because they concern the same Translation interface.

W1 / Q1. Translation evidence and compiler baseline

We specified how the Translation evidence is obtained for each benchmark (Appendix E.2, Table 35) and added a source-only compiler plus solver baseline (Table 36). The evidence is benchmark-specific: a deterministic compiler for ProofWriter-test, a unary compiler for PrOntoQA, upstream logical annotations for ProofWriter-full, and none admitted for FOLIO and ProverQA. On the supported banks the baseline reaches 100% accuracy and coverage with no model calls, so the certificate's value is auditing a model's response, not outperforming a compiler.

W1.1. Benchmark-specific Translation evidence

Translation evidence is benchmark-specific: ProofWriter-test and PrOntoQA use deterministic source compilers, whereas native ProofWriter-full certification uses benchmark-supplied upstream logical annotations. Appendix E.2 (Table 35) specifies the evidence mode for each bank:

  • PW-test: a deterministic parser for controlled-English facts, negation, relations and Horn rules.
  • PQA: a unary controlled-language compiler.
  • Native PW-full: the benchmark's supplied upstream fact, rule and query annotations, aligned to exact source text and IDs.
  • FOLIO and ProverQA: no admitted full Translation evidence.

These annotations are privileged semantic information, even though they are not answer labels. Neither they nor scorer labels enter model generation (Section 3.3, Appendix G.1); the separate Translation checker is described in Appendix E.2. The checker verifies premise and query identity and multiplicity, parses aligned formulas into typed ASTs, and compares normalized signatures sentence by sentence. This is a restricted normalization criterion, not unrestricted English semantic equivalence; its morphology and local-consistency limitations are disclosed in Appendix E.2. Without admitted evidence, full certification remains unavailable.

W1.2. Compiler-only accuracy, coverage, and cost

The source-only compiler-plus-solver baseline solves the supported banks without model calls. PQA-dev is contained in PQA-matrix, so these denominators must not be summed as independent evidence. Source-only compiler-plus-solver baseline (manuscript Table 36).

BankItemsAnswer accuracy (%)Answer coverage (%)LLM calls
PW-test600100.00100.000
PW-full10,190100.00100.000
PQA-dev500100.00100.000
PQA-matrix9,000100.00100.000

CPU median latency is approximately 0.77-1.25 ms per item. We did not measure grammar-engineering effort, and we do not claim a matched GPU latency comparison.

W1.3. Value of response certification

The compiler-only baseline shows that an LLM is unnecessary for answering items within these supported controlled grammars. RAVEN's certificate audits a supplied model response and its source-to-answer bindings. Conditional on faithful admitted Translation evidence, sound execution, and correct enforcement of all release obligations, a passed certificate establishes source-to-response validity; the empirical question is how often a model response satisfies these conditions. Native release authorizes 10,901 of 19,790 finite-Horn Core responses (55.08%) with no observed accepted error. Under the evaluated FOLIO and ProverQA profiles, no Translation evidence sufficient for full native certification is admitted, so the current implementation issues no native source-to-response certificate on these benchmarks.

W2. Controllers, protocols, and certification scope

The protocol map below separates Final's inference accuracy, Core's native certification coverage, and the common Level-C comparison. These answer different evaluation questions; FOLIO and ProverQA have zero native certification coverage under the current admitted profiles.

W2.1. Evaluation protocol map

ProtocolQuestion it answersHeadline result
Final (accuracy-first controller)How accurate is inference on four benchmarks?88.76% macro (Qwen2.5-32B)
Core, native release (finite-Horn pool)How often can a Core response be released with source-bound checks?10,901/19,790 (55.08%), no observed accepted error
Common Level C (PW-test)How do methods compare under identical certification criteria?RAVEN-Core 46.33% (278/600) vs SymbCoT 9.83% (59/600)

The table maps each protocol to its reported results. Final differs from Core only by the terminal U-rule, with no extra model call. On 1,803 primary items it changes 64 labels: 36 wrong-to-correct, 19 correct-to-wrong and nine wrong-to-different-wrong. We report both accuracies. A Final-bound certification replay also authorizes 10,924/19,790 responses with no observed accepted errors, versus 10,901 for Core (Appendix D.3).

W2.2. Seed coverage and certification scope

The five-value ordering sweep (Table 7) is explicitly a one-seed study. The additional three-seed study in Q3 compares reversal with same-order and random second orderings; it does not replicate the full five-value sweep. The original guarantee is conditional on admitted semantic evidence and sound checking. Appendix E.2 now specifies the evidence available under each benchmark profile: FOLIO and ProverQA have no admitted full Translation evidence and therefore zero native certification coverage. Model support audits do not replace Translation. Section 4 also distinguishes label soundness from Global's use of a frozen model audit to bind the inference branch; unrestricted natural-language assurance is outside the stated guarantee.

Q2. Final's U-rule: development and separation

Q2.1. Development items

Thank you for asking for this separation. Final's terminal override was selected using FOLIO-v2 validation (203 items), ProofWriter development (600), and curated FOLIO validation (202), with Core/Final correct counts of 166/172, 590/594 and 163/177. Curated FOLIO contains 121 clean, 77 ambiguity and four extra-assumption items; it shares 125 exact items with FOLIO203. These banks provide development evidence, not independent confirmation of Final's gain. In particular, the FOLIO-v2 validation score reported for Final should be interpreted as development-influenced rather than an independently held-out estimate. Appendix G.3 (Tables 41-43) reports the selected rule, development overlaps, and post-hoc exclusion sensitivities.

Q2.2. Selection stage and rule conditions

Final's terminal rule is a development-stage choice based on the labeled banks in Appendix G.3 (Table 41), separate from per-item inference and from the replay-policy comparison in Table 15. The reported Final comparisons apply the same deterministic rule to recorded Core outputs, without additional model calls or reference-label access during replay. On a three-label task, it changes Core's selected label to Unknown only when the stable-direct branch is selected and exactly two formal views, observed and reversed, both return semantic Unknown. It requires neither a confidence threshold nor full response certification. Appendix G.3 documents the implemented rule and its development overlaps; the available historical evidence does not establish its selection sequence relative to every evaluation access.

Q2.3. Development and evaluation overlap

FOLIO203 is therefore developmental for this rule. The table distinguishes exact question overlap from reuse of the same premise context; zero exact overlap does not prove a complete historical non-access record.

Declared U-rule development overlap (Appendix G.3, manuscript Tables 41-42).

BankItemsRole for U-ruleExact dev-item matchesRows sharing dev contexts
FOLIO-v2 val203Development203203
FOLIO curated-full202Development202202
PW-dev600Development600600
PW-full10,190Reported bank6007,400
PW-test600Reported bank00
PQA-dev500Reported bank00
PQA-matrix9,000Reported bank00
ProverQA-hard500Reported bank00

PQA is absent from the declared development banks for Final's terminal U-rule, and its binary label contract makes the rule inapplicable. The separate replay-policy selection uses a 500-item PQA-matrix prefix that is disjoint from reported PQA-dev (manuscript Table 15). This replay-policy selection is distinct from the development-stage selection of Final's terminal rule.

Q2.4. Exclusion sensitivity and historical limits

Excluding all development premise contexts leaves 2,790 items: Core/Final obtain 2,747/2,759 correct (+0.43 points). The table includes the exact-item exclusion as a separate sensitivity.

PW-full sensitivity without retuning (manuscript Table 43).

Remaining populationnCore correct (acc.)Final correct (acc.)Δ (pp)
Full PW-full10,1909,955 (97.69%)10,010 (98.23%)+0.54
Exclude exact development items9,5909,365 (97.65%)9,416 (98.19%)+0.53
Exclude development contexts2,7902,747 (98.46%)2,759 (98.89%)+0.43

These analyses are post-hoc sensitivities, not prospectively untouched evaluations. The overlap tables document the observed data reuse and the effect of exclusion without retuning; they do not reconstruct the full history of data access, all candidate-policy comparisons, or the rule-selection sequence. We therefore distinguish development evidence from independent evaluation and do not claim that all reported evaluation items were unseen during policy development. Appendix G.3 makes these limitations explicit.

Q3. Seed robustness and compute controls

Across three seeds on the four primary banks, reversal adds 1.63 macro points over same-order resampling; its point estimate differs from random ordering by -0.04 points, at similar aggregate cost. A separate exploratory study under equal per-item allowances improves full-denominator accuracy, with additional correct outcomes concentrated in pairs with different budget-stop status. This is descriptive evidence under the shared allowance, not a causal attribution to budget stops or an unconstrained gain. Core-SingleView remains a separate pipeline comparison that the three-seed ordering study does not re-estimate.

Q3.1. Multiple seeds and random orderings

We completed a three-seed comparison using the same fixed staged-Core controller across all four primary endpoints, with seeds 11, 23, and 37. The table reports sample SD across seeds and realized consumption; differences are computed from unrounded values.

Three-seed staged-Core ordering study (manuscript Table 16).

Secondary orderMacro acc. (%) mean ± SDCallsInput tokensOutput tokens
Same order86.35 ± 0.4629,92413,611,2316,014,035
Reverse87.97 ± 0.4129,91713,581,3425,989,055
Random88.01 ± 0.7530,01613,712,2556,064,324

Reverse minus same is +1.63 points, with approximate paired seed/context-bootstrap 95% interval [0.68, 2.61]; reverse minus random is −0.04 [−1.24, 1.15]. Costs are similar in aggregate, not identical per item. Three seeds do not establish universal endpoint gains or reversal's superiority to random. The study is reported in Appendix B.1; it is separate from the historical one-seed sweep.

Reviewer_5JBiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RAVEN, a neuro-symbolic reasoner that turns premise-order sensitivity into testable evidence. RAVEN reverses the order and combines the direct and solver answers from both orderings with a fixed decision rule. A separate certifier then runs eight checks and either releases or withholds the response. With Qwen2.5-32B, RAVEN reaches 88.76% macro accuracy across four logical-reasoning benchmarks. On 19,790 ProofWriter and PrOntoQA items, the certifier has no observed errors.

Critical strengths

The paper targets an important problem. Neuro-symbolic pipelines are often treated as trustworthy because a solver executes the logic, but the paper shows two ways this trust can break: the LLM’s translation into logic can change just from reordering the premises, and a valid proof can still use a mistranslated input

Critical weaknesses

The paper is not clearly presented. 1) Writing wise: the introduction lacks a concrete example and does not clearly connect its two contributions (Examples are given only on page 7). The key ideas are abstract. It opens with “source-equivalent premise orderings” without properly defining the term. Figure 1 has terms like dp, Zp ,sp,Gp, Sp which are not defined when the figure is referred to. There’s no exploration of which orderings help or further explanations about proof-aligned order from Chen et al. (2024) 2) Setup is not clear too. This paper seems to rely on the setup that the translation is generated in one call( the authors should explicitly mention this), but we can always first fixing a shared vocabulary for the problem and then translating each premise independently against it and then we longer has order issue.

The effectiveness of the proposed method is not justified. 1) The reordering contribution is unclear: most of that gain comes from the solver pipeline itself, not from reordering. The single-order version already has high accuracy. Reordering adds little help on average and no gain on FOLIO and PQA. 2) The usage of the proposed checks is not clear. On the 19,790-item natural pool, every withheld wrong answer first fails either Translation or LocalStep (Table 18), and no natural response is first withheld by Artifact, Profile, Display, Terminal, or Scope. The authors could consider reporting how often each check catches real errors.

I have some concern for the practical usage. On ProofWriter-full, the full certificate eliminates 137 accepted errors at the cost of withholding roughly 6000 correct answers, reducing coverage from 98.68% to 36.08% (Fig. 3a). The authors should analyze whether these rejections reflect genuine mistranslations or overly strict matching against the reference. What exact kinds of translation errors does reordering expose?

Questions for authors

See the weaknesses.

6 comments

We thank the reviewer for the detailed assessment. We respond point by point below, identifying the analyses already present in the original submission and the clarifications and additional experiments in the revision. We hope this evidence addresses your concerns. Substantive revisions are highlighted in yellow in the revised manuscript.

W1.1. Presentation

We clarify the existing premise-order sweep (Section 5.2.2, Appendix B), the SingleView-Core comparison (Table 2), and the natural-failure and certification analyses (Appendix D), and provide additional experiments addressing your concerns.

W1.1.1. Concrete example and linked contributions

We moved a concrete example earlier and made the existing connection between the two contributions explicit.

The original Introduction (Section 1) and Figure 1 already presented premise-order reasoning and source-to-response certification as two linked stages. However, concrete order-sensitive examples appeared only in Figure 2 (p. 7). Revised Figure 1 (p. 2) now illustrates both stages using a recorded ProofWriter case: the observed ordering mistranslates "Anne is quiet" as ¬nice(anne), yielding Unknown, whereas reversal preserves quiet(anne) and supports True. The selected response then passes the registered release checks. The revised Introduction further clarifies that ordering affects LLM-generated formalizations, not the solver's logical result for a fixed theory.

W1.1.2. Definitions and notation

We moved previously defined concepts earlier and clarified their notation to present the ideas clearly.

The original Section 3.1 already introduced the source (X=(P,q)), generated theories (Z_p), solver labels (s_p), and machine witnesses (G_p). Section 3.3 also specified subset-minimal source support through solver replay and premise-deletion tests. These definitions were therefore present, although they appeared after the introductory figure. The revision explicitly defines source-equivalent orderings in the opening paragraph of the Introduction, expands Figure 1's notation, and consolidates the definitions in Appendix I (Table 46). Importantly, source support reproduces a definite solver result from translated premises; it does not by itself certify translation faithfulness.

W1.1.3. Ordering choices and proof-aligned order

In Section 5.2.2 and Appendix B, Table 6 of the previous manuscript, we already evaluated the five reordering strengths which are p ∈ {0, 0.25, 0.5, 0.75, 1} with the same Qwen2.5-32B model. That experiment showed that full reversal (p = 1) had the highest macro average (88.22%) on four benchmarks while FOLIO peaked at p = 0.25. The original Appendix B also noted the task-dependent pattern: FOLIO peaks at p = 0.25 and PQA is at ceiling across all five values.

The original sweep was limited to one seed. To assess robustness, we added a three-seed experiment (revised Appendix B.1, Table 16). Reversal adds 1.63 macro points over resampling the same order (approximate 95% interval [0.68, 2.61]), but we find no evidence that reversal outperforms a random permutation (reversal minus random: −0.04, interval [−1.24, 1.15]).

We therefore do not claim that reversal is universally optimal. Its role is to provide a reproducible second ordering without requiring a gold label. The original Introduction, Related Work, and Section 3.2 already distinguished RAVEN's label-free intervention from the proof-aligned ordering studied by Chen et al. (2024), which requires information unavailable to our inference procedure. The revision makes this distinction clearer and adds multi-seed evidence, without claiming that reversal is optimal across tasks.

W1.2. Translation setup

We clarify the joint translation call structure and directly evaluate the suggested shared-vocabulary, premise-wise alternative. Original Section 3.3 already specified an ID-aligned theory for each ordering, although the call structure was not explicit. The revision clarifies the one-call joint translation setup. In our single-seed comparison, our implementation of independent translation is less accurate than joint translation using the same vocabulary stage on all four benchmarks.

W1.2.1. Joint translation calls

In the original manuscript, Section 3.3 already specified that each ordering produces a formal theory (Z_p) covering the premises and query. The revised manuscript Section 3.3 now describes more clearly that each acquired ordering uses one joint formalization call over the complete problem, separate from the direct-label call, with at most one repair call. Joint translation permits a shared predicate vocabulary and cross-premise contextual interpretation while also allowing premise order to affect the generated theory.

W1.2.2. Shared vocabulary and independent translation

We evaluated the suggested alternative in an additional single-seed experiment: a source-only stage establishes a shared vocabulary, after which each premise and the query are translated independently. With an order-invariant signature held fixed, deterministic sentence-level translation and reassembly remove dependence on sentence presentation order. The experiment below compares answer accuracy for one implementation of this alternative; it does not independently establish end-to-end order invariance of the vocabulary-generation stage. Both arms use the same source-only signature procedure, and the joint arm is separate from the main RAVEN/SingleView results.

Shared-vocabulary comparison (Qwen2.5-32B, seed 11; accuracy %).

BenchmarkJoint translationShared vocabulary, premise-wiseDifference
FOLIO67.4951.72−15.76
PrOntoQA97.0070.20−26.80
ProofWriter91.3364.83−26.50
ProverQA56.6036.60−20.00

All items remain in the denominator, with unresolved outputs counted as wrong; differences use unrounded accuracies. This one-seed implementation comparison includes different repair behavior: the joint arm has a theory-level refinement round, whereas the independent arm retries sentence-level syntax/schema errors. The signature is prompt-provided rather than enforced by constrained decoding. We therefore do not attribute the entire gap to co-reference or semantic errors, or claim every independent translator must perform worse.

A shared vocabulary does not remove errors that are not lexical: in Figure 1 the correct predicate is available, yet the fact itself is replaced. A separate shared-vocabulary FOLIO example (item 1014, Bonnie; A: attending school events, S: being a student) keeps the signature but produces S ∧ (A ↔ S), adding the unsupported assertion S; the independent answer is True instead of gold Unknown, while joint translation returns Unknown. Replacing only that formula with (A ∧ S) ∨ (¬A ∧ ¬S) in a separate CPU diagnostic restores Unknown. These are mechanism examples, not an attribution of the full accuracy gap. These results do not establish that independent translation is universally inferior, given the single-seed setting and different repair procedures. They show that a fixed shared vocabulary and premise-wise translation do not by themselves guarantee source-faithful formalization.

W2. Reordering gains and checker utility

The original submission distinguished SingleView from Core (Table 2) and reported Core-over-SingleView gains of +5.95 and +6.31 macro points on Qwen3-8B and Llama-3.3-70B, respectively (Appendix B, Tables 8-10). Our new three-seed control isolates the incremental contribution of changing the second ordering, showing a +1.63-point macro improvement over same-order resampling (revised Table 16). We also extend the original natural-failure and mutation studies with common-candidate analyses, showing that Translation and LocalStep jointly withhold all 248 wrong parent responses in the audited pool (revised Tables 23-24).

W2.1. Pipeline gains versus ordering gains

The original submission reports full-controller gains of 5.95 and 6.31 macro points over SingleView on the 8B and 70B backbones (Appendix B, Tables 8-10). Our added three-seed control isolates the changed second ordering within the controller, showing a +1.63-point macro gain over same-order resampling.

We separate the contribution of the neuro-symbolic pipeline from the incremental effect of reordering. The original Table 2 already reported CoT, SingleView, Core and Final separately, allowing this attribution. On Qwen2.5-32B, much of the improvement over CoT indeed arises before acquiring a second ordering. However, the conclusion that multi-order reasoning offers only negligible additional value does not hold uniformly across the backbones.

Full-pipeline macro accuracies (Original Table 2 and Appendix B, Tables 8-10).

BackboneCoTSingleViewCore
Qwen2.5-32B73.2686.15 (+12.89)87.48 (+1.33)
Qwen3-8B71.1775.65 (+4.48)81.60 (+5.95)
Llama-3.3-70B68.0978.37 (+10.28)84.68 (+6.31)

Core improves over SingleView by 1.33, 5.95 and 6.31 macro points, respectively. These are full-controller comparisons, not isolated ordering effects. To isolate the contribution of a changed second ordering, we additionally conducted a three-seed study using the same staged controller and resource ceilings (revised Table 16). Reversal improves macro accuracy over same-order resampling by +1.63 points (approximate 95% CI [0.68, 2.61]).

We therefore make a more precise claim: reordering provides a modest but measurable additional benefit under the evaluated protocol, while the overall neuro-symbolic pipeline contributes separately.

W2.2. Task-dependent ordering gains

The FOLIO tie concerns the 32B Core-SingleView comparison; the original sweep already showed FOLIO’s order sensitivity (Appendix B, Table 6). Our added three-seed control gives a +2.30-point FOLIO point estimate for reversal over same-order resampling, while PrOntoQA remains at ceiling, supporting a task-dependent ordering benefit.

We distinguish the staged Core-SingleView comparison from the controlled ordering experiment. The reported absence of a FOLIO gain applies to the 32B staged comparison in original Table 2. However, original Appendix B, Table 6, already showed FOLIO sensitivity to premise ordering: accuracy increased from 80.30% at p=0p=0 to 84.24% at p=0.25p=0.25. The new three-seed study further reports a +2.30-point FOLIO accuracy difference for reversal versus same-order resampling. This endpoint result is distinct from the aggregate +1.63-point macro improvement (approximate 95% CI [0.68, 2.61]). We do not infer endpoint-specific significance from the macro-level interval.

PrOntoQA is at ceiling, and the 32B Core-SingleView comparison ties on FOLIO. The reversal-minus-random estimate is -0.04 points, so these results do not establish a uniquely best second ordering. The supported claim is a modest, task-dependent gain from a second source-equivalent ordering. Realized calls and tokens in the three-seed study are similar in aggregate, not identical per item; the separate same-controller allowance study is reported in Appendix B.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.