acceptodds
Withdrawn from ICLR 2027

Beyond Executability: Schema-Adaptive Alignment for Faithful Natural-Language-to-Symbolic Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Formalize-then-solve pipelines improve LLM reasoning by translating natural-language problems into symbolic artifacts and delegating inference to external solvers. However, we identify a hidden reliability gap: a generated artifact may be syntactically executable while still violating the schema-level commitments it declares. We call this failure mode Silent Schema Inconsistency. Such errors are especially dangerous because they do not necessarily trigger parser errors, solver crashes, or execution feedback, leaving execution-driven refinement structurally blind to the underlying semantic mismatch. To address this problem, we propose Schema-Adaptive Constraint Alignment (SACA), an oracle-free, feedback-free, and inference-free pre-execution alignment layer for LLM-generated symbolic artifacts. Instead of asking whether an artifact can run, SACA asks whether it obeys its own declared schema: it detects the artifact schema, extracts a schema-specific constraint closure, and deterministically aligns recoverable violations before solver execution. We instantiate SACA for rule-style and CSP-style artifacts, enabling predicate-declaration completion, predicate-signature alignment, and domain–ordinal normalization. Across ProntoQA, LogicalDeduction, and ProofWriter, SACA improves symbolic reasoning by repairing distinct schema-level inconsistencies while introducing zero additional LLM calls. On LogicalDeduction, SACA improves FARA-CF from 70.0% to 91.0% on the diagnostic subset and improves Logic-LM from 54.7% to 80.7% on the full 300-example development split. Moreover, 7B symbolic-execution systems equipped with SACA outperform a larger 14B prompt-only model on structurally constrained settings, while SACA itself adds less than 0.5 ms of latency per example. These results show that reliable neuro-symbolic reasoning requires not only executable artifacts, but also systematic pre-execution enforcement of the schemas those artifacts declare.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
large language models, neuro-symbolic reasoning, symbolic reasoning, formalize-then-solve, schema alignment, logical reasoning, program repair

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TUmCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a reliability issue in formalize-then-solve pipelines: an LLM-generated symbolic artifact may be executable while still violating the schema-level commitments declared within the artifact, a failure mode termed Silent Schema Inconsistency. The paper proposes Schema-Adaptive Constraint Alignment (SACA), a deterministic pre-execution layer that constructs schema-specific consistency constraints from the generated artifact and repairs violations only when a unique repair is supported. The authors instantiate SACA for predicate-declaration completion, CSP domain–ordinal normalization, and predicate-signature alignment, without using gold labels, solver feedback, or additional LLM calls. Experiments on ProntoQA, LogicalDeduction, and ProofWriter show improvements over the corresponding unaligned systems, with the largest gains on LogicalDeduction; the same CSP alignment module is also applied to Logic-LM to demonstrate reuse across formalizers.

Critical strengths

1. The paper identifies a meaningful failure mode that execution-based validation can miss. The strongest evidence comes from LogicalDeduction with Logic-LM: adding SACA changes the executable rate only marginally while substantially increasing executable accuracy. This supports the paper’s central observation that a symbolic program may execute successfully while remaining internally inconsistent.

2. SACA is lightweight, conservative, and practically easy to insert into an existing formalize-then-solve pipeline. It requires no additional LLM calls, gold labels, or solver feedback, and applies a repair only when it is uniquely supported by the artifact itself. Applying the same CSP alignment module to both FARA-CF and Logic-LM also demonstrates useful reuse across different formalizers.

Critical weaknesses

1. The central claim of “schema adaptivity” is not sufficiently established because the actual consistency and repair logic remains manually specified for each schema. Extending SACA to a new symbolic schema requires defining schema-specific parsing, constraint construction, compatibility checking, repair enumeration, and repair application (Parse, Cl, Comp, R, and A). Thus, the current system appears closer to a unified interface over handcrafted schema-specific static checkers than to a method that automatically adapts to previously unseen schemas. The three evaluated datasets also correspond almost one-to-one to three predefined error types: missing predicate declarations, domain–ordinal mismatch, and predicate-signature mismatch. Reusing the same checker across FARA-CF and Logic-LM demonstrates formalizer-level reuse, but does not establish schema-level generalization. This substantially limits the novelty and generality of the proposed method.

2. The empirical evidence does not yet establish that Silent Schema Inconsistency is a broad and practically important failure mode, and the strongest result may largely arise from a fragile representation choice. SACA repairs 35/500 ProntoQA examples, 33/100 LogicalDeduction examples, and only 1/100 ProofWriter examples, while the corresponding FARA-CF gains are +2.6, +21.0, and +1.0 points. The main improvement is therefore concentrated in LogicalDeduction. In this setting, the artifact redundantly exposes both an ordinal meaning (e.g., newest) and a numeric rank, after which SACA corrects inconsistencies such as newest: bus == 5 using the already declared newest -> 1 mapping. A natural alternative is to let the LLM generate a canonical symbolic representation such as bus = NEWEST and deterministically compile NEWEST into the solver-specific numeric representation. The paper does not compare against such a representation, typed DSL, or standard static-checking baseline. Moreover, ProntoQA primarily involves render-blocking missing declarations rather than executable-but-silently-inconsistent programs, while ProofWriter triggers almost no repairs. A systematic error taxonomy is therefore needed to establish how much of the overall formalization error distribution is actually explained and repaired by the proposed failure mode.

Questions for authors
  1. For a completely new symbolic schema, how much manual work is required to implement SACA? Can the consistency and repair logic be automatically derived from a declarative schema specification, rather than manually defining Cl, Comp, R, and A? This would materially affect my assessment of the claimed schema adaptivity.

  2. Can the authors provide a complete taxonomy of formalization errors, including errors that are internally consistent but semantically incorrect? In particular, what fraction of all baseline formalization failures are Silent Schema Inconsistencies, and what fraction of those can SACA actually repair?

  3. For LogicalDeduction, can the authors compare against a canonical/typed intermediate representation in which the LLM generates semantic symbols such as NEWEST, followed by deterministic compilation to numeric ranks? If such a representation removes most of SACA’s gain, the current result would appear to reflect representation fragility rather than the necessity of a general alignment layer.

Reviewer_q7CAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Schema-Adaptive Constraint Alignment (SACA), a lightweight pre-execution layer for detecting and repairing internal schema inconsistencies in LLM-generated symbolic artifacts. Experiments on three datasets and integration with Logic-LM demonstrate improvements in reasoning accuracy with negligible overhead. However, the paper does not sufficiently establish the semantic benefits of internal consistency enforcement, clearly distinguish its methodological contributions from existing techniques, or demonstrate broad generalizability.

Critical strengths
  1. Important problem. The paper identifies a practical failure mode in formalize-then-solve pipelines: symbolic artifacts can execute successfully while violating their own declared schemas.

  2. Practical effectiveness. SACA achieves promising accuracy improvements across multiple reasoning settings, requires no additional LLM calls, and introduces less than 0.5 ms of reported alignment overhead per example.

Critical weaknesses

W1. Internal Consistency Does Not Guarantee Semantic Faithfulness

SACA enforces consistency with an artifact's own declarations and annotations, which may themselves be inconsistent with the original natural-language problem. Therefore, improved answer accuracy does not establish that the generated formalizations are more semantically faithful. The authors should evaluate repaired constraints against reference formalizations or independently annotated semantic representations, including cases where an artifact is internally consistent but semantically incorrect.

W2. Incomplete Related Work and Technical Positioning

The paper's claims regarding schema-level consistency checking and deterministic program repair are closely related to established work. Dafny provides specification-based verification of program functional correctness (Leino, 2010), while TBar uses predefined repair templates to automatically fix program defects (Liu et al., 2019). SACA appears to adapt and combine these established paradigms for LLM-generated symbolic reasoning artifacts, but the paper does not clearly establish what fundamentally new methodological insight this adaptation introduces. The authors should explicitly clarify how SACA differs from these approaches in its problem formulation, constraint extraction, repair mechanisms, and correctness guarantees, and identify which aspects constitute genuine technical contributions beyond applying existing verification and repair techniques to a new setting.

W3. Limited Evidence for Generalizability

The evaluation covers three datasets corresponding to three specific artifact schemas, with most diagnostic experiments conducted on fixed subsets. Although integration with Logic-LM provides encouraging evidence of portability, the current results do not establish generality across a broader range of formalizers, tasks, or schema types. Evaluation on additional systems and independently designed schemas would strengthen the claim that SACA is a general-purpose alignment framework rather than a collection of task-specific repair rules.

References

Leino, K. R. M. (2010). Dafny: An automatic program verifier for functional correctness. In E. M. Clarke & A. Voronkov (Eds.), Logic for Programming, Artificial Intelligence, and Reasoning (Lecture Notes in Computer Science, Vol. 6355, pp. 348–370). Springer. https://doi.org/10.1007/978-3-642-17511-4_20

Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019). TBar: Revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 31–42). ACM. https://doi.org/10.1145/3293882.3330577

Questions for authors

Please Refer to Weaknesses

Reviewer_Bqq3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds a failure mode in the LLM-based formalize-then-solve pipeline: the generated symbolic artifact can be executed by the solver, but it is internally inconsistent with its own schema declaration. The authors propose Schema-Adaptive Constraint Alignment (SACA). SACA is a deterministic pre-execution layer. It first detects the schema of the artifact, then extracts a schema-specific constraint closure, and finally applies uniquely determined repairs when the artifact violates its own declaration. SACA does not use oracle, does not use feedback, and does not use additional LLM calls.

Critical strengths
  1. This paper finds a meaningful failure mode that is not fully explored. The observation that an executable artifact can be internally inconsistent is important in practice. This observation is supported by a clear motivating example on LogicalDeduction.
  2. The method is efficient and easy to deploy. SACA does not add extra LLM calls, does not use solver feedback, and the runtime overhead is very small.
Critical weaknesses
  1. After reading the paper, my understanding is that the core operations of SACA are schema detection, constraint closure, and uniquely determined repair. I think these operations are very similar to type/schema inference, static dataflow analysis, and minimal or certainty-based program repair. The paper formalizes these operations, but except combining them for LLM-generated symbolic artifacts, it does not clearly explain what is new at the algorithm level. It is still not clear whether SACA is a new method or an engineering combination for a specific domain.
  2. This paper motivates SACA with faithful natural-language-to-symbolic reasoning, but SACA only enforces consistency with the artifact's own declaration. This creates an unchecked risk: if the schema declaration itself is wrong, SACA may "repair" other correct constraints to match the wrong declaration, and therefore strengthen the error. I did not see that the authors provide such cases for the behavior under corrupted schema declarations, missing ordinal annotations, or ambiguous repairs, so I am confused about the concrete runtime behavior. This is my biggest confusion.
  3. Too much AI involvement in writing and citation issues. This paper repeatedly restates its no-oracle / no-feedback / zero-leakage boundary in the main text and appendices, which makes the reading somewhat unnatural. The authors should be responsible for AI-generated content and guarantee that the content is valid and meaningful. In the Related Work section at line 135, there is still an unresolved citation.
Questions for authors
  1. The schema declaration and the symbolic constraints are both generated by the same model, so both may contain errors. When they disagree, how does SACA determine which part is actually incorrect? Does enforcing internal consistency necessarily improve faithfulness to the original natural-language problem? The paper motivates SACA with faithful reasoning, but the method only checks consistency inside the artifact itself.
  2. Supporting a new schema seems to require defining new parsers, consistency checks, and repair operations. What is the truly reusable component that allows the method to generalize beyond the specific symbolic representations considered in this paper? If every new schema needs a full set of hand-written operators, where is the "adaptive" part of Schema-Adaptive?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.