- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a reliability issue in formalize-then-solve pipelines: an LLM-generated symbolic artifact may be executable while still violating the schema-level commitments declared within the artifact, a failure mode termed Silent Schema Inconsistency. The paper proposes Schema-Adaptive Constraint Alignment (SACA), a deterministic pre-execution layer that constructs schema-specific consistency constraints from the generated artifact and repairs violations only when a unique repair is supported. The authors instantiate SACA for predicate-declaration completion, CSP domain–ordinal normalization, and predicate-signature alignment, without using gold labels, solver feedback, or additional LLM calls. Experiments on ProntoQA, LogicalDeduction, and ProofWriter show improvements over the corresponding unaligned systems, with the largest gains on LogicalDeduction; the same CSP alignment module is also applied to Logic-LM to demonstrate reuse across formalizers.
Critical strengths
1. The paper identifies a meaningful failure mode that execution-based validation can miss. The strongest evidence comes from LogicalDeduction with Logic-LM: adding SACA changes the executable rate only marginally while substantially increasing executable accuracy. This supports the paper’s central observation that a symbolic program may execute successfully while remaining internally inconsistent.
2. SACA is lightweight, conservative, and practically easy to insert into an existing formalize-then-solve pipeline. It requires no additional LLM calls, gold labels, or solver feedback, and applies a repair only when it is uniquely supported by the artifact itself. Applying the same CSP alignment module to both FARA-CF and Logic-LM also demonstrates useful reuse across different formalizers.
Critical weaknesses
1. The central claim of “schema adaptivity” is not sufficiently established because the actual consistency and repair logic remains manually specified for each schema. Extending SACA to a new symbolic schema requires defining schema-specific parsing, constraint construction, compatibility checking, repair enumeration, and repair application (Parse, Cl, Comp, R, and A). Thus, the current system appears closer to a unified interface over handcrafted schema-specific static checkers than to a method that automatically adapts to previously unseen schemas. The three evaluated datasets also correspond almost one-to-one to three predefined error types: missing predicate declarations, domain–ordinal mismatch, and predicate-signature mismatch. Reusing the same checker across FARA-CF and Logic-LM demonstrates formalizer-level reuse, but does not establish schema-level generalization. This substantially limits the novelty and generality of the proposed method.
2. The empirical evidence does not yet establish that Silent Schema Inconsistency is a broad and practically important failure mode, and the strongest result may largely arise from a fragile representation choice. SACA repairs 35/500 ProntoQA examples, 33/100 LogicalDeduction examples, and only 1/100 ProofWriter examples, while the corresponding FARA-CF gains are +2.6, +21.0, and +1.0 points. The main improvement is therefore concentrated in LogicalDeduction. In this setting, the artifact redundantly exposes both an ordinal meaning (e.g., newest) and a numeric rank, after which SACA corrects inconsistencies such as newest: bus == 5 using the already declared newest -> 1 mapping. A natural alternative is to let the LLM generate a canonical symbolic representation such as bus = NEWEST and deterministically compile NEWEST into the solver-specific numeric representation. The paper does not compare against such a representation, typed DSL, or standard static-checking baseline. Moreover, ProntoQA primarily involves render-blocking missing declarations rather than executable-but-silently-inconsistent programs, while ProofWriter triggers almost no repairs. A systematic error taxonomy is therefore needed to establish how much of the overall formalization error distribution is actually explained and repaired by the proposed failure mode.
Questions for authors
-
For a completely new symbolic schema, how much manual work is required to implement SACA? Can the consistency and repair logic be automatically derived from a declarative schema specification, rather than manually defining
Cl,Comp,R, andA? This would materially affect my assessment of the claimed schema adaptivity. -
Can the authors provide a complete taxonomy of formalization errors, including errors that are internally consistent but semantically incorrect? In particular, what fraction of all baseline formalization failures are Silent Schema Inconsistencies, and what fraction of those can SACA actually repair?
-
For LogicalDeduction, can the authors compare against a canonical/typed intermediate representation in which the LLM generates semantic symbols such as
NEWEST, followed by deterministic compilation to numeric ranks? If such a representation removes most of SACA’s gain, the current result would appear to reflect representation fragility rather than the necessity of a general alignment layer.