acceptodds
Under review as a conference paper at ICLR 2027

SATISFIABILITY IS NOT ENOUGH: FIDELITY-GATED REPAIR OF LLM-EXTRACTED HYBRID AUTOMATA

Abstract

Large language models can translate technical manuals into symbolic models, but verifier-guided correction creates a distinct failure mode: a proposer can make a model satisfiable by deleting the obligations that made it unsatisfiable. We call this solver gaming. We present a source-traced pipeline that extracts partially observ- able hybrid-automaton (POHA) graphs, checks discrete and algebraic constraints with an SMT (Satisfiability Modulo Theories) Solver, and places a deterministic fidelity gate before repaired graphs can become active. The gate preserves source traces, declarations, guards, effects, evidence, minimum durations, sensor status, epistemology scope, and safety classifications; source support for additions remains a separate evidence-review obligation. In a natural pre-gate run, an unconstrained loop reached Satisfiability (SAT) through ten unsupported safety downgrades; the gate rejected both affected edits. In a controlled 20-case benchmark, all 15 valid additive candidates were accepted and all 5 solver-satisfying effect deletions were rejected. Separately, we test the utility of raw, not fidelity-gated, graphs for down- stream decision support. Across a chosen dataset of 100 cyber-physical scenarios and five trials per scenario, graph augmentation improved exact decision accuracy from 77.33% to 87.33% on a challenge set and from 78.00% to 87.00% on a repre- sentative set. These results establish repair-integrity behavior and raw-graph utility as separate findings: they do not establish physical completeness or downstream benefit from repaired graphs. However, we note that formal graph structures can occasionally over-constrain valid actions, increasing False Rejection Rate (FRR) in exchange for strict safety compliance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic AI, Large Language Models, Formal Verification, Model Repair, Agentic RAG, Safety-Critical Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dqZGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the problem of auto-formalizing SMT models from technical manuals using LLMs. This is generally considered a very hard problem, specifically due to the lack of ground-truth formal models. Natural language is fairly ambiguous, and because of that, the same piece of natural language can be represented in many different ways in SMT. The authors argue that simply using an SMT solver to check if the model is satisfiable is not enough. The authors call this problem of removing constraints to make it satisfiable "solver gaming." They propose a pipeline that extracts what they call a partially observable hybrid automaton (POHA) and uses an SMT solver to identify inconsistencies. They introduce a fidelity gate, which is supposed to check whether a proposed repair does not delete existing constraints, weaken safety classification, change source references, or things like that.

There are two experiments in which the authors evaluate their method:

  1. Evaluating the fidelity gate on a small set of repair cases (15 valid additive, 5 destructive) and reporting that it accepts valid repairs and rejects destructive repairs.
  2. Checking whether including the raw extracted graphs in a RAG system improves decisions on automotive scenarios, where the authors are showing some improvements in decision accuracy.
Critical strengths
  • I think the general problem is very important. The idea of autoformalization is to convert natural language into a formal model and make sure that the representation in this formal model actually preserves the intended spirit of what was expressed in natural language. That is a pretty hard problem, and it is very important for all of neurosymbolic reasoning.
  • The observation that a solver can accept a specification that no longer reflects the source, along with the fact that lots of these reward-hacking things happen, is well documented in the literature as well, including in other formal settings with agents (for example, in math AI and the Lean language). This is an important problem statement, and that is a strength.
Critical weaknesses

Overall, I do not see sufficient scientific evidence to support the main contributions that the authors are making.

Novelty and relation to prior work From a theoretical standpoint, I do not understand what is fundamentally very new here compared to prior work. There are a lot of papers in the literature that have done more, including concepts of valid, satisfiable, invalid, and unsatisfiable [1] in prior work around auto-formalization and around finding uncertainty in many such auto-formalizations. The problem is well known.

The central observation here revolves around making a formal specification satisfiable, but that does not mean it is correct or that it preserves the original requirement. This is a fairly basic observation that we already know, but I struggle to understand why it requires a new formal framework. There is lots of existing work on how to write a specification, how to observe and maintain semantic preservation, and such problems. Could the same effect be achieved through straightforward structural comparisons? Finally, the paper spends a considerable amount of effort defining and motivating a very elaborate pipeline, much of which has already been explored in prior literature.

Necessity of the POHA formalism The paper is doing this four-dimensional decomposition and has a partially observable hybrid automaton representation, but I do not understand why this particular formalism is necessary for the repair mechanism. They are explicitly restricting their verification to graph structure and algebraic constraints. I want to know: compared to established ways of representing things in SMT, why is this additional machinery justified?

Limited evaluation of the fidelity gate On the empirical side, the validation of the fidelity gate is extremely limited. They are just adding 15 additive modifications, making 5 destructive modifications, and testing things. That is not the level of empirical evidence that is required for an ICLR venue paper.

Downstream evaluation does not test the repair method The downstream methods do not seem to evaluate the actual repair method at all. The decision accuracy can come from providing raw extracted graphs to the LLM without applying the repair process. It is hard to say that the central contribution of this paper, the fidelity gate, improves downstream behavior. If the formalization is better, of course, it will improve downstream behavior.

There is a comparison between text-only RAG and RAG augmented with the structured graph, but there is no comparison against simpler methods. All the downstream evaluation is also limited to 100 curated scenarios derived from automotive documentation. That is good because it provides a new benchmark, but it also does not test on the benchmarks in the existing neurosymbolic literature. There are a lot of papers around the same idea, and it does not actually check that it works on those datasets.

Overall, I do not think this currently meets the bar for an ICLR publication.

Please see some prior work here:

[1] A neurosymbolic approach to natural language formalization and verification. An et al.

[2] Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks. Ganguly et al.

Both papers are fairly comprehensive regarding prior literature review, and prior work and benchmarks for improvements can be found from there.

Questions for authors

If you are planning to re-submit this paper, I would recommend exploring the following questions.

  1. What is fundamentally new about this fidelity gate compared to prior work in the neurosymbolic literature on checking specification correctness and refining specifications?
  2. I would really like to understand how the proposed gate works when correct repairs require modifying or removing incorrect constraints in the original extraction, especially if you're going through many different data types. In SMT, you're going through the many different sorts you have previously defined.
  3. I would want to see a lot more evaluation across datasets used in the literature.
  4. I want to specifically understand what the proposed hybrid automaton representation adds. Would a simpler representation provide similar improvements over text-only RAG at retrieval time? I think so.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer dqZG: Novelty, Multi-Sort Gate & Baselines

We thank Reviewer dqZG for the detailed critique and references [1,2]. We address W1–W4 and Q1–Q4 below and in the revised paper.

W1 & Q1: Novelty vs. [1] An et al. & [2] Ganguly et al., and why structural checks fail

We added dedicated comparisons in Secs. 2 & 3.3:

  1. Vs. [1] An et al. (AWS ARc) & fixed-spec verifiers (Clover, nl2spec): In An et al., experts freeze an immutable model M∗\mathcal{M}^{\ast} offline and check outputs against fixed M∗\mathcal{M}^{\ast}. In reference-free spec repair (H0→H′\mathcal{H} _ 0 \to \mathcal{H}'), the LLM has write-access to the specification. Because meta-axioms take the implicational form ∀t:A(t)⇒B(t)\forall t: A(t) \Rightarrow B(t) (e.g., κt=True⇒Sensor(Pt)\kappa_t = \texttt{True} \Rightarrow \text{Sensor}(P_t)), Z3 returns SAT whether the LLM repairs B(t)B(t) or erases A(t)A(t) (flipping κt:True→False\kappa_t: \texttt{True} \to \texttt{False}, deleting tt, or injecting Pt′≡⊥P_t' \equiv \bot). Static SAT checks miss antecedent erasure; we formulate a cross-step relational refinement contract (H0⪯fidH′\mathcal{H} _ 0 \preceq _ {\text{fid}} \mathcal{H}', Eqs. 4–6).
  2. Vs. [2] Ganguly et al. (NeurIPS 2025): Ganguly et al. measure open-loop parse uncertainty (PCFG grammar entropy). Solver gaming is closed-loop reward hacking, not uncertainty: at T=0.0T=0.0, the LLM deterministically flips κt:True→False\kappa_t: \texttt{True} \to \texttt{False} or deletes guards to clear UNSAT cores.
  3. Why structural comparisons fail (Table 1): String inclusion (Pt⊆Pt′P_t \subseteq P_t') fails in both directions: (i) 85.7% FAR (30/35) when appending contradictions (speed > 0 and speed == 0) preserves strings while making Pt′≡⊥P_t' \equiv \bot vacuous (28.6% even with Z3 feasibility, solved to 0% [0/35] by Evidence-Assisted); and (ii) 100% FQR (0/10) when flawed initial guards need tightening (temp <= 31 →\to temp < 31) or revision (0% FQR [10/10] with our gate).

W2 & Q4: Necessity of the 4-pillar POHA formalism vs. simpler representations

In Sec. 3.1, we ground each pillar in classical formal methods (LTS Topology, EFSM/QF_LRA Ontology, Timed Automata δt>0\delta_t > 0 Temporality, POMDP/Shielding σt\sigma_t Epistemology) and ablate representations in Table 3 & App. F (Table 6):

  • RAG-Only: 76.3% (7 repair-affected) / 56.5% (all 13 scenarios)
  • RAG + Structured Text (JSON rules): 71.2% / 61.4% (fails topology on mirror folding 0.00)
  • RAG + Discrete FSM (states + boolean guards, no Real QF_LRA, δt\delta_t, or σt\sigma_t): 87.0% / 69.9% (solves discrete switches 1.00, but drops to 0.00 vs. POHA 1.00 on reserve-charge display and 0.375 vs. 0.625 on summer parking due to missing Real thresholds)
  • RAG + Fidelity-Gated POHA: 92.6% (7 repair-affected) / 72.9% staged base across all 13, rising to 88.3% with +Verifier diagnostics (Sliding Door 0->1) and σt\sigma_t re-anchoring (Steering 0->1; Apps. C.2 & F).

W3, W4 & Q3: Expanded repair evaluation, downstream repair impact & literature datasets

  1. Expanded Repair Evaluation (Sec. 5.1, Table 1, App. E): Beyond the 20-case 4-pillar suite (15/15 valid, 5/5 gamed), we evaluate real manual sections across splits (7/7 gamed dev/val sections quarantined, blocking 28 silent κt\kappa_t flips, plus live σt\sigma_t re-anchoring on Steering; Table 4), a 60-case multi-sort pool (35/35 invalid rejected, 20/20 valid accepted, 5/5 unsupported abstained), and 10 live closed-loop tasks (50 LLM calls across 5 policies; 10/10 repaired vs. 0/10 syntactic; Table 5).
  2. Downstream Impact & Datasets (Secs. 5.3 & 6, Table 3): RAG + Ungated Repair drops accuracy by -21.7 pp (92.6% -> 70.9%), causing unsafe ALLOW decisions on summer parking (0.625 -> 0.250). Standard NL-to-FOL datasets in [1,2] (e.g., FOLIO, PrOntoQA) test single-turn open-loop entailment rather than closed-loop hybrid state-transition repair over multi-page CPS manuals (Sec. 6).

Q2: Modifying/removing incorrect initial constraints across Bool, Real & String

Detailed in Secs. 4.1–4.2 & App. E.2 (Table 5): our Z3 compiler encodes Bool, Real (QF_LRA intervals and linear updates xnext=x+cx _ {\text{next}} = x + c), and String enums, catching 100% of cross-sort conflicts. When an initial guard is flawed: (i) valid tightenings pass via Z3 implication SAT(Pt′)∧UNSAT(Pt′∧¬p)\text{SAT}(P_t') \land \text{UNSAT}(P_t' \land \neg p); (ii) contradictory (request and not request), over-restricted (and ambient), or loose-enum (!= "blocked" -> == "ready") guards in Z3's local UNSAT core or witness check (Auth(t,F,WM)\text{Auth}(t, F, W_M)) are unlocked for replacement while Preservectx\text{Preserve} _ {\text{ctx}} locks states, invariants, effects, κt\kappa_t, and δt\delta_t (10/10 live tasks repaired; 0 context damage).

Reviewer_dqZGOpenReview ↗

Maintaining my score

Thank you for the response. It seems to me that the method under review has been changed during the rebuttal, and there is now a new Z3-based gate. This means that most of the numbers in the rebuttal come from different scenarios and a different experimental methodology compared to the original submission, so they cannot be compared with what was originally reviewed.

In fact, I am not even sure how they are all presented in the paper at once without rerunning all the experiments, since some results appear unchanged from the original submission.

As I mentioned in my original review, there is a very elaborate formal setup, and I am still not convinced that there is anything novel about it.

The rebuttal's comparison with the papers I mentioned uses terminology that is hard to follow. Overall, the rebuttal and the revision read like a lot of word spaghetti: they are unnecessarily dense, and it is hard to keep track of how much AI writing has been done.

AuthorsOpenReview ↗

Response to previous concerns/queries

Thank you for your feedback. We apologize if our previous reply was overly dense, and we clarify each point below if it helps:

  • Did the gate change, or did we add a Z3 feasibility check?

    • We did not replace our pipeline or method. In the original submission, Step 3 already compiled graphs into Z3, and Step 4 (the Fidelity Gate) already locked source traces, states, effects, durations, sensor tags, and is_safety_critical, while checking guards by string matching. During the rebuttal, we simply connected our existing Z3 compiler to that guard check—adding a Z3 transition-feasibility check (to block contradictory dead guards) and UNSAT-core guard unlocking—specifically to answer your Q2 on how to modify or remove wrong initial guards. We kept the original string-matching gate side-by-side in Table 1 so both versions are directly compared on the exact same tasks.
  • Which experiments are original (unchanged) vs. newly added for the rebuttal?

    • None of the original experiments were removed or altered:
        1. Original & unchanged: The 20-case repair benchmark (15/15 valid, 5/5 destructive), the validation manual repair audit (Section 5.1), and the 100-scenario RAG evaluation (77.33% -> 87.33% and 78.00% -> 87.00%, Section 5.2, Table 2). Table 2 did not need rerunning because it tests text-only RAG against raw Step-1 extracted graphs before the Step-4 repair loop ever runs. Moreover, on all 7 manual sections where repair ran (Table 4), both the original gate and the Z3-extended gate reject the gamed edits and keep the exact same raw graph.
        1. Added in rebuttal per reviewer requests: Table 1 (testing wrong initial guards across Bool, Real, and String sorts, per your Q2) and Table 3 (comparing Ungated Repair, Structured Text, Discrete FSM, and Fidelity-Gated POHA on repair-affected scenarios, per your W4 and Q4 and also some queries from other reviewers).
  • What is the core novelty?

    • Standard LLM + SMT repair loops stop as soon as the edited model is SAT. Unlike Lean proof repair (where a human locks the target theorem) or code verification like Clover (Sun et al., 2024, where specs are cross-checked against executable code), technical manuals have neither a human-written formal spec nor reference code: the LLM edits its own imperfect SMT model after Z3 reports UNSAT, and often reaches SAT by deleting rules or flipping is_safety_critical from True to False ("solver gaming"). Our experiments show that all three "obvious" solutions fail:
    1. An unconstrained SMT repair loop (Verifier-Only, stopping as soon as Z3 returns SAT with no fidelity gate) accepted all 7/7 solver-gamed manual sections (28 silent safety-flag flips plus guard/effect deletions) and 100% of SAT gamed candidates, dropping downstream accuracy to 70.9%—worse than plain text-only RAG (76.3%) without a solver.
    2. Anti-gaming prompts (Prompt-Only) stop flag flips, yet the LLM still games Z3 by deleting conflicting transitions and postconditions (Wipers, eCall).
    3. Straightforward structural diffing (Syntactic Gate, directly testing the question in your original review) fails in both directions: it rejects 100% (10/10) of legitimate fixes when an initial LLM guard is wrong (e.g., tightening temp <= 31 to temp < 31), while still accepting 85.7% (30/35) of bad edits when the LLM appends a contradiction (speed > 0 and speed == 0). Our contribution is identifying this failure mode and resolving the two-way trap: using Z3 UNSAT cores and text boundary witnesses to unlock only the broken clauses while checking Z3 implication and pre/post-state satisfiability on everything else. This turns 0/10 valid repair and 85.7% false accepts into 10/10 valid repair and 0.0% false accepts, lifting downstream accuracy from 70.9% to 92.6% (+21.7 points).
  • Comparison with [1] An et al. and [2] Ganguly et al.:

    • In An et al. [1], humans inspect and freeze a trusted formal specification first, and the solver checks answers against that fixed specification. In our setting, there is no human-frozen specification yet—the LLM is editing its own SMT model inside a repair loop. In Ganguly et al. [2], grammar entropy detects when an LLM is uncertain during a single-pass translation. In our repair loop, even at temperature 0 (high confidence), the LLM deletes rules or flips safety flags to make Z3 return SAT, so uncertainty scores alone do not catch it.
  • On writing and AI polishing:

    • Consistent with our paper's AI Use Statement, we designed and ran all experiments and drafted the responses, and used an LLM tool to condense and polish wording to fit OpenReview's 5,000-character limit. Compressing eight detailed answers into that limit made our previous reply read too densely; we wrote this clarification directly in simple terms so the distinction is clear. If you have any further questions, we will be happy to clarrify each sections.
Reviewer_VoqHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a neurosymbolic verification approach to ensure LLM-generated repairs guarantee the preservation of original specifications instead of satisfying an SMT solver. The core idea proposed in this paper is clearly communicated and profound in fidelity gate. The authors demonstrate the practicality of their proposed framework using elaborate set of experiments.

Critical strengths
  1. The authors articulated succinctly the existing limitations in LLM-based formal verification frameworks where satisfaction of requirements using smt solvers can be achieved at the expense of preserving the original requirements.
  2. The authors also discussed deterministic fidelity gate as a feasible mechanism to prevent potential violations.
  3. The authors provided enough motivation in the discussion of the distinction between repair integrity and downstream decision accuracy.
  4. Again, the authors demonstrated through their experiments that the proposed framework improve decision accuracy with the inclusion of unsafe scenarios.
Critical weaknesses
  1. The authors used only 20 use cases in the demonstrating the effectiveness of the proposed framework. Though elaborate, the 20 use cases are not extensive in covering all scenarios, thereby limiting the evaluation of the fidelity gate. Hence, the evaluation of the framework does not demonstrate end-to-end effectiveness. Again, the authors used only raw graphs in the experiments performed downstream. The authors should really demonstrate how effective their approach improves downstream decision-making using fidelity-gated repaired graphs. The authors should also evaluate more diverse repair failures and more complex formulae.

  2. A comparison of the proposed framework with established benchmarks and baselines are missing. Such comparisons will definitely go a long way to enhance the quality of the paper.

  3. Although the proposed framework leverages hybrid automata, the SMT verification approach proposed does not capture continuous dynamics or reachability. It is imperative that the authors discuss the implication of this gap on critical applications.

Questions for authors
  1. Doesn't the strict preservation policy place limitation on the practical deployment of the tool in real-world applications?
  2. Can the authors clarify what mechanism was employed to know if the original specification was correct?
  3. How does the framework reason about continuous dynamics?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer VoqH: Expanded Benchmarks, Baselines & Hybrid Scope

We thank Reviewer VoqH for the thoughtful review and recognition of our solver-gaming discovery, verifier-agnostic framing, and reproducible pipeline. We address W1–W3 and Q1–Q3 below and in the revised paper.

W1: Expanded repair benchmarks & downstream impact of repaired vs. ungated graphs

We expanded both upstream repair and downstream decision evaluations (Secs. 5.1–5.3, Tables 1–3, Apps. C–F):

  1. 3-Part Upstream Repair Evaluation (Sec. 5.1, Table 1):
    • Real Manual Sections (App. E.1, Table 4): Across all 7 UNSAT dev/val sections (5 Dev + 2 Val: Wipers, Mirrors) in our 17-section EV manual corpus, unconstrained repair gamed SAT (28 silent κt\kappa_t flips: 18 Dev + 10 Val, plus guard/effect removals); our gate quarantined 7/7 (100%) gamed sections while accepting the live σt\sigma_t-typed re-anchoring repair on Steering (App. C.2).
    • 60-Case Multi-Sort Pool: Across Bool, Real (QF_LRA), and String (35 invalid/vacuous, 20 valid, 5 abstentions), Z3 feasibility (Core-Targeted) blocks 25/25 vacuous/4-pillar gaming (85.7% -> 28.6% FAR) and Evidence-Assisted (+ WMW_M) rejects 35/35 (0.0% FAR), accepts 20/20 (100%) valid repairs, and safely abstains on 5/5 unsupported-syntax cases.
    • 10-Task Live Closed-Loop Campaign (App. E.2, Table 5): Across 50 live LLM calls (5 policies ×10\times 10 tasks: contradictory guards, over-restrictions, numeric bands/strict boundaries, loose enums, XOR/disjunctive overrides), Core-Targeted and Evidence-Assisted achieved 10/10 (100%) live repair (0% FQR) vs. 0/10 (100% FQR) for syntactic non-deletion.
  2. Downstream Impact of Repaired Graphs (Sec. 5.3, Table 3 & App. F, Table 6):
    • On the 7 repair-affected scenarios, RAG + Ungated Repair drops to 70.9% (-21.7 pp below RAG + Fidelity-Gated POHA at 92.6%, and below RAG-Only at 76.3%). Solver-gamed graphs downgraded safety flags and removed guards (Charging, Wipers), causing an unsafe ALLOW on summer low-SoC parking (0.625 -> 0.250) and 0.00 on wiper readiness (1.00 -> 0.00).
    • Across all 13 scenarios, staged POHA leads all baselines (72.9%), rising to 88.3% with +Verifier diagnostics (Sliding Door 0->1) and σt\sigma_t re-anchoring (Steering 0->1, 0 vs. 6 false sensor cores; Apps. C.2 & F).

W2: Comparison against Structured Text and Discrete FSM baselines

In Table 3 & App. F (Table 6), we compare 5 representations under identical frozen retrieval (top-k=10) and judge settings:

  • RAG-Only: 76.3% (7 repair-affected) / 56.5% (all 13 scenarios)
  • RAG + Structured Text (JSON rules): 71.2% / 61.4% (fails topology on mirror folding: 0.00)
  • RAG + Discrete FSM (states + boolean guards, no Real QF_LRA, δt\delta_t, or σt\sigma_t): 87.0% / 69.9% (solves discrete switches 1.00, but drops to 0.00 vs. POHA 1.00 on reserve-charge display, 0.375 vs. 0.625 on summer parking, and 0.714 vs. 0.857 on bidirectional high-temp cutoff)
  • RAG + Fidelity-Gated POHA: 92.6% (7 repair-affected) / 88.3% (all 13 scenarios; 72.9% staged base).

W3 & Q3: Formal verification scope of Hybrid Automata & safety-critical applications

In Secs. 3.2 & 4.1, we clarify that nonlinear ODE reachability (x˙=f(x)\dot{x}=f(x)) is undecidable in SMT and absent from owner's manuals. POHA operates in the decidable Timed / QF_LRA + Bool + finite-String hybrid transition fragment (Alur et al., 1995; Henzinger, 1996), where δt>0\delta_t > 0 enforces Non-Zeno realizability and Z3 checks joint pre/post-state execution (SAT(Inv(ssrc)∧Pt∧Et∧Inv(sdst)next)\text{SAT}(\text{Inv}(s _ {\text{src}}) \land P_t \land E_t \land \text{Inv}(s _ {\text{dst}})^{\text{next}}), Eq. 3). For safety-critical systems, POHA thus acts as a discrete-mode/threshold decision shield, not a continuous-time trajectory controller (Ethics Statement).

Q1: Overcoming strict syntactic preservation limits

In Table 1, syntactic non-deletion (Pt⊆Pt′P_t \subseteq P_t') had 100% FQR (0/10 repaired) on flawed initial guards. Our two-tier semantic gate (Sec. 4.2, Eqs. 4–6) uses Z3 implication (SAT(Pt′)∧UNSAT(Pt′∧¬p)\text{SAT}(P_t') \land \text{UNSAT}(P_t' \land \neg p)), UNSAT-core/witness unlocking (Auth(t,F,WM)\text{Auth}(t, F, W_M)), and source-state re-anchoring to achieve 10/10 (100%) valid repair (0% FQR) and 0% FAR.

Q2: Verifying initial extractions without a ground-truth oracle

Detailed in Sec. 4.1, our reference-free stack combines: (1) verbatim page-traced quote verification & static symbol audit (flagging undeclared/self-comparing expressions), (2) cross-section ontology alignment with co-occurrence & sort vetoes, (3) multi-sort Z3 4-pillar + invariant/guard feasibility checks, and (4) source-witness consistency checks.

I thank the authors for their response.

The response from the authors have substantially addressed my main concerns.

However, I would like to know whether the newly included repair cases considered the proposed gate?

AuthorsOpenReview ↗

Response for inclusion of repair gate

Thank you for your review!

Yes—every newly included repair evaluation and our new downstream ablation (Table 3) directly run through our proposed Fidelity & Feasibility Gate (with the original syntactic gate also evaluated side-by-side in Table 1 for comparison):

  • 10-Task Live Campaign (Table 1 & Table 5): Every live LLM repair candidate (across all 10 multi-sort tasks, 50 LLM calls) is audited directly by our proposed gate, accepting 10/10 (100%) valid guard repairs under both Core-Targeted and Evidence-Assisted (vs. 0/10 under the original syntactic gate).
  • 60-Case Multi-Sort Pool (Table 1): All 60 candidates are evaluated through the proposed gate—rejecting 25/35 invalid cases witness-free (Tier 1) and 35/35 (100%) under Tier 2, accepting 20/20 valid repairs, and abstaining on 5/5 unsupported-syntax cases (vs. 5/35 rejected by the original syntactic gate without Z3 feasibility).
  • Real Manual Sections (Table 4 & App. C.2): All 7 UNSAT dev/val sections plus Steering and eCall run through the proposed gate, quarantining 7/7 solver-gamed repairs (28 silent safety-flag flips) and destructive transition/effect removals while accepting the valid re-anchoring repair on Steering (0 violations).
  • Downstream RAG (Table 3 & Table 6): RAG + Fidelity-Gated POHA enforces the proposed gate before inference—quarantining gamed graphs to reach 92.6% vs. 70.9% for RAG + Ungated Repair on the 7 repair-affected scenarios (and 72.9% vs. 61.3% staged across all 13 scenarios, rising to 88.3% with +Verifier diagnostics and the gate-accepted Steering graph).
Reviewer_okxiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets an interesting and important problem in LLM-based formal specification generation and refinement: LLMs may modify or weaken the original constraints to achieve SAT, rather than preserving the semantics (a solver gaming problem). To address it, the paper proposes a source-traced pipeline that extracts Partially Observable Hybrid Automata (POHA) from text manuals, checks the resulting constraints using an SMT solver, and introduces a fidelity gate to prevent destructive modifications during spec. repair. The experimental results provide initial evidence that the proposed fidelity gate can detect and reject certain solver-satisfying but semantically destructive modifications. Separately, the authors show that incorporating symbolic graph representations into downstream RAG-based decision-making can improve accuracy.

Critical strengths
  • This paper identifies a meaningful limitation of existing LLM-based formal specification generation and refinement frameworks and offers an interesting perspective on addressing this issue.

  • The proposed fidelity gate provides an explicit mechanism for preventing destructive modifications during LLM-guided specification repair. The overall design is relatively straightforward to understand.

  • The authors provide initial experimental evidence supporting their approach and also acknowledge its limitations.

Critical weaknesses
  • The proposed fidelity gate appears closely related to existing "techniques" (not task settings) such as design-by-contract, specification refinement, and graph invariant preservation. The paper could better clarify its technical novelty and differences from these established approaches. I would appreciate further technique-level discussion in the rebuttal.

  • The fidelity gate mainly guarantees structural preservation, and the paper makes a minor contribution to semantic correctness (however, semantic correctness is inherently challenging)

  • The paper lacks comparisons against alternative specification-repair methods, existing proposer-verifier frameworks, or other "structured presentation" (rather than POHA); these omissions make it difficult to determine whether the reported benefits arise specifically from the proposed POHA representation and fidelity-gated repair mechanisms, or just the high-level framework.

Questions for authors

Beyond the weaknesses discussed above, I have a few additional questions that I would appreciate the authors answering during the rebuttal:

  1. Do the authors expect the choice of the underlying specification language to significantly affect the results? (>> Different specification languages have different grammars, expressiveness, and associated reasoning or solving procedures. These differences may also affect the structure of the extracted models and the types of modifications that can be detected during repair. I would be interested in understanding how dependent the proposed fidelity gate is on the specific POHA representation and whether it could be generalized to other formal specification languages.)

  2. Could the authors further clarify why POHA was chosen as the underlying formal representation? For example, would simpler representations, such as finite-state machines or other symbolic models, be sufficient for some of the evaluated tasks?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer okxi: Semantic Refinement Theory, Z3 Gate & Baselines

We thank Reviewer okxi for the constructive feedback and address W1–W3 and Q1–Q2 below and in the revised paper.

W1: Technical novelty vs. Design-by-Contract (DbC), specification refinement & invariant preservation

In Secs. 2 & 3.3, we formalize why classical refinement fails when an LLM has write-access to the specification (H0→H′\mathcal{H} _ 0 \to \mathcal{H}') without a ground-truth oracle:

  1. Inverted precondition polarity vs. DbC: DbC checks implementations against a fixed contract by weakening preconditions (P⇒P′P \Rightarrow P'). In safety spec repair, PtP_t encodes interlocks (brake_pressed == True); weakening PtP_t erases safety prohibitions.
  2. Vacuity trap of naive strengthening (Pt′⇒PtP_t' \Rightarrow P_t): Since ⊥⇒Pt\bot \Rightarrow P_t is tautological, an LLM satisfies Pt′⇒PtP_t' \Rightarrow P_t by injecting contradictions (speed > 0 and speed == 0), muting tt vacuously.
  3. Antecedent erasure on self-mutating specs: Meta-axioms take the form ∀t:A(t)⇒B(t)\forall t: A(t) \Rightarrow B(t) (e.g., κt=True⇒Sensor(Pt)\kappa_t = \texttt{True} \Rightarrow \text{Sensor}(P_t)). Z3 returns SAT if the LLM erases A(t)A(t) (flipping κt:True→False\kappa_t: \texttt{True} \to \texttt{False} or deleting tt).

We introduce Bounded Non-Vacuous Semantic Refinement (H0⪯fidH′\mathcal{H} _ 0 \preceq _ {\text{fid}} \mathcal{H}', Sec. 4.2, Eqs. 4–6):

H0⪯fidH′  ⟺  Preservectx(H0,H′)∧Feasible(H′)∧RefinesWM(H0,H′)\mathcal{H} _ 0 \preceq _ {\text{fid}} \mathcal{H}' \iff \text{Preserve} _ {\text{ctx}}(\mathcal{H} _ 0,\mathcal{H}') \land \text{Feasible}(\mathcal{H}') \land \text{Refines} _ {W_M}(\mathcal{H} _ 0,\mathcal{H}')

locking state-invariant (Inv(s)⊆Inv′(s)\text{Inv}(s) \subseteq \text{Inv}'(s)), sensor, and epistemic context (Preservectx\text{Preserve} _ {\text{ctx}}), proving Z3 pre/post-state satisfiability (Feasible\text{Feasible}), and enforcing non-vacuous Z3 guard implication with core/witness revision (RefinesWM\text{Refines} _ {W_M}).

W2: From structural preservation to semantic correctness

We upgraded the gate from string inclusion (Pt⊆Pt′P_t \subseteq P_t') to multi-sort Z3 verification (Bool, Real QF_LRA, String; Secs. 4.1–4.2, App. D):

  1. Z3 Non-Vacuous Implication: Checks SAT(Pt′)∧∀p∈Pt,UNSAT(Pt′∧¬p)\text{SAT}(P_t') \land \forall p \in P_t, \text{UNSAT}(P_t' \land \neg p), accepting valid tightenings (temp <= 31 →\to temp < 31) rejected by string matching (FQR: 100% -> 0%).
  2. Pre/Post-State Feasibility + Witnesses: SAT(Inv′(ssrc′)∧Pt′∧Et′∧Inv′(sdst′)next)\text{SAT}(\text{Inv}'(s _ {\text{src}}') \land P_t' \land E_t' \land \text{Inv}'(s _ {\text{dst}}')^{\text{next}}) blocks 100% (20/20) of vacuous/invariant-conflict guards (Wipers "OFF" vs. "AUTO"), and witnesses WMW_M catch drift (FAR: 85.7% -> 28.6% -> 0.0% [0/35]).
  3. Core/Witness Revision & Re-Anchoring: Guards in Z3's local UNSAT core or failing witnesses (Auth(t,F,WM)\text{Auth}(t, F, W_M)) are unlocked for replacement (Airbag & Steering, Apps. C–D).

W3: Spec-repair policies & representation baselines

  • 5 Repair Policies (Table 1, Sec. 5.1, App. E): On a 60-case multi-sort pool and 10 live closed-loop tasks (50 LLM calls): Verifier-Only and Prompt-Only accept 100% of gamed repairs; Syntactic Gate has 100% FQR (0/10) and 85.7% FAR (28.6% with Z3 feasibility); Core-Targeted achieves 10/10 live repair (0% FQR) and blocks 25/25 vacuous/4-pillar gaming, while Evidence-Assisted (+ WMW_M) reaches 0% FAR (35/35 rejected).
  • Downstream Baselines & Repair Impact (Table 3, Sec. 5.3, App. F): On the 7 repair-affected scenarios, RAG-Only scores 76.3%, RAG + Structured Text 71.2%, RAG + Discrete FSM 87.0%, RAG + Ungated Repair drops to 70.9% (-21.7 pp from solver gaming), and RAG + Fidelity-Gated POHA reaches 92.6%.

Q1: Generalization beyond POHA

Antecedent erasure and vacuity gaming arise whenever self-mutating specs are checked against ∀x:A(x)⇒B(x)\forall x: A(x) \Rightarrow B(x). Our contract maps directly to LTL (nl2spec: locking qq in □(p→q)\square(p \to q), checking ◊p\Diamond p, requiring p′⇒pp' \Rightarrow p unless in UNSAT core), PDDL (LLM+P: preserving :effect, checking SAT(pre(a))\text{SAT}(\text{pre}(a))), and Alloy/SMT-LIB (preventing domain collapse).

Q2: Why POHA over Finite-State Machines?

In Table 3 & App. F (Table 6), Discrete FSM matches POHA on discrete switches (Group A: 100.0%, mirror folding 1.00 vs. Structured Text 0.00), but fails when tasks need: (1) Ontology (Real QF_LRA): Reserve-charge display (1.00 POHA vs. 0.00 FSM) and summer parking (0.625 vs. 0.375); (2) Temporality (δt>0\delta_t > 0): Blocks Δt=0\Delta t = 0 jumps; and (3) Epistemology (σt\sigma_t) + Z3 Diagnostics: Eliminated 6/6 false sensor cores; staged POHA already leads all 13 scenarios (72.9% vs. 69.9% FSM), rising to 88.3% with +Verifier notes (Sliding Door 0->1) and σt\sigma_t re-anchoring (Steering 0->1; Apps. C.2 & F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.