acceptodds
Withdrawn from ICLR 2027

NeuroFORM: Neuro-Symbolic Formalization of Legal Rules with Counterexample-Guided Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A generated formal specification can execute and be satisfiable while still misrepresenting its source rule—for example, by omitting an exception or an applicability condition. We study whether such behavioural errors can be detected and used to repair specifications at test time. NeuroFORM treats a language model as an untrusted proposal generator: given a legal clause and an expert-verified structured representation, it generates a candidate specification, compares its solver outcomes with expected outcomes on source-conditioned scenarios, and feeds failures back for bounded repair. The method operates on propositional encodings of obligations, permissions, prohibitions, exceptions, and boundary conditions. We evaluate it on 238 provisions from GDPR, CCPA, DPDP, and PIPL, with 2,322 span-level annotations and six behavioural scenario categories per provision. Across three independent runs, the full system achieves mean held-out behavioural accuracies of 78.43% with an LLM generator and 75.07% with an SLM generator, exceeding the strongest corresponding functional baselines by 14.77 and 20.33 percentage points, respectively. Among cases eventually repaired, 99.5% succeed by the second attempt within the successful repair invocation in both model tiers. These findings show how behavioural verifier failures can provide effective test-time feedback for generated symbolic specifications.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-symbolic reasoning, Legal formalization, Large language models, Symbolic verification, Counterexample-guided inductive synthesis, Automated compliance, Deontic logic, Formal methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kZhjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NeuroFORM to repair generated legal formulas that omit conditions or exceptions. At inference time, pretrained models generate and repair Z3 constraints using legal clauses, verified annotations, and scenario feedback, with a separate model checking consistency with the source. The proposed benchmark covers 238 provisions from four privacy laws, with separate repair and evaluation scenarios. The 7B and 32B configurations reach 75.07% and 78.43% held-out accuracy, respectively, but the main comparisons use unequal computation budgets.

Critical strengths
  • The paper's use of solver feedback to improve legal formulas is well motivated, especially for missing conditions and exceptions.
  • The proposed benchmark covers 238 provisions across four privacy laws and includes exception and boundary cases.
  • The ablations are informative in showing that removing the combined structural warning and repair lowers accuracy, and the separate model check improves accuracy.
Critical weaknesses
  • The main comparisons use unequal inference budgets (Section 4.1). It is unclear how much of the improvement comes from the proposed design versus extra computation.
  • In the matched study, diagnostic feedback improves accuracy by 3.78% and 1.40%, but both 95% confidence intervals include zero (Table 9). Its added value over generic feedback remains unclear.
  • Existing legal systems also repair constraints after solver failures (Section 5). B7/B9 often fail to execute, but the manuscript gives little detail about the failures or which original components these adaptations preserve.
Questions for authors
  • Could the authors compare NeuroFORM with CoT-Retry and generic solver-guided repair using identical inputs and repair labels, the same generator, and comparable total computation, including auxiliary model calls?
  • Does Table 9 retain the full pipeline, including structural repair, the semantic gate, candidate search, and repair budgets, with only the feedback changed?
  • Why do B7/B9 often fail to execute, and which original components do these adaptations preserve?
  • Does B9 implement Hsia et al.'s constraint-refinement procedure? If not, what prevents a comparison with that procedure?
Reviewer_GP3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NeuroFORM, which translates legal provisions and expert-verified annotations into executable Z3 specifications. It combines scenario-based checking, diagnostic-guided repair, and a model-based semantic gate. Experiments show improved accuracy on held-out scenarios over the evaluated baselines.

Critical strengths
  • The paper addresses an important problem.
  • The framework integrates verification and iterative repair, with separate scenarios for repair and evaluation.
Critical weaknesses
  1. Insufficient validation of evaluation targets.

Although expert review is included, the reported annotation agreement concerns span-level labels rather than the scenario-level SAT/UNSAT targets used to measure accuracy. Some expected verdicts may also require context absent from the synthesis input. Independent agreement on these targets and an analysis of context-insufficient cases would help distinguish formalization errors from limitations of the evaluation setup.

  1. Weak comparisons with closely related methods.

The L4L-Schema and RAG-SMT adaptations have low executability and are excluded from the functional-baseline ranking. This limits the evidence for advantages over closely related approaches. The paper should better justify these adaptations and clarify whether their failures reflect interface incompatibilities or limitations of the underlying methods.

  1. Limited model coverage.

Generation and repair are evaluated only on two sizes of Qwen2.5-Coder. Whether the benefits generalize across model families remains untested.

Questions for authors

See Weaknesses

Reviewer_DkoBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NeuroFORM, a neuro-symbolic framework for automatically formalizing legal provisions written in natural language. The framework takes original legal provisions and expert-verified structured annotations as input, uses language models to generate candidate Python/Z3 specifications, and checks whether their behavior matches expectations through source-clause-conditioned test scenarios and the Z3 solver. When the verification results differ from the expected SAT/UNSAT outcomes, the system uses scenario-level counterexamples and structured diagnostic information to guide bounded iterative repairs. In addition, the framework introduces an LLM-based semantic consistency check to further filter candidate specifications that satisfy the internal verification conditions.

Critical strengths
  1. The research problem has practical significance. The paper focuses on an important problem in automatic formalization: formal specifications generated by LLMs may be syntactically valid and executable, yet fail to accurately capture the applicability conditions, exceptions, and logical relationships expressed in the original natural-language rules. The authors formulate this problem as a behavioral consistency verification task based on source-clause-conditioned scenarios and develop a systematic framework combining LLM generation, Z3-based symbolic checking, counterexample feedback, and iterative repair. The research problem has practical significance.

  2. The experimental evaluation is relatively comprehensive. The authors construct a benchmark covering four privacy regulations, 238 provisions, and 2,322 span-level annotations. They design six categories of behavioral scenarios to examine different logical structures, including obligations, exceptions, and boundary conditions. They also conduct experiments involving two model scales, multiple functional baselines, repair budgets, E4v structural repair, generic versus structured feedback, and semantic gating, while reporting some paired statistical analyses and confidence intervals. Overall, the experimental evaluation is relatively comprehensive.

Critical weaknesses
  1. The effectiveness of the core mechanism is not sufficiently supported by empirical evidence. The central innovation of the paper is a structured diagnostic mechanism based on legal provisions and expert annotations, designed to guide the repair of formal specifications. However, the comparative experiments show that structured diagnostics provide only a small accuracy improvement over generic counterexample feedback, and the 95% confidence intervals include zero. Therefore, a consistent improvement cannot be statistically established, which weakens the evidence supporting the contribution of the core method.

  2. The baseline comparisons are not sufficiently fair. The baselines used in the main experiments, including Zero-Shot, Few-Shot, CoT-Retry, and Naive CEGIS, are relatively basic. Some adapted versions of recent state-of-the-art approaches exhibit low execution success rates, and sufficient comparisons with strong baselines are lacking. Meanwhile, the proposed method integrates multiple components, including multi-candidate generation, multi-round search, symbolic verification, and semantic gating, without aligning the inference budgets with those of the comparison methods. This makes it difficult to determine the sources of the observed improvements.

  3. The ablation study design is incomplete. The candidate-count experiments in the existing ablation studies primarily reflect the influence of computational budgets rather than independently validating the candidate-ranking strategy. The E4v ablation conflates diagnostic information with additional repair opportunities. Furthermore, controlled experiments on semantic gating and feedback strategies are distributed across different randomized experimental blocks, without an analysis of component interactions under a unified experimental setting. This makes it difficult to isolate the contribution of each component.

Questions for authors

Q1. In Appendix Table 9, the improvements of Diagnostic Feedback over Generic Feedback are not statistically significant under either the SLM or LLM configuration. Could the authors further explain how this result relates to the contribution of structured diagnostics emphasized in the paper?

Q2. NeuroFORM employs multi-candidate generation, up to seven outer iterations, bounded repair, and an independent semantic checking model, whereas the main baselines do not fully match its inference budget. Could the authors report the average number of model calls, input and output token counts, number of repair attempts, inference time, and computational cost for each method?

Q3. The paper already provides experiments on repair budgets, E4v structural repair, and semantic gating, but some comparisons still involve coupled components. Could the authors further distinguish the independent contributions of E4v diagnostic information and the additional repair opportunities it triggers? For example, under the same repair budget, could the authors compare repair outcomes using actual E4v diagnostics, generic structural warnings, and repairs without specific diagnostic content?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.