acceptodds
Under review as a conference paper at ICLR 2027

LCSL: A Symbolic Rule Language and Reasoning Framework for Enhancing Legal Compliance in Large Language Models

Abstract

Addressing the difficulties of Large Language Models (LLMs) in handling non-monotonic logic and opaque reasoning in legal compliance, we propose the LCSL-Framework, based on a Legal Compliance Symbolic Language. The framework combines Prolog-style predicate conditions with defeasible reasoning to decouple legal rules into prohibitive rules and defeaters. Its central design connects contextual condition interpretation to targeted conclusion revision: ATTACKS links an exception to its prohibition, and OVERRIDES withdraws that inference while other prohibitions remain subject to judgment. An LLM constructs reusable rule blocks and interprets a three-phase protocol for prohibition assessment, exception verification, and conflict adjudication. Experiments on Chinese Advertising Law, Copyright Law, and Hearsay tasks show the highest among the evaluated baselines, including SymbCoT. On Hearsay and Abs-Real, LCSL exceeds SePO by 5.8 and 7.6 percentage points. Abs-Real ablations reduce by 7.1 points without the rule representation and 4.2 without staged reasoning, supporting both aspects of the design. Case analysis illustrates the connection between conditions, exceptions, and judgments; OPP-115-derived classification provides auxiliary privacy evaluation.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
legal compliance determination, large language models, reasoning enhancement, symbolic language, symbolic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Z7mnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LCSL, a symbolic rule-based reasoning framework for legal compliance assessment. The authors use an LLM to transform legal text into Prohibition Rules and Defeaters, and employ attacks and overrides to model the relationships between rules and exceptions. They then use a three-stage reasoning process to determine whether a violation has occurred. Experiments on multiple legal datasets achieved higher F1 scores than existing baselines.

Critical strengths

This paper addresses non-monotonic reasoning in legal compliance assessments and has practical applications.

Critical weaknesses
  1. Regarding the method:

1.1) I believe the methodological innovation of this paper is not sufficiently clear. Compared to existing works such as Horner et al. (2025) and SymbCoT (Xu et al., 2024), which you cited in your related work, LCSL employs a simpler rule representation and reasoning process. Horner et al. have already investigated the use of LLMs to generate defeasible logic rules and conducted a dedicated evaluation of the rules’ syntactic and semantic correctness; SymbCoT has also proposed a framework for Symbolic Translation, Structured Reasoning, and Verification. In contrast, LCSL primarily relies on attacks/overrides and a three-stage prompt to make legal judgments. The authors should further explain what specific technical innovations LCSL offers compared to these prior works and whether this simplification yields clear advantages.

1.2) The method definition and implementation details of LCSL are not very clear. Although the paper provides a rule template and a three-stage reasoning process, it does not adequately explain how complex conditions are specifically evaluated, how attacks relations are determined from legal texts, or how the correctness of automatically generated rules is verified. Furthermore, although Equation (1) defines the final decision logic, the authors do not adequately explain how they ensure that the results generated by the LLM are consistent with this logic. I believe these are important details for understanding and evaluating the proposed method and should be further explained in the paper.

1.3) The paper emphasizes Targeted Exception Handling, but the Rule Construction Prompt in the Supplementary Code uses ATTACKS ALL_STRICT_RULES by default, and the generated rules provided also make extensive use of Global Override. I also found that similar exceptions are handled using both Targeted Override and Global Override in different generated rule files, which may lead to confusion. The authors should explain the specific criteria for determining attacks relations, why Global Override is used by default, and how the correctness of these binding relationships is verified. Given that avoiding the Global Immunity Fallacy is one of the core objectives emphasized in the paper, I believe this section requires more thorough explanation and experimental validation.

  1. Regarding the experiments and presentation

2.1) The paper primarily claims to improve non-monotonic legal reasoning and targeted exception handling, but the experiments mainly report final classification accuracy and F1 scores without a systematic evaluation of these core capabilities. Although some datasets include non-monotonic reasoning scenarios, the authors do not separately analyze the model’s ability to handle rule conflicts and exception overrides. I recommend adding controlled experiments, such as adding or removing different exceptions from the same case to test whether the final decision changes correctly, and evaluating the model in complex scenarios where multiple prohibitions and multiple exceptions coexist.

2.2) The ablation experiments are insufficient. Currently, ablation has only been conducted on the Abs-Real dataset, and it primarily removes entire components such as Rule Initialization, the LCSL Rule Base, and CoT; the contributions of the attacks/overrides mechanisms have not been verified separately.

2.3) The baseline experiments are not sufficiently comprehensive. First, SymbCoT performs substantially worse than standard CoT on multiple datasets. The authors should explain the reasons for this and provide further details on SymbCoT’s implementation and prompt configuration to ensure the fairness of the comparison. Second, SePO, CONTRAPROMPT, and SymbCoT were not tested on all major datasets.

2.4) I recommend that the authors include 95% confidence intervals and significance tests to assess the reliability of the experimental results. Additionally, since LCSL involves additional rule compilation and multi-stage reasoning, we recommend reporting token consumption, latency, and model calls and comparing them with the baselines.

2.5) The paper and supplementary code lack necessary methodological and experimental details. Although the authors uploaded the code, it does not include a README file, dependency configurations, or clear instructions for running the scripts.

2.6) I believe the paper’s writing and organization are also insufficient. A significant portion of the 9-page main text is devoted to flowcharts and case studies and half of the page is ai statement and ethics statement, which do not count toward the page limit, and the Appendix is only 3 pages long. Many important methodological and experimental details, say the complete prompt templates, the specific process for rule compilation, the intermediate representation schema, the dataset construction and annotation procedures, and the baseline implementation details, are not adequately explained. I believe this core information should be clearly presented in the main text or the Appendix, rather than requiring reviewers to read the code themselves to understand the methods and experimental procedures. Also, as I mentioned in 2.5, the absence of even a basic README makes it unnecessarily difficult for reviewers to navigate the supplementary code and understand the implementation.

Questions for authors
  1. Why does the rule construction prompt use ATTACKS ALL_STRICT_RULES by default, given the emphasis on Targeted Exception Handling?

  2. How do the authors verify the correctness of the generated rules and ensure that the LLM's final decisions are consistent with Equation (1)?

  3. Could the authors clarify which scripts, prompts, and configurations were used to produce the results in Tables 2–4?

Reviewer_u4kGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose LCSL, a neuro-symbolic framework designed to improve large language models' ability to perform legal reasoning, particularly in scenarios involving non-monotonic logic and statutory exceptions. The method introduces a structured symbolic rule representation that combines contextual conditions with defeasible reasoning mechanisms, using explicit ATTACKS and OVERRIDES relationships to model conflicts between prohibitions and exceptions. Experiments across multiple legal reasoning datasets, including Chinese advertising law, copyright law, and hearsay classification, demonstrate improvements over the evaluated prompting-based and symbolic reasoning baselines.

Critical strengths

The paper addresses a realistic and important challenge in legal reasoning. The motivation is grounded in practical legal scenarios, including incorrect exception handling and premature termination of the reasoning process, which are relevant to real-world legal compliance applications. The paper also reports empirical improvements over several baselines. These results suggest that the overall framework can improve legal classification performance.

Critical weaknesses
  1. Insufficient validation of the targeted exception mechanism. One of the paper's central contributions is the use of explicit ATTACKS relationships to link exceptions to specific prohibitions, which is intended to prevent the "global immunity fallacy." However, the experiments do not clearly demonstrate this claim. The authors do not provide a metric to measure the frequency of the global immunity fallacy or establish whether it is a significant failure mode in existing LLM-based legal reasoning systems. Additionally, the authors could provide stronger justification for their dataset selection. In my view, some of the datasets are not well suited to evaluating non-monotonic reasoning. For example, Hearsay is primarily a definition-classification task that excludes exceptions, while Med-Real focuses on prohibitive reasoning. Even for the datasets categorized as non-monotonic, the authors do not report how many examples actually require resolving conflicts between multiple prohibitions and exceptions. Furthermore, although Table 3 includes an ablation that removes the entire LCSL rule representation, there is no targeted ablation that removes only the ATTACKS relationships. Consequently, it remains unclear whether the reported improvements are attributable to the targeted exception mechanism itself. Could the authors provide further evidence or explanations to support this central claim?

  2. Lack of quantitative evaluation of intermediate reasoning steps. The proposed method combines multiple components, but the paper does not directly evaluate the correctness of their intermediate outputs. All quantitative results in Tables 2–5 are based on final binary classification labels. Although Eq. (1) specifies the expected outcome of conflict adjudication, against which the model's reported judgment "can be inspected," the authors do not evaluate how often the final prediction agrees with the result implied by the model's intermediate applicability decisions. The only direct evidence regarding intermediate reasoning behavior consists of two illustrative examples in Figure 3. Therefore, it is difficult to determine whether the reported improvements reflect more faithful non-monotonic reasoning or simply better final-label predictions.

Questions for authors
  1. In Table 2, SymbCoT substantially underperforms even standard CoT on the three overlapping datasets. Could the authors provide more details about how SymbCoT was implemented and adapted to these tasks? Additionally, how do the token budgets, inference latency, and total computational costs compare across methods, including the cost of rule-base initialization?

  2. Could the authors provide more details about the annotation and quality-control protocols for the newly constructed datasets, including how labels were verified and disagreements were resolved? Since Qwen3-32B was used to generate synthetic examples and also serves as the main evaluation backbone, how do the authors verify that the synthetic evaluation data do not introduce model-specific biases? Although Table 4 includes additional model backbones, those experiments are limited to Abs-Real rather than the synthetic datasets.

  3. Given the relatively small evaluation sets (e.g., 94 examples for Hearsay and 255 for Abs-Real) and the use of a single evaluation pass, could the authors provide confidence intervals or statistical significance tests for the reported improvements? Additional robustness analyses would help establish whether the observed gains are reliable.

Reviewer_FpeeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Paper introduces a neuro-symbolic framework to help models decide legal compliance with validation across Qwen3-32B on a synthetic dataset. The paper is somewhat unclear in its writing and dataset construction. Most importantly, it doesn't quantify uncertainty and it seems like it doesn't improve upon zero-shot baselines. Therefore, I mark clear reject.

Critical strengths
  • Framework: Makes sense to use neuro-symbolic framework to decide legal compliance for certain cases and the framework is intuitive.
Critical weaknesses
  • Dataset Unclear: It is not clear what the tasks are in the generated dataset, nor is it clear how human validation was done. Further, things like US privacy law cannot be broken down into clear rules like the paper suggests. I am only familiar with the US legal system, but as far as I understand, neuro-symbolic approaches have limited applicability for US law. Further, please translate Figure 3 for readers. Currently, it is not helpful for non-English readers.
  • Lack of Uncertainty Quantification: Please quantify uncertainty and explain why your framework may work better than other methods (e.g., SePO, SymbCoT, etc.) that you compare against (minor: please elaborate about the difference between your framework and SymbCoT). Currently, without uncertainty quantification, it seems like your framework is not statistically different from zero-shot prompting (Table 2).
  • Tested One Model: The paper only evaluates Qwen3-32B. It would help to have more models evaluated to understand how well the framework generalizes to different models.
Questions for authors
  • How was your dataset generated?
  • How do other models perform with your framework?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.