- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes LCSL, a symbolic rule-based reasoning framework for legal compliance assessment. The authors use an LLM to transform legal text into Prohibition Rules and Defeaters, and employ attacks and overrides to model the relationships between rules and exceptions. They then use a three-stage reasoning process to determine whether a violation has occurred. Experiments on multiple legal datasets achieved higher F1 scores than existing baselines.
Critical strengths
This paper addresses non-monotonic reasoning in legal compliance assessments and has practical applications.
Critical weaknesses
- Regarding the method:
1.1) I believe the methodological innovation of this paper is not sufficiently clear. Compared to existing works such as Horner et al. (2025) and SymbCoT (Xu et al., 2024), which you cited in your related work, LCSL employs a simpler rule representation and reasoning process. Horner et al. have already investigated the use of LLMs to generate defeasible logic rules and conducted a dedicated evaluation of the rules’ syntactic and semantic correctness; SymbCoT has also proposed a framework for Symbolic Translation, Structured Reasoning, and Verification. In contrast, LCSL primarily relies on attacks/overrides and a three-stage prompt to make legal judgments. The authors should further explain what specific technical innovations LCSL offers compared to these prior works and whether this simplification yields clear advantages.
1.2) The method definition and implementation details of LCSL are not very clear. Although the paper provides a rule template and a three-stage reasoning process, it does not adequately explain how complex conditions are specifically evaluated, how attacks relations are determined from legal texts, or how the correctness of automatically generated rules is verified. Furthermore, although Equation (1) defines the final decision logic, the authors do not adequately explain how they ensure that the results generated by the LLM are consistent with this logic. I believe these are important details for understanding and evaluating the proposed method and should be further explained in the paper.
1.3) The paper emphasizes Targeted Exception Handling, but the Rule Construction Prompt in the Supplementary Code uses ATTACKS ALL_STRICT_RULES by default, and the generated rules provided also make extensive use of Global Override. I also found that similar exceptions are handled using both Targeted Override and Global Override in different generated rule files, which may lead to confusion. The authors should explain the specific criteria for determining attacks relations, why Global Override is used by default, and how the correctness of these binding relationships is verified. Given that avoiding the Global Immunity Fallacy is one of the core objectives emphasized in the paper, I believe this section requires more thorough explanation and experimental validation.
- Regarding the experiments and presentation
2.1) The paper primarily claims to improve non-monotonic legal reasoning and targeted exception handling, but the experiments mainly report final classification accuracy and F1 scores without a systematic evaluation of these core capabilities. Although some datasets include non-monotonic reasoning scenarios, the authors do not separately analyze the model’s ability to handle rule conflicts and exception overrides. I recommend adding controlled experiments, such as adding or removing different exceptions from the same case to test whether the final decision changes correctly, and evaluating the model in complex scenarios where multiple prohibitions and multiple exceptions coexist.
2.2) The ablation experiments are insufficient. Currently, ablation has only been conducted on the Abs-Real dataset, and it primarily removes entire components such as Rule Initialization, the LCSL Rule Base, and CoT; the contributions of the attacks/overrides mechanisms have not been verified separately.
2.3) The baseline experiments are not sufficiently comprehensive. First, SymbCoT performs substantially worse than standard CoT on multiple datasets. The authors should explain the reasons for this and provide further details on SymbCoT’s implementation and prompt configuration to ensure the fairness of the comparison. Second, SePO, CONTRAPROMPT, and SymbCoT were not tested on all major datasets.
2.4) I recommend that the authors include 95% confidence intervals and significance tests to assess the reliability of the experimental results. Additionally, since LCSL involves additional rule compilation and multi-stage reasoning, we recommend reporting token consumption, latency, and model calls and comparing them with the baselines.
2.5) The paper and supplementary code lack necessary methodological and experimental details. Although the authors uploaded the code, it does not include a README file, dependency configurations, or clear instructions for running the scripts.
2.6) I believe the paper’s writing and organization are also insufficient. A significant portion of the 9-page main text is devoted to flowcharts and case studies and half of the page is ai statement and ethics statement, which do not count toward the page limit, and the Appendix is only 3 pages long. Many important methodological and experimental details, say the complete prompt templates, the specific process for rule compilation, the intermediate representation schema, the dataset construction and annotation procedures, and the baseline implementation details, are not adequately explained. I believe this core information should be clearly presented in the main text or the Appendix, rather than requiring reviewers to read the code themselves to understand the methods and experimental procedures. Also, as I mentioned in 2.5, the absence of even a basic README makes it unnecessarily difficult for reviewers to navigate the supplementary code and understand the implementation.
Questions for authors
-
Why does the rule construction prompt use
ATTACKS ALL_STRICT_RULESby default, given the emphasis on Targeted Exception Handling? -
How do the authors verify the correctness of the generated rules and ensure that the LLM's final decisions are consistent with Equation (1)?
-
Could the authors clarify which scripts, prompts, and configurations were used to produce the results in Tables 2–4?