acceptodds
Under review as a conference paper at ICLR 2027

Is This Evidence Decision-Critical? Learning to Verify Rule-Governed Decisions

Abstract

Rule-based reasoning, as in eligibility checks and contract reviews, requires language models to assess evidence against individual conditions and combine their judgments under explicit rules. Errors in evidence assessment can leave a decision unchanged, but misinterpreting or overlooking decision-critical evidence can reverse it. Identifying such evidence allows more capable models to focus on checking the corresponding condition judgments, supporting accurate and safe decisions. Recognizing the evidence's criticality requires understanding how evidence affects a condition judgment and how that judgment affects the decision. To achieve the goal, we propose a INTERvention-based imPACT learning framework (InterPact), which enables counterfactual verification of evidence criticality in rule-governed decisions. Specifically, its evidence intervention constructor generates training pairs for a propagation verifier by editing case facts with a frozen language model while holding rules and non-target conditions fixed. Human-reviewed labels record the resulting condition and decision changes, while complete state-to-decision mappings supervise consequences beyond the observed edit. During training, the verifier weights learned conditional decision predictions by evidence-based condition probabilities through a fixed composition operation, propagating decision-change supervision into the base model. At inference, the trained base model directly judges criticality from the original case and target evidence, without human or stronger-model supervision. On single-case evidence criticality verification over adapted rule-governed decision cases, InterPact achieves 68.28% accuracy, outperforming all six baselines. These results support learned decision sensitivity as a basis for prioritizing evidence checks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Rule-Governed Reasoning, Decision-Impact Verification, Counterfactual Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iaoUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new task called Evidence Criticality Verification, in which models determine whether modifying a particular piece of evidence is likely to change the final decision. Based on the ContractNLI and ShARC datasets, the authors constructed a new dataset by modifying evidence in counterfactual scenarios and introduced InterPact, which uses propagation-aware training to enable models to learn the dependencies among evidence, conditions, and decisions, thereby improving the accuracy of identifying critical evidence.

Critical strengths
  1. The paper proposes an interesting task, Evidence Criticality Verification, and it is meaningful for real-world decision-making.

  2. The ablation experiments are quite thorough, with controlled ablation tests conducted on all core components. So I think the authors have a good analysis of the contributions of individual components of the proposed method.

Critical weaknesses
  1. Regarding the experiments:

1.1) LLM baselines should be included: GPT, DeepSeek, and open-source models larger than 4B should all be tested to facilitate comparison with the proposed method.

1.2) Sufficient backbones should be included for testing, such as Llama, and experiments with different sizes should also be added to validate the method’s generalizability.

1.3) Please report the 95% confidence intervals for the experimental results to assess statistical significance.

1.4) I recommend that the authors include an Evidence-ablated Baseline, in which the Evidence content in the input is masked, retaining only information such as Rule Type and Condition Structure, to verify whether the model can still achieve high Criticality Accuracy, thereby investigate potential shortcuts resulting from superficial statistical patterns in the dataset.

  1. Regarding the Dataset and Task

Although the authors mention interventions involving multiple Evidence Units in the Limitations section, they do not explicitly discuss cases where a single piece of Evidence simultaneously affects multiple Conditions. The current data construction process requires that non-target Conditions remain unchanged, which may limit the method’s ability to model complex Evidence Dependencies. I think supplementing the analysis with tests involving Multi-Condition Evidence would verify the model’s generalization performance in this scenario.

I am willing to increase my score if I saw some parts of my major concerns have been addressed (say more SOTA baselines, backbones), and the main conclusion of the paper is still unchanged.

Questions for authors
  1. Could you please add a proprietary LLM baseline and an open-source Qwen 32B baseline for comparison?

  2. Can you change the backbones to see the generalizability of the method?

  3. Could you add an Evidence-ablated Baseline to detect if there are any potential shortcuts?

Reviewer_dYAVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper describes work on investigating if an evidence given to an LLM for some arbitrary task is crucial or has significant influence to dictate the model’s decisions. The authors frame this task as teaching LLMs on verification or validation of evidence criticality and propose the INTERPACT framework. INTERPACT trains an LLM as a verifier in a supervised manner from an evidence intervention constructor which edits facts and records condition and decision changes while a human review assesses annotation quality. The optimized LLM can then be used to judge the criticality of an evidence from the origin case without human supervision to which the paper shows using this INTERPACT method beats weak baseline performance at 68.28% accuracy.

Critical strengths

I see the value in the line of research the authors want to pursue which is verifying how LLMs recognize some evidence or decision factor’s criticality and how this would ultimately affect its judgment for some task. This particular direction has several applications in legal compliance and high-stakes domains where LLMs are used. The overall proposed INTERPACT method looks promising and may be useful for some specific tasks, provided that the authors carefully address the major weaknesses I have highlighted from the paper in their next paper submission.

Critical weaknesses

One of the main critical issues with the paper is that the written content in the is very AI-sounding and the overall writing of the paper could be further significantly improved in quality and readability. I acknowledge that the authors did mention they use AI for the paper and take responsibility for it, but I still highly encourage the authors to edit out any AI written text in the main paper given that they are very poorly written and are obvious to readers. The paper is also full of uncontextualized technical jargons such as “propagation verifier”, “root criticality labels” “frozen estimator", the word “intervention” itself which is vague, etc which could have been rewritten in very crisp and simple language instead. It’s very hard to grasp what the authors wanted to say and hard to differentiate which were the authors’ main idea vs. what has been AI generated from this work.

Another critical issue is that the data requirements (i.e. what the data the paper needs to properly evaluate the research question it wants to investigate and its proposed methods) are poorly defined and do not give the reader enough context to learn about the paper and the data itself. Answer the simple questions: What type of dataset do you need for this task? What specific labels? What language? What domain? Can you use synthetically generated or restricted human-curated? Are you restricted to only NLI-type datasets?

Moreover, in the experiment results, testing on two datasets is weak and is hardly convincing and quite limited for ICLR papers where technical rigor is critical. There are plenty of reasoning-based tasks that the method can be applied or evaluated with including compliance assessment with legal reasoning. See https://aclanthology.org/2025.emnlp-main.44/ or https://arxiv.org/pdf/2509.23291 for example where compliance decisions must rely on reasoning from existing laws like GDPR/HIPAA which could serve as the explicit rules the task needs.

In terms of baselines in the experiments, the ones presented in the results seem weak and there are no strong baselines that the authors compared with. Why not just prompt a general frontier model like GPT-Sol or Claude as baselines in a zero-shot or few-shot manner and compare their results to the other baselines and the proposed method? If these models show comparable performance with the proposed INTERPACT approach, then that tells something about the limitations of the paper but will help the authors diagnose when and where the proposed method can actually contribute to. I believe this is basic in the state of highly-advanced LLM research these days.

Questions for authors

See supporting questions from the major weaknesses cited above.

Reviewer_6tNXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

change the decision while preserving the governing rule and non-target condition states. INTERPACT constructs controlled evidence interventions and derives complete mappings from each possible target condition state to its decision consequence. Following supervised initialization, it trains a language model to predict these conditional decisions, combines them with probabilities from a frozen condition estimator, and applies a “Hard Warrant” constraint that preserves the annotated original condition-to-decision relation. At deployment, the trained language model directly answers a criticality question without executing the paired verifier.

Experiments use 6,913 intervention pairs across 1,041 roots from structured adaptations of ContractNLI and ShARC. On 207 test roots, INTERPACT achieves 68.28% accuracy and 66.61% balanced accuracy, exceeding the strongest reported baseline by 4.51 and 5.76 percentage points, respectively. Additional experiments evaluate conditional decision predictions, intervention outcomes, and matched training ablations.

Critical strengths

1.A useful formulation of evidence importance in terms of decision consequences. The paper distinguishes evidence that changes a local condition judgment from evidence that can change the final decision. Complete condition-to-decision mappings provide supervision beyond the particular states reached by sampled edits. This is consequential because it makes decision sensitivity an explicit learning target, rather than treating local evidence relevance or decision accuracy as a sufficient proxy.

  1. Well-designed ablations provide evidence beyond an aggregate performance gain. The matched variants share initialization, condition estimates, paired inputs, and training exposure. In particular, the composition ablation retains branch supervision and the probability mass associated with the known original relation. Removing composition reduces direct balanced accuracy from 66.61% to 56.85% (Table 2), supporting a contribution beyond simply adding mapping labels. The paper also reports unfavorable results, including increased false changes on invariance controls, which helps delimit its conclusions.
Critical weaknesses

1.The main comparison does not adequately establish the advantage over alternative methods for solving the criticality task. In direct evaluation, PSE and R2-Guard use only their trained language-model components, without their defining rule-execution mechanisms (Section 5.1 and Appendix C). Applying a common prompt is a reasonable test of transfer to direct answering, but these results do not establish superiority over the complete systems adapted to criticality prediction. A particularly relevant missing comparison would estimate condition states and feasible target-state changes from the original evidence, then apply the supplied aggregation rule to assess decision sensitivity. This should respect evidence-level intervention feasibility rather than assume that every target state is reachable. The matched ablations support the proposed training design, but they do not resolve this broader comparison. This matters because the task already supplies substantial symbolic structure, and the benefit of learning conditional decision consequences over using that structure directly remains insufficiently established.

  1. The main benchmark is small, and a consequential subset of its negative labels needs stronger validation. The direct evaluation contains 207 roots in 52 connected components. Of the 122 negative roots, 88 have a constant decision mapping, which supports their labels conditional on the original annotations. The other 34 rely on reviewer judgments that no allowed edit can reach a decision-changing condition state (Appendix B.4). The paper does not report agreement or detailed adjudication evidence specifically for this subset. Moreover, the direct prompt does not explicitly enumerate the six allowed operations or the requirement to preserve non-target condition states (Appendix D.4). These omissions leave uncertainty about alignment between the question answered by the model and the reference-label definition. The separate 400-pair audit is valuable, but assumes the original labels are correct and does not directly validate these root-level existential judgments. Finally, three-run standard deviations do not characterize uncertainty from sampling test components. Given the 4.51-point accuracy improvement, these issues could materially affect the interpretation of the gain.
Questions for authors

1.Comparison request: How does INTERPACT compare with a criticality baseline that retains explicit rule execution, estimates condition states and feasible evidence-induced changes without reference labels, and receives the same original-case information? Please report inference cost alongside performance if this requires additional model calls.

  1. Clarification request: For the 34 negative roots without constant mappings, what criteria establish that every decision-changing target state is unreachable through an allowed edit? Please provide representative examples, the number of independent annotators, and any recorded agreement or adjudication information specific to these root labels.

  2. Evaluation request: Does the main advantage persist when the direct prompt explicitly specifies the allowed intervention operations and preservation of non-target condition states? Please also report results separately for positive roots, constant-mapping negative roots, and the 34 judgment-based negative roots.

  3. Statistical analysis request: Can you provide paired confidence intervals for the main accuracy and balanced-accuracy differences, accounting for the 52 connected components—for example, by resampling components? This would help distinguish run-to-run variability from uncertainty due to the small evaluation sample.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.