- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new task called Evidence Criticality Verification, in which models determine whether modifying a particular piece of evidence is likely to change the final decision. Based on the ContractNLI and ShARC datasets, the authors constructed a new dataset by modifying evidence in counterfactual scenarios and introduced InterPact, which uses propagation-aware training to enable models to learn the dependencies among evidence, conditions, and decisions, thereby improving the accuracy of identifying critical evidence.
Critical strengths
-
The paper proposes an interesting task, Evidence Criticality Verification, and it is meaningful for real-world decision-making.
-
The ablation experiments are quite thorough, with controlled ablation tests conducted on all core components. So I think the authors have a good analysis of the contributions of individual components of the proposed method.
Critical weaknesses
- Regarding the experiments:
1.1) LLM baselines should be included: GPT, DeepSeek, and open-source models larger than 4B should all be tested to facilitate comparison with the proposed method.
1.2) Sufficient backbones should be included for testing, such as Llama, and experiments with different sizes should also be added to validate the method’s generalizability.
1.3) Please report the 95% confidence intervals for the experimental results to assess statistical significance.
1.4) I recommend that the authors include an Evidence-ablated Baseline, in which the Evidence content in the input is masked, retaining only information such as Rule Type and Condition Structure, to verify whether the model can still achieve high Criticality Accuracy, thereby investigate potential shortcuts resulting from superficial statistical patterns in the dataset.
- Regarding the Dataset and Task
Although the authors mention interventions involving multiple Evidence Units in the Limitations section, they do not explicitly discuss cases where a single piece of Evidence simultaneously affects multiple Conditions. The current data construction process requires that non-target Conditions remain unchanged, which may limit the method’s ability to model complex Evidence Dependencies. I think supplementing the analysis with tests involving Multi-Condition Evidence would verify the model’s generalization performance in this scenario.
I am willing to increase my score if I saw some parts of my major concerns have been addressed (say more SOTA baselines, backbones), and the main conclusion of the paper is still unchanged.
Questions for authors
-
Could you please add a proprietary LLM baseline and an open-source Qwen 32B baseline for comparison?
-
Can you change the backbones to see the generalizability of the method?
-
Could you add an Evidence-ablated Baseline to detect if there are any potential shortcuts?