- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EviPair, a test of whether language models follow the evidence they are given. Each synthetic problem appears in several versions: two with opposite answers, versions with a helpful hint or an irrelevant sentence added, and one with key evidence removed, where the right answer is "unknown." Comparing a model's answers across these versions shows whether hints help, whether irrelevant text distracts, and whether the model avoids guessing. The authors find a clear hint benefit and distraction cost in one fine-tuned Qwen2.5 model.
Critical strengths
Each problem is tested in matched versions, so the paper can tell apart three things that are usually mixed together: whether a hint helps, whether irrelevant text hurts, and whether the model avoids guessing when evidence is missing.
Critical weaknesses
None
Questions for authors
The local models answer directly, with very few output tokens and thinking turned off. It is already known that models struggle with multi-step problems without step-by-step reasoning, and that stating an intermediate step helps them. Does the screened Qwen2.5 configuration still benefit from the hint when it is allowed to reason?
4 comments
The hint benefit persists when clear final answers are accepted
Dear Reviewer eXNP,
We would like to highlight a further result from the completed comparison that directly addresses whether the hint benefit depends on restricting reasoning or enforcing a rigid answer format. With step-by-step prompting and clear final-answer fields accepted, relevant-hint joint correctness is 256/256 versus 140/256 for the original input. The irrelevant control reaches 102/256 (Supplementary Analyses §2.2).
This experiment uses the development-selected Qwen2.5 NF4 SFT replica. Direct and step-by-step prompts share the same weights, 256 backgrounds, seven evidence conditions, greedy decoding and 4,096-token output limit. Joint correctness requires both opposite-premise answers to be correct. The reasoning prompt produces visible prose in 707/1,792 confirmation responses, with mean output length 37.16 tokens versus two for direct answers.
The complete scoring comparison is:
| Step-by-step answer scorer | Implicit J | Relevant J | Irrelevant J | Relevant–implicit gain |
|---|---|---|---|---|
| Frozen primary grammar | 69/256 | 98/256 | 46/256 | +11.33 pp |
| Prefix relaxation, specified before formal analysis | 116/256 | 175/256 | 90/256 | +23.05 pp |
| Exploratory clear-final-field sensitivity | 140/256 | 256/256 | 102/256 | +45.31 pp |
The final-field sensitivity accepts a completed bare answer with an optional period or an explicit case-insensitive FINAL: A/B/C field at the end of the output, including an inline field. It was introduced after inspecting the fixed output sample, and all original primary scores are retained. Every scorer keeps the full cohort and gives no credit to truncated responses. The hint benefit survives both relaxations, showing that strict final-line formatting does not account for the observed contrast.
The direct replica passes all four competence categories. Under clear-final-field scoring, the reasoning condition obtains 256/256 explicit decisions and 120/128 unknown relations, while positive and negative relation judgments each reach 109/128, below the 90% screen. These probes identify relation judgment as a remaining bottleneck under this reasoning policy.
These measurements make the role of explicitation more precise: on this replica, allowing explicit working and accepting clear completed answers still leaves a substantial benefit from supplying an entailed intermediate consequence. EviPair measures that benefit against both the original input and irrelevant additions, while separately checking premise tracking and warranted uncertainty. The same-weight comparison and its scoring sensitivities are provided in Supplementary Analyses §§2.1–2.3.
Thank you again for suggesting this comparison.
Thank you for the detail rebuttal. I would keep my positive score.
Thank you for your support and constructive discussion
Dear Reviewer eXNP,
Thank you for confirming your positive assessment and for the question that helped us strengthen the reasoning comparison. We are a small research group committed to contributing useful tools for understanding and improving LLMs, and your encouragement means a great deal to us.
We hope EviPair can serve as a concrete starting point for future research on LLM hallucination and interpretability: it connects how a model uses evidence, follows premise changes, and recognizes insufficient information on the same problems. An encouraging practical result is our supplementary Qwen2.5 NF4 comparison: with the same 10,240 labeled presentations, five-view SFT achieves 453/512 three-state successes versus 16/512 for fixed single-view repetition, while unsupported answers on missing-evidence inputs fall from 343/512 to 32/512 (Supplementary Analyses §4). This links the diagnostic to a substantial improvement in evidence-responsive behavior.
We hope these results and the released dataset, generator and evaluator strengthen your confidence in the paper's contribution and its value for future research. Thank you again for your recognition and support; they are a strong encouragement for us to continue this work.
Hint benefit under explicit step-by-step prompting
Dear Reviewer eXNP,
Thank you for suggesting step-by-step reasoning as an intervention. The supplementary same-weight comparison retains a relevant-hint benefit when explicit reasoning is allowed: joint correctness rises from 69/256 to 98/256, a gain of 11.33 percentage points (two-test Holm-adjusted p = 0.0038). The gain over the irrelevant control is 20.31 points (adjusted p = 1.32 × 10⁻⁷).
The comparison uses a development-selected five-view Qwen2.5 NF4 SFT replica. Direct and step-by-step prompts share exactly the same weights, all 256 backgrounds, all seven conditions, and a 4,096-token output limit. Joint correctness requires both opposite-premise answers to be correct.
The direct configuration passes all four independent competence categories. Mean output length increases from 2 to 37.16 tokens, with visible prose in 707/1,792 step-by-step confirmation responses. The supplementary tables provide the style-specific capability checks and final-answer scoring sensitivities. Table 1’s 7.42-point gain uses the original screened checkpoint with direct answers and a 64-token limit.
For assistants answering from supplied documents, extra reasoning should help recover implicit consequences while leaving genuinely insufficient evidence unresolved. In the original Qwen2.5 records, 12 of the 19 hint-rescued backgrounds also receive the correct unknown answer (Appendix N). This linkage matters when assessing an intervention: improved determined decisions and appropriate uncertainty can be checked on the same problems.
The comparison tests a concrete explanation of the gap: whether giving the model room for explicit working removes its dependence on a stated intermediate consequence. The residual benefit shows that output opportunity alone does not remove that dependence in this configuration. EviPair makes changes in inference policy assessable through premise tracking and warranted uncertainty, providing concrete response patterns against which explanations of model behavior can be tested. The supplementary materials contain the settings, linked results and recorded outputs.