acceptodds
Under review as a conference paper at ICLR 2027

EviPair: Paired Behavioral Diagnostics for Evidence-Responsive Decisions in Language Models

Abstract

Evidence-based decisions should follow changed premises and retain uncertainty when decisive information is missing. We introduce EviPair, a paired behavioral diagnostic combining entailed hints, premise reversals, irrelevant additions, and verified evidence removal on the same underlying problems. Joint correctness tests whether a model follows both premise states; a shared unknown condition tests whether it avoids unsupported assertions. This design measures Evidential Myopia as sensitivity to making an already entailed consequence explicit, while separating hint benefit from distraction. Our relational, numerical-aggregation, and natural-language studies cover two open-weight backbones and multiple hosted models, including a seven-family API extension. A screened Qwen2.5 configuration exhibits a 7.42-point hint benefit and a 7.03-point distraction cost in joint correctness. Hosted configurations show varied response profiles, including near-ceiling screened performance. Five-view SFT checkpoints improve premise tracking and uncertainty handling over unadapted backbones on new backgrounds from the same synthetic construction, without test-time gold hints. Matched five- view comparisons find no stable additional benefit from entailment consistency over SFT or paraphrase controls. Together, these results provide a paired evalua- tion protocol for distinguishing useful evidence interventions from distraction and unsupported certainty.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evidence Use, Language Model Evaluation, Behavioral Diagnostics, Semantic Interventions, Counterfactual Evaluation, Uncertainty Handling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eXNPOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviPair, a test of whether language models follow the evidence they are given. Each synthetic problem appears in several versions: two with opposite answers, versions with a helpful hint or an irrelevant sentence added, and one with key evidence removed, where the right answer is "unknown." Comparing a model's answers across these versions shows whether hints help, whether irrelevant text distracts, and whether the model avoids guessing. The authors find a clear hint benefit and distraction cost in one fine-tuned Qwen2.5 model.

Critical strengths

Each problem is tested in matched versions, so the paper can tell apart three things that are usually mixed together: whether a hint helps, whether irrelevant text hurts, and whether the model avoids guessing when evidence is missing.

Critical weaknesses

None

Questions for authors

The local models answer directly, with very few output tokens and thinking turned off. It is already known that models struggle with multi-step problems without step-by-step reasoning, and that stating an intermediate step helps them. Does the screened Qwen2.5 configuration still benefit from the hint when it is allowed to reason?

4 comments
AuthorsOpenReview ↗

Hint benefit under explicit step-by-step prompting

Dear Reviewer eXNP,

Thank you for suggesting step-by-step reasoning as an intervention. The supplementary same-weight comparison retains a relevant-hint benefit when explicit reasoning is allowed: joint correctness rises from 69/256 to 98/256, a gain of 11.33 percentage points (two-test Holm-adjusted p = 0.0038). The gain over the irrelevant control is 20.31 points (adjusted p = 1.32 × 10⁻⁷).

The comparison uses a development-selected five-view Qwen2.5 NF4 SFT replica. Direct and step-by-step prompts share exactly the same weights, all 256 backgrounds, all seven conditions, and a 4,096-token output limit. Joint correctness requires both opposite-premise answers to be correct.

Prompting styleImplicitRelevantIrrelevant
Direct242/256256/256166/256
Step-by-step69/25698/25646/256

The direct configuration passes all four independent competence categories. Mean output length increases from 2 to 37.16 tokens, with visible prose in 707/1,792 step-by-step confirmation responses. The supplementary tables provide the style-specific capability checks and final-answer scoring sensitivities. Table 1’s 7.42-point gain uses the original screened checkpoint with direct answers and a 64-token limit.

For assistants answering from supplied documents, extra reasoning should help recover implicit consequences while leaving genuinely insufficient evidence unresolved. In the original Qwen2.5 records, 12 of the 19 hint-rescued backgrounds also receive the correct unknown answer (Appendix N). This linkage matters when assessing an intervention: improved determined decisions and appropriate uncertainty can be checked on the same problems.

The comparison tests a concrete explanation of the gap: whether giving the model room for explicit working removes its dependence on a stated intermediate consequence. The residual benefit shows that output opportunity alone does not remove that dependence in this configuration. EviPair makes changes in inference policy assessable through premise tracking and warranted uncertainty, providing concrete response patterns against which explanations of model behavior can be tested. The supplementary materials contain the settings, linked results and recorded outputs.

AuthorsOpenReview ↗

The hint benefit persists when clear final answers are accepted

Dear Reviewer eXNP,

We would like to highlight a further result from the completed comparison that directly addresses whether the hint benefit depends on restricting reasoning or enforcing a rigid answer format. With step-by-step prompting and clear final-answer fields accepted, relevant-hint joint correctness is 256/256 versus 140/256 for the original input. The irrelevant control reaches 102/256 (Supplementary Analyses §2.2).

This experiment uses the development-selected Qwen2.5 NF4 SFT replica. Direct and step-by-step prompts share the same weights, 256 backgrounds, seven evidence conditions, greedy decoding and 4,096-token output limit. Joint correctness requires both opposite-premise answers to be correct. The reasoning prompt produces visible prose in 707/1,792 confirmation responses, with mean output length 37.16 tokens versus two for direct answers.

The complete scoring comparison is:

Step-by-step answer scorerImplicit JRelevant JIrrelevant JRelevant–implicit gain
Frozen primary grammar69/25698/25646/256+11.33 pp
Prefix relaxation, specified before formal analysis116/256175/25690/256+23.05 pp
Exploratory clear-final-field sensitivity140/256256/256102/256+45.31 pp

The final-field sensitivity accepts a completed bare answer with an optional period or an explicit case-insensitive FINAL: A/B/C field at the end of the output, including an inline field. It was introduced after inspecting the fixed output sample, and all original primary scores are retained. Every scorer keeps the full cohort and gives no credit to truncated responses. The hint benefit survives both relaxations, showing that strict final-line formatting does not account for the observed contrast.

The direct replica passes all four competence categories. Under clear-final-field scoring, the reasoning condition obtains 256/256 explicit decisions and 120/128 unknown relations, while positive and negative relation judgments each reach 109/128, below the 90% screen. These probes identify relation judgment as a remaining bottleneck under this reasoning policy.

These measurements make the role of explicitation more precise: on this replica, allowing explicit working and accepting clear completed answers still leaves a substantial benefit from supplying an entailed intermediate consequence. EviPair measures that benefit against both the original input and irrelevant additions, while separately checking premise tracking and warranted uncertainty. The same-weight comparison and its scoring sensitivities are provided in Supplementary Analyses §§2.1–2.3.

Thank you again for suggesting this comparison.

Thank you for the detail rebuttal. I would keep my positive score.

AuthorsOpenReview ↗

Thank you for your support and constructive discussion

Dear Reviewer eXNP,

Thank you for confirming your positive assessment and for the question that helped us strengthen the reasoning comparison. We are a small research group committed to contributing useful tools for understanding and improving LLMs, and your encouragement means a great deal to us.

We hope EviPair can serve as a concrete starting point for future research on LLM hallucination and interpretability: it connects how a model uses evidence, follows premise changes, and recognizes insufficient information on the same problems. An encouraging practical result is our supplementary Qwen2.5 NF4 comparison: with the same 10,240 labeled presentations, five-view SFT achieves 453/512 three-state successes versus 16/512 for fixed single-view repetition, while unsupported answers on missing-evidence inputs fall from 343/512 to 32/512 (Supplementary Analyses §4). This links the diagnostic to a substantial improvement in evidence-responsive behavior.

We hope these results and the released dataset, generator and evaluator strengthen your confidence in the paper's contribution and its value for future research. Thank you again for your recognition and support; they are a strong encouragement for us to continue this work.

Reviewer_rxcMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EviPair, a synthetic paired diagnostic that tests whether language models follow premise changes, benefit from entailed hints versus irrelevant additions, and answer “unknown” when decisive evidence is removed. Building on this diagnostic, the authors also propose a training approach that fine-tunes Qwen models with multiple supervised views per background and adds an entailment-consistency regularizer between implicit and explicit inputs. The authors also show that multi-view SFT improves Qwen models on new synthetic backgrounds, but the proposed entailment-consistency regularizer adds no stable benefit over matched baselines.

Critical strengths
  • Original input as a baseline. Rather than only contrasting relevant and irrelevant hints, the paper also uses the unmodified input as a baseline. This separates the improvement brought by relevant hints from the degradation caused by irrelevant context, and avoids overstating the benefit of explicit hints.

  • Transparent reporting of negative results. The authors report all configurations, seeds, and failed prespecified criteria, including the lack of a stable benefit from their own consistency regularizer and the near-ceiling performance of strong API models.

Critical weaknesses
  • Limited novelty. The effects of relevant hints and irrelevant context are already well studied (Press et al., 2023; Shi et al., 2023). The paired evaluation also resembles contrast sets (Gardner et al., 2020), which the paper does not discuss. It is unclear what the joint evaluation reveals beyond separate evaluations. Moreover, the gap after screening is only significant for one fine-tuned Qwen2.5 checkpoint, while screened API models are near ceiling.

  • Limited training insights. The consistency regularizer does not consistently outperform five-view SFT, with little analysis of why. A single-view SFT baseline is also missing, making it difficult to tell whether the gains come from multi-view training or simply adapting to the synthetic tasks.

  • Dataset availability. EviPair is presented as a new evaluation suite, but the dataset, generator, and evaluator are not provided. This makes it difficult to assess label quality and potential shortcuts. Releasing an anonymized repository during the discussion would help.

Questions for authors

Will the authors release EviPair? Since the authors present the evaluation suite as a main contribution, could they release the dataset, generator, and evaluator to support reproducibility and future research?

3 comments
AuthorsOpenReview ↗

What the linked comparisons reveal and how training changes them

Dear Reviewer rxcM,

The accompanying anonymized supplementary package provides all three requested components—EviPair's dataset, generator and evaluator—together with recorded outputs and verified offline reproduction commands. It preserves the original supplementary materials alongside Supplementary Analyses, a concise PDF explaining the linked results and controlled training comparisons. We will also release the paper, data and code on GitHub after double-blind review.

Thank you for highlighting the original-input baseline. It changes the interpretation of the result: Qwen2.5’s 14.45-point relevant–irrelevant difference contains a 7.42-point improvement over the original input and a 7.03-point distraction cost (Table 1). These point to different intervention targets, even though a single contrast would combine them.

What the joint test adds. Evidential Myopia concerns a decision improving when a consequence already entailed by the input is stated explicitly. EviPair locates that benefit within the model's responses to changed and insufficient evidence. In the same 256-background records, relevant hints rescue 19 premise pairs; 12 of these 19 also receive the correct unknown answer after decisive evidence is removed (Supplementary Analyses §1; Appendix N, pp. 35–36). Overall unknown accuracy is 91.4% (234/256). Linked scoring pinpoints the seven rescued backgrounds that also produce unsupported assertions in their shared unknown view. Separate accuracies do not locate these cases. For an assistant answering from supplied documents, this distinction matters: making a supported answer easier should also be evaluated alongside its behavior when that support is absent.

Press et al. examine composition conditional on correct component answers; Shi et al. examine distraction, including group-level accuracy; and Gardner et al.’s contrast sets examine local decision boundaries and consistency. EviPair extends these controlled and group-based comparisons by tying an answer-preserving entailed addition to an answer-reversing premise edit and verified evidence removal, then measuring which response changes co-occur. Unknown is certified by compatible completions supporting opposite answers, grounding the uncertainty test in the supplied evidence. The 19/12 result illustrates the information this linkage supplies. The revised related-work discussion will make these connections explicit.

A separately trained Qwen2.5 NF4 SFT replica passes all four standalone competence checks and retains a 5.47-point hint gain, with 14 gains and zero losses on the same 256 confirmation backgrounds. The gain also holds after full-prompt token lengths and query-entity mentions are matched (Supplementary Analyses §3). This extends the positive finding to another qualified checkpoint under tighter lexical controls.

Training across evidence states. Five-view SFT supervises original, changed-premise and unknown inputs. On 512 new backgrounds, Qwen2.5’s both-premise correctness rises from 1/512 to 469–480/512 across three seeds, and unknown correctness from 213/512 to 482–496/512, without test-time gold hints (Table 43).

A budget-matched NF4 control uses the same 2,048 training backgrounds, 10,240 labeled presentations and about 5.584 million training tokens per arm. On the same 512 validation backgrounds, five-view SFT answers both premise states and the unknown view correctly on 453/512 backgrounds, versus 16/512 when one fixed view per background is repeated five times: +85.35 points, with 437 gains and zero losses (conservative paired 95% interval [80.64, 88.68]; Supplementary Analyses §4). Covering the different evidence states produces a clear benefit under this fixed training budget.

The implicit and relevant-hint views share the same supervised target, so cross-entropy already encourages their agreement. The consistency term further aligns their output distributions without adding a new label signal. This shared-target structure helps explain why SFT supplies the main adaptation gain; the matched paraphrase comparison tests an alternative alignment signal.

Reusable materials. A version map identifies the preserved original materials and links each supplementary analysis to its inputs, recorded outputs and reproduction command. Reviewers and the AC can inspect the labels and recompute the paired results; other researchers can reuse the evidence interventions to evaluate reasoning or training improvements.

Dear authors,

Thank you for the detailed response and releasing the supplementary package. I appreciate the new single-view SFT comparison and the release of the supplementary materials, which address my concerns about the training baseline and reproducibility. My other concern is partially addressed. The 19/12 result is interesting, but I am not convinced that it provides a substantial new insight. Besides, the 19/12 linkage example is small. I will maintain my original rating.

Best regards,

AuthorsOpenReview ↗

What EviPair adds: diagnosing which evidence-response requirements improve

Dear Reviewer rxcM,

Thank you for confirming that the single-view comparison and supplementary release address your baseline and reproducibility concerns. We would like to clarify the methodological contribution and connect it to the paper's larger experiments.

EviPair contributes a semantically verified, joint intervention design. For one background and a fixed query, two opposite premise states each receive original, relevant-entailed and irrelevant-entailed presentations, followed by one shared unknown input: seven linked conditions. Added statements are checked against their own premise state; the unknown label requires compatible completions supporting opposite answers. This construction connects three requirements: using entailed information, following decisive premise changes, and withholding a definite answer when support is absent (Section 3).

Press et al. (2023) analyze composition conditional on correct component answers; Shi et al. (2023) study distraction; Gardner et al. (2020) score consistency across an original example and its edits. EviPair builds on these controlled comparisons by jointly constructing and verifying answer-preserving additions, answer-reversing edits and evidence removal. The resulting contribution is an intervention profile that identifies which evidence-response requirements improve on the same problems.

What the linkage reveals, and its replication. The 19/12 result localizes a specific discrepancy: seven of the 19 premise pairs rescued by hints still produce unsupported assertions on their shared unknown input, despite overall unknown accuracy of 234/256. The separately trained, screen-qualified Qwen2.5 NF4 replica repeats this pattern: JJ rises from 242/256 to 256/256, while TT rises from 231/256 to 241/256. Its 14 rescues split into 10 with correct unknown answers and four with unsupported assertions. Length/entity-matched additions retain these counts (Supplementary Analyses §§2–3). These are two trained configurations evaluated on the same 256 backgrounds; the unknown input receives no hint treatment.

The paper also tests the training implications on 512 independent validation backgrounds. Appendix V, Table 44 reports Qwen3 across seeds 17/29/43:

Training armBoth premise states correctAll three states correct
No CF/U483 / 455 / 4540 / 2 / 2
Five-view SFT505 / 506 / 502502 / 496 / 492
Five-view + entailment consistency489 / 501 / 503480 / 497 / 497

Each count is out of the same 512 backgrounds. No CF/U jointly removes changed-premise and unknown supervision while retaining original, explicit and paraphrased supervision and entailment consistency. Its seed-17 model follows both opposite premise states on 483/512 backgrounds (94.3%), yet asserts a definite answer on all 483 corresponding unknown inputs. The full five-view entailment arm and No CF/U even have identical original-input accuracy, 496/512, and the same consistency term, while their three-state joint counts are 480 versus 0. Table 43 shows the same separation for Qwen2.5: 266–362 correct premise pairs and zero correct unknown answers across three seeds.

These results make a training distinction concrete: answer-preserving consistency and supervision across different evidence states play different roles. Proposition 2 (Appendix F) bounds same-label disagreement by the supervised answer loss, explaining why shared-label CE already promotes implicit/explicit agreement. Changed-premise and unknown targets supply different response requirements. The new fixed-budget comparison directly tests this coverage: plain five-view SFT obtains 453/512 three-state successes versus 16/512 for a fixed single view repeated five times, under the same 10,240 labeled presentations (Supplementary Analyses §4).

The contribution therefore combines a verified diagnostic construction, reproducible evidence-response profiles, and controlled evidence about what supervision improves. We will foreground these findings and their connection to contrast sets in the revised manuscript so that the contribution is clear beyond the 19/12 illustration.

Thank you again for your consideration.

Reviewer_jxweOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Language models making evidence-based decisions must draw implicit conclusions, revise answers when premises change, and preserve uncertainty when decisive information is missing. Measuring these abilities is difficult because explicit hints can simplify reasoning, irrelevant additions can cause distraction, and correct standalone judgments do not necessarily translate into reliable decisions. EviPair addresses this problem through seven linked inputs combining premise reversals, relevant and irrelevant entailed additions, and verified evidence removal. Experiments reveal a configuration-specific hint benefit alongside many ceiling results, while supervised adaptation improves performance on new synthetic backgrounds without establishing a stable additional benefit from consistency regularization.

Critical strengths
  • The seven-condition design evaluates answer-preserving additions, answer-reversing premise changes, and evidence removal on the same underlying background. Requiring joint correctness across opposite premise states makes it possible to distinguish successful evidence tracking from an unchanged answer that happens to be correct in one state.

  • Unknown labels require compatible completions supporting opposite answers, and the public-text evaluator checks these witnesses without exposing them to the tested models. Matching record counts and endpoint mentions across determined and unknown inputs further strengthens the effort to measure warranted uncertainty rather than recognition of an obvious missing-information template.

  • Comparing relevant hints against both the original input and an irrelevant addition reveals whether an apparent benefit combines genuine improvement with distraction. The Qwen2.5 follow-up demonstrates the value of this distinction concretely, decomposing the 14.45-percentage-point relevant-versus-irrelevant difference into a 7.42-point improvement and a 7.03-point distraction cost.

Critical weaknesses
  • The clearest screened hint benefit in Section 6.1 is established for one selected Qwen2.5 SFT checkpoint, whereas Qwen3 is near ceiling and several qualified hosted configurations show no remaining benefit. Section 6.2 similarly reports that the complete qualified API configurations achieve 127 or 128 jointly correct backgrounds out of 128, limiting the diagnostic's ability to differentiate stronger systems. A more demanding, independently evaluated difficulty progression would help establish whether the benchmark can expose evidence-response limitations across a broader range of capable configurations.

  • Passing the separate behavioral screen does not establish that a model correctly derives the relevant intermediate consequence on the particular decision instances where a hint helps. Section 6.3 makes this distinction consequential: requiring correct standalone relation judgments on both premise states reduces Qwen2.5's relevant-versus-implicit gains to five backgrounds, with an adjusted p-value of 0.0625. Although the paper acknowledges this limitation, stronger matched evidence would be needed to distinguish overlooked usable knowledge from failures of derivation or integration within the full decision context.

  • Section 3 matches relevant and irrelevant additions by sentence template, relation type, and position, but explicitly leaves per-pair token length and query overlap unmatched. The measured contrast can therefore reflect differences in how easily a model associates the added sentence with the query, alongside the usefulness of the entailed intermediate conclusion. More tightly matched additions and systematic lexical counterbalancing would strengthen the attribution of the observed benefit to the intended semantic intervention.

  • The seven-condition natural-language study in Section 6.2 contains only 14 FOLIO-derived stories, and none of its paired comparisons survives the declared correction. The larger 204-example FOLIO evaluation in Section 6.4 tests ordinary accuracy using checkpoints from a separate eight-view study, whose semantic-consistency mean of 57.84% falls below the 61.11% SFT mean. These complementary checks are useful, but they do not yet provide robust external evidence that the main paired effects or five-view adaptation results extend to diverse natural-language decisions.

Questions for authors
  1. The clearest positive result is obtained for one selected Qwen2.5 SFT checkpoint. Does the effect replicate across independently trained checkpoints that pass the same behavioral screen?

  2. Can EviPair differentiate these stronger systems using an independently specified difficulty progression, such as increased reasoning depth, competing evidence, or more complex premise reversals? How would you increase difficulty while retaining the competence requirements and avoiding selection based on observed model failures?

  3. Among hint-rescued backgrounds, how often does the model correctly judge the relevant intermediate relation in both premise states? Can you report the corresponding paired effect size and uncertainty for this subset?

2 comments
AuthorsOpenReview ↗

Checkpoint robustness, controlled difficulty, and where the hint benefit arises

Dear Reviewer jxwe,

Thank you for asking where the hint effect persists and what the matched relation judgments reveal. Your questions help distinguish two explanations for an improved decision: making an intermediate consequence easier to derive, and helping a correctly judged consequence carry through to the final answer.

1. The effect replicates in a separately trained, screen-qualified checkpoint. The supplementary Qwen2.5 NF4 SFT replica passes the same four competence categories: explicit decisions 256/256, positive relations 123/128, negative relations 122/128 and unknown relations 119/128. Its weights are selected solely by development performance before confirmation. On the fixed 256-background cohort, it retains a relevant-hint gain:

Qwen2.5 SFT configurationImplicitRelevantIrrelevantNet hint gain (pp)
Original screened checkpoint237/256256/256219/256+7.42
NF4 seed-17 replica242/256256/256166/256+5.47

The replica has 14 gains and zero losses, with a conservative paired 95% interval of [1.06, 9.55] points. This is a separate training run and capability screen on the same confirmation backgrounds. Additional directional evidence across the three SFT seeds in Appendix U, Table 41 gives gains of +8.59, +5.47 and +6.25 points on its separate 128-background cohort.

2. Harder evidence chains with the same competence requirements. The four complete, qualified API configurations reach 127–128/128 joint correctness (Table 40): they already handle the implicit evidence in this regime well. Appendix S fixes independent 2–4-hop and 5–6-hop strata before collection, excludes prior structural signatures, and separates capability from confirmation backgrounds.

A prospective 2/4/6/8-hop progression would hold entity and record budgets constant and predeclare generation seeds, cohort sizes, models and stopping rules. Each level would retain disjoint capability and confirmation backgrounds, the same 90% per-category screen, checked premise reversals and opposite-answer completion witnesses for unknown inputs. Reporting every configuration–level outcome avoids choosing difficulty from observed model errors. This increases the required composition while preserving a clear interpretation of success at each level.

3. Relation correctness among hint-rescued backgrounds. Five of the 19 rescued backgrounds, 26.3%, have correct relation judgments in both premise states. To estimate the paired effect without selecting on rescue, we use all 233 backgrounds satisfying that relation condition: implicit joint correctness is 228/233 and relevant-hint correctness is 233/233, giving five gains and zero losses. The net gain is 2.15 percentage points, with a conservative 95% confidence interval of [−1.28, 5.39].

The paper’s prespecified masked analysis retains the full 256-background denominator: +1.95 points, with the original four-test Holm-adjusted p = 0.0625 (Section 6.3; Appendix N). The split gives the aggregate hint gain a more informative interpretation: five rescued backgrounds combine correct standalone relation judgments with an implicit decision error. These provide concrete cases for investigating how a correctly judged intermediate conclusion carries through to the final decision.

Matched lexical controls. Both added blocks mention the query and comparison subjects once and have identical full-prompt token lengths in all 512 pairs under both local tokenizers. The screened NF4 replica retains J = 242/256, 256/256 and 174/256 for implicit, relevant and irrelevant inputs: +5.47 points over implicit and +32.03 over irrelevant, with 14 gains and zero losses over implicit. The entailed additions preserve the original questions and labels. Natural-language coverage includes the 14-story paired FOLIO study (Appendix R) and the separate 204-example accuracy evaluation using eight-view checkpoints, with answer SFT at 61.11% versus 50.00% for the base model (Appendix Q, Table 34).

AuthorsOpenReview ↗

Why the joint diagnostic remains informative when task accuracy is high

Dear Reviewer jxwe,

Thank you for your careful questions. Your points about ceiling performance and relation-level competence raise a broader issue: what practical information does EviPair provide when ordinary task accuracy is already high? A connection to the paper's training results helps make this concrete.

In Appendix V, Table 44, two Qwen3 seed-17 configurations both obtain 496/512 original-input accuracy and use the same implicit/explicit consistency term. Their three-state joint counts are nevertheless 480/512 and 0/512. The full five-view entailment arm supervises original, changed-premise and unknown inputs; the No CF/U joint ablation omits supervision on the latter two. The joint diagnostic distinguishes these configurations by requiring correct answers under both opposite premise states and warranted uncertainty on the same background. This provides a concrete way to evaluate whether a training improvement covers the different requirements of evidence use.

The supplementary Qwen2.5 NF4 comparison demonstrates that this behavior can be improved substantially: five-view SFT achieves 453/512 three-state successes versus 16/512 for one fixed view repeated five times, using the same 2,048 training backgrounds and 10,240 labeled presentations. The gain is 85.35 percentage points, with a conservative paired 95% interval of [80.64, 88.68], without test-time gold hints (Supplementary Analyses §4).

Together with the independently screened Qwen2.5 replica and the length/entity-matched controls, which retain a 5.47-point hint gain with 14 gains and zero losses (Supplementary Analyses §3), these results connect the diagnostic to reproducible training decisions. The seven conditions have verified semantic relationships, and their linked responses show which requirements an intervention actually improves. We hope this makes clearer the contribution of EviPair as a reusable tool for studying and improving evidence-responsive behavior, and provides a stronger basis for your final assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.