acceptodds
Under review as a conference paper at ICLR 2027

Do Language Models Revise the Right Conclusions? Diagnosing and Training Evidence-Contingent Reasoning

Abstract

Aggregate answer scores cannot tell whether a language model revises the right conclusions when its evidence changes. We study evidence-contingent reasoning (ECR): withdrawing conclusions that lose support, preserving unaffected content, and retaining conclusions that have a valid alternative proof in the target world. Our evaluation suite, EviPair, links audited evidence worlds and measures Change, Preserve, and Switch on fixed, model-independent eligibility sets. On MuSiQue-Full, FEVER, and QAMPARI, reinforcement learning with only node-local semantic and evidence rewards improves answering but lowers preservation on every task, and development-set endpoint matching still leaves substantial response differences on the full test graphs. We instantiate structured intervention training with Paired Transition Policy Optimization (PTPO), which credits each node with the marginal return of its incident intervention edges while reusing independently sampled world trajectories. With Forced Switch edges (which invalidate every source-world proof of a conclusion while retaining alternative support) withheld from reinforcement learning, macro SwitchAcc on FEVER and QAMPARI rises from 17.48 after supervised finetuning (SFT) to 33.42. The gain over the strongest prior-method baseline, CRPO-Evidence, is 4.28 points (95% paired root-clustered interval [2.21, 6.35]); the Change and Preserve differences are not significant. The clearest benefit of structured intervention training over CRPO-Evidence thus lies in alternative-proof adaptation, and local revision and proof adaptation are best evaluated as separate targets.

Primary area:
reinforcement learning
Keywords:
evidence-contingent reasoning, LLM evaluation, counterfactual/behavioral evaluation, reinforcement learning for LLMs, credit assignment, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7dXPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a language model should revise its conclusions when the supporting evidence changes. Conclusions that lose their support should be withdrawn; conclusions that are unaffected should be kept; conclusions for which a different proof exists in the new world should be kept but given a new justification. The authors construct EviPair, which links audited evidence worlds through typed intervention edges and scores Change, Preserve, and Switch on eligibility sets frozen before any model runs.

The diagnostic part shows that reinforcement learning with only node-level semantic and evidence rewards improves endpoint scores while consistently hurting preservation across all three datasets. The training part proposes PTPO, which credits each node with the cross-world relational return on each of its incident edges and centers these credits with a node-local leave-one-out baseline, reusing trajectories across edges. The main experiments use Qwen3-4B-Base as the backbone, run on three datasets and five seeds, with a macro-average Change/Preserve/Switch of 57.75/66.58/33.42 against 44.63/60.95/17.48 for the SFT start. Forced Switch edges are excluded from the training curriculum.

Critical strengths
  1. The coordinate decomposition turns an unobservable response into a scorable object. Eligibility sets are frozen before evaluation and do not depend on model success, so denominators cannot drift. The endpoint-only baseline gains 5.68, 4.68, and 3.71 endpoint points on the three datasets while losing 3.18, 3.17, and 3.51 on preservation (Table S27) — exactly the "better answering, worse preservation" pattern that aggregate scores cannot see. I expect this evaluation design to outlast the training method.

  2. The pre-registered Switch result is strong and stable. SwitchAcc rises from 17.48 to 33.42, with a paired difference of +13.39 [9.90, 16.88] against the endpoint baseline and +4.28 [2.21, 6.35] against CRPO-Evidence, Holm-corrected at p = 1.02e-4; all five seeds are positive (Table S15b). Ablations over pairing and edge sampling (27.94–29.73) rule out these two implementation choices as the source of the gain.

  3. The largest gain appears on a relation never seen in training. Forced Switch edges never enter the RL curriculum, yet SwitchAcc on them rises substantially. This is the strongest argument for acceptance.

Critical weaknesses
  1. The paper is difficult to read. Worlds, edges, atoms, certificates, eligibility sets, and the four intervention types are all introduced in Section 3 at high symbol density, and the reader must shuttle between the main text and Appendices A.2–A.6 to assemble the full decision procedure. The fact that Equation (8) is an existential test rather than an equality is precisely what determines how Switch should be read, yet the main text disposes of it in half a sentence. For a paper whose contribution is a construction, this substantially raises the evaluation bar.

  2. A rise in SwitchAcc does not by itself show that the model re-grounded its proof. Equation (8) only asks whether the certificate contains some minimal sufficient set, not whether it equals one, so a policy that cites every document visible in the target world passes the same test. Equation (10) rules out any complete source proof surviving in the target world, but the new proof lies inside EvE_v, hence inside such a certificate, so the old proof cannot simply be copied and the all-cite strategy still succeeds. If that is so, the +15.94+15.94 gain over SFT and the +13.39+13.39 gain over GRPO-Marginal do not establish that the model learned to switch proofs rather than to cite more broadly. Appendix A.2 notes that over-citation is a separate citation-precision question and reports no measurement of it, and Equations (11)–(12) never require Cv≠CuC_v \neq C_u. Switch is the coordinate where this matters most: it is constructed so that the answer set is unchanged while every source proof is invalidated, which leaves the citation as the only observable difference between the two worlds. The authors need not have erred, but without the corresponding measurement the reported improvement cannot be separated from this alternative explanation.

Questions for authors
  1. I would like to understand how the reward design induces the three behaviours of withdrawing what should be withdrawn, keeping what should be kept, and changing the justification when the evidence changes, especially the last. Section 5 gives an additive return, but from the main text I could not reconstruct how the rewards on Necessary Drop and Irrelevant Preserve edges distinguish correct from incorrect withdrawal, nor why support adaptation on natural redundancy should teach reconstruction on replaced evidence. The Switch definition also raises a question of its own. An edge is eligible only if the source proof cannot be reused in the target world, which seems to require evidence that was not available before, whereas the training curriculum appears to consist only of removal-based edges. Are there training edges that require introducing new evidence? If not, where does the model learn this? Please explain the mechanism, supplementing with a control experiment if necessary

  2. Since Equation (8) only requires the certificate to contain some minimal sufficient set, a policy that cites every visible document would also pass, and its SwitchAcc could be reported directly. Could you add that baseline, together with a measure of certificate minimality on the 485 Forced Switch edges?

Reviewer_kifCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a model changes the right conclusions when its evidence changes: dropping answers that lose support, keeping the ones that don’t, and holding on to an answer when its original proof is gone but another one is available. To measure this, the authors build EviPair from MuSiQue, FEVER and QAMPARI. Each question comes with several audited evidence worlds linked by typed interventions. The model answers each world independently, and Change, Preserve and Switch are computed over edge sets that are fixed before any model is run. A diagnostic experiment shows that GRPO with only per-world rewards raises answer accuracy but lowers Preserve on all three tasks. The proposed method, PTPO, adds a typed relational reward between paired worlds, credits each node with the average return over its incoming and outgoing edges, and uses a node-local leave-one-out baseline. With Forced Switch edges kept out of RL entirely, macro SwitchAcc goes from 17.48 (SFT) to 33.42, which is 4.28 points above CRPO-Evidence (95% CI [2.21, 6.35]). The Change and Preserve differences against that baseline are not significant.

Critical strengths
  1. The formulation goes a step beyond the usual perturbation tests. Contrast sets and evidence-deletion probes mostly check whether the answer flips. Here withdrawal, retention and proof replacement are scored separately for individual answer atoms, and Forced Switch (every source proof invalidated, an alternative guaranteed) is a clean probe that I have not seen in prior work.

  2. The evaluation protocol is careful. Eligibility does not depend on the model, so source-world errors and parse failures stay in the denominator instead of being quietly filtered out, a common problem in this line of work. The paired root-clustered bootstrap, the single pre-registered test family and Proposition 3 together make a good case that these metrics cannot be read off endpoint accuracy.

  3. The training experiments are well controlled, and the claims are stated with appropriate caution. Forced Switch is held out at the root level, and the ablations vary the reward kernel, credit normalization, pairing and rollout reuse at a matched token budget. The finding that moving from joint standardization to a node-local LOO baseline helps at least as much as the typed kernel is useful for anyone training on paired environments.

Critical weaknesses
  1. I am not convinced the problem survives outside this setting. Everything is run on constructed, bounded evidence worlds with short-answer tasks and 4B/8B models, with no test on evidence that changes naturally, such as an updated retrieval index. Since Appendix B.5 notes that exact endpoint correctness already implies most of the relational constraints, the Preserve drop and PTPO’s advantage may well shrink as scale pushes endpoint accuracy up; showing how the gap moves with model or data size would address this.

  2. The motivating diagnosis is thin. The Preserve drop under GRPO-Marginal is about 3 points, there is no analysis of why it happens, and it could come from a training mix with many deletion worlds rather than from per-world reward as such. Training GRPO on unperturbed data, or with a different Drop/Preserve ratio, would separate the two explanations.

  3. The margin over the strongest baseline is small, and that baseline is not especially strong. CRPO-Evidence is the authors’ own adaptation of a method designed for video LLMs. The significant gain over it is limited to Switch (+4.28 on 485 edges across two datasets) and falls to +1.44 on FEVER at matched endpoints (Fig. 5). A RAG-oriented RL method and a plain paired-consistency loss would be more informative baselines.

  4. It is unclear what Switch actually measures. As the paper itself shows, a Forced Switch edge must add evidence, so a higher SwitchAcc may reflect better use of newly added documents rather than proof replacement. The alternative-support documents in training and test also come from the same constructor, so an add-only control (add J and Y, keep R1) and a constructor- or template-level holdout are needed before the transfer result can be read as held-out generalization.

Questions for authors

see weaknesses

Reviewer_QfvLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models respond appropriately when their evidence changes: withdrawing unsupported conclusions, preserving unaffected content, and retaining conclusions supported by alternative proofs. The authors introduce EviPair, which constructs paired evidence worlds from MuSiQue, FEVER, and QAMPARI and evaluates independently generated answers using Change, Preserve, and Switch metrics. They also propose PTPO, which combines endpoint rewards with typed relational rewards and assigns credit through incident edges using shared rollouts and node-local leave-one-out baselines. With Forced Switch edges excluded from RL training, the paper reports an increase in macro SwitchAcc from 17.48 after SFT to 33.42. The reported improvement over CRPO-Evidence is 4.28 points, while the corresponding Change and Preserve improvements are not statistically significant.

Critical strengths
  1. The paper addresses an important problem with a clear motivation.

When the evidence changes, models should withdraw claims that are no longer supported while keeping the parts that remain valid. By studying withdrawal, preservation, and alternative-proof adaptation separately, the paper reveals behavior that final-answer accuracy alone would miss.

  1. The evaluation carefully defines which examples count.

Eligibility for Change, Preserve, and Switch is fixed before the model runs. Examples with incorrect source-side answers, missing citations, or tool failures remain in the evaluation. This avoids overstating performance by evaluating only cases the model already gets right. The paper also makes a useful distinction between naturally redundant evidence and forced proof switching.

  1. The control experiments address several plausible explanations for the gains.

Endpoint-LOO, PTPO-Shared, and Typed-Pair examine the effects of the credit convention, pairing strategy, and rollout reuse, respectively. The endpoint-matched experiments also address whether the behavioral gains simply reflect better answer accuracy. Some important controls are still missing, but the overall design covers several possible explanations.

Critical weaknesses
  1. The Switch metric does not establish whether the model actually learns to select an alternative proof.

The paper allows certificates to include evidence that is available but irrelevant. As long as the answer is correct, a model could therefore satisfy the Switch certificate requirements simply by citing every input document in each world. Without a cite-all baseline, citation precision measurements, or controls on the citation budget, the reported gains in proof adaptation could partly reflect a tendency to cite more broadly.

  1. The core evaluation labels depend on proof audits whose reliability has not been sufficiently established.

Forced Switch requires every old proof to become invalid. Missing even one proof could change whether an example qualifies. Although the paper reports human review, it does not adequately describe the scope of the proof search, the limits on evidence combinations considered, the stopping criteria, or the remaining label error rate in the test set. These gaps directly affect whether the benchmark measures the intended capability.

  1. The paper does not sufficiently isolate PTPO’s contribution over existing methods.

Compared with CRPO-Evidence, PTPO changes the reward formulation, the scope of the baseline, and advantage scaling at the same time. The typed reward also adds a semantic-precision gate. The existing ablations do not show whether the gains mainly come from modeling relationships, additional correctness supervision, or changes in optimization scale. This leaves PTPO’s specific contribution unclear.

Questions for authors

please refer to the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.