acceptodds
Under review as a conference paper at ICLR 2027

Evidence, Pressure, or Neither: What Actually Moves Costly Decisions in Large Language Models

25 views

Abstract

Language models are increasingly asked to commit to costly judgements and to revise them when challenged. Evaluations report a single revision rate against a control with the pushback removed, conflating revision that tracks the argument with revision that tracks the arguer and assuming that the control does not move. We propose Matched-Presence Decomposition (MPD), which separates revision into a content, a social and a re-ask component by replacing each manipulated element of the prompt with the same element from another item, together with six audits that decide whether each component is identified. On a prediction market, a bare second ask in the same conversation changed the answer in to of episodes in four open-weight families, and neither the framing sentence, greedy decoding nor dropping the word “updated” removed most of it. Across seven families, argument content moved every one and direction-free reputation none. The audits also mark where a reading fails: a peer-review placebo carries direction, and on political items part of what a published sycophancy benchmark scores is agreement with the persona's last assertion. A prompt that makes the model state whether the material is new removed to points of drift while preserving the response to evidence; the largest reduction, which a reduction-only evaluation would select, came from an instruction that removed up to of that response.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
evaluation methodology, sycophancy, decision revision

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_zSYzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper looks at how we measure sycophancy and belief revision in two-turn LLM evaluations. Usually, we ask a model for an initial judgment in turn 1, push back in turn 2, and compare that against an empty control where the pushback is removed. The authors point out two flaws with this setup: it lumps together revising because of a good argument (which we want) and revising because of who is speaking (which we don't), and it assumes the empty control is a true zero.

To separate these effects, the authors introduce Matched-Presence Decomposition (MPD). Instead of deleting elements in the control, MPD swaps each manipulated piece of turn 2 (the argument text, the user's reputation record, or the user's stated position/shares) with the corresponding element from a different episode in the pool, and adds a bare reask_only condition that drops even the framing sentence. They also define six audits to check whether the controls and metrics are actually valid before reading the coefficients.

Testing this on ~1000 comment episodes across 86 Manifold prediction markets and seven model families, they find three main things. First, simply asking the model a second time (reask_only) causes it to change its probability estimate in 65.6% to 86.6% of episodes across the four open-weight models, and greedy decoding barely reduces this. Second, when scored on whether the revision moves in the direction of the pushback (toward), argument content has a large positive effect across all seven models stated position has a small effect in five of seven , and the user's reputation record has zero effect across the board. Third, when testing prompt mitigations to reduce re-ask drift, telling the model it can repeat its previous answer (license_to_repeat) gives the biggest drop in drift (down to 4.2% on Qwen3-8B), but does so by making the model ignore real evidence (retaining only 43.3% of the evidence effect). Asking the model to first output a boolean for whether turn 2 contains new information (evidence_gate) cuts drift by 19 to 47 points while keeping the evidence response intact.

Critical strengths
  1. I really appreciate the methodological discipline in this paper. The core point of Audit 1 is simple and important: an empty control (zero_info) and a presence-matched donor control (argument_shuffled) do not behave the same way. Because showing an unrelated comment from another market actually makes the model revise less often than showing an empty prompt (and on Gemma-4-12B, the empty control triggers a 23.4% refusal rate because the prompt says another trader responded without showing the comment), standard evaluations that subtract the empty control end up understating the true effect of the argument by 2 to 17 points.

  2. I also respect how the authors handled their negative results and pre-registered checks. When they tried to transfer MPD to 50,339 ICLR peer review episodes, Audit 2 caught that the model's initial review score (6.46) sits above the human mean (5.52), so 78% of both real and donor reviewer scores pull the model downward (pagree=0.696p_{\text{agree}} = 0.696). Rather than trying to salvage those numbers, they voided the directional coefficients from that domain and used it as an example of what the audit catches. Same goes for dropping the "ungrounded revision" metric when Audit 4 showed the LLM judge agreed 100% of the time with a trivial string check.

  3. the mitigation analysis in Section 5.3 makes a great practical point. If you only evaluate anti-sycophancy interventions on how much they reduce flipping, you end up picking prompts that make the model stubborn to everything, including good evidence. Requiring both drift reduction and evidence retention is the right way to evaluate these fixes.

Critical weaknesses
  1. My biggest issue with the paper is that the headline claim about "bare re-ask drift" (Figure 1 and Section 5.2) is confounded by how the prompts were built for reply comments. In the main text, reask_only is described as asking the model a second time with "nothing attached." However, if you read Appendices B, G, and N carefully, 73.6% of the Manifold stimuli (1,014 out of 1,378) are replies to an earlier comment, and in all of those episodes, turn 2 prepends that parent comment (Earlier in the comment thread, another trader wrote: ...) even in reask_only. The model never saw that parent comment in turn 1. So for nearly three-quarters of the data, reask_only is not a bare re-ask at all; it is feeding the model a brand new comment from the thread and asking for an updated probability.

You can see how much this matters in Appendix G. When the authors run reask_verbatim (which repeats the turn 1 question exactly and drops both the word "updated" and the parent comment), Qwen3-8B's revision rate drops from 0.784 all the way down to 0.179, whereas reask_neutral (which only drops the word "updated" and keeps the parent comment) stays at 0.623. That means roughly 44 percentage points of Qwen3-8B's drift comes from showing the parent comment in turn 2, not from asking the question twice. Since 26.4% of the dataset (364 episodes) has no parent comment, the authors already have the clean data to separate this. Those 364 episodes should be reported separately in the main text, and Figure 1 needs to be caveated or updated so readers aren't misled into thinking a pure re-ask causes an 80% flip rate when most of those prompts injected new thread context.

  1. My second concern is about how the presence-matched control (argument_shuffled) interacts with the unconditional toward metric. Putting a comment about one prediction market into a prompt about a completely different prediction market is not a neutral placebo; it is a weird, off-topic non-sequitur. As Appendix G shows, the less topically related the donor comment is, the more it suppresses any revision at all (slope of +0.645 across similarity quartiles). Because toward is defined unconditionally as I[sign(Δ)⋅d>0]\mathbb{I}[\text{sign}(\Delta) \cdot d > 0], any episode where Δ=0\Delta = 0 is scored as 0. When an off-topic donor comment causes the model to freeze and keep its old probability (Δ=0\Delta = 0), toward drops mechanically, even if the donor comment's direction is random. So part of βEcontent\beta_E^{\text{content}} on toward is just measuring the fact that models freeze up when shown off-topic text from another market, rather than measuring persuasion by the real argument. Reporting toward conditional on Δ≠0\Delta \neq 0 (which is mentioned briefly in one sentence in Section 5.2) alongside the unconditional metric would make this much cleaner.

  2. the primary toward results rely on only 32.1% of the dataset (443 of 1,378 stimuli) where two LLM labellers (DeepSeek-V4-Pro and GPT-5) agreed on the comment's direction, and the authors note in Appendix K that they cancelled the human validation check on these labels. Throwing out 68% of the comments (48% as directionless, 20% for disagreement) selects for the most strongly opinionated comments in the pool. While the post-hoc Brier score analysis against market resolution in Appendix L is a great label-free check on 1,158 stimuli, it also reveals a problem with evidence_gate that gets brushed past: on Qwen2.5-7B, evidence_gate has the highest directional retention in Table 3 (1.377), yet in Table 5 its Brier score gain (real - donor) drops from -1.53 (significant) to -0.64 (not significant). Also, in Section 5.3, models under evidence_gate still claim "new_information": true on 32% to 53% of episodes where no comment was shown at all. So evidence_gate is still hallucinating new information nearly half the time on empty re-asks.

  3. I am not convinced by how the reputation channel (βRcontent\beta_R^{\text{content}}) and its linear probe (vRv_R in Appendix O) are set up. In the prompt (Appendix N), reputation is a multi-field metadata string (account year, lifetime mana deposited, number of traders, trustworthy flag, comment likes). When you swap a commenter's metadata with a random donor's metadata from another episode, how is the "direction" of that swap defined for toward or for the linear probe? Does a donor have higher or lower reputation than the real commenter? If you don't condition on the sign of the reputation difference between the real user and the donor user, then sometimes the real user has higher reputation and sometimes the donor does, which will average out to zero on toward even if the model cares about reputation. The paper never explains how direction is assigned to βRcontent\beta_R^{content} or what binary label the linear probe vRv_R is trained to predict.

Concers on writing and presentation

the paper is written in a very compressed, telegraphic style where key definitions are tucked into the appendix. Section 3.3 summarizes Audits 3 through 6 in four sentences without giving their formulas, and the nine experimental conditions are never laid out in a clean table in the main text. Also, in Section 4.2 and Figure 3, the authors repeatedly state that coefficient point estimates cannot be compared between k=2k=2 (open-weight) and k=1k=1 (API) runs because per-stimulus noise differs. Changing the number of Monte Carlo draws per stimulus changes the standard error of the mean, not the expected value of the sample mean contrast. Unless temperature changed (as in Claude-Sonnet-5), the point estimates are estimating the exact same quantity.

Questions for authors
  1. On the 26.4% of Manifold episodes (364 stimuli) that are top-level comments with no parent comment injected in turn 2, what are the revision rates for reask_only, reask_neutral, and reask_verbatim across the four open-weight models?

  2. In Table 1, how is βRcontent\beta_R^{\text{content}} scored on toward? Specifically, is d∈{−1,+1}d \in \{-1, +1\} still just the direction of the comment text? If so, why would replacing a random donor's reputation record with the real commenter's reputation record increase alignment with dd unless the real commenter systematically has higher reputation than the donor? Shouldn't the contrast be conditioned on whether the real commenter's reputation exceeds the donor's?

  3. What exact binary target or contrast is the linear probe for vRv_R and vPv_P trained on in Appendix O?

  4. Why does evidence_gate report "new_information": true in 32% to 53% of episodes where no comment was shown, and is that rate coming primarily from the 73.6% of episodes where the parent comment appears in turn 2?

  5. Why does Qwen2.5-7B show 137.7% retention on toward under evidence_gate (Table 3), but lose more than half of its Brier score improvement (real - donor dropping from -1.53 to -0.64 in Table 5)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer zSYz (1/2)

Thank you for the detailed and thoughtful reviews.

Writing. We revise accordingly: the conditions table is added (Table 2, App. B); Audits 3–6 are defined in full in §3.3, with the realized values in App. N; the claim that k = 1 and k = 2 estimates are not comparable is removed from §4.2, and the Table 3 note, the Fig. 3 caption, App. D, App. M and Limitations now state that k changes precision only.


W1 / Q1. Parent comment; the three re-asks on the 364 no-parent stimuli; Fig. 1.

The parent accounts for 6–8 points of re-ask drift, not the 44 points inferred in the review. The verbatim fall is a wording effect, and the claim survives removing the parent. The four-placement table (full analysis in CQ1), then the requested numbers on the 364 no-parent stimuli:

P(Δ≠0), 1,014 reply stimulino parent anywhereparent in both turnsparent arrives in turn 2 (paper)
Qwen3-8B0.7160.7750.795
Qwen2.5-7B0.8230.8550.883
Gemma-4-12B0.5740.6620.656
GPT-OSS-20B0.7730.8380.853
P(Δ≠0), 364 no-parent stimulireask_onlyreask_neutralreask_verbatim
Qwen3-8B0.7510.3760.206
Qwen2.5-7B0.8190.4350.657
Gemma-4-12B0.6550.0580.563
GPT-OSS-20B0.7880.8540.873

The verbatim fall occurs where no parent exists, so the 44-point attribution does not hold. The fall is wording, and it is specific to Qwen3-8B. Fig. 1 now hatches the parent's share and gives both numbers.


W2. Donor comments are not neutral; counting "no change" as 0 mixes suppression into toward.

We agree that the donor comment suppresses revision. Conditional on revising, the donor direction is at chance, and the content contrast survives. The paper reports the suppression gradient. Conditional on revising, the donor direction is at chance (0.50–0.57) in all seven families, and the real comment's is 0.76–0.87; the content contrast is +0.24 to +0.36 on that reading (Appendix I, added in the revision):

share moving toward the comment's side, among episodes that revised (443 agreed)donor commentreal commentreal − donor
Qwen3-8B0.5570.843+0.287 [+0.211, +0.370]
Qwen2.5-7B0.5020.832+0.330 [+0.267, +0.399]
Gemma-4-12B0.5730.842+0.270 [+0.179, +0.374]
GPT-OSS-20B0.5180.761+0.242 [+0.185, +0.307]
DeepSeek-V4-Pro / Claude-Sonnet-5 / GPT-5.50.51 / 0.54 / 0.510.87 / 0.83 / 0.84+0.36 / +0.30 / +0.33

W3 / Q4 / Q5. The 443 subset; the Qwen2.5-7B Brier advantage halved while direction-scored retention is 1.38; the gate reports new information on 32–53% of empty re-asks.

The 443 subset is an acknowledged limitation. Both retentions are intact: the Brier advantage shrank because the gate removed the donor comment's harm, not the response to evidence. The 32–53% is the parent block.

Labels (W3): we agree; the submission states this limitation in Limitations (App. A) and in App. L. We are hand-checking the direction labels against the model labels now and will post the agreement rates as soon as this is done.

However, the limitation does not reach the paper's conclusions. The two labellers agree on the three-way label (YES, NO, no direction) in 80.0% of the 1,378 stimuli (Cohen's κ = 0.66). Of the 227 disagreements, 179 have one labeller reading a direction where the other reads none, and 48 have opposite directions. Every result that needs no direction (revision rate, |Δ|, re-ask drift, each mitigation's drift reduction) uses all 1,378 stimuli. The label-free Brier results on 1,158 resolved stimuli agree in sign with the direction-scored results in all seven families. If the agreed subset over-represents clearly argued comments, that can make the evidence effect look larger than it is; it cannot create a reputation effect of zero where there is one: the reputation contrast compares the same argument under a different author, and selecting on how clearly the argument is made does not touch that comparison. Brier (W3, Q5): the paired test is in CQ2; the outcome-scored retention ratio fell because its baseline value included the donor comment's harm:

Qwen2.5-7B, Brier change ×100real commentdonor commentreal − donor
baseline+0.43+1.96 [+0.76, +3.19]−1.53
evidence_gate−0.16+0.48−0.64

Gate (W3, Q4): it reports "new information" on 39.5–72.0% of empty re-asks with a parent and on 0.1–10.9% without, so the gate sees text that is new; §5.3 is corrected.


AuthorsOpenReview ↗

Response to Reviewer zSYz (2/2)

W4 / Q2 / Q3. Direction of the reputation channel; split by who outranks whom; probe target.

toward is always scored against the comment's direction; the reputation block has none. The reputation contrast asks whether the same argument is followed more closely under a different author. We split it by lifetime deposits, which every record shows; the real commenter's deposits are higher on 50% of stimuli:

β_R on towardreal > donorreal < donor
Qwen3-8B+0.012 [−0.005, +0.032]+0.013 [−0.003, +0.033]
Qwen2.5-7B+0.000 [−0.024, +0.022]+0.009 [−0.013, +0.033]
Gemma-4-12B−0.022 [−0.043, −0.004]−0.009 [−0.030, +0.015]
GPT-OSS-20B−0.005 [−0.040, +0.037]+0.019 [−0.030, +0.066]
DeepSeek / Claude / GPT-5.5−0.039 / +0.029 / +0.010−0.013 / +0.021 / +0.004

The null holds in both halves for every family. The platform's "trustworthy" badge differs between the real commenter and the donor on 400 stimuli; split that way, each cell of the labelled subset holds only 36–136 pairs, and the contrasts scatter on both sides of 0 across families (App. I). Probe target: the real reputation block of a stimulus is labelled 1 and its donor-swapped block 0; the train/test split is by stimulus, so both rows of a pair fall on the same side (App. P).


Reviewer_zSYzOpenReview ↗

Official comment by Reviewer zSYz

I thank the authors for their detailed response and extended experiments. Raising my score accordingly and also posting few followups:

  1. In your response, you noted that you were hand checking a sample of the direction labels against the two LLM labellers and would post the agreement rates once complete. Could you share those human agreement numbers before the discussion period closes?
AuthorsOpenReview ↗

Update of human agreement numbers

We sincerely thank you for your thoughtful and constructive feedback. We especially appreciate your willingness to raise your score in light of our response and additional experiments.

We have now completed the human annotation study, with the results reported in Appendix L. Specifically:

  • Human annotation: Two annotators independently labelled all 1,378 comments using blind sheets and the same annotation guidelines provided to the model labellers.
  • Inter-annotator agreement: The two human annotators achieved 83.3% agreement on the three-way labels (Cohen's κ = 0.74), compared with 80.0% (κ = 0.66) between the two model labellers. When both human annotators assigned a direction, their directional agreement was 96.1%.
  • Human–model agreement: On the 443 stimuli in the primary subset, Human Annotators 1 and 2 achieved 88.5% and 88.7% agreement with the two LLM labellers' consensus labels, respectively.
  • Directional consistency: Among the 396 stimuli for which both human annotators and both model labellers assigned a direction, both human annotators matched the LLM labellers' consensus direction in 95.2% of cases.
  • Conservative subset selection: The human annotators agreed on a directional label for 644 comments, but 443 were selected by the model labellers. This suggests that the model labellers were more conservative in assigning directional labels.
  • Example of ambiguity: A comment questioning the authenticity of a Netanyahu livestream was labelled NO by one annotator, who interpreted it as genuine skepticism, and YES by the other, who read it as sarcasm mocking such skepticism. This illustrates how different interpretations of sarcasm can lead to disagreement.
  • The original 443-stimulus subset remains unchanged.

We hope these additional results address your concern, and we sincerely appreciate your valuable feedback and continued engagement.

Reviewer_nfXuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why LLMs change their answers, separating the effects of evidence, social pressure, and simply asking the same question again. It finds that models respond strongly to evidence, minimally to reputation, and frequently change their numerical answers even when asked again without receiving new information.

Critical strengths

Introduces a rigorous experimental framework with matched controls and statistical audits to isolate what causes LLMs to change their answers.

Critical weaknesses

Does not establish whether changing answers when asked again reflects undesirable instability or legitimate reconsideration of existing information.

Questions for authors
  1. Could models changing their answers when asked again reflect meaningful reconsideration rather than arbitrary instability?

  2. Does reducing these answer changes actually improve decision accuracy, or does it simply encourage models to stick with their initial answers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nfXu

Thank you for the positive assessment and the two conceptual questions.


Q1. Could re-ask revision be legitimate reconsideration rather than instability?

Four observations say mostly not. (i) The direction is uninformative: on empty re-asks, revisions go toward the side argued by the comment that was withheld from the prompt 44.5–53.5% of the time, which is chance. (ii) It is not sampling noise: two samples of the same prompt disagree 17–68% of the time, far below the drift, and greedy decoding lowers the drift by only 0.3–3.8 points. (iii) It is not new information: with the parent in both turns, the drift is unchanged (full analysis in CQ1):

P(Δ≠0), 1,014 reply stimulino parent anywhereparent in both turnsparent arrives in turn 2 (paper)
Qwen3-8B0.7160.7750.795
Qwen2.5-7B0.8230.8550.883
Gemma-4-12B0.5740.6620.656
GPT-OSS-20B0.7730.8380.853

(iv) Scored on the resolution, the revisions neither help nor hurt on average; and when the gated model itself says "no new information", it revises in only 1.4–2.6% of episodes; GPT-OSS-20B is the exception at 42.0%, a leak the paper reports. Some individual revisions may be genuine reconsideration, which is why the paper calls it drift and not error.


Q2. Does reducing drift improve accuracy, or just anchor the model to its first answer?

The two prompts in Table 5 (Table 3 of the submission) are those two cases. license_to_repeat anchors: it has the largest drift reduction, but its evidence retention is 0.43 on Qwen3-8B and 0.65 on Gemma-4-12B, so the rule calls it an over-correction. evidence_gate keeps retention ≥0.99 in all four families. On accuracy, the honest statement is: the gate does not raise mean Brier; it removes prompt-unrelated revisions without losing the response to evidence. On Qwen2.5-7B, what the gate removed was the donor comment's harm (full analysis in CQ2):

Qwen2.5-7B, Brier change ×100real commentdonor commentreal − donor
baseline+0.43+1.96 [+0.76, +3.19]−1.53
evidence_gate−0.16+0.48−0.64

Thank you for your review, this answers most of my questions, but I will be keeping my score.

Thank you for your feedback, we are glad our clarifications addressed most of your questions.

Reviewer_pbSwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MPD, an evaluation framework that separates changes in LLM forecasts into effects of argument content, speaker attributes, and repeated questioning. They evaluate 86 forecasting questions and report that prompts requiring an explicit assessment of new information reduce the frequency of revisions while preserving responsiveness to argument content.

The study addresses the important question of why LLMs revise their answers. However, the paper does not adequately demonstrate the originality of its evaluation design and prompt modifications. The small question set and the ambiguity of the control conditions restrict the scope of the findings. Further validation is required to support broader claims.

Critical strengths

The authors evaluate answer revisions based on their frequency, magnitude and alignment with the direction of the comment. Their comparisons demonstrate that evaluators who consider only revision frequency may favour prompts that suppress responsiveness to relevant arguments.

The authors examine seven model families, conducting control audits and comparisons across multiple conditions. They exclude peer-review results that fail an audit and reject a mitigation that suppresses responsiveness to argument content.

Critical weaknesses

The study only covers 86 distinct forecasting questions, and the 1,378 comments do not represent independent questions. Furthermore, the authors restrict the primary directional analysis to just 443 cases. The direction labels depend on the agreement of two LLMs and the authors provide no human validation of these labels.

The authors retain parent comments in the re-ask condition, but do not make it clear whether the first response also contains them. This condition may therefore introduce new information.

Questions for authors

Did the authors include parent comments in the first-turn input? Can they reproduce the main findings after removing the comments from both turns?

Is the proposed prompt still effective when used with a larger set of questions with human-validated direction labels, and does it improve Brier scores?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pbSw

We thank the reviewer for providing thoughtful and constructive comments on our study.


W2 / Q1. Does turn 1 contain the parent? Do the results hold with it removed from both turns?

Turn 1 never contains the parent, and the results hold with it removed from both turns. Appendix O (exact prompts) now states that turn 1 never shows the parent. With the parent removed from both turns, the re-ask drift is 0.726 / 0.822 / 0.597 / 0.776 over all 1,378 stimuli (submission: 0.784 / 0.866 / 0.656 / 0.836). With the parent in both turns, the drift is unchanged within noise (full analysis in CQ1):

P(Δ≠0), 1,014 reply stimuliNo parent anywhereParent in both turnsParent arrives in turn 2 (paper)
Qwen3-8B0.7160.7750.795
Qwen2.5-7B0.8230.8550.883
Gemma-4-12B0.5740.6620.656
GPT-OSS-20B0.7730.8380.853

W1 / Q2. 86 markets, 1,378 dependent comments, 443 labelled, no human check. Do the prompts still work on a larger set with human labels? Does Brier improve?

We agree that the comments are not independent, which is why every interval and permutation test in the paper clusters on the market. The pool is filtered by a pre-registered activity floor: every Manifold market with at least 20 traders (fixed before any result was computed), which admits 86 markets and 1,378 comments.

We have now completed independent human annotation of all 1,378 comments. The two annotators achieved 83.3% agreement on the three-way labels (Cohen's κ = 0.74), and their labels matched the LLM consensus on the original 443-stimulus subset at 88.5% and 88.7%, respectively (Appendix L).

However, our main findings are not solely dependent on these direction labels. The two LLM labellers agree on the three-way label (YES, NO, no direction) in 80.0% of the 1,378 stimuli (Cohen's κ = 0.66). Of the 227 disagreements, 179 have one labeller reading a direction where the other reads none, and 48 have opposite directions. Every result that needs no direction (revision rate, |Δ|, re-ask drift, each mitigation's drift reduction) uses all 1,378 stimuli. The label-free Brier results on 1,158 resolved stimuli agree in sign with the direction-scored results in all seven families. If the agreed subset over-represents clearly argued comments, that can make the evidence effect look larger than it is; it cannot create a reputation effect of zero where there is one: the reputation contrast compares the same argument under a different author, and selecting on how clearly the argument is made does not touch that comparison.

With our current manpower, we cannot build a larger question set, and building a benchmark is outside the scope of this paper. Re-ask drift is, on average, neither helpful nor harmful when scored against the resolution. The gate does not improve mean forecast accuracy; instead, it reduces re-ask drift while preserving responsiveness to evidence. That advantage holds in three families; on Qwen2.5-7B, the baseline real-minus-donor advantage was mostly due to the donor comment harming the forecast, which the gate reduced (full analysis in CQ2):

Qwen2.5-7B, Brier change ×100Real commentDonor commentReal − donor
baseline+0.43+1.96 [+0.76, +3.19]−1.53
evidence_gate−0.16+0.48−0.64

W3. Novelty of the evaluation design and prompts.

We thank the reviewer for raising this important point and giving us the opportunity to clarify our contribution. The novelty of MPD lies not in individual controls or prompt modifications, but in a more rigorous and systematic framework for auditing LLM answer revisions. Our experiments reveal three overlooked issues: (i) substantial revision without new evidence, (ii) revision suppression by unrelated donor comments, and (iii) mitigation strategies that reduce revision at the cost of responsiveness to relevant arguments. We believe MPD's contribution lies in making these failure modes measurable and auditable across model families.

AuthorsOpenReview ↗

Update of human agreement numbers

We have now completed the human annotation study, with the results reported in Appendix L. Specifically:

  • Human annotation: Two annotators independently labelled all 1,378 comments on Manifold, using blind sheets and the same annotation guidelines provided to the model labellers.
  • Inter-annotator agreement: The two human annotators achieved 83.3% agreement on the three-way labels (Cohen's κ = 0.74), compared with 80.0% (κ = 0.66) between the two model labellers. When both human annotators assigned a direction, their directional agreement was 96.1%.
  • Human–model agreement: On the 443 stimuli in the primary subset, Human Annotators 1 and 2 achieved 88.5% and 88.7% agreement with the two LLM labellers' consensus labels, respectively.
  • Directional consistency: Among the 396 stimuli for which both human annotators and both model labellers assigned a direction, both human annotators matched the LLM labellers' consensus direction in 95.2% of cases.
  • Conservative subset selection: The human annotators agreed on a directional label for 644 comments, but 443 were selected by the model labellers. This suggests that the model labellers were more conservative in assigning directional labels.
  • Example of ambiguity: A comment questioning the authenticity of a Netanyahu livestream was labelled NO by one annotator, who interpreted it as genuine skepticism, and YES by the other, who read it as sarcasm mocking such skepticism. This illustrates how different interpretations of sarcasm can lead to disagreement.
  • The original 443-stimulus subset remains unchanged.

We hope these additional results address your concern, and we sincerely appreciate your valuable feedback.

Dear Reviewer pbSw,

Thank you again for your review and questions. They helped us clarify the paper and strengthen the analysis. We accordingly revised the manuscript to improve readability and clarity.

We have completed the human annotation study and added the results to Appendix L. Two annotators independently labeled all 1,378 Manifold comments. On the original 443-stimulus subset, the two annotators agreed with the model consensus on 88.5% and 88.7% of cases, respectively. We also found that re-ask drift remained substantial after removing the parent comment from both turns. The evidence gate preserved the forecasting advantage of real over donor comments in three model families.

MPD makes it possible to distinguish revisions driven by evidence from those triggered by re-asking alone, and to test whether interventions reduce the latter while preserving responsiveness to evidence.

As the discussion period draws to a close, we would appreciate hearing whether these results and our responses address your concerns. We hope you will consider them in your final assessment, and we are happy to clarify any remaining points.

Thank you for your time.

Reviewer_f7PkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Matched-Presence Decomposition (MPD), comparing real prompt elements with elements drawn from other items to distinguish content, reputation, position and re-asking effects. It combines three revision outcomes with six identification audits. Experiments use 1,378 comments on 86 prediction markets across seven reported models; peer-review data and a sycophancy benchmark provide additional audit cases. Four-model mitigation experiments compare reducing revision with retaining argument responsiveness.

Critical strengths
  1. Separating revision frequency from directional alignment produces a useful empirical distinction. In Appendix G, adding a real comment changes revision frequency by only 0.2–3.2 percentage points relative to reask_only, while changing toward by 26.9–34.7 points. The paired comparisons and market-clustered inference make the distinction more informative than an aggregate flip rate. The withdrawal of direction-scored peer-review conclusions after the placebo audit also demonstrates a consequential use of the audits (Appendix J).

  2. Evaluating mitigation through both reduction and retention exposes blanket inertia. For Qwen3-8B, license_to_repeat removes 74.2 points of revision but retains only 0.433 of the direction-scored content response, versus a 32.1-point reduction and 1.232 retention for evidence_gate (Table 3). Appendix L adds a valuable label-free outcome check rather than equating argument agreement with correctness.

Critical weaknesses
  1. The headline no-information drift is confounded by newly presented context. Turn 1 contains the question, description and price, whereas turn 2 includes a parent comment in every condition for 1,014/1,378 items (Appendices B and N). Thus the pooled 65.6–86.6% reask_only rates do not isolate re-asking without new information. Appendix G's verbatim re-ask, which drops the parent comment on replies and also changes the wording, lowers Qwen3-8B's revision rate from 78.4% to 17.9%, while the other three families still revise in 49.0–86.0% of episodes, at or above their reask_neutral rates. Appendix Q's assertion that zero-information conditions contain no comment also needs reconciliation. Holding parents constant across second-turn arms preserves those contrasts; it does not remove this turn-1/turn-2 confound. The same issue affects interpreting the gate's control judgments as erroneous and its reduction as removal of ungrounded drift.

  2. MPD establishes condition contrasts more securely than pure components. Equation 2 is an algebraic identity; a nonzero placebo-minus-baseline contrast does not establish which control is neutral. Appendices A/G show that unrelated donors actively suppress revision. Since toward equals revision probability times conditional alignment, chance donor alignment does not eliminate that suppression from the content contrast. The reported directional alignment and Brier results support genuine argument responsiveness, but do not justify treating the entire contrast as an identified correction to an empty-control estimate. One fixed donor derangement (Appendices A/P) and the restriction of primary direction-scored results to the 443 of 1,378 stimuli (32.1%) to which both API labellers assign the same direction, without a human check (Appendix K), further limit that interpretation.

Questions for authors
  1. Clarification and decisive control: Are parent comments absent from turn 1 as Appendix N indicates? Please reconcile Appendix Q and report drift and gate behavior separately on the 364 parent-free items. A control placing identical parent context in both turns would separate context arrival from re-asking.

  2. Identification clarification: What assumptions make real-minus-donor a pure content effect despite donor suppression? Please distinguish the randomized condition contrast from the proposed component interpretation and assess sensitivity to donor reassignment. Report toward, away and ties on the same labelled subset.

  3. Mitigation clarification: What is the final reduction threshold after Appendix E demotes removable-normalized reduction to the absolute drop, and what are the intervals that Table 4's rule requires? Appendix L also notes that Qwen2.5-7B's real-minus-donor Brier change (Table 5, ×100) moves from −1.53 to −0.64 under the gate, with an interval that includes zero, despite a direction-scored retention of 1.377 (Table 3); please report a paired test of whether outcome retention holds.

5 comments
AuthorsOpenReview ↗

Response to Reviewer f7Pk

Thank you for your detailed and thoughtful review.

Q1. Parent comment; both-turns control; Appendix R; does the gate misfire without a parent?

We ran the both-turns control. The parent comment accounts for a few points of re-ask drift, and the claim survives its removal. The both-turns control shows that the parent's arrival as new text is not what drives the drift. The table is below; the full analysis is in CQ1.

P(Δ≠0), 1,014 reply stimulino parent anywhereparent in both turnsparent arrives in turn 2 (paper)
Qwen3-8B0.7160.7750.795
Qwen2.5-7B0.8230.8550.883
Gemma-4-12B0.5740.6620.656
GPT-OSS-20B0.7730.8380.853

The arrival effect (turn 2 only minus both turns) is −0.6 to +2.8 points, and every interval covers 0. Without a parent in turn 2, the drift is 0.716–0.865 whether or not turn 1 showed it. On empty re-asks the gate reports "new information" in 0.1–10.9% of the no-parent rows, against 39.5–72.0% of the rows with a parent, so the "32–53%" in the submission was the parent; §5.3 now says so. Appendix R now says that zero_info carries no comment of its own; the parent is shared by every condition.

Q2. The donor comment; report toward / away / no-change; try another derangement.

We agree that the donor comment is not a pure control: what we estimate is the difference between two conditions (real comment vs donor comment), not a "pure" content effect with every other influence removed. We ran both of the checks: the toward / away / no-change shares for every condition, and the placebo regenerated under two new random donor assignments (both added to Appendix I in the revision).

The content effect does not come from the donor comment suppressing revision. Among the episodes that did revise, the direction under the donor comment is at chance and under the real comment is 0.76–0.87, so the content contrast survives (+0.24 to +0.36). The two new donor assignments move the content contrast by at most 2.3 points.

share moving toward the comment's side, among episodes that revised (443 agreed)donor commentreal commentreal − donor
Qwen3-8B0.5570.843+0.287 [+0.211, +0.370]
Qwen2.5-7B0.5020.832+0.330 [+0.267, +0.399]
Gemma-4-12B0.5730.842+0.270 [+0.179, +0.374]
GPT-OSS-20B0.5180.761+0.242 [+0.185, +0.307]
DeepSeek-V4-Pro / Claude-Sonnet-5 / GPT-5.50.51 / 0.54 / 0.510.87 / 0.83 / 0.84+0.36 / +0.30 / +0.33

The suppression you describe is in the no-change column, not in the direction:

toward / away / no-change (443 agreed)empty controldonor commentreal comment
Qwen3-8B0.43 / 0.37 / 0.190.32 / 0.25 / 0.430.76 / 0.14 / 0.10
Qwen2.5-7B0.45 / 0.43 / 0.110.39 / 0.38 / 0.230.80 / 0.16 / 0.04
Gemma-4-12B0.36 / 0.31 / 0.330.23 / 0.17 / 0.600.67 / 0.13 / 0.21
GPT-OSS-20B0.47 / 0.45 / 0.080.35 / 0.33 / 0.320.73 / 0.23 / 0.04

Under two new random donor assignments, the content contrast on toward is:

β_E on towardoriginalassignment 2assignment 3
Qwen3-8B+0.444 [+0.364, +0.523]+0.460 [+0.393, +0.515]+0.449 [+0.376, +0.531]
Qwen2.5-7B+0.410 [+0.352, +0.473]+0.410 [+0.340, +0.482]+0.407 [+0.343, +0.487]
Gemma-4-12B+0.442 [+0.366, +0.545]+0.448 [+0.363, +0.548]+0.465 [+0.377, +0.552]
GPT-OSS-20B+0.378 [+0.320, +0.441]+0.391 [+0.321, +0.452]+0.385 [+0.334, +0.447]

Every difference between assignments has an interval covering 0.


Q3. The decision rule (original Table 3, revision Table 5); intervals; paired Brier test.

The rule in force is the pre-registered one: a protocol must remove ≥30% of the removable drift and keep evidence retention ≥0.80. Removable drift is the drift above the resampling floor, i.e. above how often two samples of the same prompt already differ; retention is the evidence effect under the protocol divided by the effect at baseline. The submission quoted absolute drops without stating a numeric threshold for them. The revision states the rule in the caption and gives an interval in every cell. Under a literal "any drop and retention ≥0.80" reading, commit_then_reveal would pass on Qwen2.5-7B (+0.067 [+0.037, +0.100]) and Gemma-4-12B (+0.056 [+0.028, +0.087]). evidence_gate and license_to_repeat have the same verdict under both readings (App. H). Paired Brier test: +0.88 [−0.02, +1.83] on Qwen2.5-7B (p = 0.061), and within noise of 0 elsewhere. The Brier contrast shrank because the gate removed the donor comment's harm, not the response to the real comment. The decomposition (full analysis in CQ2):

Qwen2.5-7B, Brier change ×100real commentdonor commentreal − donor
baseline+0.43+1.96 [+0.76, +3.19]−1.53
evidence_gate−0.16+0.48−0.64

Dear Reviewer f7Pk,

Thank you again for your review and the questions, which helped us strengthen or further clarify the paper. In response, we added a control with the same parent comment in both turns, checked sensitivity to two new donor assignments, and clarified the mitigation rule alongside the paired Brier results.

As the discussion period ends soon, we would like to kindly ask whether our responses have addressed your concerns. We would greatly appreciate it if you could consider these results in your final assessment, and we are happy to provide any further clarification.

Thank you for your time.

I appreciate the control that holds the parent comment fixed across turns. Models still revise their forecasts under that control, which addresses my concern about introducing the comment only in the second turn. When forecasts change, they are more likely to move in the direction of a real comment than of a donor comment.

My reservation is with the claim in Figure 4 that the empty control understates the effect of evidence. The donor comments themselves make models less likely to revise, so part of the overall difference comes from the behavior of the control. Average forecast accuracy was broadly stable under re-asking, which also leaves the benefit of suppressing drift unclear. I am keeping my original score of 2 (weak rejection).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All the Reviewers 1 (Updated)

We thank the reviewers for their thoughtful feedback, for recognizing the strengths of the work, and for pointing out where it can be improved.

All new experiments use the same 1,378 stimuli, the same turn-1 answers and the same 95% market-cluster bootstrap as the paper. They are descriptive and change no pre-registered rule.


Common Question 1 (f7Pk, pbSw, zSYz): How much of the re-ask drift is the parent?

We ran two new conditions on the same turn-1 answers. The first is reask_only with the parent block deleted. The second is f7Pk's control: the same parent block is placed in turn 1, so turn 2 contains no text the model has not already seen.

The conclusion is: re-ask drift comes from being asked again, not from the parent comment. The parent shifts the rate by a few points, and no claim in the paper depends on it. The evidence_gate prompt (the mitigation we recommend, which first asks whether turn 2 contains new information) reacts to text that is genuinely new; it does not hallucinate new information.

P(Δ≠0), 1,014 reply stimulino parent anywhereparent in both turnsparent arrives in turn 2 (paper)
Qwen3-8B0.7160.7750.795
Qwen2.5-7B0.8230.8550.883
Gemma-4-12B0.5740.6620.656
GPT-OSS-20B0.7730.8380.853

The parent accounts for 6.0–8.2 points of re-ask drift. Its arrival as new text accounts for −0.6 to +2.8 points, and every interval covers 0. Pooled over all 1,378 stimuli, the parent-free drift is 59.7–82.2%, against 65.6–86.6% in the submission. reask_verbatim, which repeats the turn-1 question word for word, cuts Qwen3-8B's rate from 0.751 to 0.206 even on the 364 stimuli that never had a parent, so that fall is a wording effect, not the parent. The gate's "new information" reports on empty re-asks come from the parent block: it reports new information on 39.5–72.0% of the empty re-asks that show a parent and on 0.1–10.9% of those that show nothing.


Common Question 2 (f7Pk, nfXu, zSYz): on Qwen2.5-7B the gate keeps the direction-scored evidence response, but the Brier advantage halves. Which is right?

Both. Brier change is the change in Brier score from turn 1 to turn 2 (×100; negative = the forecast moved closer to the resolution). The Brier advantage is that change under the real comment minus the change under the donor comment. We split it into its two terms:

Qwen2.5-7Breal commentdonor commentreal − donor
baseline+0.43+1.96 [+0.76, +3.19]−1.53
evidence_gate−0.16+0.48−0.64

At baseline, most of the advantage was the donor comment harming the forecast. The gate removed that harm, which is what it is for. The direction-scored retention measures how closely the model follows the argument. The argument points at the resolution in only 58.1% of pairs. When the argument is right, the real comment improves the forecast (Brier change under the real comment: −6.56 at baseline, −5.76 under the gate); when it is wrong, the real comment worsens the forecast (+8.51, +7.72). The gate changes neither, so the model follows the argument as much as before. The paired test the reviewers asked for compares the advantage under the gate with the advantage at baseline: on Qwen2.5-7B the advantage shrinks by +0.88 [−0.02, +1.83], p = 0.061; for the other three families the interval covers 0.


Changes to the paper (marked blue):

  • Abstract, introduction, Fig. 1. The re-ask drift with the parent comment removed (59.7–82.2%).
  • §3.2. Points to conditions (Tab. 2, App. B), how toward is scored.
  • §3.3. Audits 3.
  • §4.2. The claim that k = 1 and k = 2 estimates are not comparable is removed.
  • §5.2. New paragraph and Tab. 1: re-ask drift by where the parent comment appears.
  • §5.3. The gate's "new information" reports are attributed to the parent block; Qwen2.5-7B Brier retention.
  • Tab. 5. The caption states the pre-registered rule, each cell has a 95% interval.
  • App. G. Tables for the parent comment: no-parent subset, parent removed, parent in both turns, gate reports by parent presence.
  • App. H. The protocol verdicts under both readings of the rule.
  • App. I (new). Toward / away / no-change shares, the direction conditional on revising, two further donor assignments, the reputation contrast split by standing.
  • App. L. Inter-labeller agreement: LLM (80.0%, κ = 0.66) and human (83.3%, κ = 0.74) on 1,378 comments. Human–LLM consensus agreement was 88.5% and 88.7% on primary subset.
  • App. M. Paired Brier test and the decomposition of the Qwen2.5-7B contrast.
  • App. O, P, R. Turn 1 never shows the parent; the probe's training target; the corrected description of zero_info.
  • Limitations. The parent share, the donor-assignment sensitivity, the sampling difference between local and API families.
  • We rephrased and reorganized parts of the paper for clarity.
AuthorsOpenReview ↗

Update of human agreement numbers (Updated)

We sincerely thank the reviewers for their thoughtful and constructive feedback. We have now completed the human annotation study,

Specifically:

  • Human annotation: Two annotators independently labelled all 1,378 comments using blind sheets and the same annotation guidelines provided to the model labellers.
  • Inter-annotator agreement: The two human annotators achieved 83.3% agreement on the three-way labels (Cohen's κ = 0.74), compared with 80.0% (κ = 0.66) between the two model labellers. When both human annotators assigned a direction, their directional agreement was 96.1%.
  • Human–model agreement: On the 443 stimuli in the primary subset, Human Annotators 1 and 2 achieved 88.5% and 88.7% agreement with the two LLM labellers' consensus labels, respectively.
  • Directional consistency: Among the 396 stimuli for which both human annotators and both model labellers assigned a direction, both human annotators matched the LLM labellers' consensus direction in 95.2% of cases.
  • Conservative subset selection: The human annotators agreed on a directional label for 644 comments, but 443 were selected by the model labellers. This suggests that the model labellers were more conservative in assigning directional labels.
  • Example of ambiguity: In S0042 (YES / NEITHER), one annotator read a question about video verification as implying that the bet had taken place, while the other read it as a request for verification. In S0081 (NO / NEITHER), one annotator read a joke about needing 78,000 participants as implying that the puzzle was unlikely to be solved in time, while the other read it as a joke about the number of possible guesses. In S1027 (YES / NO), one annotator focused on the ceasefire having lasted two days, while the other focused on the continued instability. (The sample IDs and their corresponding annotation labels are included in the supplementary ZIP file.)
  • The original 443-stimulus subset remains unchanged. Two annotators assigned the same directional label to 644 comments, and the two LLM labellers did so for 443 comments. Of the 396 comments on which both pairs reached directional agreement, the pairs agreed with each other on 377 (95.2%) and assigned opposite directions to the remaining 19.

We hope these additional results provide further evidence of annotation reliability.

AuthorsOpenReview ↗

Final Response to All Reviewers and the Area Chair

Dear Reviewers and Area Chair,

Thank you for the thoughtful feedback and discussion. MPD’s main contribution is a more rigorous and systematic framework for auditing why language models revise their answers. It (1) separates how often models revise from whether they revise in the argument’s direction, (2) audits control conditions rather than assuming they are neutral, and (3) evaluates whether interventions reduce re-ask drift while preserving responsiveness to evidence. Our experiments reveal overlooked issues, including revision suppression by unrelated comments and mitigations that discourage revision even in response to relevant evidence.

We are encouraged that the reviewers recognized the value of distinguishing revision frequency from directional responsiveness, the systematic identification audits, and the joint evaluation of drift reduction and evidence retention.

In response to the reviews:

  • Re-asking without the parent comment (f7Pk, pbSw, zSYz). With the parent comment absent from both turns, models still revised in 59.7–82.2% of episodes across the four open-weight families. We also ran a control with the same parent comment in both turns. The revision distinguishes these conditions and reports the evidence gate’s “new information” judgments separately by parent-comment presence (§§5.2–5.3, Appendix G, General Response 1).
  • Interpreting the matched control (f7Pk, zSYz). We clarify that real-minus-donor is a condition contrast, partly affected by donor-induced revision suppression, rather than a pure evidence effect. However, conditional on revision, directional alignment was near chance under donor comments (0.50–0.57) but substantially higher under real comments (0.76–0.87). Two new donor assignments changed the unconditional toward contrast by at most 2.3 percentage points (Appendix I).
  • Human validation of the direction labels (f7Pk, pbSw, zSYz). Two annotators independently labelled all 1,378 comments and agreed on 83.3% of the three-way labels (Cohen’s κ = 0.74). On the original 443-stimulus subset, their labels matched the LLM consensus in 88.5% and 88.7% of cases, respectively (Appendix L, General Response 1).
  • Mitigation and forecast accuracy (f7Pk, nfXu, pbSw, zSYz). We clarified the pre-registered rule requiring both drift reduction and evidence retention. The evidence gate met both criteria across the four tested families. For Qwen2.5-7B, the paired change in the real-minus-donor Brier contrast was +0.88 (×100), with a 95% interval of [−0.02, +1.83] (Table 5, Appendices H and M, General Response 1). While average forecast accuracy remained broadly stable, MPD reveals important differences in revision behavior that aggregate accuracy alone cannot capture.
  • Reputation contrast and probe target (zSYz). We clarified that toward is scored against the comment’s direction. The reputation contrast remained small when splitting the sample by whether the real commenter had higher or lower lifetime deposits than the donor (Appendix I). We also clarified that the linear probe distinguishes each stimulus’s original reputation or position record from its matched donor record (Appendix P).

We have also revised and reorganized the manuscript to clarify the interpretation and scope of our findings.

We sincerely thank all reviewers for their constructive feedback and hope these additional experiments and clarifications will help inform the final assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.