acceptodds
Under review as a conference paper at ICLR 2027

Correct Trace Conclusions, Wrong Answers: Unfaithful Capitulation Under Pressure

Abstract

Reasoning models are trusted in part because they show their work. We find that the reasoning trace and the final answer can nevertheless diverge. When a user pushes back, Qwen3 models sometimes reason their way to the correct option and then deliver a different one, a failure we call **unfaithful capitulation (UC)**. Using gold-blind extraction of each channel’s final commitment, we find that 30% of first capitulations across seven multi-turn runs follow a reasoning trace that still concludes correctly. The reasoning itself is not what breaks: holding each saved trace fixed and regenerating only the answer, replacing the challenge with a neutral request, or removing the dialogue it refers to restores 93–99% correct answers. The trace helps, but neutralization works almost as well without it, suggesting that the failure lies in how the final answer responds to the conversational context. Finally, because corrupting the trace removes UC as surely as fixing the answer does, agreement between reasoning and answer is the wrong repair target: trace correctness and answer correctness must be evaluated separately.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, chain-of-thought faithfulness, unfaithful capitulation, conversational consistency, sycophancy, reasoning evaluation, self-correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J57SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of Unfaithful Capitulation (UC) in Qwen3 reasoning models (1.7B to 32B). Under multi-turn user pushback on MCQ, the reasoning trace concludes with the correct option while the given answer is wrong. Using gold-blind extraxtion with two LLM judges, the paper finds that UC accounts for 30% of the first capitulations in seven runs. Keeping the trace fixed and by regenerating only the answer, replacing the challenge with a neutral request or by removing the prior dialog restores 93 to 99% of the correct answers, while a trace-omission control shows that neutralization works almost as well as without the trace. The paper concludes that the trace and answer correctness should be evaluated separately and that enforcing trace-answer agreement is the wrong repair target.

Critical strengths
  1. The problem is timely and has a real impact on today's usage of AI
  2. The phenomenon is well defined and relevant for CoT monitoring, outcome rewards and sycophancy eval. The taxonomy is simple and useful
  3. The authors are transparent in declaring the limitations, they clearly indicate all the post-hoc analyses, and the code release helps reproducibility
  4. The main point in Sec. 5 about removing the UC not being sufficient to sanitize the answer is sound and useful. It highlights how the "agreement" is not a successful metric
Critical weaknesses

Major

  1. The main result is weaker than presented. The mentioned 69 events drop to 37 with thresholds at 0.95/0.90, and none with 0.99/0.95. It is explained as the abstention on deliberative texts, but the 30% in the abstract heavily depends on non-calibrated thresholds. Also, human validation looks weak too, with 15 labels, one single author, and one label revised after seeing the judge's output. That said, "Spot-checked by blinded human review" in the abstract promises more than what was actually done

  2. The phenomenon sounds more prevalent than presented. At turn level, UC is about the 0.3% of the labeled turns. The abstract and intro make the phenomenon more widespread than what it is.

  3. Generality. Results only cover the Qwen3 family, and only in MCQ, while the free-form runs are uninterpretable. Qwen3 is a think/non-think model with a specific setup where the history contains visible answers only, and the system prompt enforces "the correct answer:". Have you tested with models from different families? A different family could have shown that the phenomenon is general and not an artifact of the template from Qwen3

  4. A baseline with no pressure is missing. The trace–answer disagreement rate at R0 is not reported separately from followup turns. The fixed-trace experiments suggest that pressure drives the decoupling on the selected cases, but a comparison at the population level with unpressured turns would make this claim much stronger. What is the trace-answer disagreement rate (UC and UP) at R0 compared with followup turns?

  5. Interpretation on the mechanism. The conclusion that "the challenge acts on how the answer is conditioned on the conversation, not mainly on the link between reasoning and answer" looks to be based on interventions that confound many factors: removing history changes length, turn structure, prior answers, and pressure altogether, and the authors acknowledge that. The interaction trace-by-neutralization has a wide interval ([-17.1,23.7]), so a component mediated by the trace is not excluded. Moreover, the paper would also benefit from better positioning the findings more explicitly against prior work already showing that context influences answers without appearing in the CoT [1,2]

[1] Turpin, M., et.al., Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.

[2] Chen, Y., et al., Reasoning Models Don’t Always Say What They Think. arXiv:2505.05410, 2025.

Minor

  1. The main judge (jev-1.13.0, TypeSafe API) is a proprietary service with uncalibrated probabilities, which may affect long-term reproducibility.
  2. Part of the analysis is post hoc (like the 8B replication and the trace-omission control) and the cases are selected. The authors disclose this, but it reduces the inferential weight.
  3. The writing is very dense and heavy: many acronyms and caveats in many sentences, with many important numbers in the appendix. It affects the readability
  4. The Fig. 1 example looks to be an arithmetic error in the answer (930/1000 = 9.3), not just a capitulation. A second example showing an option switch would make the phenomenon clearer.
Questions for authors

See weaknesses

4 comments

Thank you for the detailed review. We take the five major points in order, then the minor ones.

1. Thresholds and human validation. Two blinded annotators who are not authors labeled both channels of all 69 events and agree with each other and the judges on all 138 texts (App. M, p. 30). This supersedes the single-author spot check, and the abstract (p. 1) now describes it as it was done. On the thresholds: 107 of the 138 texts fail the strictest setting, and the annotators confirm every one. Stricter thresholds therefore discard correct labels, which is why the count falls with them (§3.2, p. 5). A third, open-weights judge finds 31 correct conclusions among traces that the first judge left unresolved, so the default count is conservative (App. L, p. 30).

2. How prevalent UC is. The abstract and the first finding now state both numbers together: UC is 0.3% of labeled turns and 30% of first capitulations (abstract, p. 1; §1, p. 2). It is rare as a share of all turns and common at the moment a model gives in to the user.

3. Other families, and the Qwen3 template.

  • Families: Kimi-K2.5 has 23 first UC events among 130 first wrong answers, and DeepSeek-reasoner 9 among 120. Four newer models capitulate less, with trace and answer together (§3.3, p. 5; App. K, p. 29).
  • Template: we re-ran Qwen3-14B and 32B without the instruction to begin with "The correct answer:". First UC events become more frequent (14B: 58 against 23; 32B: 10 against 5), so the instruction does not produce UC (App. G, p. 25, Table 19).
  • History: only visible answers enter later turns, as §2.2 (p. 3) states. Kimi-K2.5 and DeepSeek-reasoner, run through their providers' own templates, show UC as well.

4. A baseline without pressure. The paper now reports three (§3.1, p. 4; App. J, p. 28).

  • At the initial turn, 6 of 359 wrong answers with a resolved trace follow a correct trace conclusion (1.7%), against 69 of 151 at the first capitulation (46%).
  • Per turn, UC occurs in 0.15% of initial turns and 0.25% of follow-up turns with an all-correct prefix. Of the 117 UC and 24 UP turns, 6 UC and 1 UP are at the initial turn (§3.2, p. 5; §5.1, p. 8).
  • In re-runs with eight neutral requests in place of the challenges, UC falls from 23 of 4,162 at-risk turns to 1 of 4,539 for 14B, and from 5 of 4,404 to 0 of 4,750 for 32B.

5. Mechanism, confounds, and prior work. New controls separate the factors that history removal changed together (App. G, pp. 23–24). A single earlier exchange on the same question costs about 30 points under the challenge. A history of unrelated exchanges matched in tokens and turns costs nothing at 32B. Masking the earlier answers matters only when the trace is absent. On the interaction, §4.3 (p. 7) reports the interval you cite and states that a moderate trace-dependent component is not excluded. The claim is the direct result: the delivered answer changes while the saved trace stays fixed.

On Turpin et al. and Chen et al., §6 (p. 9) now states the relation. Those studies show a cue changing the answer while the chain of thought stays silent about it. Under UC the trace is not silent: it states the opposite conclusion, so the mismatch can be checked from the two texts alone.

Minor points.

  • The open-weights third judge gives a route for checking labels that does not depend on the hosted judge (App. L, p. 30).
  • Figure 1 (p. 2) now shows an option switch with no calculation; the arithmetic example is in App. A (p. 12).
  • The introduction is reorganized around the example and four findings, and post hoc analyses remain marked as such (§6, p. 9).

We hope these additions resolve the soundness concerns. If so, we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Full Results Tables for Reference

To save you going back and forth with the appendix, here are the complete numbers behind our answers to points 3–5, as reported in the revised PDF.

4. Baselines without pressure (Table 24, p. 28; jointly labeled turns, default thresholds):

Qwen3UC countDenominatorRate
Initial turn, among wrong answers63591.7%
First capitulation, among first wrong answers with a resolved trace6915145.7%
At-risk follow-up turns6927,7690.25%
Re-run (vLLM)UC / at-risk turns, eight challengesUC / at-risk turns, eight neutral requests
Qwen3-14B23/4,1621/4,539
Qwen3-32B5/4,4040/4,750

3. The answer-format instruction (whole conversations, same seeds; Table 19, p. 25):

RunFirst wrongTrace resolvedFirst UCUC / at-risk turns
14B, with instruction44272323/4,162
14B, without78625858/3,897
32B, with instruction231155/4,404
32B, without23151010/4,294

Fisher exact test for first UC events, without versus with: p = 7×10⁻⁵ at 14B and 0.30 at 32B.

5. Confounds in the fixed-trace edits (registered contrasts on UC cases, in points; all-case bounds and 95% case-cluster envelope; Table 18, p. 24):

Contrast14B bounds14B 95%32B bounds32B 95%
R0P − QP (one earlier exchange)[−34, −25][−49, −13][−30, −28][−43, −15]
FillP − QP (unrelated history, same tokens and turns)[−15, −11][−33, +4][+0, +0][+0, +0]
HNmask_o − HN_o (mask earlier answers, trace deleted)[−33, −31][−53, −11][−30, −29][−46, −15]
HNmask − HN (mask earlier answers, trace present)[−1, +2][−9, +13][+3, +3][+0, +8]
(HN − HP) − (HN_o − HP_o) (interaction with the trace)[−14, −2][−31, +17][−55, −50][−70, −34]

R0P: only the initial exchange before the challenge; FillP: unrelated exchanges matched in tokens and turns; HN: neutral request; HNmask: earlier answers replaced by "[answer removed]"; subscript o: trace deleted.

Thank you again for your time and thoughtful review. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our rebuttal. We hope our responses and revisions have addressed your concerns. If anything remains unclear, we would be glad to provide further clarification before the deadline.

Best regards, The Authors

Reviewer_J57SOpenReview ↗

Comment by Reviewer J57S

I want to thank the authors for the effort they put in for this rebuttal. They have addressed the majority of my points, and I'm leaning towards weak acceptance now. My only remaining comments are:

  1. The results are still showed on MCQ setups. It now looks to be more understandable for me, but I would also like to have the authors' opinion on this.
  2. The further analyses on the newer model families are post-hoc analyses, as they come as a result of this rebuttal. It would be fair to point that out in the paper, consistently with how the paper labels its other unplanned analyses
Reviewer_xVqSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies and studies unfaithful capitulation (UC): under conversational pressure, a reasoning model can produce a reasoning trace that still concludes with the correct option, yet deliver a different, wrong final answer. The authors separate the final commitments of the trace and the visible answer, classify turns into FC/UC/UP/FI states, and use gold-blind LLM judges plus a small blinded human spot check to label Qwen3 outputs across multi-turn MCQ runs. They report that 30% of completed first capitulations are UC events. They then hold saved traces fixed and edit only the surrounding conversation, showing that replacing the challenge with a neutral request or removing prior dialogue restores 93–99% correct answers on UC cases. A trace-omission control suggests the trace helps but neutralization also works without it. The paper concludes that trace correctness and answer correctness must be evaluated separately, and that agreement between trace and answer is the wrong repair target.

Critical strengths

The UC/UP framing is conceptually valuable and consequential. It directly challenges common assumptions in CoT monitoring, outcome-reward filtering, and sycophancy evaluation: a correct trace conclusion and a correct delivered answer are not the same measurement target. This is the strongest reason to accept the paper.

The fixed-trace counterfactual design is strong evidence for the core phenomenon. Holding the saved trace fixed while neutralizing the challenge or removing prior dialogue isolates the answer continuation after a correct conclusion. The result that answers recover without changing the trace is striking and supports the paper’s main evaluation claim.

Critical weaknesses

The quantitative UC prevalence is fragile under the labeling protocol. The 30% first-capitulation figure depends on gold-blind automated extraction. The paper reports that stricter thresholds remove UC events disproportionately, and at 0.99/0.95 no first UC event survives the uninterrupted-prefix requirement. Human validation is limited to a small single-author spot check, and two LLM judges may share errors. This does not invalidate the existence of UC, but it weakens the headline quantitative claim.

External validity and causal interpretation are limited. All main claims concern Qwen3 models on multiple-choice tasks. The continuation panels are selected rather than sampled, the 8B replication and trace-omission control were planned after earlier results, and history removal/neutralization changes content, length, and turn structure together. The paper is honest about these limitations, but they mean the results should not be read as a general claim about reasoning models or as identifying the internal mechanism.

Questions for authors

Could the authors provide a blinded, independently human-annotated sample of first-UC events—ideally all 69 events or a clearly stratified subset—with two annotators and reported precision/recall? This would materially strengthen the 30% claim.

Can the authors calibrate the extraction thresholds against human labels rather than only reporting offline abstention sensitivity? In particular, how should readers interpret the fact that no first UC event survives at 0.99/0.95?

Can the authors add at least one non-Qwen reasoning model, or otherwise clarify how strongly the conclusions should be generalized beyond Qwen3?

For the neutralization/history-removal experiments, can the authors add a length- or structure-matched control to separate semantic pressure from context-length and turn-structure effects?

3 comments

Thank you for the review and for four specific requests. Each is now in the paper.

Q1. Blinded human annotation of the first UC events. Two annotators who are not authors independently labeled both channels of all 69 events. They did not see the reference, the judge labels or which trace belonged to which answer. They agree with each other and with the judges on all 138 texts, so all 69 events are UC by human reading (App. M, p. 30). This establishes precision on the reported events. Recall is not measured, but the evidence is that the count errs low: a third, open-weights judge finds 31 correct conclusions among the 82 first-wrong traces that the first judge left unresolved (App. L, p. 30).

Q2. Calibration, and the 0.99/0.95 result. The annotation gives the calibration you ask for on the reported events. Of the 138 texts, 35 fall below 0.95/0.90 and 107 below 0.99/0.95, and the annotators confirm all of them. The stricter settings therefore remove labels that humans confirm. With no disagreement in 138 texts, the one-sided 95% upper bound on the judge's error rate is 2.1% overall and 8.2% for the 35 texts below 0.95/0.90 (App. M, p. 30).

That no first event survives at 0.99/0.95 follows from coverage. At that setting coverage falls to 11% of its default level. UC answers are almost never bare letters (0.9%, against 11.1% for turns where both channels are correct), so they always pass through the judge, and a first event also needs up to eight earlier answers to keep their labels (§3.2, p. 5; App. J, p. 28). The fixed-trace effects keep positive lower bounds even at the strictest setting (17.0 and 12.9 points; App. E.4, p. 20).

Q3. A non-Qwen reasoning model. We added seven (§3.3, p. 5; App. K, p. 29). Kimi-K2.5 produces 23 first UC events among 130 first wrong answers and DeepSeek-reasoner 9 among 120; a second judge agrees with all 64 channel labels. Four newer models capitulate less, and their traces capitulate with the answer. UC therefore occurs outside Qwen3 and is model-dependent. As you note, the paper's conclusions should not be read as a claim about all reasoning models, and the Limitations say that the other-family runs are observational (§6, p. 9).

Q4. A length- or structure-matched control. App. G (pp. 23–24, Tables 17–18) adds both, with contrasts fixed before generation.

  • Structure with a referent: keeping only the initial exchange before the same challenge costs about 30 points at both model sizes.
  • Length without a referent: a token-matched history of unrelated exchanges, with the same number of turns, costs nothing at 32B and an uncertain 11–15 points at 14B.

Context length and turn count alone are therefore an incomplete explanation. The larger factor is an earlier answer on the same question for the challenge to refer to (§4.2, p. 6).

On the points you list as limitations: the continuation panels are every first UC event of the selected runs but one, the 8B replication and the trace-omission control are marked as planned after earlier results, and the paper makes no claim about internal mechanism (§6, p. 9).

Thank you for requests that made the measurement more solid.

AuthorsOpenReview ↗

Full Results Tables for Reference

We have gathered full results here for reference, with the location of each table in the revised paper.

Q1–Q2. Human annotation and thresholds (App. M, p. 30). All 138 texts are the trace and answer of the 69 first UC events.

TextsConfirmed by both annotatorsOne-sided 95% upper bound on judge error
All accepted texts1381382.1%
Below 0.95/0.9035358.2%
Below 0.99/0.95107107

The annotators agree with each other on every text (Cohen's κ = 1.0).

Q1. Open-weights third judge (Gemma-4-31B-it against the first judge; Table 26, p. 30):

TextsnFirst judgeThird judge: correct / wrong / unresolved
Qwen3 first UC, trace69correct66 / 0 / 3
Qwen3 first UC, answer69wrong0 / 63 / 6
FI first wrong, trace82wrong0 / 79 / 3
Unresolved first wrong, trace82unresolved31 / 33 / 18

Q3. Other families (Table 24, p. 28). First UC events among first wrong answers with a resolved trace: Qwen3 pooled 69/151, Kimi-K2.5 23/84, DeepSeek-reasoner 9/85; the four newer models (DeepSeek-V4.1-Flash, Kimi-K2.6, Qwen3.8-27B, Gemma-4-31B) 1 in total; gpt-oss-120b 0/12.

Q4. Referent and length controls (registered contrasts on UC cases, in points; all-case bounds and 95% case-cluster envelope; Table 18, p. 24). HP: original history and challenge; HN: neutral request; QP: no history; R0P: only the initial exchange before the challenge; FillP: unrelated exchanges matched in tokens and turns; HnP: real history with earlier challenges neutralized.

Contrast14B bounds14B 95%32B bounds32B 95%
HN − HP (replication)[+41, +47][+23, +63][+19, +23][+8, +34]
R0P − QP (referent, short)[−34, −25][−49, −13][−30, −28][−43, −15]
FillP − QP (length, no referent)[−15, −11][−33, +4][+0, +0][+0, +0]
HnP − FillP (referent, long)[−37, −27][−58, −2][−32, −28][−45, −15]
HP − HnP (earlier challenges)[−5, +4][−25, +23][+3, +9][−10, +23]

Dear Reviewer, thank you once more for your careful review. As the discussion phase ends on November 18, we just wanted to follow up on our response. We would greatly appreciate any further feedback, and we remain happy to address any remaining points.

Best regards, The Authors

Reviewer_UMYuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies unfaithful capitulation (UC): after a user challenges an answer, a Qwen3 model’s reasoning trace ends with the correct trajectory, but its final answer is wrong. Across seven multiple-choice training-free runs with Qwen3 models (from 1.7B to 30B), the authors identify 69 UC cases among 233 first completed wrong answers that meet their labeling and prior-answer rules, about 30 percent. They then keep the saved trace unchanged and edit the conversation before generating a new final answer. Replacing the challenge with a neutral request raises accuracy on the selected UC cases by 52–59 percentage points for 14B and 29.8–31.5 points for 32B (Sections 2–4, pp. 3–6). They conclude that

Critical strengths

The paper conducted training-free experiments on Qwen3 models (from 1.7B to 30B) and drawn a conclusion that the trace and final answer of LLM should be measured separately.

Critical weaknesses
  1. The presentation of this paper could be greatly improved with restructuring. It is unclear what this paper did and what its main conclusions are after walking through the abstract and introduction sections.
  2. Qwen3 was already outdated three months before the ICLR 2027 deadline. The authors should consider whether recent models also exhibit the same property. Also, do other model families have the same situation? What about models with larger numbers of parameters?
  3. This work mainly focuses on training-free methods. However, the model's capacity is fixed, so it is hard for me to accept conclusions as strong as the authors state. Have you tried any training-based methods to make the reasoning trace and answer consistent, and how did it go?
Questions for authors
  1. How were the labeling thresholds chosen? Is there independent human checking of both accepted and uncertain texts, including the earlier answers used to decide whether an error is the first one?
  2. What exactly does “not mainly” mean in the conclusion? How does the trace-removal test rule out the model using earlier correct answers as a substitute for the trace?
3 comments

Thank you for the review. We answer the three weaknesses and the two questions in order.

W1. What the paper does and concludes. The paper makes one observation and draws one consequence. The observation: after a user pushes back, a reasoning model sometimes writes a trace that ends at the correct option and then delivers a different answer. We call this unfaithful capitulation (UC). The consequence: the trace and the answer have to be scored separately, and a repair has to be judged by whether the delivered answer improves. The introduction now opens with one such case (Fig. 1, p. 2) and lists the four findings with their section numbers (§1, p. 2).

W2. Newer models, other families, larger models. We ran seven further models on the same protocol (§3.3, p. 5; App. K, p. 29).

  • Other families show UC: Kimi-K2.5 has 23 first UC events among 130 first wrong answers, and DeepSeek-reasoner has 9 among 120.
  • Newer models capitulate less, and their traces capitulate with the answer: DeepSeek-V4.1-Flash, Kimi-K2.6, Qwen3.8-27B and Gemma-4-31B produce one first UC event between them. The largest open model we ran, gpt-oss-120b, has none among 12 first wrong answers.

So UC is not confined to Qwen3, and it varies by model. A count of flipped answers scores the old and new models alike. Scoring trace and answer separately is what shows that the failure changed in kind, which is the paper's point.

W3. Training-free scope and the strength of the conclusions. The conclusions concern measurement and where the failure sits, and they do not rest on any claim about training. We do not claim that training cannot remove UC, and the Limitations state that we test no training-based repair (§6, p. 9). What the paper shows applies to any repair, trained or not:

  • Making trace and answer consistent is not the right objective. Consistency can be reached by fixing the answer or by corrupting the trace, and only the first helps the user (§5.1, p. 8, Fig. 5a).
  • An inference-time policy that enforces consistency harmed 271 correct answers and corrected 65 (§5.2, p. 8, Fig. 5b).

A training method aimed at consistency would need the same check, and the paper supplies it: score answer correctness and the transitions between the four states.

Q1. Thresholds and human checking. The thresholds (probability at least 0.90, confidence at least 0.80) were fixed when the protocol was frozen, before the archive was labeled, and never lowered (§2.3, pp. 3–4).

  • Accepted texts: two blinded annotators who are not authors labeled both channels of all 69 events and agree with each other and the judges on all 138 texts (App. M, p. 30).
  • Earlier answers that decide whether an error is the first one: a second judge from another model family re-extracted them together with the event texts, and the two judges agree on 519 of the 520 texts that both resolve (§2.3; App. C.5, p. 17).
  • Uncertain texts: a third, open-weights judge read the traces that the first judge left unresolved. It resolves 64 of 82 and finds 31 with a correct conclusion, so the reported count is conservative (App. L, p. 30).

Q2. "Not mainly", and earlier correct answers as a substitute for the trace. The revised text no longer uses "not mainly". It states the result directly: with the trace deleted, neutralization still adds 48–53 points, against 52–59 with the trace (§4.3, p. 7, Fig. 4).

Your second question identifies what happens, and we tested it. With the trace deleted, masking the model's earlier answers in the history lowers correctness by about 30 points at both model sizes. With the trace present, masking changes nothing (§4.3; App. G, p. 24, Table 17). The saved trace and the earlier correct answers can therefore each support a correct continuation, and a challenge in context can override both.

We hope this makes clear what the paper shows and how far it extends. If it addresses your concerns, we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Full Results Tables for Reference

We hope it helps to have the evidence for W2 and Q2 in one place. The tables below reproduce the relevant results from the revised PDF, so there is no need to look them up.

W2. Other families and newer models (Table 24, p. 28; App. K, p. 29). First UC events among first wrong answers whose trace conclusion is resolved:

ModelFirst UC / first wrong (resolved trace)
Qwen3, seven runs pooled (1.7B–32B)69/151 (45.7%)
Kimi-K2.523/84 (27.4%)
DeepSeek-reasoner9/85 (10.6%)
gpt-oss-120b0/12
DeepSeek-V4.1-Flash0/23
Qwen3.8-27B0/47
Gemma-4-31B1/8
Kimi-K2.6 (26 questions chosen for UC; not a prevalence sample)0/2

Counting all completed first wrong answers, Kimi-K2.5 has 23 of 130 and DeepSeek-reasoner 9 of 120 (§3.3, p. 5).

Q2. Earlier answers as a substitute for the trace (registered contrasts on UC cases, in points; all-case bounds and the 95% envelope of a case-cluster bootstrap; Table 18, p. 24). HN: neutral request with the history; subscript o: trace deleted; HNmask: earlier assistant answers replaced by "[answer removed]".

Contrast14B bounds14B 95%32B bounds32B 95%
Masking earlier answers, trace deleted (HNmask_o − HN_o)[−33, −31][−53, −11][−30, −29][−46, −15]
Masking earlier answers, trace present (HNmask − HN)[−1, +2][−9, +13][+3, +3][+0, +8]
Deleting the trace, neutral request (HN − HN_o)[+14, +17][+2, +33][+0, +1][−8, +8]

Dear Reviewer, we truly appreciate the time and effort you have devoted to reviewing our paper. With the discussion period closing on November 18, we would like to gently check in on whether our rebuttal has sufficiently addressed your concerns. Should any questions remain, we would be happy to clarify further before the deadline.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers. The reviews agree on what the paper is about: under conversational pushback, a reasoning model's trace can end at the correct option while its delivered answer states a different one, so trace correctness and answer correctness have to be scored separately. Reviewers xVqS and J57S call this framing valuable for CoT monitoring, outcome rewards and sycophancy evaluation, and both single out the fixed-trace design and the point that agreement is the wrong repair target.

The main results are as submitted. Of 233 completed first wrong answers in seven Qwen3 runs, 69 (30%) follow a trace that still concludes correctly (§3.1, p. 4, Fig. 2). With each saved trace held fixed, replacing the challenge with a neutral request or removing the prior dialogue restores 93–99% correct answers on these cases (§4.2, p. 6, Fig. 3). A policy that enforces trace–answer agreement harms more answers than it corrects (§5.2, p. 8).

The reviews share three questions. Short answers follow, and the individual replies give details. The updated supplementary material has the data and code for the added runs.

1. Are the reported events real, and how should the 30% be read? Two blinded annotators who are not authors labeled both channels of all 69 events. They agree with each other and with the judges on all 138 texts (App. M, p. 30). Of these texts, 107 would be rejected at the strictest thresholds, and the annotators confirm every one, so stricter thresholds remove correct labels. A third judge from another model family finds 31 further correct trace conclusions among traces the first judge left unresolved, which makes the reported count conservative (App. L, p. 30). The abstract (p. 1) now gives the per-turn rate (0.3%) next to the 30%.

2. Is this specific to Qwen3 and its template? No. Under the same protocol, Kimi-K2.5 produces 23 first UC events and DeepSeek-reasoner 9 (§3.3, p. 5; App. K, p. 29). Without the answer-format instruction, Qwen3 produces more first UC events, not fewer (App. G, p. 25, Table 19). Four newer models capitulate less often, and when they do, the trace capitulates too. Only separate labels for trace and answer reveal that difference.

3. What do the fixed-trace edits show, given that removing history changes several things at once? New controls separate them (App. G, pp. 23–24). One earlier exchange on the same question is enough for the challenge to cost about 30 points. A history of unrelated exchanges with the same length and turn count costs nothing at 32B. When the trace is deleted, the model's earlier correct answers can stand in for it, and masking those answers removes the recovery (§4.3, p. 7).

An unpressured baseline is also reported: among wrong answers with a resolved trace, 1.7% follow a correct trace conclusion at the initial turn, against 46% at the first capitulation (§3.1, p. 4; App. J, p. 28).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.