acceptodds
Under review as a conference paper at ICLR 2027

Benchmarks Don't Talk Back: Learning Through Self-Disagreement

Abstract

A benchmark usually measures a language model alone with a question, and deployment rarely looks like that: answers get contested, and models give up correct ones under counter-reasoning that is itself wrong. We show that what fails there is not a shortfall of accuracy but a capability in its own right — knowing when to hold a position and when to yield to a better one — which we call dialectical competence. It stands on first-answer accuracy without following from it: a model can be made materially better at answering and no better at this, and training for accuracy alone can drift against it. Nor can it be bought at inference, by self-reflection or by sampling more. Taking a cue from the dialectical tradition in philosophy, in which productive opposition arises within a position rather than being imposed on it from outside, we hypothesize that this capability can be trained from opposition the model supplies itself. It states a position, generates the strongest case against that position — its antithesis — rules on whether the case stands, and is trained on that ruling, with no second model at training or at deployment. We call this learning through Self-Disagreement: reinforcement learning from verifiable rewards throughout, where the same check that scores an answer against its reference scores the ruling too, so no reward model, no preference data and no human annotation enter anywhere. The capability is trained rather than elicited: weak in the base model, it arrives with the reward, is not paid for out of first-answer accuracy, and holds against challengers the model never trained against, including ones stronger than itself. The opposition a model needs in order to improve, it turns out, need not come from outside it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning from verifiable rewards, large language models, RLVR, reasoning, dialectical competence, self-disagreement, robustness under disagreement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EJsbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how models respond when their answers are challenged. It proposes Self-Disagreement, where the same model generates an answer, critiques it in a separate context, and then decides whether to revise it. Training rewards correct answers and correct assessments of the critique, while penalizing giving up a correct answer or refusing a valid correction.

The main experiments use Qwen3-8B and Gemma-2 9B, with a 327-question disagreement probe and nine public benchmarks. The authors also evaluate against external challengers. They report better accuracy after challenge and better retention of correct answers, without hurting initial-answer accuracy.

Critical strengths
  1. The paper looks at a problem that matters in actual use: getting an answer right initially is not enough if the model gives it up when challenged. Average accuracy after challenge improves from 50.0% to 56.3% on Qwen and from 19.9% to 37.8% on Gemma. I found the results against external challengers especially useful, since they show that the gains extend beyond responding to the model’s own critiques.
  2. The method uses one policy and existing answer verifiers, which makes the setup practical. I also appreciated the accuracy, revision, and verification controls, along with the checks that initial-answer accuracy does not drop. These make the overall result more convincing.
Critical weaknesses
  1. I am more convinced that the model gets better at holding its answer than at deciding when to change it. Correct yielding drops from 84.1% to 50.8% on Qwen and from 93.4% to 65.5% on Gemma. The paper acknowledges this and accounts for it in the net improvement. But the “skill” decomposition does not settle the issue, since the groups depend on each model’s initial answers and the challenges they produce. The examples and arguments being compared can therefore differ across models. The odds-ratio analysis also does not compare the base and trained models. I would want to see both models evaluated on identical initial answers and critiques before concluding that their ability to judge arguments has improved.
  2. I could not work out how the CAD-RL reward is defined. Its challenge contains generic doubt but no proposed answer, while several reward terms depend on whether the challenger’s answer is correct. What happens to those terms when there is no answer? If the supervision changes too, the comparison does not isolate the effect of substantive criticism. This leaves me unconvinced by the claim that roughly two thirds of the gain comes from the objection’s content. The verdict ablations also need clearer definitions of the rewards and output requirements.
Questions for authors
  1. When you say the initial answers are frozen and replayed, are the base and trained models given exactly the same initial answer and critique, or are these fixed only across challengers within each model? If you have an evaluation that holds the full input fixed across checkpoints, please report both correct holding and correct yielding.
  2. For CAD-RL, how do you define challenger correctness when the critique contains no answer? Is the verdict target always “incorrect”? How are the correction multiplier and stubbornness penalty handled?
  3. In Table 7, what changes between “Shaping only, no judge,” “Outcome only,” and “Judge on, ungated”? Please specify the reward terms, required outputs, and format gates so it is clear which comparisons isolate the verdict requirement.
2 comments

Thank you for a careful and encouraging review.

W1 and Q1. Holding against deciding, on identical first answers and critiques. To address these, we added a new evaluation that holds the full input fixed across models, as you asked. Table 8 fixes the input only within each model, as its caption now states. In the new evaluation, every arm receives the base model's first answer and the same antithesis from Gemini-3.8-flash or Claude-Haiku-4.5, on the nine benchmarks, with each trained arm at its three training seeds, each scored on eight sampled answers per question (§4.1, Table 1's lower block; App. Q). Table 13 reports, for every arm, correct holding (keeping a right first answer against a wrong antithesis), correct yielding (taking a right antithesis over a wrong first answer), their mean and the AUC of switching. In all four model–challenger pairs, Self-Disagreement holds more than twice as often as any other arm, SCoRe included, while still taking most of the right antitheses; its mean is 8 to 19 points above every other arm, and its AUC is the highest. On Qwen3-8B against Gemini-3.8-flash, for example, holding rises from 30.6% to 78.0% and yielding falls from 97.0% to 73.4%, so the mean rises from 63.8 to 75.7.

  • The odds ratio across models: we added the same design on the disagreement probe (Table 14, App. Q). The mean of holding and yielding rises by 5.1 points [2.4, 7.7] on Qwen3-8B and 17.2 [14.8, 19.6] on Gemma-2 9B, and the odds ratio can now be compared across the base and trained models: Self-Disagreement's log odds ratio is above the base model's by 0.68 [0.40, 0.98] and 1.23 [0.89, 1.56] (Qwen3-8B, Gemma-2 9B).
  • Holding against deciding: we added a test that pushes the base model's final answer toward its first answer by amounts fixed in advance, which on Gemma-2 9B brings it to Self-Disagreement's switch rate (App. Q). It still decides worse: Self-Disagreement's mean of holding and yielding is higher by 10.4 [9.1, 11.9] and 7.0 [4.7, 9.5] points (against Gemini-3.8-flash and Claude-Haiku-4.5).

So on identical inputs, the trained model is better at deciding when to change its answer, not only at keeping it.

W2 and Q2. CAD-RL's reward when the critique has no answer. To address these, we now state in App. C how each reward term is computed for CAD-RL, and we scoped the claim. The fixed passage proposes no answer, so it is graded as not correct: the verdict target is always "incorrect", the correction multiplier applies to every wrong first answer, and the stubbornness penalty never fires, since it needs a correct challenge. As you point out, the supervision then changes with the content, so §3 and App. F now say that removing the content of the disagreement, together with the supervision that depends on it, costs roughly half to two thirds of the gain, and §4.2 that the content is also what makes the verdict checkable. So these sections no longer credit that share to the content alone.

Q3. Table 7's rows, and which comparisons isolate the verdict requirement. To address this, we added a panel to Table 7 (App. E) that lists every row's reward terms, required outputs and format gate; we relabelled the rows and named the pairs that differ in one setting. All seven runs share every generation setting, the token limit included; only the synthesis template differs, with or without the verdict request.

  • Shaping only, no judge (now Full return, no verdict): the final and first answers, both penalties and the format gate; no verdict.
  • Outcome only: the final and first answers alone, no format gate.
  • Judge on, ungated (now Full return + coherence credit): the first row plus the verdict, required and paid, and 0.2 more when the final answer is right and follows it; ungated named a different gate, off here.

So switching the judge on turns on three things at once: the requirement, its payment and that credit. The single-setting pairs separate them: the requirement alone is Full return, no verdict →\rightarrow Judge (the verdict required, not paid; App. E), and the payment alone is Judge →\rightarrow Self-Disagreement, three training seeds each, which lowers capitulation from 2.9% to 1.0% (Table 3) and, on the probe, raises correct holding by 6.2 points and lowers correct yielding by 7.6, their mean unchanged (App. Q). So each comparison now isolates one setting, and §4.2 now states what Table 3 shows.

How your review improved the paper: a new evaluation with identical inputs for every arm, with holding, yielding and the odds ratio across models (App. Q); Table 8's caption; CAD-RL's reward and the scoped claim (App. C; §3, §4.2, App. F); every Table 7 row defined (App. E); §4.2 restated by what Table 3 shows.

With all of your concerns and questions now addressed in the revision, we kindly ask you to raise your rating to 4, and your presentation, informativeness and soundness scores. Thank you.

Dear Reviewer EJsb,

We wanted to follow up, as the discussion closes on November 18. With all your concerns and questions addressed in the revision, we kindly ask you to raise your rating. If any question remains, we would be happy to answer it. Thank you for your time.

Reviewer_NVjMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models can retain correct answers under incorrect counterarguments while accepting valid corrections. The authors call this behavior dialectical competence and propose Self-Disagreement (SelfD-RL). A single policy generates an initial answer, produces an opposing argument in a separate context, and assesses the opposing answer before generating a final response. The opposing turn is sampled without direct optimization. The answering turns are trained with GRPO using reference-answer verification, an explicit verdict reward, and penalties for capitulation and stubbornness. Evaluation includes a 327-question disagreement probe, nine public benchmarks, fixed external challengers, and multiple model families. The central claim is that this training improves judgment under disagreement beyond improvements in first-answer accuracy or additional inference computation.

Critical strengths
  1. The paper identifies a useful distinction between answering accurately and responding appropriately to disagreement. Table 2 separates changes in the distribution of initial-answer/challenger outcomes from changes in conditional final-answer performance. The standard training controls improve the former without a positive estimated contribution from the latter, whereas SelfD-RL improves both models' final-answer performance. Although this decomposition has limitations, it provides a useful empirical reason to evaluate behavior under challenge separately from first-answer accuracy.
  2. The controls support a real improvement under the tested interaction metric. Evaluation relative to fixed external challengers in Table 8 helps rule out gains caused solely by a weakening self-generated opponent. The content-agnostic and external-opposition controls further examine what the training requires. Improvements across Qwen3-8B and Gemma-2 9B provide evidence that the effect extends beyond one model family. These results make the proposed training recipe worth investigating.
Critical weaknesses
  1. The incremental contribution over existing self-correction training is not sufficiently established. SCoRe (Kumar et al., ICLR 2025) already trains a single model on its own multi-turn outputs and uses progress rewards to encourage corrections while penalizing correct-to-incorrect transitions. S²R (Ma et al., ACL 2025) also trains explicit verification and correction. SelfD-RL's unconditional opposing role is a meaningful distinction, but the experiments do not establish how much it adds over strong versions of these approaches. Appendix A discusses SCoRe, yet Revise-RL is not documented as retaining its two-stage training and progress-reward design. The comparison therefore establishes an advantage over the paper's revision control more clearly than it establishes an advance over established self-correction training.
  2. Prompt compatibility further limits this interpretation. Table 5 states that all arms are evaluated with the verdict template, although several controls train with different revision contexts. Verify-RL already includes an explicit assessment, so prompt mismatch alone is not an explanation for the entire result. Nevertheless, evaluation under both the common protocol and the controls' native revision prompts would help separate the learning mechanism from familiarity with the evaluated interaction. This matters because the claimed contribution depends on what the opposing role and verdict objective add to existing training.
  3. The results leave an important alternative explanation for the claimed improvement in discrimination. In Table 9, Qwen3-8B's correct holding rate rises from 53.8% to 90.4%, while its correct yielding rate falls from 84.1% to 50.8%. Gemma-2 9B shows the same trade-off: 24.7% to 83.5% holding and 93.4% to 65.5% yielding. These results are consistent with a substantial shift toward defending the initial answer. The paper acknowledges the yielding loss and includes it in the aggregate skill term, but an aggregate gain does not establish an improved holding-versus-yielding frontier. Moreover, Table 9 defines situations using each arm's own generations, so the decomposition does not compare defenders on identical argumentative inputs.
  4. The odds-ratio analysis and fixed base-correct evaluation address some simpler explanations, but do not establish an advantage over a baseline whose propensity to revise is adjusted. The distinction between answer correctness and argument validity also remains relevant: the verdict reward uses the opposing final answer's correctness, while Appendix G's jury assesses visible reasoning behavior and explicitly does not certify logical soundness. The evidence supports improved answer selection under the tested mixture more strongly than the broader claim of improved argument-sensitive judgment.
Questions for authors
  1. How does Revise-RL differ from SCoRe's two-stage training and progress reward? What evidence establishes the additional benefit of the opposing role over this stronger correction recipe, with comparable generation budgets and appropriate revision prompts?
  2. On a fixed panel of identical initial-answer/challenge pairs, does SelfD-RL improve holding at matched yielding, or yielding at matched holding, relative to a prompting baseline that varies the propensity to revise? This comparison would materially affect my assessment of the claimed discrimination gain.
  3. What are the complete reward coefficients and output requirements for the Table 7 ablations, particularly “Shaping only, no judge” and “Outcome only”? With final-answer reward, format constraints, token budget, and opposing content fixed, what does explicit verdict supervision contribute to both holding and yielding?
  4. How many independent training runs support the main comparisons? Appendix L describes repeated decodes of fixed checkpoints, while some captions refer to seeds. Please distinguish training variability from challenger/decoding variability and clarify the evidence that the main gains reproduce across training runs.
3 comments

Thank you for a precise review.

W1 and Q1. The contribution over SCoRe and S$^2$R. To address these, we added SCoRe as an arm and defined Revise-RL against it.

  • SCoRe: trained with its two stages and progress bonus on our data, three seeds per base model, it is now an arm throughout (App. N). On identical inputs it reaches 51.6% synthesis accuracy, against 66.4% for ours (Qwen3-8B, Gemini-3.8-flash).
  • Revise-RL is a one-stage revision control, without SCoRe's stages and bonus (App. B).
  • Budgets: all arms share sampling and token limits, and Table 3's Unrew. runs the same exchange for the same steps, paid only for its first answer: it capitulates on 14.0%, against 1.0% for ours.
  • S$^2$R: its verdict is on the model's own solution, with no case put against it. Our nearest arm, Verify-RL, judges its own answer first and recovers at most about half of our gain (§4.1); App. A details both works.

So the opposing role adds a case to decide on, not more generation.

W2. Prompt compatibility. By design, the compared arms train for correctness, which does not supply dialectical competence (Table 2, as your first strength notes); their training prompts contain no antithesis (Table 5), so every arm meets it under one instruction asking for a verdict. To address this, we added a test of each without the verdict request (App. C): each does worse on Qwen3-8B (capitulation 6.6 to 7.3 points higher), and none gains beyond its seed-to-seed spread on Gemma-2 9B. So the verdict request does not handicap them; on Qwen3-8B it helps them.

W3, Q2 and W4's first part. Holding at matched yielding, against a prompting baseline. To address these, we added three new tests on identical pairs (§4.1, Table 1's lower block; App. Q):

  • Identical pairs: on the nine benchmarks, every arm gets the base model's first answer and the same antithesis from Gemini-3.8-flash or Claude-Haiku-4.5. Self-Disagreement holds a right first answer more than twice as often as any other arm while still taking most right antitheses, with the highest AUC of switching in every pair (Table 13).
  • The prompting baseline: as you suggested, on the probe we told the base model to keep its first answer, at three strengths fixed in advance. It still yields on 81–84% of the items where yielding is right on Qwen3-8B (Self-Disagreement 58%), and our mean of holding and yielding is higher at every strength, by 3.5 to 5.2 and 14.4 to 19.0 points (Qwen3-8B, Gemma-2 9B).
  • Matched switching: pushing the base model's final answer toward its first, with sizes fixed in advance, brings Gemma-2 9B to our switch rate; our mean is still higher, by 10.4 and 7.0 points (one per challenger), and on Qwen3-8B, with pushes added afterwards, by 17.7 and 9.9.

So the advantage lies in which items it holds on, not in how many.

Q3. Table 7's settings and what the verdict contributes. To address this, we added a panel to Table 7 (App. E) with every row's reward coefficients, required outputs and format gate; all runs share every generation setting, the 4,096-token budget included, except the synthesis template. Shaping only, no judge (now Full return, no verdict) pays both answers, with both penalties and the format gate; Outcome only pays both answers alone. With the final-answer reward, format, budget and opposing content fixed, only the verdict's reward varies, so we added Judge (verdict required, not paid) against Self-Disagreement, three seeds each (App. Q): paying for it raises correct holding by 6.2 points and lowers correct yielding by 7.6, their mean unchanged, and lowers capitulation from 2.9% to 1.0% (Table 3). So the verdict reward cuts capitulation by holding more; without it, Judge already decides better than the base model (mean 74.6% against 68.7%, Table 14).

Q4. Independent training runs. Three per arm in every main comparison (Table 7 is a single-run ablation). To address this, the ± is now the spread across three training seeds, each scored on its own sampled answers (eight in Table 1). So the gains reproduce across seeds: on Qwen3-8B, +5.90 points over the base model, [+5.00, +6.80] over the seeds (§4.1).

W4. Answer correctness and argument validity. To address this, §2 now states the claim as the evidence supports it, telling a correct challenge from an incorrect one, and §5 bounds it to checkable answers, as any method trained on verifiable rewards checks the final answer, not the reasoning; App. G's jury checks that what counts as capitulation is a real failure.

How your review improved the paper: SCoRe throughout and Revise-RL defined (App. N, B); the verdict-request test (App. C); the identical-pair tests (App. Q); Table 7's settings (App. E); seeds and draws distinguished.

With all of your concerns and questions now addressed in the revision, we kindly ask you to raise your rating to 4, and your presentation, informativeness and soundness scores. Thank you.

Dear Reviewer NVjM,

We wanted to follow up, as the discussion closes on November 18. With all your concerns and questions addressed in the revision, we kindly ask you to raise your rating. If any question remains, we would be happy to answer it. Thank you for your time.

I appreciate the authors' comprehensive and thoughtful responses, which have effectively addressed most of my concerns. Consequently, I have increased my rating.

Reviewer_FjekOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is talking about a scene where model is under challenge by used and after turns of interactions, the model will reconsider its answer and give the wrong answer(sycophancy), while the authors think that model should insist correct answer and reconsider wrong answer when being challenged(dialectical competence). The authod proposed a D-probe dataset and evaluate Qwen3-8b and Gemma-2 9B models, gap of accuracy between "should insist" and "should give up" is significant. This paper proposed self-disagreement method(selfD-RL) to solve this issue. The results show that models' accuracy under challenge improves and first-answer accuracy stays the same level.

Critical strengths
  1. Two finding:

    • During Accuracy-RL, thesis accuracy goes up while "give away" goes up as well.
    • Gemma-2 9B is really trained to have the discrimination for insisting and give away: ratio for hold +58.8%, ratio for yield -27.9, balanced accuracy 59.1->74.5.
  2. Diversity experiments results with 17 challenger, 9 benmarks(including math and code), 4 extra models.

  3. Ablations and controls rule out alternative explanations:

    • Contentless pressue(CAD-RL)
    • a second pass with nothing to rule on (Revise-RL, Verify-RL)
    • A decaying antithesis (fixed-weight challengers)
    • a relabelled answer (blinded jury) -- reward hacking (Appendix M)
Critical weaknesses
  1. Qwen3-8B's results doesn't support that statement "sialecttical competece is trained": Hold ratio +36.6 and Yield ratio -33.3, balanced accuracy is just from 69.0 to 70.6, and discrimination doesn't improve. In addition, the headline gain on Qwen3-8B is largely shift exploiting the probe's base rate, not a improvement in dialectical competence.

  2. Related works are missing, overstated novelty, no external baseline. Balancing resistance and acceptance of persuation is already the subject of PBT(Stengel-Eskin et al., NAACL 2025, https://arxiv.org/abs/2410.14596), the hold-if-right/revise-if-wrong target was already used by Laban et al.(2023, https://arxiv.org/abs/2311.08596), and Wang et al. (EMNLP 2023 Findings, https://arxiv.org/abs/2305.13160) already showed failure under challenge is independent of confidence; None is cited in paper. All seven arms are the authors' own design. Hence "neither accuracy-centered train not inference-time remedies supply it"is untested against published methods. The authors' own ExtD-RL matches SelfD-RL(+5.15 vs 6.30), so the self-supplied opposition saves a second model rather than improving results.

  3. Evaluation mirrors training, with no decomposition on benchmarks. All challenges use the same challenging-turn instruction, but selfD-RL is evaluated with its own training instruction while the baselines are not use the same one (table 5). Table 1 reports only synthesis accuracy, and no thesis accuracy, challenger accuracy, hold/yield split. Hence, it shows higher post-challenge accuracy, not transfer of the capability. Only single-turn antitheses are tested, neither multi-turn pushback nor content-agnostic challenge is tested.

Questions for authors
  1. Can you report a base-rate-invariant metric (like balanced accuracy, d' or AUC from the eight draws) for every arm, and compare the odds ratio across arms rather than disclaiming it?

  2. Can you position self-disagreement against PBT (Stengel-Eskin et., 2025) and Laban et al.(2023)? What's difference between them, And run comparison experiment if time allows to check if SelfD-RL still outperforms?

  3. Does the gain still hold on a 1:1 or yield-heavy hold/yield mix setting, and does varying λcap\lambda_{cap} / λstub\lambda_{stub} change the operating point? And also why is SelfD-RL evaluated with its own training prompt while the baselines are not (table 5)?

2 comments

Thank you for a thorough review.

W1 and Q1. The base rate and the odds ratio. To address these, we added a new evaluation in which every arm gets the base model's first answer on the nine benchmarks and the same antithesis from Gemini-3.8-flash or Claude-Haiku-4.5, at three training seeds per arm, each with eight sampled answers (§4.1, Table 1's lower block; App. Q). Self-Disagreement has the highest balanced accuracy (the mean of holding and yielding) and AUC of switching in every model–challenger pair (Table 13); on Qwen3-8B, from 63.8 to 75.7 against Gemini-3.8-flash and from 57.2 to 66.5 against Claude-Haiku-4.5. On the probe, with the base model's exchange, it rises from 68.7 to 73.8 on Qwen3-8B, +5.1 [2.4, 7.7] (Table 14). As you suggested, we also compare the odds ratio across arms there: Self-Disagreement's log odds ratio is above the base model's by 0.68 [0.40, 0.98] on Qwen3-8B and 1.23 [0.89, 1.56] on Gemma-2 9B, and above every other trained arm. So with the inputs fixed, the gain is a better decision, not the base rate.

W2 and Q2. Related work and external baselines. To address these, we extended the related work and added two external baselines.

  • SCoRe, the comparison Reviewer NVjM also asked for, trained with its two stages and progress bonus in our setting (three seeds), is now an arm throughout the paper: Figure 1 and Tables 1–3 and 8–14 (App. N). On identical inputs it reaches 51.6% synthesis accuracy, against 66.4% for ours (Qwen3-8B, Gemini-3.8-flash).
  • PBT: as you suggested, we ran its released model in our setting (App. R). Ours, trained from the same Llama-3.1-8B (one training run), leads it by 5.7 points [2.7, 8.6] in balanced accuracy and 0.14 in AUC; as the training data differ, this compares released models.
  • Related work (App. A compares each in full, Wang et al. included): the works you named share part of our target. FlipFlop (Laban et al.) uses challenges such as "Are you sure?", which carry no content, and fine-tunes on gold-labelled conversations. PBT trains a model to defend against persuasion about short factual answers, while accepting it when right, offline, from dialogues read by other models; by its own analysis, it learns to adopt the answer it finds more likely. Ours differs: the model must solve a multi-step problem and then weigh a worked case against its answer, which may be right or wrong, and is paid for ruling correctly. This can be learned wherever answers can be checked: the policy writes its own opposition and the verifier gives every label, so no second model or labelled dialogues are needed.
  • ExtD-RL, our own design with the larger Qwen3-14B as challenger, gains +5.09 against our +5.90 (§4.2): matching it with no second model is the point.

So ours leads both external baselines.

W3 and Q3. The evaluation design, the 1:1 mix and λ\lambda. To address these, we added new evaluations:

  • Table 5: the compared arms are trained for correctness by design, since our claim is that this does not supply dialectical competence (§1, Figure 1); their own training prompts contain no challenge, so all are tested under one instruction asking for a verdict (App. C). We added a test of each, and of the base model, without the verdict request: each does worse on Qwen3-8B (capitulation 6.6 to 7.3 points higher), and none gains beyond its seed-to-seed spread on Gemma-2 9B. So the shared instruction does not handicap them, and under it none gains more than 1.4 points over the base model (Table 1's lower block).
  • Decomposition: first-answer accuracy per benchmark (App. O), how often the outside antithesis is right (App. Q), and holding and yielding on identical inputs (Table 13).
  • Multi-turn: three rounds of challenge (App. P): after round 3 Self-Disagreement still leads in synthesis accuracy, 72.0% against at most 64.8% on Qwen3-8B and 52.4% against 41.7% on Gemma-2 9B.
  • Content-agnostic: we test it as a training signal: CAD-RL, our nearest arm to FlipFlop, trains against a fixed passage objecting without naming a defect and recovers roughly a third to a half of the gain (§4.2).
  • The 1:1 mix: the mean of holding and yielding weights the two kinds of item equally, so it does not depend on the mix; Self-Disagreement leads every arm on it, by 8 to 19 points (Table 13).
  • λ\lambda: two training seeds with λstub\lambda_{\mathrm{stub}} raised from 0.5 to 2.0 (App. E): holding falls by 5.3 points and yielding rises by 6.5, their mean unchanged. So the weight moves the operating point.

How your review improved the paper: SCoRe throughout and PBT (App. N, R); identical inputs for every arm (App. Q); the verdict-request test (App. C); three rounds of challenge (App. P); λstub\lambda_{\mathrm{stub}}'s effect (App. E).

With all of your questions and concerns now addressed in the revision, we kindly ask you to raise your rating to 4, and your presentation, informativeness and soundness scores. Thank you.

Dear Reviewer Fjek,

We wanted to follow up, as the discussion closes on November 18. With all your questions and concerns addressed in the revision, we kindly ask you to raise your rating. If any question remains, we would be happy to answer it. Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for careful and constructive reviews. They found that the paper targets a problem that matters in use, a right answer given up under challenge, with a practical setup of one policy and existing verifiers (EJsb); that it separates answering accurately from responding to disagreement (NVjM, Table 2); that the gains hold against fixed outside challengers (EJsb, NVjM) and on two model families (NVjM), in experiments spanning 17 challengers and nine benchmarks (Fjek); and that the controls rule out content-agnostic disagreement, a second pass with nothing to rule on, a weakening antithesis, a relabelled answer and reward hacking (Fjek; NVjM for the weakening antithesis). Each reply is self-contained; the revised PDF marks the changes to the text since the submission in blue.

How we addressed the concerns

  1. Does the model decide better, or only hold more? (EJsb W1/Q1, Fjek W1/Q1, NVjM W3/W4/Q2) We added a new evaluation in which every arm gets the base model's first answer and the same antithesis from Gemini-3.8-flash or Claude-Haiku-4.5 on the nine benchmarks, each trained arm at its three training seeds (App. Q). Self-Disagreement's mean of correct holding (keeping a right first answer against a wrong antithesis) and correct yielding (taking a right antithesis over a wrong first answer) is 8 to 19 points above every other arm, with the highest AUC of switching in all four model–challenger pairs (Table 13). With the same design on the disagreement probe (Table 14), the mean rises by 5.1 [2.4, 7.7] and 17.2 [14.8, 19.6] points, and the log odds ratio, now comparable across models, by 0.68 [0.40, 0.98] and 1.23 [0.89, 1.56] (Qwen3-8B, Gemma-2 9B). We also added a test that pushes the base model's final answer toward its first by amounts fixed in advance, which on Gemma-2 9B brings it to Self-Disagreement's switch rate: it still decides worse, its mean lower by 10.4 and 7.0 points (against Gemini-3.8-flash and Claude-Haiku-4.5; App. Q). So on identical inputs the trained model decides better, not only holds more.
  2. Is it more than self-correction training? (NVjM W1/Q1, Fjek W2/Q2) We added SCoRe, trained with its two stages and progress bonus in our setting at three seeds, as an arm throughout (App. N): on identical inputs, it reaches 51.6% synthesis accuracy, against 66.4% for ours (Qwen3-8B, Gemini-3.8-flash; Table 1). We also ran PBT's released model in our setting: ours, from the same Llama-3.1-8B (one training run), leads it by 5.7 points [2.7, 8.6] in that mean (App. R). We extended App. A to position PBT, FlipFlop (Laban et al.), Wang et al. and S$^2$R against our design.
  3. Does the shared protocol favor our method? (NVjM W2, Fjek W3/Q3) The arms we compare against have no challenge in their training prompts, so all are tested under one instruction asking for a verdict (Table 5). We added a test of each, and of the base model, without the verdict request (App. C): each does worse on Qwen3-8B (capitulation 6.6 to 7.3 points higher), and none gains beyond its seed-to-seed spread on Gemma-2 9B. So the shared instruction does not handicap them.
  4. What does each arm's reward contain, and what does paying for the verdict add? (EJsb W2/Q2/Q3, NVjM Q3) We defined each of CAD-RL's reward terms (App. C), scoped the claim to the disagreement's content together with the supervision that depends on it (§3, §4.2, App. F), and added a panel to Table 7 with every row's reward terms (App. E). Paying for the verdict lowers capitulation from 2.9% to 1.0% (Table 3).
  5. Do the gains reproduce? (NVjM Q4) We trained three seeds per arm for every main comparison, and the ± is now the spread across them: on Qwen3-8B, +5.90 points over the base model, [+5.00, +6.80] over the seeds (§4.1).

Where to find the changes (in blue)

  • Main text: Table 1 adds SCoRe and a lower block with identical inputs for every arm (§4.1), and Figure 4 a SCoRe curve; §2 states the claim as telling a correct challenge from an incorrect one (NVjM W4); §4.2 is restated by what Table 3 shows; Table 8's caption (App. I) states what is fixed within each model.
  • Seeds: three training seeds per arm in Tables 1–3, 6, 8–10 and 13 and Figures 1, 2 and 5 (Table 7 is a single-run ablation).
  • Challenger: Gemini-3.6-flash, one of the 16 outside challengers, was retired by its provider after the submission, so its successor, Gemini-3.8-flash, takes its place for every arm and run, and every number that depends on the challenger set is recomputed.
  • Appendices: B, Revise-RL against SCoRe; C, CAD-RL's reward and the verdict-request test; E, every Table 7 row and two training seeds with λstub\lambda_{\mathrm{stub}} at 2.0. New: N, SCoRe; O, first-answer accuracy per benchmark; P, three rounds of challenge; Q, identical inputs (Tables 13–14), the prompting baseline, matched switching and how often each outside antithesis is right; R, PBT. A to M keep their letters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.