acceptodds
Under review as a conference paper at ICLR 2027

Consensus Is Not Correctness: Reliability-Aware Self-Teaching for Unsupervised Self-Distillation

Abstract

Unsupervised on-policy self-distillation uses a model's own consensus to construct training signal, yet answer agreement does not certify the reasoning trace being imitated. We ask which quantities consensus actually certifies and introduce Reliability-Aware Self-Teaching (RAST), a control layer that separates answer confidence, group trace consistency, teacher-process reliability, and retained supervision. In process-auditable diagnosis logs, answer-correct traces can fail process audit in both Gemini 2.5 Pro and a dedicated Qwen3-4B process-gold audit subset, although process measurability is selective and the conditional rates are not population prevalence estimates. Answer confidence and teacher reliability specialize to different audit targets (AUROC 0.894 vs. 0.624 and 0.649 vs. 0.788), while joint thresholding traces a risk-coverage frontier below the audit-defined risk rate expected under random thinning of the same candidate pool. Correction-time selector rankings also disagree between answer and process objectives. A frozen five-seed LoRA study does not establish a multiplicity-corrected accuracy winner. A post-freeze three-seed MATH-500 downstream stress test likewise does not resolve a Majority@8 advantage for either self-distillation condition: the RAST-U-OPSD difference is -0.07 percentage points (95% paired two-way bootstrap CI [-1.40,+1.33]). The supported conclusion is narrower: self-distillation should monitor answer trust, process trust, and supervision coverage separately rather than collapse them into a single consensus score.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-distillation, unsupervised self-improvement, large language models, reliability, teacher selection, self-consistency, process supervision, on-policy distillation reliability-aware self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.20 over 5 reviews, above 2% of the others.
Reviewer_XnbuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper suggests the limitations of existing self-distillation by experimentally demonstrating that reasoning traces can be incorrect even when the final answers are correct (Table 2).

Based on this observation, the authors propose Reliability-Aware Self-Teaching (RAST), a distillation method that evaluates the trustworthiness of the teacher's reasoning based on three factors (1) answer confidence, (2) group trace consistency, and (3) teacher reliability. RAST then adjusts the types and strengths of the training signals according to the estimated reliability.

The paper also presents a variety of experiments to investigate whether training models with RAST leads to actual performance improvements.

Critical strengths

They demonstrate several important empirical findings:

  1. Even when the final answer is correct, the reasoning process can be incorrect.
  2. Answer confidence and teacher reliability predict different targets.
Critical weaknesses

Major This paper revisit a well-known issue: answer-level correctness does not always imply the correctness of intermediate reasoning steps. Although the paper provides diagnostic analyses of this issue, it does not convincingly demonstrate that the proposed method effectively addresses it. Specifically, the experiments fail to demostrate that RAST framework improves downstream performance.

Minor

  1. Lack of ablation studies: the separate effects of answer confidence, group trace consistency, and teacher reliability are not evaluated.
  2. Section 5.4 contains overly detailed descriptions of the statistical analysis and experimental protocol (It would be better written in Appendix to improve readability).
Questions for authors
  1. Section 3.1; what is "diagnosis tasks" and how authors get the gold label for each process.
Reviewer_RcQUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether agreement on a final answer is a sufficient basis for choosing reasoning traces for self-distillation. It separates answer confidence, group trace consistency, teacher reliability, and retained supervision through a proposed RAST control layer. Audits on constructed reasoning tasks find that correct final answers can coexist with failed intermediate-quantity checks and that answer confidence and teacher reliability predict different audit targets. Correction experiments and small LoRA studies examine possible downstream consequences. Neither the primary five-seed study nor the three-seed MATH-500 study establishes an accuracy advantage. The paper's supported claim is a diagnostic distinction between answer and process trust.

Critical strengths
  1. The paper carefully separates measured process-audit failure from causal training harm. It does not present conditional Lucky-Correct rates as population prevalence, non-significant differences as equivalence, or the strongest training point estimate as an established winner. Reporting the null downstream results and the limitations of the baseline implementations makes the conclusions substantially more credible.

  2. The answer/process distinction is supported by more than an illustrative example. The diagnostic analysis compares predictive signals using task-cluster uncertainty and checks stricter process-pass definitions. The separate Qwen audit also provides evidence that the phenomenon is not confined to the commercial audit model. Figure 1 and Table 6 provide a useful, if limited, empirical basis for evaluating answer and process objectives separately.

Critical weaknesses

The principal concern is a specific reproducibility limitation that the paper itself documents. Appendix A (p. 13, lines 663–690) states that the component list for the primary answer-confidence score CA is absent from the submission-source snapshot; Table 5 marks the raw-rollout CA/CS constructors as incomplete. Saved scores allow statistical reanalysis, but do not specify how to compute the same primary score from new rollout text. Since CA is central to the answer-versus-process comparison and filtering policy, this limits independent reconstruction of an important part of the reported method. The teacher reliability formula and policy constants are supplied, and the limitation is disclosed clearly.

I do not think the small diagnostic sample or the absence of an established training improvement as separate critical weaknesses, given the paper's explicitly limited diagnostic claim.

Questions for authors

None. The main concerns concern the demonstrated contribution and evaluation scope, rather than an ambiguity requiring an author response.

Reviewer_beFFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether answer consensus is sufficient for selecting reliable reference traces in unsupervised on-policy self-distillation. It proposes RAST, which separates answer confidence, group consistency, teacher reliability, and supervision coverage. Experiments include process audits, correction-time teacher interventions, a five-seed LoRA study, and a three-seed MATH-500 evaluation. The strongest positive finding is that answer-confidence and teacher-reliability scores predict different audit targets. The intervention studies do not establish a downstream accuracy advantage.

Critical strengths

The paired task-cluster analysis in Table 6 provides evidence that answer confidence and teacher reliability predict different audit targets. This is a useful diagnostic observation and the strongest basis for a potential contribution. The authors also appropriately distinguish conditional audit rates from population prevalence and avoid interpreting nonsignificant downstream results as equivalence or superiority.

Critical weaknesses
  1. The diagnostic evidence does not sufficiently establish the framework’s added value. The motivating failure is an answer-correct reference trace with incorrect reasoning. However, the risk–coverage analysis measures an incorrect consensus answer with a correct minority present (§3.2), so it does not directly establish improved filtering of that failure. Reference-trace correctness also does not establish the quality of the teacher distribution conditioned on that trace, and the correction and training studies leave this connection unresolved. Validation is further limited by narrow task coverage, small retained training datasets, and weight-update experiments using only one 4B model family, leaving generality across tasks and model scales unestablished.

  2. The method is incompletely specified at points essential to interpreting its results. Appendix A and Table 5 acknowledge that the primary CAC_A and CSC_S constructors cannot be reconstructed from the submitted snapshot. The manuscript also needs to explain how its separate answer and trace weights enter the distillation loss, particularly given Appendix D.2’s description of the applied sample weight and identical effective updates across variants. Archived scalar values and execution hashes do not substitute for an operational definition of the method.

Questions for authors

see weakness above

Reviewer_THRCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies unsupervised on-policy self-distillation (U-OPSD; Li et al., 2026), in which a model's majority-vote pseudo-solution conditions a same-model teacher that is distilled into the model's incorrect rollouts. It argues that answer agreement certifies the answer but not the reasoning trace being imitated. To separate the two, it proposes RAST, a control layer that computes answer confidence CAC_A (vote support and margin), group trace consistency CSC_S, and a per-candidate teacher reliability RTR_T (agreement of a trace's decisive numeric quantities with the other agreeing traces, multiplied by a deterministic self-consistency check). RAST maps these signals to full-trace, reduced-trace, answer-only, down-weighted or abstaining supervision.

Critical strengths

The statistical reporting is careful, and the paper reports its nulls as nulls. The paper uses a task-cluster bootstrap over task IDs, a pre-specified Holm family, an explicit list of 56 invalidated runs, and a clear separation between the frozen confirmatory study and the post-freeze analyses. It also shows with its own numbers how a homogeneous slice overstates an effect: seeds 1–3 give +4.27 pp (unadjusted p = 0.034), while the pooled five-seed estimate is +2.72 pp (Holm p = 0.435). Self-improvement papers routinely report single-seed gains of this size, so this is a useful methodological example for the area.

The dissociation between answer confidence and teacher reliability (Fig. 1a, Table 6) is the most informative result. Each signal predicts a different target. The selector study shows the same split at correction time: the answer-gold oracle selector has the highest answer-correction rate (0.870) but a process-correction rate (0.609) below that of the medoid selector (0.804). If the dissociation replicates on more than 16 process-auditable tasks, a self-distillation filter would have to state which of the two targets it protects.

Critical weaknesses

The training experiments cannot show whether RAST helps, because U-OPSD itself does not improve the model in the authors' setup. On MATH-500, U-OPSD lowers Majority@8 by 1.00 pp relative to the untrained model and RAST lowers it by 1.07 pp; in the weight study, the best variant gains 2.72 pp on 125 problems, close to the 1.44 pp re-measurement noise.

A reliability layer can only demonstrate value on top of a self-distillation method that works, so these null results are uninformative about RAST, and the contribution comes down to the diagnostic section. That section is small and its labels are not validated. The Lucky-Correct rates depend on an automatic rubric that matches extracted intermediate quantities, and the authors report no human check of its precision. RTR_T is computed from the same extraction, which may inflate its AUROC on the process target, and the RT>CAR_T > C_A separation uses only 16 tasks and excludes zero only at the 50% threshold. In the risk–coverage analysis, the joint gate's advantage over thresholding on answer confidence alone lies within overlapping intervals and is never tested directly.

Finally, Uesato et al. (2022) and ProcessBench (Zheng et al., 2024) have already quantified how often correct final answers come with flawed reasoning, and the paper cites neither.

Questions for authors

Most of my concerns about the diagnostic section could be settled with a few additional numbers. What fraction of the traces labeled Lucky Correct (for example, 50 of the 311 Gemini traces and 30 of the 72 Qwen traces) does a human annotator or an independent strong judge confirm as containing a real reasoning error? How does the process-target AUROC of RTR_T change when the depth gate and the representativeness term are removed, or when RTR_T is computed on quantities the rubric does not score? And at matched coverage, how much lower is the risk of the joint gate than that of thresholding on CAC_A alone, with a task-cluster interval, and what does the risk–coverage curve look like when risk is defined by the process target, the quantity RAST is meant to protect?

On the training side, why does your U-OPSD implementation fail to improve over the untrained model on MATH-500 when Li et al. (2026) report gains, and can you rerun it in their reported setting (training schedule, KL estimator and sampling) so that RAST is compared with U-OPSD where U-OPSD actually helps? An answer here would change my assessment more than any of the diagnostic numbers. Please also give the full definition of CAC_A and say whether the 133-problem suite and its process annotations will be released.

Reviewer_dbmsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies unsupervised on-policy self-distillation, where a majority-consistent pseudo-solution conditions a teacher distilled into prefixes of the student's own incorrect completion. Authors argue that answer consensus certifies which outcome recurs, not which reasoning trace is safe to imitate and propose RAST, a control layer separating answer confidence, group trace consistency, and per-candidate teacher reliability, mapped to full-trace, reduced-trace, answer-only, down-weighted, or abstaining supervision.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The paper addresses an important problem: consensus certifies which answer recurs, not which trace is safe to imitate. Splitting this into answer trust, trace trust, and retained coverage is a cleaner unit of control than the consensus-level admission it replaces.

Critical weaknesses
  1. My main concern is that the paper never shows the proposed RAST method actually helps. In the five-seed study no variant beats the no-training baseline once multiple comparisons are corrected, and on MATH-500 the method is no better than the baseline or than the simpler U-OPSD.

  2. Everything rests on one small student, one self-built reasoning suite, and one pass over a public benchmark, with the best gain inside re-measurement noise. There is no larger-scale or second-model validation, which limits how far the conclusion can be pushed.

  3. A related concern is recency. All of the analysis and experiments are run on Gemini 2.5 Pro and Qwen3-4B, and both now look somewhat dated, so it is unclear whether the same separation still holds for current models.

  4. The general claim is not new. That a correct final answer does not certify the reasoning behind it is already established in process supervision and in work on chain-of-thought faithfulness, and so is the worry that agreement can reinforce a shared error rather than average it out. The paper's increment is to quantify this once inside a particular self-distillation construction, where the teacher's privileged context is itself built from the same consensus being checked, which is a useful localization of a known risk, but not a new finding.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.