acceptodds
Under review as a conference paper at ICLR 2027

Consensus as Privileged Context for Label-Free Self-Distillation

40 views

Abstract

Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 15 points, outperforming label-free reinforcement learning by 6 points at 15% of its compute and approaching a teacher conditioned on gold solutions. Trained on unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis attributes most of the improvement to the consensus being distilled into the weights and finds no significant narrowing of coverage. On the mathematics benchmarks, CANON's pass@ advantage over the base model persists at every sampling budget measured, reaching 512 samples per prompt, and its majority vote itself becomes more accurate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, post-training, self-distillation, on-policy distillation, self-consistency, test-time training, test-time scaling, label-free learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tK1POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CANON (Consensus-ANchored self-distillatiON), a method that uses the model’s own consensus as privileged context for label-free self-distillation. For each unlabeled prompt, CANON samples multiple rollouts, identifies the majority answer, and selects the corresponding solution with the highest mean token log-probability. A frozen snapshot teacher receives this consensus solution, and the student minimizes a full-vocabulary Jensen–Shannon divergence over all initially sampled rollouts for one epoch.

The paper evaluates both transductive test-time training and inductive transfer. On Qwen3-4B-Instruct-2507, CANON improves AMC avg@32 from 66.0 to 76.5 using 4.8 GPU-hours, compared with 70.3 and 32.4 GPU-hours for TTRL. Additional experiments examine transfer across tasks and models, conditioning choices, coverage, and training stability.

Critical strengths
  • A useful problem and a simple method. Conditioning a teacher on a consensus solution provides richer supervision than a scalar pseudo-reward. Reusing the same rollouts for consensus construction and distillation makes the approach practically attractive.

  • Convincing improvements on the main comparison. The AMC gains over the base model and TTRL are substantial, with paired bootstrap intervals of [+5.8, +15.6] and [+2.9, +9.9], respectively.

Critical weaknesses
  • Applicability to full fine-tuning remains unverified. Nowadays, full SFT is the predominant approach for postraining in most of the frontier labs. In contrast, all CANON experiments use LoRA, leaving it unclear whether the reported benefits persist under full fine-tuning. Restricting parameter updates to a low-rank subspace may affect sensitivity to incorrect consensus, training stability, and coverage preservation.
Questions for authors
  • The main concern is about lora fintuning vs full SFT - are we confident to claim CANON will also work for full SFT?
3 comments

We thank the reviewer for the constructive review and suggestions. We ran CANON with full fine-tuning during the discussion period. We address the concern below.

"Applicability to full fine-tuning remains unverified [...] all CANON experiments use LoRA, leaving it unclear whether the reported benefits persist under full fine-tuning." and "are we confident to claim CANON will also work for full SFT?"

Yes. With all parameters updated and no adapter, CANON reaches 81.3 avg@32 on AMC for Qwen3-4B-Instruct-2507, against 76.5 with LoRA and 66.0 for the base model. We ran this on three models with the rest of the recipe unchanged, and selected the learning rate on the OmniMath development slice.

AMC, avg@32BaseCANON, LoRA (Table 3)CANON, full fine-tuning
Qwen3-4B-Instruct-250766.076.581.3
Qwen3.5-4B67.675.780.6
Gemma-4-E4B-it63.772.174.9

Full fine-tuning improves over LoRA on every model. The result for Qwen3-4B-Instruct-2507 is the mean of three training seeds. The same checkpoints also improve the held-out benchmarks of Table 1.

Qwen3-4B-Instruct-2507, avg@32AIME 2024AIME 2025GPQA
Base32.530.143.9
CANON, full fine-tuning40.034.149.0

"Restricting parameter updates to a low-rank subspace may affect sensitivity to incorrect consensus, training stability, and coverage preservation."

We checked the three properties with full fine-tuning. Training is stable. For Qwen3-4B-Instruct-2507, accuracy on the development slice stays between 72.6 and 76.9 when the learning rate varies by a factor of ten, and for Qwen3.5-4B accuracy on AMC stays between 79.8 and 80.8 when it varies by a factor of five. Over training, the loss decreases from 0.020 to 0.009, and the entropy of the student stays at 0.21. Coverage is preserved. On AMC, maj@32 rises from 83.1 to 86.7 and pass@32 from 85.5 to 88.8. The sensitivity to an incorrect consensus is the one reported for LoRA in Section 5.2. The base model holds a wrong majority answer with confidence on two AMC prompts, and both remain wrong after training.

Full fine-tuning is also the setting of the RL baselines. With the learning rate of TTRL selected on the same development slice, CANON reaches 81.3 against 76.3 for TTRL (three training seeds each), and one CANON run takes about one sixth of the GPU-hours of a TTRL run.

Changes in the revised manuscript

  • New Appendix L (Table 28) and Section 4.2 (Table 2): CANON with full fine-tuning on three models, with the tuning protocol, the seed variance and the held-out results.
  • Section 4.2: the full fine-tuning results.

We thank the reviewer again. We have uploaded an updated revision. We are happy to answer any remaining questions before the discussion closes. If our response addresses the reviewer's concern, we would be grateful if the reviewer would consider raising the score.

Reviewer_tK1POpenReview ↗

Thanks for the update

Thanks for the update. My main concerns are resolved and I raised my score

Reviewer_FpH4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CANON performs label-free self-distillation by conditioning a frozen teacher on a model-generated consensus solution and matching its token-level distributions over reused student rollouts. It achieves substantial accuracy gains at low training cost, with positive evidence of held-out transfer.

Critical strengths
  1. Simple and practical supervision construction without external teachers or gold training answers.

  2. Strong accuracy–computing tradeoff against the evaluated non-label baselines.

  3. Broad transductive validation across 7 models and 5 benchmarks, improving 34/35 settings.

  4. Useful conditioning ablations, uncertainty estimates, and large-budget coverage analyses.

Critical weaknesses
  1. The independent contribution of consensus selection is untested: there is no control that conditions on a random solution from the same prompt. This matters because gold and consensus anchors perform almost identically on math. The SFT comparison also changes data exposure (83 sequences vs. 2,656 rollouts).

  2. Comparable development-set tuning across baselines is not established. CANON’s learning rate is tuned per model on dev slices with LoRA, whereas the RL baselines use reference hyperparameters from other models, full fine-tuning, and an older software stack. This also confounds the GPU-hour comparison.

  3. Held-out evidence is limited and setting-dependent. Gains concentrate on AIME 2024, n=30, plausibly seen in pretraining, and several differences are insignificant, e.g. on AIME 2025, CANON-pool vs. TTRL-pool. Voting accuracy can decline, and the post-cutoff AIME 2026 set shows essentially no avg@32 gain.

Questions for authors
  1. How does consensus conditioning compare with a randomly selected solution from the same prompt?

  2. Can soft versus hard supervision be compared with identical trajectories and training exposure?

  3. Do the baseline gaps persist under comparable development-set tuning and stopping rules?

  4. Do the gains hold on larger sets of truly unseen problems (cause AIME 2026 shows almost none)? When training and testing on the same prompts, is CANON actually better than just taking a majority vote over the base model’s 32 samples?

4 comments

We thank the reviewer for the careful and constructive review. We ran the controls the reviewer asked for during the discussion period. We address each concern below.

"The independent contribution of consensus selection is untested: there is no control that conditions on a random solution from the same prompt." and "How does consensus conditioning compare with a randomly selected solution from the same prompt?"

We ran this control. The teacher is conditioned on a random solution of the same prompt in place of the consensus solution, and everything else is unchanged. The consensus solution is above the random solution in all five settings, by 8 points where the two often differ. The table reports consensus minus random in avg@32 (three training seeds, same student rollouts for both, 95% bootstrap intervals in brackets).

Model, benchmarkPrompts where the two solutions differConsensus − random
SmolLM3-3B, AMC43 of 83+8.0 [+4.0, +12.4]
SmolLM3-3B, MATH500134 of 500+8.5 [+6.8, +10.2]
Qwen3-4B-Instruct-2507, AIME 20255 of 30+2.0 [−1.4, +5.5]
Qwen3-4B-Instruct-2507, GPQA19 of 198+0.2 [−2.3, +2.7]
Qwen3-4B-Instruct-2507, AMC6 of 83+0.4 [−1.8, +2.0]

For SmolLM3-3B, the two solutions often differ, and the vote is what produces the improvement. With a random solution, SmolLM3-3B does not improve over the base model (−0.8 on AMC and −1.1 on MATH500). For Qwen3-4B-Instruct-2507, the two solutions carry the same final answer on most prompts, so the difference is small. On the prompts where they differ, the consensus solution is correct 35 times and the random solution 14 times over the three seeds. The majority vote needs no additional generation, since the 32 rollouts are already used for distillation.

"The SFT comparison also changes data exposure (83 sequences vs. 2,656 rollouts)."

We agree that the two differ in exposure, and the difference to SFT remains when exposure is matched. An exact match is difficult, because CANON is defined on several rollouts per prompt and SFT on one target solution per prompt. The closest comparison we could construct is the following. The same teacher generates 32 solutions per prompt, and the student is fine-tuned on them with the same number of sequences and optimizer steps as CANON.

Difference in avg@32 to CANONAMCGPQAAIME 2025
SFT with matched exposure−6.0 [−8.4, −3.8]−4.7 [−7.3, −2.2]−5.2 [−10.3, −0.6]

SFT with matched exposure is 4.7 to 6.0 points below CANON. The LMSI baseline of Table 1 is a second comparison of this kind. It fine-tunes on all 1,661 majority rollouts of the student and reaches 66.5 on AMC.

"Comparable development-set tuning across baselines is not established. CANON's learning rate is tuned per model on dev slices with LoRA, whereas the RL baselines use reference hyperparameters from other models, full fine-tuning, and an older software stack. This also confounds the GPU-hour comparison." and "Do the baseline gaps persist under comparable development-set tuning and stopping rules?"

With both methods updating all parameters and the learning rate of each selected on the same development slice, CANON is 5.0 points above TTRL on AMC (95% CI [+0.8, +9.5]), and above it on the held-out sets.

All parameters updated, avg@32AMCAIME 2024AIME 2025GPQAPost-cutoff
TTRL76.337.731.448.130.7
CANON81.340.034.149.033.1

Each row is the mean of three training seeds. The last column is a set of 148 competition problems released after the model (see our answer on held-out evidence below). In this comparison both methods use full fine-tuning, the same development slice, the same grid for the learning rate ({5e-7, 1e-6, 2e-6, 5e-6}, with 2e-6 selected for both) and the same evaluation. TTRL runs in a current version of its software stack (verl 0.9). Both methods are evaluated at the end of training, and neither uses the test set to select a checkpoint.

Regarding compute, one CANON run takes 1.4 GPU-hours and one TTRL run 8.1, with rollout generation included. The ratio is about one sixth. Note that hours are given for H100 GPUs, while Table 1 reports a node with RTX 6000 Ada GPUs.

"Held-out evidence is limited and setting-dependent. Gains concentrate on AIME 2024, n=30, plausibly seen in pretraining [...] Voting accuracy can decline, and the post-cutoff AIME 2026 set shows essentially no avg@32 gain." and "Do the gains hold on larger sets of truly unseen problems (cause AIME 2026 shows almost none)?"

On 148 post-cutoff problems, the AMC-trained checkpoint of the paper improves avg@32 by 4.1 points, maj@32 by 6.8 and pass@32 by 7.4 over the base model, and the three intervals exclude zero. We pooled five competitions held after the release of the model (AIME 2026, BRUMO 2026, CMIMC 2026, HMMT February 2026 and HMMT November 2025). AIME 2026 alone, with 29 problems, was too small to show this gain. The table reports the difference to the base model.

Checkpoint of the paperavg@32maj@32pass@32
CANON, trained on AMC+4.1 [+2.0, +6.4]+6.8 [+1.4, +12.2]+7.4 [+2.7, +12.8]
CANON, trained on the 339-prompt pool+3.7 [+1.3, +6.3]+2.7 [−2.0, +8.1]+6.8 [+2.0, +12.2]

The gain also holds on two larger sets. ArXivMath has 340 problems, each built from a mathematics paper posted to arXiv between December 2025 and September 2026, so it also post-dates the model. On this set avg@32 improves from 4.2 to 6.5 (+2.3 [+1.4, +3.3]). On OlympiadBench, with 672 problems, it improves from 57.4 to 62.1 (+4.7 [+3.5, +6.0]).

The difference to TTRL also holds on the post-cutoff problems. In the comparison with full fine-tuning and the same tuning, CANON is 2.4 points above TTRL in avg@32 (95% CI [+0.8, +4.1]) and 4.7 points in maj@32 (95% CI [+1.6, +8.1]).

"Can soft versus hard supervision be compared with identical trajectories and training exposure?"

Yes. We trained on exactly the 2,656 rollouts used by CANON, with the same frozen teacher, and replaced the teacher's full distribution at each token with its argmax, which is a hard target.

Difference in avg@32 to CANONAMCGPQAAIME 2025
Argmax token (hard target)−0.4 [−1.2, +0.5]−0.6 [−1.4, +0.2]+1.2 [−1.2, +4.4]

The two perform the same, so CANON is not sensitive to the form of the per-token target. In both cases, the supervision is equally dense, with one target from the consensus-conditioned teacher at every token of every rollout of the student. This is the property that separates CANON from a scalar reward per rollout and from sequence-level fine-tuning (Sections 3.2 and 6). Our answer on the SFT comparison above shows the complementary result. When the same teacher supervises through solutions that it generates itself, accuracy is 4.7 to 6.0 points lower.

"When training and testing on the same prompts, is CANON actually better than just taking a majority vote over the base model's 32 samples?"

With the same number of samples, yes. With 32 samples per prompt, the majority vote of CANON reaches 85.5 on AMC against 83.1 for the base model (Table 6), and with 512 samples 86.8 against 83.1 (Table 26 in the revised manuscript). pass@k is also above the base model at every k we measured, with 96.4 against 89.2 at k = 512. A single sample from CANON does not reach the 32-sample vote of the base model (76.7 against 83.1), but it removes more than half of the difference at 1/32 of the inference cost.

Changes in the revised manuscript

  • New Appendix M, Section 6: the teacher conditioned on a random solution of the same prompt (Table 30), hard targets on identical rollouts and SFT with matched exposure (Table 8).
  • New Appendix L, Section 4.2: CANON and the baselines with full fine-tuning and the same tuning procedure (Table 2).
  • Appendix I, Section 4.4: the pooled post-cutoff set (Table 22) and the two larger held-out sets.
  • Appendix E: confidence intervals for the new comparisons.

Thank you for the targeted controls. The matched TTRL comparison and the gains on 148 post-release problems substantially address my concerns about baseline fairness and held-out transfer. I am raising my rating from 2 to 3 (Weak Accept). I encourage the authors to narrow the mechanism claim: consensus selection clearly helps SmolLM3, but its advantage over a random same-prompt solution is inconclusive on Qwen3-4B. Likewise, the identical-rollout control shows no clear advantage of full-distribution targets over tokenwise argmax targets.

We thank the reviewer for the updated assessment and for the two suggestions. We agree with both. The revised Section 6 states that for Qwen3-4B-Instruct-2507, the random and the consensus solutions perform similarly, and that the form of the per-token target has little effect. We will make the rest of the paper consistent with this in the final version.

Reviewer_6sG1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CANON, a label-free self-distillation method that converts a model’s self-consensus into dense token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, identifies the majority answer, and selects a high-confidence solution supporting that answer as privileged information for the teacher. The student is guided by the teacher with the privileged information using full-token distribution with JS divergence. Experiments on mathematrical reasoning and GPQA demonstrate the effectiveness of the proposed CANON method.

Critical strengths
  • Compared to existing label-free RL methods that mainly reduce self-consensus to a binary pseudo answer, the paper uses a consensus-supported solution as privileged information for dense token-level self-distillation.
  • The idea of the paper is intuitive and easy to understand: it converts the information contained in self-consistent reasoning trajectories into a richer training signal without requiring gold labels or external verifiers.
Critical weaknesses
  • The main experimental comparison is limited in scale, using only Qwen3-4B-Instruct-2507 and 83 unlabeled AMC prompts for training. Although the paper evaluates the resulting checkpoint on additional benchmarks, these evaluations mainly assess transfer from the small AMC training set rather than provide equally comprehensive comparisons with competing methods across diverse training settings. - It therefore remains unclear whether CANON’s advantage consistently generalizes across model families and larger or more diverse unlabeled training sets.
  • The novelty is limited given the growing body of label-free RL and self-rewarding methods, such as TTRL [1], Intuitor [2], EMPO [3], Co-rewarding [4], and CoVo [5]. In contrast, CANON mainly repurposes self-consistency as privileged context for token-level self-distillation. The paper does not sufficiently clarify the unique motivation for, or advantages of, this formulation over existing label-free post-training paradigms.
    [1] TTRL: Test-Time Reinforcement Learning.
    [2] Learning to Reason without External Rewards.
    [3] EMPO: Fully Unsupervised LLM Reasoning Incentivization.
    [4] Co-rewarding: Stable self-supervised rl for eliciting reasoning in large language models.
    [5] Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning.
  • The experimental evaluation lacks a large-scale inductive setting, where the model is trained on a relatively larger datasets, such as DAPO-Math-17K or DeepScaleR, and then evaluate on a series of benchmarks.
Questions for authors
  • Could the authors more comprehensively review recent label-free RL and self-rewarding methods, and clarify position of CANON relative to these approaches?
  • Could the authors evaluate CANON in a larger-scale inductive setting, with more model families and scales?
  • Could the authors provide further analysis to show whether CANON can avoid the training collapse often observed in OPSD-style methods?
7 comments

We thank the reviewer for the detailed and constructive review. We ran the larger-scale inductive experiments and the additional baselines the reviewer requested during the discussion period. We address each concern below.

"The main experimental comparison is limited in scale, using only Qwen3-4B-Instruct-2507 and 83 unlabeled AMC prompts for training [...] It therefore remains unclear whether CANON's advantage consistently generalizes across model families and larger or more diverse unlabeled training sets.", "The experimental evaluation lacks a large-scale inductive setting, where the model is trained on a relatively larger datasets, such as DAPO-Math-17K or DeepScaleR, and then evaluate on a series of benchmarks." and "Could the authors evaluate CANON in a larger-scale inductive setting, with more model families and scales?"

In the large-scale inductive setting, CANON improves every held-out benchmark for Qwen3-4B-Instruct-2507, by up to 10 points on AIME 2024 and 7.8 on AMC, and the improvement holds for a larger model and for a second model family. We trained on unlabeled prompts from DAPO-Math-17K, DeepScaleR, and OmniMath, with up to 4,000 prompts, which is ~50 times the size of the AMC training set, and evaluated on six sets that are all held out. The last one consists of 148 problems from five competitions held after the release of the model (AIME 2026, BRUMO 2026, CMIMC 2026, HMMT February 2026 and HMMT November 2025) to control for training leakage. The following table reports the difference in avg@32 to the base model for Qwen3-4B-Instruct-2507.

Training setPromptsAMCAIME 2024AIME 2025GPQAMATH500Post-cutoff
DAPO-Math-17K4,000+7.8+7.4+4.1+5.7+1.8+4.5
DeepScaleR2,000+7.3+10.0+0.3+5.8+1.5+5.2
OmniMath2,000+6.7+9.3+1.9+5.7+1.9+4.9

The improvement persists for the three training sets, and it transfers from mathematics to GPQA. The second table reports the same differences for two further models trained on 2,000 DAPO-Math-17K prompts.

ModelAMCAIME 2024AIME 2025GPQAMATH500Post-cutoff
Qwen3.5-9B+1.8+8.6+5.7+2.9+1.2+5.6
Gemma-4-E4B-it+7.8+7.2+3.1−0.2+4.1+6.2

CANON improves in 11 of these 12 settings. On model families, Table 3 of the revised paper also trains CANON on seven models from four families, between 2B and 9B parameters, and 34 of the 35 settings improve. For the comparison with competing methods, see our answer on label-free RL methods in the second part of this response, where every method is tuned in the same way.

We used subsets of the proposed datasets because a full pass did not fit in the compute available to us during the discussion period. Further runs are in progress, and we will post their results if they complete in time.

"The novelty is limited given the growing body of label-free RL and self-rewarding methods, such as TTRL, Intuitor, EMPO, Co-rewarding, and CoVo [...] The paper does not sufficiently clarify the unique motivation for, or advantages of, this formulation over existing label-free post-training paradigms." and "Could the authors more comprehensively review recent label-free RL and self-rewarding methods, and clarify position of CANON relative to these approaches?"

The five methods share one design. Each assigns a scalar reward to every sampled response without gold labels and optimizes it with policy gradient, with new responses sampled throughout training. The reward is agreement with the majority-voted answer in TTRL, self-certainty in Intuitor, the frequency of the response's semantic cluster in EMPO, agreement with the majority answer on a rephrased question or of a slowly updated copy of the policy in Co-rewarding, and how consistently the intermediate reasoning states support the final answer in CoVo. CANON uses the consensus differently. The consensus solution is given as privileged context to a frozen copy of the model, and this teacher provides a target at every token of every rollout. A rollout therefore receives a few thousand token-level targets where the RL methods assign one number and disperse the reward indiscriminately. CANON needs a single generation pass and one epoch of training. The revised Section 2 states this positioning and covers the five methods.

We also extended the comparison with these methods. The submission compares CANON with TTRL, EMPO and three further RL baselines (Table 1). We added Intuitor, Co-rewarding-I and CoVo. In the table below, every method is trained with all parameters updated, and its learning rate is selected on the same development slice. CANON has the highest accuracy on AMC, in avg@32 and in maj@32, and on the post-cutoff problems, at the lowest training compute.

MethodAMC avg@32AMC maj@32AIME 2024AIME 2025GPQAPost-cutoffGPU-hours per run
Base66.083.132.530.143.928.8
CANON81.386.740.034.149.033.11.4
Co-rewarding-I79.183.540.130.849.232.019.7
TTRL76.381.937.731.448.130.78.1
Intuitor71.283.937.230.347.229.94.6
CoVo66.282.334.530.244.629.43.8

Each row is the mean of three training seeds. On AIME 2024, AIME 2025 and GPQA, CANON is the best method or within 0.2 points of it. The post-cutoff column uses the same evaluation benchmark described in the first answer. On these problems, CANON also improves maj@32 by 3.8 points, and Co-rewarding-I and TTRL lower it by 0.5 and 0.9. Note that GPU-hours are given for H100 GPUs, while Table 1 reports hours on RTX 6000 Ada GPUs.

"Could the authors provide further analysis to show whether CANON can avoid the training collapse often observed in OPSD-style methods?"

We do not observe this collapse with CANON. CANON follows two choices that prevent it in our experiments. The teacher is a frozen snapshot that is not updated during training, and the student is trained for a single pass. OPSD makes the same choice and fixes the teacher to the initial policy (Zhao et al., 2026). The collapses reported by Yuan et al. (2026) and Guo et al. (2026) occur when the teacher is the current policy or is refreshed from it, and their runs with a teacher fixed at the initial policy do not collapse. The new Appendix O reports the training loss and the entropy of the student over training. On AMC the loss decreases from about 0.020 to 0.010 without reaching zero, and the entropy of the student stays at its initial value of about 0.2. The same holds over the much longer training regimes. On the DAPO-Math-17K prompts, the loss decreases from 0.016 to 0.013, the entropy of the student stays at 0.24, and held-out accuracy is stable between the checkpoints at 2,000 and 4,000 prompts (AMC +6.1 and +7.8 over the base model, AIME 2024 +7.7 and +7.4). After training, pass@k on AMC remains above the base model up to k = 512 (96.4 against 89.2), so the set of problems that the model can solve does not shrink the longer we train.

Changes in the revised manuscript

  • Section 2: positioning of CANON relative to label-free RL and self-rewarding methods, with the five methods the reviewer lists.
  • Section 4.2 (Table 2) and new Appendix L: Intuitor, Co-rewarding and CoVo, with full fine-tuning and the same tuning procedure for every method.
  • Section 4.4 (Table 5) and new Appendix N: training on up to 4,000 prompts from DAPO-Math-17K, DeepScaleR and OmniMath, on three models.
  • New Appendix O (Table 31): training loss and student entropy over training.
Reviewer_6sG1OpenReview ↗

Response by the reviewer

Thanks for your responses.

I have a few folow-up questions:

  • Could the authors clarify why CANON’s reported computational cost is so low and whether the 1.4 GPU-hours includes the time required to generate the consensus solutions used as privileged context?
  • Could the authors clarify the training details, including training dataset and the number of training examples, epochs, steps used for this comparison, and clarify whether all methods are trained on the same set of prompts?

Additionally, it seems that there is room in the main text of the revised paper. Why not move some of the key results, including those added during the revision, from the appendix to the main text?

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback.

We have uploaded a revised version of the paper. It includes additional experiments and the discussion from our responses in the main text and in the appendix. All changes are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.