- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies a simple approach to label-free self-distillation. Specifically, the authors first generate 16 independent reasoning trajectories, apply Recursive Self-Aggregation (RSA), and use majority voting to select a pseudo-answer. The aggregated result determines the pseudo-answer, while a matching trajectory from the initial generation is preferred as privileged context for on-policy self-distillation. The authors evaluate CASD on Qwen3-4B and Qwen3-8B across seven mathematical reasoning benchmarks. Experimental results show that CASD outperforms the evaluated label-free baselines and achieves better performance than methods using ground-truth supervision on most benchmarks.
Critical strengths
-
The paper addresses an important research problem with a simple and straightforward approach, achieving relatively consistent improvements across multiple benchmarks.
-
The idea of using the final aggregated results to select a reliable pseudo-answer while preferring an initial reasoning trajectory as privileged context for student training is interesting and insightful.
Critical weaknesses
-
RSA and OPSD are both established methods, and the main novelty of CASD appears to lie in separating pseudo-answer selection from privileged-context selection. Although the paper cites CANON [1] and other consensus-based self-distillation methods in Section 2.2, the label-free baselines in Table 3 only include TTRL and Intuitor. A direct comparison with CANON under matched generation budgets would help clarify the methodological contribution.
-
The paper observes that aggregation reduces success coverage and shortens reasoning trajectories, but the relationship between these changes and downstream student performance remains unclear. More controlled experiments or theoretical analysis would help explain why selecting earlier reasoning trajectories improves self-distillation and identify the actual sources of the performance gains.
-
CASD generates up to 64 trajectories per problem, with later generations conditioned on previous responses, introducing substantial computational overhead. It is unclear whether simpler approaches, such as increasing parallel sampling or using standard consensus-based self-distillation, could achieve comparable improvements under the same computational budget. Furthermore, experiments are limited to two Qwen models, and evaluations on additional model families would strengthen the evidence for generalizability.
[1] Gkountouras, John, Josip Jukić, and Ivan Titov. "Consensus as Privileged Context for Label-Free Self-Distillation." arXiv preprint arXiv:2607.13643 (2026).
Questions for authors
-
Does CASD outperform CANON under the same generation compute budget?
-
Could the authors provide more controlled experiments or theoretical analysis to better explain the sources of the performance improvements?
Response to Reviewer ZY2Y
We appreciate the reviewer’s suggestions and discuss each point below.
W1: Lack direct comparison with CANON.
Thanks for pointing out this! Please check Comparison to other label free baselines in our global rebuttal.
W2: The relationship between rollout changes and downstream student performance remains unclear. Could the authors provide more controlled experiments or theoretical analysis to better explain the sources of the performance improvements?
Thanks for raising this concern. We add token level controlled experiments For the student-generated responses, we check the output token length, the markers per response, and the markers per 10k tokens of the proposed method and several baselines.
Here are the results:
Markers per response measures how often explicit reflection appears in an answer; markers per 10k tokens measures its density, accounting for differences in response length.
The SFT and OPSD models produce reasoning traces that are substantially longer than necessary and use reflection markers far more frequently than needed, exhibiting signs of overthinking.
Appendix D further provides matched-prefix teacher case studies, illustrating how initial references can increase support for diverse reflective expressions and guide the teacher back to checking intermediate errors.
W3.1 && Q1: Does CASD outperform CANON and increase parallel sampling under the same generation compute budget?
Thank you for highlighting this point. The generation compute budget is an important consideration, and we agree that incorporating this context strengthens our evaluation. We have added comparisons with additional baselines under matched sampling budgets in the Comparison to other label-free baselines section of our global rebuttal. Specifically, we evaluate two label-free approaches: CANON using an equivalent sampling budget and an expanded parallel sampling baseline. CASD consistently outperforms both methods.
W3.2: Generalizability of our method. To further assess the generalizability of our approach across generation modules, we evaluated CASD after replacing RSA with PDR. The results demonstrate that:
The results indicate that the CASD approach remains robust even when the generation module is swapped. While PDR-CASD shows a slight decrease in overall average performance (51.57%) compared to the original RSA-CASD (52.55%), it remains highly competitive, demonstrating that the CASD framework is effectively generalizable across different generation methods.