acceptodds
Under review as a conference paper at ICLR 2027

CoReD: Council-Weighted Reasoning Distillation

Abstract

Standard chain-of-thought distillation transfers reasoning ability to small models, but typically treats all reasoning steps uniformly despite substantial differences in their importance within a reasoning trajectory. Existing step-aware methods largely use importance signals for hard selection or pruning, leaving the retained steps weighted uniformly. We propose Council-Weighted Reasoning Distillation (CoReD), which derives continuous step weights from a deliberately diversified council of LoRA experts. For each step, we combine the experts' average predictive uncertainty with their normalized disagreement, capturing whether a step is difficult for individual experts or controversial across the council. To make disagreement informative, the experts are explicitly diversified at uncertain reasoning steps before being consolidated in weight space. The merged adapter initializes the student and acts as a frozen distributional anchor during subsequent step-weighted optimization. After training, the learned adapter is merged into the backbone, introducing no additional inference overhead. Across mathematical reasoning benchmarks, CoReD consistently outperforms standard CoT distillation and recent step-aware reasoning methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, Chain-of-thought Reasoning, LoRA, Mixture of Experts, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Swa7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoReD assigns continuous training weights to reasoning steps using a council of three LoRA experts. It first trains the experts on adaptively truncated teacher trajectories and encourages diverse predictions at high-entropy steps. It then combines mean predictive uncertainty and relative disagreement to set the step weights. A merged model initializes the student and provides a frozen KL reference during weighted training. The final system uses one model at inference. Experiments on three backbones and four math benchmarks report the highest average score among the compared methods.

Critical strengths
  1. The motivation is clear, and the content is accessible and easy to follow. Besides, the paper provides clear mathematical definitions and formal proofs for the key properties of the proposed method, especially the information-theoretic interpretation of council disagreement.

  2. The combination of LoRA and model merging is a good design choice. It makes training multiple experts efficient while adding no extra inference overhead. The experiments are also fairly comprehensive. Detailed ablation studies help explain how different components contribute to the final performance.

Critical weaknesses
  1. The contribution of the design choices to the final student needs clarification. Table 2 shows that functional repulsion drives most of the predictive diversity. Orthogonality and step-level dropout have little effect on JS. The final student benefit of orthogonality and dropout remains unclear.

  2. The stability of the evaluation and the total training cost require further analysis. Table 3 reports results based on only three samples per problem. I think there is still significant fluctuation regarding AIME24/25 and AMC12, rendering the evaluation results and conclusions unreliable. I suggest the authors report results for avg@32 or avg@64 and analyze whether the conclusions remain consistent.

Questions for authors
  1. How do one or two alternative merge operators affect final CoReD performance under the same setting? Do higher merge-only scores also lead to higher final scores?

  2. Whether each diversity-oriented design is meaningful? How does removing orthogonality or step-level dropout affect the final CoReD student after Phase 2?

  3. How does the total cost of compare with standard CoT distillation and other step-aware baselines?

Reviewer_GShwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoReD, a framework for CoT distillation based on the disagreement of experts. The method proposes the weighting of reasoning steps for the CoT distillation, where the weights are based not only on the perceived difficulty but also on disagreement of council (ensemble) of expert LoRA modules. The LoRA experts are trained on pruned rationales from teachers to train the council for both accuracy and diversity. Next, the trained council is merged to compute the weight of reasoning steps capturing sample difficulty and council disagreement for given steps. The student is finally trained on the weighted reasoning steps in a teacher-forcing manner. The method is evaluated on several math benchmarks to demonstrate its effectiveness.

Critical strengths
  1. The paper is well-written, and the idea is clearly presented.

  2. The approach of utilizing the LoRA ensembles combined with epistemic uncertainty (disagreement among expert ensembles) appears novel and effective for weighted distillation of teacher rationales. This key approach is properly backed by the analysis, although the derivation itself is somewhat straightforward in the framework of Bayesian active learning (also cited by the authors, e.g., BALD).

  3. The proposed method achieves significant gains over recent baseline methods on distillation dealing with long CoT.

Critical weaknesses

My concerns are mostly about design choices.

  1. Ensemble size. One of important hyperparameters in this work would be the size of council or MM. The setting in the experiments is M=3M=3. The number appears somewhat small, which may increase the variance of importance estimate. Why not use larger MM? A related discussion or experiments are not provided. Empirically, a small MM may suffice, but it would be good if some study is provided on various values of MM.

  2. Heuristic weighting. The weighting in Eq. (12) uses UiU_i and ρi\rho_i, based on a function of a somewhat heuristic form. Why not use ViV_i instead of ρi\rho_i, or use ViV_i only? Isn't ViV_i in principle sufficient to measure the informativeness of a sample reasoning step as in BALD? Heuristics is fine, but it would be good if stronger justification or analysis is provided.

Questions for authors

It would be appreciated if the authors can address the concerns in Weaknesses.

3 comments
AuthorsOpenReview ↗

Reminder to engage in discussion

Dear Reviewer GShw,

We have provided detailed responses and additional experimental results addressing the concerns raised in your review. We would greatly appreciate it if you could take a moment to review our responses and let us know if any questions or concerns remain. We would be happy to provide further clarification if needed.

As the rebuttal deadline is approaching, we kindly invite you to engage in further discussion.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_GShwOpenReview ↗

Thanks

Thank you for the thorough rebuttal. I will keep my positive rating.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer GShw,

Thank you for your response and for taking the time to consider our rebuttal.

We sincerely appreciate your constructive comments and suggestions, which have helped us improve the justification of our design choices and strengthen the experimental analysis.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_md1hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Council-Weighted Reasoning Distillation (CoReD), a step-weighted chain-of-thought distillation framework. It trains a council of LoRA experts on adaptively truncated teacher trajectories, encouraging diversity through step-level supervision dropout, orthogonality regularization, and functional repulsion. The council's mean predictive entropy and normalized disagreement are combined into continuous reasoning-step weights. The experts are merged using TIES; the merged adapter initializes the student and serves as a frozen distributional anchor during weighted optimization. The council is needed only during training, so the final model has no additional inference-time routing overhead. Experiments use s1K-1.1 with DeepSeek-R1 as teacher and evaluate Qwen2.5-7B-Instruct, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-1.5B on MATH500, AIME24, AIME25, and AMC12. The paper reports the best average Pass@1 and Pass@3 among the evaluated baselines, with some benchmark-specific exceptions for the smallest student. This paper proposes Council-Weighted Reasoning Distillation (CoReD), a step-weighted chain-of-thought distillation framework. It trains a council of LoRA experts on adaptively truncated teacher trajectories, encouraging diversity through step-level supervision dropout, orthogonality regularization, and functional repulsion. The council's mean predictive entropy and normalized disagreement are combined into continuous reasoning-step weights. The experts are merged using TIES; the merged adapter initializes the student and serves as a frozen distributional anchor during weighted optimization. The council is needed only during training, so the final model has no additional inference-time routing overhead. Experiments use s1K-1.1 with DeepSeek-R1 as teacher and evaluate Qwen2.5-7B-Instruct, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-1.5B on MATH500, AIME24, AIME25, and AMC12. The paper reports the best average Pass@1 and Pass@3 among the evaluated baselines, with some benchmark-specific exceptions for the smallest student.

Critical strengths
  1. Continuous step weighting is a reasonable alternative to binary pruning. The paper distinguishes a step's own uncertainty from disagreement between plausible predictors. This may capture different reasons a reasoning step is informative, while retaining the complete trajectory rather than deleting selected segments.
  2. The training-only council avoids inference-time architectural overhead. The proposed use of TIES merging both initializes the student and supplies a frozen distributional target. If robust, this is a practical way to benefit from an ensemble-like signal without deploying multiple experts.
  3. The evaluation spans several student scales and mathematical benchmarks. Testing three backbones, including a 1.5B model, is useful evidence that the method is not tied to one model size. CoReD reports the strongest average Pass@1 and Pass@3 across the four benchmarks, although individual benchmark wins are not universal.
  4. The paper includes component-level ablations. The analysis of council diversity objectives and weighting components helps make the proposed multi-stage method more interpretable than a single opaque training recipe.
Critical weaknesses
  1. The source of the gains is not fully isolated. CoReD changes several elements simultaneously: adaptive teacher-trace truncation, council diversity training, uncertainty/disagreement weighting, TIES merging, merged-adapter initialization, and KL anchoring. A matched baseline using the same truncation and initialization but uniform weights is essential to establish how much improvement comes specifically from council-derived weights.
  2. The relationship between council disagreement and training value is assumed more than demonstrated. Disagreement can indicate an informative decision point, but it can also arise from unstable or poorly calibrated experts. The paper should show whether high-weight steps are actually more consequential for final answer correctness, ideally using counterfactual step removal or intervention-based measures.
  3. The council and merging procedure introduce considerable training complexity. Multiple LoRA experts require additional optimization and forward passes to obtain council signals. The paper argues that there is no inference overhead after training, but the overall training compute and wall-clock cost are not sufficiently central to the comparison.
  4. The benchmark and data scope is narrow. Training uses s1K-1.1 with DeepSeek-R1-generated reasoning, and evaluation is confined to four mathematical reasoning benchmarks. The results do not establish whether the method generalizes to non-mathematical reasoning, different teacher distributions, or teachers with different styles and error profiles.
  5. Average improvements obscure heterogeneous benchmark results. CoReD is not best on every metric for the 1.5B student: Segment Selective SFT and P-ALIGN win on some individual benchmark metrics. More detailed uncertainty estimates and consistent multi-seed comparisons would help establish the reliability of the average gains.
  6. The continuous-weight construction needs clearer sensitivity analysis. The normalization and combination of entropy and disagreement determine which steps receive high weight. The paper should compare entropy alone, disagreement alone, random weights, and uniform weights under otherwise identical training.
Questions for authors
  1. What is the performance of the full CoReD pipeline with uniform step weights, keeping adaptive truncation, merged initialization, and KL anchoring unchanged?
  2. Do learned weights identify causally important reasoning steps? Can you validate them using controlled step ablations or counterfactual interventions?
  3. What are the total training FLOPs, GPU hours, and memory costs of training the council and computing weights compared with each baseline?
  4. How sensitive are results to the number of council experts, the entropy/disagreement combination, and weight normalization?
  5. Can the approach be evaluated with a different teacher or on at least one non-mathematical reasoning task?
  6. Are benchmark improvements consistent across multiple random seeds, and are confidence intervals available for Pass@1 and Pass@3?
1 comment
AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer md1h,

Thank you for your thoughtful follow-up and for taking the time to carefully consider our additional experiments and clarifications.

We sincerely appreciate your constructive feedback throughout the discussion, which has helped us strengthen both the experimental evidence and the presentation of our work. We also acknowledge the remaining limitations you highlighted and will keep them in mind for future research.

Thank you again for your time, consideration, and continued support.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.