acceptodds
Under review as a conference paper at ICLR 2027

DOR: Dual On-Policy Self-Distillation for Reflection and Reasoning

Abstract

On-policy self-distillation (OPSD) improves language model reasoning by transferring privileged information from a self-teacher to an unprivileged student. Recent reflection-guided approaches leverage self-generated reflections as privileged information, allowing models to learn from their own reasoning errors. However, these methods primarily optimize the reasoning policy to internalize reflections, while leaving the generation of useful reflections largely unoptimized. We propose DOR, a dual on-policy self-distillation framework that jointly improves reflection generation and reasoning. Our key insight is that self-generated reflections are themselves on-policy sequences and can therefore be directly improved through privileged self-distillation. To assess reflection quality, we compare how reflection conditioning changes the likelihood of the model's own generated response and use its verified correctness to determine whether this change is desirable: a useful reflection should preserve or increase support for a correct response, while reducing support for an incorrect one. The resulting qualitative feedback serves as privileged information for an evaluation-conditioned reflection teacher, whose token-level distribution is distilled into the unprivileged reflector alongside conventional reflection-guided reasoning distillation. Across a suite of reasoning benchmarks, DOR consistently outperforms standard OPSD and reflection-guided OPSD baselines across multiple model scales, demonstrating the benefit of explicitly optimizing both reflection generation and reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, Self-Reflection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rSXnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DOR² jointly trains reasoning and reflection generation through dual on-policy self-distillation, using correctness-aware likelihood changes to provide reflection-quality feedback. Experiments across five tasks and three Qwen3 model sizes show improved average in-domain accuracy, while ablations support the benefits of quality feedback and joint training

Critical strengths
  1. Clear methodological contribution. Explicitly training reflection generation addresses a gap in reflection-guided self-distillation, providing a well-motivated extension.
  2. Practical feedback construction. Correctness-aware likelihood comparisons provide reflection-quality feedback without an additional response rollout, avoiding that sampling cost.
  3. Consistent empirical gains. Average in-domain mean@8 improves across three model sizes, and ablations support the contribution of reflection distillation and informative feedback.
Critical weaknesses
  1. Limited validation of reflection quality. Suppressing an incorrect response does not guarantee better alternatives, and the reported helpful rate reuses the training criterion. This weakens support for the central claim of improved reflection quality.
  2. Insufficient evidence of robustness. Key ablations use only Qwen3-1.7B on the 50-question Biology test set, without repeated-run uncertainty, leaving the stability of the gains unclear.
  3. Limited model coverage. Experiments cover only Qwen3 models with at most 8B parameters, leaving generalization to other model families and larger models untested.
  4. Unclear computational trade-off. Training costs are not reported, leaving it unclear whether the gains justify the additional reflection-distillation computation.
Questions for authors
  1. Improved downstream performance alone does not establish that reflection quality has improved. Can you independently evaluate the accuracy and usefulness of the generated reflections, and provide evidence showing how improvements in reflection quality contribute to the training gains?
  2. Can you report uncertainty estimates for the key ablation gains, given the small, 50-question Biology test set? For example, report the mean and standard deviation of mean@8 across seeds, along with 95% paired bootstrap confidence intervals for DOR²’s improvement over each ablation.
  3. How much additional training time or GPU computation does DOR² require compared with ROSD under the same experimental setup?
8 comments
AuthorsOpenReview ↗

Author response to Reviewer rSXn - Part 1

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the positive assessment of our methodological contribution, efficient reflection-quality evaluation, and empirical improvements. To address the reviewer's concerns regarding reflection utility, statistical robustness, model generalization, and computational efficiency, we conducted additional independent reflection-utility evaluations, three-seed ablation studies with paired bootstrap confidence intervals, experiments on a distinct model family (OLMo-3-7B), and controlled computational-cost comparisons. We present these new results below.

Q1 / W1. Independent Reflection Utility and Its Contribution to Training

We appreciate this important suggestion. We agree that suppressing an incorrect response does not guarantee a better alternative. As clarified in Section 4.1.2, our correctness-aware criterion (Eq. 6) is a task-aligned proxy for reflection utility, rather than a guarantee of accurate error diagnosis. We conducted additional experiments examining (1) the independent utility of learned reflections and (2) their contribution to downstream training improvements.

(1) Evaluation of reflection utility.

We compare reflections generated by the trained ROSD and DOR$^2$ reflectors on matched questions and original responses sampled from the initial Qwen3-1.7B checkpoint. We freeze this initial checkpoint as a common re-solving model and evaluate three conditions: no reflection, ROSD reflection, and DOR$^2$ reflection.

Unlike the training criterion, which measures likelihood changes on the original response, this evaluation measures actual correction success on newly generated responses and gold-answer log-probability after re-reasoning.

Biology (253 originally incorrect responses from 49 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection0.68%—−16.23
ROSD28.36%+27.68 pp [+20.08, +35.54]−12.32
DOR$^2$41.98%+41.30 pp [+32.07, +50.79]−10.86

Material (386 originally incorrect responses from 74 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection2.42%—−14.01
ROSD28.88%+26.46 pp [+18.30, +35.09]−10.37
DOR$^2$32.25%+29.83 pp [+21.59, +38.45]−10.03

Correction success is evaluated on originally incorrect responses, while gold-answer LogP is averaged over scorable re-answers from all original responses. These results demonstrate that DOR$^2$ improves reflection utility beyond merely optimizing the training-time helpfulness criterion. Compared with ROSD, DOR$^2$ improves correction success by 13.62 percentage points on Biology and 3.37 percentage points on Material, alongside higher gold-answer likelihood.

Crucially, the original responses and re-solving model are identical and frozen across conditions; only the supplied reflections differ. Thus, the improvements reflect more useful guidance for actual error correction, rather than changes in the solver's capabilities or better alignment with Eq. 6.

(2) Contribution of reflection optimization to training improvements.

To further distinguish explicit reflection optimization from merely exploiting reflection-quality feedback, we compare DOR$^2$ against Filter-only ROSD, which uses the same reflection-quality signal only to select reflections for reasoning distillation, without directly optimizing the reflector.

DatasetFilter-only ROSD mean@8DOR$^2$ mean@8Improvement
Biology55.75%60.00%+4.25 pp
Material65.82%74.87%+9.05 pp

DOR$^2$ consistently outperforms Filter-only ROSD, supporting the benefit of explicitly distilling reflection-quality feedback into the reflector, beyond using that feedback merely to select reflection guidance.

Together, these experiments provide complementary evidence that DOR$^2$ generates reflections with greater practical correction utility and that explicitly learning to generate useful reflections contributes to improved final reasoning performance. This directly supports our central motivation for jointly optimizing reflection generation and reasoning.

AuthorsOpenReview ↗

Author response to Reviewer rSXn - Part 2

Q2 / W2. Robustness and Statistical Uncertainty of Ablation Results

We thank the reviewer for highlighting the importance of statistical robustness, particularly given the 50-question Biology test set. To address this concern, we repeated the key ablations using three independent training seeds (42, 123, and 2026) and report mean@8 (mean ±\pm sample standard deviation) alongside 95% question-paired bootstrap confidence intervals for DOR$^2$'s improvements.

Additional three-seed results (Qwen3-1.7B, Biology):

Ablation Settingmean@8 (%) ↑\uparrowΔ\Delta vs DOR$^2$ (95% CI), pp
Reflection evaluation and feedback
Full DOR$^2$58.67 ±\pm 1.59—
Single-rewrite verification53.17 ±\pm 2.27+5.50 [−1.25, 12.00]
Numerical logP feedback51.33 ±\pm 2.32+7.34 [−2.83, 17.25]
Reflection-quality labels
50% random labels53.25 ±\pm 3.97+5.42 [−3.75, 14.92]
Always helpful55.33 ±\pm 0.88+3.34 [−6.83, 13.08]
Always unhelpful55.50 ±\pm 4.52+3.17 [−2.58, 9.00]
Reflection-distillation weight
λ\lambda = 056.58 ±\pm 2.32+2.09 [−6.92, 11.00]
λ\lambda = 0.550.58 ±\pm 3.09+8.09 [1.50, 14.50]
λ\lambda = 0.8552.67 ±\pm 3.17+6.00 [−0.58, 12.25]
λ\lambda = 133.42 ±\pm 7.67+25.25 [13.75, 36.67]

Here, Δ\Delta denotes the difference between full DOR$^2$ (λ\lambda = 0.15) and the corresponding ablation. For the paired bootstrap, we first average each question's score across the three seeds, then resample the 50 questions with replacement 10,000 times to construct percentile confidence intervals. The reported standard deviations quantify variation across training seeds, whereas the bootstrap intervals quantify question-level uncertainty conditional on these runs.

The additional experiments provide three key findings.

First, DOR$^2$ achieves the highest three-seed average mean@8 across all nine ablation comparisons, with a relatively modest standard deviation of 1.59 pp, supporting the consistency of its favorable average performance.

Second, alternative reflection evaluations and random or constant quality labels consistently yield lower average mean@8. Although their 95% CIs include zero, the favorable trends across these configurations provide complementary evidence for the importance of correctness-aware, sample-specific reflection feedback.

Third, the reflection-distillation weight ablation provides stronger statistical evidence for balancing reflection and reasoning distillation. DOR$^2$ outperforms λ=0.5\lambda = 0.5 by 8.09 pp (95% CI [1.50, 14.50]) and reflection-only distillation (λ=1\lambda = 1) by 25.25 pp (95% CI [13.75, 36.67]). The improvement over λ=0\lambda = 0 is also positive on average, although its CI includes zero.

Overall, the repeated-run experiments reinforce the favorable average performance of our full configuration while providing a more transparent characterization of uncertainty. We believe these complementary results strengthen the robustness evidence beyond the original single-run experiments and provide a more comprehensive characterization of statistical uncertainty.

AuthorsOpenReview ↗

Author response to Reviewer rSXn - Part 3

W3. Generalization Beyond the Qwen3 Model Family

We appreciate the reviewer's concern regarding model-family coverage. To investigate whether the improvements extend beyond Qwen3, we additionally conducted experiments using OLMo-3-7B, a distinct model family, on Biology and Material.

Additional OLMo-3-7B results (%):

DatasetMethodpass@8 ↑\uparrowmean@8 ↑\uparrowmajority@8 ↑\uparrow
BiologyBase Model32.6911.7514.46
GRPO73.0548.2549.72
SDPO73.1655.8857.02
ROSD56.4351.7552.68
DOR$^2$77.3654.5054.24
MaterialBase Model55.2934.7139.09
GRPO81.6377.1978.13
SDPO81.2376.9377.94
ROSD81.3977.3978.48
DOR$^2$82.5477.6678.79

These results provide encouraging evidence that DOR$^2$'s benefits extend beyond the Qwen3 model family. Notably, DOR$^2$ achieves the highest pass@8 on both datasets. On Biology, it improves pass@8 over ROSD by 20.93 percentage points (56.43% →\rightarrow 77.36%), while also improving mean@8 by 2.75 pp and majority@8 by 1.56 pp. Its pass@8 also exceeds the strongest competing baseline, SDPO, by 4.20 pp. On Material, DOR$^2$ achieves the best performance across all three metrics, outperforming GRPO, SDPO, and ROSD. Although SDPO achieves higher mean@8 and majority@8 on Biology, DOR$^2$ consistently outperforms ROSD across all three metrics on both datasets, indicating that explicit reflection optimization remains beneficial on a different model family.

Together with our original results across Qwen3-1.7B, 4B, and 8B, these additional experiments strengthen the evidence for the broader applicability of dual reflection–reasoning self-distillation. We acknowledge that generalization to substantially larger model scales remains an important direction for future investigation.


Q3 / W4. Computational Overhead and Performance–Cost Trade-off

We conducted controlled reruns of ROSD and DOR$^2$ using eight NVIDIA H100 GPUs, matching training batch sizes, rollout counts, optimization settings, sequence limits, learning rate, sampling configurations, and inference settings. Both methods completed experiments under the same experimental configuration.

Additional compute measurements:

DatasetMethodWall TimeGPU-HoursRelative Overhead
BiologyROSD1h 35m 10s12.69—
DOR$^2$1h 51m 20s14.84+16.9%
MaterialROSD2h 19m 13s18.56—
DOR$^2$2h 53m 40s23.16+24.8%

These measurements show that explicit reflection optimization requires approximately 17-25% additional training computation. For context, our main experiments (Table 1) show improvements over ROSD by 6.00 percentage points on Biology and 5.85 percentage points on Material with Qwen3-1.7B.

Importantly, DOR$^2$ evaluates reflection quality through teacher-forced likelihood comparisons of existing on-policy responses, avoiding additional reflection-conditioned response rollouts during training. Moreover, the additional computation is confined to training: reflection and reasoning guidance are distilled into a single shared model, requiring no additional reflection evaluation or privileged teacher at inference.

We acknowledge that these measurements compare matched experimental configurations without enforcing equal GPU-hour budgets. Nevertheless, they directly quantify the overhead under matched configurations and contextualize the accuracy improvements observed in our main experiments. Overall, DOR$^2$ achieves meaningful reasoning improvements with moderate additional training computation while preserving the inference efficiency of ROSD.


We appreciate the reviewer's feedback and hope the additional experiments and clarifications address the concerns raised.

Reviewer_XPx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DOR², a dual on-policy self-distillation framework that jointly optimizes reasoning and reflection generation. Building on reflection-guided OPSD methods such as ROSD, the paper identifies a limitation of existing approaches: reflections are used as privileged information to supervise the reasoning policy, but the reflector itself is not explicitly trained to produce more useful reflections.

DOR² addresses this by introducing an additional reflection self-distillation objective. It evaluates each generated reflection by comparing the likelihood of the original on-policy response with and without reflection conditioning, using the verified correctness of the response to determine whether the likelihood change is desirable. The resulting binary quality judgment is verbalized and provided as privileged context to a reflection teacher, whose token-level distribution supervises the original reflector through JSD-based distillation. This objective is combined with conventional reflection-guided reasoning distillation.

Experiments on four SciKnowEval domains and ToolAlpaca, using Qwen3-1.7B, 4B, and 8B, show improvements in mean@8 over ROSD across all model-task combinations. The paper also provides ablations on reflection evaluation, quality labels, and the balance between the two distillation objectives.

Critical strengths

S1. Well-motivated extension of reflection-guided self-distillation.

The paper identifies a meaningful limitation in existing reflection-guided OPSD methods: reflection quality directly determines the supervision provided by the privileged reasoning teacher, yet reflection generation receives no dedicated training objective. Treating reflection as another on-policy sequence and introducing a corresponding self-distillation objective is a natural and coherent extension of ROSD. This addresses a relevant problem in improving the reliability of reflection-based supervision.

S2. Practical quality signal and consistent empirical improvements.

The proposed reflection-quality evaluation requires only two teacher-forced likelihood evaluations of an existing response, without an additional reflection-conditioned rollout. This makes the evaluation signal relatively inexpensive to obtain. Empirically, DOR² improves mean@8 over ROSD in every model-task combination reported in Table 1, with particularly notable gains on Qwen3-1.7B. The ablations on quality-label construction and the reflection-distillation weight also provide useful initial evidence for the proposed design.

Critical weaknesses

W1. The reflection-quality criterion may conflate answer information with genuine reflection quality, and the paper lacks independent evidence of improved reflection generation.

The central concern is whether the proposed quality label actually measures useful error diagnosis. According to Appendix F, the reflector has access to the reference solution and the correct answer, and generated reflections can explicitly state the correct option. The quality criterion (Eq. 6) then measures the likelihood shift of the original response under reflection conditioning. For an incorrect response, a reflection that simply states the correct answer can reduce the likelihood of the original incorrect answer, regardless of whether it correctly identifies the reasoning error. Conversely, stating the correct answer may increase the likelihood of a correct response.

Consequently, the binary quality label may largely reflect whether the reflection reveals the gold answer rather than whether it provides an accurate and useful explanation. This concern is reinforced by the case study in Appendix F.3, where the ROSD reflection appears to misdiagnose the error but still identifies the correct option, potentially satisfying the proposed helpfulness criterion.

Moreover, Figure 3 evaluates reflection quality using the same criterion that DOR² is trained to optimize. Thus, a higher helpful rate does not independently demonstrate better reflections. The near-saturation of helpful rates also raises questions about how informative the binary labels remain during training. The paper lacks independent measures of error-localization accuracy or reflection-conditioned correction success, as well as direct baselines that isolate the benefit of training the reflector from simply using the quality signal.

Furthermore, the effectiveness of the reflection-distillation mechanism itself is insufficiently established. The privileged reflection teacher is conditioned on a binary helpful/unhelpful label and evaluated on the same original reflection tokens as the student. It is unclear whether this additional conditioning actually shifts the teacher's token-level distribution toward more accurate error diagnoses, particularly for reflections labeled unhelpful. Since the teacher does not generate an independently verified improved reflection, the paper provides limited evidence that the resulting distillation targets are genuinely better than the student's original predictions.

This is consequential because improving reflection generation is the paper's primary claimed contribution, but the current evidence does not convincingly establish that the learned reflections are better in a task-aligned sense.

W2. The evaluation protocol is insufficiently specified, and several reported metrics are inconsistent with the stated definitions.

Several reported metric values appear inconsistent with the stated evaluation protocol. The paper defines pass@8 as the fraction of questions for which at least one of eight sampled responses is correct. With 50 Biology test questions, a single such evaluation should produce pass@8 values in increments of 2%, yet Table 1 reports values such as 66.39% and 67.75%. Similar inconsistencies appear in other domains and in majority@8. It is unclear whether these numbers involve repeated evaluations, averaging across checkpoints, or a different estimator.

There is also an unexplained discrepancy between the ROSD result in Table 1 (54.00 mean@8 on Qwen3-1.7B Biology) and the λ = 0 ablation in Table 8 (56.25), despite λ = 0 removing the reflection-distillation objective. The paper does not establish whether these configurations are algorithmically identical.

Finally, the paper describes only training and test splits, while evaluation is performed on the test set every 10 training steps. It does not specify how checkpoints or hyperparameters, particularly λ = 0.15, were selected. If test-set performance influenced these decisions, the final reported results may be affected by test-set selection bias.

These issues substantially weaken confidence in the magnitude and reliability of the empirical improvements, and the evaluation protocol needs to be clarified before the main conclusions can be fully supported.

Questions for authors

Q1. Does the proposed reflection-quality criterion measure useful reflection rather than answer leakage?

Since the reflector has access to the reference answer, could the authors report the distribution of helpful/unhelpful labels throughout training, separately for correct and incorrect responses? More importantly, could the authors evaluate an answer-only reflection baseline (e.g., "The correct answer is C") and compare its helpful rate with that of full reflections? Could the authors also recompute the likelihood shift after excluding the final answer span and report the agreement between the original and answer-masked quality labels? These controls would help determine whether the criterion captures meaningful reasoning corrections beyond privileged answer information.

Q2. Is there independent evidence that DOR² produces better reflections?

Figure 3 measures helpfulness using the criterion that DOR² directly optimizes, and the model scale used in that figure is not specified. Could the authors clarify the backbone model and provide an independent reflection-quality evaluation, such as error-localization accuracy or the success rate of reflection-conditioned re-solving? Additionally, under Eq. 6, would the ROSD reflection in Appendix F.3 be classified as helpful despite its incorrect error diagnosis?

Q3. How does reflection self-distillation compare with simpler ways of using the same quality signal?

Could the authors compare DOR² against (i) RL applied directly to the reflector using the binary quality label as reward, (ii) rejection-sampling SFT on helpful reflections, and (iii) ROSD where the quality label is used only to select reflections for reasoning distillation? These comparisons would help isolate the contribution of the proposed reflection-distillation mechanism.

Q4. How exactly are the evaluation metrics computed?

The reported pass@8 and majority@8 values do not appear consistent with the stated test-set sizes and single-evaluation definitions. For example, Biology contains 50 test questions, implying that pass@8 should change in increments of 2% under the described protocol. How are values such as 67.75% obtained? Are the metrics averaged across multiple evaluation runs or checkpoints, or is a different estimator used? Please provide the exact calculation procedures.

Q5. How were checkpoints and hyperparameters selected?

The paper reports evaluation on the test set every 10 training steps but does not describe a separate validation split. Which checkpoint is used for Table 1: the final checkpoint, the best checkpoint according to test performance, or another selection rule? Was λ = 0.15 selected based on the Biology test set? Please clarify how model selection and hyperparameter tuning were conducted without using the final test set.

Q6. How does the λ = 0 configuration in Table 8 relate to the ROSD baseline in Table 1?

In particular, is the λ = 0 configuration in Table 8 algorithmically and experimentally identical to the ROSD baseline in Table 1? If so, why does it achieve 56.25 rather than 54.00 mean@8 on Qwen3-1.7B Biology? If not, please explain the implementation differences.

4 comments
AuthorsOpenReview ↗

Author response to Reviewer XPx3 - Part 1

We sincerely thank the reviewer for the careful and constructive assessment. We appreciate the recognition of our motivation, efficient reflection-quality evaluation, and empirical improvements. To address the concerns regarding reflection-quality validity, the effectiveness of reflection self-distillation, and evaluation reliability, we conducted additional experiments examining (1) potential answer leakage, (2) independent downstream reflection utility, and (3) alternative approaches to leveraging the same reflection-quality feedback. We also provide detailed clarifications of the evaluation metrics, checkpoint and hyperparameter selection, and the relationship between the λ=0\lambda = 0 ablation and ROSD.

Q1 / W1. Does the Reflection-Quality Criterion Measure Useful Reflection Beyond Answer Leakage?

We appreciate this important concern. We agree that Eq. 6 is a task-aligned directional proxy rather than a direct measure of error-diagnosis accuracy: suppressing an incorrect response does not necessarily identify its underlying reasoning error. To investigate whether the criterion captures information beyond revealing the correct answer, we conducted two additional analyses.

(1) Distribution of reflection-quality labels during training.

We examine helpful-label rates separately for originally correct and incorrect responses:

DatasetTraining StepsCorrect Responses: HelpfulIncorrect Responses: Helpful
Biology1–3567.79%100.00%
36–7094.95%100.00%
71–10580.57%99.87%
106–14089.53%100.00%
Overall84.08% (15,744/18,726)99.95% (3,919/3,921)
Material1–6579.33%99.96%
66–13089.61%99.78%
131–19590.96%100.00%
196–26093.12%99.93%
Overall88.56% (37,957/42,860)99.92% (7,221/7,227)

These results reveal an important asymmetry. The criterion is nearly saturated for incorrect responses, providing limited discrimination within this subset. For correct responses, however, it remains more selective, classifying 15.92% and 11.44% of reflections as unhelpful on Biology and Material, respectively. We therefore do not interpret the near-saturated helpful rates as evidence of accurate error diagnosis. Instead, we further examine whether the criterion captures effects beyond answer information.

(2) Answer-only baseline and answer-masked evaluation.

We compare full DOR$^2$ reflections against an answer-only baseline that simply reveals the correct option (e.g., "The correct answer is C"). To isolate the contribution of final-answer likelihood, we recompute Eq. 6 excluding the final-answer span of the original response, while retaining the original reflection conditioning. This evaluates whether the signal also captures changes in the preceding reasoning-token likelihoods.

The evaluation uses matched questions and original responses sampled from the initial Qwen3-1.7B checkpoint.

DatasetFeedbackOriginal Helpful RateAnswer-Masked Helpful RateLabel Agreement
BiologyFull DOR$^2$ reflection65.17%65.17%99.00%
Answer-only67.00%59.00%90.50%

Under the original criterion, answer-only feedback achieves a slightly higher helpful rate (67.00% vs. 65.17%), confirming the reviewer's concern that answer information can influence the quality label.

However, excluding the final-answer span reverses this ordering: the answer-only helpful rate decreases to 59.00%, whereas the full-reflection rate remains at 65.17%. Full reflections therefore achieve a 6.17-percentage-point advantage in answer-masked helpfulness (95% question-paired CI [+0.17, +12.67]). Moreover, full-reflection labels exhibit substantially higher agreement before and after masking (99.00% vs. 90.50%).

These findings suggest that full reflections induce more stable changes in reasoning-token likelihoods beyond their influence on the final answer, whereas answer-only feedback is more sensitive to the final-answer span. Although answer masking does not establish accurate error diagnosis, it provides evidence that the reflection-quality signal is not entirely explained by answer leakage.

We further examine whether these reflections genuinely improve downstream correction success and gold-answer likelihood through an independent frozen-solver evaluation in Q2.


AuthorsOpenReview ↗

Author response to Reviewer XPx3 - Part 2

Q2 / W1. Independent Evidence of Improved Reflection Quality

Thank you for this suggestion. We clarify that Fig. 3 uses Qwen3-1.7B and reports per-step helpful rates under the training criterion, together with trailing 10-step averages. We agree that this metric alone cannot establish whether learned reflections genuinely improve subsequent reasoning.

We therefore conducted an independent evaluation using reflection-conditioned correction success and gold-answer log-likelihood, neither of which is directly used as the training-time helpfulness label.

Experimental setup. We use matched questions and original responses generated by the initial Qwen3-1.7B checkpoint and freeze this checkpoint as a common re-solving model. We compare no reflection, ROSD-generated reflections, and DOR$^2$-generated reflections. Each condition produces one new answer, and we additionally measure the gold-answer log-probability conditioned on the newly generated reasoning.

Biology (253 originally incorrect responses from 49 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection0.68%—−16.23
ROSD28.36%+27.68 pp [+20.08, +35.54]−12.32
DOR$^2$41.98%+41.30 pp [+32.07, +50.79]−10.86

Material (386 originally incorrect responses from 74 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection2.42%—−14.01
ROSD28.88%+26.46 pp [+18.30, +35.09]−10.37
DOR$^2$32.25%+29.83 pp [+21.59, +38.45]−10.03

Correction success is measured on originally incorrect responses, while gold-answer LogP is averaged over scorable re-answers from all original responses.

DOR$^2$ improves correction success over ROSD by 13.62 percentage points on Biology and 3.37 points on Material, accompanied by consistently higher gold-answer likelihood. Crucially, the original responses and frozen solver are identical across conditions; only the supplied reflections differ. Unlike Eq. 6, these experiments measure actual correction outcomes on newly generated responses, providing direct evidence that DOR$^2$ learns reflections that are more useful for subsequent problem solving, independently of the helpfulness criterion used during training.

Regarding Appendix F.3: The original ROSD response incorrectly selects A (gold answer C), and its reflection misdiagnoses the reasoning error while identifying the correct option. Nevertheless, reflection conditioning reduces the original response's log-likelihood from −29.19 to −105.20 (Δ ≈ −76.01), causing Eq. 6 to classify it as helpful.

We acknowledge that our criterion measures a correctness-aligned likelihood shift rather than verified error-diagnosis accuracy. However, the independent re-solving experiments demonstrate that optimizing this proxy produces reflections with greater practical correction utility on average. We will clarify this distinction and avoid equating helpfulness labels with accurate error diagnosis.

AuthorsOpenReview ↗

Author response to Reviewer XPx3 - Part 3

Q3 / W1. Effectiveness of Reflection Self-Distillation Compared with Alternatives

We appreciate this suggestion. To isolate the contribution of our reflection-distillation mechanism, we implemented the three requested baselines, all using the same reflection-quality signal:

  • Binary RL: Directly optimizes the reflector using the binary quality label as a reward.
  • Helpful SFT: Applies rejection-sampling SFT to reflections labeled helpful.
  • Filter-only ROSD: Uses the quality label only to select reflections for reasoning distillation, without explicitly optimizing the reflector.

Additional results (Qwen3-1.7B):

DatasetMethodmean@8 ↑\uparrowGPU-Hours ↓\downarrow
BiologyBinary RL38.2535.59
Helpful SFT46.0018.25
Filter-only ROSD55.7510.64
DOR$^2$60.0014.84
MaterialBinary RL69.2844.64
Helpful SFT58.7820.23
Filter-only ROSD65.8218.51
DOR$^2$74.8723.16

DOR$^2$ achieves the highest reasoning accuracy on both datasets, outperforming direct binary-reward RL, rejection-sampling SFT, and reflection filtering.

Importantly, the comparison with Filter-only ROSD directly examines whether the quality signal should merely select reflections or also supervise their generation. DOR$^2$ improves mean@8 by 4.25 percentage points on Biology and 9.05 percentage points on Material, supporting the additional benefit of quality-conditioned reflection self-distillation.

Furthermore, Binary RL requires substantially greater GPU computation while achieving lower accuracy on both datasets. Although Filter-only ROSD is computationally cheaper, DOR$^2$ provides consistently higher reasoning performance through explicit reflection optimization.

Together, these comparisons demonstrate that the benefits of DOR$^2$ cannot be explained solely by access to the reflection-quality signal. Instead, they provide direct empirical support for our central contribution: transferring reflection-quality feedback into the reflector through privileged token-level self-distillation, rather than relying only on scalar rewards, supervised filtering, or reflection selection.


Q4 / W2. Clarification of Evaluation Metrics

We thank the reviewer for identifying this ambiguity. We apologize that the manuscript did not sufficiently clarify our evaluation procedure. The reported pass@8 and majority@8 values use bootstrap resampling of the eight generated responses per question, rather than a single binary evaluation. We clarify the computation and motivation below.

For each evaluation question, we independently generate eight responses and compute three metrics:

  1. mean@8: Average correctness of the eight generated responses, subsequently averaged over questions.
  2. Bootstrap pass@8: Resample eight responses with replacement from the original eight, repeat 1,000 times, and average the indicator that at least one response is correct across resamples and questions.
  3. Bootstrap majority@8: Using the same resampling procedure, evaluate the correctness of the majority-voted answer in each resample (ties resolved by the first encountered answer), then average across resamples and questions.

Conventional pass@8, when evaluated directly from exactly eight generated responses, assigns each question a binary score depending on whether at least one response is correct. We adopted bootstrap resampling to obtain a smoother empirical measure of multi-sample success that also reflects the observed frequency of correct responses, rather than relying on a single binary outcome.

Specifically, given cc correct responses among the original eight, the expected bootstrap pass@8 is pass@8^boot=1−(1−c8)8\widehat{\mathrm{pass@8}}_{\mathrm{boot}}=1-\left(1-\frac{c}{8}\right)^8. For example, when only one of eight original responses is correct, conventional observed pass@8 assigns 100%, whereas bootstrap pass@8 assigns approximately 65.64%. Consequently, the bootstrap statistic is not restricted to increments of 1/N1/N, where NN is the number of evaluation questions. Additionally, the Biology evaluation log records: Bootstrap pass@8: 0.67748 →\rightarrow 67.75%; mean@8: 0.60000 →\rightarrow 60.00%; Bootstrap majority@8: 0.59706 →\rightarrow 59.71%. These values reproduce the corresponding entries in our tables.

We acknowledge that bootstrap pass@8 is a plug-in estimate and differs from the conventional unbiased pass@k estimator. We will revise the manuscript to explicitly document the resampling procedure, distinguish the bootstrap metrics from their conventional counterparts, and use consistent terminology throughout. Importantly, mean@8, our primary evaluation metric, is computed directly from the original responses and is unaffected by this clarification.

AuthorsOpenReview ↗

Author response to Reviewer XPx3 - Part 4

Q5 / W2. How were checkpoints and hyperparameters selected?

We thank the reviewer for highlighting the importance of a transparent model-selection protocol. We clarify our procedures as follows.

Checkpoint selection. We evaluate the model every 10 training steps on the corresponding benchmark test set and report the checkpoint with the highest observed mean@8 in Table 1.

Hyperparameter selection. We selected λ=0.15\lambda=0.15 based on aggregate performance across the five evaluated domains, rather than being tuned independently for Biology or individual tasks. The same value was subsequently applied uniformly across datasets and model scales.

We acknowledge that the benchmark test sets were used for both checkpoint selection and hyperparameter tuning, which may introduce selection bias. Consequently, the reported results should be interpreted as test-selected performance rather than unbiased held-out estimates. We will explicitly clarify this protocol and its limitations in the revised manuscript.

Importantly, our additional experiments in Q1-Q3 provide complementary evidence for the proposed mechanism, including answer-masked reflection evaluation, independent frozen-solver correction success, and comparisons with alternative uses of the same quality signal. While these results strengthen the empirical support for reflection self-distillation, we recognize that they do not eliminate the potential bias associated with the original model-selection protocol.

We appreciate the reviewer's observation and will revise the manuscript to ensure that the experimental procedure and interpretation of the results are clearly documented.


Q6 / W2. Relationship Between λ=0\lambda = 0 and ROSD

We thank the reviewer for identifying this inconsistency and apologize for the lack of clarification.

Setting λ=0\lambda = 0 in Eq. 10 removes reflection self-distillation, leaving the same reflection-guided reasoning-distillation objective as ROSD. The λ=0\lambda = 0 ablation in Table 8 and the ROSD baseline in Table 1 are therefore algorithmically equivalent. However, the two reported results were obtained from independent training runs, rather than the same trained checkpoint. Consequently, their mean@8 values differ. We acknowledge that we should have explicitly distinguished these independent experimental realizations in the manuscript.

Importantly, our additional three-seed experiments (Reviewer rSXn, Q2 / W2) provide empirical evidence of run-to-run variability: λ=0\lambda = 0 achieves 56.58 ±\pm 2.32% mean@8 across three independent seeds. The original 2.25-percentage-point discrepancy is therefore plausible given the observed training variability, although this does not establish its exact source.

We will clarify the equivalence of the two configurations and explicitly document the independent-run provenance of the reported results, avoiding ambiguity between an algorithmic difference and stochastic variation across experiments.


We hope these additional experiments and clarifications strengthen the empirical evidence for DOR$^2$ while improving the rigor and transparency of our evaluation.

Reviewer_zGxJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+2)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DOR^2, a dual on-policy self-distillation framework that jointly optimizes reasoning and reflection generation. It evaluates reflection helpfulness based on correctness-aware likelihood changes and uses the resulting feedback to guide reflection self-distillation. Experiments show consistent improvements over existing baselines across multiple reasoning tasks and model scales.

Critical strengths
  • The proposed framework, DOR^2, achieves consistent improvements in mean@8 over strong baselines across five tasks and three model scales.

  • The proposed reflection evaluation is practical, requiring no additional reflection-conditioned rollouts.

Critical weaknesses
  • The proposed framework appears to be an extension of existing reflection-guided on-policy self-distillation methods, particularly ROSD, by applying an additional self-distillation objective to reflection generation. The overall framework essentially combines standard on-policy self-distillation with a heuristic reflection-quality feedback mechanism. While jointly optimizing reasoning and reflection generation is a reasonable extension, the methodological novelty beyond existing approaches seems relatively limited.

  • The proposed reflection distillation assumes that helpfulness feedback provides better supervision, but it is unclear whether an "Unhelpful" label actually shifts the teacher's distribution toward more useful reflections. Moreover, the improved helpful reflection rate (Fig. 3) is evaluated using the same criterion as the training feedback, making it unclear whether reflection quality genuinely improves or the model simply aligns with this metric. An independent evaluation, such as correction accuracy or correct-answer likelihood, would provide stronger evidence.

Questions for authors
  • An independent evaluation of reflection quality, such as correction accuracy or correct-answer likelihood, would be helpful.

  • It would be useful to evaluate whether the privileged reflection teacher generates better reflections than the unprivileged reflector.

  • A comparison with baselines under an equal training compute budget would help clarify the efficiency of DOR^2.

4 comments
AuthorsOpenReview ↗

Author response to Reviewer zGxJ - Part 1

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our empirical improvements and the opportunity to clarify the methodological contributions of DOR$^2$. To address these concerns, we conducted additional experiments evaluating (1) the independent utility of learned reflections, (2) the effectiveness of quality-conditioned privileged reflection supervision, and (3) the computational overhead relative to ROSD. We present these results and clarify the methodological novelty of DOR$^2$ below.

W1. Methodological Novelty Beyond ROSD

We agree that DOR$^2$ builds upon reflection-guided OPSD, particularly ROSD. However, we would like to clarify that our contribution goes beyond introducing an auxiliary self-distillation objective: while ROSD treats reflection as privileged supervision, DOR$^2$ makes the generation of that privileged supervision itself an explicit learning target.

In existing reflection-guided OPSD methods, reflections guide the reasoning policy, but their generation receives no dedicated quality-aware supervision. This creates a fundamental asymmetry: the quality of reflections directly determines the supervision provided by the privileged reasoning teacher, yet the reflector itself is not explicitly optimized for producing useful guidance. DOR$^2$ addresses this limitation through three interconnected contributions:

  1. Reflection generation as an on-policy self-distillation problem. Our key insight is that self-generated reflections are themselves on-policy sequences and can therefore be explicitly optimized through privileged self-distillation. This introduces a second teacher-student relationship over reflection generation, rather than treating reflections solely as a source of supervision for reasoning.

  2. Task-aligned reflection evaluation. A large teacher–student distributional shift does not necessarily indicate useful reflection. We formalize this distinction through rationalization gain (Eqs. 2–3) and develop a correctness-aware likelihood criterion (Eq. 6) that evaluates whether reflection conditioning changes the likelihood of the original response in the direction indicated by its verified correctness. Importantly, this provides reflection-quality feedback through teacher-forced likelihood comparisons without additional reflection-conditioned response rollouts.

  3. Quality-conditioned reflection self-distillation. Rather than using reflection-quality feedback merely as a scalar reward or a reflection-selection criterion, we incorporate it as privileged information for a reflection teacher. Its token-level distribution is distilled into the unprivileged reflector along the original on-policy reflection prefixes (Eq. 8), enabling the model to internalize reflection-quality feedback without generating improved teacher reflections. Combined with reasoning distillation, this explicitly optimizes both reflection generation and its utilization within a shared model.

Importantly, our experimental evidence demonstrates that the improvements cannot be explained merely by adding another objective or using a quality signal. Replacing the quality-aware labels with random, always-helpful, or always-unhelpful labels reduces Biology mean@8 from 60.00% to 54.75%, 53.25%, and 53.00%, respectively (Table 3). Furthermore, in our additional rebuttal experiments (see our response to Reviewer XPx3, Q3), we introduce a Filter-only ROSD baseline that uses the same quality signal solely to select reflections for reasoning distillation. It achieves 55.75% and 65.82% mean@8 on Biology and Material, compared with 60.00% and 74.87% for DOR$^2$, respectively. These comparisons support the value of explicitly learning reflection generation through quality-conditioned self-distillation, beyond simply adding auxiliary training or filtering reflection guidance.

In essence, DOR$^2$ moves beyond treating reflection as a source of privileged supervision and explicitly optimizes how that supervision is generated and subsequently utilized. We believe this shift in the learning formulation, together with our task-aligned evaluation and quality-conditioned distillation mechanism, constitutes a meaningful methodological contribution beyond simply adding an auxiliary loss to ROSD.

AuthorsOpenReview ↗

Author response to Reviewer zGxJ - Part 2

Q1 / W2. Independent Evaluation of Reflection Quality

We appreciate this important suggestion. We agree that the helpful reflection rate in Fig. 3, being measured using the training criterion, does not by itself establish whether the learned reflections genuinely improve downstream reasoning. To address this concern, we conducted an independent evaluation of reflection utility through actual correction success and gold-answer likelihood, rather than relying on the training-time helpfulness metric.

Experimental setup. We use matched questions and original responses generated by the Qwen3-1.7B base checkpoint before task training. We freeze this initial checkpoint as a common re-solving model and evaluate three conditions: no reflection, ROSD-generated reflections, and DOR$^2$-generated reflections. Each condition produces a new answer, and we additionally measure the gold-answer log-probability conditioned on the newly generated reasoning.

Biology (253 originally incorrect responses from 49 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection0.68%—−16.23
ROSD28.36%+27.68 pp [+20.08, +35.54]−12.32
DOR$^2$41.98%+41.30 pp [+32.07, +50.79]−10.86

Material (386 originally incorrect responses from 74 questions):

ReflectionCorrection Success ↑\uparrowGain vs. No Reflection (95% CI)Gold-Answer LogP ↑\uparrow
No reflection2.42%—−14.01
ROSD28.88%+26.46 pp [+18.30, +35.09]−10.37
DOR$^2$32.25%+29.83 pp [+21.59, +38.45]−10.03

Correction success is evaluated on originally incorrect responses, while gold-answer LogP is averaged over scorable re-answers from all original responses.

The results provide direct evidence that DOR$^2$ improves reflection utility beyond the training feedback criterion. Compared with ROSD, DOR$^2$ increases correction success by 13.62 percentage points on Biology (28.36% →\rightarrow 41.98%) and 3.37 percentage points on Material (28.88% →\rightarrow 32.25%), while consistently improving the gold-answer log-likelihood.

Crucially, these results are not obtained by reapplying Eq. 6 to the original responses. Instead, they evaluate whether the reflections help the solver generate correct new responses. Since the questions, original responses, and frozen re-solving model are identical across conditions, the observed differences can be attributed to the supplied reflections rather than changes in the solver's capabilities.

Together, the two complementary metrics demonstrate that DOR$^2$-generated reflections not only improve the training-time helpfulness measure but also provide more effective guidance for actual error correction and increase the solver's conditional likelihood of the correct answer after regeneration. This provides empirical support for the practical utility of explicitly optimizing reflection generation.

AuthorsOpenReview ↗

Author response to Reviewer zGxJ - Part 3

Q2 / W2. Effectiveness of Quality-Conditioned Reflection Supervision

Thank you for raising this concern. We agree that an "Unhelpful" label does not theoretically guarantee that the teacher will produce better reflections in every instance. Our approach instead uses this label as post-hoc privileged information that conditions the teacher's token-level distribution, allowing reflection-quality feedback to be transferred to the unprivileged reflector.

To directly examine whether privileged conditioning produces more useful reflections, we conducted an additional experiment comparing newly generated reflections from the privileged teacher and the unprivileged reflector, evaluated using the same frozen Qwen3-1.7B re-solving model.

The privileged teacher receives the original solution, error information, reference answer, and reflection-quality feedback. We evaluate reflection utility through downstream correction success and gold-answer log-likelihood.

Additional results:

DatasetReflection SourceCorrection Success ↑\uparrowGold-Answer LogP ↑\uparrow
BiologyUnprivileged reflector17.74%−12.09
Privileged teacher61.31%−6.78
MaterialUnprivileged reflector18.67%−10.14
Privileged teacher71.29%−2.21

The privileged teacher achieves substantially higher correction success, improving over the unprivileged reflector by 43.57 percentage points on Biology and 52.62 percentage points on Material. The corresponding improvements in gold-answer log-likelihood further confirm that its reflections provide more useful guidance for subsequent reasoning. These results establish that the privileged reflection teacher provides a stronger source of reflection supervision, rather than merely inducing an arbitrary distributional shift.

Moreover, the quality-label ablations in Table 3 provide complementary evidence: replacing sample-specific feedback with random or constant labels substantially degrades final reasoning performance. This supports the importance of informative reflection-quality feedback, rather than the label's mere presence.

We emphasize that our method does not require explicitly sampling an improved teacher reflection during training. Instead, the privileged teacher provides dense token-level supervision on the original on-policy reflection trajectory (Eq. 8). The newly generated teacher reflections in this experiment serve specifically to validate the utility of the privileged teacher.


Q3. Computational Efficiency Relative to ROSD

We appreciate the suggestion to clarify the computational trade-off. We conducted additional controlled reruns of ROSD and DOR$^2$ on eight NVIDIA H100 GPUs, matching training batch sizes, rollout counts, optimization settings, sequence limits, sampling configurations, and inference settings.

Additional results:

DatasetMethodTraining TimeGPU-HoursRelative Overhead
BiologyROSD1h 35m 10s12.69—
DOR$^2$1h 51m 20s14.84+16.9%
MaterialROSD2h 19m 13s18.56—
DOR$^2$2h 53m 40s23.16+24.8%

These results show that DOR$^2$ introduces approximately 17-25% additional training computation, corresponding to 2.15 and 4.60 extra GPU-hours on Biology and Material, respectively.

For context, our main experiments (Table 1) show that DOR$^2$ improves mean@8 over ROSD by 6.00 percentage points on Biology and 5.85 percentage points on Material with Qwen3-1.7B. Importantly, DOR$^2$ evaluates reflection quality through teacher-forced likelihood comparisons of existing on-policy responses, avoiding additional reflection-conditioned response rollouts during training. Consequently, DOR$^2$ provides meaningful reasoning improvements with a relatively moderate increase in training computation.

We acknowledge that matching experimental configurations may not ensure equal GPU-hour budgets. Nevertheless, these controlled measurements quantify the additional computational cost and help contextualize the observed performance improvements.


We hope these clarifications and additional experimental results adequately address the reviewer's concerns.

I thank the authors for their extensive and thoughtful rebuttal.

  • [W1] Methodological Novelty: Thank you for the clarification. I acknowledge the methodological contribution of explicitly optimizing reflection generation through a dedicated self-distillation objective, beyond using reflections solely as privileged supervision for reasoning.

  • [Q1/W2] Independent Evaluation of Reflection Quality: I appreciate the additional experiments comparing downstream correction success and gold-answer likelihood, and the revision of the manuscript as well. These additional experiments provide convincing evidence of improved reflection utility beyond the training-time helpfulness metric. I believe this analysis would be a valuable replacement of the current Fig. 3.

  • [Q2/W2] Effectiveness of Quality-Conditioned Reflection Supervision: Thank you for the extensive additional analysis. However, Appendix D.4.4 reveals a substantial imbalance in the quality labels, with almost 99.9% of reflections for incorrect responses classified as helpful. This raises concerns about the discriminative power of the proposed criterion, making it difficult to conclude that the quality-conditioned supervision reliably distinguishes useful from unhelpful reflections.

  • [Q3] Computational Efficiency: Thank you for the additional computational cost comparison. The reported overhead appears reasonable given the performance gains. However, I would still appreciate an equal compute comparison, as originally suggested. Moreover, the performance gains cited from Table 1 are based on test-set-selected checkpoints, which makes it difficult to assess the reliability of the reported improvements and the resulting accuracy-compute trade-off.

Based on the rebuttal results, I will raise my Informativeness score from 1 to 3 while keeping my overall rating unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We sincerely thank the reviewers for their constructive feedback. We have uploaded a revised manuscript incorporating the additional experiments, methodological clarifications, and analyses presented in our rebuttal. The main revisions are summarized below, with corresponding locations for each reviewer.

Reviewer zGxJ

  • W1 (Section 1, Paragraph 3): Strengthened the conceptual motivation and clarified the methodological novelty of DOR² beyond ROSD.
  • Q1/W2 (Section 5.3): Added an independent evaluation of learned reflection utility using downstream correction success and gold-answer likelihood.
  • Q2/W2 (Appendix D.4.5): Added experiments validating the utility of privileged reflection supervision.
  • Q3 (Appendix D.4.3): Added controlled computational-overhead comparisons with ROSD.

Reviewer XPx3

  • Q1/W1 (Appendix D.4.4): Added analyses of reflection-quality label distributions, answer-only feedback, and answer-masked evaluation.
  • Q2/W1 (Section 5.3; Appendix F.6): Added independent reflection-utility results and clarified the limitations of the reflection-quality criterion through case analysis.
  • Q3/W1 (Section 5.4): Added comparisons with Binary RL, Helpful SFT, and Filter-only ROSD to examine alternative uses of reflection-quality feedback.
  • Q4/W2 (Section 5.1): Clarified the evaluation metrics and bootstrap-resampling procedure.
  • Q5/W2 (Appendix D.1): Documented checkpoint and hyperparameter selection procedures and acknowledged potential selection bias.
  • Q6/W2 (Appendix D.3.3): Clarified the relationship between the λ = 0 ablation and ROSD, including independent-run variability.

Reviewer rSXn

  • Q1/W1 (Section 5.3): Added independent reflection-utility evaluation to validate the practical benefits of learned reflections.
  • Q2/W2 (Appendix D.4.2): Added three-seed ablation experiments with standard deviations and paired bootstrap confidence intervals.
  • W3 (Appendix D.4.1): Added experiments with OLMo-3-7B to examine generalization beyond the Qwen3 model family.
  • Q3/W4 (Appendix D.4.3): Added controlled computational-cost measurements and clarified the performance–cost trade-off.

We believe these revisions strengthen the empirical evidence for quality-conditioned reflection self-distillation while improving the rigor, transparency, and reproducibility of our evaluation. We sincerely appreciate the reviewers' constructive suggestions, which have helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.