- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DOR² jointly trains reasoning and reflection generation through dual on-policy self-distillation, using correctness-aware likelihood changes to provide reflection-quality feedback. Experiments across five tasks and three Qwen3 model sizes show improved average in-domain accuracy, while ablations support the benefits of quality feedback and joint training
Critical strengths
- Clear methodological contribution. Explicitly training reflection generation addresses a gap in reflection-guided self-distillation, providing a well-motivated extension.
- Practical feedback construction. Correctness-aware likelihood comparisons provide reflection-quality feedback without an additional response rollout, avoiding that sampling cost.
- Consistent empirical gains. Average in-domain mean@8 improves across three model sizes, and ablations support the contribution of reflection distillation and informative feedback.
Critical weaknesses
- Limited validation of reflection quality. Suppressing an incorrect response does not guarantee better alternatives, and the reported helpful rate reuses the training criterion. This weakens support for the central claim of improved reflection quality.
- Insufficient evidence of robustness. Key ablations use only Qwen3-1.7B on the 50-question Biology test set, without repeated-run uncertainty, leaving the stability of the gains unclear.
- Limited model coverage. Experiments cover only Qwen3 models with at most 8B parameters, leaving generalization to other model families and larger models untested.
- Unclear computational trade-off. Training costs are not reported, leaving it unclear whether the gains justify the additional reflection-distillation computation.
Questions for authors
- Improved downstream performance alone does not establish that reflection quality has improved. Can you independently evaluate the accuracy and usefulness of the generated reflections, and provide evidence showing how improvements in reflection quality contribute to the training gains?
- Can you report uncertainty estimates for the key ablation gains, given the small, 50-question Biology test set? For example, report the mean and standard deviation of mean@8 across seeds, along with 95% paired bootstrap confidence intervals for DOR²’s improvement over each ablation.
- How much additional training time or GPU computation does DOR² require compared with ROSD under the same experimental setup?
8 comments
Author response to Reviewer rSXn - Part 2
Q2 / W2. Robustness and Statistical Uncertainty of Ablation Results
We thank the reviewer for highlighting the importance of statistical robustness, particularly given the 50-question Biology test set. To address this concern, we repeated the key ablations using three independent training seeds (42, 123, and 2026) and report mean@8 (mean sample standard deviation) alongside 95% question-paired bootstrap confidence intervals for DOR$^2$'s improvements.
Additional three-seed results (Qwen3-1.7B, Biology):
| Ablation Setting | mean@8 (%) | vs DOR$^2$ (95% CI), pp |
|---|---|---|
| Reflection evaluation and feedback | ||
| Full DOR$^2$ | 58.67 1.59 | — |
| Single-rewrite verification | 53.17 2.27 | +5.50 [−1.25, 12.00] |
| Numerical logP feedback | 51.33 2.32 | +7.34 [−2.83, 17.25] |
| Reflection-quality labels | ||
| 50% random labels | 53.25 3.97 | +5.42 [−3.75, 14.92] |
| Always helpful | 55.33 0.88 | +3.34 [−6.83, 13.08] |
| Always unhelpful | 55.50 4.52 | +3.17 [−2.58, 9.00] |
| Reflection-distillation weight | ||
| = 0 | 56.58 2.32 | +2.09 [−6.92, 11.00] |
| = 0.5 | 50.58 3.09 | +8.09 [1.50, 14.50] |
| = 0.85 | 52.67 3.17 | +6.00 [−0.58, 12.25] |
| = 1 | 33.42 7.67 | +25.25 [13.75, 36.67] |
Here, denotes the difference between full DOR$^2$ ( = 0.15) and the corresponding ablation. For the paired bootstrap, we first average each question's score across the three seeds, then resample the 50 questions with replacement 10,000 times to construct percentile confidence intervals. The reported standard deviations quantify variation across training seeds, whereas the bootstrap intervals quantify question-level uncertainty conditional on these runs.
The additional experiments provide three key findings.
First, DOR$^2$ achieves the highest three-seed average mean@8 across all nine ablation comparisons, with a relatively modest standard deviation of 1.59 pp, supporting the consistency of its favorable average performance.
Second, alternative reflection evaluations and random or constant quality labels consistently yield lower average mean@8. Although their 95% CIs include zero, the favorable trends across these configurations provide complementary evidence for the importance of correctness-aware, sample-specific reflection feedback.
Third, the reflection-distillation weight ablation provides stronger statistical evidence for balancing reflection and reasoning distillation. DOR$^2$ outperforms by 8.09 pp (95% CI [1.50, 14.50]) and reflection-only distillation () by 25.25 pp (95% CI [13.75, 36.67]). The improvement over is also positive on average, although its CI includes zero.
Overall, the repeated-run experiments reinforce the favorable average performance of our full configuration while providing a more transparent characterization of uncertainty. We believe these complementary results strengthen the robustness evidence beyond the original single-run experiments and provide a more comprehensive characterization of statistical uncertainty.
Author response to Reviewer rSXn - Part 3
W3. Generalization Beyond the Qwen3 Model Family
We appreciate the reviewer's concern regarding model-family coverage. To investigate whether the improvements extend beyond Qwen3, we additionally conducted experiments using OLMo-3-7B, a distinct model family, on Biology and Material.
Additional OLMo-3-7B results (%):
| Dataset | Method | pass@8 | mean@8 | majority@8 |
|---|---|---|---|---|
| Biology | Base Model | 32.69 | 11.75 | 14.46 |
| GRPO | 73.05 | 48.25 | 49.72 | |
| SDPO | 73.16 | 55.88 | 57.02 | |
| ROSD | 56.43 | 51.75 | 52.68 | |
| DOR$^2$ | 77.36 | 54.50 | 54.24 | |
| Material | Base Model | 55.29 | 34.71 | 39.09 |
| GRPO | 81.63 | 77.19 | 78.13 | |
| SDPO | 81.23 | 76.93 | 77.94 | |
| ROSD | 81.39 | 77.39 | 78.48 | |
| DOR$^2$ | 82.54 | 77.66 | 78.79 |
These results provide encouraging evidence that DOR$^2$'s benefits extend beyond the Qwen3 model family. Notably, DOR$^2$ achieves the highest pass@8 on both datasets. On Biology, it improves pass@8 over ROSD by 20.93 percentage points (56.43% 77.36%), while also improving mean@8 by 2.75 pp and majority@8 by 1.56 pp. Its pass@8 also exceeds the strongest competing baseline, SDPO, by 4.20 pp. On Material, DOR$^2$ achieves the best performance across all three metrics, outperforming GRPO, SDPO, and ROSD. Although SDPO achieves higher mean@8 and majority@8 on Biology, DOR$^2$ consistently outperforms ROSD across all three metrics on both datasets, indicating that explicit reflection optimization remains beneficial on a different model family.
Together with our original results across Qwen3-1.7B, 4B, and 8B, these additional experiments strengthen the evidence for the broader applicability of dual reflection–reasoning self-distillation. We acknowledge that generalization to substantially larger model scales remains an important direction for future investigation.
Q3 / W4. Computational Overhead and Performance–Cost Trade-off
We conducted controlled reruns of ROSD and DOR$^2$ using eight NVIDIA H100 GPUs, matching training batch sizes, rollout counts, optimization settings, sequence limits, learning rate, sampling configurations, and inference settings. Both methods completed experiments under the same experimental configuration.
Additional compute measurements:
| Dataset | Method | Wall Time | GPU-Hours | Relative Overhead |
|---|---|---|---|---|
| Biology | ROSD | 1h 35m 10s | 12.69 | — |
| DOR$^2$ | 1h 51m 20s | 14.84 | +16.9% | |
| Material | ROSD | 2h 19m 13s | 18.56 | — |
| DOR$^2$ | 2h 53m 40s | 23.16 | +24.8% |
These measurements show that explicit reflection optimization requires approximately 17-25% additional training computation. For context, our main experiments (Table 1) show improvements over ROSD by 6.00 percentage points on Biology and 5.85 percentage points on Material with Qwen3-1.7B.
Importantly, DOR$^2$ evaluates reflection quality through teacher-forced likelihood comparisons of existing on-policy responses, avoiding additional reflection-conditioned response rollouts during training. Moreover, the additional computation is confined to training: reflection and reasoning guidance are distilled into a single shared model, requiring no additional reflection evaluation or privileged teacher at inference.
We acknowledge that these measurements compare matched experimental configurations without enforcing equal GPU-hour budgets. Nevertheless, they directly quantify the overhead under matched configurations and contextualize the accuracy improvements observed in our main experiments. Overall, DOR$^2$ achieves meaningful reasoning improvements with moderate additional training computation while preserving the inference efficiency of ROSD.
We appreciate the reviewer's feedback and hope the additional experiments and clarifications address the concerns raised.
Author response to Reviewer rSXn - Part 1
We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the positive assessment of our methodological contribution, efficient reflection-quality evaluation, and empirical improvements. To address the reviewer's concerns regarding reflection utility, statistical robustness, model generalization, and computational efficiency, we conducted additional independent reflection-utility evaluations, three-seed ablation studies with paired bootstrap confidence intervals, experiments on a distinct model family (OLMo-3-7B), and controlled computational-cost comparisons. We present these new results below.
We appreciate this important suggestion. We agree that suppressing an incorrect response does not guarantee a better alternative. As clarified in Section 4.1.2, our correctness-aware criterion (Eq. 6) is a task-aligned proxy for reflection utility, rather than a guarantee of accurate error diagnosis. We conducted additional experiments examining (1) the independent utility of learned reflections and (2) their contribution to downstream training improvements.
(1) Evaluation of reflection utility.
We compare reflections generated by the trained ROSD and DOR$^2$ reflectors on matched questions and original responses sampled from the initial Qwen3-1.7B checkpoint. We freeze this initial checkpoint as a common re-solving model and evaluate three conditions: no reflection, ROSD reflection, and DOR$^2$ reflection.
Unlike the training criterion, which measures likelihood changes on the original response, this evaluation measures actual correction success on newly generated responses and gold-answer log-probability after re-reasoning.
Biology (253 originally incorrect responses from 49 questions):
Material (386 originally incorrect responses from 74 questions):
Correction success is evaluated on originally incorrect responses, while gold-answer LogP is averaged over scorable re-answers from all original responses. These results demonstrate that DOR$^2$ improves reflection utility beyond merely optimizing the training-time helpfulness criterion. Compared with ROSD, DOR$^2$ improves correction success by 13.62 percentage points on Biology and 3.37 percentage points on Material, alongside higher gold-answer likelihood.
Crucially, the original responses and re-solving model are identical and frozen across conditions; only the supplied reflections differ. Thus, the improvements reflect more useful guidance for actual error correction, rather than changes in the solver's capabilities or better alignment with Eq. 6.
(2) Contribution of reflection optimization to training improvements.
To further distinguish explicit reflection optimization from merely exploiting reflection-quality feedback, we compare DOR$^2$ against Filter-only ROSD, which uses the same reflection-quality signal only to select reflections for reasoning distillation, without directly optimizing the reflector.
DOR$^2$ consistently outperforms Filter-only ROSD, supporting the benefit of explicitly distilling reflection-quality feedback into the reflector, beyond using that feedback merely to select reflection guidance.
Together, these experiments provide complementary evidence that DOR$^2$ generates reflections with greater practical correction utility and that explicitly learning to generate useful reflections contributes to improved final reasoning performance. This directly supports our central motivation for jointly optimizing reflection generation and reasoning.