- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work dives deep into how training data, specifically the training questions, affects the final performance of OPD. They conduct comprehensive and interesting experiments with the number of training questions and teacher generated questions. Their findings, including OPD performance can be mostly achieved using 8 questions, teacher generated questions can obtain on-par performance with annotated questions, are very interesting and provide novel perspectives into how OPD data should be curated.
Critical strengths
1, The findings from this work are novel and interesting. I believe they are great contributions to the community.
2, Besides empirically answering research questions, this work also conducts reasonable investigation into the observations, which provides preliminary insights and encourages further studies.
3, The manuscript is well-written and easy to follow.
Critical weaknesses
1, The critical weakness, as acknowledged in the conclusion section, is the scale of the experiments. It is not guaranteed that the conclusions could straightforwardly generalize to larger models. However, considering the complexity of such experiments, these limitations are acceptable; and the work still provides meaningful insights.
Questions for authors
From my understanding, the 8 samples in the THE NUMBER OF TRAINING QUESTIONS section and 64 samples in DF-OPD are randomly sampled. It's actually surprising to me that without any advanced sampling heuristics, those few samples could already recover the OPD performance on the whole set.
That said, I am wondering how the random seed would affect the results? Could you try some other random seeds, and see if the conclusions will change?
5 comments
Response to Reviewer 7AdL (2/2)
Additional robustness check across checkpoint-selection rules.
To further verify that our small-pool OPD and DF-OPD results are also insensitive to the choice of checkpoint-selection rule, we evaluate two additional rules alongside the original one used in our paper:
- Original: selects the checkpoint achieving the highest overall score on the evaluation benchmarks across training, as used in the paper (
Appendix B.3). - Final checkpoint: directly reports the last checkpoint at the end of training, with no checkpoint selection.
- Held-out val: selects the checkpoint scoring highest on an independent validation set formed by holding out a random 20% of the real training corpus (3,583 questions in the single-teacher settings and 1,492 stratified by domain in MOPD), sampled to avoid any overlap with the small pools and disjoint from all evaluation benchmarks. To prevent any overlap with this validation set, the full-data baseline is retrained on the remaining 80% questions.
Table R2 below reports performance under all three rules for our key configurations:
| Setting | Question pool | Original | Final checkpoint | Held-out val |
|---|---|---|---|---|
| RL-tuned teacher | Real, |D|=17,917 (full data) | 56.41 | 56.05 | — |
| (init 47.08, teacher 61.44) | Real, |D|=14,334 (full data minus held-out val) | 56.39 | 55.72 | 56.39 |
| Real, |D|=8 | 56.33 | 55.77 | 56.33 | |
| Self-generated, |D|=64 | 56.61 | 56.53 | 56.61 | |
| Larger teacher | Real, |D|=17,917 (full data) | 47.36 | 46.61 | — |
| (init 34.49, teacher 75.46) | Real, |D|=14,334 (full data minus held-out val) | 47.57 | 47.07 | 47.25 |
| Real, |D|=8 | 47.73 | 47.61 | 47.56 | |
| Self-generated, |D|=64 | 49.16 | 49.04 | 49.04 | |
| Multi-teacher | Real, |D|=7,458 (full data) | 31.18 | 30.53 | — |
| (init 24.21, teachers 31.42) | Real, |D|=5,966 (full data minus held-out val) | 31.24 | 30.39 | 31.24 |
| Self-generated, |D|=1k | 31.31 | 31.31 | 31.31 |
As shown in Table R2, both small-pool OPD and DF-OPD remain comparable to the full-data baselines whether evaluated directly at the final checkpoint (55.77 / 56.53 vs. 56.05, 47.61 / 49.04 vs. 46.61,
and 31.31 vs. 30.53) or selected via held-out validation (56.33 / 56.61 vs. 56.39, 47.56 / 49.04 vs. 47.25, and 31.31 vs. 31.24),
further confirming the stability of our findings.
W1. Generalization to larger model scales.
Thank you very much for your kind understanding of the complexity of such experiments! We fully agree that testing larger student models to expand the scale of our experiments is an important direction. We are actively running experiments with larger student models (e.g., 7B) during the discussion period. However, because multi-step on-policy rollout generation and multi-benchmark evaluation across pool sizes are computationally intensive, they may not finish before the discussion period ends. Should we unfortunately be unable to share them in time during the discussion phase, we commit to including results on at least one to two larger student models in the final version of the paper!
Thank you once again for your generous and encouraging review! Your feedback has not only strengthened our work, but also motivated us to keep moving forward on our research. Please let us know if you have any further questions, and we would be more than happy to address them! :-D
Best regards,
Authors of Paper 36946
I thank the authors for their detailed responses, and decide to keep my positive rating.
Response to Reviewer 7AdL (1/2)
Dear Reviewer 7AdL,
Thank you very much for your positive assessment and strong support of our work, and for recognizing the novelty of our findings, our mechanistic investigations, and our contributions to the community! Below, we address your question on random-seed sensitivity by evaluating independently sampled, mutually disjoint question pools as well as independent training seeds, further corroborated by evaluating the impact of different checkpoint-selection rules on the results (
Q1), and briefly discuss our ongoing efforts toward larger model scales (W1). The corresponding results and analyses are highlighted in blue inSections 3.1 and 4.2andAppendices G.1 and G.2of the revised manuscript.Q1. How does the random seed affect the results? Could you try some other random seeds, and see if the conclusions will change?
Thank you for raising this important question! As you noted, our small question pools are drawn purely at random without heuristic selection. Following your suggestion, to test whether our conclusions depend on a fortunate random draw—or on the random seed used during training—we repeated the two key small-pool and DF-OPD conditions (
Real, |D|=8andSelf-generated, |D|=64in the two single-teacher settings), as well asSelf-generated, |D|=1kin the multi-teacher setting (MOPD), along two independent random-seed axes:Varying question-sampling seed: three runs on mutually disjoint question pools randomly sampled without replacement using seeds 42, 43, and 44 (with 42 as the paper default) from the same source (DAPO-17k for real questions, and the teacher-generated corpus for self-generated ones, with domain proportions held fixed in MOPD), while keeping the training seed fixed at the default.
Varying training seed: three runs on the original question pool using different random seeds (0, 1, and 2, with 0 as the paper default) for student trajectory sampling, which affects the sampled on-policy rollouts and in turn alters the teacher supervision computed along them at every step. (We vary this seed rather than the data-loader seed because in the single-teacher settings, each update of 64 questions (batch size = 64) already covers the entire 8- or 64-question pool, so data-loader shuffling would not change which questions enter a batch.)
Sharing the original paper run, each condition comprises five distinct runs. Table R1 below reports the paper score, the mean ± std (with all three runs in parentheses) along each axis, and the full real-data baseline:
Two clear takeaways emerge from Table R1:
Varying the question-sampling seed does not change the conclusions: Across three mutually disjoint question pools drawn purely at random without heuristics, both
Real, |D|=8(56.61 ± 0.57vs.56.41for the RL-tuned teacher, and48.02 ± 0.43vs.47.36for the larger teacher) andSelf-generated, |D|=64(56.95 ± 0.41vs.56.41, and49.76 ± 0.52vs.47.36) consistently match the17,917-question baseline, whileSelf-generated, |D|=1kin MOPD (31.21 ± 0.21) matches the7,458-question baseline (31.18), with relatively small standard deviations throughout (0.21–0.57points).Varying the training seed likewise yields stable results: Across independent rollout-sampling seeds on the same question pool, both
Real, |D|=8(57.12 ± 0.69vs.56.41, and47.93 ± 0.19vs.47.36) andSelf-generated, |D|=64(56.58 ± 0.33vs.56.41, and48.70 ± 0.48vs.47.36) again stay on par with full-data OPD, as doesSelf-generated, |D|=1kin MOPD (31.28 ± 0.13vs.31.18), with similarly low standard deviations (0.13–0.69points).Together, these results confirm that small-pool OPD and DF-OPD do not rely on a fortunate question draw or training seed.