- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies strategic diversity in LLM self-training, arguing that IID sampling followed by correctness filtering may overrepresent strategies the model already favors. The authors introduce GROOT, which constructs a hierarchical tree of approaches, and adapt Verbalized Sampling to generate alternative approaches before producing complete responses. Across competitive programming and next-chapter prediction, they report that training on strategically sampled data improves performance on difficult tasks and provides stronger initializations for reinforcement learning and test-time scaling than IID-based training.
Critical strengths
- The authors investigate the issue of diversity in model sampling, which is an interesting research question.
Critical weaknesses
- The paper’s motivation is somewhat limited. There is already a substantial body of research on diversity in data sampling, and this work does not clearly distinguish itself from prior studies or demonstrate the specific advantages offered by its proposed methods.
- The writing is somewhat rough. The introduction does not clearly articulate the unique advantages of the proposed method within the broader line of research on sampling diversity. The methodology section is overly brief and omits many important technical details; the method would benefit from a more formal mathematical formulation.
- The experimental evaluation considers a rather limited set of baselines, relying primarily on IID sampling. It would be helpful to include comparisons with more recent methods.
Questions for authors
See weaknesses.
3 comments
Your responses regarding the method description and the adequacy of the experiments have addressed my concerns. However, I remain unconvinced by the motivation. In your response, you distinguish your approach from prior work by saying that existing studies focus primarily on the correctness of the data, whereas your method focuses on improving data diversity. However, research on data diversity has a long history. For example:
[1] Measuring Diversity in Synthetic Datasets, ICML 2025 [2] Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning, NeurIPS 2025
Your method may indeed have distinctive aspects, but I do not think these are clearly emphasized in the abstract, introduction, or subsequent rebuttal. the benefits of “increasing diversity through the training data during the warmup stage, while keeping the training objective and test-time inference fixed” are not sufficiently clear to me. Taking this into account, along with the other reviewers’ comments, I have decided to maintain my score.
We thank the reviewer for engaging with our rebuttal, and are glad our responses on the method description and experiments addressed their concerns.
We agree that data diversity is an active area, which we see as highlighting the relevance of our work’s analysis. However, [1] and [2] differ significantly from our work:
[1] proposes a metric (DCScore) for the diversity of synthetic text datasets, such as classification and story completion data.
[2] generates new math and NLI problems with larger models (for math, 72B for problems and 32B for solutions, to train a 7B student), keeps only majority-vote answers (a correctness filter, so training on incorrect data is not studied), and diversifies the dataset in gradient space.
Neither focuses on approach-level diversity, self-training, the effect of incorrect data, or the effect of this fine-tuning on RL or test-time aggregation. Our domains are also different, as we focus on Next-Chapter Prediction and coding tasks.
We would like to highlight that prior work has also explicitly described the need for work in our specific setting: Lee et al. (2026), for example, leave ‘direct optimization of approach-level diversity as an open problem’.
As for the benefits of 'increasing diversity through the training data during the warmup stage, while keeping the training objective and test-time inference fixed': we show that changing only the data to be more strategically diverse improves pass@k performance, test-time aggregation, and RL initialization (Section 5). Keeping the training objective and test-time inference fixed makes our proposed method flexible and ensures that existing training pipeline can can still be used.
We will add [1] and [2] to the related work, alongside a more detailed discussion.
We thank the reviewer for finding the question of sampling diversity interesting. We answer each question below.
W1: 'The paper's motivation is somewhat limited.'
Section 2 (Related Work) discusses how our paper relates to prior research. We argue that most work on improving sampling diversity focuses on changing the training objective, while we instead 'increase diversity through the training data during the warmup stage, keeping the training objective and test-time inference fixed'.
We also analyse what makes self-training data useful, challenging common assumptions that the data should be correct, that more samples from the model's own distribution help, and that a larger teacher gives better data (Key Takeaways 1-3). Our reply to Reviewer YQ7n compares our work in detail with GuidedSampling and PlanSearch, highlighting differences in both the focus of analysis (we focus on what features make training data useful and they focus on finding more correct solutions) and specific sampling procedures. We will add this comparison to the paper. We will also state the distinction early in the introduction.
If there are any specific comparisons you would like us to make we are happy to answer any questions.
W2: 'The methodology section is overly brief [...] the method would benefit from a more formal mathematical formulation.'
We will add a formal definition to Section 3, and move some implementation details about prompts (Appendix A.9) and data filters (Appendix A.2) into the main text.
The formal definitions of our tested sampling procedures, given an input x and model π, where pname refers to a named prompt:
Feel free to ask about any other specific points of confusion.
W3: 'The experimental evaluation considers a rather limited set of baselines, relying primarily on IID sampling.'
Our evaluation tests four assumptions about self-training data through six experimental comparisons, supplemented by two new controls that isolate the contribution of strategic diversity:
We challenge each of these assumptions through our ablations and comparisons against strategic sampling, showing how strategically diverse data can be a better basis for post-training.
Other sampling methods may be complementary to our work, as our focus is on analysing what makes training data useful, not on identifying the optimal sampling method.