acceptodds
Under review as a conference paper at ICLR 2027

Strategically Diverse Sampling for Self-Training

Abstract

Many LLM training and inference methods, including RL and test-time scaling, depend on repeated sampling, but benefit only when the responses meaningfully differ. Self-training faces the same challenge: training data is typically constructed by sampling IID responses and filtering primarily for correctness, thereby over-representing strategies a model already favours. We investigate strategic diversity, or substantive variation among approaches to a problem, as an alternative principle for constructing self-training data. We generate strategically diverse data with two sampling methods: GROOT, a new method which constructs a hierarchical tree of approaches and samples distinct paths, and Verbalized Sampling (VS), adapted to produce an unstructured set of approaches. Across competitive programming and Next-Chapter Prediction domains, models trained on strategically sampled data outperform IID-trained counterparts on difficult tasks and provide strong initializations for RL and test-time scaling. Most strikingly, self-training on strategically diverse but incorrect traces from Qwen3-4B outperforms IID distillation from a 235B teacher. These results challenge prevailing assumptions about what makes useful self-training data and show that diversity of approaches can matter more than correctness or teacher scale.

Primary area:
generative models
Keywords:
diversity, fine-tuning, RL, test-time scaling, aggregation, narrative

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oWjxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies strategic diversity in LLM self-training, arguing that IID sampling followed by correctness filtering may overrepresent strategies the model already favors. The authors introduce GROOT, which constructs a hierarchical tree of approaches, and adapt Verbalized Sampling to generate alternative approaches before producing complete responses. Across competitive programming and next-chapter prediction, they report that training on strategically sampled data improves performance on difficult tasks and provides stronger initializations for reinforcement learning and test-time scaling than IID-based training.

Critical strengths
  1. The authors investigate the issue of diversity in model sampling, which is an interesting research question.
Critical weaknesses
  1. The paper’s motivation is somewhat limited. There is already a substantial body of research on diversity in data sampling, and this work does not clearly distinguish itself from prior studies or demonstrate the specific advantages offered by its proposed methods.
  2. The writing is somewhat rough. The introduction does not clearly articulate the unique advantages of the proposed method within the broader line of research on sampling diversity. The methodology section is overly brief and omits many important technical details; the method would benefit from a more formal mathematical formulation.
  3. The experimental evaluation considers a rather limited set of baselines, relying primarily on IID sampling. It would be helpful to include comparisons with more recent methods.
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for finding the question of sampling diversity interesting. We answer each question below.

W1: 'The paper's motivation is somewhat limited.'

Section 2 (Related Work) discusses how our paper relates to prior research. We argue that most work on improving sampling diversity focuses on changing the training objective, while we instead 'increase diversity through the training data during the warmup stage, keeping the training objective and test-time inference fixed'.

We also analyse what makes self-training data useful, challenging common assumptions that the data should be correct, that more samples from the model's own distribution help, and that a larger teacher gives better data (Key Takeaways 1-3). Our reply to Reviewer YQ7n compares our work in detail with GuidedSampling and PlanSearch, highlighting differences in both the focus of analysis (we focus on what features make training data useful and they focus on finding more correct solutions) and specific sampling procedures. We will add this comparison to the paper. We will also state the distinction early in the introduction.

If there are any specific comparisons you would like us to make we are happy to answer any questions.

W2: 'The methodology section is overly brief [...] the method would benefit from a more formal mathematical formulation.'

We will add a formal definition to Section 3, and move some implementation details about prompts (Appendix A.9) and data filters (Appendix A.2) into the main text.

The formal definitions of our tested sampling procedures, given an input xx and model π\pi, where pnamep_{name} refers to a named prompt:

  • IID sampling draws nn responses independently: y1,…,yn∼π(⋅∣x)y_1, \dots, y_n \sim \pi(\cdot \mid x).
  • Strategic sampling first draws nn approaches in one planning call, (a1,…,an)∼π(⋅∣x,pplan)(a_1, \dots, a_n) \sim \pi(\cdot \mid x, p_{\text{plan}}). The planning prompt pplanp_{\text{plan}} asks for a list of distinct approaches with estimated probabilities (VS), or for nn distinct paths through a tree of decisions (GROOT). We then sample a response conditioned on both the question and each individual approach: yi∼π(⋅∣x,ai,pexec)y_i \sim \pi(\cdot \mid x, a_i, p_{\text{exec}}), where pexecp_{\text{exec}} asks the model to reason as if it had found aia_i itself.

Feel free to ask about any other specific points of confusion.

W3: 'The experimental evaluation considers a rather limited set of baselines, relying primarily on IID sampling.'

Our evaluation tests four assumptions about self-training data through six experimental comparisons, supplemented by two new controls that isolate the contribution of strategic diversity:

  • Correctness matters most: training on correct (RFT), all (SFT), or only incorrect (ANTI) samples.
  • More samples help: IID with a 16 times larger budget (IID-64).
  • Higher temperature gives more diversity: IID at a tuned temperature (T=1.5).
  • Stronger teachers give better data: IID distillation from a 235B teacher (120B for Nemotron).
  • New controls: testing non-diverse, but approach-conditioned, data that reuses a single approach for all samples (see our response to Reviewer YQ7n). Both new controls result in weaker models than GROOT-4 and VS-4, indicating that diversity in the approach space does improve performance.

We challenge each of these assumptions through our ablations and comparisons against strategic sampling, showing how strategically diverse data can be a better basis for post-training.

Other sampling methods may be complementary to our work, as our focus is on analysing what makes training data useful, not on identifying the optimal sampling method.

Your responses regarding the method description and the adequacy of the experiments have addressed my concerns. However, I remain unconvinced by the motivation. In your response, you distinguish your approach from prior work by saying that existing studies focus primarily on the correctness of the data, whereas your method focuses on improving data diversity. However, research on data diversity has a long history. For example:

[1] Measuring Diversity in Synthetic Datasets, ICML 2025 [2] Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning, NeurIPS 2025

Your method may indeed have distinctive aspects, but I do not think these are clearly emphasized in the abstract, introduction, or subsequent rebuttal. the benefits of “increasing diversity through the training data during the warmup stage, while keeping the training objective and test-time inference fixed” are not sufficiently clear to me. Taking this into account, along with the other reviewers’ comments, I have decided to maintain my score.

We thank the reviewer for engaging with our rebuttal, and are glad our responses on the method description and experiments addressed their concerns.

We agree that data diversity is an active area, which we see as highlighting the relevance of our work’s analysis. However, [1] and [2] differ significantly from our work:

[1] proposes a metric (DCScore) for the diversity of synthetic text datasets, such as classification and story completion data.

[2] generates new math and NLI problems with larger models (for math, 72B for problems and 32B for solutions, to train a 7B student), keeps only majority-vote answers (a correctness filter, so training on incorrect data is not studied), and diversifies the dataset in gradient space.

Neither focuses on approach-level diversity, self-training, the effect of incorrect data, or the effect of this fine-tuning on RL or test-time aggregation. Our domains are also different, as we focus on Next-Chapter Prediction and coding tasks.

We would like to highlight that prior work has also explicitly described the need for work in our specific setting: Lee et al. (2026), for example, leave ‘direct optimization of approach-level diversity as an open problem’.

As for the benefits of 'increasing diversity through the training data during the warmup stage, while keeping the training objective and test-time inference fixed': we show that changing only the data to be more strategically diverse improves pass@k performance, test-time aggregation, and RL initialization (Section 5). Keeping the training objective and test-time inference fixed makes our proposed method flexible and ensures that existing training pipeline can can still be used.

We will add [1] and [2] to the related work, alongside a more detailed discussion.

Reviewer_kyZAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes strategically diverse self-training: generate different approaches to each problem, use them to produce answers, then train on the answers without the approach prompts. Compared with ordinary sampling, this covers more problems and improves the trained model’s code and story-planning performance. The resulting models also benefit more from test-time aggregation and provide stronger starting points for RL, which generally learns faster and sometimes reaches better final performance.

Critical strengths
  • Strong empirical results. Strategically sampled self-training improves performance over conventional sampling across code and story planning. The paper also tests whether these gains carry through to test-time aggregation and subsequent RL.

  • Informative ablations. The comparison among SFT, RFT, and ANTI is especially useful. Models trained only on unsuccessful but strategically diverse responses can still outperform the base model, suggesting that the benefit is not solely due to collecting more correct answers.

  • Simple, well-presented idea. The approach is easy to follow and fits into an otherwise standard self-training pipeline, making it a promising method to explore at larger scale.

Critical weaknesses
  • The central mechanism is not measured directly. The paper shows that strategic sampling solves more distinct problems and improves downstream performance, but it does not measure whether the generated answers—or the trained model’s answers to the same problem—use more distinct strategies. These results support the method’s effectiveness, but leave the proposed explanation of preserved strategic diversity less certain.

  • The final RL benefit varies across settings. The strategic models often provide a better starting point and learn faster, but a higher final performance is less consistent. In the Qwen code results, VS’s held-out pass@64 falls slightly after RL (Table 6); in the Nemotron results, strategic and IID initializations finish close together (Figure 3 and Table 8). The evidence is stronger for faster RL progress than for a reliably higher final ceiling.

Questions for authors
  1. RL dynamics: Do the strategically trained models enter RL with higher output entropy or greater approach-level diversity than the IID-trained models? In Figure 2b, they start at a much higher pass@8, but their additional improvement during RL appears smaller. Is this mainly due to reduced headroom, or does their initial output distribution change how RL learns?

  2. Sampling scale: How does the marginal benefit change as the number of proposed strategies increases? Appendix A.2 already shows limited improvement from four to eight samples. Does this indicate that the model has exhausted useful distinct approaches, or might larger sampling budgets still improve coverage?

  3. Difficulty of learning diverse data: Is strategically diverse data harder to fit than IID-generated data under the same training setup? For example, how do their training and validation loss curves compare, and does fine-tuning on strategic data increase the model’s output entropy?

1 comment

We thank the reviewer for highlighting our empirical results, ablations, and the promise of the method at larger scale. We answer each question below, with new experiments for W1, Q1 and Q3.

W1: 'The central mechanism is not measured directly.'

To address this concern, we measure strategy diversity in both training data and model-generated answers using an LLM judge (gpt-oss-120b), following Lee et al. (2026, arXiv:2606.29985).

Training data: All four samples use the same strategy for 84% of problems under IID sampling, compared with 54% for GROOT and 62% for VS. The average number of distinct strategies per problem is correspondingly higher for GROOT (1.63) and VS (1.50) than for IID (1.21).

Trained models: Among eight generated answers per problem, all use the same strategy for 64% of problems with IID-4, comparable to the base model (62%), but only 39% with GROOT-4 and 32% with VS-4. The average number of distinct strategies is also higher for GROOT-4 (2.18) and VS-4 (2.32) than for IID-4 (1.61) or the base model (1.59).

In summary, we find that strategically sampled data is more diverse in approaches, and that training on this data makes the model's answers more diverse. In contrast, training on IID data has little effect on diversity.

W2: 'The final RL benefit varies across settings.'

We agree that the primary conclusion from our results is faster RL progress, and will state the RL claim more precisely: strategic models start higher in every setting, and GROOT in particular keeps improving during RL and finishes highest with Qwen3-4B on code (Table 6) and NCP (Table 32). With the Nemotron model on code (Table 8), the performance after RL converges.

Q1: 'Do the strategically trained models enter RL with higher output entropy or greater approach-level diversity?'

Yes to both. We measure, for Qwen3-4B on code, the mean next-token entropy of each model on its own samples, and strategy diversity with the judge from W1:

ModelEntropy before RLEntropy after RLAll 8 answers share one strategy, before RL[...] after RL
base0.230.2962%60%
IID-40.190.3164%57%
GROOT-40.310.4139%31%
VS-40.320.5232%38%

The strategic models enter RL with higher entropy and lower strategy collapse than the base and IID-4 models, and finish RL with higher entropy and more diversity as well. However, as the reviewer notes, these models do not necessarily gain more from RL.

We cannot definitively say what causes the relative plateau from initial performance (i.e. what causes the gain from RL to be lower). It's possible that there is a ceiling to RL-improvement with our given model and hyperparameter combination, and training on strategically sampled data simply moves models closer to that ceiling at the start.

Prior work by Yue et al. (2025, arXiv:2504.13837) argues that the problems RL solves are within some solvability threshold of the original base model, possibly indicating that new information would need to be introduced from outside the model's own knowledge to see further improvements.

Q2: 'How does the marginal benefit change as the number of proposed strategies increases?'

Larger sampling budgets do seem to improve the number of unique solved problems in the training set (GROOT solves 155 with budget 4 and 240 with 8), but training on these increasingly large datasets does not seem to continue improving the model. One hypothesis for why this is the case could be that the primary driver of improvement from strategically diverse data is a reasoning style that better considers the possible approach space for a given question, and the benefits from that reasoning style plateau at low approach numbers. It's also possible that the number of reasonable approaches to any given problem is small and as a result more training data contains rare or unhelpful examples. However, it is worth mentioning that on NCP where the number of reasonable approaches is much larger than code, raising the budget from 8 to 16 continued to improve the VS model (Appendix A.7).

Q3: 'Is strategically diverse data harder to fit than IID-generated data?'

Yes. The training loss on GROOT-4 and VS-4 data starts at 0.36 and 0.34 per token, compared to 0.18 for IID-4, and remains about twice as high (0.29 and 0.26 vs 0.15 at the end). This is explained by IID data being close to the base model's distribution. Fine-tuning on strategic data also increases the model's entropy over its own responses (to around 0.31 for GROOT-4, compared to the base model's 0.23). In contrast, fine-tuning on IID data reduces entropy to 0.19.

We will add the loss curves to the Appendix.

Reviewer_YQ7nOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that IID self-training data over-represent the strategies a model already prefers, and proposes to sample for strategic diversity instead. A planner call proposes several approaches to a problem, either as distinct paths of a decision tree in the new method GROOT, or as a list with estimated probabilities, adapted from Verbalized Sampling (VS). A solver call then writes one solution per approach, and the approach is left out of the training prompt. Models are fine-tuned on the correct, all, or only the incorrect traces. On competitive programming and next-chapter prediction, mainly with Qwen3-4B, the paper reports gains over IID data on hard problems, under test-time aggregation and as an RL starting point, even compared with much larger IID budgets. Training only on incorrect strategic traces is also reported to beat IID distillation from a 235B teacher.

Critical strengths
  1. Training on the strategically sampled data gives clear gains on hard problems, even when the traces are incorrect. On Qwen3-4B, models trained on GROOT or VS data reach a much higher pass@k on hard held-out problems than models trained on IID data, including IID data with a much larger budget and a higher temperature. The same holds when only the incorrect strategic traces are kept, and these models remain ahead of the IID-trained models after RL, as Appendix A.5 shows. For building warm-up data before RL, this is a useful practical finding.

  2. The evaluation is broad. It covers two domains, competitive programming with a verifier and next-chapter prediction without one, two student models, and three uses of the trained model, namely repeated sampling, test-time aggregation and RL. The comparisons with IID data are also extensive, with budgets up to 16 times larger, a higher temperature and distillation from a 235B teacher. So the main gains do not seem to come from a weak IID baseline.

Critical weaknesses
  1. The novelty over prior work on approach-conditioned sampling is limited. GuidedSampling [1] already samples solutions conditioned on distinct concepts for each problem and fine-tunes on the correct ones, also without the concept. It reports gains over repeated sampling at pass@5. For competitive programming, PlanSearch [2] shows that conditioning code on diverse natural-language plans raises pass@k. VS is an existing method, here applied to approaches, and deriving a hidden approach as if the model found it is close to the rationalization step of STaR. So the main new element of the method seems to be the tree planner GROOT, which does not consistently beat VS in Table 2. Neither [1] nor [2] is cited, and the contribution should be positioned against them.

  2. Strategic diversity is neither guaranteed nor measured. The paper attributes its gains to strategic diversity, but nothing ensures that the sampled approaches differ in strategy. Section 3 notes that VS does not enforce differences, and the GROOT prompt only encourages paths from different top-level strategies. In the prompt's own example tree, two paths can differ only in the sort order. The paper also never measures diversity, even though Section 2 lists such measures. Meanwhile, most approach texts in the coding task examples of Appendix A.10 already say whether a method is fast enough for the input size, which works as a hint that an IID sample never gets. So part of the gain may come from this guidance rather than from diversity, and a control with approach-conditioned but non-diverse data is needed to tell the two apart.

  3. Whether the solutions follow their approaches is not checked. The solver must "closely align with the given approach", but the only filter tied to the approach removes responses that mention it. The code examples in Appendix A.10 suggest that solutions can depart from their approaches. For GROOT, the program for the first correct path, A1, is an offline sweep with a Fenwick tree rather than the planned loop over all trains. The D2 path is also tagged correct, although its own text says the approach "is invalid". A weak approach can also push the solver to justify steps that do not hold, and no filter checks the reasoning itself. As the diversity claim depends on it, the rate of adherence should be reported.

  4. The RL evidence depends on one objective per task. For coding task, RL uses only MaxRL, which targets pass@k. Without a GRPO run on coding, it is unclear whether the advantage of the strategic starting points holds under a standard objective. On next-chapter prediction, which does use GRPO, the GROOT and VS starts end ahead of the IID and base starts, but only the GROOT start improves after RL on the threshold-only metric in Table 31.

[1] Handa, Divij, et al. "GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time." The Fourteenth International Conference on Learning Representations. 2026.

[2] Wang, Evan, et al. "Planning in Natural Language Improves LLM Search for Code Generation." The Thirteenth International Conference on Learning Representations. 2025.

Questions for authors
  1. Clarification on W1. Please position the method against [1] and [2], and explain what the decision tree adds over the VS list, given that GROOT is not consistently better.

  2. Requested comparison for W2. Please train Qwen3-4B on approach-conditioned but non-diverse code data of the same size, for example with one approach reused for all samples of a problem. Please also report a direct diversity measure, such as the number of distinct approaches per problem, for the training data and the trained models. This would show how much of the gain comes from diversity rather than from the conditioning itself.

  3. Clarification on W3. Please report, on a sample of the code training data, how often the solution follows its assigned approach and how often the approach itself is valid, for example with an LLM judge.

  4. Requested comparison for W4. If possible, please run RL with GRPO on code from the IID-4, GROOT-4 and VS-4 models with the same settings as the MaxRL runs. This would show whether the advantage of the strategic starting points depends on MaxRL.

  5. Clarification on W4. On Nemotron3-Nano-4B, the IID-4, GROOT-4 and VS-4 models end close together after RL, and the strategic methods also benefit from the larger teacher. Please clarify whether the Key Takeaways, such as the advice to sample from the student rather than a larger teacher, hold beyond Qwen3-4B.

3 comments

We thank the reviewer for recognizing the utility of strategically sampled data for hard problems and RL warm-up. We address questions below, with new experiments for W2, Q2, W3 and Q5.

W1/Q1: 'The novelty over prior work on approach-conditioned sampling is limited.'

GuidedSampling [1] and PlanSearch [2] mainly show that diverse concepts can help find correct answers during inference. Our paper instead asks what makes self-training data useful and challenges assumptions like the importance of correct traces or a larger teacher.

GuidedSamplingPlanSearchOurs
Conditioned onconcept or theorem nameplan from combined observationsfull approach
Training datacorrect samplesnonecorrect, all, or incorrect samples, on hard problems
Compared to high-budget IID and a larger teacher✗✗✓
Model used for RL and test-time aggregation✗✗✓
Includes a non-verifiable domain✗✗✓ (NCP)

PlanSearch [2] leaves training on generated plans for future work, which is the focus of our paper. Unlike STaR's [4] rationalization step, we never give the model the correct answer, and unlike VS [3] as originally proposed, we focus on approach-level diversity and training data. We will add [1], [2], and this comparison to the paper.

GROOT's tree of explicit decisions is a structured alternative to VS's unstructured list and tends to give more distinct approaches (all four share one strategy on 30% of problems, vs 50% for VS) and a better initialization for RL (Table 6).

[1] Handa et al. 'GuidedSampling [...]'

[2] Wang et al. 'Planning in Natural Language [...]'

[3] Zhang et al. 'Verbalized Sampling [...]'

[4] Zelikman et al. 'STaR [...]'

W2/Q2: 'Strategic diversity is neither guaranteed nor measured.'

We count distinct strategies with an LLM judge (gpt-oss-120b), following [5], on 4 samples for 488 training problems and 8 answers for 203 test problems. Sampling cannot guarantee distinct strategies, but strategic sampling induces far more.

All four IID training samples share one strategy on 84% of problems, versus 54% for GROOT and 62% for VS (avg. distinct strategies 1.21, 1.63 and 1.50). After training, all 8 answers share one strategy on 64% of problems for IID-4, versus 39% for GROOT-4 and 32% for VS-4 (avg. 1.61, 2.18 and 2.32).

We also push back on the idea that the approaches are 'a hint that an IID sample never gets'. The model gets no extra information when writing approaches, so IID samples are free to reason in the same way (and truncate less often, 5% versus GROOT and VS's 32%).

[5] Lee et al. 'Are We Measuring Strategy [...]'

Q2: 'approach-conditioned but non-diverse code data'

We perform two experiments that reuse one approach for all four samples, using either VS's most likely approach or one sampled IID. Both produce weaker models (pass@64):

Training dataCobaltLiveCodeBenchOJBench
VS's most likely approach, x425.319.111.3
one IID approach, x426.419.110.3
GROOT-430.519.511.5
VS-430.223.111.9

Reusing one approach is worse than four distinct ones, so the gap comes from diversity.

W3/Q3: 'Whether the solutions follow their approaches is not checked.'

We use gpt-oss-120b to judge whether each solution follows, modifies, or departs from its approach, and whether the approach is valid. It agrees with the reviewer on A1 (modifies) and D2 (departs). However, departures are rare: 10% (GROOT-4) and 5% (VS-4) of correct solutions and 14% of incorrect ones. The judge also finds that about 80% of approaches for correct solutions are valid, versus 43% for incorrect ones. Importantly, even training only on incorrect solutions outperforms IID training (Table 2).

We will add this analysis and clarify that D2’s 'correct' means the program passed the tests.

W4/Q4: 'The RL evidence depends on one objective per task.'

We chose MaxRL for code because it targets pass@k; it differs from GRPO only in dividing advantages by the mean reward. We cannot re-run GRPO from all starts during the rebuttal, but earlier experiments with slightly different data found the two comparable (13.1 pass@8 vs 12.9 from GROOT, 8.8 vs 10.1 from base).

Q5: 'clarify whether the Key Takeaways [...] hold beyond Qwen3-4B.'

Our Nemotron3-Nano-4B experiments (A.1) largely reproduce the Qwen3-4B findings.

Strategic sampling outperforms IID training (held-out pass@64 of 19.9/20.8 for GROOT-4/VS-4 versus at most 14.9 for IID; Table 7), even when training on incorrect samples (16.2/19.7 vs 7.8 IID-4; new rows). Self-generated strategic data also outperforms IID distillation from a 120B teacher (13.8), although strategic sampling benefits further from a stronger teacher (25.7/27.4; Table 12). Strategic data also speeds up RL: VS-4 and GROOT-4 reach 40% validation pass@8 in 28 and 48 steps, versus 64 for IID-4 (Figure 3), though final performance converges (Table 8).

Thank you for the detailed response and the new experiments. I appreciate the diversity measure, the controls that reuse one approach and the adherence analysis.

However, some concerns remain. For example regarding Q2, the newly added controls already reach 19.1 pass@64 on LCB, against 7.0 for IID-4 in Table 2 and 19.5 and 23.1 for GROOT-4 and VS-4, and OJBench shows similar pattern. So on the held-out benchmarks, most of the gain over IID seems to come from conditioning on an approach rather than from diversity, and the claim that the gains are driven by diversity should be narrowed. Considering the results of GROOT for RL initialization on Nemotron3-Nano-4B, where GROOT-4 ends below VS-4 and IID-4 on the held-out benchmarks after RL in Table 8, the takeaway to use GROOT to initialize RL does not seem to be supported. The same holds for the advice to sample from the student rather than a larger teacher, since GROOT and VS both improve with data from the larger teacher in Table 12. So, based on the current experiments, these two takeaways should be narrowed to Qwen3-4B. Given the new results and considering the whole novelty and quality of this work, I raise my score from 1 to 2.

We thank the reviewer for increasing their score.

To respond to the first point, we believe that the worse in-distribution performance in our Q2 experiments shows the benefit of a more approach-diverse set of inputs, but our results also show that approach-conditioned generation performs much better than IID, which is in line with our claims. Perhaps the reasoning performed when given an approach, chosen by the model from the space of possible approaches, is qualitatively different, and can improve performance even when incorrect. In revisions we will add these hypotheses to Section 5.

For the second set of concerns about Takeaways 3 and 4:

We agree that strategically diverse data also benefits from sampling from a larger teacher on Nemotron specifically. As this is another benefit of our method (strategically diverse sampling performs better than traditional IID sampling whether from the same student or a larger teacher), we will adjust the language of Takeaway 3 in revisions.

After RL, GROOT-4, VS-4 and IID-4 on Nemotron are within seed noise of each other (held-out pass@64 of 31.0 ± 2.0, 33.6 ± 3.0 and 33.3 ± 1.8 over three RL runs), so Table 8 shows convergence rather than VS-4 significantly outperforming GROOT-4. We will add the standard deviations in revisions. We believe these results show that GROOT-4 performs comparably to or better than VS-4 after RL, in line with our recommendation in Takeaway 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.