acceptodds
Under review as a conference paper at ICLR 2027

Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

Abstract

Large language models increasingly tackle hard reasoning problems by spending more test-time compute, yet the dominant strategy remains naive repeated sampling: draw many independent solutions and hope one is correct. Because such sampling explores only through local decoding noise, it tends to produce many near duplicate attempts rather than genuinely different ideas. We ask whether exploration can instead be steered at a semantic level, by first sampling problem specific concepts, hints, or strategies and then conditioning answer generation on them. We refine this into a simple, more exploratory procedure that emits many diverse concepts in a single trajectory, and evaluate it on hard problems where repeated sampling struggles. We then go a step further and make concept generation trainable: a small concept generator is optimized with reinforcement learning so that its concepts maximize the downstream success of a larger, frozen answer generator. On hard mathematical reasoning problems, the trained concept generator substantially improves the answer generator's pass@k over naive repeated sampling at the same answer generation allocation, surpasses concepts drawn from much larger untuned models, and transfers to answer generators it was never trained against, including a model from a different family. A small model can thus be trained into an effective, reusable search policy for a much larger one.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, LLMs, test-time scaling, search policies, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6tYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work explores how to improve LLM reasoning beyond repeated sampling, which often produces redundant duplicate solution attempts. The authors propose to generate multiple problem-specific concepts in a single trajectory and use them to guide a frozen answer model. They further train a small concept generator with reinforcement learning, rewarding concepts according to downstream answer success. On difficult mathematical problems, the trained 7B generator raises a 32B model’s pass@128 from 19.0% to 39.2%, outperforming larger untrained concept generators.

Critical strengths

The paper identifies that GuidedSampling’s reported gains largely result from an overly restrictive repeated-sampling baseline, and thus proposes an improved approach of concept-guided sampling. Furthermore, it shows that a finetuned small concept-generator can effectively guide the reasoning of a much larger model.

Critical weaknesses
  1. Narrow and outdated experimental scope. The main RL experiments are only limited to two mathematical reasoning benchmarks, DeepMath and Omni-MATH 2, one of which even shares the same distribution with the training data, and it's hard to the effectiveness on general math reasoning. And the selected LLM backbones are Qwen2.5 and Llama-3.3-70B, both released in 2024. It is therefore unclear whether the method remains effective for newer reasoning models or tasks beyond mathematics.

  2. Limited evidence of out-of-distribution generalization. When synthesizing the RL data, Qwen2.5-32B is used to generate the answer rollouts that determine the RL reward, and to judge answer correctness. Consequently, the concept generator (CG) learns to exploit Qwen2.5-32B’s specific reasoning priors, failure modes, prompt sensitivities, and judge preferences. Therefore, all evaluation using DeepMath benchmark or Qwen2.5-32B as the answer generator (AG) can be considered in-domain. The only OOD setting that changes both the dataset and answer-model family (Llama-3.3-70B as AG on Omni-MATH 2 in Table 4) reduces pass@1 from 1.92% to 1.73%, while improving pass@128 only modestly from 12.50% to 13.97%.

  3. Weak baseline coverage. The paper lacks against closely related approaches. For example, Step-Back Prompting[1] first elicits high-level principles to guide further reasoning. Diversity of Thought[2] generates and ensembles diverse problem-solving approaches before reasoning. Tree of Thoughts[3] explores and evaluates multiple reasoning branches. These methods have reported strong reasoning improvements in their respective settings. It seems that the idea of concept guidance in this ppaer has little difference from the ideas of finding a good high-level principle to initialize reasoning in the related works. So I prefer seeing the comparison with these approaches in methodology and performance.

[1]: Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

[2]: Diversity of Thought Improves Reasoning Abilities of LLMs

[3]: Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Questions for authors

1. What exactly is a ‘concept‘, and how is it distinguished from a partial solution?

  • Appendix C.3 provides only a syntactic definition. However, the examples in Appendix D.10 vary substantially in semantic content. Some are mathematical principles or theorem names, while others provide a detailed solution guideline, and even multiple intermediate reasoning steps. Moreover, the trained generator produces substantially more concrete derivations than the untrained generator. How can the authors distinguish whether the CG learns to generate high-level concepts or instead learns substantial parts of the solution trajectories required by the AG?
  • In addition, how are the average concept counts in Tables 1 and 2 calculated, and how to define a duplicate concept?

2. Can the trained CG output effective concept guide to newer and stronger answer models? A central claim is that a small CG learns a reusable search policy for larger models. Could the authors evaluate the trained CG with more recent closed-source models, such as GPT-5.4/5.5, or Claude Opus 4.6/4.7? Such experiments would help determine whether the generated concepts are broadly useful or primarily adapted to the reasoning behavior and failure modes of Qwen2.5-32B.

3. In the fully OOD cross-model setting in Table 4, the improvement only appears at pass@128 and pass@64. In other words, I need to rollout dozens of times to reach the correct answer. I wonder with so many rollout outputs, how would the system identify and return the correct answer when the ground truth is unavailable?

5 comments
AuthorsOpenReview ↗

Response to official comment by reviewer T6tY (1/4)

We thank the reviewer for the feedback. Below, we address the concerns and questions with additional experiments and clarifications.

W1. Experimental scope: newer models and tasks beyond mathematics

We started from the same models as the prior GuidedSampling work of Handa et al., so that our re-evaluation in Section 3 is directly comparable, and we use mathematics because it is a verifiable domain that is usually used in RL as answers can be scored to obtain a training signal.

Our analysis (Appendix A) predicts that concepts have more room to improve the performance where the answer generator (AG) has a low success probability, and Section 4 confirms it empirically. Evaluating a newer model therefore requires a benchmark that is hard for that model and still large enough after filtering to give a statistically meaningful signal. Small or saturated benchmarks mostly measure noise.

We have now added three new sets of experiments on benchmarks and models in response to your request:

(a) GPQA-Diamond and AIME.

We evaluate the paper's CG, trained with the frozen Qwen2.5-32B AG on the full AIME 2024, 2025, and 2026 (pooled, 90 problems) and GPQA-Diamond (198 problems). Values are paired differences vs repeated sampling in pass@k points, with 95% bootstrap CIs over problems.

BenchmarkΔ\Deltap@8Δ\Deltap@32Δ\Deltap@128
AIME 24+25+26 (90)+0.8+0.8 [−1.6-1.6, +3.3+3.3]+1.2+1.2 [−3.1-3.1, +5.6+5.6]+2.2+2.2 [−5.6-5.6, +10.0+10.0]
GPQA-Diamond (198)+3.7+3.7 [+1.7+1.7, +5.8+5.8]+4.8+4.8 [+2.5+2.5, +7.2+7.2]+3.0+3.0 [+0.5+0.5, +6.1+6.1]
  • GPQA: better performance. The CG never saw this science domain questions during training. This supports the claim that the CG learns transferable guidance.
  • AIME: shows positive improvement, but all confidence intervals include zero, so the results are within noise. With 90 problems, the 95% CI at pass@128 is about ±8\pm 8 points, so we mostly measure noise due to the dataset size. And importantly, AIME is not in the regime our method targets of very low success probability: Qwen2.5-32B-Instruct baseline already solves 52%52\% of AIME at pass@128.

(b) Two recent reasoning models, gpt-oss-20b (2025) and Qwen3.5-35B-A3B (2026), on hard mathematics.

  • Hard-set construction: For each model, we filter Omni-MATH-2 to the problems it fails on all 128 attempts. This gives 601 problems for gpt-oss-20b and 402 for Qwen3.5-35B-A3B.
  • Evaluation: We then evaluate with fresh samples under a new seed, so repeated sampling is not exactly 0%0\% (the finite-sample effect of Appendix D.6).
  • CGs compared:
    • the model's own concepts (CG = AG).
    • the untuned Qwen2.5-7B CG.
    • our trained Qwen2.5-7B CG, unchanged: trained once against Qwen2.5-32B on DeepMath.
AG (Omni-MATH-2 hard set)Methodp@32p@128 (Δ\Delta vs repeated sampling)
gpt-oss-20b (601)Repeated sampling4.713.5
Own concepts (gpt-oss-20b as CG)8.622.5 (+9.0+9.0 [+5.7+5.7, +12.5+12.5])
Untuned Qwen2.5-7B CG6.818.3 (+4.8+4.8 [+1.7+1.7, +8.0+8.0])
Our trained Qwen2.5-7B CG7.118.3 (+4.8+4.8 [+1.7+1.7, +8.2+8.2])
Qwen3.5-35B-A3B (402)Repeated sampling3.310.0
Own concepts (Qwen3.5 as CG)5.312.7 (+2.7+2.7 [−0.7-0.7, +6.0+6.0])
Untuned Qwen2.5-7B CG4.510.7 (+0.7+0.7 [−2.5-2.5, +4.0+4.0])
Our trained Qwen2.5-7B CG5.513.4 (+3.5+3.5 [+0.0+0.0, +7.5+7.5])

(c) Other domain: physics (UGPhysics).

  • Hard-set construction: We filter the 4,443 single answer English problems of UGPhysics with the same protocol.
  • Grading reliability: We kept only problems with a numeric, multiple-choice or yes/no reference answer. A manual check showed that symbolic physics answers written with different notation (e.g. μ\mu vs mm, AA vs πr2\pi r^2) cannot be graded reliably.
  • Resulting hard sets: 316 problems for gpt-oss-20b, 380 for Qwen3.5 and 434 for Qwen2.5-32B.
AG (UGPhysics hard set)Repeated samplingOwn conceptsUntuned 7B CGOur trained 7B CG
gpt-oss-20b (316), p@12814.219.6 (+5.4+5.4 [+0.3+0.3, +10.4+10.4])16.8 (+2.5+2.5 [−1.9-1.9, +7.0+7.0])23.4 (+9.2+9.2 [+4.7+4.7, +13.9+13.9])
Qwen3.5-35B-A3B (380), p@1287.410.5 (+3.2+3.2 [+0.0+0.0, +6.1+6.1])13.2 (+5.8+5.8 [+2.9+2.9, +8.7+8.7])16.6 (+9.2+9.2 [+5.8+5.8, +12.6+12.6])
Qwen2.5-32B (434), p@12815.217.1 (+1.8+1.8 [−2.3-2.3, +6.0+6.0])17.3 (+2.1+2.1 [−2.1-2.1, +6.2+6.2])19.4 (+4.1+4.1 [+0.0+0.0, +8.3+8.3])

These settings change both the dataset and the answer model family, which is the fully out of distribution experiment requested. And this setting shows that we effectively learn a transferable policy that performs better overall to propose concepts than using other bigger and newer models that were not trained to do this.

AuthorsOpenReview ↗

Response to official comment by reviewer T6tY (2/4)

W2. Out-of-distribution generalization and the judge correctness

We use an LLM judge because rule based matching rejects many correct answers written in a different form. Real examples from our baseline rollouts that exact match (EM, with string normalization) marks wrong: \text{No} for No, e^{2/e} for e^{\frac{2}{e}}, \sinh(1) for \sinh 1, \frac{7-3\sqrt5}{2} for \frac12(7-3\sqrt5).

So we therefore opted to use an LLM as a judge. But the judge only sees the problem statement (for context), the ground-truth answer, and the final answer extracted from the rollout's \boxed{}. Rollouts without a boxed answer are marked incorrect without consulting the judge. Thus the judge never sees the reasoning, so it doesn't judge the entire chain of thought. Using this version of LLM as a judge allows us to avoid incorrectly penalizing trajectories and further introducing noise.

Rule-based evaluation and a CG trained without the judge. We re-graded every rollout with EM, and we also retrained the CG with the EM reward instead of the judge.

Pass@128 (%), frozen Qwen2.5-32B AG:

MethodDeepMath hard EMLLM judgeOmni-MATH-2 hard EMLLM judge
Repeated sampling18.019.08.511.3
Untuned 7B CG24.828.911.915.0
Untuned 32B CG28.733.811.815.3
Trained 7B CG, EM reward29.135.113.918.2
Trained 7B CG, judge reward (paper)32.739.214.318.6

This evidence goes against the claim that the CG "learns to exploit Qwen2.5-32B's [...] judge preferences". The judge-trained CG retains its advantage under independent EM scoring, and it is also better under EM scoring (32.7%32.7\% vs 29.1%29.1\% on DeepMath, 14.3%14.3\% vs 13.9%13.9\% on Omni-MATH-2). Its stronger performance than the EM-trained CG is consistent with false negatives in EM rewards making the training reward noisier.

Crucially, the conclusions from the paper do not change under exact match scoring. We include this experiment in the Appendix.

On the Llama-3.3-70B result. The method targets exploration: finding at least one valid solution within a sampling budget on problems the model essentially never solves. Gains are therefore expected mostly at moderate to large kk, as predicted by Appendix A. A change at pass@1 within noise is consistent with this goal. With the newer models above, gains already appear at pass@32 (+5.5+5.5 and +6.5+6.5 points on UGPhysics).

AuthorsOpenReview ↗

Response to official comment by reviewer T6tY (3/4)

W3. Comparison with Step-Back Prompting, Diversity of Thought, and Tree of Thoughts

All three methods aim to improve reasoning with intermediate structure. They differ from ours in what they optimize and in how they scale to large sampling budgets.

  • Step-Back Prompting makes the model first state the principles behind the given question, then solve it grounded on them. For math (GSM8K), the paper does this in a single call with a one-shot exemplar whose solution first states the principles it applies, decoded greedily and evaluated by accuracy. The principles are problem-specific, but they come from the solver itself, are not trained, and each answer gets a single abstraction. We use the same single call prompt format (ask the model to first recall principles then answer) with a zero-shot instruction, sampled at T=1.0T=1.0 with the same frozen Qwen2.5-32B AG and the same 128-rollout budget:
Method (frozen Qwen2.5-32B AG)DeepMath hard p@1p@32p@128Omni-MATH-2 hard p@1p@32p@128
Repeated sampling0.26.419.00.13.811.3
Step-Back prompting (zero-shot, single call)0.49.324.30.23.911.6
Untuned Qwen2.5-7B CG (no training)0.813.728.90.35.915.0
Trained Qwen2.5-7B CG2.221.939.20.48.318.6

Our untuned CG has better performance than Step-Back at every reported budget. Step-Back improves DeepMath performance (comparably to generic prompt modification baseline) but provides little improvement on Omni-MATH-2. Each Step-Back rollout generates its principles independently, without access to those proposed in other rollouts. Diversity across these principles therefore relies on token noise sampling. Our CG generates concepts jointly in one trajectory, so later concepts are conditioned on earlier ones, allowing the model to explicitly diversify the set.

  • Diversity of Thought works at the task level, not at the problem level. It queries an LLM for approaches on random questions of a benchmark, keeps the most frequent ones, selects the best 3–5 approach/persona prompts on a validation set, and reuses these same prompts for every problem of the task. Answers are decoded greedily and aggregated by majority vote, so the approach targets ensemble accuracy of a 3–5-call ensemble, not coverage under a large sampling budget. Our untuned CG is the analogue of this idea on the problem level: the LLM proposes approaches for each problem, and answers are conditioned on them.

  • Tree of Thoughts searches a tree of partial solutions where the same LLM proposes the candidate next steps and also scores intermediate states, either by valuing each state or by voting for the most promising one among several. Its main tasks are Game of 24, Creative Writing and Mini Crosswords. The biggest limitation is that the search is sequential: each level waits for generation and evaluator calls on the previous one, so it is not inference friendly. On GSM8K, zero-shot ToT samples five strategies and votes for the best one, then samples five solutions conditioned on that strategy and votes for the best solution. Like our untuned approach, it uses high-level strategies to guide solution generation. Our method keeps the inference pattern of repeated sampling: one extra small-model call per problem (0.3% overhead in Section 7.3), then fully parallel rollouts.

We also compare against a trained single-concept generator variant of our approach, in the spirit of DSP/RLAD, in our response to Reviewer mjYZ section 2.

Reviewer_mjYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a small concept generator to improve exploration by a larger frozen LLM. It generates multiple problem-specific hints in one trajectory and optimizes them through downstream answer correctness. On hard DeepMath problems, a trained 7B generator raises a frozen 32B solver’s pass@128 from 19.0% to 39.2%, versus 28.9% with untuned concepts. Gains also extend to Omni-MATH 2 and a different solver family. The paper additionally shows that prior GuidedSampling gains largely disappear against a stronger decoding baseline.

Critical strengths
  • Large gains with a small trainable component. Training improves pass@128 by 10.3 points over untuned concepts and outperforms a larger untuned generator. Cross-dataset and cross-family transfer support the usefulness of the learned guidance.
Critical weaknesses
  • The central correctness measurements lack an independent evaluation. Qwen2.5-32B provides training rewards and evaluation judgments and is also used for dataset filtering and leakage detection. No human evaluation or independently validated verifier is reported. Rare false positives can potentially affect pass@128, and optimization against the same judge could favor its particular errors. This uncertainty affects both the reported gains and the strength of the exploration-versus-leakage interpretation.

  • The incremental contribution over prior learned guidance is insufficiently isolated. Directional Stimulus Prompting already trains a small policy to guide a frozen LLM through downstream rewards, while RLAD trains abstraction generation for reasoning. Its empirical comparisons emphasize repeated sampling and untuned concepts. A separately trained single-hint or plan generator would better establish the benefit of jointly generating and rewarding concept sets.

Questions for authors
  • Can authors independently evaluate a representative sample of problems counted as solved, especially those solved only by the trained method, and report whether the gains persist under corrected judgments?
  • Do authors have a matched comparison with a separately trained single-hint/plan generator, or a frozen-solver adaptation of a closely related learned guidance method?
  • Were the 200-step schedule and other methodological choices selected using a validation set separate from the reported 1,000-problem evaluation set?
4 comments
AuthorsOpenReview ↗

Response to official comment by reviewer mjYZ (1/2)

We appreciate the reviewer's recognition of the gains from training a small concept generator and its transfer across datasets and model families. We address the concerns and questions below.

1. Independent evaluation of the judge

Correctness is defined by a reference final answer, so grading only requires deciding whether two final answers are equivalent. Concretely, the judge only sees the problem statement (for context), the ground truth answer, and the final answer extracted from the rollout's \boxed{}. It never sees the reasoning, so it cannot be persuaded by a convincing chain of thought, and rollouts without a \boxed{} answer are marked incorrect.

Independent rule based verifier. We re-graded every rollout with rule based exact match (EM with some string normalization, e.g. \dfrac to \frac). We also retrained the concept generator (CG) with the EM reward instead of the judge.

Pass@128 (%), frozen Qwen2.5-32B AG:

MethodDeepMath hard EMLLM judgeOmni-MATH-2 hard EMLLM judge
Repeated sampling18.019.08.511.3
Untuned 7B CG24.828.911.915.0
Untuned 32B CG28.733.811.815.3
Trained 7B CG, EM reward29.135.113.918.2
Trained 7B CG, judge reward (paper)32.739.214.318.6
  • The gains persist under the independent verifier. The ranking repeated sampling < untuned CG < trained CG is the same under EM on both datasets. The gains persists even when counting a rollout as correct only when both EM and the judge accept it (14.9%→29.9%14.9\% \to 29.9\% on DeepMath, 7.4%→12.7%7.4\% \to 12.7\% on Omni-MATH-2).
  • Judge-trained gains persist under independent scoring. The judge-trained CG retains its advantage under independent EM scoring. Its stronger performance than the EM-trained CG is consistent with false negatives in EM rewards making the training reward noisier.
AuthorsOpenReview ↗

Response to official comment by reviewer mjYZ (2/2)

2. Relation to prior learned guidance, and a single concept baseline

Both Directional Stimulus prompting and RLAD are close works to our approach, and are cited in the related work. But our setting differs from both as follows:

DSP trains a small policy model to generate a directional stimulus specific to the input that guides a frozen black-box LLM toward a desired output. Depending on the task, the stimulus consists of summary keywords, dialogue acts, or a chain-of-thought trigger prompt. Their approach target accuracy (pass@1) in its reasoning experiments whereas we target coverage measured by pass@k. DSP also uses an SFT warm start for it small policy model before RL training, whereas our CG is trained directly with RL without an earlier SFT.

RLAD uses an SFT warm start as well for its abstraction generator and also trains the answer generator, whereas our AG remains frozen. At test time, RLAD scales by sampling several abstractions independently from the abstraction generator and this raises the diversity question again: independent abstractions from the same generator can repeat the same underlying idea. It leads to a chicken and egg situation: abstractions are meant to provide more diversity than token level sampling noise, yet their own diversity again relies on token sampling noise. We therefore propose to generate concepts in one trajectory, so later concepts are conditioned on earlier ones. And this one trajectory generation is more efficient and inference friendly than the previous iterative concepts generation procedure by GuidedSampling of Handa et al.

Matched comparison with generators trained to emit one concept. We ran the comparison requested by the reviewer, with the same CG model (Qwen2.5-7B-Instruct), the same frozen 32B AG, the same budget of 128 rollouts, and the same judge:

  • Single concept CG: our pipeline, with the CG prompted to emit one concept per trajectory. The trained variants use either the max-of-mean or max-of-max reward. At inference, "1 concept" allocates all 128 rollouts to one sampled concept, and "8 concepts" allocates 16 rollouts to each of eight independently sampled concepts.

Pass@k (%), frozen Qwen2.5-32B AG, B=128B=128:

MethodDeepMath hard p@1p@32p@128Omni-MATH-2 hard p@1p@32p@128
Repeated sampling0.26.419.00.13.811.3
Single concept CG, untuned (1 concept)0.88.418.30.24.19.8
Single concept CG, trained, max-of-mean (1 concept)7.917.725.60.74.810.0
Single concept CG, trained, max-of-max (1 concept)5.820.634.00.57.316.4
Single concept CG, trained, max-of-mean (8 concepts)7.924.436.70.98.516.9
Single concept CG, trained, max-of-max (8 concepts)6.224.339.60.68.518.8
trained CG (max-of-mean, paper)2.221.939.20.48.318.6
  • Multiple concepts improve performance at large sampling budgets. Training a CG to emit one concept per trajectory, then sampling eight concepts independently at inference, can achieve pass@128 comparable to our jointly generated concept sets. Spreading the same 128-rollout budget across multiple concepts improves pass@128 over using all rollouts with one concept. A single concept from a trained CG can perform well at small kk, while multiple concepts allow exploration of additional solution directions and make better use of larger sampling budgets.

  • Training a CG to emit one concept reduces the diversity of its independently sampled concepts. The Vendi score, using a cosine kernel, of 8 independently sampled concepts drops from 2.812.81 (untuned) to 2.202.20 after training (cosine kernel). Our trained CG scores 3.033.03 calculated on the first eight concepts of each jointly generated set. The concepts are sampled for the 1k problems on DeepMath hard test set form the paper.

We added this experiment to the appendix.

3. Choice of the 200-step schedule and other methodological choices

The ideal step schedule should indeed be picked by a separate validation dataset, in our case the 200 step was simply the checkpoint reached after about one day of runtime so it allowed for comparison after an equivalent amount of compute between rewards, and we didn't select it specifically as the best point over any separate set. As shown in Figure 7 of Appendix D.9 letting the training run longer does give later checkpoints with occasional higher test scores (the best checkpoint is about 41% pass@128 on the DeepMath val-hard). The RL hyperparameters are standard GRPO settings, identical across all methods and rewards (Appendix C).

We sincerely hope that our responses answer the reviewers questions and satisfy the requests made, and hope you will consider accepting our work.

I appreciate the authors for the clarifications and additional experiments. Some of my concerns (e.g., the exact-match and learned single-hint comparison) have been addressed. I will keep my score.

AuthorsOpenReview ↗

Response to official comment by reviewer mjYZ

We thank the reviewer for their response and for acknowledging that our additional clarifications and experiments address some of their concerns. Given that the discussion period is still open, could the reviewer clarify which concerns remain and what additional evidence or revisions would address them? If our responses have addressed their main concerns, we ask the reviewer to consider raising their score.

Reviewer_CS5bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains a small concept generator to guide a larger, frozen answer generator on mathematical reasoning problems. It generates multiple concepts in one trajectory and uses downstream answer correctness as the RL reward. The trained generator improves pass@k and is also evaluated with a different answer-model family.

Critical strengths
  • This approach improves a larger model without requiring updates to its weights. Notably, the trained 7B concept generator outperforms an untuned 32B generator, suggesting that learning effective guidance may be more beneficial than simply scaling up the generator.

  • Extensive analyses examine the impact of decoding parameters on repeated sampling. The mismatched-concept and concept-count studies further clarify how problem-specific guidance and the use of multiple approaches contribute to the observed gains.

Critical weaknesses
  • The baseline comparisons are insufficient to establish the method’s advantages over existing approaches. Could the authors include stronger learned guidance and direct RL training baselines under matched model capacity and compute budgets? The existing direct-training comparison uses a 7B solver against a system backed by a 32B solver. Could the authors provide comparisons with matched solver capacity and compute budgets to establish the benefit of the proposed concept-generation and reward design?

  • The empirical evaluation is too limited to support the broader claims. It relies on older Qwen2.5 and Llama-3.3 models; do the gains persist with newer, stronger reasoning models? The main benchmarks are also few and confined to mathematics. Could the authors include AIME and GPQA to establish broader applicability?

  • Qwen2.5-32B serves as both the primary answer generator and the correctness judge used for training and evaluation. How reliable are its judgments on these difficult problems? Could the authors validate a subset with an independent judge, symbolic verification, or human inspection, and check whether the gains remain?

Questions for authors

Same as the “Critical Weaknesses” section.

5 comments
AuthorsOpenReview ↗

Response to official comment by reviewer CS5b (1/3)

We appreciate the reviewer's recognition of the value of improving a larger model without updating its weights. We address the three weaknesses below:

1. Baselines comparisons

The main comparison is already capacity and compute matched. Our headline baseline is naive repeated sampling with the same frozen 32B answer generator (AG), at the same budget of B=128B=128 rollouts per problem. The only change is how those rollouts are prompted. It is important to distinguish between two cost questions:

  • Inference cost. Generating the concepts takes one call to the small CG per problem. This is under 0.3%0.3\% of the 128-rollout per problem answer budget (Section 7.3). Concept-guided sampling therefore runs at essentially the same inference compute as the baseline it beats.
  • Training cost. Training the CG is a one time cost and we don't retrain for each evaluation. Training is also optional: the untuned CG already improves over repeated sampling (19.0%→28.9%19.0\% \to 28.9\% pass@128 on our DeepMath hard set), and training adds a further +10.3+10.3 points (39.2%39.2\%).

The main motivation we put forward is that we train (once) a small model to guide and improve the performances of a larger frozen model or a closed source model that might be accessible only through API. The direct-training comparison in Appendix D.1 answers "why not train the small model to solve the problems itself?" It was not meant as a capacity matched baseline.

1.1 Matched solver capacity: Qwen2.5-7B-Instruct as both CG and AG

For completness we add a matched capacity comparison experiment as requested:

Filtering for the 7B. Our theory in Appendix A, predicts that concepts help where the AG success probability is low. Gains are therefore only meaningful on problems that are hard for the AG under evaluation. We filter the datasets for this eval to only keep samples that have low success probability for Qwen2.5-7B-Instruct using the same protocol of the paper's 32B hard sets: 128 rollouts at temperature 1.0, Qwen2.5-32B LLM judge, keeping only problems with 0/128.

  • DeepMath: 1000 held-out problems.
  • Omni-MATH-2: 1460 problems.

All approaches below use a 7B solver. The CG adds one 7B call per problem (under 1%1\% of the budget). Direct RL uses the same RL algoritm and the same judge to provide binary rewards, we pick the oracle checkpoint over the test set DeepMath-7B-hard.

Pass@k (%) with Qwen2.5-7B-Instruct as the solver:

Method (7B solver)DeepMath-7B-hard p@1p@32p@128Omni-MATH-2-7B-hard p@1p@32p@128
Repeated sampling (7B)0.26.317.00.24.813.7
Oracle Direct RL on the 7B4.919.430.71.011.022.4
7B AG + untuned 7B CG0.48.720.10.26.116.0
7B AG + 7B CG trained against the 7B AG0.910.621.40.36.817.5
7B AG + 7B CG trained against the 32B AG***0.36.916.4

(* the numbers are not included because some of the test problems were included in the training set of the 7B CG trained against the 32B AG)

With matched 7B solver capacity, direct RL achieves the highest pass@k at all reported budgets. The trained CG improves the frozen 7B solver's pass@128 from 17.0%17.0\% to 21.4%21.4\% on DeepMath and from 13.7%13.7\% to 17.5%17.5\% on Omni-MATH-2, with CG training adding a further 1.31.3 and 1.51.5 percentage points over untuned guidance, respectively.

When solver updates are feasible, directly training the solver performs better, which is expected. Our main motivation is to train a small CG to improve a larger frozen or API-accessed solver, allowing for better search policy through learned guidance while keeping the solver's weights fixed.

AuthorsOpenReview ↗

Response to official comment by reviewer CS5b (2/3)

2. Broader evaluation: math, physics, and newer models

We evaluated the paper's CG, trained, with the frozen Qwen2.5-32B AG on the full AIME 2024, 2025, and 2026 (pooled, 90 problems) and GPQA-Diamond (198 problems). Values are paired differences vs repeated sampling in pass@k points, with 95% bootstrap CIs over problems.

BenchmarkΔ\Deltap@8Δ\Deltap@32Δ\Deltap@128
AIME 24+25+26 (90)+0.8+0.8 [−1.6-1.6, +3.3+3.3]+1.2+1.2 [−3.1-3.1, +5.6+5.6]+2.2+2.2 [−5.6-5.6, +10.0+10.0]
GPQA-Diamond (198)+3.7+3.7 [+1.7+1.7, +5.8+5.8]+4.8+4.8 [+2.5+2.5, +7.2+7.2]+3.0+3.0 [+0.5+0.5, +6.1+6.1]
  • GPQA: better performance. The CG never saw this science domain questions during training. This supports the claim that the CG learns transferable guidance.
  • AIME: shows positive improvement, but it is within noise. With 90 problems, the 95% CI at pass@128 is about ±8\pm 8 points, so we mostly measure noise due to the dataset size. And importantly, AIME is not in the regime our method targets: Qwen2.5-32B-Instruct baseline already solves 52%52\% of AIME at pass@128, whereas in Appendix A we show that concepts help better in the regime where the model has a very low success probability.

This is also an issue with choosing the newer models that saturate these benchmarks which makes finding a large dataset that contain enough hard problems after filtering challenging. We added 2 new experiments, using GPT-OSS-20B (2025) and Qwen3.5-35B-A3B (2026) on OMNI-MATH-2 and UGPhysics datasets where we first filter each dataset to only keep problems where the model has a low success probability (0/128 correct, UGPhysics is further filtered to keep only problems whose reference answer is a multiple-choice letter, a true/false, or numerical), and obtain:

  • GPT-OSS-20B: 601 math problems and 316 physics problems
  • Qwen3.5-35B-A3B: 402 math problems and 380 physics problems
AG (hard set)Methodp@32p@128
gpt-oss-20b (Omni-hard, 601)Repeated sampling4.713.5
Own concepts (gpt-oss-20b as CG)8.622.5 (+9.0+9.0 [+5.7+5.7, +12.5+12.5])
Untuned Qwen2.5-7B CG6.818.3 (+4.8+4.8 [+1.7+1.7, +8.0+8.0])
Our trained Qwen2.5-7B CG7.118.3 (+4.8+4.8 [+1.7+1.7, +8.2+8.2])
Qwen3.5-35B-A3B (Omni-hard, 402)Repeated sampling3.310.0
Own concepts (Qwen3.5 as CG)5.312.7 (+2.7+2.7 [−0.7-0.7, +6.0+6.0])
Untuned Qwen2.5-7B CG4.510.7 (+0.7+0.7 [−2.5-2.5, +4.0+4.0])
Our trained Qwen2.5-7B CG5.513.4 (+3.5+3.5 [+0.0+0.0, +7.5+7.5])

Pass@k (%) on UGPhysics hard sets:

AG (hard set)Methodp@32p@128
gpt-oss-20b (UGPhysics-hard, 316)Repeated sampling4.914.2
Own concepts (gpt-oss-20b as CG)7.919.6 (+5.4+5.4 [+0.3+0.3, +10.4+10.4])
Untuned Qwen2.5-7B CG7.016.8 (+2.5+2.5 [−1.9-1.9, +7.0+7.0])
Our trained Qwen2.5-7B CG10.423.4 (+9.2+9.2 [+4.7+4.7, +13.9+13.9])
Qwen3.5-35B-A3B (UGPhysics-hard, 380)Repeated sampling2.27.4
Own concepts (Qwen3.5 as CG)4.910.5 (+3.2+3.2 [+0.0+0.0, +6.1+6.1])
Untuned Qwen2.5-7B CG5.813.2 (+5.8+5.8 [+2.9+2.9, +8.7+8.7])
Our trained Qwen2.5-7B CG8.716.6 (+9.2+9.2 [+5.8+5.8, +12.6+12.6])
Qwen2.5-32B (UGPhysics-hard, 434)Repeated sampling5.215.2
Own concepts (untuned Qwen2.5-32B as CG)7.717.1 (+1.8+1.8 [−2.3-2.3, +6.0+6.0])
Untuned Qwen2.5-7B CG7.117.3 (+2.1+2.1 [−2.1-2.1, +6.2+6.2])
Our trained Qwen2.5-7B CG9.019.4 (+4.1+4.1 [+0.0+0.0, +8.3+8.3])

The trained CG is used unchanged: it was trained only on DeepMath against Qwen2.5-32B and never saw these models nor physics domain. Every evaluation uses fresh samples under a new seed, so repeated sampling is not exactly 0%0\% on these hard sets (Appendix D.6).

  • Physics: the trained CG performs best for all three AGs. For both newer models it adds +9.2+9.2 points pass@128, with confidence intervals well above zero, and already +5.5+5.5 and +6.5+6.5 points at pass@32. It also beats the concepts that gpt-oss-20b and Qwen3.5 write for themselves (+5.4+5.4 and +3.2+3.2): a 7B model trained to give guidance helps more than the larger models that were not trained on this task.
  • Mathematics: on Qwen3.5 the trained CG is again the best (+3.5+3.5 at pass@128). On gpt-oss-20b it gives a significant +4.8+4.8, but here the model's own concepts are better (+9.0+9.0), and the untuned 7B CG matches the trained one.

Overall, concept guidance helps the newer reasoning models on their own hard problems in both domains. The trained CG transfers across model families and reasoning domains.

AuthorsOpenReview ↗

Response to official comment by reviewer CS5b (3/3)

3. Reliability of the Qwen2.5-32B-Instruct as judge

It is true that Qwen2.5-32B serves both to generate the answer rollouts and to judge the final answers, and it is mostly for 2 reasons:

  • The first is because the rule base was failing to correctly grade a lot of responses. Examples are shown in the table below:
Ground truthModel final answerrule basedLLM judge
No\text{No}01
Yes\text{true}01
e^{\frac{2}{e}}e^{2/e}01
\sinh 1\sinh(1)01
\frac{1}{2}(7-3\sqrt{5})\frac{7 - 3\sqrt{5}}{2}01
-2010012 / 2010013-\frac{2010012}{2010013}01

So we therefore opted to use an LLM as a judge. But the judge only sees the problem statement (for context), the ground-truth answer, and the final answer extracted from the rollout's \boxed{}. It never sees the reasoning, so it doesn't judge the entire chain of thought. Using this version of LLM as a judge allows us to avoid incorrectly penalizing trajectories and further introducing noise.

  • The second reason why we use the same model as the answer generator is practical. As using a separate capable judge would need its own GPUs in an already expensive RL loop. Using the same frozen model as judge allow us to do this efficiently.

Rule Based evaluation We re-evaluate using rule based which is exact match EM (with some string normalization like \dfrac to \frac etc) and we also redo the training using EM instead of LLM as a judge.

Pass@128 (%), frozen Qwen2.5-32B AG:

MethodDeepMath hard EMLLM judgeOmni-MATH-2 hard EMLLM judge
Repeated sampling18.019.08.511.3
Untuned 7B CG24.828.911.915.0
Untuned 32B CG28.733.811.815.3
Trained 7B CG, EM reward29.135.113.918.2
Trained 7B CG, judge reward (paper)32.739.214.318.6

The conclusions from the paper do not change under exact match scoring, and there is no evidence of judge exploitation during the training of the CG. Moreover as we noted, the EM trained concept generator performs worse compared to the LLM judge trained CG even when both are scored by exact match. This is consistent with EM false negatives making the training reward noisier and slowing learning.

Summary of contributions

Concepts or hint-guided sampling exists in prior work. Our contributions are what makes it principled and trainable:

  1. An analysis of when conditioning helps pass@k. Concept-conditioned pass@k (or passcpass^c@k as we introduced it) depends on the geometric mean of per-concept failure probabilities. Our analysis in Appendix A motivates focusing on problems with low baseline success probability, where there is greater headroom for improvement. The new results confirm this prediction: the trained CG improves pass@128 on every model's hard set, significantly in most cases, while gains on AIME, which is not in this regime, are mostly noise.
  2. A correct pass@k estimator for concept-conditioned sampling. We give the exact unbiased estimator for split budgets and state when it is close and when it coincides with the unbiased estimator of Chen et al.
  3. RL training of the CG against a frozen AG, with rewards whose targets we characterize. Max-of-max is unbiased for concept conditioned passcpass^c@k but has higher variance. Max-of-mean is a lower variance biaised reward that targets the oracle concept.
  4. Evidence that the approach works at essentially no extra inference cost (<0.3%) and generalizes beyond its training setup: across datasets (Omni-MATH-2, and the now added GPQA, physics), across AG families (Llama-3.3-70B, and the now added gpt-oss-20b and Qwen3.5 experiments), and under independent rule based grading (which we add in the paper Appendix).

Changes in the revised version are marked in blue. We hope these additions address the reviewer's concerns about baselines, scope and judging. We are happy to discuss anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.