- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work explores how to improve LLM reasoning beyond repeated sampling, which often produces redundant duplicate solution attempts. The authors propose to generate multiple problem-specific concepts in a single trajectory and use them to guide a frozen answer model. They further train a small concept generator with reinforcement learning, rewarding concepts according to downstream answer success. On difficult mathematical problems, the trained 7B generator raises a 32B model’s pass@128 from 19.0% to 39.2%, outperforming larger untrained concept generators.
Critical strengths
The paper identifies that GuidedSampling’s reported gains largely result from an overly restrictive repeated-sampling baseline, and thus proposes an improved approach of concept-guided sampling. Furthermore, it shows that a finetuned small concept-generator can effectively guide the reasoning of a much larger model.
Critical weaknesses
-
Narrow and outdated experimental scope. The main RL experiments are only limited to two mathematical reasoning benchmarks, DeepMath and Omni-MATH 2, one of which even shares the same distribution with the training data, and it's hard to the effectiveness on general math reasoning. And the selected LLM backbones are Qwen2.5 and Llama-3.3-70B, both released in 2024. It is therefore unclear whether the method remains effective for newer reasoning models or tasks beyond mathematics.
-
Limited evidence of out-of-distribution generalization. When synthesizing the RL data, Qwen2.5-32B is used to generate the answer rollouts that determine the RL reward, and to judge answer correctness. Consequently, the concept generator (CG) learns to exploit Qwen2.5-32B’s specific reasoning priors, failure modes, prompt sensitivities, and judge preferences. Therefore, all evaluation using DeepMath benchmark or Qwen2.5-32B as the answer generator (AG) can be considered in-domain. The only OOD setting that changes both the dataset and answer-model family (Llama-3.3-70B as AG on Omni-MATH 2 in Table 4) reduces pass@1 from 1.92% to 1.73%, while improving pass@128 only modestly from 12.50% to 13.97%.
-
Weak baseline coverage. The paper lacks against closely related approaches. For example, Step-Back Prompting[1] first elicits high-level principles to guide further reasoning. Diversity of Thought[2] generates and ensembles diverse problem-solving approaches before reasoning. Tree of Thoughts[3] explores and evaluates multiple reasoning branches. These methods have reported strong reasoning improvements in their respective settings. It seems that the idea of concept guidance in this ppaer has little difference from the ideas of finding a good high-level principle to initialize reasoning in the related works. So I prefer seeing the comparison with these approaches in methodology and performance.
[1]: Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
[2]: Diversity of Thought Improves Reasoning Abilities of LLMs
[3]: Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Questions for authors
1. What exactly is a ‘concept‘, and how is it distinguished from a partial solution?
- Appendix C.3 provides only a syntactic definition. However, the examples in Appendix D.10 vary substantially in semantic content. Some are mathematical principles or theorem names, while others provide a detailed solution guideline, and even multiple intermediate reasoning steps. Moreover, the trained generator produces substantially more concrete derivations than the untrained generator. How can the authors distinguish whether the CG learns to generate high-level concepts or instead learns substantial parts of the solution trajectories required by the AG?
- In addition, how are the average concept counts in Tables 1 and 2 calculated, and how to define a duplicate concept?
2. Can the trained CG output effective concept guide to newer and stronger answer models? A central claim is that a small CG learns a reusable search policy for larger models. Could the authors evaluate the trained CG with more recent closed-source models, such as GPT-5.4/5.5, or Claude Opus 4.6/4.7? Such experiments would help determine whether the generated concepts are broadly useful or primarily adapted to the reasoning behavior and failure modes of Qwen2.5-32B.
3. In the fully OOD cross-model setting in Table 4, the improvement only appears at pass@128 and pass@64. In other words, I need to rollout dozens of times to reach the correct answer. I wonder with so many rollout outputs, how would the system identify and return the correct answer when the ground truth is unavailable?
5 comments
Response to official comment by reviewer T6tY (2/4)
W2. Out-of-distribution generalization and the judge correctness
We use an LLM judge because rule based matching rejects many correct answers written in a different form. Real examples from our baseline rollouts that exact match (EM, with string normalization) marks wrong: \text{No} for No, e^{2/e} for e^{\frac{2}{e}}, \sinh(1) for \sinh 1, \frac{7-3\sqrt5}{2} for \frac12(7-3\sqrt5).
So we therefore opted to use an LLM as a judge. But the judge only sees the problem statement (for context), the ground-truth answer, and the final answer extracted from the rollout's \boxed{}. Rollouts without a boxed answer are marked incorrect without consulting the judge. Thus the judge never sees the reasoning, so it doesn't judge the entire chain of thought. Using this version of LLM as a judge allows us to avoid incorrectly penalizing trajectories and further introducing noise.
Rule-based evaluation and a CG trained without the judge. We re-graded every rollout with EM, and we also retrained the CG with the EM reward instead of the judge.
Pass@128 (%), frozen Qwen2.5-32B AG:
| Method | DeepMath hard EM | LLM judge | Omni-MATH-2 hard EM | LLM judge |
|---|---|---|---|---|
| Repeated sampling | 18.0 | 19.0 | 8.5 | 11.3 |
| Untuned 7B CG | 24.8 | 28.9 | 11.9 | 15.0 |
| Untuned 32B CG | 28.7 | 33.8 | 11.8 | 15.3 |
| Trained 7B CG, EM reward | 29.1 | 35.1 | 13.9 | 18.2 |
| Trained 7B CG, judge reward (paper) | 32.7 | 39.2 | 14.3 | 18.6 |
This evidence goes against the claim that the CG "learns to exploit Qwen2.5-32B's [...] judge preferences". The judge-trained CG retains its advantage under independent EM scoring, and it is also better under EM scoring ( vs on DeepMath, vs on Omni-MATH-2). Its stronger performance than the EM-trained CG is consistent with false negatives in EM rewards making the training reward noisier.
Crucially, the conclusions from the paper do not change under exact match scoring. We include this experiment in the Appendix.
On the Llama-3.3-70B result. The method targets exploration: finding at least one valid solution within a sampling budget on problems the model essentially never solves. Gains are therefore expected mostly at moderate to large , as predicted by Appendix A. A change at pass@1 within noise is consistent with this goal. With the newer models above, gains already appear at pass@32 ( and points on UGPhysics).
Response to official comment by reviewer T6tY (3/4)
W3. Comparison with Step-Back Prompting, Diversity of Thought, and Tree of Thoughts
All three methods aim to improve reasoning with intermediate structure. They differ from ours in what they optimize and in how they scale to large sampling budgets.
- Step-Back Prompting makes the model first state the principles behind the given question, then solve it grounded on them. For math (GSM8K), the paper does this in a single call with a one-shot exemplar whose solution first states the principles it applies, decoded greedily and evaluated by accuracy. The principles are problem-specific, but they come from the solver itself, are not trained, and each answer gets a single abstraction. We use the same single call prompt format (ask the model to first recall principles then answer) with a zero-shot instruction, sampled at with the same frozen Qwen2.5-32B AG and the same 128-rollout budget:
| Method (frozen Qwen2.5-32B AG) | DeepMath hard p@1 | p@32 | p@128 | Omni-MATH-2 hard p@1 | p@32 | p@128 |
|---|---|---|---|---|---|---|
| Repeated sampling | 0.2 | 6.4 | 19.0 | 0.1 | 3.8 | 11.3 |
| Step-Back prompting (zero-shot, single call) | 0.4 | 9.3 | 24.3 | 0.2 | 3.9 | 11.6 |
| Untuned Qwen2.5-7B CG (no training) | 0.8 | 13.7 | 28.9 | 0.3 | 5.9 | 15.0 |
| Trained Qwen2.5-7B CG | 2.2 | 21.9 | 39.2 | 0.4 | 8.3 | 18.6 |
Our untuned CG has better performance than Step-Back at every reported budget. Step-Back improves DeepMath performance (comparably to generic prompt modification baseline) but provides little improvement on Omni-MATH-2. Each Step-Back rollout generates its principles independently, without access to those proposed in other rollouts. Diversity across these principles therefore relies on token noise sampling. Our CG generates concepts jointly in one trajectory, so later concepts are conditioned on earlier ones, allowing the model to explicitly diversify the set.
-
Diversity of Thought works at the task level, not at the problem level. It queries an LLM for approaches on random questions of a benchmark, keeps the most frequent ones, selects the best 3–5 approach/persona prompts on a validation set, and reuses these same prompts for every problem of the task. Answers are decoded greedily and aggregated by majority vote, so the approach targets ensemble accuracy of a 3–5-call ensemble, not coverage under a large sampling budget. Our untuned CG is the analogue of this idea on the problem level: the LLM proposes approaches for each problem, and answers are conditioned on them.
-
Tree of Thoughts searches a tree of partial solutions where the same LLM proposes the candidate next steps and also scores intermediate states, either by valuing each state or by voting for the most promising one among several. Its main tasks are Game of 24, Creative Writing and Mini Crosswords. The biggest limitation is that the search is sequential: each level waits for generation and evaluator calls on the previous one, so it is not inference friendly. On GSM8K, zero-shot ToT samples five strategies and votes for the best one, then samples five solutions conditioned on that strategy and votes for the best solution. Like our untuned approach, it uses high-level strategies to guide solution generation. Our method keeps the inference pattern of repeated sampling: one extra small-model call per problem (0.3% overhead in Section 7.3), then fully parallel rollouts.
We also compare against a trained single-concept generator variant of our approach, in the spirit of DSP/RLAD, in our response to Reviewer mjYZ section 2.
Response to official comment by reviewer T6tY (1/4)
We thank the reviewer for the feedback. Below, we address the concerns and questions with additional experiments and clarifications.
W1. Experimental scope: newer models and tasks beyond mathematics
We started from the same models as the prior GuidedSampling work of Handa et al., so that our re-evaluation in Section 3 is directly comparable, and we use mathematics because it is a verifiable domain that is usually used in RL as answers can be scored to obtain a training signal.
Our analysis (Appendix A) predicts that concepts have more room to improve the performance where the answer generator (AG) has a low success probability, and Section 4 confirms it empirically. Evaluating a newer model therefore requires a benchmark that is hard for that model and still large enough after filtering to give a statistically meaningful signal. Small or saturated benchmarks mostly measure noise.
We have now added three new sets of experiments on benchmarks and models in response to your request:
(a) GPQA-Diamond and AIME.
We evaluate the paper's CG, trained with the frozen Qwen2.5-32B AG on the full AIME 2024, 2025, and 2026 (pooled, 90 problems) and GPQA-Diamond (198 problems). Values are paired differences vs repeated sampling in pass@k points, with 95% bootstrap CIs over problems.
(b) Two recent reasoning models, gpt-oss-20b (2025) and Qwen3.5-35B-A3B (2026), on hard mathematics.
(c) Other domain: physics (UGPhysics).
These settings change both the dataset and the answer model family, which is the fully out of distribution experiment requested. And this setting shows that we effectively learn a transferable policy that performs better overall to propose concepts than using other bigger and newer models that were not trained to do this.