- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RACS, a prompt-selection criterion for GRPO training. For each prompt, it samples several responses and measures how directionally aligned their hidden-state vectors are. To avoid scoring the whole corpus, consistency is computed on a small reference set each epoch and propagated to the rest via nearest-neighbour regression over frozen prompt embeddings, and prompts with high estimated consistency are favored for training.
On four Qwen models and five math benchmarks under a fixed update budget, RACS improves pooled accuracy by 1–3 points over random selection and the strongest baseline. Analyses show that the estimator preserves coarse score rankings, that consistency is largely independent of pass-rate difficulty, and that high-consistency prompts have more separable correct and incorrect response representations.
Critical strengths
-
Clear presentation, simple idea, and seems to be better than pure GRPO. The method is simple and easy to understand, and it improves over GRPO with random selection. The experimental setup is good in terms of model coverage: results on two model families (Qwen2.5-Math and Qwen3) and multiple scales show that the effect generalizes across both.
-
Well written and well structured. The paper provides both interesting empirical results and intuitive justification for why consistency might matter. The lack of theoretical justification is not a concern for me.
Critical weaknesses
- The evidence that consistency-based selection helps is not yet convincing in terms of empirical setup:
- The pilot study (Figure 1) is a single run per group with no error bars. The high, medium and low curves overlap for most of training and separate by only a few points at the end, which is within typical GRPO run-to-run variation. Since this figure motivates the whole paper, it needs repeated runs or a significance test. Note that I only ask this for Figure 1, not the main experiments.
- The baselines are in comprehensive. DOTS+RR scores below random baseline, which is very very strange to me. Representative prompt-selection methods are also missing: DAPO[1] and GRESO[2].
- Compute is not matched. Budgets match update steps, not rollouts (Appendix A). RACS spends extra rollouts on the reference set and on rejected candidates, so part of the gain may come from more sampling.
- Only math is evaluated. Code generation and scientific reasoning are not considered, so the generality of the signal is untested.
- No ablation of the representation. The score uses the last-token hidden state from three fixed layers. There is no ablation over layer choice, token position or pooling, nor a comparison with a simple answer-agreement score, so it is unclear whether this specific representation matters.
- The explanation for why high consistency helps is unconvincing and is contradicted by the paper's own analysis.
- Intuitively, I would think high-consistency prompts should be less informative. If a prompt is easy and the model answers it correctly in all responses, the response contents will be similar and the consistency score will be high. In that case the prompt carries little learning signal. The paper does not explain why prompts with such similar responses are better for training.
- Figure 4(b) is strange. I would expect RACS to be higher for moderate prompts than for hard and easy ones. Instead, the RACS distributions are essentially the same across all three groups. This implies RACS has no correlation with prompt difficulty at all, which is hard to reconcile with the claim that it identifies prompts useful for learning.
- Line 469 contradicts Figure 4b. The paper explains that high consistency reflects prompts for which the model already has a basic solution strategy. By that logic, easy prompts should receive the highest consistency scores, even more so than moderate prompts, so the method would favor easy prompts. This directly contradicts both the flat distributions in Figure 4(b) and the claim that RACS selects useful training prompts.
REFERENCES:
[1] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.
[2] Zheng, Haizhong, Yang Zhou, Brian Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. "Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts." Advances in Neural Information Processing Systems 38 (2026): 124321-124346.
Questions for authors
See weakneses
7 comments
Part 2-3
Q6: The explanation for why high consistency helps is unconvincing and is contradicted by the paper's own analysis.
A6: Four examples from recorded training rollouts illustrate why final-answer correctness and response consistency need not coincide.
1. High pass rate and high consistency need not coincide
(i) High pass rate does not imply similar reasoning or a high RACS selection score
Case A1 — Octahedron volume
Low RACS: . Correct responses: .
Question: A regular octahedron is formed by joining the centers of adjoining faces of a cube. What is the ratio of the octahedron's volume to the cube's volume?
(A) ; (B) ; (C) ; (D) ; (E) .
Correct answer: .
| Responses | Reward per response | Reasoning and outcome |
|---|---|---|
| R1, R2, R5 | 1 | Use the correct edge length and the octahedron volume formula to obtain . |
| R4, R6 | 1 | R4 initially gives the edge length as , but then uses the correct pyramid dimensions. R6 calls “half the cube side,” although its calculation is correct. Both give the correct final answer despite errors in their geometric explanations. |
| R3 | 0 | First obtains from an incorrect base size. It later derives , but mistakenly says this answer is absent from the options and selects . |
| R7 | 1 | Uses incorrect dimensions to obtain , then changes this to the correct answer through invalid equalities. |
| R8 | 0 | First obtains , then repeatedly recomputes using incompatible dimensions and formulas. The saved response ends without a final boxed answer. |
The group mixes valid solutions, conflicting geometric assumptions and invalid derivations. Its success rate therefore masks substantial variation in reasoning, alongside a low RACS.
Case A2 — Inscribed squares
Low RACS: . Correct responses: .
Question: A square is inscribed in an isosceles right triangle in two ways. In the first configuration, two sides of the square lie along the triangle's legs, and its area is . In the second, one side lies on the hypotenuse and the opposite two vertices lie on the legs. What is the area of the second square?
(A) ; (B) ; (C) ; (D) ; (E) .
Correct answer: .
| Responses | Reward per response | Reasoning and outcome |
|---|---|---|
| R1 | 0 | Uses an incorrect geometric relation, eventually obtains an area of about , and then chooses without justification. |
| R2, R7 | 1 | Incorrect geometric relations lead to in R2 and in R7. Both select as the “closest” option, even though is closer. |
| R3, R5 | 1 | R3 incorrectly turns into . R5 incorrectly infers from . Both eventually select without deriving it. |
| R4, R6 | 1 | Use contradictory dimensions and areas. R4 reaches and R6 reaches , but both then assert that the answer is without a valid derivation. |
| R8 | 1 | Treats drawing coordinates as physical lengths, then asserts the false equality . The left-hand side is approximately . |
Seven of eight final answers are correct, yet none of the eight written solutions validly derives . Different geometric errors lead to unsupported correct answers. This case shows that a high observed pass rate can coexist with varied, unreliable reasoning and a low recorded selection score.
Part 3-3
(ii) A shared approach and a high predicted RACS do not guarantee a high pass rate
Case B1 — Integer divisors
High RACS: . Correct responses: .
Question: How many integer divisors does have?
Correct answer: .
| Responses | Reward per response | Reasoning and outcome |
|---|---|---|
| R1, R3–R8 | 0 | Count only and and answer , overlooking and . They mistake the positive divisors for all integer divisors. |
| R2 | 1 | Includes the negative divisors, correctly lists , and answers . |
All eight responses identify and count divisors, but seven apply the same mistaken restriction to positive divisors. A shared approach can therefore preserve a conceptual error; here, it coexists with a high predicted RACS and only correct answers.
Case B2 — Polar-curve asymptote
High RACS: . Correct responses: .
Question: Find the equation of the asymptote of the graph of .
Correct answer: .
All eight responses attempt a similar broad plan: rewrite the equation, examine limiting behavior, and infer a Cartesian asymptote.
| Responses | Reward per response | Reasoning and outcome |
|---|---|---|
| R1, R3, R6 | 0 | Rewrite: Obtain . Limits: Examine , where diverges. Asymptote: Incorrectly set to zero by direct substitution and answer . |
| R2, R8 | 0 | Rewrite: R2 incorrectly replaces with ; R8 incorrectly simplifies . Limits: Both examine . R2 obtains from its incorrect equation, while R8 considers divergent . Asymptote: Both incorrectly infer from the limiting direction. |
| R4 | 0 | Rewrite: Starts from , but introduces an extra factor of when converting to Cartesian coordinates. Limits: Mistakenly treats as a condition for divergence, although at those angles. Asymptote: Uses those angles to incorrectly infer . |
| R7 | 0 | Rewrite: Obtains and a correct implicit Cartesian equation. Limits: Identifies , but also incorrectly drops the terms involving when analyzing the Cartesian equation at large . Asymptote: Returns to the limiting-angle argument and incorrectly concludes . |
| R5 | 1 | Rewrite: Obtains . Limits: Examines and simplifies before taking the limit. Asymptote: Correctly obtains , although its statements about infinite limits still contain errors. |
These responses follow a similar broad plan, but make different errors in the transformations and limits. In particular, a vertical limiting direction does not determine the asymptote's position. The high predicted score and recurring approach therefore coexist with only correct final answers.
2. Responses to the specific concerns
(1) Does high consistency imply a high pass rate?
Not necessarily. A shared solution approach can still contain conceptual errors or be applied incorrectly. In B1 and B2, the predicted RACS scores are and , respectively, yet both groups have an observed pass rate of only . In B1, although all responses follow the same broad approach of identifying and counting the divisors of , most overlook the negative divisors. In B2, the responses follow a similar broad plan but make errors in transformations and limits. These cases illustrate that high predicted consistency can coexist with low observed correctness.
These cases also help interpret Figure 4(b), where difficulty is measured by pass rate. A1–A2 illustrate that correct answers can involve varied reasoning. B1–B2 illustrate that a shared approach can still produce mostly incorrect answers. A higher pass rate therefore does not necessarily imply a higher RACS score. The similar distributions in Figure 4(b) do not contradict this point.
(2) Does the “basic solution strategy” explanation contradict Figure 4(b)?
By “basic solution strategy,” we mean a shared approach, not a fully mastered solution. In B1, the responses identify divisors but mostly misinterpret which divisors to count. In B2, they use a similar broad approach but make errors in transformations and coordinate limits. A shared approach can therefore coexist with low accuracy, because following the same basic solution strategy does not guarantee that every step is carried out correctly.
Part 1-3
Thank you for your feedback. Given rebuttal time and compute limits, we first address the concerns covered by our completed experiments.
All additional training experiments below use Qwen2.5-Math-1.5B.
Q1: The pilot study (Figure 1) is a single run per group and needs repeated runs.
A1: We repeated the high- and low-consistency runs from Figure 1 with seed 43. High-RACS selection remains 3.80 percentage points ahead, reproducing the high-over-low ordering at the end of training.
Q2: DOTS+RR scores below Random, which is strange. Representative prompt-selection methods DAPO and GRESO are also missing.
A2: We examined DOTS+RR's saved rollouts across 60 steps. Its share of all-correct or all-incorrect groups averaged 18.33%, versus 24.96% for Random, suggesting that difficulty-targeted selection reduces zero-advantage groups as intended. However, fewer such groups do not guarantee higher final accuracy. The original DOTS+RR paper (§4.3) notes that replaying historical rollouts can introduce off-policy bias, which may partly explain the lower accuracy in our setting.
We also evaluated DAPO and GRESO under the original 60-update budget. RACS outperforms them by 1.49 and 1.25 percentage points, respectively.
Q3: Budgets match update steps, not rollouts. RACS spends extra rollouts on the reference set and on rejected candidates.
A3: To test whether additional GPU time closes the accuracy gap, we extended training for DAPO and GRESO, our two strongest reproduced baselines. Each ran to the first completed update at or beyond RACS's GPU-hour budget. RACS remains higher by 1.25 and 0.79 percentage points, respectively.
All methods used the same resource budget of one A800 GPU. Timing covers the full selection and training pipeline; for RACS, this includes reference generation/scoring, rejected rollouts, activation extraction, prompt encoding, and updates.
Q4: The score uses the last-token hidden state. There is no ablation over pooling.
A4: Replacing last-token representations with mean pooling retains an advantage of 0.92 percentage points over Random with the same filter.
Q5: There is no comparison with a simple answer-agreement score.
A5: We changed only the selection score from RACS to pass rate: measured reference-set pass rates were propagated to other prompts by the unchanged activation-similarity kNN estimator, followed by the same sampling, filtering, rollout reuse, and training. Pooled accuracy fell by 1.86 percentage points.