acceptodds
Under review as a conference paper at ICLR 2027

RACS: Response Activation Consistency-guided Data Selection for Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning abilities of language models. Training requires many sampled responses, so selecting useful prompts is important for making effective use of the training budget. A common approach to RLVR data selection relies on prompt difficulty estimated from pass rates. Pass rate summarizes response correctness but does not capture the current policy’s behavior during generation. Unlike prior work, we study response activation consistency, the directional agreement among hidden-state vectors of multiple responses sampled from the current policy for the same prompt. In our pilot study, training on prompts with higher response activation consistency yields better test accuracy. Inspired by this, we propose Response Activation Consistency-guided Data Selection, which combines response activation consistency scoring, reference-based score estimation, and consistency-guided prompt sampling. Each epoch, it measures consistency on a small reference set containing fewer than 2% of the corpus, estimates scores for the rest, and favors high-scoring prompts for training. On five mathematical reasoning benchmarks, our method improves pooled accuracy on Qwen2.5-Math-1.5B by 3.24 percentage points over random selection and 2.92 points over the strongest evaluated baseline under the same training-update budget. Further analysis shows that high-consistency prompts tend to have more separable correct and incorrect response representations, suggesting a possible explanation for the observed gains. Our work identifies response activation consistency across sampled responses as a useful signal for RLVR data selection and provides a practical procedure for using it during training.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning with Verifiable Rewards, Data Selection, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_b74jOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RACS, a prompt-selection criterion for GRPO training. For each prompt, it samples several responses and measures how directionally aligned their hidden-state vectors are. To avoid scoring the whole corpus, consistency is computed on a small reference set each epoch and propagated to the rest via nearest-neighbour regression over frozen prompt embeddings, and prompts with high estimated consistency are favored for training.

On four Qwen models and five math benchmarks under a fixed update budget, RACS improves pooled accuracy by 1–3 points over random selection and the strongest baseline. Analyses show that the estimator preserves coarse score rankings, that consistency is largely independent of pass-rate difficulty, and that high-consistency prompts have more separable correct and incorrect response representations.

Critical strengths
  1. Clear presentation, simple idea, and seems to be better than pure GRPO. The method is simple and easy to understand, and it improves over GRPO with random selection. The experimental setup is good in terms of model coverage: results on two model families (Qwen2.5-Math and Qwen3) and multiple scales show that the effect generalizes across both.

  2. Well written and well structured. The paper provides both interesting empirical results and intuitive justification for why consistency might matter. The lack of theoretical justification is not a concern for me.

Critical weaknesses
  1. The evidence that consistency-based selection helps is not yet convincing in terms of empirical setup:
  • The pilot study (Figure 1) is a single run per group with no error bars. The high, medium and low curves overlap for most of training and separate by only a few points at the end, which is within typical GRPO run-to-run variation. Since this figure motivates the whole paper, it needs repeated runs or a significance test. Note that I only ask this for Figure 1, not the main experiments.
  • The baselines are in comprehensive. DOTS+RR scores below random baseline, which is very very strange to me. Representative prompt-selection methods are also missing: DAPO[1] and GRESO[2].
  • Compute is not matched. Budgets match update steps, not rollouts (Appendix A). RACS spends extra rollouts on the reference set and on rejected candidates, so part of the gain may come from more sampling.
  • Only math is evaluated. Code generation and scientific reasoning are not considered, so the generality of the signal is untested.
  • No ablation of the representation. The score uses the last-token hidden state from three fixed layers. There is no ablation over layer choice, token position or pooling, nor a comparison with a simple answer-agreement score, so it is unclear whether this specific representation matters.
  1. The explanation for why high consistency helps is unconvincing and is contradicted by the paper's own analysis.
  • Intuitively, I would think high-consistency prompts should be less informative. If a prompt is easy and the model answers it correctly in all responses, the response contents will be similar and the consistency score will be high. In that case the prompt carries little learning signal. The paper does not explain why prompts with such similar responses are better for training.
  • Figure 4(b) is strange. I would expect RACS to be higher for moderate prompts than for hard and easy ones. Instead, the RACS distributions are essentially the same across all three groups. This implies RACS has no correlation with prompt difficulty at all, which is hard to reconcile with the claim that it identifies prompts useful for learning.
  • Line 469 contradicts Figure 4b. The paper explains that high consistency reflects prompts for which the model already has a basic solution strategy. By that logic, easy prompts should receive the highest consistency scores, even more so than moderate prompts, so the method would favor easy prompts. This directly contradicts both the flat distributions in Figure 4(b) and the claim that RACS selects useful training prompts.

REFERENCES:

[1] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.

[2] Zheng, Haizhong, Yang Zhou, Brian Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. "Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts." Advances in Neural Information Processing Systems 38 (2026): 124321-124346.

Questions for authors

See weakneses

7 comments

Thank you for your feedback. Given rebuttal time and compute limits, we first address the concerns covered by our completed experiments.

All additional training experiments below use Qwen2.5-Math-1.5B.

Q1: The pilot study (Figure 1) is a single run per group and needs repeated runs.

A1: We repeated the high- and low-consistency runs from Figure 1 with seed 43. High-RACS selection remains 3.80 percentage points ahead, reproducing the high-over-low ordering at the end of training.

SelectionGSM8KMATH500MinervaAIME24AMC23All (2,161)
High-RACS86.7377.0027.5713.3350.0075.34
Low-RACS83.9370.6023.9010.0045.0071.54

Q2: DOTS+RR scores below Random, which is strange. Representative prompt-selection methods DAPO and GRESO are also missing.

A2: We examined DOTS+RR's saved rollouts across 60 steps. Its share of all-correct or all-incorrect groups averaged 18.33%, versus 24.96% for Random, suggesting that difficulty-targeted selection reduces zero-advantage groups as intended. However, fewer such groups do not guarantee higher final accuracy. The original DOTS+RR paper (§4.3) notes that replaying historical rollouts can introduce off-policy bias, which may partly explain the lower accuracy in our setting.

We also evaluated DAPO and GRESO under the original 60-update budget. RACS outperforms them by 1.49 and 1.25 percentage points, respectively.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS86.8176.4026.8413.3360.0075.34
DAPO86.1373.0025.7410.0055.0073.85
GRESO86.5873.2024.6310.0057.5074.09

Q3: Budgets match update steps, not rollouts. RACS spends extra rollouts on the reference set and on rejected candidates.

A3: To test whether additional GPU time closes the accuracy gap, we extended training for DAPO and GRESO, our two strongest reproduced baselines. Each ran to the first completed update at or beyond RACS's GPU-hour budget. RACS remains higher by 1.25 and 0.79 percentage points, respectively.

All methods used the same resource budget of one A800 GPU. Timing covers the full selection and training pipeline; for RACS, this includes reference generation/scoring, rejected rollouts, activation extraction, prompt encoding, and updates.

MethodUpdatesGPU-hoursGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS6036.2586.8176.4026.8413.3360.0075.34
DAPO (extended)78≥36.25\geq 36.2586.7372.2027.2110.0047.5074.09
GRESO (extended)67≥36.25\geq 36.2586.2876.2025.7413.3345.0074.55

Q4: The score uses the last-token hidden state. There is no ablation over pooling.

A4: Replacing last-token representations with mean pooling retains an advantage of 0.92 percentage points over Random with the same filter.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS (original last token)86.8176.4026.8413.3360.0075.34
RACS (mean pooling)84.6976.0026.1013.3342.5073.53
Random + identical filtering and rollout reuse85.8268.8025.7413.3347.5072.61

Q5: There is no comparison with a simple answer-agreement score.

A5: We changed only the selection score from RACS to pass rate: measured reference-set pass rates were propagated to other prompts by the unchanged activation-similarity kNN estimator, followed by the same sampling, filtering, rollout reuse, and training. Pooled accuracy fell by 1.86 percentage points.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS (original last token)86.8176.4026.8413.3360.0075.34
Pass-rate selection86.5071.6025.746.6742.5073.48

Q6: The explanation for why high consistency helps is unconvincing and is contradicted by the paper's own analysis.

A6: Four examples from recorded training rollouts illustrate why final-answer correctness and response consistency need not coincide.

1. High pass rate and high consistency need not coincide

(i) High pass rate does not imply similar reasoning or a high RACS selection score

Case A1 — Octahedron volume

Low RACS: 0.8483510.848351. Correct responses: 6/86/8.

Question: A regular octahedron is formed by joining the centers of adjoining faces of a cube. What is the ratio of the octahedron's volume to the cube's volume?

(A) 312\frac{\sqrt{3}}{12}; (B) 616\frac{\sqrt{6}}{16}; (C) 16\frac{1}{6}; (D) 28\frac{\sqrt{2}}{8}; (E) 14\frac{1}{4}.

Correct answer: 16\frac{1}{6}.

ResponsesReward per responseReasoning and outcome
R1, R2, R51Use the correct edge length a/2a/\sqrt{2} and the octahedron volume formula to obtain 1/61/6.
R4, R61R4 initially gives the edge length as a/2a/2, but then uses the correct pyramid dimensions. R6 calls a/2a/\sqrt{2} “half the cube side,” although its calculation is correct. Both give the correct final answer 1/61/6 despite errors in their geometric explanations.
R30First obtains 1/121/12 from an incorrect base size. It later derives 1/61/6, but mistakenly says this answer is absent from the options and selects 2/8\sqrt{2}/8.
R71Uses incorrect dimensions to obtain 2/32/3, then changes this to the correct answer 1/61/6 through invalid equalities.
R80First obtains 1/121/12, then repeatedly recomputes 2/24\sqrt{2}/24 using incompatible dimensions and formulas. The saved response ends without a final boxed answer.

The group mixes valid solutions, conflicting geometric assumptions and invalid derivations. Its 6/86/8 success rate therefore masks substantial variation in reasoning, alongside a low RACS.

Case A2 — Inscribed squares

Low RACS: 0.6927750.692775. Correct responses: 7/87/8.

Question: A square is inscribed in an isosceles right triangle in two ways. In the first configuration, two sides of the square lie along the triangle's legs, and its area is 441 cm2441\,\mathrm{cm}^2. In the second, one side lies on the hypotenuse and the opposite two vertices lie on the legs. What is the area of the second square?

(A) 378378; (B) 392392; (C) 400400; (D) 441441; (E) 484484.

Correct answer: 392 cm2392\,\mathrm{cm}^2.

ResponsesReward per responseReasoning and outcome
R10Uses an incorrect geometric relation, eventually obtains an area of about 75.6675.66, and then chooses 400400 without justification.
R2, R71Incorrect geometric relations lead to 220.5220.5 in R2 and 162+722162+72\sqrt{2} in R7. Both select 392392 as the “closest” option, even though 378378 is closer.
R3, R51R3 incorrectly turns (a−s)/s=a/s(a-s)/s=a/s into a=2sa=2s. R5 incorrectly infers a=441a=441 from 21a+441=21a21a+441=21a. Both eventually select 392392 without deriving it.
R4, R61Use contradictory dimensions and areas. R4 reaches 196196 and R6 reaches 441441, but both then assert that the answer is 392392 without a valid derivation.
R81Treats drawing coordinates as physical lengths, then asserts the false equality 10000/441=39210000/441=392. The left-hand side is approximately 22.67622.676.

Seven of eight final answers are correct, yet none of the eight written solutions validly derives 392392. Different geometric errors lead to unsupported correct answers. This case shows that a high observed pass rate can coexist with varied, unreliable reasoning and a low recorded selection score.

(ii) A shared approach and a high predicted RACS do not guarantee a high pass rate

Case B1 — Integer divisors

High RACS: 0.9986340.998634. Correct responses: 1/81/8.

Question: How many integer divisors does 77 have?

Correct answer: 44.

ResponsesReward per responseReasoning and outcome
R1, R3–R80Count only 11 and 77 and answer 22, overlooking −1-1 and −7-7. They mistake the positive divisors for all integer divisors.
R21Includes the negative divisors, correctly lists 1,−1,7,−71,-1,7,-7, and answers 44.

All eight responses identify and count divisors, but seven apply the same mistaken restriction to positive divisors. A shared approach can therefore preserve a conceptual error; here, it coexists with a high predicted RACS and only 1/81/8 correct answers.

Case B2 — Polar-curve asymptote

High RACS: 0.9994090.999409. Correct responses: 1/81/8.

Question: Find the equation of the asymptote of the graph of r=cos⁡(2θ)sec⁡θr=\cos(2\theta)\sec\theta.

Correct answer: x=−1x=-1.

All eight responses attempt a similar broad plan: rewrite the equation, examine limiting behavior, and infer a Cartesian asymptote.

ResponsesReward per responseReasoning and outcome
R1, R3, R60Rewrite: Obtain r=2cos⁡θ−sec⁡θr=2\cos\theta-\sec\theta. Limits: Examine cos⁡θ→0\cos\theta\to0, where rr diverges. Asymptote: Incorrectly set x=rcos⁡θx=r\cos\theta to zero by direct substitution and answer x=0x=0.
R2, R80Rewrite: R2 incorrectly replaces sec⁡θ\sec\theta with cos⁡θ\cos\theta; R8 incorrectly simplifies sin⁡2θ/cos⁡θ\sin^2\theta/\cos\theta. Limits: Both examine cos⁡θ→0\cos\theta\to0. R2 obtains r→0r\to0 from its incorrect equation, while R8 considers divergent rr. Asymptote: Both incorrectly infer x=0x=0 from the limiting direction.
R40Rewrite: Starts from r=2cos⁡θ−sec⁡θr=2\cos\theta-\sec\theta, but introduces an extra factor of rr when converting to Cartesian coordinates. Limits: Mistakenly treats cos⁡(2θ)=0\cos(2\theta)=0 as a condition for divergence, although r→0r\to0 at those angles. Asymptote: Uses those angles to incorrectly infer y=xy=x.
R70Rewrite: Obtains r=2cos⁡θ−sec⁡θr=2\cos\theta-\sec\theta and a correct implicit Cartesian equation. Limits: Identifies cos⁡θ=0\cos\theta=0, but also incorrectly drops the terms involving yy when analyzing the Cartesian equation at large xx. Asymptote: Returns to the limiting-angle argument and incorrectly concludes x=0x=0.
R51Rewrite: Obtains r=2cos⁡θ−sec⁡θr=2\cos\theta-\sec\theta. Limits: Examines cos⁡θ→0\cos\theta\to0 and simplifies x=rcos⁡θ=2cos⁡2θ−1x=r\cos\theta=2\cos^2\theta-1 before taking the limit. Asymptote: Correctly obtains x=−1x=-1, although its statements about infinite limits still contain errors.

These responses follow a similar broad plan, but make different errors in the transformations and limits. In particular, a vertical limiting direction does not determine the asymptote's position. The high predicted score and recurring approach therefore coexist with only 1/81/8 correct final answers.

2. Responses to the specific concerns

(1) Does high consistency imply a high pass rate?

Not necessarily. A shared solution approach can still contain conceptual errors or be applied incorrectly. In B1 and B2, the predicted RACS scores are 0.9986340.998634 and 0.9994090.999409, respectively, yet both groups have an observed pass rate of only 1/81/8. In B1, although all responses follow the same broad approach of identifying and counting the divisors of 77, most overlook the negative divisors. In B2, the responses follow a similar broad plan but make errors in transformations and limits. These cases illustrate that high predicted consistency can coexist with low observed correctness.

These cases also help interpret Figure 4(b), where difficulty is measured by pass rate. A1–A2 illustrate that correct answers can involve varied reasoning. B1–B2 illustrate that a shared approach can still produce mostly incorrect answers. A higher pass rate therefore does not necessarily imply a higher RACS score. The similar distributions in Figure 4(b) do not contradict this point.

(2) Does the “basic solution strategy” explanation contradict Figure 4(b)?

By “basic solution strategy,” we mean a shared approach, not a fully mastered solution. In B1, the responses identify divisors but mostly misinterpret which divisors to count. In B2, they use a similar broad approach but make errors in transformations and coordinate limits. A shared approach can therefore coexist with low accuracy, because following the same basic solution strategy does not guarantee that every step is carried out correctly.

Reviewer_BbiZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RACS selects RLVR prompts using the directional consistency of hidden representations across multiple sampled responses. The score is the mean resultant length of normalized response vectors, averaged across selected layers. To avoid generating responses for the entire corpus, the method measures scores on a small reference set and predicts other scores using nearest-neighbor regression over frozen prompt representations. Prompts with high predicted consistency are sampled preferentially, and only mixed-correctness response groups are retained for GRPO. Experiments report improved pooled accuracy across four Qwen models and five mathematical reasoning benchmarks under a fixed update budget.

Critical strengths

The selection statistic is simple, interpretable, and does not itself require verifier labels. The paper clearly distinguishes consistency among responses to one prompt from prompt difficulty. Reference-based estimation makes the method more practical than exhaustive response-level scoring. The manuscript reports both ranking diagnostics and the substantial reduction in scoring cost relative to scoring the entire corpus. Table 1 shows improvements in pooled accuracy across all four evaluated models. The 1.5B result improves from 72.10% with random selection to 75.34% with RACS. The paper is appropriately cautious about its explanation. Section 6.3 and the limitations distinguish the observed representation-separability association from a demonstrated account of policy-gradient improvement.

Critical weaknesses
  1. The computational benefit is unclear. RACS requires extra rollouts and activation extraction, but the comparisons only match training updates. It would be useful to compare methods under the same total compute budget.

  2. The benefit beyond prompt difficulty is not fully established. Showing that RACS varies within difficulty groups does not demonstrate that it selects better training examples at the same difficulty. A difficulty-matched comparison would help.

  3. The stability of the improvements is unclear. The experiments use 60 training steps and do not clearly report results across independent training seeds. Longer runs and uncertainty estimates would make the conclusions more convincing.

Questions for authors

refer to weakness

2 comments

Thank you for your feedback. Given rebuttal time and compute limits, we first address the concerns covered by our completed experiments.

All additional training experiments below use Qwen2.5-Math-1.5B.

Q1: It would be useful to compare methods under the same total compute budget.

A1: To test whether additional GPU time closes the accuracy gap, we extended training for DAPO and GRESO, our two strongest reproduced baselines. Each ran to the first completed update at or beyond RACS's GPU-hour budget. RACS remains higher by 1.25 and 0.79 percentage points, respectively.

All methods used the same resource budget of one A800 GPU. Timing covers the full selection and training pipeline; for RACS, this includes reference generation/scoring, rejected rollouts, activation extraction, prompt encoding, and updates.

MethodUpdatesGPU-hoursGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS6036.2586.8176.4026.8413.3360.0075.34
DAPO (extended)78≥36.25\geq 36.2586.7372.2027.2110.0047.5074.09
GRESO (extended)67≥36.25\geq 36.2586.2876.2025.7413.3345.0074.55

Q2: Showing that RACS varies within difficulty groups does not demonstrate that it selects better training examples at the same difficulty. A difficulty-matched comparison would help.

A2: We matched the proportions of hard, moderate, and easy prompts at every epoch, measuring difficulty under each run's current policy. The low-RACS control sampled with Appendix Eq. (9) until it filled the main run's difficulty quotas, discarding candidates from groups already full. All other training settings were unchanged.

In short, both runs used identical hard/moderate/easy proportions at every epoch. One favored high-RACS prompts, while the other favored low-RACS prompts under the same difficulty quotas. High-RACS selection retains an advantage of 2.55 percentage points.

SelectionGSM8KMATH500MinervaAIME24AMC23All (2,161)
High-RACS (main experiment)86.8176.4026.8413.3360.0075.34
Low-RACS (difficulty matched)84.3873.2025.7410.0052.5072.79

Q3: The experiments do not clearly report results across independent training seeds. Uncertainty estimates would make the conclusions more convincing.

A3: We added two training seeds for RACS. Pooled accuracy varies little across the three runs.

SD is the sample standard deviation across seeds.

SeedGSM8KMATH500MinervaAIME24AMC23All (2,161)
4186.8176.2028.6810.0050.0075.29
42 (original run)86.8176.4026.8413.3360.0075.34
4386.7377.0027.5713.3350.0075.34
Mean ±\pm SD86.78±0.0586.78 \pm 0.0576.53±0.4276.53 \pm 0.4227.70±0.9327.70 \pm 0.9312.22±1.9212.22 \pm 1.9253.33±5.7753.33 \pm 5.7775.32±0.0375.32 \pm 0.03

Q4: The experiments use 60 training steps. Longer runs would make the conclusions more convincing.

A4: To test whether RACS's advantage persists with longer training, we extended both RACS and Random with identical filtering and rollout reuse to 100 steps on Qwen2.5-Math-1.5B. Both methods improve over their original 60-step results, and RACS achieves 1.57 percentage points higher overall accuracy than Random.

Method (100 steps)GSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS86.6679.4028.3113.3355.0076.03
Random + identical filtering and rollout reuse87.1173.8024.6310.0052.5074.46

Thank you again for your time and effort in reviewing our submission. As the discussion period is drawing to a close, we wanted to gently follow up to see if our response has addressed your main concerns. If so, we would greatly appreciate it if you could consider raising your score.

Reviewer_JvcwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes response activation consistency as a criterion for online prompt selection in reinforcement learning with verifiable rewards. For each prompt, it computes the mean resultant length of normalized hidden states across sampled responses and averages scores over three layers. At each epoch, the current policy scores 512 reference prompts, and similarity-weighted nearest-neighbor regression over frozen prompt representations estimates scores for the remaining corpus. Training favors high estimated scores and retains response groups containing both correct and incorrect answers for GRPO. Experiments cover four Qwen checkpoints and five mathematical reasoning benchmarks under a fixed update budget. RACS achieves the highest reported pooled accuracy for each checkpoint; analyses examine estimator quality, difficulty distributions, and representation separability.

Critical strengths
  1. The proposed selection signal is interesting and has encouraging downstream evidence. Table 1 reports the highest pooled accuracy for RACS across all four checkpoints, with gains of 2.92, 1.99, 1.90, and 0.97 percentage points over the best listed baseline. The shared training pool, hyperparameters, and update budget make the observed improvements meaningful evidence for its potential utility.

  2. The paper provides a concrete online procedure and tests the direction of its selection preference. The reference-set estimator avoids exhaustive response-based scoring, and the high-versus-low consistency comparison improves pooled accuracy by five percentage points (Table 2). Together, these results motivate further investigation of response geometry as a training-data selection signal.

Critical weaknesses
  1. The incremental value of consistency is not sufficiently isolated from the surrounding pipeline. The paper lacks a difficulty-only control using the same estimator, mixed-reward filtering, rollout reuse, and sampling procedure; it is also unclear whether all listed baselines receive identical filtering. Figure 4 establishes within-difficulty variation in RACS, but not incremental training utility after controlling for difficulty. Moreover, reference prompts retain direct scores while other prompts receive averaged predictions (Eq. 6); the low sampling temperature could amplify differences between these score distributions. The selected reference-prompt share and a control using uniformly estimated scores are not reported. Multiple training seeds and uncertainty estimates are also absent, limiting confidence in the smaller gains.

  2. Practical computational efficiency remains unestablished. Appendix A explicitly matches update counts rather than total rollouts or wall-clock time. Across 30 epochs, reference scoring generates 30 x 512 x 8 = 122,880 responses, equal in count to the responses in the retained training groups, before accounting for rejected candidates, activation extraction, and prompt encoding. Algorithm 1 does not specify reuse of reference rollouts for training. The 98.8% saving in Section 6.1 is relative to exhaustive RACS scoring, not to end-to-end baseline training. Thus, the experiments support improved accuracy per update but leave the practical accuracy-versus-compute tradeoff unresolved.

Questions for authors
  1. Can you report accuracy under a matched total generated-token or GPU-time budget against Random and difficulty-based selection? Please break down reference scoring, candidate rejection, activation extraction, prompt encoding, and training costs, and clarify whether reference rollouts are reused for updates.

  2. Are mixed-reward filtering and rollout reuse identical across methods? Under the same full pipeline, does replacing RACS with pass-rate/difficulty preserve or remove the gain? Results across multiple training seeds, together with the number of stochastic evaluation samples, would materially affect my confidence in the comparison.

  3. What fraction of selected and retained prompts belongs to the current reference set? Does the improvement persist when reference prompts are excluded from training candidates or are assigned leave-one-out predictions? Please also specify the automatic temperature-relaxation rule mentioned in Appendix A, including its trigger and update schedule, which are not given in the subsequent algorithm.

4 comments

Thank you for your feedback. Given rebuttal time and compute limits, we first address the concerns covered by our completed experiments.

All additional training experiments below use Qwen2.5-Math-1.5B.

Q1: Can you report accuracy under a matched total generated-token or GPU-time budget?

A1: To test whether the gain persists at matched GPU time, we extended Random with identical filtering and rollout reuse, as well as DAPO and GRESO, our two strongest reproduced baselines. Random, DAPO and GRESO ran to the first completed update at or beyond RACS's budget. RACS remains higher by 1.21, 1.25, and 0.79 percentage points, respectively.

All methods used the same resource budget of one A800 GPU. Timing covers the full selection and training pipeline; for RACS, this includes reference generation/scoring, rejected rollouts, activation extraction, prompt encoding, and updates.

Before extension, Random, DAPO and GRESO used 19 h 32 min, 28 h 08 min and 32 h 44 min, respectively, for 60 updates.

MethodUpdatesGPU-hoursGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS6036.2586.8176.4026.8413.3360.0075.34
Random + identical filtering and rollout reuse (extended)110≥36.25\geq 36.2586.1374.2027.2110.0045.0074.13
DAPO (extended)78≥36.25\geq 36.2586.7372.2027.2110.0047.5074.09
GRESO (extended)67≥36.25\geq 36.2586.2876.2025.7413.3345.0074.55

Q2: Please break down reference scoring, candidate rejection, activation extraction, prompt encoding, and training costs.

A2: The available stage timings from the main-run logs are:

Stageelapsed time
Reference rollout loop23.41 h across 30 epochs; 46.82 min/epoch on average
Prompt encoding8.48 min in total
Score propagationApproximately 2 s/epoch across 30 epochs
Training-candidate sampling and mixed-reward filtering≈5.45 h across 30 epochs; ≈10.91 min/epoch (estimated from progress logs)
GRPO updates7.09 min/update, averaged over 58 recorded updates

Q3: Are reference rollouts reused for updates?

A3: No. Reference-scoring responses are used only for scoring, even if the same prompt is later selected. Only accepted training rollouts are cached for updates.


Q4: The 98.8% saving is relative to exhaustive RACS scoring, not end-to-end baseline training.

A4: Your interpretation is correct. We apologize for the ambiguity: this saving shows that reference-based estimation makes RACS scoring practical, not that end-to-end training is faster than the baselines.


Q5: Are mixed-reward filtering and rollout reuse identical across methods?

A5: The original baselines follow their published procedures. To test whether the gain remains under identical filtering and rollout reuse, we added a Random control. RACS remains 2.73 percentage points higher. Both cache accepted training rollouts for updates.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
Random + identical filtering and rollout reuse85.8268.8025.7413.3347.5072.61
RACS (original run, seed 42)86.8176.4026.8413.3360.0075.34

Q6: Under the same full pipeline, does replacing RACS with pass-rate/difficulty preserve or remove the gain?

A6: We changed only the selection score from RACS to pass rate. Each epoch, we measured pass rates from reference-set rollouts and used the unchanged similarity-weighted nearest-neighbor estimator over frozen prompt activation signatures to predict pass rates for the remaining prompts. The same score-based sampling, mixed-reward filtering, rollout reuse, and training settings were retained. Pooled accuracy decreases by 1.86 percentage points.

Selection scoreGSM8KMATH500MinervaAIME24AMC23All (2,161)
Pass rate86.5071.6025.746.6742.5073.48
RACS (main experiment)86.8176.4026.8413.3360.0075.34

Q7: Figure 4 shows within-difficulty variation in RACS, but not incremental training utility after controlling for difficulty.

A7: To test RACS's training value beyond difficulty, we matched the proportions of hard, moderate, and easy prompts at each epoch. Difficulty was measured under each run's current policy because it changes as the policy learns. The low-RACS control sampled with Appendix Eq. (9) until it filled the main run's difficulty quotas, discarding candidates from groups already full. All other training settings were unchanged.

In short, both runs used identical hard/moderate/easy proportions at every epoch. One favored high-RACS prompts, while the other favored low-RACS prompts under the same difficulty quotas. High-RACS selection retains an advantage of 2.55 percentage points.

SelectionGSM8KMATH500MinervaAIME24AMC23All (2,161)
High-RACS (main experiment)86.8176.4026.8413.3360.0075.34
Low-RACS (difficulty matched)84.3873.2025.7410.0052.5072.79

Q8: Results across multiple training seeds would materially affect confidence in the comparison.

A8: We added two training seeds for RACS. Pooled accuracy varies little across the three runs.

SD is the sample standard deviation across seeds.

SeedGSM8KMATH500MinervaAIME24AMC23All (2,161)
4186.8176.2028.6810.0050.0075.29
42 (original run)86.8176.4026.8413.3360.0075.34
4386.7377.0027.5713.3350.0075.34
Mean ±\pm SD86.78±0.0586.78 \pm 0.0576.53±0.4276.53 \pm 0.4227.70±0.9327.70 \pm 0.9312.22±1.9212.22 \pm 1.9253.33±5.7753.33 \pm 5.7775.32±0.0375.32 \pm 0.03

Q9: What fraction of retained prompts belongs to the current reference set? Does the improvement persist when reference prompts are excluded from training candidates?

A9: Across 30 epochs, we retained 15,360 training-prompt selections, of which 1,146 came from the corresponding epoch's reference set (7.46%). The per-epoch reference share had a median of 5.86% (range: 2.93%–29.88%).

Excluding each epoch's reference prompts from training candidates leaves an advantage of 1.11 percentage points over Random with the same filter.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS (main experiment)86.8176.4026.8413.3360.0075.34
RACS (exclude current reference set)84.8475.6027.216.6750.0073.72
Random + identical filtering and rollout reuse85.8268.8025.7413.3347.5072.61

Thanks for your response.Your response has addressed some part of my question. However, there is still lack of some part of the details, especially in Q9. So, the reviewers will maintain the original score.

Thank you for your follow-up. We would like to clarify that our previous comment provided detailed responses to all nine questions, including additional experiments and quantitative results.

For example, regarding Q9, we provided detailed statistics on the proportion of retained training prompts belonging to the current reference set and conducted an additional experiment excluding reference-set prompts from training candidates to verify whether the improvement persists.

It is difficult for us to identify what remains unresolved from the general statement that details are lacking. If you still have concerns, particularly regarding Q9, please let us know which specific details are missing. If not, we would greatly appreciate it if you could consider raising your score.

Reviewer_3ryfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RACS scores RLVR prompts by the directional agreement (mean resultant length) of the last-token hidden states of K sampled responses. Scores for most of the corpus are predicted by kNN from a small reference set. Training samples high-RACS prompts and keeps only mixed-reward groups. The method improves pooled accuracy over Random on four Qwen models and five math benchmarks.

Critical strengths

1.A signal distinct from pass rate: RACS varies substantially within the same difficulty level. 2.Consistent gains over Random on all four models, and high-consistency selection beats low-consistency selection by 5 points. 3.A cost-aware reference-set estimator, with code released.

Critical weaknesses

1.The gains are confounded by mixed-reward filtering (dynamic sampling). The paper does not state whether the baselines use the same filtering and rollout reuse. 2.Surface-level factors are not ruled out. Hidden states are anisotropic (mean 0.877 among retained groups) and are not centered. The last token is often a formatting token, so the score may mostly encode answer agreement, truncation, or length. 3.Matching updates does not match compute. Reference-set scoring and generation for rejected candidates are not counted. 4.The estimator and statistics are weak. Predicted deciles span only 0.85–0.88 and are non-monotonic after D5. There is one seed and only 60 steps. The baselines are weak (DOTS+RR < Random on 3/4 models). The temperature-relaxation rule and train–test deduplication are missing.

Questions for authors

1.Do all baselines use identical filtering and rollout reuse? What is the gain under unified filtering and matched GPU-hours? 2.Does RACS keep its value after centering (with τ recalibrated), with alternative pooling, and after controlling for length, truncation, and answer agreement? 3.Please report the estimator's rank correlation, multi-seed results, the temperature-relaxation rule, and deduplication.

3 comments

Thank you for your feedback. Given rebuttal time and compute limits, we first address the concerns covered by our completed experiments.

All additional training experiments below use Qwen2.5-Math-1.5B.

Q1: Do all baselines use identical filtering and rollout reuse? What is the gain under unified filtering?

A1: The original baselines follow their own procedures and do not all use identical filtering and rollout reuse. To isolate the effect of the selection signal, we added a Random control with filtering and rollout reuse identical to those used by RACS. RACS remains 2.73 percentage points higher.

We also compared with methods that already filter zero-variance groups and train on retained rollouts: DAPO and GRESO. RACS exceeds them by 1.49 and 1.25 percentage points under the original 60-update budget, respectively.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
Random + identical filtering and rollout reuse85.8268.8025.7413.3347.5072.61
DAPO (with filtering)86.1373.0025.7410.0055.0073.85
GRESO (with filtering)86.5873.2024.6310.0057.5074.09
RACS (original run, seed 42)86.8176.4026.8413.3360.0075.34

Q2: What is the gain under matched GPU-hours?

A2: To test whether additional GPU time closes the accuracy gap, we extended training for DAPO and GRESO, our two strongest reproduced baselines. Each ran to the first completed update at or beyond RACS's GPU-hour budget. RACS remains higher by 1.25 and 0.79 percentage points, respectively.

All methods used the same resource budget of one A800 GPU. Timing covers the full selection and training pipeline; for RACS, this includes reference generation/scoring, rejected rollouts, activation extraction, prompt encoding, and updates.

MethodUpdatesGPU-hoursGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS6036.2586.8176.4026.8413.3360.0075.34
DAPO (with filtering, extended)78≥36.25\geq 36.2586.7372.2027.2110.0047.5074.09
GRESO (with filtering, extended)67≥36.25\geq 36.2586.2876.2025.7413.3345.0074.55

Q3: Does RACS keep its value after centering and with alternative pooling?

A3: We tested mean pooling and within-prompt centering separately. Centering subtracts the mean of the eight response vectors before L2 normalization; the sampling temperature τ\tau remains unchanged.

Mean pooling and centering remain 0.92 and 2.03 percentage points above Random with the same filter, respectively, although both score below the original representation.

MethodGSM8KMATH500MinervaAIME24AMC23All (2,161)
RACS (original last token)86.8176.4026.8413.3360.0075.34
RACS (mean pooling)84.6976.0026.1013.3342.5073.53
RACS (within-prompt centering)86.3574.4026.8420.0057.5074.64
Random + identical filtering and rollout reuse85.8268.8025.7413.3347.5072.61

Q4: Predicted deciles are non-monotonic after D5.

A4: Non-monotonicity among the upper deciles need not prevent useful selection: our sampler mainly needs to distinguish high- from low-consistency prompts. Figure 3(b) shows that estimated-score sampling still favors high actual RACS.


Q5: There is one seed. Please report multi-seed results.

A5: We added two training seeds for RACS. Pooled accuracy varies little across the three runs.

SD is the sample standard deviation across seeds.

SeedGSM8KMATH500MinervaAIME24AMC23All (2,161)
4186.8176.2028.6810.0050.0075.29
42 (original run)86.8176.4026.8413.3360.0075.34
4386.7377.0027.5713.3350.0075.34
Mean ±\pm SD86.78±0.0586.78 \pm 0.0576.53±0.4276.53 \pm 0.4227.70±0.9327.70 \pm 0.9312.22±1.9212.22 \pm 1.9253.33±5.7753.33 \pm 5.7775.32±0.0375.32 \pm 0.03

Thanks for running the extra baselines and seeds on the 1.5B model. While the unified-filter comparison helps, the matched-compute margin over GRESO narrows to 0.79%, and several core questions remain open, including controlling for surface factors (answer agreement, truncation, length), the kNN rank correlation, and train-test deduplication. I will keep my current score.

Thank you for your follow-up and for considering our additional experiments. We would like to clarify four points.

(Q1). The matched-compute margin over GRESO.

(A1) Under matched GPU-hour budgets, RACS still achieves higher overall accuracy than both added baselines, DAPO (+1.25 percentage points) and GRESO (+0.79 percentage points). Importantly, RACS also introduces a different perspective on data selection: it prioritizes prompts based on response activation consistency, whereas GRESO, like other reward-based selection methods, relies on rewards from sampled responses to decide which rollouts to skip.

(Q2). Last-token representations and surface factors.

(A2) As discussed in our paper, we use the last valid token because, under causal attention, its hidden state can incorporate information from the entire preceding response. Prior work has analyzed how sentence-level features are aggregated into last-token representations (Kirsanov et al., 2025) and demonstrated effective text embeddings using last-token pooling (Wang et al., 2024). Although last-token representations may be influenced by surface factors, they also encode global features of the response that our method aims to capture.

(Q3). kNN rank correlation.

(A3) As explained in A4 of our previous response, our use case does not require the estimator to recover an exact ranking of all prompts. It only needs to capture coarse-grained differences in RACS so that sampling is more likely to favor higher-RACS prompts. We believe Figure 3(b) already provides evidence of this ability, as sampling with estimated scores favors prompts with higher measured RACS.

(Q4). Train–test deduplication.

(A4) We performed train–test deduplication in our experiments.

Comments on the paper

1 comment

We thank the AC and all reviewers for their constructive feedback. Reviewers recognized our interpretable consistency-based selection signal (3ryf, Jvcw, BbiZ, b74j) that is distinct from prompt difficulty (3ryf, BbiZ), its consistent gains across models and scales (3ryf, Jvcw, BbiZ, b74j), our practical reference-set estimator (3ryf, Jvcw, BbiZ), and the paper's clear presentation (3ryf, Jvcw, b74j).

During the rebuttal, we ran controlled experiments to directly address the reviewers' core concerns.

  • We added strong baselines (DAPO and GRESO), and RACS still outperforms them under matched end-to-end GPU-hour budgets.
  • Difficulty-matched, pass-rate, and unified-filtering controls further show that the gains come from our proposed response activation consistency signal rather than prompt difficulty or pipeline settings.
  • We also added multi-seed runs, pilot-study replication, representation ablations, reference-set exclusion, and rollout case studies to confirm the stability and soundness of our method.

Notably, Reviewer b74j (Confidence: 4, the highest among all reviewers) engaged across two rounds of detailed empirical and conceptual discussion. After our responses addressed these concerns, the reviewer raised the rating from 2 to 3 and recommended acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.