- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the problem of data selection for offline reinforcement learning with verifiable rewards (RLVR), where different responses and tokens may contribute differently to model training. The authors analyze the mismatch between the current policy and the Gibbs-optimal policy through bidirectional KL divergence and propose GIST, a hierarchical data selection method operating at both response and token levels. At the response level, GIST selects correct responses from high-temperature samples and incorrect responses from low-temperature samples. At the token level, it selects low-probability tokens from correct responses and high-probability tokens from incorrect responses.
Critical strengths
- The paper investigates data selection for offline RLVR, which is a practical and relevant problem.
Critical weaknesses
-
The benefits of incorporating incorrect responses remain insufficiently investigated. Although the paper includes an RFT-style baseline that retains only correct responses, this comparison is conducted under the BOLT objective and does not fully isolate the contribution of incorrect responses under different training objectives. In particular, comparing GIST against a positive-only SFT baseline with the same token selection strategy and comparable training budgets would help clarify whether incorporating incorrect responses provides additional benefits over training exclusively on correct responses.
-
The choice of sampling temperatures and their allocation is not sufficiently justified. GIST generates an equal number of responses at temperatures 1.0 and 0.3, but the paper does not investigate whether this allocation is optimal or even necessary. Although the authors provide some intuition for selecting correct responses from high-temperature samples and incorrect responses from low-temperature samples, the experimental evaluation lacks comparisons with alternative sampling strategies, such as using only high-temperature or low-temperature samples, or varying their proportions. Such experiments would help clarify the contribution of temperature pairing and strengthen the empirical justification for the proposed design.
Questions for authors
Please see the weaknesses.
4 comments
W2: Justification and Sensitivity Analysis of Sampling Temperatures and Selection Quotas
Pairing and targets distinct mismatch regimes without extra rollout cost:
- On-policy positives (): Sampling from retains low-probability reasoning steps () with high forward-KL soft-value potential (, Prop. 3.7), whereas low temperature collapses positives onto high-probability tokens (mean for below-mean tokens at vs. at , Appendix B.5).
- Low-temperature negatives (): By Prop. 3.4, reverse-KL mismatch on incorrect responses scales with ; concentrates rollouts on high-likelihood near-mode mistakes (raising median error log-probability from at to at ).
To test temperature pairing, alternative values, single-temperature pools, and varying the proportion of selected -correct vs. -incorrect responses at a fixed 8-response budget (drawn from the 16:16 pool on the 339 shared prompts satisfying all five quotas), we ran controlled ablations on Qwen3-0.6B / GSM8K (BOLT , 8/32 selected responses, ~33% loss-bearing tokens):
Table 2: Ablation on Temperature Pairing, Temperature Values, and Selected -Pos : -Neg Quotas (Qwen3-0.6B / GSM8K, BOLT)
| Ablation Dimension & Setting | Configuration | Val. Strict (%) | Val. Trunc. (%) |
|---|---|---|---|
| Pairing ablation (1,593 shared prompts, 256 cap) | With pairing ( pos. / neg., Table 4f) | 45.79 | — |
| Pairing ablation (1,593 shared prompts, 256 cap) | Without pairing (pooled + GIST mask, Table 4c) | 43.59 | — |
| Temperature value sweep (1,826 prompts, 1,024 cap) | Default (, 4:4 quota) | 46.02 | 5.16 |
| Temperature value sweep (1,024 cap) | Grid (2:2 & 8:8) | 39.12-45.11 | — |
| Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap) | 8 : 0 (8 -pos. + 0 -neg., exhausting half of pool) | 43.14 | 5.61 |
| Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap) | 6 : 2 (6 -pos. + 2 -neg.) | 44.50 | 7.28 |
| Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap) | 4 : 4 (GIST default: 4 -pos. + 4 -neg.) | 46.10 | 4.47 |
| Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap) | 2 : 6 (2 -pos. + 6 -neg.) | 45.72 | 35.18 |
| Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap) | 0 : 8 (0 -pos. + 8 -neg.) | 41.55 | 3.87 |
All 8-response configurations train on 25% of candidates (8/32) and ~33% of loss-bearing tokens (5.4x faster training to Val. than All-candidates on 0.6B: 0.80h vs. 4.31h):
- Temperature pairing and single-temperature controls: Removing temperature pairing on the 1,593 shared prompts drops Strict accuracy by 2.20 pp (45.79% vs. 43.59%); at 1,024 cap, the default (0.3, 1.0) setting (46.02% at Val., 45.72% at R5) outperforms all grid alternatives (39.12%-45.11%) as well as full-dataset single-temperature pools (32:0 at only: 43.56% at R5; 0:32 at only: 43.61% at R5).
- Selected -positive : -negative proportions: Because GIST draws positives from and negatives from , an equal 16:16 rollout pool maximizes joint prompt eligibility for both classes. Unlike lightweight GIST-Pos (which selects just 4 selective positives, 4/32), forcing 8:0 at fixed budget 8 requires taking 8 of 16 rollouts and dilutes selectivity on the 339-prompt subset (43.14%), whereas the balanced 4:4 default achieves both the highest Val. accuracy (46.10%) and low non-trivial truncation (4.47% vs. 35.18% for 2:6).
Manuscript revisions
We have added the three-seed CE + Unlikelihood ablation and positive-only SFT/BOLT controls (Table 1) as well as the temperature-value grid, single-temperature comparisons, and selected response-quota sensitivity analyses (Table 2) to Section 5.4 and Appendix B of the revised manuscript.
Thanks for the rebuttal, which has addressed most of my concerns. I will raise my score to 3.
Dear Reviewer AFyz,
Thank you very much for taking the time to read our rebuttal. We genuinely appreciate your constructive feedback, which helped us clarify the role of the negative-response branch and strengthen our ablations in Section 5.4 and Appendix B.
Thank you again for your support and thoughtful evaluation!
We thank Reviewer AFyz for the helpful review and focused suggestions to isolate the contribution of incorrect responses and evaluate sampling temperature choices and allocations. Below we report controlled experiments addressing both weaknesses, which we have incorporated into the revised manuscript.
W1: Validating the Contribution of Incorrect Responses Across Positive-Only and Negative-Gradient Objectives
As an upstream data selector, GIST pairs cleanly with two downstream objective families depending on whether the objective applies negative gradients to incorrect responses:
Table 1: Three-Seed Controlled Ablation Isolating GIST's Incorrect Responses on the Same Codebase & Benchmark (Qwen3.5-2B / GSM8K, 1,024 Cap)
Key findings: