acceptodds
Under review as a conference paper at ICLR 2027

GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

Abstract

Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative. We study which ones to supervise through the Gibbs-optimal policy of KL-regularized RLVR, which offline training targets, and measure the mismatch between a policy and this target with a bidirectional KL divergence. At the reference policy, this mismatch factorizes exactly and separates the two outcomes: correct responses contribute mainly through the forward term, concentrated on their low-probability tokens, and incorrect responses through the reverse term, concentrated on the responses the policy generates often and on their high-probability tokens. Based on this analysis, we propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with signals available before training. GIST uses the sampling temperature to select correct responses on-policy and incorrect responses near the policy's mode, and it uses the initial policy's token probabilities to supervise only the tokens where the mismatch lies. Applied to two offline training objectives, GIST raises the average accuracy over GSM8K, MATH-500, and MBPP for all three models under both objectives. In a controlled comparison on Qwen3-0.6B/GSM8K, it outperforms seven selection baselines that choose from the same candidates while supervising 66.6% fewer tokens than training on all candidates of the same prompts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, offline RLVR, data selection, Gibbs-optimal policy

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_AFyzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of data selection for offline reinforcement learning with verifiable rewards (RLVR), where different responses and tokens may contribute differently to model training. The authors analyze the mismatch between the current policy and the Gibbs-optimal policy through bidirectional KL divergence and propose GIST, a hierarchical data selection method operating at both response and token levels. At the response level, GIST selects correct responses from high-temperature samples and incorrect responses from low-temperature samples. At the token level, it selects low-probability tokens from correct responses and high-probability tokens from incorrect responses.

Critical strengths
  1. The paper investigates data selection for offline RLVR, which is a practical and relevant problem.
Critical weaknesses
  1. The benefits of incorporating incorrect responses remain insufficiently investigated. Although the paper includes an RFT-style baseline that retains only correct responses, this comparison is conducted under the BOLT objective and does not fully isolate the contribution of incorrect responses under different training objectives. In particular, comparing GIST against a positive-only SFT baseline with the same token selection strategy and comparable training budgets would help clarify whether incorporating incorrect responses provides additional benefits over training exclusively on correct responses.

  2. The choice of sampling temperatures and their allocation is not sufficiently justified. GIST generates an equal number of responses at temperatures 1.0 and 0.3, but the paper does not investigate whether this allocation is optimal or even necessary. Although the authors provide some intuition for selecting correct responses from high-temperature samples and incorrect responses from low-temperature samples, the experimental evaluation lacks comparisons with alternative sampling strategies, such as using only high-temperature or low-temperature samples, or varying their proportions. Such experiments would help clarify the contribution of temperature pairing and strengthen the empirical justification for the proposed design.

Questions for authors

Please see the weaknesses.

4 comments

We thank Reviewer AFyz for the helpful review and focused suggestions to isolate the contribution of incorrect responses and evaluate sampling temperature choices and allocations. Below we report controlled experiments addressing both weaknesses, which we have incorporated into the revised manuscript.

W1: Validating the Contribution of Incorrect Responses Across Positive-Only and Negative-Gradient Objectives

As an upstream data selector, GIST pairs cleanly with two downstream objective families depending on whether the objective applies negative gradients to incorrect responses:

  1. Positive-likelihood objectives (BOLT, SFT): Standard positive-likelihood objectives apply only positive gradients (−log⁡πθ(yt∣st)-\log \pi_\theta(y_t|s_t)): BOLT (β=0.1\beta=0.1) assigns incorrect responses a near-zero weight of e−10≈4.5×10−5e^{-10} \approx 4.5 \times 10^{-5} (effectively ignoring them), whereas unweighted SFT assigns unit positive weight (w=1w=1) to all retained tokens (positively imitating errors). Consequently, on Qwen3-0.6B / GSM8K (1,826 shared prompts, 1,024 cap), training exclusively on GIST's 4 τh\tau_h correct responses with the m+m^+ mask (GIST-Pos, just 4/32 responses without imitating errors) reaches 46.02% Strict accuracy (up from 43.97% when positively imitating incorrect responses under w=1w=1 SFT, and beating full-pool 32/32 training at 42.46% while cutting data volume by 87.5%).
  2. Negative-gradient objectives (CE + Unlikelihood): Because GIST's 4 τl\tau_l incorrect responses and m−m^- mask identify high-confidence error tokens (mean qt=0.974q_t = 0.974, Prop. 3.4), unlocking their active benefit requires a negative-gradient operator that suppresses rather than imitates those tokens. On the exact same codebase, shared eligible candidate pool, and GSM8K (1,024-token cap) benchmark on Qwen3.5-2B (where BOLT All is 60.65% and BOLT Random is 60.88%), we ran three independent training seeds comparing Pos.-Only CE (equivalent to positive-only SFT/BOLT on the 4 τh\tau_h positives with m+m^+) against Full GIST pairing cross-entropy on m+m^+ with unlikelihood loss −log⁡(1−πθ(yt∣st))-\log(1 - \pi_\theta(y_t|s_t)) on m−m^-:

Table 1: Three-Seed Controlled Ablation Isolating GIST's Incorrect Responses on the Same Codebase & Benchmark (Qwen3.5-2B / GSM8K, 1,024 Cap)

Training Configuration (Shared Eligible Pool)Selected BudgetSeed 1 / 2 / 3 Val. (%)Val. Mean ±\pm SD (%)Val. Gain vs. Pos.-OnlyR5 Gain vs. Pos.-Only
Pos.-Only CE (== GIST-Pos SFT/BOLT, 4 τh\tau_h pos. with m+m^+)4/3260.88 / 60.34 / 60.8860.70 ±\pm 0.31+0.00 pp+0.00 pp
CE + Unlikelihood w/ Random Neg. Mask (equal neg. token budget)8/3261.57 / 62.92 / 62.6362.37 ±\pm 0.71+1.68 ±\pm 0.62 pp+0.15 ±\pm 1.05 pp
CE + Unlikelihood (Full GIST) (4 τh\tau_h pos. m+m^+ + 4 τl\tau_l neg. m−m^-)8/3262.02 / 62.84 / 63.0862.65 ±\pm 0.56+1.95 ±\pm 0.71 pp+1.54 ±\pm 0.22 pp

Key findings:

  • Consistent gains from incorrect responses across all 3 seeds: On the exact same Qwen3.5-2B codebase and candidate pool, adding GIST's τl\tau_l incorrect responses with m−m^- unlikelihood updates outperforms Pos.-Only CE in 3/3 seeds at both the validation-selected checkpoint (+1.14, +2.50, +2.20 pp; mean +1.95 ±\pm 0.71 pp) and Round 5 (+1.29, +1.67, +1.67 pp; mean +1.54 ±\pm 0.22 pp), while cutting Round-5 generation truncation from 49%-58% down to 29%-37%.
  • Superiority of m−m^- over random negative token masking: While random negative token masking degrades by Round 5 (+0.15 ±\pm 1.05 pp over Pos.-Only CE), GIST's targeted m−m^- mask maintains a +1.54 ±\pm 0.22 pp gain (outperforming random negative masking by +1.39 ±\pm 1.22 pp at R5).

W2: Justification and Sensitivity Analysis of Sampling Temperatures and Selection Quotas

Pairing τh=1.0\tau_h=1.0 and τl=0.3\tau_l=0.3 targets distinct mismatch regimes without extra rollout cost:

  • On-policy positives (τh=1.0\tau_h = 1.0): Sampling from πref\pi_{ref} retains low-probability reasoning steps (qtq_t) with high forward-KL soft-value potential (ρt≤1/qt\rho_t \le 1/q_t, Prop. 3.7), whereas low temperature collapses positives onto high-probability tokens (mean qt=0.583q_t = 0.583 for below-mean tokens at τl\tau_l vs. 0.4110.411 at τh\tau_h, Appendix B.5).
  • Low-temperature negatives (τl=0.3\tau_l = 0.3): By Prop. 3.4, reverse-KL mismatch on incorrect responses scales with qyq_y; τl=0.3\tau_l=0.3 concentrates rollouts on high-likelihood near-mode mistakes (raising median error log-probability from −80.60-80.60 at τh\tau_h to −33.80-33.80 at τl\tau_l).

To test temperature pairing, alternative (τl,τh)(\tau_l, \tau_h) values, single-temperature pools, and varying the proportion of selected τh\tau_h-correct vs. τl\tau_l-incorrect responses at a fixed 8-response budget (drawn from the 16:16 pool on the 339 shared prompts satisfying all five quotas), we ran controlled ablations on Qwen3-0.6B / GSM8K (BOLT β=0.1\beta=0.1, 8/32 selected responses, ~33% loss-bearing tokens):

Table 2: Ablation on Temperature Pairing, Temperature Values, and Selected τh\tau_h-Pos : τl\tau_l-Neg Quotas (Qwen3-0.6B / GSM8K, BOLT)

Ablation Dimension & SettingConfigurationVal. Strict (%) ↑\uparrowVal. Trunc. (%) ↓\downarrow
Pairing ablation (1,593 shared prompts, 256 cap)With pairing (τh=1.0\tau_h=1.0 pos. / τl=0.3\tau_l=0.3 neg., Table 4f)45.79—
Pairing ablation (1,593 shared prompts, 256 cap)Without pairing (pooled τh/τl\tau_h/\tau_l + GIST mask, Table 4c)43.59—
Temperature value sweep (1,826 prompts, 1,024 cap)Default (τl=0.3,τh=1.0\tau_l = 0.3, \tau_h = 1.0, 4:4 quota)46.025.16
Temperature value sweep (1,024 cap)Grid (τl,τh)∈{0.1,0.6}×{0.8,1.2}(\tau_l, \tau_h) \in \{0.1, 0.6\} \times \{0.8, 1.2\} (2:2 & 8:8)39.12-45.11—
Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap)8 : 0 (8 τh\tau_h-pos. + 0 τl\tau_l-neg., exhausting half of τh\tau_h pool)43.145.61
Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap)6 : 2 (6 τh\tau_h-pos. + 2 τl\tau_l-neg.)44.507.28
Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap)4 : 4 (GIST default: 4 τh\tau_h-pos. + 4 τl\tau_l-neg.)46.104.47
Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap)2 : 6 (2 τh\tau_h-pos. + 6 τl\tau_l-neg.)45.7235.18
Selected Quota (of 8 from 16:16 pool, 339 prompts, 1,024 cap)0 : 8 (0 τh\tau_h-pos. + 8 τl\tau_l-neg.)41.553.87

All 8-response configurations train on 25% of candidates (8/32) and ~33% of loss-bearing tokens (5.4x faster training to Val. than All-candidates on 0.6B: 0.80h vs. 4.31h):

  1. Temperature pairing and single-temperature controls: Removing temperature pairing on the 1,593 shared prompts drops Strict accuracy by 2.20 pp (45.79% vs. 43.59%); at 1,024 cap, the default (0.3, 1.0) setting (46.02% at Val., 45.72% at R5) outperforms all grid alternatives (39.12%-45.11%) as well as full-dataset single-temperature pools (32:0 at τh=1.0\tau_h=1.0 only: 43.56% at R5; 0:32 at τl=0.3\tau_l=0.3 only: 43.61% at R5).
  2. Selected τh\tau_h-positive : τl\tau_l-negative proportions: Because GIST draws positives from τh\tau_h and negatives from τl\tau_l, an equal 16:16 rollout pool maximizes joint prompt eligibility for both classes. Unlike lightweight GIST-Pos (which selects just 4 selective τh\tau_h positives, 4/32), forcing 8:0 at fixed budget 8 requires taking 8 of 16 τh\tau_h rollouts and dilutes selectivity on the 339-prompt subset (43.14%), whereas the balanced 4:4 default achieves both the highest Val. accuracy (46.10%) and low non-trivial truncation (4.47% vs. 35.18% for 2:6).

Manuscript revisions

We have added the three-seed CE + Unlikelihood ablation and positive-only SFT/BOLT controls (Table 1) as well as the temperature-value grid, single-temperature comparisons, and selected response-quota sensitivity analyses (Table 2) to Section 5.4 and Appendix B of the revised manuscript.

Thanks for the rebuttal, which has addressed most of my concerns. I will raise my score to 3.

Dear Reviewer AFyz,

Thank you very much for taking the time to read our rebuttal. We genuinely appreciate your constructive feedback, which helped us clarify the role of the negative-response branch and strengthen our ablations in Section 5.4 and Appendix B.

Thank you again for your support and thoughtful evaluation!

Reviewer_zZx4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies which responses and tokens to use for training in offline RLVR. The authors proposed GIST that selects prompts and responses using sampling temperature and correctness. It then uses the initial policy’s token probabilities to select positions for loss computation. GIST improves average accuracy across three models and two training objectives. In the controlled Qwen3-0.6B/GSM8K experiment, it achieves higher accuracy while using 66.6% fewer cumulative supervised tokens than training on all candidates.

Critical strengths
  1. GIST applies to both BOLT and SFT and improves average accuracy across three models. This provides empirical evidence that its benefits are not limited to a single training objective.
  2. GIST improves average accuracy across three models and two training objectives. In the controlled Qwen3-0.6B/GSM8K experiment, it achieves higher accuracy while using 66.6% fewer cumulative supervised tokens than training on all candidates.
Critical weaknesses
  1. The practical context for the offline setting is insufficiently specified. The introduction explains why data selection matters when training on a fixed response pool. However, it could better explain the practical conditions that require offline RLVR and the situations that make online RLVR difficult. This would clarify the scope and importance of the work.

  2. The connection between the theoretical selection guarantee and the implemented GIST procedure is not quantified. Corollary 3.2 analyzes a procedure that selects the BB responses with the largest UU. In practice, GIST filters candidates by temperature and correctness, then samples uniformly within each group. For token selection, it uses the mean token probability within each response as the threshold. The exact factorization at the initial policy and the token-level bounds motivate these rules. However, they do not guarantee how much mismatch the actual selection captures. I acknowledge that the authors explicitly limit the scope of their guarantee. Nevertheless, it remains difficult to assess how strongly the theory justifies the specific implementation rules.

  3. I am unsure whether BOLT is a suitable baseline. I believe BOLT was chosen because the paper assumes an offline setting. However, it is not a widely used algorithm. SFT, the other baseline, also does not optimize reward under a KL constraint. Therefore, it cannot be considered to fall under the setting assumed in Equation 1.

  4. Sensitivity to the evaluation generation limit is not examined. For GSM8K, data collection allows up to 1,024 new tokens, whereas evaluation limits generation to 256 new tokens. If training uses a maximum length of 1,024, I would expect the model’s behavior to be optimized for that setting. I therefore do not understand why evaluation uses a different setting. Consequently, it is difficult to regard this evaluation as an accurate assessment of training effectiveness. The evaluation generation limits for MATH and MBPP should also be specified.

Questions for authors
  1. Can you explain how closely temperature pairing and uniform sampling approximate top-UU selection? Can you report the mismatch captured by the actual selection or compare it with a computable reference selection?

  2. Do the performance differences between methods persist when the GSM8K evaluation generation limit increases? For example, can you report results and truncation rates with limits of 256 and 1,024 new tokens? What are the evaluation generation limits for MATH and MBPP?

  3. How were τh=1.0,τl=0.3\tau_h=1.0,\tau_l=0.3 and the quotas of four correct and four incorrect responses chosen? Beyond the pairing ablation, can you provide sensitivity analyses for the temperature values and response quotas?

12 comments

We thank Reviewer zZx4 for the constructive questions regarding practical motivation, theoretical approximation quality, baseline choice, evaluation length limits, and hyperparameter sensitivity. Below we address each weakness and question with supporting literature, mathematical clarifications, and controlled empirical diagnostics, and we have updated the manuscript accordingly.

W1: Practical Motivation and Compute Accessibility of Offline RLVR

While online RLVR (e.g., PPO, GRPO; Schulman et al., 2017; Shao et al., 2024; Yu et al., 2025, cited in Section 1) is effective when large-scale synchronous rollout compute is available, offline and iterative-offline RLVR (e.g., RFT, RAFT, ReST, OREO, and BOLT; Yuan et al., 2023; Dong et al., 2023; Gulcehre et al., 2023; Wang et al., 2024; Shu et al., 2026, cited in Sections 1 and 2.2) decouples rollout generation from policy optimization. By generating a candidate pool once, auditing it deterministically, and reusing it across training rounds and hyperparameter sweeps, offline RLVR avoids regenerating millions of tokens per run and co-locating inference engines in GPU memory during training. Within this offline paradigm, GIST further reduces 0.6B training time to the best validation checkpoint (Val.) by 5.4x on a single RTX 4090 (0.80 vs. 4.31 GPU-hours) and 4B 5-round training (R5) time by 4.9x (1.67 vs. 8.24 GPU-hours).

W2 & Q1: Approximating Top-U Selection & Comparison with a Computable Reference Selection

GIST is designed as a zero-extra-rollout tractable approximation to our reference-policy analysis. Computing exact soft values Z(s)Z(s) requires expensive Monte Carlo prefix rollouts; instead, Section 3 characterizes the probability regimes carrying bidirectional KL mismatch at q=πrefq = \pi_{\text{ref}}:

  • High-likelihood failure targeting (τl=0.3\tau_l = 0.3): Proposition 3.4 and Remark 3.5 show that within each outcome class at πref\pi_{\text{ref}}, sequence mismatch KλK^\lambda is proportional to sequence likelihood q(y)q(y) and the ratio Kλ/UλK^\lambda / U^\lambda is constant, so within-class top-BB by UU is exact for top-BB by KK. Sampling failures at τl=0.3\tau_l = 0.3 concentrates rollouts near the mode (raising median error log-probability from −80.60-80.60 at τh\tau_h to −33.80-33.80) with no extra rollouts.
  • Low-probability positive targeting (τh=1.0\tau_h = 1.0 with mask m+m^+): Proposition 3.7 (ρ≤1/q\rho \le 1/q) and Eq. (9) (q⋅h(1/q)q \cdot h(1/q), strictly decreasing in token probability qq) show that high-probability positive tokens cannot carry large mismatch. Sampling positives at τh=1.0\tau_h = 1.0 preserves low-probability steps (mean token probability 0.4110.411 vs. 0.5830.583 under τl\tau_l, Appendix B.5), and the sequence mean qˉ\bar{q} removes ~67% of boilerplate tokens without threshold tuning, contributing a +2.04 pp exact-format accuracy (Strict) gain on paired responses (Table 4).

To answer Q1 directly, we computed exact likelihood-weighted bidirectional mismatch captured within the finite candidate pool at an equal 8-of-32 response budget on GSM8K, comparing GIST against Random selection relative to the combinatorial best same-size subset (computable reference upper bound):

Model & Pool Setting (GSM8K, 32 Cand./Prompt)Selector (Budget)Total Mismatch Captured (%)Fraction of Best Same-Size Subset
Qwen3-0.6B (1,826 prompts)GIST (8/32)35.400.36
Qwen3-0.6B (1,826 prompts)Random (8/32 uniform)30.100.30
Qwen3.5-2B (Shared pool)GIST (8/32)31.400.32
Qwen3.5-2B (Shared pool)Random (8/32 uniform)27.100.27

Temperature pairing plus within-group sampling captures 0.36 (0.6B) and 0.32 (2B) of the combinatorial reference optimum, outperforming Random (0.30 and 0.27) while requiring only one reference forward pass (0.08 GPU-hours at 4B scale).

W3: Mathematical Alignment of Eq. (1) with RWR/AWR/BOLT, Why Not DPO, and Cross-Objective Generality

Rather than being a specific baseline choice, our evaluation spans four distinct post-training objective families while anchoring our primary analysis in the exact variational solution of Eq. (1):

  1. Exact mathematical unity between Eq. (1), Gibbs target, and RWR/AWR/BOLT: The unique closed-form maximizer of the canonical KL-regularized RLVR objective in Eq. (1) is the Gibbs policy π⋆(y∣x)∝q(y∣x)exp⁡(r(x,y)/β)\pi^\star(y|x) \propto q(y|x) \exp(r(x,y)/\beta) (Prop. 2.1, where q=πrefq = \pi_{\text{ref}}), which maintains a finite likelihood ratio π⋆(y+∣x)/π⋆(y−∣x)=(q(y+∣x)/q(y−∣x))exp⁡(1/β)\pi^\star(y^+|x)/\pi^\star(y^-|x) = (q(y^+|x)/q(y^-|x)) \exp(1/\beta) between correct (r=1r=1) and incorrect (r=0r=0) trajectories. In the offline RL literature, fitting this forward-KL Boltzmann projection KL(π⋆∥π)\text{KL}(\pi^\star \| \pi) via reward-weighted likelihood is the classic RWR / REPS / AWR (Reward- / Advantage-Weighted Regression) paradigm (Peters & Schaal, 2007; Peters et al., 2010; Peng et al., 2019, cited in Section 2.2). BOLT (Shu et al., 2026) is the exact instantiation of this classical Boltzmann projection for binary verifiable rewards under reference rollouts (Eq. 4). Thus, Eq. (1), Section 3's Gibbs mismatch decomposition, and BOLT form a single, mathematically unified framework.
  2. Why DPO does not match Eq. (1) under deterministic binary verifiers: DPO assumes pairwise preferences generated by a Bradley-Terry (BT) model P(y+≻y−∣x)=σ(ΔR)P(y^+ \succ y^- | x) = \sigma(\Delta R) and enforces βlog⁡(π⋆(y+∣x)/q(y+∣x))−βlog⁡(π⋆(y−∣x)/q(y−∣x))=ΔR\beta \log(\pi^\star(y^+|x)/q(y^+|x)) - \beta \log(\pi^\star(y^-|x)/q(y^-|x)) = \Delta R. With deterministic binary verifier rewards r∈{0,1}r \in \{0,1\}, every correct rollout beats every incorrect rollout with empirical probability P(y+≻y−∣x)=1P(y^+ \succ y^- | x) = 1. Since σ(ΔR)=1\sigma(\Delta R) = 1 requires an infinite BT reward gap ΔR=logit(1)=+∞\Delta R = \text{logit}(1) = +\infty, DPO's population optimum drives π⋆(y−∣x)→0\pi^\star(y^-|x) \to 0 regardless of the regularization strength β>0\beta > 0 (as formally proved in the IPO analysis of Azar et al., 2024, Section 3.1), departing from the finite Gibbs target π⋆\pi^\star of Eq. (1).
  3. Empirical generality across four training paradigms: Because GIST is an upstream data selector, we verified that its gains hold across BOLT, RFT, SFT, and negative-gradient Unlikelihood on GSM8K:
Objective Family & Setting (GSM8K)GIST Acc. (%)Baseline Acc. (%)Net Gain
BOLT (0.6B, 1,024 cap, Val.)46.02 (8/32)42.99 (All 32/32)+3.03 pp
Rejection Sampling RFT (0.6B, Table 3, 256 cap)45.79 (8/32)44.05 (RFT-style)+1.74 pp
Unweighted SFT (0.6B, Table 2, 256 cap)43.82 (8/32)42.08 (All 32/32)+1.74 pp
Positive-Only SFT (0.6B, 1,024 cap)46.02 (4/32)42.46 (All 32/32)+3.56 pp
CE + Unlikelihood (2B, 1,024 cap, 3 seeds Val.)62.65 +- 0.56 (8/32)60.70 +- 0.31 (Pos.-Only)+1.95 +- 0.71 pp

Furthermore, sweeping the BOLT KL temperature across β∈{0.3,0.5,1.0}\beta \in \{0.3, 0.5, 1.0\} on Qwen3-0.6B / GSM8K (1,826 prompts, 1,024 cap) shows that GIST (8/32) consistently outperforms All-candidates (32/32) by +2.12 pp (β=0.3\beta=0.3), +2.27 pp (β=0.5\beta=0.5), and +1.97 pp (β=1.0\beta=1.0) while training on 4x fewer responses.

W4 & Q2: Matched 1,024-Token GSM8K Evaluation and Benchmark Generation Limits

We agree that evaluation token limits should match training rollout horizons and be stated explicitly for all benchmarks. In the submission, MATH-500 and MBPP were evaluated with caps of 1,024 and 2,048 new tokens (now stated explicitly in Appendix B.4), while GSM8K used the default 256-token 8-shot Q:/A: setting of lm-evaluation-harness. Re-evaluating identical checkpoints at both 256 and 1,024 new tokens confirms that GIST's gains hold and widen on Qwen3.5-2B when matching the 1,024-token rollout horizon:

Table 3: Paired Evaluation of Identical Checkpoints at 256 vs. 1,024 New Tokens on GSM8K (Strict %)

Model & Shared Eligible PoolMethod (Selected Budget)256-Token Acc. (%)1,024-Token Acc. (%)Gain at 1,024 Cap
Qwen3-0.6B (1,826 prompts, Val.)BOLT, All candidates (32/32)42.5342.99baseline
Qwen3-0.6B (1,826 prompts, Val.)BOLT + GIST (8/32)45.0345.34+2.35 pp (5.16% cap hit)
Qwen3.5-2B (1,826 prompts, R5)BOLT + Random (8/32)60.4060.88baseline
Qwen3.5-2B (1,826 prompts, R5)BOLT + GIST (8/32)60.8062.62+1.74 pp (widened by +1.34 pp)

Raising the evaluation limit from 256 to 1,024 tokens preserves GIST's +2.35 pp lead on Qwen3-0.6B (with a low 5.16% cap-hit rate) and expands GIST's lead over equal-budget Random on Qwen3.5-2B from +0.40 pp to +1.74 pp. Under the template-free 8-shot Q:/A: continuation format, cap-hit rates on larger models remain comparable to or lower than full-pool and random baselines. We have adopted the 1,024-token cap as our primary GSM8K evaluation protocol in the revised manuscript.

Q3: Sensitivity Analyses for Temperature Values and Response Quotas

Our default hyperparameters (τh=1.0,τl=0.3\tau_h=1.0, \tau_l=0.3, 16:16 rollout split, and 4:4 selected quota) were fixed prior to experimentation without test-set tuning: τh=1.0\tau_h=1.0 samples directly from πref\pi_{ref} without off-policy distortion, while τl=0.3\tau_l=0.3 concentrates near-mode errors. Removing temperature pairing on the shared 1,593-prompt pool drops Strict accuracy by 2.20 pp (Table 4c: 45.79% vs. 43.59%). To systematically test sensitivity to temperature values and response quotas under BOLT (β=0.1\beta=0.1, Qwen3-0.6B / GSM8K, 1,024 cap), we conducted two controlled sweeps:

  1. Temperature grid sweep: Evaluating (τl,τh)∈{0.1,0.6}×{0.8,1.2}(\tau_l, \tau_h) \in \{0.1, 0.6\} \times \{0.8, 1.2\} across 2+2 and 8+8 quotas on the 1,826-prompt pool yields validation-selected accuracies of 39.12% to 45.11%, remaining consistently below our default (τl=0.3,τh=1.0)(\tau_l=0.3, \tau_h=1.0) setting (45.56%).
  2. Selected response quota sweep (fixing total budget at 8/32 responses on the shared 339-prompt subset satisfying all tested quotas):

Table 4: Sensitivity to Selected Response Quota (Pos. : Neg.) at Fixed 8/32 Budget (Qwen3-0.6B / GSM8K, 1,024 Cap)

Selected Quota (Pos. : Neg., Total 8/32)Val.-Selected Strict Acc. (%)Val. Cap-Hit / Truncation Rate (%)
8:0 (8 correct, 0 incorrect)43.145.61
6:2 (6 correct, 2 incorrect)44.507.28
4:4 (default balanced quota)46.104.47
2:6 (2 correct, 6 incorrect)45.7235.18
0:8 (0 correct, 8 incorrect)41.553.87

Across the quota sweep, the default 4:4 setting achieves both the highest validation-selected accuracy (46.10%) and low truncation (4.47%), confirming that GIST's default hyperparameters sit in a stable, well-behaved region.

Manuscript revisions

We have expanded the offline RLVR literature discussion (Sections 1 & 2.2), added the three-tier theory-to-algorithm table and finite-pool mismatch capture diagnostics (Section 4), clarified the mathematical connection between Eq. (1), BOLT, and DPO along with the 3-seed Unlikelihood evaluation (Section 4.3), updated primary GSM8K evaluations to 1,024 tokens with explicit benchmark caps (Section 5 & Appendix B.4), and added the temperature and quota sensitivity sweeps (Appendix C).

Reviewer_th2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GIST, a hierarchical data-selection method for offline reinforcement learning with verifiable rewards. Starting from the Gibbs-optimal policy of KL-regularized RLVR, the authors use a bidirectional KL mismatch to argue that informative correct responses contain underrepresented, low-probability tokens, whereas informative incorrect responses are frequently generated failures containing high-probability tokens. GIST accordingly selects correct responses from temperature-1 samples and incorrect responses from low-temperature samples, then supervises below-mean-probability tokens in correct responses and above-mean-probability tokens in incorrect ones. The method is combined with BOLT and SFT and evaluated on three small models across GSM8K, MATH-500, and MBPP. The authors report consistent average improvements and reduced supervised-token counts, with additional comparisons and ablations on Qwen3-0.6B/GSM8K.

Critical strengths
  • GIST uses only verifier outcomes, sampling temperature, and frozen-policy token probabilities. It is easy to implement and can be combined with BOLT or SFT without modifying their optimization procedures.
  • The paper considers data selection at two levels rather than treating every response and token equally. The distinction between underrepresented correct behavior and confidently generated errors is intuitively appealing.
Critical weaknesses
  • The theoretical selection scores depend on unavailable target probabilities and soft-value ratios. GIST replaces them with temperature and a response-relative probability threshold. Low token probability does not imply high soft value, while high probability in a failed response does not establish that the token caused the failure. The proposed rules are therefore heuristics loosely inspired by, rather than derived from, the analysis.

  • The theorem guarantees only the amount of pre-training mismatch captured, not improved performance after optimization. Moreover, the appendix reports that its approximation factor can fall below (10^{-7}) under the experimental setting. This provides no meaningful guarantee for the actual method.

  • GIST simultaneously changes prompt eligibility, response temperature, response composition, token masking, and the selected training checkpoint. It excludes prompts that lack enough correct and incorrect candidates, thereby removing especially easy or difficult examples. The reported gains therefore cannot be attributed cleanly to the claimed Gibbs-targeted selection principle.

  • Masked tokens remain in the forward and backward passes, so the claimed reduction in “supervised tokens” is not equivalent to proportional compute savings. Most of the actual savings come from discarding responses. The paper does not report wall-clock time, FLOPs, memory consumption, or end-to-end cost including candidate generation and frozen-policy scoring.

  • BOLT is derived for samples from the reference policy, yet the candidate pool mixes two sampling temperatures and the paper explicitly applies no off-policy correction to the low-temperature responses. The resulting objective is no longer the exact Gibbs-target fitting objective used to motivate the method.

  • With (\beta=0.1), an incorrect response receives only (e^{-10}) times the weight of a correct response under BOLT. Its contribution is therefore almost zero. Consequently, the main BOLT experiments provide little evidence that selecting confident errors or masking their high-probability tokens is useful. The most detailed token-rule ablation examines correct responses instead of directly validating the paper’s claims about incorrect ones.

Questions for authors

See Weaknesses.

4 comments

We thank Reviewer th2Y for the detailed feedback on our theoretical framing, experimental controls, compute accounting, and training objective. Below we address each weakness with new diagnostics and controlled ablations, and we have incorporated these revisions into the updated manuscript.

W1: Theory-Guided Probability-Regime Targeting and Tractable Design

We agree that GIST should be described as theory-guided rather than closed-form derived. Computing exact per-prefix soft values Z(st)Z(s_t) requires expensive Monte Carlo rollouts at every token; instead, GIST is designed as a zero-extra-rollout tractable approximation that targets the exact high-mismatch probability regimes identified by our Section 3 analysis using a single frozen-policy forward pass (0.08 GPU-hours at 4B scale):

  • Targeting high-qyq_y failures via τl=0.3\tau_l = 0.3 (Prop. 3.4): At q=πrefq = \pi_{ref}, incorrect-response reverse-KL mismatch is exactly proportional to sequence likelihood, Kyλ=qyhλ(κ0)∝qyK_y^\lambda = q_y h_\lambda(\kappa_0) \propto q_y, requiring neither π∗\pi^* nor soft values. Sampling at τh=1.0\tau_h = 1.0 wastes rollout budget on low-likelihood tail noise (median error log-probability −80.60-80.60), whereas mode-seeking sampling at τl=0.3\tau_l = 0.3 (πτl∝πref1/τl\pi_{\tau_l} \propto \pi_{ref}^{1/\tau_l}) concentrates rollouts directly on high-qyq_y near-mode errors (median log-probability −33.80-33.80).
  • Targeting low-qtq_t positive steps via τh=1.0\tau_h = 1.0 + m+m^+ (Prop. 3.7 & Eq. 9): In correct responses, large soft-value gains require low qtq_t (ρt≤1/qt\rho_t \le 1/q_t), and the maximum attainable token mismatch qthλ(1/qt)q_t h_\lambda(1/q_t) strictly decreases in qtq_t (Eq. 9), ruling out high-qtq_t tokens. Sampling positives at τl\tau_l collapses trajectories onto already high-probability steps (mean qt=0.583q_t = 0.583 even among below-mean tokens at τl\tau_l vs. 0.4110.411 at τh\tau_h, Appendix B.5). Sampling positives on-policy (τh=1.0\tau_h = 1.0) preserves rare low-qtq_t reasoning steps while the length-adaptive sequence mean qˉ\bar{q} removes ~67% of high-qtq_t boilerplate tokens without threshold tuning (whereas flipping m+m^+ to keep top-qtq_t tokens drops accuracy by 1.74 pp in Table 5).
ComponentTheoretical StatusBasis & Empirical Validation
Within-class failure ranking (∝qy\propto q_y)ExactAt q=πrefq = \pi_{ref}, Kyλ=qyhλ(κr)K_y^\lambda = q_y h_\lambda(\kappa_r) (Prop. 3.4); ranking errors by qyq_y requires no target probabilities π∗\pi^*.
Low-qtq_t focus on correct responsesBound-supportedHigh soft-value gain requires low qtq_t (ρt≤1/qt\rho_t \le 1/q_t, Prop. 3.7; Eq. 9); keeping top-qtq_t drops 1.74 pp (Table 5, 1,593 prompts).
qˉ\bar{q} threshold, τl/τh\tau_l/\tau_h sampling, 4+4 quotasTractable ProxyZero-rollout screening rules; m−m^- targets high-confidence over-commitment (mean qt=0.974q_t=0.974) without claiming causal credit assignment.

W2: Selection Bound Scope, the 10−710^{-7} Constant, and Finite-Pool Mismatch Capture

We agree that Corollary 3.2 is a pre-training Selection Bound rather than a post-optimization convergence guarantee, and we have renamed it accordingly. Importantly, the factor c−/c+=γ2<10−7c_-/c_+ = \gamma^2 < 10^{-7} in Appendix A.3 arises only if positive and negative responses are ranked together in one mixed pool. Within each outcome class at πref\pi_{ref}, density ratios equal κr\kappa_r, so Kyλ/UyλK_y^\lambda / U_y^\lambda is constant and top-BB by UU is exact for top-BB by KK (Remark 3.5). Because GIST separates classes via fixed 4+4 quotas and never ranks across classes by UU, the cross-class γ2\gamma^2 factor never enters GIST's selection.

To measure how much exact bidirectional KL mismatch GIST captures in practice (likelihood-weighted over pooled candidates at an equal 8-of-32 response budget), we evaluated finite-pool mismatch capture on GSM8K:

Model & Candidate Pool Setting (GSM8K)Selector (Budget)Total Mismatch Captured (%) ↑\uparrowFraction of Best Same-Size Subset ↑\uparrow
Qwen3-0.6B (1,826 prompts, 32 cand./prompt)GIST (8/32, τh/τl\tau_h/\tau_l)35.400.36
Qwen3-0.6B (1,826 prompts, 32 cand./prompt)Random (8/32 uniform)30.100.30
Qwen3.5-2B (Shared pool, 32 cand./prompt)GIST (8/32, τh/τl\tau_h/\tau_l)31.400.32
Qwen3.5-2B (Shared pool, 32 cand./prompt)Random (8/32 uniform)27.100.27

GIST captures substantially more pool mismatch than Random while using only a single frozen-policy qtq_t forward pass (20x faster than full-vocabulary entropy scoring in HES: 0.08 vs. 1.66 GPU-hours at 4B scale).

W3: Step-by-Step Controlled Ablations Isolating Response and Token Selection

To separate GIST's response-level and token-level rules from prompt eligibility and checkpoint choice, we evaluate all variants on the exact same eligible prompt set under BOLT (β=0.1\beta=0.1) on Qwen3-0.6B / GSM8K, reporting exact-format match accuracy (Strict %) and change relative to the full 32/32 pool at both the validation-selected checkpoint (Val., 1,593 prompts, 256 cap) and fixed Round-5 optimizer steps (R5, 2,285 steps, 1,826 prompts, 1,024 cap):

Variant (Qwen3-0.6B / GSM8K, BOLT)Response Selection (Budget)Token Selection (% Pos. Kept)Val. (1,593 Prompts, 256 Cap)R5 (1,826 Prompts, 2,285 Steps)
(a) Full pool (Table 4a)All 32 candidates (32/32)All tokens (100%)43.82 (+0.00 pp)42.46 (+0.00 pp)
(b) Unpaired 4+4 (Table 4b)Random 4 pos. + 4 neg. (8/32)All tokens (100%)43.14 (-0.68 pp)43.21 (+0.75 pp)
(c) Token mask only (Table 4c)Random 4 pos. + 4 neg. (8/32)GIST m+/m−m^+/m^- (33.4%)43.59 (-0.23 pp)43.59 (+1.13 pp)
(d) Temp. pairing only (Table 4d)Paired τh\tau_h pos. + τl\tau_l neg. (8/32)All pos. tokens (100%) + m−m^-43.75 (-0.07 pp)43.90 (+1.44 pp)
(f) Full GIST (Table 4f)Paired τh\tau_h pos. + τl\tau_l neg. (8/32)GIST m+/m−m^+/m^- (33.4%)45.79 (+1.97 pp)45.72 (+3.26 pp)

The ablation directly confirms our theoretical mechanism:

  1. Strong 2x2 synergy between response pairing and token masking: Adding m+m^+ without temperature pairing improves accuracy by only +0.45 pp over Unpaired (Row c vs. b, since τl\tau_l positives lack low-qtq_t steps), and adding temperature pairing without m+m^+ improves accuracy by +0.61 pp (Row d vs. b, since ~67% high-qtq_t boilerplate tokens dilute gradients). Combining both yields +2.65 pp over Row b (+2.04 pp from m+m^+ on paired responses, Row f vs. d) and +1.97 to +3.26 pp over Full pool.
  2. Matched-step superiority & long-horizon stability: Across the 5-round trajectory on 1,826 prompts, GIST beats All-candidates at every round (+0.6 to +3.26 pp). While unmasked Random is competitive in early rounds before overfitting, continued offline optimization causes Random to degrade sharply (falling to 43.21% at R5 and 41.62% at Round 10, vs. 45.72% for GIST at R5). Separately, 70.9%-85% of excluded GSM8K prompts are already-mastered easy items lacking τl\tau_l errors (P(x)→1P(x) \to 1), while excluding all-failure prompts improves matched-step BOLT accuracy by +2.04 pp on 0.6B and +2.27 pp on 2B with no loss on the hardest test bin (3/148 vs. 2/148 on 0.6B; 5/80 vs. 5/80 on 2B).

W4: Loss-Bearing Tokens vs. Compute Savings, Wall-Clock Time, and Memory

We agree on per-step mechanics: under standard dense training, masked positions stay in the attention context during forward and backward passes (Section 5.4), so token masking does not reduce per-sequence FLOPs or peak VRAM (9.3 GB on 0.6B). Instead, total wall-clock savings come from two distinct mechanisms:

  1. Response subselection (32 to 8): Selecting 8 of 32 responses directly cuts per-round sequence volume by 4x (reducing fixed 5-round training time on 4B GSM8K from 8.24h to 1.67h vs. 1.68h for Random, and on 4B Code from 3.70h to 1.16h vs. 1.18h for Random).
  2. Faster convergence via token masking at the same 8-response budget: Holding the response budget fixed at 8/32 responses, unmasked Random (8/32) requires 28.9M processed tokens (2.09 GPU-hours) to reach its validation peak, whereas GIST (8/32)—by masking ~67% of boilerplate tokens to reduce gradient variance—converges in 2.6x fewer steps/hours (10.8M processed tokens, 0.80 GPU-hours) while achieving +2.65 pp higher accuracy.

Table 4A: Qwen3-0.6B Training Wall-Clock Time, Token Volume, and Peak VRAM (GSM8K, 1x RTX 4090 to Val. Ckpt)

Method (Response Budget)Train GPU-Hours ↓\downarrowProcessed Tokens (incl. masked)Loss-Bearing TokensPeak VRAM
All candidates (32/32)4.3157.6M100% of 32 cand.9.3 GB
Random (8/32, all tokens)2.0928.9M100% of 8 cand.9.3 GB
GIST (8/32, m+/m−m^+/m^- mask)0.8010.8M33.4% of 8 cand.9.3 GB

Table 4B: Qwen3-4B End-to-End Pipeline Stage Breakdown on GSM8K (5 Rounds R5, 1,024 Cap)

Pipeline StageAll Candidates (32/32)GIST (8/32)
Shared Rollout Generation (16 τh\tau_h + 16 τl\tau_l/prompt)3.80 GPU-h3.80 GPU-h
Frozen-Policy Scoring (reference forward pass)0.00 GPU-h0.08 GPU-h
Offline Policy Optimization (5 rounds, BOLT)8.24 GPU-h1.67 GPU-h
Total End-to-End Cost12.04 GPU-h5.55 GPU-h

W5: Mixed-Temperature Pool and Off-Policy Importance Weighting

In GIST, all correct responses are drawn exclusively from τh=1.0\tau_h = 1.0 (directly from πref\pi_{ref}), while only incorrect responses come from τl=0.3\tau_l = 0.3 (carrying e−10e^{-10} weight under BOLT with β=0.1\beta=0.1).

To directly test whether GIST is sensitive to off-policy weighting under the mixed-temperature proposal μ(y)=0.5π0.3(y)+0.5π1.0(y)\mu(y) = 0.5\pi_{0.3}(y) + 0.5\pi_{1.0}(y), we applied exact sequence-level importance weights a(y)=πref(y)/μ(y)a(y) = \pi_{ref}(y)/\mu(y) to GIST on Qwen3-0.6B / GSM8K (1,826 shared prompts, BOLT β=0.1\beta=0.1, 8/32 responses, 1,024-token cap, Strict accuracy):

GIST Importance-Weighting Variant (1,826 Prompts, 1,024 Cap)Val. Acc. (%) ↑\uparrowR5 Acc. (%) ↑\uparrow
GIST, no IS correction (default)46.0245.72
GIST, self-normalized IS45.8745.87
GIST, plain IS46.1046.10
GIST, truncated IS (cap 1.5)46.2545.41

Adding exact or variance-reduced importance weights shifts GIST's accuracy by only −0.31-0.31 to +0.23+0.23 pp, confirming that GIST's gains are insensitive to off-policy importance correction.

W6: Validating the Incorrect-Response Branch Across Positive-Only and Negative-Gradient Objectives

GIST is an upstream data selector that pairs seamlessly with two downstream objective families depending on their gradient sign:

  1. Positive-likelihood objectives (BOLT, SFT): Standard BOLT (β=0.1\beta=0.1) assigns incorrect responses a near-zero weight of e−10≈4.5×10−5e^{-10} \approx 4.5 \times 10^{-5}, whereas unweighted SFT assigns a positive imitation weight (w=1w=1) to retained tokens (so avoiding positive error imitation by training on GIST's 4 τh\tau_h positives with m+m^+—GIST-Pos, just 4/32 responses—reaches 46.02% Strict on Qwen3-0.6B / GSM8K, vs. 43.97% when positively imitating errors under w=1w=1 SFT, and 62.62% on Qwen3.5-2B BOLT).
  2. Negative-gradient objectives (CE + Unlikelihood): Evaluating the active contribution of GIST's τl\tau_l incorrect responses and m−m^- mask (mean qt=0.974q_t = 0.974) requires a negative-gradient operator. Using the exact same codebase, shared eligible candidate pool, and GSM8K (1,024-token cap) protocol on Qwen3.5-2B (where BOLT All is 60.65% and BOLT Random is 60.88%), we ran three independent training seeds comparing Pos.-Only CE (equivalent to positive-only SFT/BOLT on the 4 τh\tau_h positives with m+m^+) against Full GIST adding unlikelihood loss −log⁡(1−πθ(yt∣st))-\log(1 - \pi_\theta(y_t|s_t)) on m−m^- tokens of the 4 τl\tau_l incorrect responses:
Method & Token Supervision (Qwen3.5-2B / GSM8K, 1,024 Cap, 3 Seeds)Response BudgetSeed 1 / 2 / 3 Val. (%)Val. Mean ±\pm SD (%)Val. GainR5 Gain
Pos.-Only CE (== GIST-Pos SFT/BOLT, 4 τh\tau_h pos. with m+m^+)4/3260.88 / 60.34 / 60.8860.70 ±\pm 0.31+0.00 pp+0.00 pp
CE + Unlikelihood (Rand. Neg. Mask) (equal neg. token budget)8/3261.57 / 62.92 / 62.6362.37 ±\pm 0.71+1.68 ±\pm 0.62 pp+0.15 ±\pm 1.05 pp
CE + Unlikelihood (Full GIST) (4 τh\tau_h pos. m+m^+ + 4 τl\tau_l neg. m−m^-)8/3262.02 / 62.84 / 63.0862.65 ±\pm 0.56+1.95 ±\pm 0.71 pp+1.54 ±\pm 0.22 pp

(Under larger BOLT temperatures β∈{0.3,0.5,1.0}\beta \in \{0.3, 0.5, 1.0\} on Qwen3-0.6B / GSM8K, GIST also consistently outperforms All-candidates by +1.97 to +2.27 pp at 4x lower response cost.)

When paired with a negative-gradient update on the exact same benchmark and codebase, GIST's incorrect-response branch delivers consistent gains across all 3/3 seeds (+1.95 ±\pm 0.71 pp at Val. and +1.54 ±\pm 0.22 pp at R5 over Pos.-Only CE), outperforms equal-budget random negative token masking by +1.39 ±\pm 1.22 pp at R5 (+1.54 vs. +0.15 pp), and cuts R5 generation truncation from 49%-58% down to 29%-37%.

Manuscript revisions

We have renamed Corollary 3.2 as a Pre-Training Selection Bound and clarified Remark 3.5 in Section 3 / Appendix A.3, replaced "supervised tokens" with "loss-bearing tokens", and added the wall-clock GPU-hour and memory tables (Tables 4A-4B), the three-seed CE + Unlikelihood ablations, the finite-pool mismatch capture diagnostics, and the importance-sampling sensitivity results to Section 5 and Appendix B of the revised manuscript.

Dear Reviewer th2Y,

Thank you again for your detailed and thoughtful review. With the discussion period coming to a close, we wanted to kindly follow up in case you have had a chance to look over our response. In the revised manuscript, we have made several substantial updates based on your suggestions:

  • We reframed GIST as theory-guided, explicitly separating exact reference identities, bound-supported token rules, and tractable proxies, alongside finite-pool mismatch capture diagnostics.
  • We added controlled ablations isolating response and token selection on fixed prompt sets, verified robustness to off-policy importance-sampling weights, and reported complete wall-clock GPU-hours, peak memory, and scoring overhead.
  • We clarified how our selection rules align with the update sign of downstream objectives, supported by three-seed CE + Unlikelihood experiments on Qwen3.5-2B.

We hope these revisions resolve your concerns, and we look forward to any feedback you may have. We would be glad to clarify anything further if needed.

Reviewer_zj1JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GIST, a response- and token-level selection method for offline RL with verifiable rewards. The method is motivated by a bidirectional KL analysis of the mismatch between the current policy and the Gibbs-optimal target. GIST selects correct responses from on-policy samples and incorrect responses from low-temperature samples, then supervises low-probability tokens in correct responses and high-probability tokens in incorrect responses. GIST improves average performance under both BOLT and SFT. In Qwen3-0.6B/GSM8K comparison, it achieves higher accuracy while using 66.6% fewer supervised tokens than training on all candidates.

Critical strengths
  1. the proposed method is pratical and it is straightforward to integrate into existing offline training pipelines.

  2. the paper had unfied analytical perspective to connect response selectionn and outcome dependent token superision. The exact factorization at the reference policy provides useful insight into the selection problem.

  3. the paper demonstrates consistent empirical improvement.

Critical weaknesses

The biggest concern is the limited scope of evaluation: baseline comparisons and ablations focus on one model–dataset setting, and each result comes from a single training run.

Questions for authors

Could you extend the controlled baseline comparison beyond Qwen3-0.6B/GSM8K, particularly to code generation and larger models, and evaluate whether filtering out the hardest prompts affects performance on difficult test examples?

3 comments

We thank Reviewer zj1J for the positive assessment of GIST's practicality, unified analytical perspective, and empirical gains, as well as the helpful suggestions to expand our controlled evaluations across model scales, code generation, and prompt-filtering diagnostics. We have incorporated all corresponding revisions into the updated manuscript.

W1 & Q1 (Part 1): Evaluation Scope, Scaling to Larger Models / Code, and Multi-Seed Robustness

To broaden our evaluation scope and assess run-to-run stability, we extended the full 9-method controlled comparison from Table 3 (where all selectors train on the exact same eligible prompt set under BOLT, β=0.1\beta=0.1, selecting 8/32 responses vs. 32/32 for All) to Qwen3.5-2B, Gemma-4-E2B-it, and Qwen3-4B, and further evaluated MATH-500 and out-of-distribution code generation (training on TACO Python problems and testing zero-shot on MBPP-500 and HumanEval-164).

Within each row, all methods use the exact same eligible prompt pool, generation token limit (cap), and checkpoint rule: either the best validation-selected checkpoint (Val.) or the fixed Round-5 checkpoint (R5) at matched optimizer steps (Matched steps). Math tasks report exact-format match accuracy (Strict), and code tasks report unit-test Pass@1.

Table 1A: Controlled Baseline Comparison on GSM8K Across Four Model Scales (Strict %, BOLT β=0.1\beta=0.1; 8/32 Selected except All = 32/32)

Model & Setting (Cap / Checkpoint)GISTAllRandomETSHESPODSRFTRSI-SProFit
Qwen3-0.6B (1,593 prompts, 256 cap, Val.)45.7943.8243.1443.7544.2043.5244.0543.2142.46
Qwen3.5-2B (Shared pool, 1,024 cap, Matched steps)62.6260.6560.8860.3560.7360.5060.5860.2060.05
Gemma-4-E2B-it (Shared pool, 1,024 cap, Val.)14.2513.1213.0413.9513.7213.2713.4912.9612.81
Qwen3-4B (Shared pool, 1,024 cap, R5)88.1787.7288.1787.0487.2687.5787.3486.9686.73

Table 1B: Controlled Scaling to MATH-500 (Strict %) and Code Generation (Pass@1 %, BOLT β=0.1\beta=0.1)

Model & TaskExperimental Setting (Cap / Checkpoint)GIST (8/32) (%) ↑\uparrowAll Cand. (32/32) (%) ↑\uparrowBest Other (8/32) (%) ↑\uparrow
Qwen3-4B / MATH-500Shared pool, 1,024 cap, Val., Strict66.0065.0066.00 (RFT)
Qwen3-0.6B / MBPP-500TACO to MBPP, 2,048 cap, R523.0018.4021.20 (Random)
Qwen3.5-2B / MBPP-500TACO to MBPP, 2,048 cap, R539.8037.0035.00 (Random)
Qwen3-4B / MBPP-500TACO to MBPP, 2,048 cap, R564.8064.8063.60 (Random)
Qwen3-0.6B / HumanEvalTACO to HE-164, 2,048 cap, Val.39.6339.6335.37 (Random)
Qwen3.5-2B / HumanEvalTACO to HE-164, 2,048 cap, Val.60.3755.4960.37 (Random)
Qwen3-4B / HumanEvalTACO to HE-164, 2,048 cap, Val.82.9382.9382.32 (Random)
  • Accuracy across scales and domains: In Table 1A, GIST achieves the highest score across all four models, outperforming all eight baselines on 0.6B, 2B, and Gemma-E2B, and exceeding every heuristic selector (+0.60 to +1.44 pp) as well as full-pool training (+0.45 pp) on Qwen3-4B. In Table 1B, GIST also improves Qwen3-4B on MATH-500 (+1.00 pp over All-candidates; +1.40 pp over Random/HES) and yields clear zero-shot transfer gains on MBPP-500 (+4.60 pp on 0.6B and +2.80 pp on 2B at Round 5).
  • Training compute and data savings (3.2x-5.4x speedup): Every GIST run trains on only 25% of candidate responses (8/32) and ~33% of loss-bearing tokens. Where Qwen3-4B approaches saturation (e.g., 88.17% on GSM8K and 82.93% on HumanEval), GIST matches or slightly exceeds full-pool accuracy while cutting training wall-clock time by 5.4x on 0.6B GSM8K (0.80 vs. 4.31 GPU-hours to Val., and 2.6x faster than Random's 2.09h), 4.9x on 4B GSM8K (1.67 vs. 8.24 GPU-hours over 5 rounds, with frozen-policy scoring taking 0.08 GPU-hours vs. 1.66h for HES), and 3.2x on 4B Code (1.16 vs. 3.70 GPU-hours).

W1 (Continued): Multi-Seed Training Stability

Beyond the default single-seed benchmark runs in Table 1A, we ran three additional independent training seeds under BOLT (β=0.1\beta=0.1) comparing GIST (8/32) against All (32/32) on both Qwen3-0.6B and Qwen3.5-2B (GSM8K, 1,319 test problems):

Table 2: Three-Seed Training Replication on GSM8K (Strict %, BOLT β=0.1\beta=0.1)

Model & SettingMethod (Budget)Seed 1 / 2 / 3 Acc. (%)Mean ±\pm SD (%)Gain vs. All
Qwen3-0.6B (256 cap, Val.)GIST (8/32)45.56 / 45.79 / 46.0245.79 ±\pm 0.23+1.97 pp
Qwen3-0.6B (256 cap, Val.)All (32/32)43.82 / 43.59 / 44.0543.82 ±\pm 0.23+0.00 pp
Qwen3.5-2B (1,024 cap, Matched steps)GIST (8/32)62.32 / 62.93 / 62.6262.62 ±\pm 0.30+1.89 pp
Qwen3.5-2B (1,024 cap, Matched steps)All (32/32)60.58 / 61.03 / 60.5860.73 ±\pm 0.26+0.00 pp

Variance across seeds is small (SD ≤0.30%\le 0.30\%): on Qwen3-0.6B, GIST solves 601, 604, and 607/1,319 items vs. 578, 575, and 581/1,319 for All; on Qwen3.5-2B, GIST solves 822, 830, and 826/1,319 vs. 799, 805, and 799/1,319 for All, with the lowest GIST seed above the highest All seed on both models. Together with our three-seed negative-gradient ablation on Qwen3.5-2B (+1.95 ±\pm 0.71 pp over positive-only training) and multi-β\beta sweeps (β∈{0.1,0.3,0.5,1.0}\beta \in \{0.1, 0.3, 0.5, 1.0\}), these runs confirm consistent improvements at 4x lower response cost.

Q1 (Part 2): Impact of Filtering Out the Hardest Prompts

Filtering out unsolvable prompts improves overall test accuracy (+2.04 to +2.27 pp) while avoiding wasted gradient updates and preserving accuracy on the hardest test questions:

  1. Preventing error imitation without hurting hard-bin performance: On prompts where all 32 candidates fail (n1=0n_1=0), standard BOLT assigns unit weight to every failed rollout and imitates them (Appendix B.3). Under matched optimizer steps on GSM8K (1,024-token cap, Strict accuracy), excluding all-failure prompts improves overall accuracy while leaving solved counts in the hardest test bucket unchanged or slightly higher:

Table 3: Effect of Excluding All-Failure Prompts (GSM8K, 1,024 Cap, Strict %, Matched-Update BOLT β=0.1\beta=0.1)

Model & Experimental SettingRetain All-Failure (%)Exclude All-Failure (%)Overall GainHardest Test Bin (Retain vs. Exclude)
Qwen3-0.6B (GSM8K, 1,024 cap, matched steps)42.9945.03+2.04 pp2/148 vs. 3/148
Qwen3.5-2B (GSM8K, 1,024 cap, matched steps)59.4461.71+2.27 pp5/80 vs. 5/80
  1. Most filtered prompts are already-mastered easy items: Decomposing excluded GSM8K prompts shows that 70.9% (3,649/5,147 on 0.6B), 83% (2B), and 85% (Gemma) are dropped solely for lacking four τl\tau_l incorrect responses (already-mastered items where P(x)→1P(x) \to 1, κ1(x)→1\kappa_1(x) \to 1, and KL mismatch vanishes per Prop. 3.4), rather than lacking correct responses (29.0% on 0.6B). Skipping these zero-mismatch prompts saves over 70% of optimization steps on already-solved problems, while GIST also solves more middle-difficulty 0.6B test items (50/396 and 114/264 vs. 47/396 and 90/264 for BOLT).

Manuscript revisions:

We have added the expanded model-scale and code-generation comparisons (Tables 1A-1B), the three-seed training replications (Table 2), and the prompt-filtering analyses (Table 3) to Section 5 and Appendix B of the revised manuscript.

Dear Reviewer zj1J,

Thank you again for your constructive feedback on our submission. As the discussion period is nearing its end and we have not yet heard back from you, we wanted to send a brief, polite follow-up. We have revised our manuscript to directly address your comments, including the following key updates:

  • We extended our shared-pool experiments to larger models (Qwen3.5-2B, Gemma-4-E2B-it, and Qwen3-4B) across eight baselines, as well as MATH-500 and code generation (MBPP and HumanEval).
  • We added three-seed training replications on Qwen3-0.6B and Qwen3.5-2B to verify low variance and consistent gains across runs.
  • We analyzed prompt filtering in detail, showing that most excluded prompts are already-mastered easy questions and that filtering out all-failure prompts improves overall accuracy without hurting performance on the hardest test cases.

We hope these additions address your concerns and would greatly appreciate hearing your thoughts before the discussion closes. Please let us know if you have any remaining questions.

Comments on the paper

2 comments

Final Summary of Revisions and Rebuttal Discussion (Part 1/2)

We sincerely thank the Area Chair and all four reviewers for their time, careful reading, and constructive feedback throughout the review and discussion period. We are encouraged that the reviewers highlighted several core strengths of our work:

  • Practical relevance and drop-in usability: addressing a practical and relevant data-selection problem for offline RLVR (AFyz) with a lightweight method relying only on verifier outcomes, sampling temperature, and frozen-policy token probabilities, making it straightforward to integrate into existing BOLT or SFT pipelines without modifying their optimization procedures (zj1J, th2Y).
  • Unified two-level perspective: connecting response-level selection and outcome-dependent token supervision through a unified analytical lens and exact reference-policy factorization (zj1J), with an intuitive distinction between underrepresented correct behavior and confidently generated errors (th2Y).
  • Consistent cross-objective gains and efficiency: delivering consistent empirical improvements across models and multiple training objectives—confirming that the benefits are not limited to a single objective—while achieving higher accuracy with substantially fewer loss-bearing tokens (zZx4, zj1J).

During the discussion, we greatly appreciated the constructive engagements that led to score increases from Reviewer AFyz (2 to 3) and Reviewer zZx4 (1 to 2), with Reviewer zZx4 explicitly confirming that our controlled length evaluation (W4/Q2) and finite-pool ∑Ky\sum K_y / (328)\binom{32}{8} diagnostics (W2/Q1) fully resolved their concerns. While not all reviewers were able to provide a final response before the discussion window closed, we believe that our manuscript revisions and additional experiments have thoroughly addressed every concern raised across all four reviews.

Round-1 Revisions (Incorporated in the Updated Manuscript)

  1. Theory Framing & Cross-Objective Alignment (th2Y W1/W2/W6, AFyz W1, zZx4 W2/W3): We reframed GIST as theory-guided (separating exact within-class identities, bound-supported token rules, and tractable proxies) and clarified the scope of the cross-class constant. Controlled ablations confirm that positive-likelihood objectives (BOLT β=0.1\beta=0.1, SFT) benefit from the positive branch (GIST-Pos, 4/32 responses), while pairing Full GIST with a negative-gradient objective (CE + Unlikelihood on m−m^- across 3 seeds on Qwen3.5-2B / GSM8K) improves accuracy by +1.95 ± 0.71 pp Val. (62.65% vs. 60.70%) and +1.54 ± 0.22 pp R5 while reducing late-round truncation.
  2. Scaling, Code Benchmarks, Multi-Seed Stability, and Prompt Filtering (zj1J W1/Q1, th2Y W3): We extended shared-pool comparisons across eight baselines to Qwen3.5-2B, Gemma-4-E2B-it, and Qwen3-4B as well as MATH-500 and code generation (MBPP and HumanEval), added three-seed replications on Qwen3-0.6B and Qwen3.5-2B, and showed that 70.9%–85% of excluded prompts are already-mastered easy items while excluding all-failure prompts improves accuracy (+2.04 pp on 0.6B; +2.27 pp on 2B) with zero degradation on the hardest test buckets.
  3. Evaluation Length, Off-Policy IS, and Compute Accounting (th2Y W4/W5, zZx4 W4/Q2, AFyz W2): We upgraded primary GSM8K evaluations to a 1,024-token cap (widening the Qwen3.5-2B gain to +1.74 pp), stated generation caps for all benchmarks, verified robustness to sequence-level importance sampling, renamed "supervised tokens" to "loss-bearing tokens", and reported complete wall-clock GPU-hours (3.2x–4.9x training speedup at 4B scale with 0.08h scoring overhead) and peak memory.

Final Summary of Revisions and Rebuttal Discussion (Part 2/2)

Round-2 Targeted Follow-Up Diagnostics (To Be Included in the Final Version)

  1. Direct Token- and Response-Level Mismatch Verification (th2Y Follow-Up): Using unbiased Rao-Blackwellized prefix rollouts across 150 GSM8K prompts (9,600 prefixes, ~585k continuations on Qwen3-0.6B), we directly verified that the 29.2% of tokens retained by m+m^+ capture 78.39% (β=0.1\beta=0.1) and 97.08% (β=0.5\beta=0.5) of empirical token mismatch (vs. 34.81% / 33.97% for random masking and 3.64% / 1.29% for inverted masking). Across all 1,826 eligible prompts, τl=0.3\tau_l=0.3 captures 57.28% of incorrect-class mismatch (>2.0x random), while τh=1.0\tau_h=1.0 deliberately trades sequence-level capture for 3.6x higher token-level mismatch capacity and 13.0x more rare reasoning tokens.
  2. Unnormalized ∑Ky\sum K_y and (328)\binom{32}{8} Percentile Diagnostics (zZx4 W2/Q1 Follow-Up): In M=10,000M=10,000 Monte Carlo (328)\binom{32}{8} diagnostics, GIST's unnormalized ∑Ky\sum K_y exceeds all 10,000 random draws at the dataset level (top <0.01% on both 0.6B and 2B) and ranks at a per-prompt median of Top-24.71% / 24.52% overall and Top-10.33% / 15.11% on incorrect responses—fully resolving Reviewer zZx4's concern.
  3. Fixed-Quota Temperature Breakdown (zZx4 Q3 Follow-Up): Holding the quota fixed at 4:4 confirms that gains hold across valid temperatures (τl<τh≤1.0\tau_l < \tau_h \le 1.0: 44.47%–45.56% vs. 42.99% without GIST on Val.), with drops occurring only when using improper τh=1.2>1.0\tau_h = 1.2 > 1.0.

Overall, we believe that our Round-1 revisions and Round-2 diagnostics have thoroughly addressed every concern raised. All Round-1 updates have already been incorporated into the revised manuscript, and we will incorporate all Round-2 diagnostics and discussion clarifications into the final version. Thank you again to the Area Chair and all reviewers for the valuable feedback and for helping us strengthen the paper. Thank you again to the Area Chair and all reviewers for the valuable discussion and for helping us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.