acceptodds
Under review as a conference paper at ICLR 2027

Learning from the Gap Between Pass@K and Pass@1

Abstract

Sampling many responses and keeping one that passes a verifier lets large language models solve problems beyond their single-response ability, but this search must be paid again for every query, while many deployments answer with a single response. Post-training on verified responses can transfer the benefit of search into the model. With a fixed budget, selecting by correctness alone spends slots on problems the model already answers correctly, leaving fewer to correct its failures. To address this imbalance, we propose GapFT, which **trains on the gap between Pass@ and Pass@1**: problems that the source model fails with one response but solves within samples. GapFT keeps the objective and training budget fixed and changes only which verified responses enter training; an exact decomposition splits the resulting Pass@1 change into corrected failures and regressions on problems the source model already solved. On LogiQA 2.0 and ReClor with three model families, GapFT is above budget-matched uniform rejection-sampling fine-tuning (RFT) in every setting, with a positive pooled effect, and on Llama-3.1-8B and Mistral-7B it recovers about two thirds to four fifths of the gain of fine-tuning on the entire verified pool with 11–34% of its problems. Further analyses reveal that the gain comes from failures that the first few search samples recover, while failures found only by deeper search displace replay and add no net gain, that filling the same budget with gold-labeled failures search cannot reach lowers accuracy, and that the gain is bounded by how many transferable failures search exposes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training, Rejection Sampling Fine-Tuning, Data Selection, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fAgjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which verified responses should fill a fixed post-training budget when search results are distilled back into a model. It splits problems by the source model’s behaviour: already correct with one greedy response (G), wrong but recovered within K samples (S), and still wrong (U). GapFT trains only on S problems, with the objective, number of examples, exposures and update steps matched to the baselines, and an exact identity splits the Pass@1 change into corrections on S and U and regressions on G. On LogiQA 2.0 and ReClor with three 7–8B models, GapFT is numerically above matched-budget uniform rejection-sampling fine-tuning in all six settings and recovers most of the gain of training on the whole verified pool for two of the models. Controls on one model vary search depth, add late-found failures, and replace the selection with gold-labelled failures.

Critical strengths

A clean matched-budget design for a practical question. All conditions share the objective, the number of problems, exposures and optimizer steps, selection is fixed before training, and audit data never enters selection. This isolates which problems fill the budget, which is the paper’s question. Unusually candid reporting. The paper states which intervals include zero, that the pooled estimate was added after the fact, that Qwen shows no effect, that the gold-label result reverses under another recipe, and that training is not bitwise reproducible. This is why I do not recommend clear rejection.

Critical weaknesses

On the main benchmarks the method is supervised fine-tuning on gold labels for a selected subset, and the framing and baselines do not match that. LogiQA 2.0 and ReClor are multiple choice, the verifier is an exact match with the gold option, and GapFT’s training targets are answer-only strings such as “A” (Section 5, Table 4). The “verified response” for a recovered failure is therefore the gold label: search contributes the selection, not the training signal. For such tasks the probability the source gives the gold option should be available from one forward pass, and the S set is a noisy threshold on it: 36–44% of the source’s wrong items switch between S and U across independent resamples (Table 14). The natural baselines are a random labelled subset that includes U problems, and selection ranked by gold-option probability. Neither is run, and the classic work on ambiguous versus hard-to-learn examples (Swayamdipta et al., 2020) is not cited. The claim that search reachability is a useful signal holds in one recipe only. With a replayed core, gold-selected failures beat search-selected ones by +5.27 [2.93, 7.53] on ReClor (Appendix B.3, Table 10), which the main text summarises as “not worse”. In a two-seed ReClor diagnostic that adds a replayed core to the main recipe, the two are indistinguishable (−0.50 [−3.10, 2.00]). Where responses do carry reasoning (MATH), the contrast with uniform RFT has an interval that includes zero. Table 3 also has two undefined rows, “positive SFT” and “verified correction”, one of which matches GapFT (66.17 against 66.11).

The evidence for the headline and for the proposed account of the gain is weak. Only one of six GapFT–uniform intervals excludes zero, and the pooled +1.57 was added after the fact, as the paper says. Training on all verified problems is better than GapFT in all six rows. The budget excludes search (K = 8 on the whole pool), which every search-based condition pays, so the “three to almost fifty times” ratio overstates the saving in total compute. The main comparison uses a single budget, n = |S_8|, the one point at which GapFT uses every recovered failure. There is no budget sweep. The abstract and conclusion say the gain is bounded by the failures search exposes, and Section 6 says this share limits the gain. The identity bounds only the S term. Reconstructing the terms from Tables 1 and 11, correction on failures search missed (U) is comparable to or larger than correction on recovered ones (S) in four of six cells, for example about 6.5 against 2.9 points on Mistral–ReClor. The total gain does stay below the S share in every cell, but the claim needs a decomposition that accounts for the instability of the S/U split, or it should be dropped. The term differences have no intervals, while a byte-identical retrain changes correctness on 24% of LogiQA and 14% of ReClor audit items (Appendix A). Solved replay, which trains on no failures, already corrects 27–37% of S items in the three cells with at least 100 of them. The Section 5 controls use one model.

Questions for authors

Clarifications:

Are the training targets on LogiQA and ReClor the option label only, for all conditions including the greedy responses used for G? If so, how does GapFT differ from supervised fine-tuning on the gold labels of the same problems? What are “positive SFT” and “verified correction” in Table 3, and where is the code experiment of Figure 6 described? How does the claim that the gain is bounded by the S share fit with U corrections matching or exceeding S corrections on Mistral and Qwen? Why was n = |S_8| chosen as the only budget?

Additional evidence that would change my assessment (within the scope of the current experiments):

Two baselines at the same n: a random labelled subset of the whole pool (G, S and U), and wrong-greedy problems ranked by the source’s gold-option probability from one forward pass. Results at smaller and larger budgets, with replay filling the remainder. Intervals for the term differences in Table 11, and the same decomposition for a byte-identical rerun as a noise floor. Total cost including search for each condition, and the Section 5 controls on Mistral.

7 comments
AuthorsOpenReview ↗

Response to Reviewer fAgj (1/2)

Thank you for your careful review and for recognizing our matched-budget experimental design, our strict separation of selection and audit data, and our candid empirical reporting. We also deeply appreciate the time, effort, and careful thought you devoted to reviewing our manuscript.

Overview:

  • R1 (to W1, in Part 1/2). Clarifies controlled option-label targets, adds single-pass probability and Standard RFT baselines, and evaluates two open-ended generative tasks.
  • R2 (to W2, in Part 2/2). Reports S/G mixture and recipe sweeps around n=∣S8∣n = |S_8| and accounts for total search plus training compute.
  • R3 (to W2, in Part 2/2). Clarifies the within-support pSp_S bound and resampling invariance in Table 14.
  • R4 (to W1, in Part 2/2). Defines the rows in Table 3 and links Figure 6 to its protocol in Appendix A.

We hope these clarifications and new experiments resolve your concerns and earn your support for a higher score.


R1 (to W1). Targets, Single-Pass Baselines, and Open-Ended Tasks

Controlled option-label targets on MC tasks. On LogiQA 2.0 and ReClor, all conditions including greedy GG responses use the option-letter target by design. Keeping target strings and token counts identical across GG, SS, and gold-selected UU removes response-length and style differences, so any accuracy change comes strictly from which problems enter the training set. Even when targets match gold labels, which failures are selected without GG replay matters sharply. Unscreened gold failures from S∪US \cup U drop Pass@1 by 9.49 pp on LogiQA and 6.87 pp on ReClor relative to GapFT in Table 2C because unreachable items trigger heavy GG regressions. We will state this design rationale upfront in Section 4.1 and cite Swayamdipta et al. [2020] to connect GKG_K, SKS_K, and UKU_K to easy, ambiguous, and hard regions estimated before training.

Added single-pass probability baselines and Standard RFT. To compare KK-sample reachability against single-forward-pass option probabilities and practical KK-sample-only Standard RFT (Yuan et al., 2023) without greedy deduplication, we evaluated Low Confidence using lowest maximum option probability and High Entropy using highest option entropy [Settles, 2009] from one forward pass on Llama-3.1-8B-Instruct under the same sample count and cross-entropy recipe.

DatasetGold FailuresStandard RFTLow Confidence, 1-passHigh Entropy, 1-passGapFT
LogiQA 2.053.1057.3061.0360.5062.45
ReClor63.8062.4068.3368.1370.60

GapFT outperforms both single-pass probability baselines, Standard RFT, and unscreened gold failures on both benchmarks. In Appendix B.3 and Table 10, the shift when adding a shared GG core reflects a clear regime boundary. Reachability filtering prevents GG regression when training holds no GG replay, whereas replaying the full GG core at triple the sample count already anchors GG and makes gold failures and GapFT indistinguishable at −0.50-0.50 pp in our matched main-protocol diagnostic. We will report Table 10's exact +5.27 pp ReClor figure from the older protocol alongside this matched diagnostic directly in Section 5.

New open-ended tasks carrying full programs and answers. To test domains where single-token gold probabilities do not exist and search provides complete generated solutions, we added FinQA [Z. Chen et al., 2021] program synthesis on Qwen2.5-Coder-7B-Instruct and HotpotQA [Yang et al., 2018] multi-hop QA on Qwen2.5-7B-Instruct across two epochs with matched tokens and update steps over three seeds, comparing against DART-style (Tong et al., 2024) and DAPO-style (Yu et al., 2025) controls.

DatasetEpochStandard RFTDART-styleDAPO-styleGapFT
FinQAEpoch 1 / Epoch 239.06 / 44.3544.46 / 55.2742.89 / 55.0447.63 / 57.51
HotpotQAEpoch 1 / Epoch 251.44 / 52.4253.00 / 53.9752.43 / 54.5754.17 / 56.20

GapFT beats Standard RFT in every seed across both tasks and epochs and outperforms both difficulty-aware controls, showing clear gains when search supplies multi-step outputs.


AuthorsOpenReview ↗

Response to Reviewer fAgj (2/2)


R2 (to W2). Budget Choice, Mixtures, and Total Compute

Choice of n=∣S8∣n = |S_8|, S/GS/G mixtures, and recipe sweeps. Table 1 sets n=∣S8∣n = |S_8| because it uses every recovered failure once without replacement and zero GG replay. To verify that our findings hold across compositions and recipes, we swept clean SS-to-GG mixtures on LogiQA and ReClor, plus repetition counts R∈{1,2,4,8}R \in \{1, 2, 4, 8\} and learning rates on ReClor with Llama-3.1-8B-Instruct over three seeds.

S-to-G RatioLogiQA 2.0 (%)ReClor (%)
25:7559.6767.07
50:5062.0169.73
75:2561.1168.27
100:0, GapFT62.3369.93
ReClor Recipe SweepStandard RFT (%)GapFT (%)
R = 1 / 2 / 4 / 8 at LR 5e-561.07 / 62.27 / 63.60 / 62.0066.60 / 74.93 / 70.53 / 70.27
LR 2e-5 / 5e-5 / 1e-4 at R = 860.13 / 62.00 / 64.8770.80 / 70.27 / 69.27

Accuracy peaks at 100:0 on both benchmarks while 50:50 captures most of the gain, and GapFT beats Standard RFT across every repetition count and learning rate. At full-pool scale All, Table 11 shows that the remaining gap comes mainly from lower ρG\rho_G via extra GG replay rather than higher κS\kappa_S.

Total compute including search. Section 3.3 reports search depth KK separately from training budget B\mathcal{B} because candidate search cost is identical across Uniform RFT, GapFT, and All-verified at K=8K=8, and adds only one greedy pass or 12.5% extra generations over KK-only Standard RFT. Stopping search at K=1K=1 cuts search decodes from 9 to 2 per candidate, saving 78% of search compute, and filling remaining examples with GG in Table 2A already beats K=8K=8 Uniform RFT on LogiQA and K=8K=8 Standard RFT on both tasks. We will state both training-only and combined search-plus-training ratios in Section 4.2.


R3 (to W2). Decomposition Scope and Resampling Stability

Within-support pSp_S bound and UKU_K generalization. As written in Section 3.4 at line 229, the algebraic bound pSκS≤pSp_S\kappa_S \le p_S applies specifically to correction within observed support SKS_K. Sizable pUκUp_U\kappa_U gains appear because S8S_8 and U8U_8 are finite-sample observations rather than hard capability cutoffs as noted in Section 3.2, so updates on S8S_8 generalize to failures missed in eight draws. When pSp_S approaches zero as on Qwen logic, pUκUp_U\kappa_U is also tiny and overall headroom disappears. We will align the summary wording in the Abstract, Section 6, and the Conclusion with Section 3.4's within-support statement.

Resampling stability and retraining variance. Table 14 Panel B already shows that while individual items move between S8S_8 and U8U_8 across four independent resamples, combined correction, GG regression, and net Pass@1 gain have an exact 0.00 pp range across partitions. Appendix A also shows that retraining with identical data and settings changes mean Pass@1 by only 0.07 pp on LogiQA and 1.4 pp on ReClor, well below GapFT's multi-point gains over Standard RFT and generative baselines.


R4 (to W1). Table 3 and Figure 6 Clarifications

Table 3 pairing-loss ablation. "Positive SFT" in Table 3 is GapFT itself, named differently in the paired-loss ablation where "positive" denotes training only on verified correct responses to SS problems, as verifiable in our submitted anonymous code repository. Its paired counterpart, "verified correction", uses the same SS problems and correct targets while adding a pairing loss against each failed greedy response, lowering accuracy from 66.17% to 64.83%. We will unify the naming and define both rows in Section 6.1.

Figure 6 code experiment. Full settings and the contrast formula for Figure 6 appear in Appendix A Table 7 and Equation 10. Under a matched 64-update budget on LiveCodeBench with Qwen2.5-Coder-7B and DeepSeek-Coder-6.7B, selected SS items gain +38.36 pp over unselected SS items, broader SS coverage raises pass rates on newly covered coding failures by +15.09 pp in Figure 6A, and held-out audit differences stay positive on both models in Figure 6B. We will add a clear text paragraph linking Section 6.2 to Table 7 and Figure 6. Please let us know if any question remains during the discussion.


References

  • Z. Chen et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP, 2021.
  • B. Settles. Active Learning Literature Survey. CS Technical Report 1648, UW-Madison, 2009.
  • S. Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. EMNLP, 2020.
  • Z. Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.

AuthorsOpenReview ↗

Follow-up on Rebuttal - Paper 21974

Dear Reviewer fAgj,

Thank you again for your insightful feedback on our manuscript. With the discussion phase nearing its end, we are writing to confirm whether our responses have resolved all of your concerns. Should you have any additional questions, we are happy to provide further clarifications. If you are satisfied with our replies, we would appreciate your consideration to adjust your scores.

Best regards,

The Authors

Reviewer_hogBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "GapFT", which trains on correct responses from problems the model fails with greedy sampling but solves within K samples. The authors compare it with some rejecting sampling baselines under a fixed training budget. The paper considered two MCP datasets with three 7B base models.

Critical strengths
  1. Simple idea. The problem is well motivated, and the proposed method is simple and easy to understand.
  2. Good experimental controls. The authors provide several ablation studies to explain the results.
  3. Useful analysis. The paper analyzes corrected failures and regressions on previously correct problems. I also appreciate that the authors report negative results.
Critical weaknesses

Below contains both technical and presentation concerns I had during the review.

  1. Weak experimental results. Only one of the six main comparisons with uniform RFT shows a significant improvement. The MATH and code results also provide limited support, and positive SFT performs similarly to GapFT on MATH.
  2. Weak baselines. Uniform RFT includes correct greedy responses, but practical RFT may only collect K sampled responses. It is unclear how GapFT compares with this setting or other data selection methods.
  3. Lack of uncertainty analysis. A correct greedy answer does not mean the model is confident. Such examples may still benefit from replay, but the paper does not compare with confidence or entropy based selection.
  4. Data collection cost. GapFT requires greedy decoding and K samples for every candidate problem, but the comparison mainly considers the training budget. The total cost may be much larger than the cost of fine-tuning, so why training budget becomes an issue?
  5. Limited budget settings. The authors set n=|E_S|, which requires |E_G|>=|E_S| for Solved Replay. This may limit the model and dataset pairs that can be evaluated, and different budgets or mixing ratios are not well studied.
  6. Limited training analysis. The experiments use only three training seeds in the main setting. It is unclear whether the results remain similar with more training or different repetition settings.
  7. Limited evaluation. The main experiments only use two multiple-choice reasoning datasets. More results on other tasks and model sizes would make the findings more convincing.
  8. Presentation. Figure 1 provides little additional information, and some variables are used before being defined. The paper also introduces many symbols that are difficult to remember.
Questions for authors

Here are some questions I'm more interested to get feedback on. But please also feel free to refer back to the weaknesses above.

  1. How does GapFT compare with uncertainty or difficulty based data selection? For example, could replaying correct but low-confidence greedy answers be more useful?
  2. Could the authors compare with standard RFT that only collects K sampled responses? Also, why not compare with DPO using the correct and incorrect generations?
  3. What happens when |E_G|<|E_S|? How would the authors keep the budget matched, and does this limit the choice of models or datasets?
  4. Could the authors compare the total computation cost, including data generation and fine-tuning? Does GapFT still have an advantage under the same total budget?
  5. Could the authors study different ratios of E_G and E_S, different training budgets, and more training epochs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer hogB (1/2)

Thank you for recognizing our well-motivated and simple approach, our experimental controls and ablations, and our candid analysis of corrections, regressions, and negative results.We also deeply appreciate the time, effort, and careful thought you devoted to reviewing our manuscript.

Overview:

  • R1 (to W1-W3, W7; Q1, Q2, in Part 1/2). Adds KK-sample-only Standard RFT, uncertainty selection baselines, two generative tasks (FinQA, HotpotQA), and clarifies how GapFT relates to DPO.
  • R2 (to W4; Q4, in Part 2/2). Accounts for total search plus fine-tuning compute and shows advantages at equal and lower generation cost.
  • R3 (to W5, W6; Q3, Q5, in Part 2/2). Addresses ∣EG∣<∣ES∣|E_G| < |E_S| and reports full S/G mixture, repetition, and learning-rate sweeps.
  • R4 (to W8, in Part 2/2). Outlines notation and Figure 1 revisions.

We hope these new experiments and clarifications resolve your concerns and earn your support for a higher score.


R1 (to W1-W3, W7; Q1, Q2). Added Selection Baselines, Non-MC Tasks, and Loss Scope

Added Standard RFT and uncertainty baselines. Following your suggestion, we added Standard RFT (Yuan et al., 2023; Tong et al., 2024), sampling verified responses uniformly from the KK generations without adding greedy responses. We also evaluated single-pass Low Confidence using lowest maximum option probability and High Entropy using highest option entropy [Settles, 2009], keeping greedy-correct items eligible under the same cross-entropy recipe on Llama-3.1-8B-Instruct across three seeds.

DatasetStandard RFTLow ConfidenceHigh EntropyGapFT
LogiQA 2.057.3061.0360.5062.45
ReClor62.4068.3368.1370.60

GapFT beats Standard RFT by +5.15 pp on LogiQA 2.0 and +8.20 pp on ReClor, and outperforms both uncertainty baselines. Our paper's Uniform RFT kept only one verified response per solved problem, which already curbed easy-problem oversampling. Prioritizing actual greedy failures in SKS_K goes further than replaying uncertain greedy successes in GKG_K.

New open-ended non-multiple-choice tasks. Beyond MATH and code, we added FinQA [Z. Chen et al., 2021], where Qwen2.5-Coder-7B-Instruct generates executable financial calculation programs scored by execution accuracy, and HotpotQA [Yang et al., 2018], where Qwen2.5-7B-Instruct generates free-form multi-hop answers scored by exact match. On shared pools of 1 greedy and 8 sampled responses across 2,048 and 4,096 candidates, we compared against DART allocation (Tong et al., 2024) and DAPO filtering (Yu et al., 2025) with matched tokens and update steps over two epochs and three seeds.

DatasetEpochStandard RFTDARTDAPOGapFT
FinQAEpoch 1 / Epoch 239.06 / 44.3544.46 / 55.2742.89 / 55.0447.63 / 57.51
HotpotQAEpoch 1 / Epoch 251.44 / 52.4253.00 / 53.9752.43 / 54.5754.17 / 56.20

GapFT beats Standard RFT in every seed across both tasks and epochs, and achieves the highest mean accuracy over both difficulty-aware controls. We also clarify that "positive SFT" in Table 3 is GapFT itself, named differently in the paired-loss ablation where "positive" denotes training only on verified correct responses to SS problems, as can also be verified in our submitted anonymous code repository.

Relationship with DPO. GapFT and DPO address two complementary aspects of post-training.

  • Problem selection versus preference optimization. GapFT determines which problems enter training, whereas DPO [Rafailov et al., 2023] specifies how to update from paired correct and incorrect generations on a selected problem. Forming on-policy preference pairs requires prompts where the model produces both passed and failed responses, which is the SKS_K set identified by GapFT.
  • Negative-pair comparison in Table 3. On the same S8S_8 problems in MATH, Table 3 already compares adding a pairing loss against each problem's failed greedy response in "verified correction" at 64.83% with training solely on verified correct responses in "positive SFT" at 66.17%, showing that positive supervision alone performs better.

AuthorsOpenReview ↗

Response to Reviewer hogB (2/2)


R2 (to W4; Q4). Total Compute Cost Accounting

Matching the fine-tuning budget isolates the per-example value of each selection rule during training, and extending this accounting to end-to-end pipeline cost shows that GapFT keeps its advantage when data generation and fine-tuning are counted together.

CRFT=Csample-K+CFT,CGapFT=Csample-K+Cgreedy+CFT.C_{\text{RFT}} = C_{\text{sample-}K} + C_{\text{FT}}, \qquad C_{\text{GapFT}} = C_{\text{sample-}K} + C_{\text{greedy}} + C_{\text{FT}}.

Standard RFT already generates KK samples per candidate, and GapFT adds just one greedy decode, increasing generation count by 12.5% at K=8K=8. On FinQA and HotpotQA in Part 1/2, all methods share the exact same pre-collected pool and update steps, so GapFT wins at identical total compute. Even at 75% lower generation cost, stopping search at K=1K=1 with 2 generations per candidate instead of 8 and filling remaining examples with GG in Table 2A already beats K=8K=8 Standard RFT by over +4 pp on both logic tasks.


R3 (to W5, W6; Q3, Q5). Mixtures, Repetitions, and Continued Training

Handling ∣EG∣<∣ES∣|E_G| < |E_S| and S/GS/G mixtures. Requiring ∣EG∣≥∣ES∣|E_G| \ge |E_S| was only needed to build an equal-sized GG-only Solved Replay baseline at n=∣ES∣n = |E_S| without replacement. GapFT itself has no such requirement, and diagnostic comparisons when ∣EG∣<∣ES∣|E_G| < |E_S| simply set n=min⁡(∣EG∣,∣ES∣)n = \min(|E_G|, |E_S|). Practical runs also do not need to discard GG. Sweeping clean SS-to-GG ratios on Llama-3.1-8B-Instruct across three seeds at learning rate 5×10−55\times 10^{-5} shows smooth gains.

S-to-G RatioLogiQA 2.0 (%)ReClor (%)
25:7559.6767.07
50:5062.0169.73
75:2561.1168.27
100:0, GapFT62.3369.93

Accuracy peaks at 100:0 on both benchmarks, while 50:50 already captures most of the gain because added SS corrections outweigh regressions from reduced GG replay.

Repetition, learning-rate, and multi-epoch training sweeps. To directly test different repetition settings and additional training epochs on ReClor across three seeds, we swept repetition counts R∈{1,2,4,8}R \in \{1, 2, 4, 8\} at learning rate 5×10−55\times 10^{-5} and learning rates at R=8R=8.

Sweep AxisSettingStandard RFT (%)GapFT (%)
Repetition R at LR 5e-5R = 1 / 2 / 4 / 861.07 / 62.27 / 63.60 / 62.0066.60 / 74.93 / 70.53 / 70.27
Learning rate at R = 82e-5 / 5e-5 / 1e-460.13 / 62.00 / 64.8770.80 / 70.27 / 69.27

GapFT outperforms Standard RFT across every repetition setting and learning rate, and Epoch 2 on FinQA and HotpotQA in Part 1/2 likewise confirms that GapFT's lead persists under multi-epoch training in every seed.


R4 (to W8). Notation and Figure 1 Revisions

In the revision, we will define every symbol before its first appearance, including Table 3's ablation labels in Section 6.1, merge redundant subscripts into a short notation reference, and redraw Figure 1 so the GKG_K, SKS_K, and UKU_K split, data selection, and decomposition are clear on their own. Please let us know if any question remains during the discussion.


References

  • Z. Chen et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP, 2021.
  • R. Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS, 2023.
  • B. Settles. Active Learning Literature Survey. CS Technical Report 1648, UW-Madison, 2009.
  • Z. Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.

AuthorsOpenReview ↗

Follow-up on Rebuttal - Paper 21974

Dear Reviewer hogB,

Thank you again for your insightful feedback on our manuscript. With the discussion phase nearing its end, we are writing to confirm whether our responses have resolved all of your concerns. Should you have any additional questions, we are happy to provide further clarifications. If you are satisfied with our replies, we would appreciate your consideration to adjust your scores.

Best regards,

The Authors

Reviewer_Zq44OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of how to convert the accuracy a model gains from repeated sampling (pass@k) to better single response accuracy (pass@1). It studies the problem under a fixed budget fine-tuning setting. It proposes a method called GapFT, which works similarly to rejection-sampling fine-tuning, but changes how to select the data to fill the budget. Specifically, it trains only on problems the model gets wrong with one generation, but solves within K generations, instead of a uniformed sampling (their baseline). The authors perform experiments on two multi-choice logical reasoning datasets (LogiQA 2.0 and ReClor), with Llama-3.1-8B, Qwen2.5-7B and Mistral-7B. The proposed method is compared with 3 baselines, i.e., uniform rejection sampling fine tuning, replay of already solved problems and full verified pool. The results show that GapFT performs better than uniform rejection sampling in all 6 settings (though only one of them is statistically significant).

Critical strengths

The paper sheds practical light on how to pick examples for rejection sampling fine tuning, and supports it with controlled experiments. It shows that, at a fixed budget, training on problems the model fails but can solve by sampling is better than re-training on problems it already solves. On Llama-3.1-8B this swap raises pass@1 accuracy by 6.75% on LogiQA and 9.4% on ReClor. Also, experiment show that GapFT recovers most of the full verified pool's gain with 11-34% of its problems. This finding is useful in practice when building rejection sampling data for training.

Critical weaknesses
  1. The technical novelty is limited, and the results to support the paper's method is mostly not statistically significant. GapFT is essentially a standard rejection sampling fine-tuning with a selection rule w.r.t the problem difficulty. This direction (i.e., difficulty and learnability based filtering) is already proposed and studied in previous paper like DART-Math, DAPO etc. I appreciate the author to perform experiments and do more analysis here, but I don't see that justify enough as a new paper.

  2. the experiment setup is narrow, it is unclear whether the finding generalize to realistic post training tasks. The main experiments results are based on 4 option multi-choice tasks and the training target is only the answer letter (very unlikely for real tasks). Thus, the idea of transferring search into reasoning is not tested here. Also, other things: the experiments are done only with LoRA but not full fine-tuning, which I would argue full weight training is more often in post training. It only trains the model with a single learning rate for 1 epoch, and the impact of hyper-parameter is not ruled out at all. The model size used are all 8B, it is not clear whether results will transfer well when model size go up. All of the above will limit the paper's significance.

Questions for authors
  1. How does the results change if we do the model training with full weight fine-tuning, and with different hyper-parameters?
  2. Have you done any experiments with non multi-choice tasks? It will be very useful to see how it will transfer to more realistic tasks rather than multi-choice.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Zq44 (1/2)

Thank you for recognizing the practical value of our data-selection question, our controlled experimental design, and the data efficiency of recovering most full-pool gains from a small subset of problems. We also deeply appreciate the time, effort, and careful thought you devoted to reviewing our manuscript.

Overview:

  • R1 (to W1, in Part 1/2). Clarifies the selection stage and object of GapFT versus DART-Math and DAPO, adds Standard RFT and uncertainty baselines, and outlines our full chain of evidence.
  • R2 (to W2, Q2, in Part 1/2). Adds two two-epoch generative benchmarks, FinQA and HotpotQA, against difficulty-aware controls.
  • R3 (to W2, Q1, in Part 2/2). Evaluates full-weight fine-tuning versus LoRA through our correction and regression decomposition.
  • R4 (to W2, Q1, in Part 2/2). Reports learning-rate and repetition sweeps, S/G mixtures, and model scaling analysis.

We hope these new experiments and analyses resolve your concerns and earn your support for a higher score.


R1 (to W1). Method Distinction, New Strong Baselines, and Significance

Different stage and selection object. Prior methods filter by prompt difficulty or rollout variance at other stages rather than what the target model already gets right in one decode.

  • DART-Math (Tong et al., 2024) acts during synthetic data generation, allocating extra sampling trials to prompts with high fail rates under an external synthesis model rather than testing selection by whether the target policy itself already solves a prompt greedily.
  • DAPO (Yu et al., 2025) acts during online RL, dropping uniform-reward groups at sampling temperature, which removes only 29.1% of greedy-correct items in Appendix B.5 and does not isolate single-decode redundancy.
  • GapFT acts on offline training-data selection, choosing problems directly from the gap between the target model's own single-decode and multi-sample outcomes (SKS_K versus GKG_K) to focus on recoverable greedy failures rather than single-attempt successes, while matching examples, tokens, and steps ensures a fair comparison across selection rules.

Stronger performance against practical RFT and uncertainty baselines. In the paper, Uniform RFT kept at most one verified response per solved problem, which already curbed easy-problem oversampling. To test practical significance directly, we added Standard RFT (Yuan et al., 2023), sampling verified responses uniformly from the KK-sample pool without problem deduplication, plus single-pass Low Confidence and High Entropy selection [Settles, 2009] on Llama-3.1-8B-Instruct across three seeds.

DatasetStandard RFTLow ConfidenceHigh EntropyGapFT
LogiQA 2.057.3061.0360.5062.45
ReClor62.4068.3368.1370.60

GapFT beats Standard RFT by +5.15 pp on LogiQA 2.0 and +8.20 pp on ReClor, outperforms both uncertainty baselines, and also surpasses matched-pool DART and DAPO controls in R2 below. Together with our exact decomposition showing that gains come from reachable SKS_K failures while unreachable UKU_K items trigger heavy GKG_K regressions, combining these matched-pool baselines (R1), two-epoch generative tasks (R2), full-weight analysis (R3), and hyperparameter and mixture sweeps (R4) establishes a systematic empirical principle for verified post-training data selection.


R2 (to W2, Q2). New Non-MC Reasoning and Generation Tasks

Option-letter targets were used on the logic tasks to hold target length constant during causal controls such as answer permutation and gold-failure replacement. To test distilling search into multi-step generation beyond MATH and LiveCodeBench, we added FinQA [Z. Chen et al., 2021], where Qwen2.5-Coder-7B-Instruct writes executable financial calculation programs scored by execution accuracy, and HotpotQA [Yang et al., 2018], where Qwen2.5-7B-Instruct generates free-form multi-hop answers scored by exact match. On shared pools of 1 greedy and 8 sampled responses across 2,048 and 4,096 candidates, we compared against DART allocation and DAPO filtering with matched tokens and steps over two epochs and three seeds.

DatasetEpochStandard RFTDARTDAPOGapFT
FinQAEpoch 1 / Epoch 239.06 / 44.3544.46 / 55.2742.89 / 55.0447.63 / 57.51
HotpotQAEpoch 1 / Epoch 251.44 / 52.4253.00 / 53.9752.43 / 54.5754.17 / 56.20

GapFT outperforms Standard RFT in every seed across both tasks and epochs, and achieves the highest mean accuracy over both difficulty-aware controls. State-aware selection transfers reliably when models learn complete calculation programs and multi-hop answers rather than option letters.


AuthorsOpenReview ↗

Response to Reviewer Zq44 (2/2)

R3 (to W2, Q1). Full-Weight Fine-Tuning Versus LoRA

To directly answer how full-weight training affects our findings, we evaluated full-weight fine-tuning alongside LoRA under the same ReClor setup on Llama-3.1-8B-Instruct and decomposed the resulting Pass@1 changes.

Method on ReClorPass@1 (%)Δ\Delta vs. Source (pp)SS correction κS\kappa_S (%)GG regression ρG\rho_G (%)
Source55.40---
GapFT with LoRA70.67+15.2763.1017.10
GapFT with Full FT59.87+4.4750.2631.53

Full fine-tuning still corrects over half of unseen SS failures and improves over the source model, confirming that the SKS_K selection signal holds beyond LoRA. At the same time, updating all 8B parameters on a small pure-SS set without GG replay nearly doubles regressions on previously solved GG items, empirically explaining why low-rank adaptation or pairing full-weight updates with GG replay better protects prior capabilities when training on targeted failure sets.


R4 (to W2, Q1). Hyperparameter Sweeps, Mixtures, and Model Scale

Repetition and learning-rate sweeps. On ReClor with Llama-3.1-8B-Instruct across three seeds, we swept repetition count RR at our default learning rate of 5×10−55\times 10^{-5} and swept the learning rate at R=8R=8.

Sweep axisSettingStandard RFT (%)GapFT (%)
Repetition RR at LR 5×10−55\times 10^{-5}R=1R = 161.0766.60
R=2R = 262.2774.93
R=4R = 463.6070.53
R=8R = 8, paper default62.0070.27
Learning rate at R=8R = 82×10−52\times 10^{-5}60.1370.80
5×10−55\times 10^{-5}, paper default62.0070.27
1×10−41\times 10^{-4}64.8769.27

GapFT beats Standard RFT at every repetition count and learning rate, and GapFT's lowest score across the grid still exceeds Standard RFT's best tuned score. Peaking at R=2R=2 shows that our untuned default of R=8R=8 was conservative rather than cherry-picked. Extra passes improve accuracy while exposures remain unsaturated, as seen from Epoch 1 to Epoch 2 in R2 at R=1R=1, whereas heavy repetition on a fixed pure-SS set adds mild forgetting without reversing GapFT's lead.

Data composition mixtures. Sweeping clean SS-to-GG mixtures on Llama-3.1-8B-Instruct across three seeds at learning rate 5×10−55\times 10^{-5} under matched sample counts confirms smooth behavior across compositions.

SS-to-GG ratioLogiQA 2.0 (%)ReClor (%)
25:7559.6767.07
50:5062.0169.73
75:2561.1168.27
100:0, GapFT62.3369.93

All mixtures improve substantially over the base model and peak at 100:0 on both benchmarks, while 50:50 already captures most of the gain because added SS corrections outweigh regressions from reduced GG replay.

Transfer across model scales. Across five 6.7B to 8B backbones of varying base strength, gains track the size of the Pass@K minus Pass@1 gap on the target task rather than parameter count itself. Strong Qwen2.5-7B leaves little gap on multiple-choice logic, yet achieves clear gains on harder MATH and HotpotQA tasks where search uncovers fresh recoverable failures. Larger models face this exact same dynamic on harder frontier tasks where search still outperforms single-pass decoding. Please let us know if any question remains during the discussion.


References

  • Z. Chen et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP, 2021.
  • B. Settles. Active Learning Literature Survey. CS Technical Report 1648, UW-Madison, 2009.
  • Z. Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.

AuthorsOpenReview ↗

Follow-up on Rebuttal - Paper 21974

Dear Reviewer Zq44,

Thank you again for your insightful feedback on our manuscript.

With the discussion phase nearing its end, we are writing to confirm whether our responses have resolved all of your concerns. Should you have any additional questions, we are happy to provide further clarifications. If you are satisfied with our replies, we would appreciate your consideration to adjust your scores.

Best regards,

The Authors

AuthorsOpenReview ↗

Additional 14B and Multi-Model HotpotQA Results for Reviewer Zq44

Dear Reviewer Zq44,

As a brief update to our earlier Response to Reviewer Zq44 (1/2) and Response to Reviewer Zq44 (2/2), we completed additional three-seed HotpotQA experiments across Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen3-14B in non-thinking mode, using learning rate 5e-5 and LoRA r = 8, alpha = 16, dropout = 0.05. Adding Qwen3-14B directly addresses your question on scaling beyond 8B models and tests a newer model generation on an open-ended task against Standard RFT, DART, and DAPO.

ModelEpochStandard RFT (%)DART (%)DAPO (%)GapFT (%)
Llama-3.1-8B-InstructEpoch 154.97 ± 0.4557.52 ± 0.4856.84 ± 0.5258.96 ± 0.53
Llama-3.1-8B-InstructEpoch 255.77 ± 0.4658.67 ± 0.4957.88 ± 0.4360.69 ± 0.42
Mistral-7B-Instruct-v0.3Epoch 148.68 ± 0.5451.65 ± 0.5151.46 ± 0.5553.12 ± 0.53
Mistral-7B-Instruct-v0.3Epoch 248.19 ± 0.6348.19 ± 0.5252.13 ± 0.5354.34 ± 0.53
Qwen3-14B (non-thinking)Epoch 153.97 ± 0.4855.73 ± 0.4854.73 ± 0.4556.19 ± 0.37
Qwen3-14B (non-thinking)Epoch 253.85 ± 0.5056.86 ± 0.4056.05 ± 0.4357.66 ± 0.52

GapFT outperforms Standard RFT, DART, and DAPO across all three models and both epochs. Together with Qwen2.5-7B on HotpotQA and Qwen2.5-Coder-7B on FinQA from our first reply, these results confirm that GapFT transfers reliably to realistic open-ended generation and scales to 14B models.

Thank you again for your time and thoughtful feedback. We sincerely hope these additional 14B and multi-model results address your concerns and earn your support for a higher score. Please let us know if any question remains before the discussion closes.

Best regards,

Authors of Submission 21974

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.