- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks which verified responses should fill a fixed post-training budget when search results are distilled back into a model. It splits problems by the source model’s behaviour: already correct with one greedy response (G), wrong but recovered within K samples (S), and still wrong (U). GapFT trains only on S problems, with the objective, number of examples, exposures and update steps matched to the baselines, and an exact identity splits the Pass@1 change into corrections on S and U and regressions on G. On LogiQA 2.0 and ReClor with three 7–8B models, GapFT is numerically above matched-budget uniform rejection-sampling fine-tuning in all six settings and recovers most of the gain of training on the whole verified pool for two of the models. Controls on one model vary search depth, add late-found failures, and replace the selection with gold-labelled failures.
Critical strengths
A clean matched-budget design for a practical question. All conditions share the objective, the number of problems, exposures and optimizer steps, selection is fixed before training, and audit data never enters selection. This isolates which problems fill the budget, which is the paper’s question. Unusually candid reporting. The paper states which intervals include zero, that the pooled estimate was added after the fact, that Qwen shows no effect, that the gold-label result reverses under another recipe, and that training is not bitwise reproducible. This is why I do not recommend clear rejection.
Critical weaknesses
On the main benchmarks the method is supervised fine-tuning on gold labels for a selected subset, and the framing and baselines do not match that. LogiQA 2.0 and ReClor are multiple choice, the verifier is an exact match with the gold option, and GapFT’s training targets are answer-only strings such as “A” (Section 5, Table 4). The “verified response” for a recovered failure is therefore the gold label: search contributes the selection, not the training signal. For such tasks the probability the source gives the gold option should be available from one forward pass, and the S set is a noisy threshold on it: 36–44% of the source’s wrong items switch between S and U across independent resamples (Table 14). The natural baselines are a random labelled subset that includes U problems, and selection ranked by gold-option probability. Neither is run, and the classic work on ambiguous versus hard-to-learn examples (Swayamdipta et al., 2020) is not cited. The claim that search reachability is a useful signal holds in one recipe only. With a replayed core, gold-selected failures beat search-selected ones by +5.27 [2.93, 7.53] on ReClor (Appendix B.3, Table 10), which the main text summarises as “not worse”. In a two-seed ReClor diagnostic that adds a replayed core to the main recipe, the two are indistinguishable (−0.50 [−3.10, 2.00]). Where responses do carry reasoning (MATH), the contrast with uniform RFT has an interval that includes zero. Table 3 also has two undefined rows, “positive SFT” and “verified correction”, one of which matches GapFT (66.17 against 66.11).
The evidence for the headline and for the proposed account of the gain is weak. Only one of six GapFT–uniform intervals excludes zero, and the pooled +1.57 was added after the fact, as the paper says. Training on all verified problems is better than GapFT in all six rows. The budget excludes search (K = 8 on the whole pool), which every search-based condition pays, so the “three to almost fifty times” ratio overstates the saving in total compute. The main comparison uses a single budget, n = |S_8|, the one point at which GapFT uses every recovered failure. There is no budget sweep. The abstract and conclusion say the gain is bounded by the failures search exposes, and Section 6 says this share limits the gain. The identity bounds only the S term. Reconstructing the terms from Tables 1 and 11, correction on failures search missed (U) is comparable to or larger than correction on recovered ones (S) in four of six cells, for example about 6.5 against 2.9 points on Mistral–ReClor. The total gain does stay below the S share in every cell, but the claim needs a decomposition that accounts for the instability of the S/U split, or it should be dropped. The term differences have no intervals, while a byte-identical retrain changes correctness on 24% of LogiQA and 14% of ReClor audit items (Appendix A). Solved replay, which trains on no failures, already corrects 27–37% of S items in the three cells with at least 100 of them. The Section 5 controls use one model.
Questions for authors
Clarifications:
Are the training targets on LogiQA and ReClor the option label only, for all conditions including the greedy responses used for G? If so, how does GapFT differ from supervised fine-tuning on the gold labels of the same problems? What are “positive SFT” and “verified correction” in Table 3, and where is the code experiment of Figure 6 described? How does the claim that the gain is bounded by the S share fit with U corrections matching or exceeding S corrections on Mistral and Qwen? Why was n = |S_8| chosen as the only budget?
Additional evidence that would change my assessment (within the scope of the current experiments):
Two baselines at the same n: a random labelled subset of the whole pool (G, S and U), and wrong-greedy problems ranked by the source’s gold-option probability from one forward pass. Results at smaller and larger budgets, with replay filling the remainder. Intervals for the term differences in Table 11, and the same decomposition for a byte-identical rerun as a noise floor. Total cost including search for each condition, and the Section 5 controls on Mistral.
7 comments
Response to Reviewer fAgj (2/2)
R2 (to W2). Budget Choice, Mixtures, and Total Compute
Choice of , mixtures, and recipe sweeps. Table 1 sets because it uses every recovered failure once without replacement and zero replay. To verify that our findings hold across compositions and recipes, we swept clean -to- mixtures on LogiQA and ReClor, plus repetition counts and learning rates on ReClor with Llama-3.1-8B-Instruct over three seeds.
| S-to-G Ratio | LogiQA 2.0 (%) | ReClor (%) |
|---|---|---|
| 25:75 | 59.67 | 67.07 |
| 50:50 | 62.01 | 69.73 |
| 75:25 | 61.11 | 68.27 |
| 100:0, GapFT | 62.33 | 69.93 |
| ReClor Recipe Sweep | Standard RFT (%) | GapFT (%) |
|---|---|---|
| R = 1 / 2 / 4 / 8 at LR 5e-5 | 61.07 / 62.27 / 63.60 / 62.00 | 66.60 / 74.93 / 70.53 / 70.27 |
| LR 2e-5 / 5e-5 / 1e-4 at R = 8 | 60.13 / 62.00 / 64.87 | 70.80 / 70.27 / 69.27 |
Accuracy peaks at 100:0 on both benchmarks while 50:50 captures most of the gain, and GapFT beats Standard RFT across every repetition count and learning rate. At full-pool scale All, Table 11 shows that the remaining gap comes mainly from lower via extra replay rather than higher .
Total compute including search. Section 3.3 reports search depth separately from training budget because candidate search cost is identical across Uniform RFT, GapFT, and All-verified at , and adds only one greedy pass or 12.5% extra generations over -only Standard RFT. Stopping search at cuts search decodes from 9 to 2 per candidate, saving 78% of search compute, and filling remaining examples with in Table 2A already beats Uniform RFT on LogiQA and Standard RFT on both tasks. We will state both training-only and combined search-plus-training ratios in Section 4.2.
R3 (to W2). Decomposition Scope and Resampling Stability
Within-support bound and generalization. As written in Section 3.4 at line 229, the algebraic bound applies specifically to correction within observed support . Sizable gains appear because and are finite-sample observations rather than hard capability cutoffs as noted in Section 3.2, so updates on generalize to failures missed in eight draws. When approaches zero as on Qwen logic, is also tiny and overall headroom disappears. We will align the summary wording in the Abstract, Section 6, and the Conclusion with Section 3.4's within-support statement.
Resampling stability and retraining variance. Table 14 Panel B already shows that while individual items move between and across four independent resamples, combined correction, regression, and net Pass@1 gain have an exact 0.00 pp range across partitions. Appendix A also shows that retraining with identical data and settings changes mean Pass@1 by only 0.07 pp on LogiQA and 1.4 pp on ReClor, well below GapFT's multi-point gains over Standard RFT and generative baselines.
R4 (to W1). Table 3 and Figure 6 Clarifications
Table 3 pairing-loss ablation. "Positive SFT" in Table 3 is GapFT itself, named differently in the paired-loss ablation where "positive" denotes training only on verified correct responses to problems, as verifiable in our submitted anonymous code repository. Its paired counterpart, "verified correction", uses the same problems and correct targets while adding a pairing loss against each failed greedy response, lowering accuracy from 66.17% to 64.83%. We will unify the naming and define both rows in Section 6.1.
Figure 6 code experiment. Full settings and the contrast formula for Figure 6 appear in Appendix A Table 7 and Equation 10. Under a matched 64-update budget on LiveCodeBench with Qwen2.5-Coder-7B and DeepSeek-Coder-6.7B, selected items gain +38.36 pp over unselected items, broader coverage raises pass rates on newly covered coding failures by +15.09 pp in Figure 6A, and held-out audit differences stay positive on both models in Figure 6B. We will add a clear text paragraph linking Section 6.2 to Table 7 and Figure 6. Please let us know if any question remains during the discussion.
References
- Z. Chen et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP, 2021.
- B. Settles. Active Learning Literature Survey. CS Technical Report 1648, UW-Madison, 2009.
- S. Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. EMNLP, 2020.
- Z. Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.
Follow-up on Rebuttal - Paper 21974
Dear Reviewer fAgj,
Thank you again for your insightful feedback on our manuscript. With the discussion phase nearing its end, we are writing to confirm whether our responses have resolved all of your concerns. Should you have any additional questions, we are happy to provide further clarifications. If you are satisfied with our replies, we would appreciate your consideration to adjust your scores.
Best regards,
The Authors
Response to Reviewer fAgj (1/2)
Thank you for your careful review and for recognizing our
matched-budget experimental design, ourstrict separation of selection and audit data, and ourcandid empirical reporting. We also deeply appreciate the time, effort, and careful thought you devoted to reviewing our manuscript.Overview:
We hope these clarifications and new experiments resolve your concerns and earn your support for a higher score.
R1 (to W1). Targets, Single-Pass Baselines, and Open-Ended Tasks
Controlled option-label targets on MC tasks. On LogiQA 2.0 and ReClor, all conditions including greedy G responses use the option-letter target by design. Keeping target strings and token counts identical across G, S, and gold-selected U removes response-length and style differences, so any accuracy change comes strictly from which problems enter the training set. Even when targets match gold labels, which failures are selected without G replay matters sharply. Unscreened gold failures from S∪U drop Pass@1 by 9.49 pp on LogiQA and 6.87 pp on ReClor relative to GapFT in Table 2C because unreachable items trigger heavy G regressions. We will state this design rationale upfront in Section 4.1 and cite Swayamdipta et al. [2020] to connect GK, SK, and UK to easy, ambiguous, and hard regions estimated before training.
Added single-pass probability baselines and Standard RFT. To compare K-sample reachability against single-forward-pass option probabilities and practical K-sample-only Standard RFT (Yuan et al., 2023) without greedy deduplication, we evaluated Low Confidence using lowest maximum option probability and High Entropy using highest option entropy [Settles, 2009] from one forward pass on Llama-3.1-8B-Instruct under the same sample count and cross-entropy recipe.
GapFT outperforms both single-pass probability baselines, Standard RFT, and unscreened gold failures on both benchmarks. In Appendix B.3 and Table 10, the shift when adding a shared G core reflects a clear regime boundary. Reachability filtering prevents G regression when training holds no G replay, whereas replaying the full G core at triple the sample count already anchors G and makes gold failures and GapFT indistinguishable at −0.50 pp in our matched main-protocol diagnostic. We will report Table 10's exact +5.27 pp ReClor figure from the older protocol alongside this matched diagnostic directly in Section 5.
New open-ended tasks carrying full programs and answers. To test domains where single-token gold probabilities do not exist and search provides complete generated solutions, we added FinQA [Z. Chen et al., 2021] program synthesis on Qwen2.5-Coder-7B-Instruct and HotpotQA [Yang et al., 2018] multi-hop QA on Qwen2.5-7B-Instruct across two epochs with matched tokens and update steps over three seeds, comparing against DART-style (Tong et al., 2024) and DAPO-style (Yu et al., 2025) controls.
GapFT beats Standard RFT in every seed across both tasks and epochs and outperforms both difficulty-aware controls, showing clear gains when search supplies multi-step outputs.