- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SPLICE, a way to build SFT data for mathematical reasoning from a demonstrator’s failed attempts. While Qwen3-30B-A3B-Thinking generates a trace, a tutor model (Gemini 2.5 Pro) that has a reference solution checks whether the trace is making progress. When it stalls, the trace is cut back to the last constructive paragraph, a hint rewritten in the demonstrator’s style is appended, and generation resumes; hint strength goes up after stalls and down after progress. Qwen3-4B-Instruct is then finetuned on the accepted traces. Compared with distillation on untutored traces, the filtered SPLICE data raises AIME 2024–2026 accuracy from 60.6% to 64.4% and the mean proof score on 45 olympiad problems from 0.97 to 1.15 (out of 7). The authors also find that the trained student writes hint-like passages at test time, followed by text with higher predictive entropy.
Critical strengths
The idea is simple and sensible. Checking a trace against a reference solution is much easier than solving the problem, and SPLICE uses this to get training signal from trajectories where rejection sampling gives nothing. The design details (truncating instead of revising, adapting hint strength, suppressing the end-of-thinking and pivot tokens so that the demonstrator actually uses the hint) are well thought out, and Section 2.3 is useful in practice on its own. The experiments are reported carefully: the same student and recipe across datasets, paired bootstrap intervals, several proof metrics rather than one, and the authors say plainly that the self-hint analysis does not show causality. The comparison with Untutored-Correct (+0.17 mean proof score, CI [+0.09, +0.27]) is the most direct evidence that the hinted traces add something.
Critical weaknesses
- Limited novelty. The main ingredients already exist. Using privileged information from a reference solution to guide a model on problems it cannot solve is the idea behind the hint-based methods the paper itself discusses (POPE, ADHint, HiLL), and generating a rationale with access to the answer for failed problems goes back to STaR’s rationalization step. Verifying a trace against a reference solution is also standard. SPLICE moves the hints from RL rollouts into offline SFT data and inserts them in the middle of the trace, which is a reasonable engineering choice, but the paper does not show that this particular way of inserting hints is what makes the difference.
- The baselines do not make a convincing case. Every comparison is against variants of the paper’s own pipeline (Untutored, Untutored-Correct, SPLICE-NF/HC). There is no external method: no STaR-style rationalization, no hint-in-prompt generation, no hint-based method from the related work, and no existing proof-training recipe. Even the internal rejection-sampling baseline uses only one sample per problem, although the paper says a failure is “a trajectory that stalled, not a problem the demonstrator could never solve”, so Untutored-Correct (768 traces) vs. SPLICE (2,009) mixes repair with dataset size and coverage. The paper also gives no reference points: the student before finetuning, the demonstrator itself, or a same-size reasoning model (the Qwen3-4B-Thinking-2507 model card reports 81.3% on AIME 2025, against 64.6% for the best student here). On top of this, the evidence is thin: 45 proof problems with mean scores around 1/7, “doubles the attempts scoring ≥ 6” coming from 5 to 8 problems, and apparently a single finetuning run. Given the limited novelty and the lack of external baselines, I do not think the paper is ready for acceptance in its current form.
Questions for authors
- Is Untutored-Correct a subset of the Untutored traces (Section 4.1) or the SPLICE traces accepted without a hint (Section 2.2)? Figure 3 suggests these differ (median NLL 0.25 vs. 0.29). If they do, where does the difference come from?
- How does ASSESS handle a correct approach that differs from the reference solution? And how reliable is ACCEPT on hinted proofs, given that 10 accepted traces answered a different problem?
- What does one accepted SPLICE trace cost in tutor calls, compared with sampling k untutored traces per problem?
2 comments
4. Is Untutored-Correct the same as the SPLICE traces accepted without a hint?
Yes. Both labels refer to the same 768 traces. In Figure 3(c), 0.25 nats per token is the median for all 2,500 Untutored traces, and 0.29 is the median for the 768-trace subset accepted without hints. We will make this mapping explicit in the caption.
5. How does ASSESS handle alternative proofs, and what did the audit find?
ASSESS uses the reference solution as context and judges the generated proof on its own merits. A valid proof may use a different approach. This design is informed by RefGrader [2], where reference solutions helped proof grading. After a failed attempt, SPLICE uses the reference to generate a targeted hint, then checks the resulting proof again on its own merits.
The 10 flagged traces came from ambiguous input statements. The demonstrator solved a different interpretation of the statement from the intended one. We identified these cases through automated flags followed by manual review by one of the authors, then removed them from the filtered training set.
6. How many tutor calls does SPLICE use?
We cap SPLICE at 10 iterations per trace, with one Gemini 2.5 Pro call for verification and one additional Pro call if a hint is needed. This caps usage at 20 Pro calls per trace. The remaining stages each use a single Gemini 2.5 Flash or Flash-Lite call. Verified rejection sampling with attempts uses demonstrator generations and one verification call per attempt checked. The SPLICE cap is per trace entering the workflow. An observed average per accepted trace would also include calls spent on rejected traces.
References
[1] Du et al. The Challenge of Teaching Reasoning to LLMs Without RL or Distillation. 2025. Paper
[2] Mahdavi et al. RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows. 2025. Paper
Response to Reviewer bVdW on SPLICE, Baselines, and Evaluation
We thank the reviewer for the detailed feedback and for highlighting the practical value of Section 2.3. We address the concerns below and report a new eight-sample rejection-sampling baseline.
1. What does SPLICE contribute beyond prior work on hinting and rationalization?
Reference-guided hinting and answer-conditioned rationalization have precedents in POPE, ADHint, HiLL, and STaR. Our contribution is a workflow that turns failed attempts at hard proofs into SFT traces. SPLICE preserves the demonstrator’s progress up to the last useful step, removes the failed continuation, and resumes generation with a targeted hint written in the demonstrator’s style. This lets us use a stronger closed-source tutor to improve traces from an open-weight demonstrator even when the tutor’s own reasoning traces are unavailable.
With adaptive hinting and truncation both disabled, the workflow solved only 102 additional problems beyond those the demonstrator solved unaided. This run measures the yield of the simplified workflow. The effect of mid-trace hint insertion on its own remains open.
2. What do the baselines test, and what happens with eight unaided samples per problem?
We included two direct-distillation baselines. Untutored trains the student on all original demonstrator traces. Untutored-Correct trains it on the verified correct subset. SPLICE outperforms both in the reported experiments. The SPLICE-NF and SPLICE-HC comparisons also show how filtering changes downstream performance.
To address the single-sample concern, we generated eight unaided attempts per problem and applied verification. This yielded correct traces for 944 distinct problems, most of them from the older AIME portion of the corpus. We finetuned the same student on the resulting data.
Recovering traces for more hard problems is an intended outcome of SPLICE. The comparison between 768 Untutored-Correct traces and 2,009 SPLICE traces measures the downstream effect of the resulting datasets, including their different sizes and coverage. A size- and coverage-matched comparison would estimate the effect of repair at fixed coverage.
We prioritized direct distillation from a reasoning model because prior work on competition mathematics found it more effective than finetuning on the prompted or edited CoTs tested there [1]. Our experiments ask whether repair improves on direct distillation from the same demonstrator. STaR-style rationalization, hint-in-prompt generation, and existing proof-training recipes would address additional comparisons.
3. How should readers interpret the reference points and proof results?
The starting student is Qwen3-4B-Instruct-2507. Its model card reports 47.4% on AIME 2025, about 10 percentage points below our weakest finetuned baseline. This is a model-card reference point, and the evaluation protocols may differ. The demonstrator and Qwen3-4B-Thinking-2507 provide context for absolute performance. Our main comparison holds the student and finetuning recipe fixed and varies the training data.
The proof evaluation covers 45 problems, and absolute mean scores remain low. We will clarify that 20 to 41 refers to individual attempts scoring at least 6 for Untutored versus SPLICE. The 5 to 8 comparison counts distinct problems with at least one such attempt for Untutored versus SPLICE-HC. The paired bootstrap intervals quantify uncertainty across evaluation problems. Variation across finetuning runs remains open, and our evaluation budget limited the size of the proof benchmark.