acceptodds
Under review as a conference paper at ICLR 2027

Splicing Hints into Reasoning Traces: Learning from Demonstrator Failures

Abstract

Reinforcement learning with verifiable rewards improves mathematical reasoning when a final answer can be checked, but a correct answer does not certify the derivation behind it, and proof problems offer no answer to check. Replacing the check with a reasoning model as an in-loop verifier faces a fundamental obstacle: the policy must already produce occasional correct attempts for any reward to exist, which it may never do at olympiad level. We instead move verification and repair out of the training loop and into offline data generation with SPLICE, a workflow in which a second model, given only a reference solution, edits a reasoning model's traces as they are generated: it verifies progress, truncates a failed trace at its last verified step, and splices in an adaptively strengthened hint until progress resumes. The resulting dataset includes repaired examples whose sampled unaided trajectory failed, with failed continuations removed and progress repeatedly checked against the reference solution. A model finetuned on SPLICE traces outperforms the same model distilled from the reasoning model's unedited traces on AIME and, under a grading protocol validated against human grades, on olympiad proofs. The trained model's own traces show a corresponding behavioral signature: at difficult points it announces a branch in the style of the spliced hints and takes riskier steps, with predictive entropy rising there and just after, a pattern absent from the model trained on unedited traces.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning traces, distillation, mathematical reasoning, olympiad proofs, synthetic data, supervised finetuning, verification, hint insertion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bVdWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPLICE, a way to build SFT data for mathematical reasoning from a demonstrator’s failed attempts. While Qwen3-30B-A3B-Thinking generates a trace, a tutor model (Gemini 2.5 Pro) that has a reference solution checks whether the trace is making progress. When it stalls, the trace is cut back to the last constructive paragraph, a hint rewritten in the demonstrator’s style is appended, and generation resumes; hint strength goes up after stalls and down after progress. Qwen3-4B-Instruct is then finetuned on the accepted traces. Compared with distillation on untutored traces, the filtered SPLICE data raises AIME 2024–2026 accuracy from 60.6% to 64.4% and the mean proof score on 45 olympiad problems from 0.97 to 1.15 (out of 7). The authors also find that the trained student writes hint-like passages at test time, followed by text with higher predictive entropy.

Critical strengths

The idea is simple and sensible. Checking a trace against a reference solution is much easier than solving the problem, and SPLICE uses this to get training signal from trajectories where rejection sampling gives nothing. The design details (truncating instead of revising, adapting hint strength, suppressing the end-of-thinking and pivot tokens so that the demonstrator actually uses the hint) are well thought out, and Section 2.3 is useful in practice on its own. The experiments are reported carefully: the same student and recipe across datasets, paired bootstrap intervals, several proof metrics rather than one, and the authors say plainly that the self-hint analysis does not show causality. The comparison with Untutored-Correct (+0.17 mean proof score, CI [+0.09, +0.27]) is the most direct evidence that the hinted traces add something.

Critical weaknesses
  1. Limited novelty. The main ingredients already exist. Using privileged information from a reference solution to guide a model on problems it cannot solve is the idea behind the hint-based methods the paper itself discusses (POPE, ADHint, HiLL), and generating a rationale with access to the answer for failed problems goes back to STaR’s rationalization step. Verifying a trace against a reference solution is also standard. SPLICE moves the hints from RL rollouts into offline SFT data and inserts them in the middle of the trace, which is a reasonable engineering choice, but the paper does not show that this particular way of inserting hints is what makes the difference.
  2. The baselines do not make a convincing case. Every comparison is against variants of the paper’s own pipeline (Untutored, Untutored-Correct, SPLICE-NF/HC). There is no external method: no STaR-style rationalization, no hint-in-prompt generation, no hint-based method from the related work, and no existing proof-training recipe. Even the internal rejection-sampling baseline uses only one sample per problem, although the paper says a failure is “a trajectory that stalled, not a problem the demonstrator could never solve”, so Untutored-Correct (768 traces) vs. SPLICE (2,009) mixes repair with dataset size and coverage. The paper also gives no reference points: the student before finetuning, the demonstrator itself, or a same-size reasoning model (the Qwen3-4B-Thinking-2507 model card reports 81.3% on AIME 2025, against 64.6% for the best student here). On top of this, the evidence is thin: 45 proof problems with mean scores around 1/7, “doubles the attempts scoring ≥ 6” coming from 5 to 8 problems, and apparently a single finetuning run. Given the limited novelty and the lack of external baselines, I do not think the paper is ready for acceptance in its current form.
Questions for authors
  1. Is Untutored-Correct a subset of the Untutored traces (Section 4.1) or the SPLICE traces accepted without a hint (Section 2.2)? Figure 3 suggests these differ (median NLL 0.25 vs. 0.29). If they do, where does the difference come from?
  2. How does ASSESS handle a correct approach that differs from the reference solution? And how reliable is ACCEPT on hinted proofs, given that 10 accepted traces answered a different problem?
  3. What does one accepted SPLICE trace cost in tutor calls, compared with sampling k untutored traces per problem?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bVdW on SPLICE, Baselines, and Evaluation

We thank the reviewer for the detailed feedback and for highlighting the practical value of Section 2.3. We address the concerns below and report a new eight-sample rejection-sampling baseline.

1. What does SPLICE contribute beyond prior work on hinting and rationalization?

Reference-guided hinting and answer-conditioned rationalization have precedents in POPE, ADHint, HiLL, and STaR. Our contribution is a workflow that turns failed attempts at hard proofs into SFT traces. SPLICE preserves the demonstrator’s progress up to the last useful step, removes the failed continuation, and resumes generation with a targeted hint written in the demonstrator’s style. This lets us use a stronger closed-source tutor to improve traces from an open-weight demonstrator even when the tutor’s own reasoning traces are unavailable.

With adaptive hinting and truncation both disabled, the workflow solved only 102 additional problems beyond those the demonstrator solved unaided. This run measures the yield of the simplified workflow. The effect of mid-trace hint insertion on its own remains open.

2. What do the baselines test, and what happens with eight unaided samples per problem?

We included two direct-distillation baselines. Untutored trains the student on all original demonstrator traces. Untutored-Correct trains it on the verified correct subset. SPLICE outperforms both in the reported experiments. The SPLICE-NF and SPLICE-HC comparisons also show how filtering changes downstream performance.

To address the single-sample concern, we generated eight unaided attempts per problem and applied verification. This yielded correct traces for 944 distinct problems, most of them from the older AIME portion of the corpus. We finetuned the same student on the resulting data.

Training dataAIME 2024AIME 2025AIME 2026
Untutored-Correct, 8 samples per problem65.057.957.5
SPLICE67.964.660.8

Recovering traces for more hard problems is an intended outcome of SPLICE. The comparison between 768 Untutored-Correct traces and 2,009 SPLICE traces measures the downstream effect of the resulting datasets, including their different sizes and coverage. A size- and coverage-matched comparison would estimate the effect of repair at fixed coverage.

We prioritized direct distillation from a reasoning model because prior work on competition mathematics found it more effective than finetuning on the prompted or edited CoTs tested there [1]. Our experiments ask whether repair improves on direct distillation from the same demonstrator. STaR-style rationalization, hint-in-prompt generation, and existing proof-training recipes would address additional comparisons.

3. How should readers interpret the reference points and proof results?

The starting student is Qwen3-4B-Instruct-2507. Its model card reports 47.4% on AIME 2025, about 10 percentage points below our weakest finetuned baseline. This is a model-card reference point, and the evaluation protocols may differ. The demonstrator and Qwen3-4B-Thinking-2507 provide context for absolute performance. Our main comparison holds the student and finetuning recipe fixed and varies the training data.

The proof evaluation covers 45 problems, and absolute mean scores remain low. We will clarify that 20 to 41 refers to individual attempts scoring at least 6 for Untutored versus SPLICE. The 5 to 8 comparison counts distinct problems with at least one such attempt for Untutored versus SPLICE-HC. The paired bootstrap intervals quantify uncertainty across evaluation problems. Variation across finetuning runs remains open, and our evaluation budget limited the size of the proof benchmark.

4. Is Untutored-Correct the same as the SPLICE traces accepted without a hint?

Yes. Both labels refer to the same 768 traces. In Figure 3(c), 0.25 nats per token is the median for all 2,500 Untutored traces, and 0.29 is the median for the 768-trace subset accepted without hints. We will make this mapping explicit in the caption.

5. How does ASSESS handle alternative proofs, and what did the audit find?

ASSESS uses the reference solution as context and judges the generated proof on its own merits. A valid proof may use a different approach. This design is informed by RefGrader [2], where reference solutions helped proof grading. After a failed attempt, SPLICE uses the reference to generate a targeted hint, then checks the resulting proof again on its own merits.

The 10 flagged traces came from ambiguous input statements. The demonstrator solved a different interpretation of the statement from the intended one. We identified these cases through automated flags followed by manual review by one of the authors, then removed them from the filtered training set.

6. How many tutor calls does SPLICE use?

We cap SPLICE at 10 iterations per trace, with one Gemini 2.5 Pro call for verification and one additional Pro call if a hint is needed. This caps usage at 20 Pro calls per trace. The remaining stages each use a single Gemini 2.5 Flash or Flash-Lite call. Verified rejection sampling with kk attempts uses kk demonstrator generations and one verification call per attempt checked. The SPLICE cap is per trace entering the workflow. An observed average per accepted trace would also include calls spent on rejected traces.

References

[1] Du et al. The Challenge of Teaching Reasoning to LLMs Without RL or Distillation. 2025. Paper

[2] Mahdavi et al. RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows. 2025. Paper

Reviewer_qzkCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPLICT, a workflow that uses a stronger model, given the reference, to provide hints to guide a student model to generate correct traces.

Critical strengths

The method is interesting, and the paper is generally easy to follow.

Critical weaknesses

I am not very familiar with the hint-based method, so my concerns lie in the motivation. If you have access to a stronger model and the reference, why not just use the strong model to generate reasoning traces given the reference and use that as the training data? One advantage of SPLICE, I think, is that leaving generation to the student model would have reasoning traces that are closer to its original distribution, which might enable better training stability. In this case, I think a comparison with SPLICE with a stronger model given reference-generated traces would be beneficial.

You also mention that a student model trained this way might learn to explore when it encounters difficulty. This might be another advantage, but you need stronger experiments to support this claim. I suggest comparing the self-hint frequency, post-hint predictive entropy, and the accuracy against a student trained on reasoning traces directly generated by a stronger teacher conditioned on reference solutions.

Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer qzkC on Direct Distillation and Exploration

We thank the reviewer for raising the comparison with direct distillation and for suggesting additional analyses of the student’s reasoning behavior.

1. Why use the tutor for hints instead of asking it to generate full rationales from the reference solutions?

SPLICE uses three models. The demonstrator generates the reasoning traces, the tutor uses the reference solution to assess progress and provide hints, and the student is finetuned on the resulting traces. We designed SPLICE for a practical setting where we can collect reasoning traces from an open-weight demonstrator and query a stronger closed-source tutor whose API does not expose its own reasoning traces. The tutor can still use the reference solution to identify where the demonstrator stalled and suggest a next step. SPLICE uses that feedback to improve the traces we can collect.

Finetuning on prompted CoTs is an established approach [1]. STaR [2] also explored prompting a model to generate a rationale given the correct answer after a failed attempt. For competition mathematics, Du et al. [3, Table 1] found that finetuning Qwen2.5-32B on 50 traces from QwQ-32B-Preview achieved 19.29% pass@1 on Comp-Math-24-25. The best result using thousands of prompted or edited CoTs from Qwen2.5-32B-Instruct was 12.36%. This evidence motivates our choice of direct distillation from a reasoning model as the primary comparison.

To test whether repairing the demonstrator’s traces improves on direct distillation, we included two baselines that finetune the student on traces from the same reasoning demonstrator. Untutored uses all its unaided traces, and Untutored-Correct uses only the unaided traces with correct solutions. SPLICE outperforms both. We chose these baselines because prior work on competition mathematics [3, Table 1] found that distilling reasoning-model traces was more effective than finetuning on prompted or edited CoTs. That result motivates our choice to benchmark against direct distillation from a reasoning model. The reviewer’s specific variant, which prompts our tutor with the full reference solution, remains untested in our setting.

2. Would a student trained on tutor-written rationales show similar self-hinting behavior and accuracy?

The SPLICE-trained student produces hint-like branches at difficult points in its reasoning, and predictive entropy rises around those branches. We interpret this pattern as consistent with exploration. Whether the branches themselves cause more exploration or improve accuracy remains open. The suggested comparison of self-hint frequency, post-hint predictive entropy, and accuracy would show whether a student trained on tutor-written rationales exhibits a similar pattern and how its accuracy compares.

We appreciate the reviewer’s questions about how the tutor is used and what the self-hint analysis establishes. We will clarify the roles of the three models and the motivation for our direct-distillation baselines in the revision. We will also make clear that the observed self-hinting and entropy patterns are consistent with exploration, while their effect on accuracy remains open.

References

[1] Ho et al. Large Language Models Are Reasoning Teachers. ACL, 2023. Paper

[2] Zelikman et al. STaR: Bootstrapping Reasoning With Reasoning. NeurIPS, 2022. Paper

[3] Du et al. The Challenge of Teaching Reasoning to LLMs Without RL or Distillation. 2025. Paper

Reviewer_3HhpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve traditional reasoning distillation by refining the trajectories generated by a demonstrator (teacher) before training a student model. Specifically, when the demonstrator fails to solve a problem, a separate tutor provides progressively more detailed hints to repair the failed reasoning trajectory. The method preserves successful reasoning prefixes, removes unsuccessful continuations, and allows the demonstrator to resume generation with the inserted hints. The resulting trajectories are then used for supervised fine-tuning of the student model.

The proposed method outperforms standard distillation and rejection-sampling baselines on mathematical reasoning benchmarks.

Critical strengths

S1 Interesting perspective on improving demonstrator trajectories

Several related studies focus on selecting or adapting teacher trajectories to improve their learnability for students [1,2], while others investigate how external teacher guidance can help student policies overcome difficult problems during RL [3,4].

In contrast, SPLICE takes a distinctive perspective by explicitly repairing failed demonstrator trajectories before distillation. I find this problem formulation interesting and well motivated in the introduction.

S2 Analysis of learned reasoning behavior

Beyond downstream performance improvements, the paper provides an insightful analysis of the student's learned reasoning behavior. In particular, SPLICE-trained students exhibit self-hint-like branching behavior, accompanied by increased predictive entropy, even without external guidance at inference time. This analysis provides an interesting perspective on how guided reasoning trajectories may influence the student's subsequent reasoning behavior.

[1]Yu, Tianrun, et al. "LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation." arXiv preprint arXiv:2605.30651 (2026).

[2]He, Chongyang, et al. "Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning." arXiv preprint arXiv:2606.04466 (2026).

[3]Qu, Yuxiao, et al. "Pope: Learning to reason on hard problems via privileged on-policy exploration." arXiv preprint arXiv:2601.18779 (2026).

[4]Yan, Jianhao, et al. "Learning to reason under off-policy guidance." Advances in Neural Information Processing Systems 38 (2026): 117157-117186.

Critical weaknesses

W1 Limited novelty of the hierarchical hinting mechanism The hierarchical hinting mechanism closely resembles prior scaffolding-based approaches such as Scaf-GRPO [5], which also progressively increase hint specificity to overcome reasoning failures.

Although SPLICE applies this idea to offline demonstrator trajectory repair rather than online RL, the paper could better clarify its methodological contributions beyond adapting hierarchical hinting to the distillation setting.

W2 Lack of ablation studies on hierarchical hinting

The paper introduces a five-level hierarchical hinting mechanism as a central component of SPLICE. However, it does not provide ablations comparing adaptive hierarchical hinting against fixed-level hints or simpler intervention strategies.

Consequently, it remains unclear whether the performance improvements arise from the adaptive hinting mechanism itself or simply from providing sufficiently detailed corrective information.

Although the SPLICE-HC experiment investigates the effects of excessive hinting, it does not establish the necessity or effectiveness of the hierarchical hinting mechanism.

[5]Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026

Questions for authors

My main questions are closely related to the weaknesses mentioned above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Hhp on Scaf-GRPO and Adaptive Hinting

We thank the reviewer for their insightful feedback.

W1. Relationship to Scaf-GRPO

Scaf-GRPO [1] adds progressively more explicit hints to the prompt for fresh rollouts. SPLICE writes a hint for the specific point where an existing rollout stalled, truncates the trace at the last useful step, and resumes generation with the hint in the CoT. Both methods use progressively stronger hints when needed. In SPLICE, we assess progress within the existing rollout so we can preserve the reasoning that is already useful.

Our hinting schema is designed for proof generation. Final-answer rewards, as used in outcome-supervised GRPO [2], can check an answer, while verifying a proof requires assessing the arguments that support it. We move these checks into offline trace generation to avoid the cost of using a reasoning model as a judge in the RL loop. We will cite Scaf-GRPO in the revised paper and clarify how the two methods use hierarchical hinting.

W2. Ablation of adaptive hinting

We agree that the SPLICE-HC experiment leaves open whether adaptive hinting outperforms fixed-level hinting. In our generation logs, with both adaptive hinting and truncation at the last useful step disabled, the workflow solved 102 additional problems in our corpus beyond those the demonstrator solved without hints. This run changed both components together. To isolate the effect of adaptive hinting, we would compare it with fixed-level hinting under the same tutor-call budget and keep truncation, hint placement, and acceptance criteria fixed.

Adaptive hinting. We adapt both how much information the hint reveals and which information it provides. A fixed-level hint can give the demonstrator too little information to make progress, even after repeated attempts. Revealing the whole solution can yield a trace that largely rewrites the reference solution. Muennighoff et al. [3] used 1,000 curated reasoning traces for supervised finetuning in s1. In SPLICE, our goal is to preserve the useful reasoning already present in the demonstrator’s trace and edit it only where needed.

Truncation and hint placement. We insert the hint into the CoT so the SFT corpus contains traces that show how the demonstrator proceeds after a hint. We also rewrite each hint in the demonstrator’s style. In our experiments, when hints diverged too much from that style, the finetuned model sometimes looped and repeated the same paragraph at test time.

Truncating too much removes useful progress, which slows data generation and can prevent the demonstrator from finishing the proof. When we kept the entire trace, we observed that it could quickly reach the length limit and that the demonstrator could ignore the hint and continue along its flawed solution. To identify the last useful step, we break the reference solution into main steps and substeps and locate the latest substep the demonstrator has successfully derived.

We appreciate the reviewer’s suggestion to clarify the relationship to Scaf-GRPO and examine the contribution of adaptive hinting. References

[1] Zhang et al. Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning. ICLR, 2026. arXiv:2510.19807

[2] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300

[3] Muennighoff et al. s1: Simple test-time scaling. 2025. arXiv:2501.19393

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.