acceptodds
Under review as a conference paper at ICLR 2027

Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction

Abstract

Large language models struggle to reliably identify and correct errors in their own reasoning. Unlike prior training-based approaches that fine-tune an LLM to generate revisions or feedback, we learn where to intervene in failed rollouts and determine how to redirect them. To choose where to intervene, we train a 130K-parameter MLP head on a frozen encoder using only final-answer correctness. For redirection, we find that under a fixed generation budget, the tested inline injections fail to reliably redirect reasoning, whereas presenting the same instruction in a new user turn changes the model's course. The resulting correction framework, UPTURN (User-Prompt TURN injection at a learned position), requires no LLM fine-tuning: it retains a selected prefix, injects an instruction as a new user turn, and regenerates the suffix. Across four reasoners and six math benchmarks, UPTURN improves average accuracy in every setting over a 32-rollout baseline. It outperforms self-correction and reflection-token injection baselines under matched correction budgets, transfers across reasoners and to GPQA-Diamond without retraining, and complements GRPO and self-correction methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM reasoning, self-correction, multi-turn injection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HWVgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UPTURN corrects reasoning trajectories by selecting a paragraph boundary, retaining the prefix, inserting a fixed instruction as a new user turn, and regenerating the suffix. It trains lightweight position-selection and gating heads while keeping the reasoning model and encoder frozen. Experiments cover multiple mathematical benchmarks, cross-model transfer, and generalization to GPQA-Diamond.

Critical strengths
  1. Lightweight and practical design. UPTURN learns intervention positions from final-answer correctness without step-level annotations or full-model fine-tuning. This makes it a potentially useful addition to existing reasoning models.

  2. Substantial tight-budget improvements with broad experimental support. Results across multiple models and benchmarks, together with injection, position-selection, gating, and seed ablations, support the approach. End-to-end runtime measurements further strengthen its practical relevance.

Critical weaknesses
  1. The advantage diminishes substantially as the generation budget increases. At 24K, Avg gains over Base@32 are +0.6 percentage points for Qwen3-4B, −1.6 for Qwen3-8B, and +7.0 for DeepSeek-R1-Qwen3-8B, compared with +14.1, +16.9, and +30.3 at 8K. Both methods generally improve in absolute accuracy, but independent sampling catches up. It therefore remains unclear how much of UPTURN’s benefit comes from recovering length-truncated trajectories versus correcting completed but incorrect solutions. Reusing the 8K-trained position policy also complicates the interpretation of larger-budget results.

  2. Comparative evaluation and reporting need strengthening. Closely related rollback-based approaches such as TAAR, Thought-ICS, and StepCo warrant discussion and representative comparisons. The manuscript and supplementary main-table code also differ in gate threshold and position selection, limiting reproducibility. As a presentation improvement, Figure 1 should be redrawn: its decorative design obscures the mechanism. A simpler side-by-side diagram should clearly identify the cut position, retained prefix, turn boundary, and regenerated suffix for inline versus user-turn injection.

TAAR https://aclanthology.org/2026.findings-acl.1930

Thought-ICS https://arxiv.org/abs/2602.02416

StepCo https://aclanthology.org/2025.acl-long.1048

Questions for authors
  1. Could you report length-truncation rates at each budget, together with correction success rates separately for truncated trajectories and completed but incorrect trajectories? These statistics would help explain the different outcomes at 24K.

  2. Could you extend evaluation to at least 32K, and preferably 48K or 64K where supported, reporting accuracy, uncertainty, actual generated tokens, and runtime? Comparing the reused 8K policy with a budget-specific policy at one larger budget would help isolate transfer effects.

  3. Please clarify whether the reported results use the manuscript’s 0.4 gate threshold and sampled positions or the supplementary script’s 1.0 threshold and argmax selection, and provide matching evaluation commands. A representative comparison with one of the closely related methods above would also strengthen the assessment.

2 comments

Thank you for your thoughtful comments and constructive suggestions. We address your concerns below.

 

R1. Budget scaling and correction success (W1, Q1)

We report equal-weight averages over six benchmarks and Qwen3-4B/8B and DeepSeek-R1-Qwen3-8B, using our learned gate (γ=0.4). T→C / selected T and W→C / selected W measure correction success among selected truncated and wrong responses, respectively. Undefined transition rates are excluded from averaging.

Generation budgetCorrect / all (%)Wrong / all (%)Truncated / all (%)T→C / selected T (%)W→C / selected W (%)
8K34.511.3664.1337.839.0
16K57.003.8639.1433.022.8
24K67.889.9122.2027.610.0

The declining gains cannot be explained solely by the increasing share of wrong responses: T→C success also decreases with the budget, and wrong responses remain less frequent than truncated responses through 24K. Moreover, the SCoRe experiment below shows that UPTURN can correct wrong answers. These findings suggest that the decline may reflect limitations in transferring the 8K-trained policy to longer rollouts; the comparison in R2 examines this possibility.

To evaluate wrong-answer correction with an 8K initial-generation budget, we examine the response pools from our complementary experiments, under the same averaging and gating settings:

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
Base34.311.4064.29
GRPO54.486.3839.14
Reflexion43.602.0254.38
SCoRe52.2532.2915.45

SCoRe produces more wrong than truncated responses. Applying the existing UPTURN policy with “Resolve this concisely.” (γ=0.4), without additional training, gives:

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
SCoRe52.2532.2915.45
SCoRe + UPTURN64.2925.4710.24

C, W, and T denote correct answers, wrong extracted answers, and responses with no extracted answer, respectively. W→C gain and T→C gain are the accuracy gains from correcting W and T responses; C→W/T loss is the accuracy lost when initially correct responses become wrong or yield no extracted answer. Net gain is the sum of the two gains minus this loss. The contributions below are percentage points across all responses, rather than the conditional success rates above:

W→C gainT→C gainC→W/T lossNet gain
+8.04 pp+5.14 pp−1.15 pp+12.03 pp

Wrong-answer correction contributes more to this gain than recovery of truncated responses, showing that UPTURN also turns wrong extracted answers into correct ones.

 

R2. 32K evaluation and budget-specific policy comparison (Q2)

The requested 48K/64K response budgets exceed Qwen3-4B’s configured 40,960-token context window, which includes both the prompt and response. We therefore extended evaluation to 32K on three reasoners across all six benchmarks. Accuracy denotes Avg., with equal weights across benchmarks and reasoners. We use γ=0.1, selected from a post-hoc threshold sweep.

BudgetMethodAccuracy95% bootstrap CI
32KBase@1673.02[68.33, 77.41]
32KUPTURN@1674.37[69.74, 78.75]

UPTURN improves accuracy by 1.35 pp over Base@16, the same 16 initial responses without correction (95% CI [0.71, 2.07]). To compute the 95% CIs, we use 10,000 bootstrap resamples within each benchmark, sharing the sampled questions across reasoners and methods. These CIs reflect variation in evaluation questions only; models, generated responses, and γ are held fixed.

Representative computation cost. We measured Qwen3-4B on 30 AIME24 questions at 32K using one H200 (TP=1). Base@32 samples 32 responses in one batch; Base@16 measures the generation of UPTURN’s 16 initial responses, and UPTURN@16 additionally applies correction using the 8K-trained policy. Results are averaged over two runs; runtime is reported as mean ± SD.

MethodTokens / questionRuntime (s)
Base@16238,5863,393.5 ± 76.5
Base@32466,7616,370.7 ± 134.9
UPTURN@16263,7184,152.0 ± 165.1

UPTURN uses 43.5% fewer generated tokens and 34.8% less runtime than Base@32. Token counts include newly generated initial responses and correction suffixes, excluding retained prefixes.

At 16K evaluation, we compare 8K-trained and 16K-trained policies with γ=0.4. Scores are equal-weight averages over six benchmarks and three reasoners.

Eval. budgetMethodTrain budgetStepAccuracy (%)
16KUPTURN@168K3066.6
16KUPTURN@1616K3067.1

The 16K-trained policy achieves higher mean accuracy than the 8K-trained policy (67.1% vs. 66.6%, a gain of 0.5 pp).

R3. Gate threshold and position selection (W2, Q3)

Gate threshold. To avoid repeating generation across analyses, we intentionally use γ=1.0 to precompute one correction for every rollout. We subsequently recomputed the metrics from the saved rollout dumps, which contain the original and corrected responses and their gate scores. For the reported main results, we apply γ=0.4 to the original rollout scores and replace only responses below this threshold.

Position selection. Thank you for pointing this out. The sampling description was our mistake: training samples G=8 cuts without replacement, while inference uses argmax. We have corrected Sec. 5.3 and Appendix C. Revisions are highlighted in blue.

 

R4. Comparison with related rollback methods (W2)

We have added TAAR, Thought-ICS, and StepCo to Related Work, with the additions highlighted in blue.

TAAR [1] learns trap locations and escape probabilities to control truncation and restart with a frozen reasoner. Our reduced reimplementation achieves 44.14% Avg / 57.61% Pass, compared with UPTURN's 56.2% / 73.4% (six-benchmark means; separate rollout pools). Keeping TAAR's cuts and English instruction fixed, moving the instruction into a new user turn improves performance to 48.19% / 67.46%. The full comparison is in KVSn R1.

Thought-ICS [2] structures generation into discrete thoughts, verifies the response, localizes the first erroneous thought, and iteratively backtracks to resample from the preceding prefix. Its explicit error-localization loop differs from UPTURN's policy trained on regeneration outcomes from existing responses.

StepCo [3] alternates verification by a process-supervised verifier with revision of incorrect reasoning steps. UPTURN learns where to cut from final-answer rewards and introduces a new user turn for suffix regeneration, without requiring step-level error labels for policy training.

Our contribution combines outcome-trained cut selection with user-turn injection, supported by the TAAR comparison and our inline/user-turn ablation.

 

R5. Figure 1 revision (W2)

We have updated Figure 1 in the revised paper with a simpler side-by-side comparison of inline and user-turn injection, explicitly labeling the cut position, retained prefix, turn boundary, and regenerated suffix. The caption has also been updated, with revised text highlighted in blue.

 

[1] Chen, Kang, et al. "Thinking Traps in Long Chain-of-Thought: A Measurable Study and Trap-Aware Adaptive Restart." arXiv preprint arXiv:2601.11940 (2026).

[2] Samanta, Ankur, et al. "Structure enables effective self-localization of errors in LLMs." arXiv preprint arXiv:2602.02416 (2026).

[3] Wu, Zhenyu, et al. "Enhancing mathematical reasoning in llms by stepwise correction." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

 

Please let us know if you have any further questions or would like additional clarification.

Reviewer_KVSnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UPTURN, a lightweight framework for correcting failed LLM reasoning trajectories without fine-tuning the reasoner. It uses a learned position policy to select where to truncate a reasoning rollout, then inserts a fixed instruction as a new user turn to regenerate the remaining reasoning.

Critical strengths
  1. The paper presents a simple correction framework that does not require fine-tuning the underlying reasoner. The improvements across multiple mathematical benchmarks are substantial, and the finding that new user turns are more effective than inline instructions is interesting.

  2. The authors provide useful ablations on intervention positions, injection formats, and gating. The transfer experiments also suggest that the learned position policy can generalize beyond the model used during training.

Critical weaknesses
  1. The novelty relative to recent reasoning intervention methods is not sufficiently discussed. Several closely related studies, including TAAR (ACL 2026), GPO (NeurIPS 2025), and Reasoning Can Be Restored by Correcting a Few Decision Tokens (ICML 2026), have already investigated identifying critical reasoning positions and redirecting unsuccessful trajectories. In particular, TAAR also learns where to truncate failed reasoning and restart generation without fine-tuning the reasoner. These works are missing from the related work discussion, and the lack of direct comparisons makes it difficult to assess UPTURN's additional contribution.

  2. The experiments do not fully establish the reliability or underlying mechanism of the proposed correction. In Appendix G, even an empty user turn or an irrelevant instruction such as "Turtle" substantially improves accuracy, while some explicit correction instructions perform worse. It is unclear why "Resolve this concisely" is preferred and whether its advantage comes from encouraging shorter reasoning under the fixed token budget. The instruction analysis is also limited to Qwen3-4B and Qwen3-8B. Furthermore, in Table 2, UPTURN's advantage over the best baseline decreases substantially when the generation budget is doubled, suggesting that part of the improvement may come from preserving earlier tokens rather than more effective correction. The evaluation would also benefit from reporting wrong-to-correct, correct-to-wrong, and wrong-to-wrong transitions, as well as distinguishing genuinely incorrect reasoning from truncated responses.

Questions for authors
  1. Why does "Resolve this concisely" outperform more explicit correction instructions, and does this finding generalize to other model families?

  2. How much of the improvement comes from correcting actual reasoning errors rather than completing truncated responses?

  3. Do the learned cut positions correspond to actual reasoning errors, or simply to positions where regeneration is more likely to succeed?

5 comments

Thank you for your thoughtful comments and constructive suggestions. We address your concerns below.

 

R1. Novelty and comparison with related methods (W1)

We have added TAAR, GPO, and Few Decision Tokens to Related Work, with the additions highlighted in blue. TAAR [1] learns trap locations and escape probabilities for inline restarts. UPTURN learns cuts from regeneration rewards without trap-location annotations and regenerates through a new user turn. Both keep the reasoner frozen.

As official TAAR code and checkpoints were unavailable at reproduction time, we independently implemented a reduced version using two source models due to computational constraints. We explored training/restart variants and report our strongest completed R+E result (Native cold19: English strong-restart instruction, no additional gate), trained on 152 prefix examples from 116 DAPO questions.

We report equal-weight means over six benchmarks using Qwen3-4B with thinking enabled and an 8K response budget. TAAR variants share identical initial rollouts, while the paper’s UPTURN and Base@32 results use a separate pool. This reduced reimplementation comparison does not match total compute across methods.

MethodMean Avg (%)Mean Pass (%)
Base@3242.161.1
TAAR (ours, R+E)@1644.1457.61
TAAR + multi-turn, same instruction@1648.1967.46
TAAR + UPTURN injection@1649.8168.94
UPTURN@1656.273.4

With TAAR cuts and intervention decisions fixed, multi-turn injection improves Avg by 4.05 pp with the same English instruction, and 5.67 pp with “Resolve this concisely.” UPTURN has the highest reported mean Avg and Pass in the table.

GPO [2] fine-tunes the reasoner using critical-step resampling; Few Decision Tokens [3] delegates selected token decisions to a reasoning model based on distributional disagreement. UPTURN trains a separate cut policy and regenerates paragraph-level suffixes with the same frozen reasoner through user-turn injection.

 

[1] Chen, Kang, et al. "Thinking Traps in Long Chain-of-Thought: A Measurable Study and Trap-Aware Adaptive Restart." arXiv preprint arXiv:2601.11940 (2026).

[2] Yu, Jiahao, et al. "Gpo: Learning from critical steps to improve llm reasoning." Advances in Neural Information Processing Systems 38 (2026): 1052-1086.

[3] Shen, Changshuo, et al. "Reasoning Can Be Restored by Correcting a Few Decision Tokens." arXiv preprint arXiv:2605.16874 (2026).

R2. Instruction choice and generalization across reasoners (W2, Q1)

We extended the instruction comparison to DeepSeek-R1-Qwen3-8B and added “Continue concisely.” as a brevity-focused control. We have incorporated the DeepSeek instruction comparison into Appendix G, with the corresponding revisions highlighted in blue. We report equal-weight averages over six benchmarks with an 8K budget and γ=0.4:

InstructionQwen3-4B AvgQwen3-8B AvgDeepSeek Avg / Pass
Resolve this concisely.56.258.349.50 / 71.24
Continue concisely.55.2556.6848.62 / 66.55
(empty user turn)51.554.249.45 / 67.66
Turtle.51.053.027.16 / 60.37

“Resolve this concisely.” achieves the highest Avg among these four instructions on all three reasoners, although its Avg advantage over the empty turn is small on DeepSeek. These results measure instruction sensitivity under a fixed position policy trained with “Resolve this concisely.” All conditions, including the empty turn and “Turtle,” therefore use positions optimized for regeneration under the default instruction. Comparing policies trained separately for each instruction would assess performance when position selection is optimized for that instruction.

A complementary comparison follows Figure 2(b): measure recovery at fixed relative positions without learned position selection. Here, V is plain-continuation success and R is success after multi-turn injection from the same prefix. We report both R and the position-wise rescue gap R−V.

To allow each instruction its own best position, we select, separately for each question and instruction, the relative position with the highest measured R among {0.1, 0.3, 0.5, 0.7, 0.9}. We then report R−V at that same position, rather than selecting the position that maximizes R−V. If a question has multiple selected failed rollouts, their scores are averaged at each position before selection.

We report equal-weight averages over AIME24, AIME25, AIME26, AMC23, and HMMT Feb 2026; Avg gives each model equal weight. All conditions use matched prefixes and generation budgets, with an 8K initial response budget and 16 continuations per condition. These are empirical oracle estimates over selected failed responses: the same continuation samples are used to select and score the position.

R−V at the highest-R position (percentage points)

InstructionQwen3-4BQwen3-8BDeepSeekAvg
Resolve this concisely.23.5125.7336.5428.60
Continue concisely.19.5016.4334.3723.43
(empty user turn)12.3413.2434.4520.01
Turtle.11.0612.6221.3014.99

“Resolve this concisely.” has the highest mean R−V among these four conditions on each model, even when each instruction can choose its own highest-R position. This suggests that a brevity-focused instruction alone does not reproduce its full benefit; it does not isolate the effect of actual generated length.

R3. Wrong-answer correction and truncation recovery (W2, Q2)

We distinguish two types of incorrect responses: wrong responses with a wrong extracted answer, and truncated responses with no extracted answer. Our budget-scaling experiments test the 8K-trained policy at longer generation budgets. To investigate how the balance between wrong and truncated responses changes with the budget, and whether the smaller gains at longer budgets reflect limited wrong-answer correction, we first examine the composition of incorrect responses at each budget:

Wrong/all and Truncated/all denote the percentages with wrong extracted answers and no extracted answers, respectively, measured across all rollouts. Correct/all, Wrong/all, and Truncated/all sum to 100, up to rounding. All results below are equal-weight averages over six benchmarks and three reasoners.

Generation budgetCorrect / all (%)Wrong / all (%)Truncated / all (%)
8K34.511.3664.13
16K57.003.8639.14
24K67.889.9122.20

Wrong answers become more prevalent as the budget increases, but truncated responses remain more frequent through 24K. At 16K, as at 8K, incorrect responses are still predominantly truncated. Training budget-specific policies at 24K would require substantially longer responses and higher generation costs. We therefore sought another way to evaluate wrong-answer correction with an 8K initial-generation budget, examining the post-training and self-correction settings used in our complementary experiments:

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
Base34.311.4064.29
GRPO54.486.3839.14
Reflexion43.602.0254.38
SCoRe52.2532.2915.45

SCoRe produces a pool with more wrong answers than truncated responses. We therefore apply the existing UPTURN policy to this pool without additional training, using “Resolve this concisely.” (γ=0.4):

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
SCoRe52.2532.2915.45
SCoRe + UPTURN64.2925.4710.24

C, W, and T denote correct answers, wrong extracted answers, and responses with no extracted answer, respectively. W→C gain and T→C gain are the accuracy gains from correcting W and T responses; C→W/T loss is the accuracy lost when initially correct responses become wrong or yield no extracted answer. Net gain is the sum of the two gains minus this loss. The table decomposes the change in the correct-answer rate relative to SCoRe into gains and losses, in percentage points across all rollouts. W→W/all is the percentage remaining wrong, including responses not selected for correction; it does not contribute to the net gain:

W→C gainT→C gainC→W/T lossNet gainW→W / all (%)
+8.04 pp+5.14 pp−1.15 pp+12.03 pp20.50

Wrong-answer correction contributes more to this gain than recovery of truncated responses. These results show that UPTURN’s gains are not limited to helping truncated responses produce a final answer: it also turns wrong extracted answers into correct ones.

 

R4. Learned cut positions and reasoning errors (Q3)

UPTURN learns where regeneration is likely to succeed, rather than directly identifying where a reasoning error occurs. The policy conditions on each question and rollout and is trained on regeneration success, without error-location annotations.

To examine this distinction, we compared learned cuts with cuts before the lowest-scored step from a process reward model (PRM; Qwen2.5-Math-PRM-7B) [4]. Since the PRM estimates step-level correctness, we treat its lowest-scored step as a candidate error location. To isolate cut-position effects, all methods use ground-truth gating to correct only initially incorrect responses. We report Avg gains over Base@32 (pp) at 8K, equally averaged over three reasoners across the six main benchmarks, using matched initial pools.

MetricFreshStartRandomPRM cutLearned cut
Avg Δ+4.6+8.3+15.5+13.7+21.2

Learned cuts yield larger gains than PRM-based cuts (+21.2 vs. +13.7 pp), supporting the benefit of selecting positions for successful regeneration. This comparison does not establish whether learned cuts coincide with actual reasoning errors.

 

[4] Zhang, Zhenru, et al. "The lessons of developing process reward models in mathematical reasoning." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

 

Please let us know if you have any further questions or would like additional clarification.

Reviewer_MmHcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-hoc correction of math reasoning under a fixed per-rollout response-length limit. UPTURN uses a 66K-parameter gate to flag likely-wrong rollouts, then a 130K-parameter position head on a frozen Qwen3-0.6B encoder to select a paragraph-boundary cut. It retains the prefix, closes the assistant turn, inserts "Resolve this concisely." as a new user turn, and lets the frozen reasoner regenerate the suffix. The position head is trained with a GRPO-style objective using only final-answer correctness. Across six math benchmarks and four reasoners, 16 initial rollouts plus at most 16 corrections outperform 32 independent rollouts in aggregate candidate accuracy (Avg: 37.1% to 55.3%) and success coverage (Pass: 53.5% to 71.3%). Ablations examine learned versus heuristic cuts and inline versus user-turn injection, including an empty-turn control. Further experiments evaluate cross-reasoner and GPQA transfer, larger budgets, and gate, runtime, and seed robustness. Gains are largest under tight budgets; at 24K tokens, they largely disappear for the two tested Qwen3 reasoners, while DeepSeek retains a positive advantage.

Critical strengths
  1. Controlled ablations support both learned cut selection and user-turn injection. With the gate and replacement rule fixed, learned cuts improve Avg by 18.2 points, compared with 12.1 for random cuts and 1.3 for fresh regeneration. Multi-turn injection also outperforms inline injection with a separately trained position policy, while an empty-turn control shows that conversational structure contributes beyond instruction wording.
  2. The gains extend across models and remain useful in practical inference. Avg improves in all 24 model–benchmark settings, and gate-based single-answer accuracy rises from 52.0% to 63.0%. Transfer to GPQA-Diamond without retraining, threshold checks, and three-seed repeats further support robustness; the measured Qwen3-4B/AIME24 pipeline also uses 36% fewer decoded tokens and about 15% less wall-clock time than Base@32.
Critical weaknesses
  1. The analysis does not clearly separate reasoning correction from reduced truncation. Correcting all rollouts gives nearly the same Avg as default gating, but gains are not broken down between unfinished responses and wrong answers. As the advantage shrinks at larger budgets, it remains unclear how much comes from fixing reasoning rather than helping responses finish.
  2. The learned cut policy needs stronger heuristic baselines. The comparisons include random cuts, start injection, and fresh regeneration, but omit simple rules such as fixed-fraction cuts or the last paragraph boundary. Without analyzing where the policy cuts relative to drops in recoverability, it is unclear what it learns and whether a simple rule could match it.
Questions for authors
  1. How does the rescue rate differ between rollouts that ended without a final answer and those that produced a wrong answer? To what extent do the gains reflect improved completion rather than reasoning correction?
  2. Does the learned cut policy outperform deterministic heuristics, such as fixed-fraction cuts or the last paragraph boundary? What distinguishes its selected positions from those simple rules?
1 comment

Thank you for your thoughtful comments and constructive suggestions. We address your concerns below.

 

R1. Reasoning correction and truncation recovery (W1, Q1)

We distinguish two types of incorrect responses: wrong responses with a wrong extracted answer, and truncated responses with no extracted answer. Our budget-scaling experiments test the 8K-trained policy at longer generation budgets. To investigate how the balance between wrong and truncated responses changes with the budget, and whether the smaller gains at longer budgets reflect limited wrong-answer correction, we first examine the composition of incorrect responses at each budget:

Wrong/all and Truncated/all denote the percentages with wrong extracted answers and no extracted answers, respectively, measured across all rollouts. Correct/all, Wrong/all, and Truncated/all sum to 100, up to rounding. All results below are equal-weight averages over six benchmarks and three reasoners (Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Qwen3-8B).

Generation budgetCorrect / all (%)Wrong / all (%)Truncated / all (%)
8K34.511.3664.13
16K57.003.8639.14
24K67.889.9122.20

Wrong answers become more prevalent as the budget increases, but truncated responses remain more frequent through 24K. At 16K, as at 8K, incorrect responses are still predominantly truncated. Training budget-specific policies at 24K would require substantially longer responses and higher generation costs. We therefore sought another way to evaluate wrong-answer correction with an 8K initial-generation budget, examining the post-training and self-correction settings used in our complementary experiments:

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
Base34.311.4064.29
GRPO54.486.3839.14
Reflexion43.602.0254.38
SCoRe52.2532.2915.45

SCoRe produces a pool with more wrong answers than truncated responses. We therefore apply the existing UPTURN policy to this pool without additional training, using “Resolve this concisely.” (γ=0.4):

MethodCorrect / all (%)Wrong / all (%)Truncated / all (%)
SCoRe52.2532.2915.45
SCoRe + UPTURN64.2925.4710.24

C, W, and T denote correct answers, wrong extracted answers, and responses with no extracted answer, respectively. W→C gain and T→C gain are the accuracy gains from correcting W and T responses; C→W/T loss is the accuracy lost when initially correct responses become wrong or yield no extracted answer. Net gain is the sum of the two gains minus this loss. The table decomposes the change in the correct-answer rate relative to SCoRe into gains and losses, in percentage points across all rollouts:

W→C gainT→C gainC→W/T lossNet gain
+8.04 pp+5.14 pp−1.15 pp+12.03 pp

Wrong-answer correction contributes more to this gain than recovery of truncated responses. These results show that UPTURN’s gains are not limited to helping truncated responses produce a final answer: it also turns wrong extracted answers into correct ones.

 

R2. Heuristic baselines and learned cut selection (W2, Q2)

The table compares learned cuts with the nearest paragraph boundaries to 30/50/70% of the rollout and the last paragraph boundary, alongside Fresh, Start, and Random. To isolate cut-position effects, all methods use ground-truth gating to correct only initially incorrect responses. We report Avg gains over Base@32 (pp) at 8K, equally averaged over six benchmarks and three reasoners, using matched initial pools.

MetricFreshStartFixed (30% / 50% / 70%)Last boundaryRandomLearned
Avg Δ+4.6+8.3+10.0 / +16.5 / +13.2+0.3+15.5+21.2

The learned policy outperforms the baselines shown, exceeding the best of the three fixed rules in this table (50%) by 4.7 percentage points in Avg. Last-boundary cuts yield only a small gain over Base@32 (+0.3 pp). Unlike fixed rules, the policy conditions on each question and rollout and is trained to select positions that lead to successful regeneration. These results support the benefit of selecting a cut based on the individual rollout rather than applying a single fixed rule.

 

Please let us know if you have any further questions or would like additional clarification.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful feedback.

As the discussion period closes today, we would appreciate it if you could take a look at our responses and the additional experiments we ran to address your questions.

If any concerns remain, we’d be happy to clarify.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.