acceptodds
Under review as a conference paper at ICLR 2027

Learning to Redirect and Stop: Self-Teaching Control for Reasoning Models

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities through extended chain-of-thought generation, yet they may take suboptimal directions at critical reasoning transitions or continue reasoning after a correct answer has already emerged. We introduce Self-teaching Decision and Exit for Reasoning (SeDER), a lightweight framework that adaptively controls both the direction and length of reasoning while keeping the underlying language model frozen. At its core, a decision-token head selectively intervenes at uncertain reasoning transitions. Using successful rollouts generated by the base model itself, we construct counterfactual alternatives and evaluate them based on their compatibility with successful continuations and stability under short stochastic rollouts. The resulting utility differences yield weighted pairwise preferences for training the decision-token head to rerank plausible reasoning operators, without requiring a stronger teacher, a separate process reward model, or reinforcement learning. We further incorporate a lightweight stopping head that terminates reasoning once a sufficient solution has been reached. Experiments across multiple reasoning benchmarks with DeepSeek-R1-Distill-Qwen-1.5B and 7B demonstrate consistent accuracy improvements while substantially reducing reasoning length. SeDER achieves the highest accuracy among the evaluated baselines across all benchmarks, while ablation studies validate the effectiveness of its utility construction and selective intervention strategy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Large Reasoning Models, Efficient Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EBkQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SeDER controls reasoning direction and length while keeping the base language model frozen. A decision-token head reranks uncertain reasoning operators using preferences derived from successful model-generated trajectories, while a stopping head predicts when reasoning can end. Experiments on two DeepSeek-R1-Distill models report accuracy improvements and shorter reasoning traces across four math benchmarks, supplemented by ablations and GPQA transfer evaluation.

Critical strengths
  1. Practical self-teaching framework. Training small auxiliary modules on the model’s own successful trajectories avoids stronger-teacher supervision and reinforcement learning, making the approach potentially useful for adapting existing reasoning models.

  2. Consistent reported gains. All eight mathematical benchmark/model combinations improve over the original models in accuracy and reasoning length. The no-stopping ablations help distinguish the modules’ contributions to accuracy and token savings.

Critical weaknesses
  1. The distinctive learning mechanism is insufficiently isolated. All retained preferences favor operators from successful reference trajectories. A matched decision head trained by ordinary operator-level imitation is therefore an essential comparison; full-model SFT does not provide this control. The stability term also shows mixed benefits.

  2. Evaluation and reproducibility gaps limit confidence. Several AIME24 percentages do not match ordinary averaging over 30 problems with eight rollouts each, suggesting an undocumented aggregation procedure. Important construction parameters and repetition-fallback rules are unspecified. Token reductions support token efficiency, but additional model processing and discarded generations are not accounted for through end-to-end runtime measurements.

Questions for authors

In addition to the critical weaknesses, the following questions remain:

  1. What sample counts, seeds, and aggregation procedure produce the AIME24 results? Please provide raw success counts and uncertainty estimates.

  2. Does the proposed decision head outperform the same architecture trained to imitate successful reference operators, using identical trajectories?

  3. What are the missing construction parameters and exact fallback rules? How are truncated-context representations computed, and what is the measured inference cost including discarded generations?

7 comments
AuthorsOpenReview ↗

Response to Reviewer EBkQ

Response to Reviewer EBkQ

We thank the reviewer for the careful reading and specific suggestions on matched baselines and reproducibility.

Weakness 1 / Q2: Matched operator-level imitation

We agree that full-model SFT does not isolate the contribution of the decision-token learning mechanism. We have therefore added a matched operator-level imitation baseline using the same successful reference rollouts, uncertain operator positions, frozen backbone, linear decision-token head (including initialization), and inference policy as SeDER. In place of the counterfactual utility-based pair construction and weighted pairwise objective, this baseline trains the same head with cross-entropy to predict the reference operator. Both methods use identical inference-time intervention criteria, top-K operator candidate restriction, reranking, and stopping head.

Operator imitation obtains 82.5% / 91.3% (1.5B / 7B) on MATH-500 and 73.8% / 90.0% on AMC23, while SeDER obtains 87.1% / 93.7% and 77.2% / 92.2%. SeDER outperforms operator imitation in all four settings. This comparison directly tests the added value of the full counterfactual preference-training procedure beyond ordinary supervised operator imitation.

We agree that stability contributes less consistently than alignment. For the 1.5B model (Table 2), removing stability changes accuracy from 87.1% to 86.9% on MATH-500 and 77.2% to 75.0% on AMC23; removing alignment yields 85.6% and 72.5%, respectively. We will clarify that alignment is the primary measured signal while stability is complementary, with a dataset-dependent contribution. These ablations do not by themselves establish that every separate component of preference construction is necessary.

Weakness 2 / Q1: AIME24 aggregation, counts, and uncertainty

Thank you for noting that the AIME24 percentages do not correspond to a denominator of 30 questions × 8 rollouts = 240. The difference arises from a problem-level repetition fallback that can leave a question with four, rather than eight, retained outputs. The rule was introduced after the original evaluation protocol and was insufficiently documented.

We originally targeted eight rollouts per AIME24 problem. If one of the initial SeDER rollouts enters a repetition loop, we discard that rollout and regenerate it with the original frozen model. If all four initial SeDER rollouts for one problem loop, we stop SeDER generation for that problem and instead retain only four original-model responses, rather than proceeding to eight. This gives 29 × 8 + 1 × 4 = 236 valid evaluated rollouts for each model.

For 1.5B, one problem triggered the four-attempt problem-level rule; among its four original-model replacements, three were incorrect or reached the maximum generation length. Three other individual SeDER rollouts on other problems looped but were individually replaced without changing those questions' final rollout counts. The reported 36.9% is consistent with 87/236 = 36.864%.

For 7B, one problem similarly triggered the problem-level rule, and one additional individual SeDER rollout looped and was replaced. The reported 59.7% is consistent with 141/236 = 59.746%. We have not identified a systematic problem category associated with looping, although loops were more frequent with 1.5B than with 7B.

The random seed was 0. To reduce the variability associated with stochastic generation, we evaluated each AIME24 problem using multiple rollouts (eight per problem, except for the two model-specific fallback cases described above, which retained four). The reported accuracy is computed by aggregating correctness across all valid rollouts, yielding 87/236 = 36.9% for the 1.5B model and 141/236 = 59.7% for the 7B model. We have clarified the evaluation protocol, effective sample counts, and fallback procedure in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer EBkQ (Continued)

Response to Reviewer EBkQ (Continued)

Q3: Construction details, truncated context, repetition rules, and inference cost

Preference construction. The training procedure is detailed in Sections 3.2 and 4.1 and Appendix B.1. For each model, we sample 5,000 training questions, generate four rollouts per question, and select the shortest correct rollout as the reference. At uncertain decision-token positions, we evaluate alternative operators using continuation alignment and short-rollout stability, with a local continuation length of L=32L=32 and α=β=1\alpha=\beta=1. The confidence and operator-margin thresholds are 0.2 and 0.3, respectively. We retain utility-weighted preference pairs that satisfy the filtering criterion in Eq. (6). This produces 42,676 and 39,117 preference pairs for the 1.5B and 7B models, respectively. As detailed in Appendix B.1, the decision head is trained with a training margin of 0, learning rate of 10−410^{-4}, weight decay of 0.01, and effective batch size of 32 for 16 epochs.

Context representations. As described in Eqs. (7), (8), and (14), the decision head uses the problem input together with the most recent 512 tokens of the pre-decision reasoning prefix. The final non-padding hidden state from the frozen backbone, immediately preceding the operator to be generated, serves as the input to the linear decision head. The same context construction is used during training and inference. For the stopping head, Appendix A.2 specifies a maximum context window of 2,048 tokens, with a stride of 1,024 during training. During inference, the stopping head uses at most the most recent 2,048 tokens and applies a majority-vote criterion requiring at least four positive predictions among the five most recent predictions.

Repetition safeguard. As clarified in the revised Appendix B.2, we monitor each generated SeDER rollout for consecutive repetitions of text segments of up to 32 characters, with a threshold of three occurrences. When repetition is detected, the affected rollout is discarded and replaced by a generation from the original frozen model without SeDER intervention. If all four initial SeDER rollouts for a problem encounter repetition, we stop further SeDER generation for that problem and instead generate four responses using the original model.

Inference cost. On MATH-500 with the 1.5B model, 31 of the 2,000 rollouts (1.55%) triggered repetition fallback. The discarded generations contribute an additional 61,690 reasoning tokens, increasing SeDER's total attempted reasoning-token count from 6,332,000 to 6,393,690. Compared with 10,084,000 tokens for the original model, this corresponds to a 36.59% reduction, versus the originally reported 37.21%. The measured end-to-end inference latency is 102.0 seconds per rollout for the original model and 143.7 seconds per rollout for SeDER (one rollout worker per GPU on 3 GPUs, including all discarded and regenerated attempts).

Thank you for the detailed rebuttal.

The matched operator-level imitation baseline addresses my main methodological concern. However, despite reducing attempted reasoning tokens by 36.59%, SeDER increases end-to-end latency from 102.0 to 143.7 seconds per rollout, raising concerns about its practical efficiency.

I will therefore maintain my rating of 2, Weak Rejection.

Reviewer_LWLQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SeDER (Self-teaching Decision and Exit for Reasoning), a method designed to make reasoning language models more accurate and less wasteful. The main idea is that models often make a bad choice at a few important reasoning steps or keep thinking even after they already have the correct answer. SeDER adds two small components while keeping the original model frozen: one redirects the model at uncertain reasoning points toward a better next step, and the other learns when to stop reasoning early. Importantly, it learns these behaviors from the model’s own successful solutions rather than relying on a stronger teacher model or reinforcement learning. In experiments on several math reasoning benchmarks using DeepSeek-R1-Distill models, SeDER generally achieved higher accuracy while using substantially fewer reasoning tokens, suggesting that better control over where reasoning goes and when it ends can improve both performance and efficiency.

Critical strengths

SeDER keeps the underlying LLM frozen and adds only a lightweight decision-token head plus a stopping head, trained from the model’s own successful rollouts rather than from a stronger teacher.

The main novelty is a self-teaching reranker trained from the model’s successful rollouts, where counterfactual choices are ranked through continuation compatibility and short-rollout stability using utility-weighted pairwise preferences.

Critical weaknesses

Selective intervention at uncertain transitions is substantially anticipated by the paper "Reasoning Can Be Restored by Correcting a Few Decision Tokens", while operator-level choice and downstream evaluation is similar to paper Neural Chain-of-Thought Search (NCoTS). SeDER’s decision head appears primarily to amortize local search rather than introduce a new control formulation. Its stopping classifier is also closely related to TERMINATOR, which likewise freezes the base model and trains a lightweight sufficiency probe. The novelty is where the rollout is ranked, but not the broad redirection-and-stopping paradigm.

Successful-rollout mining still needs a correctness signal, e.g., labels or exact-match checks. And it can learn only from strategies the frozen base model already generates successfully. This weakness is important because it weakens claims of self-supervision.

Questions for authors

A key limitation is the incomplete compute-controlled evaluation and mechanism isolation. Although the paper reports intervention frequency and several useful ablations, it does not provide matched-compute comparisons against especially close uncertainty-triggered branching approaches, nor a detailed accounting of the additional rollout, counterfactual-evaluation, training, and inference costs introduced by SeDER. Thus, it remains difficult to determine how much of the improvement comes specifically from the proposed self-derived weighted-preference mechanism versus additional sampling/computation.

The approach is mostly related to math tasks that are exact-match checks. How can this method be applied to real world reasoning question where the answer is free-text (and not exact-match)?

The experiment is only valiadated by two deepseek-r1-distilled-qwen models with size less than 8B, can the finding of the paper be validated with larger model such as qwen3 or 3.5 model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer LWLQ

Response to Reviewer LWLQ

We thank the reviewer for highlighting the relationship to earlier decision-token correction, NCoTS, and TERMINATOR, and for asking for clearer mechanism isolation and cost accounting.

Weakness 1 / Q1: Novelty and mechanism isolation

We agree that selecting uncertain reasoning transitions, reasoning over discrete operators, and learning when to stop are not new paradigms in themselves. We will sharpen our novelty statement accordingly. The distinctive contribution of SeDER is the offline construction of self-derived, utility-weighted counterfactual operator preferences using both compatibility with a successful continuation and stability under short stochastic continuations, followed by training a small decision-token head to amortize these comparisons at inference time. We do not claim to invent uncertainty-triggered intervention or learned early stopping.

To test whether these preferences contribute beyond simply imitating successful operators, we added a matched operator-level imitation baseline. It uses the same successful reference rollouts, uncertain operator positions, frozen backbone, linear head, and inference-time candidate restriction and reranking. Instead of utility-filtered, utility-weighted pairwise preferences, it trains with cross-entropy to predict the reference operator. The imitation baseline achieves 82.5% / 91.3% (1.5B / 7B) on the MATH-500 dataset and 73.8% / 90.0% on the AMC23 dataset versus 87.1% / 93.7% and 77.2% / 92.2% for SeDER. SeDER achieves higher accuracy than operator imitation in all four settings. This isolates the overall value of our counterfactual preference-learning procedure relative to direct operator imitation, although not every ingredient of the procedure separately.

We also agree that successful-rollout mining requires an outcome-level correctness signal and is constrained by which successful solutions the frozen model can generate. Our use of self-teaching means that local operator preferences are constructed from the base model's own successful trajectories, without a stronger teacher, separately trained process reward model, or reinforcement learning; it does not mean correctness-free or fully unsupervised learning. We will make this limitation explicit.

Q1 (continued): Offline training cost and comparison with inference-time branching

SeDER's counterfactual comparisons are performed offline, once, rather than as repeated branching during each inference call. The offline pipeline comprises generation of successful reference rollouts, construction and evaluation of alternative operator continuations, construction of stopping labels, and training the two auxiliary heads. The complete data-construction-and-training pipeline can be completed within 24 hours on four NVIDIA H200 GPUs. These costs are not zero; they are incurred once and can be amortized over subsequent inference requests.

In contrast, uncertainty-triggered branching approaches explore additional continuations at test time. For example, [1] reports accuracy gains but, in Table 1, higher total generated-token consumption than the original model. Because SeDER targets improved accuracy and reduced inference-time token consumption, we did not include this particular branching approach among our primary token-efficiency baselines. This is not a substitute for a strictly matched-compute comparison, and we will state this limitation and clarify the distinction in the revised experimental discussion.

For concrete inference-time accounting, on MATH-500 (1.5B), 31 of 2,000 initial rollouts (1.55%) triggered repetition fallback. Table 1 implies 10,084,000 original-model reasoning tokens and 6,332,000 retained SeDER reasoning tokens; the discarded attempts add 61,690, giving 6,393,690 attempted reasoning tokens for SeDER and a 36.59% reduction over the original model (rather than 37.21% without discarded attempts). The fallback detects consecutive repetitions of text segments of up to 32 characters, with a threshold of three occurrences. When repetition is detected, the affected rollout is discarded and regenerated using the original frozen model without SeDER intervention. If all four initial SeDER rollouts for a problem encounter repetition, we stop further SeDER generation for that problem and instead generate four responses using the original model. End-to-end latency is 102.0 seconds per rollout for the original model and 143.7 seconds per rollout for SeDER with the 1.5B model, using one rollout worker per GPU on 3 GPUs and including all discarded and regenerated attempts.

[1] Zhu et al. (2026), Dissecting Failure Dynamics in Large Language Model Reasoning, ACL 2026.

AuthorsOpenReview ↗

Response to Reviewer LWLQ (Continued)

Response to Reviewer LWLQ (Continued)

Q2: Extension beyond exact-match mathematics

SeDER is not intrinsically restricted to exact-match final answers. Exact-match is used in our present evaluation to identify successful reference rollouts and to derive stopping labels; the counterfactual alignment/stability computation, preference training, and decision-head reranking do not themselves depend on exact-match. For free-text problems, successful trajectories could instead be selected through task-specific verifiers, calibrated LLM-based judges, or human assessments. A separate sufficiency or quality signal would likewise be needed to construct reliable stopping supervision. These signals introduce noise and possible evaluation bias, so free-text effectiveness remains to be validated.

We already provide preliminary cross-domain, though not free-text, evidence in the GPQA Diamond transfer evaluation (Table 4): without retraining, SeDER changes accuracy from 33.8% to 34.4% for 1.5B and from 44.4% to 46.8% for 7B on questions spanning physics, chemistry, and biology. GPQA is multiple-choice, so we do not present it as evidence of free-text generalization.

Q3: Extension to larger reasoning models

We agree that larger-model and cross-family experiments, including Qwen3-family models, would strengthen the evidence. Our reported results cover only the 1.5B and 7B DeepSeek-R1-Distill-Qwen models; both improve on all four math benchmarks, but this does not establish performance beyond the evaluated family or scale.

Architecturally, SeDER keeps the backbone frozen. The decision-token head contains only a linear mapping from hidden state to the operator vocabulary, so its parameter count scales with hidden dimension rather than the full backbone parameter count. The stopping head and offline generation/evaluation nevertheless add nontrivial costs, especially as the backbone grows. We will state clearly that effectiveness and compute–accuracy trade-offs on larger models remain open empirical questions, rather than claiming they are already validated.

AuthorsOpenReview ↗

Response to Reviewer LWLQ - Additional Clarification on Inference Latency

We would like to clarify an important limitation of the latency measurements reported in our earlier response. The reported values of 102.0 and 143.7 seconds per rollout were obtained from a small timing experiment involving four MATH-500 problems, with two rollouts per problem (eight rollouts in total). The SeDER timing included additional fallback generations, although fallback occurred in only 1.55% of rollouts in the full MATH-500 evaluation.

This fallback-heavy, limited timing sample likely overestimates SeDER's average wall-clock latency across the full benchmark. Therefore, the reported latency difference should not be interpreted as a representative measure of inference efficiency. Importantly, SeDER's 36.59% reduction in attempted reasoning tokens is calculated over the full MATH-500 evaluation, including discarded generations. Together with the one-time offline construction of counterfactual preferences, these results demonstrate that SeDER reduces inference-time token consumption without requiring repeated counterfactual branching during generation. We acknowledge that the overall wall-clock efficiency, including auxiliary-head overhead, requires more representative measurements.

Reviewer_3xgvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SeDER controls a frozen reasoning model through a linear head that reranks uncertain reasoning operators and a stopping head that ends redundant deliberation. Both learn from the model's own correct rollouts. Experiments on 1.5B and 7B models report improved mathematical accuracy and shorter reasoning, with a GPQA transfer evaluation.

Critical strengths
  • The framework separates reasoning direction from stopping and requires no stronger teacher or backbone updates. Sparse operator intervention offers a practical, focused alternative to full-model training.
  • Results improve on both model sizes, and the component/intervention ablations help distinguish accuracy improvements from the stopping head's contribution to shorter traces.
Critical weaknesses
  • The counterfactual utility measures compatibility with one successful continuation and local stability, rather than the alternative's eventual correctness. Retaining only pairs favoring the reference operator may mainly teach imitation; a matched supervised operator-head baseline is missing.
  • Efficiency is measured through retained reasoning length, without latency or total attempted-token accounting. The stopping head evaluates an extra decoder block per token, and Appendix B.2 discards looping generations and regenerates with the base model (about 1.5% on MATH-500). Reported output lengths do not explicitly account for discarded attempts.
  • Stopping supervision uses only successful trajectories and labels every position after the first gold-answer match as safe to stop. Despite answer normalization and temporal voting, false stopping on unsuccessful trajectories remains unquantified.
Questions for authors
  1. How often do utility preferences agree with final-answer correctness after completing each alternative, and does the method outperform a supervised operator head trained on the same reference traces?
  2. Can you report end-to-end latency and total generated tokens including fallback attempts, alongside stopping false-positive rates on unsuccessful traces?

Thank you for the detailed response and experiments, which have addressed most concerns, so I have raised my score.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3xgv

Response to Reviewer 3xgv

We thank the reviewer for recognizing the practical benefits of sparse intervention and the complementary roles of reasoning-direction control and early stopping. We address the concerns about the learning mechanism and actual inference cost below.

Weakness 1 / Q1: Operator imitation and validity of local utility

We agree that, because SeDER retains preference pairs only when the reference operator has higher local utility than an alternative, it is important to distinguish the effect of counterfactual preference learning from simply imitating operators in successful traces. We therefore added a matched operator-level imitation baseline. It uses the identical successful reference rollouts, uncertain decision-token positions, frozen backbone, linear head architecture, and inference-time intervention and reranking rules. The only change to the learning scheme is that it trains the head to predict the observed reference operator via cross-entropy, with no counterfactual utility evaluation, utility-based filtering, or preference weighting; the stopping head is held fixed.

Operator imitation obtains 82.5% / 91.3% (1.5B / 7B) on MATH-500 and 73.8% / 90.0% on AMC23, compared with 87.1% / 93.7% and 77.2% / 92.2% for SeDER. SeDER outperforms operator imitation in all four settings. This is a direct control for whether learning from successful operator labels alone is sufficient.

We also agree that our local utility is not a direct estimator of eventual correctness: alignment measures compatibility with the successful reference continuation, and stability measures agreement among short stochastic continuations. They are computationally tractable local proxies for constructing preferences, not guarantees of better final answers. We will explicitly acknowledge that the agreement between local utility rankings and full counterfactual rollout correctness has not yet been directly measured. We will clarify this limitation in the revised paper.

Weaknesses 2–3 / Q2: End-to-end latency, fallback tokens, and stopping false positives

Runtime. On MATH-500 using DeepSeek-R1-Distill-Qwen-1.5B, the measured end-to-end latency, including both auxiliary heads and any discarded/replacement generations, is 102.0 seconds per rollout for the original model versus 143.7 seconds per rollout for SeDER, measured with one rollout worker per GPU on 3 GPUs and CUDA-synchronized timing of every decoding attempt; this is a 40.9% increase.

Repetition fallback. Appendix B.2 briefly mentions this safeguard. For every rollout, we check for consecutive repetitions of text segments of up to 32 characters, with a threshold of three occurrences. When repetition is detected, we discard the affected SeDER generation and regenerate the response using the original frozen model without SeDER intervention. If all four initial SeDER rollouts for a question encounter repetition, we discontinue SeDER for that question and instead generate four responses using the original model.

On MATH-500 (1.5B), 31/2,000 initial rollouts (1.55%) triggered fallback. Table 1's averages imply 10,084,000 original-model reasoning tokens (2,000 × 5,042) versus 6,332,000 retained SeDER reasoning tokens (2,000 × 3,166). Discarded SeDER attempts add 61,690 tokens, yielding 6,393,690 total attempted reasoning tokens for SeDER. The resulting token reduction is 36.59%, versus 37.21% before accounting for discarded attempts. We distinguish this reasoning-token accounting from measured wall-clock latency and from any non-reasoning answer tokens.

Stopping false positives. At the trajectory level, on 80 trajectories (20 MATH-500 problems stratified by difficulty, 4 rollouts each, 1.5B) with the inference stop-vote criterion (threshold 0.7, at least 4 positive votes among the last 5), 21/80 stops (26.2%, 95% Wilson CI 17.9–36.8%) produced a wrong answer; only 2/80 (2.5%, 95% CI 0.7–8.7%) were trajectories that would have reached a correct answer had reasoning continued. The stopping head uses a conservative threshold and temporal voting to reduce isolated false triggers, but these mechanisms are not substitutes for an empirical false-positive measurement.

AuthorsOpenReview ↗

Response to Reviewer 3xgv - Additional Clarification on Inference Latency

We would like to clarify the scope of the latency measurements in our earlier response. The reported values of 102.0 seconds for the original model and 143.7 seconds for SeDER were obtained from only four MATH-500 problems, with two rollouts per problem (eight rollouts in total). The SeDER runs included additional fallback generations, whereas fallback occurred in only 1.55% of rollouts in the full MATH-500 evaluation.

Consequently, the additional fallback generations likely led to an overestimation of SeDER's average latency relative to its performance across the full benchmark. Given the limited sample size, these measurements are insufficient to draw a reliable conclusion about benchmark-wide wall-clock efficiency. However, the 36.59% reduction in total attempted reasoning tokens is based on the full evaluation and remains valid. We acknowledge that auxiliary-head overhead may offset some of the runtime benefits of shorter reasoning traces, and we will clarify this distinction in the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We thank all three reviewers for their careful assessments and constructive suggestions. We have updated the manuscript to incorporate additional experiments and clarify important implementation details. We provide two cross-cutting clarifications below: a matched operator-level imitation baseline to isolate the contribution of our preference-learning mechanism, and a more explicit account of repetition fallback and its computational cost.

1. Additional experiment: matched operator imitation baseline

To investigate whether SeDER's improvements arise from counterfactual utility-based preference learning rather than merely imitating successful trajectories, we have added a matched operator-level imitation baseline. It uses the same successful reference rollouts (the shortest correct rollout for each training question), eligible uncertain operator positions, frozen backbone, linear decision-token head architecture, and head initialization as SeDER. Instead of evaluating counterfactual alternatives and constructing utility-filtered, utility-weighted preference pairs, it directly predicts the reference operator using cross-entropy over the canonical operator vocabulary.

At inference time, we hold the intervention trigger (operator proposal, low confidence, and small top-operator probability gap), top-K candidate set, base-model-plus-head reranking, sampling settings, and stopping head fixed. Thus, the comparison specifically tests ordinary operator imitation versus the complete counterfactual preference-learning procedure, without confounding changes to the backbone or inference-time intervention policy.

ModelBenchmarkOperator imitation (Acc.)SeDER (Acc.)
1.5BMATH-50082.5%87.1%
1.5BAMC2373.8%77.2%
7BMATH-50091.3%93.7%
7BAMC2390.0%92.2%

The results obtained so far show that SeDER outperforms the matched operator-level imitation baseline, providing further evidence that counterfactual utility-based preference learning offers benefits beyond simply imitating successful reasoning trajectories. We have included this comparison and the corresponding results in the newly added Section 4.5 of the revised manuscript. Together with the existing ablation studies on alignment and stability, these results further support the effectiveness of our preference-learning mechanism.

2. Repetition fallback and token accounting

As briefly noted in Appendix B.2, we monitor every SeDER rollout for consecutive repetitions of text segments of up to 32 characters, with a threshold of three occurrences. When a loop is detected, the affected SeDER generation is discarded and replaced by a new generation from the original frozen backbone without SeDER intervention. If all four initial SeDER rollouts for a problem exhibit repetition, we stop SeDER generation for that problem and instead retain four original-model rollouts. This is a rare safeguard against looping, not an additional inference-time branching strategy.

On MATH-500 with DeepSeek-R1-Distill-Qwen-1.5B, 31 of 2,000 initial rollouts (1.55%) triggered repetition handling. Table 1 reports an average of 5,042 reasoning tokens per original-model rollout and 3,166 per retained SeDER rollout, equivalent to 10,084,000 and 6,332,000 reported reasoning tokens across 2,000 rollouts. The discarded attempts consumed 61,690 further reasoning tokens, bringing the attempted-token total for SeDER to 6,393,690. The reduction relative to the original model is therefore 36.59%, versus 37.21% when counting only the retained outputs. Repetition fallback changes the reported saving by only about 0.62 percentage points. We have expanded Appendix B.2 to provide the detailed repetition detection and fallback procedures, together with an explicit accounting of the additional token consumption.

With one rollout worker per GPU on 3 GPUs, including all fallback and discarded attempts, the measured end-to-end latency is 102.0 seconds per rollout for the original model and 143.7 seconds per rollout for SeDER. We will incorporate the precise fallback procedure and attempted-token accounting into the revised manuscript.

AuthorsOpenReview ↗

Additional Clarification on Inference Latency

We would like to clarify the scope of the end-to-end inference latency measurements reported in our previous responses.

The reported values of 102.0 seconds per rollout for the original model and 143.7 seconds for SeDER were obtained from a limited timing experiment involving four MATH-500 problems, with two rollouts per problem (eight rollouts in total). The SeDER timing included additional fallback generations, whereas repetition fallback occurred in only 1.55% of rollouts in the full MATH-500 evaluation.

Consequently, although these measurements reflect the actual execution time in this small sample, the additional fallback generations likely led to an overestimation of SeDER's average latency relative to its performance across the full benchmark. The limited sample size also makes the reported latency comparison insufficient to establish representative wall-clock performance.

We acknowledge that SeDER incurs additional computational overhead from its auxiliary heads, which may offset the runtime benefits of shorter reasoning traces. Therefore, we do not claim a demonstrated reduction in end-to-end wall-clock latency.

The 36.59% reduction in attempted reasoning tokens, however, is based on the full MATH-500 evaluation, including discarded generations, and remains valid. We will clarify the distinction between reasoning-token efficiency and wall-clock efficiency in the revised manuscript and explicitly acknowledge the limitations of the current latency measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.