- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SeDER controls reasoning direction and length while keeping the base language model frozen. A decision-token head reranks uncertain reasoning operators using preferences derived from successful model-generated trajectories, while a stopping head predicts when reasoning can end. Experiments on two DeepSeek-R1-Distill models report accuracy improvements and shorter reasoning traces across four math benchmarks, supplemented by ablations and GPQA transfer evaluation.
Critical strengths
-
Practical self-teaching framework. Training small auxiliary modules on the model’s own successful trajectories avoids stronger-teacher supervision and reinforcement learning, making the approach potentially useful for adapting existing reasoning models.
-
Consistent reported gains. All eight mathematical benchmark/model combinations improve over the original models in accuracy and reasoning length. The no-stopping ablations help distinguish the modules’ contributions to accuracy and token savings.
Critical weaknesses
-
The distinctive learning mechanism is insufficiently isolated. All retained preferences favor operators from successful reference trajectories. A matched decision head trained by ordinary operator-level imitation is therefore an essential comparison; full-model SFT does not provide this control. The stability term also shows mixed benefits.
-
Evaluation and reproducibility gaps limit confidence. Several AIME24 percentages do not match ordinary averaging over 30 problems with eight rollouts each, suggesting an undocumented aggregation procedure. Important construction parameters and repetition-fallback rules are unspecified. Token reductions support token efficiency, but additional model processing and discarded generations are not accounted for through end-to-end runtime measurements.
Questions for authors
In addition to the critical weaknesses, the following questions remain:
-
What sample counts, seeds, and aggregation procedure produce the AIME24 results? Please provide raw success counts and uncertainty estimates.
-
Does the proposed decision head outperform the same architecture trained to imitate successful reference operators, using identical trajectories?
-
What are the missing construction parameters and exact fallback rules? How are truncated-context representations computed, and what is the measured inference cost including discarded generations?
7 comments
Response to Reviewer EBkQ (Continued)
Response to Reviewer EBkQ (Continued)
Q3: Construction details, truncated context, repetition rules, and inference cost
Preference construction. The training procedure is detailed in Sections 3.2 and 4.1 and Appendix B.1. For each model, we sample 5,000 training questions, generate four rollouts per question, and select the shortest correct rollout as the reference. At uncertain decision-token positions, we evaluate alternative operators using continuation alignment and short-rollout stability, with a local continuation length of and . The confidence and operator-margin thresholds are 0.2 and 0.3, respectively. We retain utility-weighted preference pairs that satisfy the filtering criterion in Eq. (6). This produces 42,676 and 39,117 preference pairs for the 1.5B and 7B models, respectively. As detailed in Appendix B.1, the decision head is trained with a training margin of 0, learning rate of , weight decay of 0.01, and effective batch size of 32 for 16 epochs.
Context representations. As described in Eqs. (7), (8), and (14), the decision head uses the problem input together with the most recent 512 tokens of the pre-decision reasoning prefix. The final non-padding hidden state from the frozen backbone, immediately preceding the operator to be generated, serves as the input to the linear decision head. The same context construction is used during training and inference. For the stopping head, Appendix A.2 specifies a maximum context window of 2,048 tokens, with a stride of 1,024 during training. During inference, the stopping head uses at most the most recent 2,048 tokens and applies a majority-vote criterion requiring at least four positive predictions among the five most recent predictions.
Repetition safeguard. As clarified in the revised Appendix B.2, we monitor each generated SeDER rollout for consecutive repetitions of text segments of up to 32 characters, with a threshold of three occurrences. When repetition is detected, the affected rollout is discarded and replaced by a generation from the original frozen model without SeDER intervention. If all four initial SeDER rollouts for a problem encounter repetition, we stop further SeDER generation for that problem and instead generate four responses using the original model.
Inference cost. On MATH-500 with the 1.5B model, 31 of the 2,000 rollouts (1.55%) triggered repetition fallback. The discarded generations contribute an additional 61,690 reasoning tokens, increasing SeDER's total attempted reasoning-token count from 6,332,000 to 6,393,690. Compared with 10,084,000 tokens for the original model, this corresponds to a 36.59% reduction, versus the originally reported 37.21%. The measured end-to-end inference latency is 102.0 seconds per rollout for the original model and 143.7 seconds per rollout for SeDER (one rollout worker per GPU on 3 GPUs, including all discarded and regenerated attempts).
Thank you for the detailed rebuttal.
The matched operator-level imitation baseline addresses my main methodological concern. However, despite reducing attempted reasoning tokens by 36.59%, SeDER increases end-to-end latency from 102.0 to 143.7 seconds per rollout, raising concerns about its practical efficiency.
I will therefore maintain my rating of 2, Weak Rejection.
Response to Reviewer EBkQ
Response to Reviewer EBkQ
We thank the reviewer for the careful reading and specific suggestions on matched baselines and reproducibility.
Weakness 1 / Q2: Matched operator-level imitation
We agree that full-model SFT does not isolate the contribution of the decision-token learning mechanism. We have therefore added a matched operator-level imitation baseline using the same successful reference rollouts, uncertain operator positions, frozen backbone, linear decision-token head (including initialization), and inference policy as SeDER. In place of the counterfactual utility-based pair construction and weighted pairwise objective, this baseline trains the same head with cross-entropy to predict the reference operator. Both methods use identical inference-time intervention criteria, top-K operator candidate restriction, reranking, and stopping head.
Operator imitation obtains 82.5% / 91.3% (1.5B / 7B) on MATH-500 and 73.8% / 90.0% on AMC23, while SeDER obtains 87.1% / 93.7% and 77.2% / 92.2%. SeDER outperforms operator imitation in all four settings. This comparison directly tests the added value of the full counterfactual preference-training procedure beyond ordinary supervised operator imitation.
We agree that stability contributes less consistently than alignment. For the 1.5B model (Table 2), removing stability changes accuracy from 87.1% to 86.9% on MATH-500 and 77.2% to 75.0% on AMC23; removing alignment yields 85.6% and 72.5%, respectively. We will clarify that alignment is the primary measured signal while stability is complementary, with a dataset-dependent contribution. These ablations do not by themselves establish that every separate component of preference construction is necessary.
Weakness 2 / Q1: AIME24 aggregation, counts, and uncertainty
Thank you for noting that the AIME24 percentages do not correspond to a denominator of 30 questions × 8 rollouts = 240. The difference arises from a problem-level repetition fallback that can leave a question with four, rather than eight, retained outputs. The rule was introduced after the original evaluation protocol and was insufficiently documented.
We originally targeted eight rollouts per AIME24 problem. If one of the initial SeDER rollouts enters a repetition loop, we discard that rollout and regenerate it with the original frozen model. If all four initial SeDER rollouts for one problem loop, we stop SeDER generation for that problem and instead retain only four original-model responses, rather than proceeding to eight. This gives 29 × 8 + 1 × 4 = 236 valid evaluated rollouts for each model.
For 1.5B, one problem triggered the four-attempt problem-level rule; among its four original-model replacements, three were incorrect or reached the maximum generation length. Three other individual SeDER rollouts on other problems looped but were individually replaced without changing those questions' final rollout counts. The reported 36.9% is consistent with 87/236 = 36.864%.
For 7B, one problem similarly triggered the problem-level rule, and one additional individual SeDER rollout looped and was replaced. The reported 59.7% is consistent with 141/236 = 59.746%. We have not identified a systematic problem category associated with looping, although loops were more frequent with 1.5B than with 7B.
The random seed was 0. To reduce the variability associated with stochastic generation, we evaluated each AIME24 problem using multiple rollouts (eight per problem, except for the two model-specific fallback cases described above, which retained four). The reported accuracy is computed by aggregating correctness across all valid rollouts, yielding 87/236 = 36.9% for the 1.5B model and 141/236 = 59.7% for the 7B model. We have clarified the evaluation protocol, effective sample counts, and fallback procedure in the revised manuscript.