acceptodds
Under review as a conference paper at ICLR 2027

Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation

Abstract

Large language models (LLMs) have achieved remarkable success in complex reasoning tasks via long chain-of-thought (CoT), yet their immense computational overhead hinders real-world deployment. LLM reasoning distillation addresses this by transferring reasoning capabilities from formidable teacher models to compact student models. However, existing distillation paradigms face a fundamental dilemma. Typical off-policy distillation strictly utilizes teacher-generated golden trajectories, suffering from an exposure bias due to the mismatch between training distributions and student-generated inference contexts, which leads to error cascades in long CoT reasoning. To address this, on-policy distillation allows students to explore their own trajectories, but we demonstrate that it inherently introduces a reciprocal reversed exposure bias: the teacher model also struggles to provide positive guidance when conditioned on student-generated sub-optimal contexts. To resolve this dual exposure biases problem, we propose Monitoring Trajectories and Backtracking when it strays (MOTAB), a new LLM reasoning distillation pipeline. Specifically, MOTAB dynamically monitors the student's on-policy generation against an adaptive safety boundary. When the generation strays and exceeds this threshold, MOTAB backtracks to the last safe state and leverages teacher intervention to correct the course. This approach inherently tolerates minor student errors to mitigate exposure bias, while preventing sub-optimal contexts to circumvent reversed exposure bias. Extensive experiments on the LIMO-v2 and AceReason datasets demonstrate that MOTAB effectively alleviates the dual exposure biases, yielding a roughly 3% average performance improvement in reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM reasoning, knowledge distillation, chain-of-thoughts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qVmaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MOTAB, which monitors student reasoning using teacher likelihood and an adaptive threshold, backtracks when a trajectory strays, and appends a teacher continuation to the student's failed exploration for distillation. It aims to address both the mismatch between student training and inference distributions and unreliable teacher supervision on erroneous student prefixes. Results across multiple models and datasets show improvements over the main offline baselines and performance comparable to top-k OPD.

Critical strengths
  1. The method combines teacher continuations from backtracked prefixes with failed student exploration, preserving both exploration and correction in the distillation data.

  2. In Table 1, the Avg. score improves by 7.27 and 2.50 percentage points over the best SKD configuration under the two teacher settings, respectively. Tables 6 and 7 in the main text also report gains with other student models and on tasks from other domains.

Critical weaknesses
  1. Direct evidence of teacher supervision failure is missing. The low likelihood in Figure 1 could reflect the teacher correctly rejecting erroneous steps. The distributional differences in Figure 2 do not establish supervision failure either. Figure 4 evaluates the method using the same teacher likelihood used to select trajectories, without independent validation such as the correctness of teacher continuations.

  2. The backtracking ablation does not control for the amount of teacher generation. Removing backtracking in Table 2 also changes where teacher continuations begin and how long they are. Without comparisons against fixed depth or random backtracking under matched teacher token and training budgets, the gains could come from more teacher generation.

  3. The baseline comparisons remain insufficient. The main experiments modify SKD's acceptance criterion. In Table 4, the AIME24 and AIME25 pass@1 gains over top-k OPD are only 0.84 and 1.26 percentage points, respectively, while MATH500 performance is slightly lower. Uncertainty estimates for these differences are not reported.

Questions for authors
  1. Could the authors compare the correctness of teacher continuations from the original and backtracked prefixes on the same set of student trajectories? Can likelihood and entropy distinguish valid negative feedback from ineffective supervision?

  2. With teacher token counts and student training budgets matched, does backtracking based on the largest likelihood drop still outperform fixed depth or random backtracking?

  3. Which sampling and training procedures were changed from the original SKD method? Can the small differences from top-k OPD in Table 4 be supported by independent runs or an uncertainty analysis paired by problem?

  4. Do erroneous student prefixes contribute to the SFT loss? If so, how does training avoid reinforcing incorrect steps? If not, how does the implemented loss correspond to Equation (11)?

Reviewer_dNg3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that long chain-of-thought distillation suffers from two exposure biases: the familiar off-policy one, and a reversed on-policy one in which a teacher conditioned on a flawed student prefix gives suppressive, uninformative supervision. MOTAB checks each student step by comparing its length-normalized teacher likelihood with an entropy-adaptive boundary. On a breach it backtracks to a safe step and stitches the student prefix, a transition token and a teacher continuation; rollouts without a breach are kept as they are. The resulting data feed ordinary SFT, and appendix theorems formalize the biases and claim coverage and validity guarantees. Distilling Qwen3-32B and gpt-oss-120b into smaller students, MOTAB averages 66.64 with the Qwen3-32B teacher against 59.37 for the strongest SKD baseline.

Critical strengths
  1. Training stays plain SFT. The teacher only scores steps and writes continuations offline, so training needs no joint teacher and student logits, and the reported synthesis cost is comparable to SKD. If the gains hold, this makes the method cheap to adopt.

  2. The evaluation is broad. It spans two teacher families, two data sources, several students and recent on-policy baselines, and on a small student MOTAB is comparable to top-k on-policy distillation while using only SFT. My concerns are therefore about attribution and rigor, not scope.

Critical weaknesses
  1. The gains are not shown to come from backtracking. Rollouts without a breach are kept with no described correctness filter, and at the smallest gamma0 the backtracking rate is 0.005, so if it is measured per trajectory the training data are almost all unfiltered student outputs. Yet this setting beats every Table 1 baseline on AIME25 pass@1, and the ablation without backtracking nearly matches it. SFT on the student's own outputs explains this equally well, and nothing separates the two. The remaining evidence does not close the gap: the near-perfect teacher and student alignment at the largest gamma0 (Fig. 4), which a teacher-written suffix would produce anyway, comes with the sweep's lowest score, below plain SFT; gamma0 appears to be chosen on the test set; and all results are single runs whose key gaps are about one standard error. Since backtracking is the contribution, its central claim remains unsupported.

  2. The theoretical analysis does not hold as stated. Theorem 1 claims that sequence-level total variation grows quadratically in length, but its proof sums L terms each at most 1, which gives at most L, and the proof of Theorem 3 then asserts a lower bound of order L^2 epsilon, impossible for long sequences because total variation never exceeds 1. Theorem 2 infers a uniform next-step distribution from a vanishing joint probability, although the conditional divides by the prefix probability and can stay sharply peaked; Fig. 1 itself shows teacher log-probabilities on student contexts near zero, far from uniform. Theorem 5 assumes the prefix at the backtracking point keeps probability bounded away from zero, which a length-normalized threshold does not ensure. The theorems are offered as the method's formal support, so without them the claims rest on the experiments questioned above.

Questions for authors
  1. Could you report the untrained students, and SFT on token-matched trajectories the student generates itself with no teacher intervention? If that control comes close to MOTAB, what gain remains for backtracking?
  2. How was gamma0 chosen for each table, and how does MOTAB compare with SKD when both are tuned to the same extent on a validation split? The Table 6 SKD rows appear to average their AIME scores over beta.
  3. Could you report means and standard deviations over several seeds for Tables 1 and 2? With ties to baselines on some MATH500 levels, the claim that MOTAB consistently and significantly outperforms all baselines needs this.
  4. Eq. 6 caps the boundary at gamma0, at most 0.5, yet several thresholds in the case studies of Figs. 7 and 8 exceed 0.5. Which boundary was actually used, and is the reported backtracking rate per trajectory?
Reviewer_CRUuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MOTAB, a new LLM reasoning distillation pipeline that stitches the student-generated prefix and teacher-generated suffix to build training trajectories for better distillation performance, and aims to resolve the dual exposure biases introduced in the original OPD. The proposed methods are empirically evaluated on different student and teacher models and compared with common baselines like SFT, ImitKD, and SKD.

Critical strengths
  1. The proposed method is intuitive. It is reasonable to use the teacher to resume generation when the student's reasoning deviates from the safety boundary.
Critical weaknesses
  1. Some claims of this paper are not aligned with what is actually implemented. In the abstract and Section 1, the authors claim that "MOTAB dynamically monitors the student’s on-policy generation". However, according to Algorithm 1, MOTAB actually builds an offline dataset of stitched trajectories first and then performs SFT distillation, which is not on-policy. The authors also admit this limitation in Section F and attribute their implementation to computational constraints, but the motivation of this paper is to eliminate dual exposure biases, while the actual implementation brings exposure bias back again, which is self-contradictory.

  2. Some experimental settings raise concerns about the faithfulness of the baseline reproduction. It is unclear whether MOTAB can really outperform those baselines rather than their variants with unofficial settings. Please see my questions for details.

Questions for authors
  1. What are the detailed experimental settings for ImitKD and RFT? For instance, do they use online rollout trajectories during training or offline trajectories curated before training? To my knowledge, the original ImitKD applies online rollout trajectories during training, and it is unclear whether the authors faithfully reproduce the original algorithm. And for ImitKD, what are the probabilities of using teacher sampling and student sampling at each decoded position?

  2. To my knowledge, the original SKD uses a token-level top-K coverage acceptance criterion, while this paper applies a step-level acceptance criterion based on the teacher's and student's geometric-mean token probabilities. These changes affect both the acceptance rule and the granularity of teacher intervention and result in an unfaithful reproduction of the original SKD. Could the authors report results from a faithful implementation of the original SKD, and distinguish these results from the step-level variant?

  3. In addition, the original SKD applies online interleaved sampling with the student and teacher during training. The current manuscript does not clearly specify whether the SKD baseline follows this online procedure or instead uses trajectories generated offline for SFT. If using offline trajectories, could the authors provide results from a faithful implementation of the original SKD, including the token-level top-K coverage acceptance criterion and online rollout trajectories during training?

  4. Compared with top-k OPD, MOTAB does not show a significant improvement. Besides, MOTAB is claimed to be an on-policy method, but the only comparison with the most common top-k OPD baseline is in Table 4. Other tables seem to intentionally avoid this comparison. Could the authors provide an explanation?

  5. In Table 4, RFT and SKD are included as on-policy methods. However, just as mentioned in my previous questions, are their real implementations in this paper on-policy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.