acceptodds
Under review as a conference paper at ICLR 2027

Distilled Reasoning Models Bypass Uncorrected Errors in Their Own Chains of Thought

Abstract

Reasoning models are increasingly built by distillation, that is supervised fine-tuning on chains of thought sampled from an RL-trained teacher. We study chain-of-thought faithfulness in these models through a regime we call bypass, in which the model produces the correct final answer while an uncorrected error stands in its chain of thought, and we ask how often it occurs on the models' own errors, what governs its rate, where the answer then comes from, and what training changes it. We measure bypass by injecting a wrong step mid-thought and regenerating, and by auditing the models' own arithmetic errors across roughly thirty thousand sampled rollouts, executing every equation they write. Both instruments are checked against 492 blind human labels. On its own validated errors the 7B distill bypasses a quarter to a third of them depending on the detector, with blind human annotation corroborating the effect, and two thirds on a memorization-proof synthetic set. Under injection, bypass reaches half to three-quarters of injected errors on GSM8K and on the synthetic set across the DeepSeek-R1-Distill-Qwen family under the judge, whereas the base-matched RL model QwQ-32B shows near zero on GSM8K. It falls to roughly 3% on AIME and is several times lower on BBH state-tracking, a pattern consistent with answer recomputability across our benchmarks, and scale does not remediate it. Attention knockout, forced-answer, and interruption probes locate the correct answer in a post-boundary re-derivation with no detectable direct influence of the corrupted step, and injecting at a fixed early position overstates bypass for the distilled model by fourteen points relative to its own errors on the same items, a gap that closes when the injection is placed at the natural error position. RL post-training on the same distilled checkpoint cuts bypass three- to four-fold under injection and by a third to a half on the model's own errors. Bypass marks a regime in which the visible chain ends in the wrong state while the correct answer is re-derived after the reasoning boundary, so a monitor reading the chain sees wrong reasoning and a right answer with no correction between them, and RL post-training of the same checkpoint reduces it substantially.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
chain-of-thought faithfulness, chain-of-thought monitorability, reasoning models, distillation, self-correction, faithfulness evaluation, LLM evaluation, post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cSMbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the bypass effect in distilled reasoning models, which describes the phenomenon where a model produces a correct final answer despite containing uncorrected errors within its chain-of-thought reasoning. The authors observe this phenomenon via two approaches: manually injecting errors and auditing natural errors spontaneously generated by the model. A keyword marker detector and an LLM judge are adopted to classify bypass cases, and human annotation validation is used to evaluate classification agreement. The experiments mainly cover DeepSeek-R1 distilled models of varying sizes, involving mathematical and logical reasoning tasks. Experimental results show that the prevalence of the bypass effect depends on task type; simply scaling up model size does not substantially reduce the conditional bypass rate. The experiments reveal that models can derive the correct answer in the generation phase after the end of the thought block. A reduction in bypass rate is also observed in the RL post-trained models tested in the work.

Critical strengths
  1. The paper defines the bypass rate and conditional bypass rate, which respectively measure the fraction of bypass cases among all samples and the proportion of samples with uncorrected errors among all samples yielding correct final answers. The definitions distinguish whether errors within the chain of thought are corrected and whether the final answer is correct. The authors employ a keyword marker detector and an LLM judge to categorize different outcomes, and perform human verification on sampled cases to check the agreement of classification results. This provides relatively clear definitions and measurement pipelines for the bypass effect.

  2. The work considers both artificially injected errors and natural errors produced by the model itself. By auditing validated natural arithmetic errors, the paper examines whether the bypass phenomenon only exists in artificially constructed error settings, which supplies additional empirical evidence.

  3. The paper uses attention interventions and interruption tests during the answer generation of reasoning models, to investigate whether the correct answer in bypass cases is re-derived inside the answer block. Furthermore, the authors compare distilled models with their corresponding RL post-trained versions and observe a reduced bypass rate in the latter. These experiments analyze the behavioral characteristics of the bypass effect to some extent and explore potential mitigation strategies for this phenomenon.

Critical weaknesses
  1. Prior work has already investigated the phenomenon where reasoning models produce correct answers despite perturbed chain-of-thought reasoning or unfaithful rationales. Although this work defines clear metrics for the bypass effect, accounts for natural errors, conducts behavioral interventions and performs RL post-training experiments, it remains unclear what novel insights these incremental contributions deliver beyond prior work. The paper has not sufficiently illustrated how these additional analyses substantially change our understanding of this phenomenon, nor their concrete implications for the evaluation of reasoning models.

  2. Classification of bypass cases relies on judgments of error-correction behavior, and the keyword marker detector and LLM judge may yield divergent classification outcomes. While the authors evaluate classification agreement via human annotations, the audit of natural errors is largely restricted to arithmetic errors, and the associated human-validated sample size is relatively small. Moreover, the comparison between fixed-position injection and injection at natural-error positions uses different item subsets. Thus, the observed differences may be affected by sample composition and cannot be fully attributed to the error injection position itself.

  3. Comparisons between the distilled model and its RL post-trained counterpart show a substantial drop in bypass rate for the latter. However, current experiments do not conduct precise attribution for this observation. Differences in RL training data, prompt sets and post-training configurations can all influence the results. The paper notes that control experiments with continued SFT on identical prompts may help disentangle these competing explanations, yet such experiments are not provided. Therefore, the present results only support a correlation between RL post-training and reduced bypass rate, and the exact cause driving this change cannot be confirmed.

Questions for authors
  1. In Table 7, the comparison between fixed-position injection and injection at natural-error positions uses different subsets of problems. Could the authors explain the criteria for selecting these two groups of samples, and whether compositional differences between the samples affect the observed differences in "bypass" rates? Could paired analysis on a shared subset of problems be provided to better disentangle the effects of injection position versus sample composition?

  2. The results presented in the paper show that "bypass" rates drop substantially after post-training with RL on distilled model checkpoints. However, existing experiments do not fully disentangle the exact source of this change. Could the authors further elaborate on the concrete differences between the RL post-trained models and the original distilled models in terms of prompt sets, training data, and training configurations? To what extent can the observed reduction in "bypass" rate be attributed to RL post-training itself?

  3. The paper conducts experiments mainly on the DeepSeek-R1 distilled model family, with tasks largely limited to mathematical and arithmetic reasoning. Could the authors generalize the conclusions of this work to other model families and a broader range of reasoning tasks?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer cSMb (1/2)

Thank you for the detailed review. We answer in your order. Section and table numbers refer to the revised PDF.

W1. What is new beyond prior work, and what follows for evaluation

Table 1 (Sec. 2) places the paper against the six closest studies. Our contribution beyond them has three parts.

  1. Rates on the models' own errors, paired with injection on the same items. The closest study, Phan et al. (2026), reports a base rate over all unperturbed samples. Ours are shares of validated errors. On GSM8K an injected error is bypassed 14 to 22 points more often than the model's own error on the same item (Table 7).
  2. What governs the rate. One instruction sentence moves the 7B distill from 59.3% to 2.3% with the same weights. Forcing the first word of the think block moves the rate in both directions (Table 3). Lin et al. (2025) elicit short think blocks this way. What a short think block does to an uncorrected error is new.
  3. Transfer. Supervised training on another model's traces carries its bypass rate over, in both directions (Figure 2b).

For evaluation (Sec. 5, Sec. 4.6):

  • A faithfulness number holds only for the prompt format it was measured with. Here 59.3% and 2.3% come from the same weights on the same items.
  • Injected-error rates should not be read as rates on a model's own errors.
  • A low rate measured on an RL-trained model cannot be credited to RL from that comparison alone. Supervised training on its traces reproduces the rate.

W2. Detectors, scope and sample size

  • Detectors may diverge. Table 2 now reports the GSM8K model comparison under three measures: the marker, the scoped judge, and a count that reads only numbers and needs no judge. The count gives the same class as the judge on 92.8% of 21,909 traces (Table 9). A second judge, Claude Sonnet 5.5, agrees with GPT-4o on 97.7% of 600 traces (kappa = 0.96). The cells added in the revision are reported under the count and the marker (Table 10), not under the judge. The main effects of the prompt, the forced first word and the training traces hold under both measures. The dependence on the form of the think block holds under the judge as well: for the six distills, 69.9% in short-form traces against 2.8% in long-form ones (Table 9).
  • Arithmetic only. Correct. The audit catches only errors that an execution verifier can check. The paper states this as scope (Sec. 3.3, Limitations) and claims nothing about errors of strategy or logic.
  • Human sample. You are right that it is small. The 492 blind labels are spread over several audits: 162 injection traces for the judge against the marker, and 330 rows in the natural-error pipeline. The audit of corrections on natural errors has 21 validated rows. All labels come from one annotator, who is not independent of the authors (Limitations). The automated checks above support the instruments at scale. They do not replace a larger human audit.
  • Different item subsets. You are right. See Q1 in part 2.

W3. RL attribution

We agree: the evidence supports a correlation only. The submission called the result a comparison of checkpoints, not a property of the RL objective, but its abstract and section heading read more strongly. Both are reworded, and Limitations calls the comparisons observational.

We added a control with supervised fine-tuning on fixed prompts (Sec. 4.6, Table 12). Every arm uses the same recipe, the same 1,800 GSM8K training problems and 339 updates.

  • Continued SFT of the 7B distill on its own traces: 59.3% to 61.3% (+2.0 points [-2.3, +6.7]).
  • SFT on the traces of the RL post-trained model: 17.3%. That model itself gives 18.0% on the same items.
  • The other direction: a student at 0.3% reaches 58.0% after SFT on the 7B distill's traces.

So supervised training on traces is sufficient to move the rate either way. This does not isolate what the RL objective contributes, and the paper says so.

AuthorsOpenReview ↗

Reply to Reviewer cSMb (2/2)

Q1. Paired analysis on a shared subset

Done (Sec. 4.5, Table 7). Every row uses validated errors and is paired by item. The rows for the 7B distill on GSM8K (marker; bypass rate on the model's own errors and on injected errors, same items):

Injected atItemsOwn errorsInjected errorsDifference (points)
Token 40548.3%22.7%+14.4 [+5.9, +23.6]
Position of a validated own error285.4%26.8%+21.4 [+8.0, +35.7]
Token 40, the same 28 items285.4%27.7%+22.3 [+10.1, +36.3]

Redoing Table 7, we found two faults in the row we had reported for the position-matched arm. Its natural rate included error flags that had not passed validation. And its injection positions came from an older version of the verifier, so only 28 of the 52 validated items were injected at a validated error. We therefore withdraw the claim that matching the position of the natural error closes the gap. Appendix E gives the details.

Position still matters for injected errors on their own. Over all 450 items of that arm, moving the injection to token 40 raises bypass by 4.9 points [+1.8, +8.1].

Q2. How the RL post-trained models differ from the distills, and how much is due to RL

  • Training. Skywork-OR1-7B starts from the DeepSeek-R1-Distill-Qwen-7B checkpoint and is trained with rule-based RL on about 105K math and 14K code prompts (Sec. 3.5). Appendix N gives the stated recipe of every added model.
  • Behavior. The difference is in how often the short form of think block is used. Skywork-OR1-7B writes one in 13.0% of its GSM8K traces, the distill in 81.2%. Within short-form traces it bypasses as often as the distill (69.0% against 62.4%). Forcing its first word to "First," raises its bypass from 18.0% to 70.7% (Table 3).
  • Other post-trained models show the same pattern (Figure 2a, Table 11). The two made by further SFT stay at their parent's rate (+0.2 and -0.9 points). The three whose post-training includes RL are lower (17.4%, 37.8% and 21.2%), each in step with its share of short-form traces.
  • How much is due to RL itself. We cannot say, and the paper does not claim it. What the control shows is that supervised training on the RL model's traces is sufficient to produce its rate (W3).

Q3. Other model families and tasks

  • Families. We added ten public models (Sec. 3.5, Table 11). The five from other pipelines bypass at most 0.4%. They include two SFT students of DeepSeek-R1 made by other groups and two Qwen3 models. So the finding does not generalize to distillation at large, and the paper no longer says it does. It holds along one lineage: the six distills and the six models post-trained from them bypass 49.9 to 69.0% within short-form traces.
  • Tasks. On AIME and BBH nearly every think block is long-form, and bypass is low (about 2 to 10%). A bare-prompt rerun on AIME gives 1 bypass trace of 60. This agrees with the form account. We also say that the task comparison cannot test it separately, because BBH was not rerun with a bare prompt.

We hope this answers your concerns, on novelty in particular. If it does, we would be grateful if you would consider raising your score. If something is still open, please tell us what it is, and we will address it in the remaining days of the discussion.

Dear Reviewer cSMb, a short reminder that the discussion closes for authors on 18 November. If something is still open after our response, on novelty or on the paired analysis in particular, please tell us and we will address it before then. Thank you for your time.

Reviewer_wpL4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a phenomenon called “bypass,” in which reasoning models produce correct final answers despite leaving errors uncorrected in their chains of thought. Through experiments involving injected errors and naturally occurring errors, the authors find that distilled reasoning models frequently exhibit bypass on arithmetic tasks, particularly when the correct answer can be independently recomputed, while bypass is much rarer on more complex mathematical and state-tracking tasks. Comparisons between distilled and reinforcement-learning-trained models suggest that RL substantially reduces bypass, with RL post-training reducing the injected-error bypass rate of a 7B model from 61.5% to 14.4% on GSM8K. Additional interventions suggest that bypass often occurs because models independently re-derive the correct answer after the reasoning block, rather than explicitly correcting their earlier mistakes.

Critical strengths
  • The evaluation methodology employed in the paper is rigorous, using approximately 30,000 sampled rollouts, an execution-based arithmetic verifier, two correction detectors, and blind human annotations. The authors also explicitly investigate detector errors and show that correction rates can change substantially depending on the evaluation method
  • Authors investigate several alternative explanations, including memorization, model scale, injection position, and generation truncation. Their synthetic arithmetic benchmark and explicit analysis of incomplete traces make the experimental evidence more convincing
  • The mechanistic investigations are interesting
Critical weaknesses
  • Although the same-checkpoint comparison provides strong evidence that RL-based post-training reduces bypass, it does not isolate the contribution of the RL objective from additional training on curated reasoning problems. A compute- and data-matched continued-SFT baseline would be good to establish whether RL specifically improves chain-of-thought faithfulness beyond what could be achieved through further supervised training
Questions for authors

Would it be possible to have a better compute/data matched SFT baseline for the RL bypass reduction result?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer wpL4

Thank you for the careful reading and for your support of the paper.

Your request: a compute- and data-matched continued-SFT baseline

We ran a fine-tuning control (Sec. 4.6, Figure 2b, Table 12). It answers part of your question, and we want to be exact about which part.

What we ran. Two students: DeepSeek-R1-Distill-Qwen-7B, and OpenR1-Distill-7B, an SFT student of the same teacher with 0.3% bypass on this test set. Training traces from three sources: the 7B distill, Skywork-OR1-7B (the RL post-trained model) and OpenR1-Distill-7B. Every arm uses the same LoRA recipe, the same 1,800 GSM8K training problems and 339 updates. The test is the injection protocol on 300 GSM8K test items.

StudentTrained on traces ofBypassChange from untrained
Distill-Qwen-7B(untrained)59.3%
itself61.3%+2.0 [-2.3, +6.7]
itself, tokens matched to the longest arm60.3%+1.0 [-3.7, +6.0]
Skywork-OR1-7B17.3%-42.0 [-48.3, -35.0]
OpenR1-Distill-7B0.0%-59.3 [-65.0, -53.7]
OpenR1-Distill-7B(untrained)0.3%
Distill-Qwen-7B58.0%+57.7 [+52.0, +63.3]
itself0.7%+0.3 [-0.7, +1.7]

What it shows.

  • Further SFT on the model's own traces does not lower bypass in these arms, also with the training tokens matched.
  • Supervised training on the RL model's traces is enough to reach the RL model's rate: 17.3%, against 18.0% for Skywork-OR1-7B itself on the same items (difference -0.7 points [-4.3, +3.3]).
  • It works in the other direction too: 0.3% to 58.0%.
  • The marker detector shows the same main effects (Table 10).

What it does not show. We did not rerun the RL stage of Skywork-OR1 against SFT on its own prompt set at matched compute. So the contribution of the RL objective itself is still not isolated, and the paper says so (end of Sec. 4.6, Limitations). The submission already stated this result as a comparison of checkpoints, not a property of the RL objective. In the revision the abstract and the section heading no longer say that RL post-training reduces bypass. The same-checkpoint contrast itself stands: 61.5% against 14.4% of injected errors under the judge, and 25.8% against 12.6% of the models' own errors.

One change you should know about

Your summary mentions recomputability. We no longer make that claim. The benchmarks also differed in prompt format, and a new control shows that the format alone moves the rate: one instruction sentence takes the 7B distill from 59.3% to 2.3% on the same items (Sec. 4.3, Table 3). The AIME and BBH prompts carried an instruction and the GSM8K prompt did not. The benchmark comparison is now reported as confounded.

We think the paper is more useful this way. What governs bypass is now tested by intervention (the prompt, the first word of the think block, the training traces) and no longer inferred from a comparison of benchmarks.

Thank you again for your support. Does the added control, together with the plainer statement of what this comparison can show, resolve your concern? If it does, we would be grateful if you would consider raising your score.

Dear Reviewer wpL4, thank you again for your support. The discussion closes for authors on 18 November. If you have looked at the added fine-tuning control, we would be glad to hear whether it resolves your concern, and to answer any further question before then.

Dear authors, thank you for the update, it resolves most of my concerns. I maintain my positive score.

Dear Reviewer wpL4, thank you for reading the revision and for your continued support. We are glad the added control was useful, and we appreciate the time you gave to our paper.

Reviewer_QbbLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a regime called "bypass" to study faithfulness in chain-of-thought reasonings in models constructed through distillation. In particular, they look at cases where a model produces a correct final answer even when an error occurred in its chain-of-thought. They measure this by injecting errors and then allowing the model to continue to produce an answer. They evaluate on several math reasoning benchmarks.

Critical strengths

Figure 1 is very illustrative for understanding the methodology.

The findings are interesting. It's particularly interesting that RL models are more likely to fix the injected errors than distilled errors. I assume this is because their on-policy learning process encourages the model to correct their mistakes.

Critical weaknesses

The writing is hard to parse. It is often convoluted likely due to the use of LLM for writing. An example is below:

“On identical items (Distill-14B, N=300) the same trap delivered as a user instruction doubles the wrong rate (25.7% to 49.0%) because the model complies with the instructed error, and delivered as an assistant prefix before any self-generated text it is corrected far more often (Appendix B).”

The definition of bypass is ambiguous. Bypass that occurs naturally from the reasoning model's CoT can be hard to measure. Say there are two strategies A and B to derive the same answer. The model tried A first, made a mistake, gave up and tried B instead and got the right answer. The model never corrected its error in strategy A - is this bypass? How about when the model backtracks and yet never corrects the problematic step?

The abstract makes an incorrect assumption in the first sentence. It says: "Reasoning models are increasingly built by distillation, that is supervised finetuning on chains of thought sampled from an RL-trained teacher." This is not true. On-policy distillation is trained on generation from the student, not the teacher. Reasoning models may also adopt mixed RL

It is questionable how reliable the correction detectors are. Both the heuristic based marker and GPT-4o judge may not able to detect recovery behavior. Most notably, GPT-4o is not stronger than the distilled Qwen on math puzzles like AIME (Qwen Distills: 50%; GPT-4o: 10%) so you are using much weaker models to supervise stronger models. This might also explain different bypass rates between RLed models and Distills: maybe GPT-4o is just better at detecting errors from QwQ because their CoTs are more readable than DeepSeek R1 distills.

Why not study the behavior of DeepSeek R1 which is the teacher model for the distilled model, and see what behaviors have been transferred from the teacher to the student and what has not?

The natural-error analysis covers only a narrow class of reasoning errors. Your method only catches certain arithmetic error (those involving equations), but does not catch other type of reasoning errors, such as geometry, algebra, or an error expressed only in natural language reasoning. It is therefore unclear whether this part of analysis can lead to any conclusions.

Questions for authors

Why do you use a greedy approach for experiments in the injected error bypass study, instead of using the developer's recommended value or temp and top p aligned with the model's RL training?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer QbbL (1/2)

Thank you for the review. Your points were specific, and each of them led to a change in the paper. We go through them in your order. Section and table numbers refer to the revised PDF.

W1. Writing

You were right. We rewrote the abstract and the list of contributions, and we split the long sentences throughout the main text. The sentence you quoted now reads (Sec. 3.2):

On the same 300 items (Distill-14B, Appendix B) the mid-thought trap ends in a wrong answer in 25.7% of traces. Given as a user instruction, the same trap does so in 49.0%, because the model follows the instruction.

Figure 1, which you found illustrative, is redrawn at full width, with larger text and no overlapping boxes.

W2. Definition of bypass

Sec. 3.1 now defines the classes by two yes-or-no questions.

  • Q1: does the think text after the error e dispute e, or recompute what e got wrong?
  • Q2: does the answer block state the true answer?

Corrected is yes and yes. Bypass is no and yes. Wrong is Q2 no.

Your two cases are now in the text. A chain that drops approach A, or backtracks, and solves the problem another way without returning to e is corrected under the lenient construct, where any correction attempt counts. It is bypass under the strict construct, where the correction must target e. Headline judge rates are lenient, so such chains are not counted as bypass. The two constructs differ by at most 2.3 points for the distills (Table 2).

In these models bypass is usually not a change of strategy. The think block is a short plan with the wrong step left standing, and the arithmetic is done again in the answer block. Figure 3 shows one such trace verbatim.

W3. First sentence of the abstract

You are right, and the sentence is gone. The abstract now opens: "Many open-weight reasoning models are trained by supervised fine-tuning on chains of thought sampled from a stronger reasoning model. We study the DeepSeek-R1-Distill family, which is built this way ..." Sec. 1 says that on-policy distillation and logit matching are outside our scope.

The conclusion is narrower too. Two other SFT students of the same teacher show very low rates (0.4% and 0.1% on GSM8K). The paper therefore ties the behavior to this family and its descendants, "not to distillation in general".

W4. Detectors

  • The judge does not have to solve the problem. It is given the injected step and the true answer. It then reads the think text after the step and says whether that text disputes or recomputes the step (Sec. 3.4). Its own strength on AIME is not used.
  • A measure with no judge. A trace is counted as bypass if the think block closed, the answer block states the true number, and that number appears nowhere in the think text after the injected step (Sec. 3.4, Appendix M). It reads no wording, so it does not depend on how well a judge reads a model's chains. It gives 38.7 to 62.7% for the six distills and 0 of 1,319 traces for QwQ-32B (Table 2).
  • Agreement. The count and the judge give the same class on 92.8% of 21,909 traces (Table 9). The count is not perfect in either direction, and Appendix M says where it errs.
  • A second judge. Claude Sonnet 5.5 agrees with GPT-4o on 97.7% of 600 traces (kappa = 0.96).

On recovery that a detector might miss: the lenient judge counts any correction attempt, targeted or not, and the count does not call a trace bypass if the true value appears anywhere in the think text after the error. Against blind human labels on 162 audit traces the judge reaches kappa = 0.824.

AuthorsOpenReview ↗

Reply to Reviewer QbbL (2/2)

W5. The teacher

We ran DeepSeek-R1 under the same injection protocol (Sec. 4.1, Appendix O). It bypasses 0 of 100 injected errors (95% interval 0.0 to 3.7%). Of its traces, 98 end correct, and each of them states the true value in the think block after the injected step. So the students' behavior is not something the teacher shows on these items.

Two limits are stated in the paper. The run has 100 items. And it went through a hosted endpoint listed as an 8-bit quantization, because the model does not fit our hardware.

W6. Natural errors are arithmetic only

Yes, and the paper now states this as scope (Sec. 3.3, Limitations). The audit supports two conclusions.

  • Bypass is not an artifact of injection. On its own validated arithmetic errors the 7B distill bypasses 25.8% under the marker and 35.9% under the judge (n = 124).
  • Injection overstates it. On the same 54 GSM8K items, an injected error is bypassed in 22.7% of traces and the model's own error in 8.3% (+14.4 points [+5.9, +23.6], Table 7). The paper therefore says that injected-error rates should not be read as rates on a model's own errors (Sec. 5).

We make no claim about errors of strategy or logic.

Q. Why greedy decoding

Our injection experiments used one greedy continuation per item. You are right that the recommended settings belong in the paper, and they are in it now.

  • At T = 0.6 and top-p 0.95, with three seeds on 300 items, the 32B distill bypasses 60.4% [55.8, 64.8] of injected errors and QwQ-32B 0 of 900 traces (Sec. 4.1).
  • The natural-error rollouts were already sampled at these settings.

The model cards also recommend an instruction sentence for math problems. We tested it, and it turned out to be the largest effect in the paper. It takes the 7B distill from 59.3% to 2.3% on the same items (Table 3), and to 4.8% [3.2, 6.4] with the recommended sampling as well. Sec. 4.3 is built around this result.

Closing

We have responded to every point you raised: with new experiments (detectors, teacher, decoding), with rewritten text (writing, definition, abstract) and with an explicit statement of scope (natural errors). If this addresses your concerns, we would be grateful if you would reconsider your rating. If a point is still open, please tell us which one, and we will respond right away.

Dear Reviewer QbbL, a short reminder that the discussion closes for authors on 18 November. If any of your points is still open after our response, we would be glad to address it before then. Thank you for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision (to all reviewers and the AC)

Thank you for the three reviews. In the revised PDF, new and changed text is in blue, and the main text is 10 pages. The blue text will be made black again in the camera ready version.

The revision keeps the question, the definition, the instruments and the main results of the submission. It adds controls on those results, most of them requested in the reviews. Two of our conclusions did not hold up, and we withdraw them below.

Kept from the submission

  • The definition of bypass, the injection protocol and the two detectors (Sec. 3).
  • The GSM8K results under the marker and the judge (Table 2). No submitted number in it has changed. The count is a new column.
  • The audit of the models' own errors (25.8% bypass under the marker, 35.9% under the judge, n = 124), and that injection overstates bypass by 14 points on the same items.
  • The probes that place the answer after the think block (Sec. 4.4).
  • The same-checkpoint comparison (61.5% against 14.4% under the judge). As in the submission, it is stated as a comparison of checkpoints, not a property of the RL objective. The abstract and section heading no longer say that RL post-training reduces bypass.

What the added controls show

  1. Bypass holds under a measure that needs no judge (QbbL, cSMb). A count that reads only numbers gives 38.7 to 62.7% for the six DeepSeek-R1 distills on GSM8K. The base-matched RL model QwQ-32B bypasses 0 of 1,319 injected errors, and the DeepSeek-R1 teacher 0 of 100.
  2. The rate depends on the prompt (QbbL). The sentence that the model cards recommend for math takes the 7B distill from 59.3% to 2.3% on the same items (-57.0 points [-62.3, -51.0]).
  3. It follows the form of the think block. With a bare question these models mostly write a short plan and do the arithmetic in the answer. They bypass 49.9 to 68.6% of injected errors in such traces and 0.0 to 9.8% in long ones. Forcing the first word of the think block moves the rate both ways.
  4. It transfers with training traces (wpL4, cSMb). A student at 0.3% reaches 58.0% after supervised training on the 7B distill's traces. The distill falls to 17.3% after training on the traces of the RL post-trained model.
  5. It is not distillation in general (cSMb). Five models from other pipelines, two of them SFT students of the same teacher, bypass at most 0.4%.

What we withdrew or narrowed

  • Recomputability, withdrawn. We no longer claim that bypass tracks whether the answer can be recomputed from the prompt. The benchmarks also differed in prompt format, and point 2 shows that the format alone moves the rate (Sec. 4.3).
  • Injection position, withdrawn. We withdraw the claim that injecting at the position of the natural error closes the gap between injected and natural errors (cSMb). The earlier row mixed validated and unvalidated error flags, and its positions came from an older version of the verifier. Table 7 now uses validated errors in every row and pairs every comparison by item.
  • Scope, narrowed. The opening sentence on distillation is gone (QbbL). The claims concern the DeepSeek-R1-Distill family and models trained from it or on its traces.

Where each concern is answered

ConcernFromWhere
WritingQbbLAbstract, contributions, Figure 1; long sentences split
Definition of bypassQbbLSec. 3.1
First sentence on distillationQbbLAbstract, Sec. 1
Detector reliabilityQbbL, cSMbSec. 3.4, App. M (Tables 9, 10)
Teacher not studiedQbbLSec. 4.1, App. O
Greedy decodingQbbLSec. 4.1, Table 3
Natural errors are arithmetic onlyQbbL, cSMbSec. 3.3, 4.5, Limitations
Matched SFT baseline, RL attributionwpL4, cSMbSec. 4.6, Fig. 2b, Table 12
Paired analysis on shared itemscSMbSec. 4.5, Table 7, App. E
Novelty, implications for evaluationcSMbSec. 2 (Table 1), Sec. 5
Other model families and taskscSMbSec. 3.5, 4.3, 4.6, Table 11

Dear Reviewers

The author-reviewer discussion period will close soon. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.