acceptodds
Under review as a conference paper at ICLR 2027

SelfCAD: Protecting Your Efficient Reasoning Capabilities via Self-Cautious Insertion

Abstract

Large reasoning models (LRMs) are valued for their accuracy and transparency, but publishing their reasoning traces also enables unauthorized distillation, raising copyright and security risks. Existing defenses either detect misuse after the fact or obscure reasoning traces, making them impractical when transparency is required. We propose SelfCAD (Self-Cautious Anti-Distillation), a lightweight post-processing defense that preserves the original reasoning content while hindering the performance and reasoning efficiency of distilled models. Our analysis shows that self-cautious sentences strongly affect both accuracy and efficiency: inserting such sentences can lengthen chain-of-thought reasoning while reducing accuracy in distilled models. SelfCAD therefore manipulates these components after traces are generated, requiring minimal extra cost and no additional training. Experiments on Llama and Qwen show that SelfCAD consistently increases student inference cost and reduces distilled-model accuracy. On Qwen with GSM8K, it increases response length by and reduces accuracy by 23 percentage points.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
distillation, chain-of-thought, security, privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gn4vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SelfCAD, a training-free, post-hoc defense against unauthorized distillation of large reasoning models. The authors find that self-cautious sentences mainly lengthen reasoning traces without improving accuracy. SelfCAD therefore inserts such sentences, drawn from the protected model’s own outputs, after every reasoning step of the released trace. This degrades students trained on the processed traces while keeping the original reasoning visible. The authors distill GPT-OSS-120B and R1 into 1B–7B Llama and Qwen students and report accuracy drops of 5–23 points and outputs 1.3–9.1× longer on four math benchmarks.

Critical strengths
  1. The motivation for the black-box API distillation setting is clear: how to release useful reasoning traces while protecting the underlying model’s capabilities. Protecting outputs without modifying the teacher model’s parameters is an attractive design direction.

  2. The reasoning-trajectory component removal experiments (removing reasoning, self-cautious and conclusion sentences) have some exploratory value. Instead of reporting only final benchmark scores, they offer more mechanism-level insight.

  3. The human readability study and the use of multiple LLM judges show that the paper cares about the experience of benign users who rely on the protected outputs.

Critical weaknesses
  1. The most critical adaptive attack is not evaluated. The inserted sentences are a uniform, structured artifact that the paper’s own “Remove SC” pipeline in Sec. 3.2 could strip. Table 2 shows this keeps accuracy nearly unchanged at 37% versus 39% with 0.4 times the length, so SelfCAD may be easily bypassed.

  2. The claim that SelfCAD makes distilled models “less confident even on correct steps” is not supported by any error analysis of student outputs, such as how often correct intermediate results are overturned.

  3. The cost to benign users is not quantified. The paper does not report how many tokens the processed reasoning traces add, or the resulting latency overhead.

  4. The definition of self-cautious sentences is ambiguous, and what is analyzed does not match what is inserted.

  5. The protection claims go beyond the validated results. Underperforming distillation with original responses does not mean the protected outputs are no longer worth collecting. Results for the student before distillation are needed to quantify the benefit that remains.

Questions for authors
  1. After SelfCAD processing, how many tokens on average are added to the reasoning traces returned to users, and how much latency overhead does this introduce?

  2. Could you report the truncation rate, the accuracy on non-truncated generations, and how accuracy and length change under different generation limits?

  3. Have you evaluated an inference-time bypass in which the attacker uses logit bias to suppress reflection tokens such as “Wait” and “Hmm”?

  4. See weakness.

4 comments

W1: “The most critical adaptive attack is not evaluated. ... the paper’s own ‘Remove SC’ pipeline ... could strip ... . Table 2 shows this keeps accuracy nearly unchanged at 37% versus 39% with 0.4 times the length ...”

We thank the reviewer for raising this attack. The 37% versus 39% result is an analysis of removing complete self-cautious segments from ordinary teacher traces using a strong external decomposer, GPT-4.1. It is not an experiment that identifies and strips SelfCAD insertions from protected traces. Such decomposition requires a highly capable model to process every long trace, incurring substantial additional inference cost. A cleaner such as GPT-5.4 would assume access to capabilities beyond those of both the student and the protected teacher in our setting. We use strong-model decomposition as an ablation tool; access to such a model is not assumed in our practical attacker setting. For an adaptive attack under this setting, a rewriter comparable in scale to the student is a relevant test. We therefore conducted additional experiments with a student-scale model.

We tested Qwen2.5-1.5B-Instruct on the same 4,000 Bespoke traces (Appendix B.1, Table 13). Only 1,084 outputs (27.10%) were parseable, only 13 (0.325%) preserved the complete original content after ignoring layout whitespace, and none reconstructed it byte-exactly. On the matched 1,084-example training subset, Original / Remove-SC GSM8K accuracy was 20.3942% / 16.9067%, and mean output length was 7,072.09 / 8,175.90 tokens. The practical small-model pipeline introduces substantial omissions and rewriting rather than reproducing the strong decomposer's clean intervention.

We also attempted targeted removal with the instruction to remove redundant doubt and re-check requests while preserving calculations, logical steps, substantive checks, corrections, and the final answer. Qwen2.5-1.5B-Instruct nevertheless compressed approximately 5,000-token traces to approximately 500 tokens, losing substantive content.

W2: “The claim that SelfCAD makes distilled models ‘less confident even on correct steps’ is not supported by any error analysis ... such as how often correct intermediate results are overturned.”

Thank you for requesting direct behavioral evidence. We added two experiments with Original- and SelfCAD-distilled Qwen2.5-1.5B-Instruct on GSM8K, using identical chat templates and greedy decoding. These are reported in Section 5.3 (Re-checking and Reversal of Correct Results), Table 8, with evaluation and annotation details in Appendix A.6 (Behavioral Evaluation Details).

In natural trajectories, both models solve the same 100 randomly sampled questions with a maximum of 10,000 new tokens. We identify the first correct candidate and examine subsequent checking and reversal. In shared-prefix continuation, both models receive exactly the same correct, non-final intermediate prefix, represented by identical input token IDs. Ninety prefixes pass arithmetic and semantic validation, and we score behavior within the next 256 tokens.

Experiment and behaviorOriginalSelfCAD
Natural: re-check after reaching a correct candidate48/57 (84.21%)62/65 (95.38%)
Natural: reverse a correct candidate1/57 (1.75%)11/65 (16.92%)
Shared prefix: re-check within 256 tokens2/90 (2.22%)23/90 (25.56%)
Shared prefix: reversal within 256 tokens1/90 (1.11%)1/90 (1.11%)

The shared-prefix test is particularly informative: given the same correct reasoning state and the same observation window, SelfCAD increases re-checking by 23.34 percentage points. The natural trajectories additionally show a 15.17-point increase in reversal among trajectories reaching a correct candidate. The unchanged short-window reversal rate shows that reversal depends on the surrounding trajectory; we do not claim that every extra check overturns a correct step.

For annotation, anonymized LLM agents use the question, reference answer, and output context, and every positive event requires a locatable quotation. Numerical matching only proposes candidates. Independent review covers all 12 natural-trajectory reversals plus 20 random cases, and both shared-prefix reversals plus 18 random cases; reversal labels remain unchanged. These experiments provide observable evidence of excessive re-checking and context-dependent reversal. The increased re-checking after the same correct intermediate steps indicates that SelfCAD reduces confidence in correct results.

W3: “The cost to benign users is not quantified. ... how many tokens the processed reasoning traces add, or the resulting latency overhead.”

Q1: “After SelfCAD processing, how many tokens on average are added to the reasoning traces returned to users, and how much latency overhead does this introduce?”

We emphasize that SelfCAD requires only CPU text processing, with no additional teacher-model inference to generate the inserted cues. The manuscript reports processing the full distillation dataset within several minutes on a single CPU. This avoids additional teacher-decoding cost, although it does not imply zero processing latency or zero overhead from the longer returned text; per-response latency has not been separately measured. We counted the original and protected assistant text for all responses with add_special_tokens=False, defining inserted tokens as the difference (Appendix A.7, Token Overhead to Users; Table 12):

TokenizerOriginal tokensSelfCAD tokensAdded tokensMean added per responseAdded / OriginalAdded / SelfCAD
Qwen2.5-1.5B13,068,09517,068,3504,000,255500.930.61%23.44%
Llama3.212,440,89716,292,9323,852,035482.330.96%23.64%

The protected traces are therefore approximately 1.31 times the original length. These inserted words are produced by CPU text processing, not by additional teacher decoding; original derivations and final answers are retained. Thus, the provider avoids an additional teacher-generation pass, while the user receives approximately 483–501 additional tokens per response on this corpus. This cost should not be confused with the 1.3–9.1-fold student inference length inflation in the main experiments.

W4: “The definition of self-cautious sentences is ambiguous, and what is analyzed does not match what is inserted.”

We thank the reviewer for identifying a distinction that needed to be explicit. A complete self-cautious unit has two components:

  • A self-cautious cue, xsccuex_{sc}^{\mathrm{cue}}, expressing doubt or initiating a check, such as “Wait, let's re-examine...” or “Let me verify:”.
  • The subsequent verification reasoning, xscverx_{sc}^{\mathrm{ver}}, such as the recalculation “pi is approximately 3.14, so 25 × 3.14 is approximately 78.5.”

The full segment is xsc=[xsccue,xscver]x_{sc}=[x_{sc}^{\mathrm{cue}},x_{sc}^{\mathrm{ver}}]. The revised definition distinguishes the two throughout the analysis and algorithm. Table 1 measures termination immediately after a cue; the reported value of 0.13 is consistent with continued generation after that trigger. In contrast, the Remove-SC analysis in Table 2 and the classification appendix remove complete cue–verification segments, explaining why removal can substantially reduce output length to 0.4 times the original.

SelfCAD deliberately inserts cues rather than synthesizing new problem-specific calculations. Generating new verification steps would require additional model inference and could introduce mathematical errors. In the protected training trace, the cue is placed next to preserved reasoning, using the adjacent step as context. A cue can incorporate a variable assignment or numerical value from the preceding original line, for example, “Wait, did I compute {var = val} correctly?” or “Hmm, let me verify {val}.” Before the first conclusion, a confirmation closer such as “After carefully checking, I'm confident the answer is correct” can form a check-and-confirm unit while preserving the final answer.

The implemented pool combines natural teacher expressions, fixed manual cues, and anchored templates. Natural teacher lines are selected using caution-related keywords, restricted to 15–200 characters, and frequency-weighted with at most five copies per sentence. In the controlled source ablation, the pool contains 45,531 extracted instances, 42,264 distinct sentences, and 44,273 sampling entries after capping. This construction is more varied and context-sensitive than a single repeated trigger.

At student inference time, the trained student generates its own continuation after a cue, including further checks. Thus, the analysis and defense address complementary parts of the same behavior: the analysis measures the role and cost of the full segment, while the defense changes the frequency of its initiating cues without fabricating new mathematical content. The theoretical model describes a shift toward the cautious continuation pattern under its assumptions; the new behavioral tests provide empirical evidence of increased re-checking. Neither the termination statistic nor the theory alone proves that every cue causes a reversal or an accuracy loss.

W5: “Underperforming distillation with original responses does not mean the protected outputs are no longer worth collecting. Results for the student before distillation are needed to quantify the benefit that remains.”

Our evaluation targets the chain-of-thought reasoning capability acquired through distillation. The students are non-reasoning instruction checkpoints rather than specialized reasoning models, so there is no corresponding distilled chain-of-thought capability to evaluate before distillation. We therefore compare students trained on the original and protected teacher responses under the same protocol.

On GSM8K, accuracy falls from 31% to 23%, 55% to 32%, and 60% to 38% for Llama3.2-1B-Instruct, Qwen2.5-1.5B-Instruct, and Llama3.2-3B-Instruct, respectively, while generation becomes longer. These results show that protection degrades the effectiveness and efficiency of acquiring reasoning from these teacher traces.

Q2: “Could you report the truncation rate, the accuracy on non-truncated generations, and how accuracy and length change under different generation limits?”

We added a matched-cap sweep for Original- and SelfCAD-distilled Qwen2.5-1.5B-Instruct on GSM8K (Section 5.4, Evaluation under Matched Generation Budgets; Table 9). Entries are Original / SelfCAD.

Maximum new tokensAccuracy (%)Actual mean generated tokensToken-limit hit rate (%)
2,04854 / 31560 / 1,90017 / 91
4,09655 / 31600 / 3,8002 / 83
8,19255 / 31690 / 6,2001 / 31
10,00055 / 32710 / 6,4001 / 6

The accuracy deficit persists at every tested cap. In particular, increasing the cap to 10,000 reduces SelfCAD's limit-hit rate to 6%, yet accuracy remains 32%, compared with Original's 55%. This supports the conclusion that increasing the available generation budget does not remove the observed protection. Matched caps do not mean matched actual computation, which is why we report actual mean lengths as well.

Q3: “Have you evaluated an inference-time bypass in which the attacker uses logit bias to suppress reflection tokens such as ‘Wait’ and ‘Hmm’?”

We evaluated this inference-time bypass on GSM8K by applying a logit bias of -100 to "Wait" and "Hmm" in SelfCAD-distilled students. The intervention changes decoding without rewriting training data or retraining. This experiment is described in Section 5.5, with results for all three students in Appendix B.5 (Inference-Time Suppression of Reflection Markers), Table 17. Each cell reports accuracy (%) / mean generated tokens.

StudentOriginalSelfCADSelfCAD + logit bias
Llama3.2-1B31 / 60023 / 1,30019 / 350
Qwen2.5-1.5B55 / 70032 / 6,40035 / 6,200
Llama3.2-3B60 / 40038 / 65024 / 800

Neither targeted marker appears in the biased outputs, but the students continue to express doubt and re-check using other wording. Qwen gains three accuracy points yet remains 20 points below Original, while accuracy decreases for both Llama students. Suppressing these two markers therefore does not consistently restore performance, although this experiment does not rule out other decoding interventions.

I thank the authors for their response. I will maintain my original score.

Reviewer_vZySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SelfCAD, a training-free defense against distillation from released reasoning traces. It inserts self-cautious sentences drawn from the teacher's own outputs while preserving the original reasoning steps and final answers, aiming to induce excessive checking in distilled students. Experiments across Llama and Qwen students using GPT-OSS-generated and Bespoke reasoning data report lower accuracy and longer responses after distillation from protected traces. The paper also evaluates readability and several adaptive strategies, presenting SelfCAD as a lightweight approach to protecting efficient reasoning capabilities.

Critical strengths
  1. A simple but effective method. SelfCAD uses lightweight post-processing without updating the teacher or accessing a proxy student's gradients. The experiments show that this simple intervention reduces student accuracy and increases generation length across different models and reasoning-data sources.

  2. The paper examines relevant trade-offs rather than only reporting an attack on naive distillation. Sentence-removal experiments, a human readability study, and evaluations of inference-time shortening and preprocessing attacks provide useful initial controls. Preserving the original answer text by construction is also a practical advantage of post-processing, although it does not establish all aspects of user-facing utility.

Critical weaknesses

W1. Resistance to targeted cleaning remains untested. Algorithm 1 preserves the original reasoning and adds self-cautious sentences. An attacker could therefore ask GPT-5.4 to rewrite protected traces: "Remove redundant expressions of doubt or requests to re-check, such as 'Wait, let me check again,' while preserving calculations, logical steps, substantive checks, corrections, and the final answer." This requires no access to the original trace or insertion labels. Appendix A.1 and Table 2 motivate this test through sentence removal on ordinary data, but do not establish that it would bypass SelfCAD. Section 5.3 does not separately evaluate this attack. Applying the same cleaner to original and protected data would test whether SelfCAD retains an advantage.

W2. Preserved readability does not rule out detectable modification. The human study provides useful evidence that SelfCAD preserves readability and often goes unnoticed. However, someone deliberately inspecting the collected traces may recognize repetitive or contextually unnecessary self-cautious insertions and infer that the dataset has been modified. Whether informed evaluators can detect these patterns through targeted inspection remains unclear. Appendix A.3's semantic-equivalence judgment explicitly disregards expression differences and cannot resolve this question. A targeted human detection study would better support the stealth claim.

W3. Self-cautious content is not isolated from length and truncation effects. Added sentences may alter effective training supervision under sequence limits or loss normalization. Section 4.3 also acknowledges truncation at the 10,000-token generation cap, so accuracy drops may partly reflect unfinished answers. A length-matched neutral-insertion control, training and generation truncation rates, and separate results for unfinished versus completed but incorrect responses would help distinguish these explanations. Their contributions remain unresolved.

W4. Key experimental details and preprocessing controls are missing. Table 7 does not specify the rewriting model, prompt, retained-data counts, or handling of failed rewrites, and lacks the same preprocessing on original data. Its low accuracies therefore cannot distinguish residual protection from preprocessing damage. Training sequence length, truncation, packing, and loss masking or normalization also need clarification, particularly because SelfCAD changes sequence length. Full configurations and matched controls are needed to interpret and reproduce these comparisons.

Questions for authors
  1. Does SelfCAD remain effective after targeted cleaning with GPT-5.4, using the same cleaner on original and protected data while preserving substantive reasoning and corrections?
  2. Can evaluators familiar with SelfCAD detect systematic modification or identify inserted sentences through targeted inspection?
  3. How much of the degradation is attributable to self-cautious content versus added length or truncation, and does it persist with a length-matched neutral-insertion control?
  4. What rewriting model, prompt, sample-retention criteria, and training sequence, packing, and loss settings were used for Table 7, and how does the same preprocessing affect original-data distillation?
  5. Can SelfCAD protect closed-source reasoning models, such as GPT or Claude, when they expose only reasoning summaries or no reasoning trace? Does its effectiveness require full traces, or can self-cautious insertions into summaries also hinder distillation?
3 comments

W1: “Resistance to targeted cleaning remains untested. ... Applying the same cleaner to original and protected data would test whether SelfCAD retains an advantage.”

Q1: “Does SelfCAD remain effective after targeted cleaning with GPT-5.4, using the same cleaner on original and protected data while preserving substantive reasoning and corrections?”

We thank the reviewer for this concrete attack proposal. We agree that the clean sentence-removal ablation on ordinary data does not itself establish resistance to adaptive cleaning. Equally, that ablation does not establish that an attacker can faithfully identify and remove the inserted content: the original decomposition uses GPT-4.1, whereas practical cleaning must preserve the useful reasoning while separating genuine verification from inserted cues.

Using GPT-4.1 or GPT-5.4 as a cleaner would assume that the attacker has access to a model more capable than the protected teacher. Such access cannot be assumed in the practical deployment setting we consider, where the attacker seeks to acquire the teacher's reasoning capabilities through distillation. We therefore tested targeted cleaning with Qwen2.5-1.5B-Instruct, a rewriter at the student scale, using the suggested instruction:

Remove redundant expressions of doubt or requests to re-check, such as 'Wait, let me check again,' while preserving calculations, logical steps, substantive checks, corrections, and the final answer.

Despite the preservation instruction, the rewriter compressed approximately 5,000-token CoTs to approximately 500 tokens and omitted or rewrote substantive content. This indicates that, in our tested setting, the student-scale rewriter cannot faithfully remove self-cautious sentences while preserving the substantive reasoning.

We further repeated the four-category decomposition on the same 4,000 Bespoke traces using Qwen2.5-1.5B-Instruct (Appendix B.1, Sentence Removal with a Student-Scale Decomposer; Table 13):

Fidelity criterionCountFraction of all 4,000 inputs
Parseable classification1,08427.10%
Full original content preserved, ignoring layout whitespace130.325%
Byte-exact reconstruction00%

Paired training on the 1,084 parseable examples gives GSM8K accuracy of 20.3942% with Original versus 16.9067% with Remove-SC, and mean generation lengths of 7,072.09 versus 8,175.90 tokens. Thus, the student-scale implementation does not reproduce the clean removal operation of the stronger decomposer.

Our practical argument is that an attacker using a much stronger model to rewrite every long trace incurs additional inference cost and relies on capabilities beyond those of the small student.

W2: “Preserved readability does not rule out detectable modification. ... A targeted human detection study would better support the stealth claim.”

Q2: “Can evaluators familiar with SelfCAD detect systematic modification or identify inserted sentences through targeted inspection?”

Thank you for distinguishing readability from detectability. We performed an additional informed inspection of 20 responses, explicitly explaining SelfCAD to the evaluators. In 6/20 responses, evaluators could confidently point to an insertion that appeared contextually implausible. The remaining responses did not yield such a confident identification under this inspection protocol.

This result is consistent with the motivation for using natural teacher expressions: genuine reasoning traces already contain frequent checking and repeated cautious phrases, making an individual insertion difficult to distinguish from native self-caution even when the method is known.

W3: “Self-cautious content is not isolated from length and truncation effects. ... A length-matched neutral-insertion control, training and generation truncation rates, and separate results for unfinished versus completed but incorrect responses would help ...”

Q3: “How much of the degradation is attributable to self-cautious content versus added length or truncation, and does it persist with a length-matched neutral-insertion control?”

We agree that training-data expansion, inference-time truncation, and self-cautious behavior should be distinguished. We added a matched-generation-cap experiment on GSM8K with Qwen2.5-1.5B-Instruct (Section 5.4, Table 9). Each entry below is Original / SelfCAD.

Maximum new tokensAccuracy (%)Mean generated tokensToken-limit hit rate (%)
2,04854 / 31560 / 1,90017 / 91
4,09655 / 31600 / 3,8002 / 83
8,19255 / 31690 / 6,2001 / 31
10,00055 / 32710 / 6,4001 / 6

SelfCAD accuracy remains 31–32% while its limit-hit rate falls from 91% to 6%; the Original–SelfCAD accuracy gap remains 22–24 points. This argues against inference-time truncation being the sole explanation. Even shortening SelfCAD outputs through the evaluated efficient-reasoning prompt or DEER [1] does not restore accuracy: the former yields 31% accuracy with 5,500 tokens, and the latter 29% with 3,200 tokens, compared with Original's 55% and 700 tokens.

The behavioral analysis in Section 5.3 (Table 8; evaluation details in Appendix A.6) also tests identical correct intermediate prefixes with identical input token IDs and a common 256-token observation window. Re-checking rises from 2/90 (2.22%) to 23/90 (25.56%) under SelfCAD, while reversal remains 1/90 in both conditions. This directly supports a change in checking behavior after the same correct step. In natural trajectories, reversal after reaching a correct candidate rises from 1/57 (1.75%) to 11/65 (16.92%).

We also added a neutral-insertion control (Appendix B.6, Neutral Sentence Insertion Control; Table 18) that uses the same insertion positions, counts, anchoring decisions, and numerical or variable anchors as SelfCAD, but replaces caution cues with neutral statements and transitions. On GSM8K, neutral insertion yields 28.58%, 44.66%, and 59.36% accuracy for Llama3.2-1B, Qwen2.5-1.5B, and Llama3.2-3B, compared with 23%, 32%, and 38% under SelfCAD. Neutral insertion also yields higher accuracy on each of the other three benchmarks for all three students. This supports a contribution from self-cautious content beyond generic insertion.

For training, approximately 5% of examples are excluded when insertions make them too long. This exclusion rate is distinct from generation-time limit hits. Together, the controls show persistent degradation under relaxed generation limits, excess re-checking, and a difference between neutral and self-cautious insertions.

W4: “Key experimental details and preprocessing controls are missing. Table 7 does not specify the rewriting model, prompt, retained-data counts, or handling of failed rewrites, and lacks the same preprocessing on original data.”

Q4: “What rewriting model, prompt, sample-retention criteria, and training sequence, packing, and loss settings were used for Table 7, and how does the same preprocessing affect original-data distillation?”

Thank you for requesting the matched control. The prompt and matched-control results are reported in Appendix A.5 (Adaptive Rewriting: Prompt and Matched Control), Table 11, supplementing Section 5.5 (Ablation Study on Adaptive Distillation). The rewriter is Qwen2.5-1.5B, using the following prompt:

I will give you a question and a chain of thought reasoning process. Your task is to rewrite the chain of thought to make it more concise and accurate, while preserving the original meaning. Please ensure that the rewritten reasoning is clear and easy to understand.
Question:

{question}

Chain of Thought:

{cot}

Now, please provide the rewritten chain of thought below. Please only output the rewritten chain of thought without any additional text.

All rewritten examples are used, without rejection or correctness-based filtering; there were no failed outputs. Applying the same rewriter and prompt to original data gives the following GSM8K results for the Qwen2.5-1.5B student:

Training dataAccuracy (%)Mean generated tokens
Original55700
SelfCAD326,400
Original + Rewrite35327
SelfCAD + Rewrite15500

The control reveals both preprocessing damage and residual protection: rewriting lowers Original accuracy by 20 points, but SelfCAD + Rewrite remains another 20 points below Original + Rewrite. Thus, our conclusion does not rely on comparing a heavily rewritten protected dataset only against an untouched original dataset. The gap persists even when both rewritten students generate short responses.

The rewrite experiment uses the same maximum sequence length as the prior experiments, and no examples were truncated in this experiment. Main distillation uses three epochs, a constant learning rate of 2e-5, and batch size 64 on A100 GPUs. The all-retained rewrite policy is separate from ordinary SelfCAD preparation, where approximately 5% of examples are excluded for exceeding the training-length limit after insertion.

Q5: “Can SelfCAD protect closed-source reasoning models ... when they expose only reasoning summaries or no reasoning trace? ... can self-cautious insertions into summaries also hinder distillation?”

Our motivation is to enable providers, including closed-source providers, to expose useful reasoning traces while reducing their value for unauthorized distillation. Making such traces available can improve readability and provide users with more information about the displayed derivation. The defense operates on the returned text and does not require access to model weights or teacher logits, so closed-source deployment is compatible with the mechanism when the provider exposes a suitable reasoning trace.

The evaluated setting uses full visible reasoning traces. If no trace is exposed, there is no such trace for SelfCAD to modify. A short reasoning summary that reports only the results of reasoning, rather than the intermediate reflection and verification process, is not the full chain of thought considered in our study. We have not been able to conduct an experiment in this setting and therefore cannot establish whether inserting self-cautious cues into such summaries would hinder distillation.

References:

  1. Yang et al. Dynamic Early Exit in Reasoning Models. arXiv:2504.15895, 2025.
Reviewer_gGEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SelfCAD, a data-side anti-distillation method that inserts self-cautious sentences sampled from a protected model’s natural outputs into reasoning traces before they are used to train a student. The authors evaluate student accuracy and response length across several student models and math benchmarks, and compare against anti-distillation sampling (ADS) in a self-distillation setting. SelfCAD often lowers student accuracy while increasing response length; some settings show very large length increases. The paper also studies sentence selection and reports supplementary analyses of the teacher and generated data.

Critical strengths
  1. The method is lightweight at the data-processing stage and does not require modifying or fine-tuning the protected teacher. This is a practical property for providers who can curate the released traces but cannot change teacher weights.
  2. The evaluation reports response length alongside accuracy and covers several student sizes and benchmarks. This reveals an important deployment cost that an accuracy-only evaluation would hide.
Critical weaknesses

The baseline and sentence-source ablations are insufficient to establish the method’s distinct contribution. ADS is the sole reported baseline in the main comparison. Further, the method samples cautions from the protected model’s own outputs without testing cross-model, generic sentences. To fix this, please compare against other applicable anti-distillation strategies, or narrow the claim to the tested construction.

Questions for authors
  1. SelfCAD makes student responses much longer. For example, Qwen2.5-1.5B’s average answer on GSM8K jumps from 700 to 6,400 tokens, while its accuracy drops from 55% to 32%. Because of this, I'm a bit curious: does SelfCAD actually harm the student’s underlying reasoning, or does it simply teach the model to write longer, less efficient answers? To help clarify this, could you possibly report the accuracy using matched token budgets? Ultimately, this will show how much of the accuracy drop remains when you control for output length and inference costs.
  2. How do teacher quality and generation cost compare across the original and SelfCAD pipelines? Does SelfCAD require additional teacher generations or filtering, and what are the resulting total tokens and compute?
  3. Are sentences from a different model, fixed generic caution sentences, or alternative sentence pools similarly effective? It would be really helpful to see what evidence makes it necessary to sample directly from the protected model’s own natural outputs.
2 comments

W1: “The baseline and sentence-source ablations are insufficient to establish the method’s distinct contribution. ADS is the sole reported baseline ... without testing cross-model, generic sentences.”

Q3: “Are sentences from a different model, fixed generic caution sentences, or alternative sentence pools similarly effective? ... what evidence makes it necessary to sample directly from the protected model’s own natural outputs.”

We thank the reviewer for requesting a clearer comparison and a controlled examination of sentence sources. SelfCAD's distinct contribution is a post-processing defense that preserves the teacher's existing derivations and answer while making distillation less accurate and more computationally expensive. It does not require modifying teacher sampling or generating additional reasoning.

Baselines. In addition to anti-distillation sampling (ADS) [1], we attempted DOGe [2]. However, its optimization was sensitive to hyperparameters and unstable in our setting, preventing a reliable quantitative comparison. We disclose this in the appendix on baseline implementation rather than reporting an unreliable result or treating reproduction difficulty as evidence of superiority.

Sentence-source control. We added Teacher-only, Manual-only, Template-only, and Mixed variants on the same teacher responses (Appendix B.2, Controlled Ablation of Caution-Sentence Sources; Tables 14–15). The manual pool contains 25 fixed caution sentences; the template pool contains 10 templates using a variable assignment or numerical value from the preceding original reasoning line, with a generic fallback when no anchor is available. Across variants, insertion positions, counts, anchors, confirmation closers, original reasoning lines, and final answers are fixed. Each variant inserts 175,065 sentences, or 21.92 per example; only the caution wording changes.

The following results are equally weighted means over GSM8K, MATH-500, MATH algebra, and AQUA-RAT. Each cell reports accuracy (%) / mean generated tokens.

StudentMixedTeacher-onlyManual-onlyTemplate-only
Llama3.2-1B15.78 / 3,321.114.67 / 5,157.614.03 / 4,673.413.96 / 3,858.4
Qwen2.5-1.5B29.35 / 7,159.326.69 / 7,111.331.66 / 3,463.129.90 / 4,244.2
Llama3.2-3B31.03 / 3,679.433.45 / 2,940.428.77 / 3,644.830.64 / 3,619.7

These results show that the effect is not restricted to teacher-derived wording. Its magnitude depends on both the source and the student. The teacher's own expressions provide a natural source of stylistically compatible cues, but we do not claim that this source is necessary. Equal insertion counts also do not imply equal token budgets: mean cue lengths are 73.4, 94.4, 42.0, and 32.2 characters for Mixed, Teacher-only, Manual-only, and Template-only, respectively.

Q1: “Does SelfCAD actually harm the student’s underlying reasoning, or does it simply teach the model to write longer, less efficient answers? ... report the accuracy using matched token budgets.”

Thank you for highlighting this distinction. We added a generation-budget sweep (Section 5.4, Evaluation under Matched Generation Budgets; Table 9) and a behavioral analysis of correct reasoning steps (Section 5.3, Re-checking and Reversal of Correct Results; Table 8, with evaluation details in Appendix A.6). The budget sweep evaluates Original- and SelfCAD-distilled Qwen2.5-1.5B-Instruct on GSM8K with identical maximum generation budgets. Entries below are Original / SelfCAD.

Maximum new tokensAccuracy (%)Actual mean generated tokensToken-limit hit rate (%)
2,04854 / 31560 / 1,90017 / 91
4,09655 / 31600 / 3,8002 / 83
8,19255 / 31690 / 6,2001 / 31
10,00055 / 32710 / 6,4001 / 6

Increasing the SelfCAD budget from 2048 to 10,000 tokens reduces its limit-hit rate from 91% to 6%, but accuracy improves by only one point. Thus, relaxing the generation limit does not recover the lost performance in this experiment. These are matched maximum budgets; the actual token consumption is reported separately and remains substantially higher under SelfCAD.

To examine behavior beyond answer length, we also provide both students with identical, validated correct intermediate prefixes. Within the next 256 tokens, SelfCAD re-checks the correct result in 23/90 cases (25.56%), versus 2/90 (2.22%) for Original. Reversals are 1/90 in both conditions, so this controlled experiment supports increased re-checking, not increased reversal in every context. In a separate 100-question natural-trajectory analysis, correct candidates are subsequently reversed in 11/65 SelfCAD trajectories (16.92%), versus 1/57 Original trajectories (1.75%), conditional on reaching a correct candidate. These observations support a behavioral impairment in reaching and retaining a final answer, beyond merely adding harmless verbosity. The increased re-checking after identical correct steps indicates reduced confidence in those correct results.

Q2: “How do teacher quality and generation cost compare across the original and SelfCAD pipelines? Does SelfCAD require additional teacher generations or filtering, and what are the resulting total tokens and compute?”

SelfCAD reuses the original teacher responses and inserts cues through CPU text processing. It preserves their original reasoning lines and final answers; it does not regenerate calculations or require an additional teacher-model call. The manuscript reports processing the distillation dataset on a single CPU within several minutes. The original data collection already filters incorrect and overly long teacher responses; SelfCAD does not require a second teacher-based correctness filter.

We distinguish additional model-generated tokens, which SelfCAD does not require, from additional delivered text tokens, which it does introduce. Counting all assistant responses with add_special_tokens=False gives the following results, reported in Appendix A.7 (Token Overhead to Users), Table 12:

TokenizerOriginal tokensProtected tokensAdded tokensAdded per responseIncrease
Qwen2.5-1.5B13,068,09517,068,3504,000,255500.930.61%
Llama3.212,440,89716,292,9323,852,035482.330.96%

The added tokens constitute 23.44% and 23.64% of the protected text, respectively. Thus, preserving teacher content and avoiding additional teacher inference do not mean zero text overhead. In ordinary SelfCAD training-data preparation, approximately 5% of examples are discarded because insertion pushes them beyond the training-length limit; this is a training constraint, separate from preserving the content of each processed response.

References:

  1. Savani et al. Antidistillation Sampling. arXiv:2504.13146, 2025.
  2. Li et al. DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation. arXiv:2505.19504, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.