- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SelfCAD, a training-free, post-hoc defense against unauthorized distillation of large reasoning models. The authors find that self-cautious sentences mainly lengthen reasoning traces without improving accuracy. SelfCAD therefore inserts such sentences, drawn from the protected model’s own outputs, after every reasoning step of the released trace. This degrades students trained on the processed traces while keeping the original reasoning visible. The authors distill GPT-OSS-120B and R1 into 1B–7B Llama and Qwen students and report accuracy drops of 5–23 points and outputs 1.3–9.1× longer on four math benchmarks.
Critical strengths
-
The motivation for the black-box API distillation setting is clear: how to release useful reasoning traces while protecting the underlying model’s capabilities. Protecting outputs without modifying the teacher model’s parameters is an attractive design direction.
-
The reasoning-trajectory component removal experiments (removing reasoning, self-cautious and conclusion sentences) have some exploratory value. Instead of reporting only final benchmark scores, they offer more mechanism-level insight.
-
The human readability study and the use of multiple LLM judges show that the paper cares about the experience of benign users who rely on the protected outputs.
Critical weaknesses
-
The most critical adaptive attack is not evaluated. The inserted sentences are a uniform, structured artifact that the paper’s own “Remove SC” pipeline in Sec. 3.2 could strip. Table 2 shows this keeps accuracy nearly unchanged at 37% versus 39% with 0.4 times the length, so SelfCAD may be easily bypassed.
-
The claim that SelfCAD makes distilled models “less confident even on correct steps” is not supported by any error analysis of student outputs, such as how often correct intermediate results are overturned.
-
The cost to benign users is not quantified. The paper does not report how many tokens the processed reasoning traces add, or the resulting latency overhead.
-
The definition of self-cautious sentences is ambiguous, and what is analyzed does not match what is inserted.
-
The protection claims go beyond the validated results. Underperforming distillation with original responses does not mean the protected outputs are no longer worth collecting. Results for the student before distillation are needed to quantify the benefit that remains.
Questions for authors
-
After SelfCAD processing, how many tokens on average are added to the reasoning traces returned to users, and how much latency overhead does this introduce?
-
Could you report the truncation rate, the accuracy on non-truncated generations, and how accuracy and length change under different generation limits?
-
Have you evaluated an inference-time bypass in which the attacker uses logit bias to suppress reflection tokens such as “Wait” and “Hmm”?
-
See weakness.
4 comments
W3: “The cost to benign users is not quantified. ... how many tokens the processed reasoning traces add, or the resulting latency overhead.”
Q1: “After SelfCAD processing, how many tokens on average are added to the reasoning traces returned to users, and how much latency overhead does this introduce?”
We emphasize that SelfCAD requires only CPU text processing, with no additional teacher-model inference to generate the inserted cues. The manuscript reports processing the full distillation dataset within several minutes on a single CPU. This avoids additional teacher-decoding cost, although it does not imply zero processing latency or zero overhead from the longer returned text; per-response latency has not been separately measured. We counted the original and protected assistant text for all responses with add_special_tokens=False, defining inserted tokens as the difference (Appendix A.7, Token Overhead to Users; Table 12):
| Tokenizer | Original tokens | SelfCAD tokens | Added tokens | Mean added per response | Added / Original | Added / SelfCAD |
|---|---|---|---|---|---|---|
| Qwen2.5-1.5B | 13,068,095 | 17,068,350 | 4,000,255 | 500.9 | 30.61% | 23.44% |
| Llama3.2 | 12,440,897 | 16,292,932 | 3,852,035 | 482.3 | 30.96% | 23.64% |
The protected traces are therefore approximately 1.31 times the original length. These inserted words are produced by CPU text processing, not by additional teacher decoding; original derivations and final answers are retained. Thus, the provider avoids an additional teacher-generation pass, while the user receives approximately 483–501 additional tokens per response on this corpus. This cost should not be confused with the 1.3–9.1-fold student inference length inflation in the main experiments.
W4: “The definition of self-cautious sentences is ambiguous, and what is analyzed does not match what is inserted.”
We thank the reviewer for identifying a distinction that needed to be explicit. A complete self-cautious unit has two components:
- A self-cautious cue, , expressing doubt or initiating a check, such as “Wait, let's re-examine...” or “Let me verify:”.
- The subsequent verification reasoning, , such as the recalculation “pi is approximately 3.14, so 25 × 3.14 is approximately 78.5.”
The full segment is . The revised definition distinguishes the two throughout the analysis and algorithm. Table 1 measures termination immediately after a cue; the reported value of 0.13 is consistent with continued generation after that trigger. In contrast, the Remove-SC analysis in Table 2 and the classification appendix remove complete cue–verification segments, explaining why removal can substantially reduce output length to 0.4 times the original.
SelfCAD deliberately inserts cues rather than synthesizing new problem-specific calculations. Generating new verification steps would require additional model inference and could introduce mathematical errors. In the protected training trace, the cue is placed next to preserved reasoning, using the adjacent step as context. A cue can incorporate a variable assignment or numerical value from the preceding original line, for example, “Wait, did I compute {var = val} correctly?” or “Hmm, let me verify {val}.” Before the first conclusion, a confirmation closer such as “After carefully checking, I'm confident the answer is correct” can form a check-and-confirm unit while preserving the final answer.
The implemented pool combines natural teacher expressions, fixed manual cues, and anchored templates. Natural teacher lines are selected using caution-related keywords, restricted to 15–200 characters, and frequency-weighted with at most five copies per sentence. In the controlled source ablation, the pool contains 45,531 extracted instances, 42,264 distinct sentences, and 44,273 sampling entries after capping. This construction is more varied and context-sensitive than a single repeated trigger.
At student inference time, the trained student generates its own continuation after a cue, including further checks. Thus, the analysis and defense address complementary parts of the same behavior: the analysis measures the role and cost of the full segment, while the defense changes the frequency of its initiating cues without fabricating new mathematical content. The theoretical model describes a shift toward the cautious continuation pattern under its assumptions; the new behavioral tests provide empirical evidence of increased re-checking. Neither the termination statistic nor the theory alone proves that every cue causes a reversal or an accuracy loss.
W5: “Underperforming distillation with original responses does not mean the protected outputs are no longer worth collecting. Results for the student before distillation are needed to quantify the benefit that remains.”
Our evaluation targets the chain-of-thought reasoning capability acquired through distillation. The students are non-reasoning instruction checkpoints rather than specialized reasoning models, so there is no corresponding distilled chain-of-thought capability to evaluate before distillation. We therefore compare students trained on the original and protected teacher responses under the same protocol.
On GSM8K, accuracy falls from 31% to 23%, 55% to 32%, and 60% to 38% for Llama3.2-1B-Instruct, Qwen2.5-1.5B-Instruct, and Llama3.2-3B-Instruct, respectively, while generation becomes longer. These results show that protection degrades the effectiveness and efficiency of acquiring reasoning from these teacher traces.
Q2: “Could you report the truncation rate, the accuracy on non-truncated generations, and how accuracy and length change under different generation limits?”
We added a matched-cap sweep for Original- and SelfCAD-distilled Qwen2.5-1.5B-Instruct on GSM8K (Section 5.4, Evaluation under Matched Generation Budgets; Table 9). Entries are Original / SelfCAD.
| Maximum new tokens | Accuracy (%) | Actual mean generated tokens | Token-limit hit rate (%) |
|---|---|---|---|
| 2,048 | 54 / 31 | 560 / 1,900 | 17 / 91 |
| 4,096 | 55 / 31 | 600 / 3,800 | 2 / 83 |
| 8,192 | 55 / 31 | 690 / 6,200 | 1 / 31 |
| 10,000 | 55 / 32 | 710 / 6,400 | 1 / 6 |
The accuracy deficit persists at every tested cap. In particular, increasing the cap to 10,000 reduces SelfCAD's limit-hit rate to 6%, yet accuracy remains 32%, compared with Original's 55%. This supports the conclusion that increasing the available generation budget does not remove the observed protection. Matched caps do not mean matched actual computation, which is why we report actual mean lengths as well.
Q3: “Have you evaluated an inference-time bypass in which the attacker uses logit bias to suppress reflection tokens such as ‘Wait’ and ‘Hmm’?”
We evaluated this inference-time bypass on GSM8K by applying a logit bias of -100 to "Wait" and "Hmm" in SelfCAD-distilled students. The intervention changes decoding without rewriting training data or retraining. This experiment is described in Section 5.5, with results for all three students in Appendix B.5 (Inference-Time Suppression of Reflection Markers), Table 17. Each cell reports accuracy (%) / mean generated tokens.
| Student | Original | SelfCAD | SelfCAD + logit bias |
|---|---|---|---|
| Llama3.2-1B | 31 / 600 | 23 / 1,300 | 19 / 350 |
| Qwen2.5-1.5B | 55 / 700 | 32 / 6,400 | 35 / 6,200 |
| Llama3.2-3B | 60 / 400 | 38 / 650 | 24 / 800 |
Neither targeted marker appears in the biased outputs, but the students continue to express doubt and re-check using other wording. Qwen gains three accuracy points yet remains 20 points below Original, while accuracy decreases for both Llama students. Suppressing these two markers therefore does not consistently restore performance, although this experiment does not rule out other decoding interventions.
I thank the authors for their response. I will maintain my original score.
We thank the reviewer for raising this attack. The 37% versus 39% result is an analysis of removing complete self-cautious segments from ordinary teacher traces using a strong external decomposer, GPT-4.1. It is not an experiment that identifies and strips SelfCAD insertions from protected traces. Such decomposition requires a highly capable model to process every long trace, incurring substantial additional inference cost. A cleaner such as GPT-5.4 would assume access to capabilities beyond those of both the student and the protected teacher in our setting. We use strong-model decomposition as an ablation tool; access to such a model is not assumed in our practical attacker setting. For an adaptive attack under this setting, a rewriter comparable in scale to the student is a relevant test. We therefore conducted additional experiments with a student-scale model.
We tested Qwen2.5-1.5B-Instruct on the same 4,000 Bespoke traces (Appendix B.1, Table 13). Only 1,084 outputs (27.10%) were parseable, only 13 (0.325%) preserved the complete original content after ignoring layout whitespace, and none reconstructed it byte-exactly. On the matched 1,084-example training subset, Original / Remove-SC GSM8K accuracy was 20.3942% / 16.9067%, and mean output length was 7,072.09 / 8,175.90 tokens. The practical small-model pipeline introduces substantial omissions and rewriting rather than reproducing the strong decomposer's clean intervention.
We also attempted targeted removal with the instruction to remove redundant doubt and re-check requests while preserving calculations, logical steps, substantive checks, corrections, and the final answer. Qwen2.5-1.5B-Instruct nevertheless compressed approximately 5,000-token traces to approximately 500 tokens, losing substantive content.
Thank you for requesting direct behavioral evidence. We added two experiments with Original- and SelfCAD-distilled Qwen2.5-1.5B-Instruct on GSM8K, using identical chat templates and greedy decoding. These are reported in Section 5.3 (Re-checking and Reversal of Correct Results), Table 8, with evaluation and annotation details in Appendix A.6 (Behavioral Evaluation Details).
In natural trajectories, both models solve the same 100 randomly sampled questions with a maximum of 10,000 new tokens. We identify the first correct candidate and examine subsequent checking and reversal. In shared-prefix continuation, both models receive exactly the same correct, non-final intermediate prefix, represented by identical input token IDs. Ninety prefixes pass arithmetic and semantic validation, and we score behavior within the next 256 tokens.
The shared-prefix test is particularly informative: given the same correct reasoning state and the same observation window, SelfCAD increases re-checking by 23.34 percentage points. The natural trajectories additionally show a 15.17-point increase in reversal among trajectories reaching a correct candidate. The unchanged short-window reversal rate shows that reversal depends on the surrounding trajectory; we do not claim that every extra check overturns a correct step.
For annotation, anonymized LLM agents use the question, reference answer, and output context, and every positive event requires a locatable quotation. Numerical matching only proposes candidates. Independent review covers all 12 natural-trajectory reversals plus 20 random cases, and both shared-prefix reversals plus 18 random cases; reversal labels remain unchanged. These experiments provide observable evidence of excessive re-checking and context-dependent reversal. The increased re-checking after the same correct intermediate steps indicates that SelfCAD reduces confidence in correct results.