acceptodds
Under review as a conference paper at ICLR 2027

How to Steal Reasoning Without Reasoning Traces

Abstract

Many large language models (LLMs) generate multi-step reasoning traces (a.k.a. chains of thought) but expose only final answers and brief reasoning summaries. Recent work recovered reasoning traces through provider-specific API vulnerabilities that were subsequently patched. We ask whether properly hiding reasoning traces is sufficient to prevent extraction of a model's reasoning capabilities. The answer is no. We introduce Trace Inversion models that synthesize detailed reasoning traces with only ordinary black-box access to models' inputs and outputs. We show that (1) fine-tuning student models on synthesized traces substantially improves downstream reasoning, enabling distillation from proprietary black-box LLMs, and, in many cases, (2) the synthesized traces heavily overlap with the ground-truth traces (where the latter are available).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model inversion, reasoning distillation, model stealing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C7eKOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the reasoning capability of black-box LLMs can be extracted without seeing their internal chain-of-thought traces. The authors propose Trace Inversion. It first trains an inversion model on surrogate reasoning traces. Then it uses this model to synthesize detailed reasoning traces from the victim model's final answers and optional reasoning summaries. These synthesized traces are used to fine-tune student models. Experiments on DeepSeek-R1 and GPT-5.4 mini show that synthesized traces can give better supervision than directly using answers, summaries, or weak surrogate traces, especially on math and science reasoning tasks.

Critical strengths
  1. The problem studied in this paper is interesting and has practical meaning. The paper shows that hiding internal reasoning traces is not enough to prevent reasoning capability extraction through black-box access.

  2. The pipeline is simple and easy to understand. I also appreciate that the authors clearly distinguish two things: reconstructing the victim's real reasoning traces, and generating useful reasoning supervision for distillation. This distinction is important.

  3. The experiments are relatively comprehensive. They cover open-weight and commercial black-box victims, different surrogate strengths, multiple student models, and both summary and no-summary settings. The gain over surrogate-trace distillation is convincing when the surrogate is weak, although it is less stable when the surrogate is already strong.

Critical weaknesses
  1. The primary issue with this paper lies in its experimental design. The paper’s central claim is that computation is more valuable than memory; however, the experimental design compares only direct/cmd, think/cmd, think/body, and think/full, omitting the direct/full condition. This is surprising, as the current experiments only demonstrate that when a model engages in "re-thinking," the original Chain-of-Thought (CoT) may offer little additional value. Conversely, when the model does not re-think, the original CoT could The cross-model generalization of the method is not fully understood. The inversion model is trained to reconstruct reasoning from the surrogate's own answers, but at inference time it must generate reasoning for potentially different answers provided by the victim. It is unclear how well this works when the surrogate fails to solve a problem that the victim answers correctly. The paper provides overall cross-model evidence, but it does not analyze the boundary conditions of this generalization. An analysis separating examples where surrogate and victim answers agree or disagree would be useful.

  2. The paper does not fully establish the necessity of training a dedicated inversion model. The current zero-shot baseline uses an instruction-tuned Qwen model, but a stronger comparison would be to directly prompt an existing reasoning-capable model to generate detailed traces conditioned on the victim's answers, without additional inversion training. Since the trained inverter also produces substantially longer traces, controlling for reasoning length and student training-token budget would help determine whether the improvements come from inversion training itself or simply from richer reasoning supervision.

Questions for authors

I would be very grateful if the authors could address the issues I raised in the "Weaknesses" section.

2 comments

We thank the reviewer for the positive assessment and constructive suggestions.

Clarification. We respectfully note that the opening discussion of “computation versus memory” and the direct/cmd, think/cmd, and think/body conditions appears to refer to another paper, as these are not discussed in our work.

Cross-model generalization. Our inverter learns from surrogate reasoning traces but conditions on the victim's observable answers and optional summaries, without accessing its hidden reasoning traces. On JEEBench, GPT-5.4 mini achieves 85.1%, versus only 32.6% for R1-Weak, whose traces train our inverter. Despite this substantial capability gap, victim-conditioned inversion improves Qwen student accuracy from 19.7% (Surrogate-Trace) to 31.6%. This demonstrates cross-model capability transfer, although per-example surrogate–victim disagreement remains an interesting direction for further analysis.

Stronger zero-shot baseline. We additionally evaluated Qwen3.5-35B-A3B, a substantially stronger reasoning model, using the same observable victim information. With 10k training examples, its student achieves 54.2% on MATH500 and 22.8% on JEEBench, compared with 76.0% and 43.7% for our original inverter-based student. Thus, even a stronger pretrained reasoning model falls short of the downstream student performance achieved with our synthesized traces.

Reasoning length and supervision. The original zero-shot baseline generates approximately 979 tokens per trace, and Qwen3.5-35B generates 2,034, compared with 4,971 and 5,767 tokens from our R1-Weak- and R1-trained inverters, respectively. This difference is not merely an experimental confound: synthesizing richer reasoning supervision from limited black-box outputs is a central capability of our attack. While matching supervision-token budgets would help isolate the contribution of inversion training, simply truncating synthesized traces would discard potentially useful reasoning steps. To further assess the quality of this additional supervision, we conducted an LLM-as-Judge evaluation of potential distillation utility, which preferred our R1-trained inverted traces over zero-shot (56–14).

Thank you for the clarifications and additional experiments. I maintain my score.

Reviewer_GyeLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper's core problem is interesting. Specifically, when a teacher model (like the paper's GPT-5.4 mini) doesn't reveal its reasoning process, can we still distill it by using another model to synthesize CoTs? In the method, they collect a dataset of full CoTs, summaries, and final answers. Then they use this dataset to train an open-source surrogate model. Therefore, they can generate the full CoT of the victim model based on the summary of reasoning and the final answer.

Critical strengths

The paper's core problem is interesting.

It considers two scenarios. The first includes a summary of the reasoning process, and the second doesn't.

Figure 1 is great and helps readers understand the paper.

Critical weaknesses

Although the problem in this paper is interesting, the proposed method is relatively straightforward and offers limited insights. Specifically, training models using the full CoTs, reasoning summaries, or final answers should be considered standard baselines, similar to the zero-shot approach, rather than a novel methodological contribution.

It would be better if the authors could add more detailed explanations in the experimental analysis section.

The paper is difficult to follow and would benefit from further polishing to improve clarity and readability. In particular, several formatting issues should be addressed, such as changing the ** to bold in Appendix A and correcting the ¡think¿ tags.

At the same time, it would be helpful to improve the paper's structure. For example, in Section 3, it may be better to introduce the THREAT MODEL more generally and move model and training-dataset details to the Experimental setting.

Questions for authors

In particular, the paper claims that hiding reasoning traces is not sufficient to prevent extraction of a model’s reasoning capabilities. What is the reasoning capabilities refer to? Does it refer to similar CoTs, or to downstream task accuracy?

If it refers to similar CoTs. However, Table 2 doesn't include baselines. I'm not sure whether BLEU 26.64 means high similarity. It would be better to include the similarity of different CoTs generated by the victim model. The authors could also use an LLM-as-Judge to measure similarity, since BLEU may be influenced by common tokens like 'wait' or 'let me think'.

If this refers to downstream tasks, it may be better to try harder problems where GPT-5.4 and DeepSeek have a larger gap.

I think the question matters because it relates to the paper's core claim.

6 comments

We thank the reviewer for the constructive feedback.

Novelty and contribution. To our knowledge, ours is the first demonstration of reasoning-capability extraction from black-box models without accessing their hidden CoT or exploiting API vulnerabilities. Subsequent studies, EchoCoT [1] and Stealing Reasoning Traces from Proprietary LLM APIs [2], extract hidden CoTs through tool-call injection or vulnerabilities involving encrypted reasoning blocks. In contrast, our approach trains a trace inverter solely on an accessible surrogate and transfers it to a stronger victim using only legitimate API outputs—answers and optional summaries. No jailbreak, leaked reasoning, or privileged access is required. This exposes a more fundamental security risk: even when reasoning traces are successfully concealed, an adversary can synthesize effective reasoning supervision and distill the victim's capabilities. Thus, preventing CoT disclosure alone is insufficient to prevent capability extraction.

Table 2 and LLM judgments. To contextualize BLEU, unrelated R1 traces score 4.2 (token-F1 32.2) and zero-shot expansion scores 3.04 (token-F1 35.36), versus 26.6 (token-F1 64.4) for our inversion. We additionally used Qwen3.5-35B-A3B-FP8 to evaluate potential distillation utility. The judge preferred our R1-trained inverter over zero-shot 56–14 and R1-Weak inversion over zero-shot 41–29. These judgments provide complementary evidence of the training utility of R1-inverted traces. Importantly, we define reasoning capability by downstream student accuracy, rather than reconstruction of hidden CoT, and use it as our primary measure of extraction success.

Harder problems and experimental analysis. Our JEEBench setting already features a substantial capability gap: GPT-5.4 mini achieves 85.1%, compared with only 32.6% for R1-Weak. Despite this gap, victim-conditioned inversion improves Qwen student accuracy from 19.7% to 31.6% over Surrogate-Trace. We will highlight this challenging setting and expand our analysis of baseline differences, cross-model transfer, and trace-quality limitations. We welcome suggestions on any specific additional analysis the reviewer considers valuable.

Presentation. We will correct the formatting issues, clarify the general threat model in §3, and consolidate implementation details in §5.1.

[1] Qu et al. EchoCoT: Extracting Hidden Chain-of-Thought from Large Reasoning Models.

[2] Panfilov et al. Stealing Reasoning Traces from Proprietary LLM APIs.

Thanks to the authors for the explanations. My biggest concern isn't the novelty or technical depth. My concern is that the results don't support the claim.

First, I have some concerns about the experimental results and setup, as fine-tuning brings almost no improvement over the original model. This raises two questions: (1) whether there are any issues or overlooked factors in the experimental or evaluation pipeline, and (2) whether an attacker would have any incentive to steal reasoning capabilities through distillation if the performance gains are so limited.

Concretely, in Table 3, Surrogate-Trace falls below No fine-tuning: for Qwen on MATH500, accuracy is 71.2 without fine-tuning and 63.2 after. The same problem appears for Qwen on JEEBench (28.3 → 19.7) and LiveCodeBench (28.9 → 22.9). I am confused by this part. The only explanation I could find concerns the MATH500 results, where the authors say that the base Qwen model "appears to have been optimized for the MATH500 benchmark," so fine-tuning can degrade its performance. However, it cannot explain why Surrogate-Trace also lowers Qwen’s accuracy on JEEBench (28.3 → 19.7) and LiveCodeBench (28.9 → 22.9).

Therefore, it would be helpful if the authors could provide further analysis or detailed examples to explain why distillation may negatively affect the student's performance in these settings. Even so, I still recommend switching a setting or dataset, because when distillation harms the student's ability, distillation becomes less meaningful, and attackers would not want to steal the reasoning ability.

Second, the results do not show that reasoning capability has actually been "stolen." In Table 3, Synthesized-Trace (ours, R1-Weak) surrogate shows surprisingly little improvement over No fine-tuning (e.g., 71.2 → 71.8 for Qwen on MATH500). Given the above concerns about the Qwen results, I look at Llama's results instead. Synthesized-Trace achieves only 50.2 on MATH500, substantially lower than the 59.6 obtained using the victim's real CoT. How does this support the claim that reasoning capability has been successfully stolen?

Only Synthesized-Trace with the R1 surrogate approaches the oracle performance, but this improvement may partly stem from R1's own capabilities. Moreover, the R1 surrogate is fine-tuned on R1's real CoT during data preparation. Since R1 is the victim and its CoT is assumed to be inaccessible, I am unclear how this setup is consistent with the proposed threat model.

Finally, the Introduction states that "fine-tuning on synthesized traces can sometimes outperform using ground-truth traces, as the latter may contain backtracking and dead ends". Is this claim supported by experimental evidence? Moreover, if this is indeed the case, researchers with access to the original CoT could simply clean or refine the traces before distillation, as part of standard data preprocessing. Have the authors considered comparing their method against such cleaned or refined traces?

I hope my comments can be helpful. These are genuine concerns I had while reading the paper. I think the research question is interesting and important, but a compelling research question alone is not sufficient to convince me that reasoning capabilities can be effectively stolen when the victim's reasoning traces are hidden. I hope the authors can further strengthen the experimental evidence to support this claim, and that my feedback will help improve the paper.

We thank the reviewer for the thoughtful follow-up and address the three concerns below.

  1. Why does Surrogate-Trace degrade student performance, and what incentive does an attacker have if distillation provides limited gains?

We appreciate the reviewer's concern regarding the performance degradation of Surrogate-Trace. However, such degradation does not necessarily indicate a faulty pipeline. Prior work has shown that stronger reasoning teachers do not always produce better students, as distillation effectiveness depends on supervision quality and compatibility [1, 2]. R1-Weak outperforms untuned Qwen on MATH500 (81.4% vs. 71.2%), yet directly distilling its traces reduces Qwen accuracy to 63.2%.

Our central contribution is demonstrating that the same surrogate reasoning can be leveraged more effectively through victim-conditioned trace inversion. With R1-Weak as the surrogate, our method achieves 71.8%/36.3% (R1 victim) and 73.2%/31.6% (GPT-5.4 mini victim) on MATH500/JEEBench, outperforming both direct Surrogate-Trace distillation (63.2%/19.7%) and the untuned student (71.2%/28.3%). Thus, even when direct distillation fails, inversion can transform accessible surrogate reasoning into useful supervision conditioned on a stronger victim's outputs.

Importantly, the benefits are not limited to marginal improvements. With GPT-5.4 mini as a black-box victim and an R1-trained inverter, Qwen reaches 76.0%/43.7%, further increasing to 80.8%/44.5% with 15k victim queries (Figure 3). While gains vary across settings, these results demonstrate meaningful capability transfer without access to the victim's hidden reasoning.

  1. How do these results establish capability stealing, and does using R1 as a surrogate violate the threat model?

Our goal is to demonstrate reasoning-capability extraction, not necessarily to reproduce the victim's full performance. For example, with R1-Weak as the surrogate and R1 as the victim, our method improves Llama's MATH500 accuracy from 42.1% to 50.2%, recovering approximately 46% of the performance improvement achievable with the victim's actual reasoning traces (59.6%). This demonstrates meaningful capability transfer without directly using the victim's reasoning traces for student training.

Our experiments serve two complementary purposes. The R1 setting provides a controlled evaluation where ground-truth reasoning traces are available for measuring inversion fidelity and establishing an oracle reference. Using R1 as both the surrogate and victim is primarily intended for evaluating our method rather than representing our main attack scenario, although it may also approximate settings where an attacker obtains limited victim reasoning traces through jailbreaks or prompt injection. Our primary threat model targets GPT-5.4 mini as an independent black-box victim, with the inverter trained exclusively on accessible R1 or R1-Weak surrogate traces. The attacker never accesses GPT-5.4 mini's hidden reasoning traces. We will clarify the distinction between these evaluation settings and emphasize the GPT-5.4 mini attack.

  1. Why can synthesized traces outperform ground-truth traces, and what about cleaned CoT?

The observation that synthesized traces occasionally outperform raw ground-truth traces is an interesting empirical finding, rather than the central focus of our work. We hypothesize that backtracking and dead ends in raw traces may contribute to this difference, although investigating the underlying mechanism is beyond the scope of our study.

More importantly, cleaning or refining the victim's actual reasoning traces requires access to those traces, which is unavailable to the attacker in our primary black-box threat model. Our central contribution is demonstrating that, even without such access, an attacker can synthesize useful reasoning supervision from observable victim outputs and achieve meaningful improvements in downstream student performance. We will clarify this distinction.

[1] Chen et al. Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning.

[2] Li et al. Small Models Struggle to Learn from Strong Reasoners.

Reviewer_mbfQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reasoning-capability distillation when a victim exposes answers and optionally reasoning summaries, but not its full reasoning traces. Trace Inversion trains an answer-conditioned trace generator on an accessible surrogate's traces, applies it to the victim's observable outputs, and uses the synthesized traces to fine-tune a student. Experiments cover R1 and GPT-5.4 mini as victims, strong and weak surrogates, two students, and mathematical, scientific, and coding benchmarks. The most convincing gains occur on mathematical reasoning with summaries. The paper appropriately distinguishes useful synthetic supervision from faithful recovery of the victim's internal reasoning.

Critical strengths
  1. A relevant practical question. The paper examines whether hiding reasoning traces prevents capability transfer through observable outputs, and appropriately evaluates downstream student performance rather than requiring faithful trace recovery.

  2. Useful empirical evidence. The results show that even a weaker surrogate can help construct useful supervision for some mathematical tasks. The paper also reports limitations in coding and explores adjustments to the supervision format.

Critical weaknesses

W1 · Limited methodological novelty. This is my main concern. STaR [1], RLT [2], and REER [3] already establish answer-conditioned rationale generation and the use of synthesized reasoning for training. The paper adapts this recipe by training an inverter on surrogate traces and applying it to a victim's observable outputs, but does not establish a substantial methodological advance over these precedents. RLT and REER are also absent from the discussion. The positive results demonstrate useful supervision, but leave the distinct contribution of surrogate-trained inversion unclear.

W2 · Confounded attribution of the gains. R1-Weak is distilled from R1, and the compressor, inverter, and one student share the Qwen2.5 family, potentially favoring inherited styles and model compatibility. The GPT and Llama experiments partly address this concern, but the inverter family remains fixed. Separately, the zero-shot generator lacks comparable reasoning training, while Surrogate-Trace does not exploit the same victim outputs. These comparisons therefore do not isolate inversion-specific gains from ordinary reasoning adaptation or better information access. Matched training, inputs, and budgets, alongside alternative model lineages, are needed for clearer attribution.

W3 · Unexplained collapse of direct-distillation baselines. Answer-only and Summary+Answer perform far below untuned models, with near-zero coding scores in Table 4. The domain-gating experiment shows that supervision format matters, but does not explain these failures. Shared hyperparameters alone do not ensure appropriate optimization across target formats. Without checks of output validity, truncation, and format-sensitive training or evaluation, the large margins over these baselines may partly reflect avoidable pipeline failures rather than limitations of direct supervision.

References

[1] STaR: Bootstrapping Reasoning With Reasoning. 2022. arXiv:2203.14465.

[2] Reinforcement Learning Teachers of Test Time Scaling. 2025. arXiv:2506.08388.

[3] Reverse-Engineered Reasoning for Open-Ended Generation. 2025. arXiv:2509.06160.

Questions for authors
  1. What substantive methodological advance does surrogate-trained inversion provide over STaR-style rationalization, RLT, and REER?
  2. Do the gains persist with a comparably trained rationale generator given identical victim outputs and matched budgets, and when the surrogate is not distilled from the victim or the inverter comes from another model family?
  3. What causes the Table 4 baseline collapses, and do the gains persist after validating target formatting, truncation, and training settings?
1 comment

We thank the reviewer for the constructive feedback.

W1. Novelty relative to STaR, RLT, and REER. We agree that answer-conditioned rationale generation is not new, and will add RLT and REER to the related work. However, our contribution goes beyond applying this technique to another distillation task. We introduce and evaluate a distinct capability-extraction attack in which a model trained exclusively on an accessible surrogate transfers reasoning supervision from a different, black-box victim whose actual reasoning trace is never observed.

The distinction from prior work is important:

  • STaR bootstraps a model's own reasoning through iterative rationale generation and self-training.
  • RLT trains a cooperative teacher to generate student-oriented explanations using student-feedback reinforcement learning.
  • REER reverse-engineers reasoning trajectories from known solutions to improve open-ended generation.
  • Our attack learns a conditional trace inverter offline from surrogate traces, then applies it to an independently queried victim using only its answers and optional summaries. Neither victim traces nor victim-model weight access are required.

Thus, the novelty is not answer-conditioned generation, but demonstrating that a transferable learned inverter can exploit a black-box reasoning API to synthesize effective distillation supervision, undermining the assumption that hiding CoT protects reasoning capabilities. We will sharpen this distinction in the introduction and related work.

W2. Attribution and model compatibility. Surrogate-Trace represents a natural attacker baseline that directly distills reasoning from an accessible surrogate, whereas our approach additionally leverages the victim's observable outputs. To assess the contribution of inversion training, we compare against zero-shot generation using identical victim questions, answers, and summaries. We further evaluated Qwen3.5-35B-A3B, a substantially stronger pretrained reasoning model. Its student achieves 54.2% on MATH500 and 22.8% on JEEBench, compared with 76.0% and 43.7% for our original inverter-based student. This provides further evidence of the effectiveness of our synthesized traces.

Regarding model compatibility, GPT-5.4 mini is independent of the R1-distilled surrogate lineage, and our results also generalize to a Llama student. We additionally trained a Llama-3.1-8B inverter using the same surrogate reasoning data. On 500 held-out problems, its generated traces achieve 20.7 BLEU, compared with 26.0 for the Qwen inverter and approximately 4.4 for unrelated traces. The Llama inverter also generates reasoning traces of comparable length, averaging 7,161 tokens, versus 5,868 tokens for the Qwen inverter. These results provide preliminary evidence that trace inversion can be learned across different model families. Due to time and computational constraints, this experiment is currently limited to evaluating trace generation and fidelity rather than establishing comparable downstream extraction performance. We are actively working to address this through further evaluation.

W3. Baseline collapse. We investigated the reviewer's concern through a detailed audit and conservative retraining. The audit confirmed correct target serialization, loss masking, and no truncation, but identified severe repetitive generation in the original short-target baselines. Further conservative retraining (one epoch, learning rate 5e-6, no packing) largely eliminated this degeneration. Qwen Summary+Answer improved from 19.8% to 40.2% on MATH500, but remained substantially below the untuned student's 71.2%; Llama Answer-only decreased slightly from 13.0% to 11.2%, compared with 42.1% untuned. Inspection of the generated outputs reveals that these students predominantly imitate their short training targets, producing brief summaries or direct answers rather than explicit reasoning steps. This suggests that fine-tuning an already instruction-tuned model on short answer- or summary-based targets can suppress its existing reasoning behavior without providing sufficient intermediate reasoning supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.