acceptodds
Under review as a conference paper at ICLR 2027

Hidden Reasoning Must Leak, but Need Not Be Readable: Fundamental Opportunities and Limits for Chain-of-Thought Monitoring

Abstract

Can reasoning models trick chain of thought (CoT) monitors and perform hidden computation without revealing it in their thinking traces? We show that the answer depends on the underlying task difficulty and the model size. Simple computations can be performed covertly; however, beyond a threshold depending on model size, successfully solving the task necessarily leaks a near-linear amount of information about the covert task input into the CoT. Therefore, sufficiently complex hidden computation always leaves an information-theoretic footprint. However, concerningly, this leakage need not be readable: Under plausible cryptographic assumptions, even a one-layer Transformer can encrypt its reasoning online so that no polynomial-time monitor can extract information about the hidden computation. Overall, our theoretical and empirical results provide a holistic view of both the opportunities and the limitations of CoT monitoring.

Primary area:
learning theory
Keywords:
theory, transformers, reasoning, expressivity, safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qMdYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how much information about a private task input must enter a Transformer’s visible chain of thought (CoT), when the monitor sees the CoT but not the private input, private randomness, or final answer. For four task families and a fixed-precision Transformer model, it proves an information-theoretic footprint lower bound. For fixed depth and polynomially bounded model size and CoT length, successful exact computation requires near-linear input–trace mutual information, yielding an average-case CoT-length lower bound. A related entropy argument gives a qualified covert-computation length tax when the public input can be recovered from the trace. Finally, assuming suitable NC0 pseudorandom generators, a single-layer Transformer can encrypt intermediate computation online, making properties of the private input computationally inaccessible despite substantial mutual information. Validation includes three-seed LoRA experiments on two pretrained models and four synthetic tasks, prompting experiments across several model families, and separately trained one-layer encrypted-PARITY models. The latter reach 99.83% joint trajectory-and-answer accuracy at length 1,024 with substantial intermediate supervision. These results illustrate the mechanisms; they do not establish practical cryptographic security or spontaneous emergence of encrypted reasoning.

Critical strengths
  1. A substantive information-flow result with a useful conceptual separation. The footprint theorem strengthens an architectural requirement for intermediate computation into a lower bound on input–trace information, with an average-case CoT-length consequence. The subsequent encryption result shows why this information need not be available to a computationally bounded monitor. This directly sharpens what a CoT-monitoring guarantee would have to mean.
  2. An explicit architectural construction accompanied by informative controls. Online decryption, state update, and re-encryption are implemented within a single Transformer layer under a clearly identified cryptographic assumption. The empirical study includes matched Full-CoT controls, three-seed finetuning, payload recovery, negative results on Llama multiplication, and supervision ablations for encrypted PARITY. These controls make the constructive mechanism more convincing and help delimit its learnability.
Critical weaknesses
  1. The prompting study does not cleanly estimate unconditional covert-task success or the benefit of serial filler computation. Appendix D.1 reports accuracy only on nonleaking, appropriately parsed samples, while Table 11 shows that complete parsing failures can be large (for example, 61.3% for Qwen 3.5 0.8B on S5 Public-only CoT, averaged across difficulties). Calibration examples used to tune prompts are retained in the reported evaluation (D.1, pp. 46–47). Moreover, prompting fillers are appended to the user prompt, whereas finetuning fillers are generated autoregressively. This makes the prompting curves and scale fits less decisive as validation of the theoretical CoT regimes. Per-cell denominators, unconditional success rates, and results on the untouched post-calibration portion would materially improve the evidence.
  2. The practical monitoring implications need narrower wording. Corollary 3.2 assumes the public input is uniquely recoverable from the trace and compares against a minimal aligned construction; Figure 5 compares two prescribed protocols on differently correlated task pairs, so it illustrates the tradeoff rather than demonstrating a general measurable token penalty. The encrypted-PARITY experiment establishes execution of a supervised mechanism, while Appendix E explicitly reports one run, development use of test aggregates, failure of end-to-end-only training, and no monitor-attack evaluation. Together with the fact that an encrypted bitstring may be readily flagged as suspicious, the evidence supports limits on decoding private input properties, but not a general inability to detect hidden computation.
  3. The presentation needs a careful structural and copy-editing pass.The paper also has visible editing and layout issues: the results discussion contains a duplicated sentence (PDF p. 37); a result is referred to as “Theorem 3.2” where “Corollary 3.2” appears to be intended, alongside the phrase “extra extra tokens” (p. 7); and there are errors such as “an monitor,” “human-readbable,” “address this questions,” and “see alo” (pp. 1, 9, 9, and 27). Some figures are difficult to read at their presented size, Figure 4 appears before Figure 3, and the statement box for Corollary 3.2 breaks across pages.
Questions for authors

Q1 — Protocol clarification. Are all prompting fillers supplied in the user message, as D.1 and D.12 state, rather than emitted during autoregressive decoding? If so, please distinguish this input-padding intervention from the generated-Filler condition and qualify the shared Agnostic-CoT interpretation. A small generated-filler comparison would directly resolve this issue.

Q2 — Theorem scope. Can the main-text length-tax claim explicitly retain the requirement that the public input is recoverable from the trace and specify the asymptotic regime in which the bound actually separates the aligned and covert constructions? Please clarify what follows for equal-sized public/private tasks and ordinary nonminimal language-model traces.

Q3— Security and learnability scope. Please state alongside Figure 4 that finite-length cryptographic privacy was not tested, that each length uses a separate trained model, and that the apparent no-CoT threshold is partly tied to the 16-position preparatory supervision.

1 comment

We thank the reviewer for their positive evaluation and would like to address their concerns.

The prompting ... difficulties).

We observed that smaller models (under 2B) had great difficulty in following the instructions properly, and resulted in leaking both questions in most of their responses, as highlighted in Appendix D.9. Most of the models still do not leak significantly, with the leakage contained within 15%. However, we saw in separate plots (Figures 31, 32, and 33) that the unconditional plots still follow the trends established by the conditional results. A similar fit is observed for both max-passing-dd and the joint-correct accuracy in Figure 30.

prompting fillers ... regimes.

This is in fact intentional: LMs do not produce filler CoTs by default. We thus deliberately put the prompting fillers in the user message so we could control content and length across models without relying on the models to generate the requested sequence. This follows prior input-padding work such as [1].

We also performed an experiment where we prompt Kimi K3 to produce fillers (Appendix E). Results are similar (Table 19) to those obtained when appending the fillers to the prompt; generating fillers does not improve accuracy compared to appending them.

Q1 — Protocol clarification ... issue.

Yes, prompting fillers are passed in the user message rather than emitted during autoregressive decoding. However, as per our testing, generated fillers do not significantly increase the joint-correct or private-only correct accuracies. They instead reduce the private-only accuracy when tested with Kimi K3 on multiplication. The experiments are outlined in Appendix E.

Per-cell ... evidence.

We have added the requested tables that refer to per-cell numerators and denominators that in Appendix D.11 as Tables 12, 13, 14, 15, 16, and 17 with the conditional accuracy and the numerator and denominator for their calculation. The unconditional plots have been added in Appendix F (figures 30, 31, 32, and 33), and, as highlighted above, follow the same trends as observed for the conditional case. For untouched, post-calibration accuracy figures, we observed the results to be very similar to our current setup. The accuracy deviations in both unconditional and conditional post-calibration-only settings are nominal, as observed in Figure 35. We see similar trends for post-calibration-only samples in Figure 34 for max-passing-dd as we saw in Figures 3 and 30.

The practical ... penalty.

We agree. We have expanded the caption of Figure 5 to carefully make this point.

The encrypted-PARITY ... computation.

Yes, indeed, we do not mean to claim that the presence of covert computation can itself be hidden -- only its content. We are happy to highlight this more if the reviewer thinks that would be good.

The presentation ... pages.

Thank you for the careful reading. We have fixed most of them, and will take a careful pass to fix any remaining copy-editing issues. Regarding the order of Figure 3 and 4, LaTeX rendering does this to optimize the layout and we are inclined to keep it as is.

Q2 — Theorem scope. Can the main-text ... traces.

We address this as follows. First, we have now mentioned the requirement that the public input is recoverable from the trace in the text. Second, regarding the asymptotic regimes, we write "there is a substantial unavoidable length tax when Ncovert≫NpublicN_{covert} \gg N_{public}"; e.g., in the asymptotic where Ncovert=Npublic2N_{covert} = N_{public}^2, Eqs. 4 ≫\gg Eq 5 . We also do explicity state that an asymptotic separation is only claimed compared to a minimal-length aligned model ("minimally have to", line 205). Let us know if you think any further changes to the phrasing would help.

Q3— Security ... supervision.

We have rephrased to acknowledge these points: we mention that each length uses a separate trained model, that the threshold may relate to the supervision signal, and that the claim about conjectural security is only in the N→∞N \rightarrow \infty asymptotic.

[1] Brauer et al. Reading between the dots: Decoding hidden computation across filler tokens, 2026

Reviewer_bmN2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether Transformers can perform covert computation without revealing it in their chain of thought. It proves that beyond a capacity threshold, solving the task forces the CoT to leak a near-linear amount of information about the hidden input. It also shows that, under a cryptographic assumption, a one-layer Transformer can encrypt that leakage so polynomial-time monitors can't read it. Finetuning and prompting experiments also show that CoTs that carry information about the hidden input outperform filler or public-only CoTs at longer inputs.

Critical strengths

The paper offers a clean, unifying view of hidden reasoning. Its three modes (No CoT, Agnostic CoT, Informative CoT) place filler tokens, piggybacking and steganographic channels in one quantitative framework. The leakage result gives a concrete condition under which CoT monitoring has something to work with, and the encryption construction is notable because it happens inside a single Transformer layer rather than at sampling time.

Critical weaknesses

I don't have major concerns, and the main limitations are already discussed in the paper. The biggest remaining gap is between the theory and practical monitoring. The paper shows that leakage must exist and that it can in principle be made unreadable, but it doesn't test whether a realistic monitor, such as an LLM reading the CoT, can actually pick up the leaked information in the informative-CoT settings. Even a small experiment of this kind would make the safety implications more concrete. The model-size scaling evidence in Figure 3 is also somewhat limited. Each family contains only a small number of models, some reported fits are weak, and the R^2=1 fits for two-model families are not informative. More importantly, it is not obvious that Theorem 3.1 predicts a linear relationship between maximum difficulty and log parameter count, since depth also enters the bound strongly. I therefore find the qualitative scaling trend interesting, but the quantitative linear-fit evidence less convincing.

Questions for authors
  • In Figure 3, is model size measured in total or active parameters? Several of the models are mixture-of-experts, where the two differ a lot (Table 12).
  • The monitor is assumed to know the hidden task in advance. In practice it may not know what to look for. Do the results, for example the length tax, say anything about detecting that hidden reasoning is happening when the hidden task is unknown?
  • FAQ 6 explains qualitatively that deeper models can hide more. Can you give a rough sense of how large a hidden task would need to be before a model of realistic effective depth is forced to leak?

[Suggestions for future work] Would be interesting to test whether optimisation pressure such as RL against a monitor ever pushes models toward encrypted or informative-but-unreadable CoTs without explicit supervision.

1 comment

We thank the reviewer for their positive evaluation and would like to address their concerns.

We number the questions as Q1-Q4.

The paper shows that leakage ... implications more concrete.

We have conducted a new experiment with LLM judges in Appendix I -- Terra 6.1 (High) and Kimi K3 (high) -- over the reasoning traces by Kimi K3 performing in the covert-public setting. We had the judges recover the covert question given the trace, a description of the protocol, and 10 fewshot examples. Results are shown in Table 21 in Appendix I.2.2. The judges had zero accuracy on the Agnostic mode, in agreement with the theoretical expectation that I[Xcovert,CoT]=0I[X_{covert}, CoT]=0 for these. In the Informative mode, recovery was substantial, again in agreement with the theoretical expectation that I[Xcovert,CoT]I[X_{covert}, CoT] is large. Recovery on piggybacking has ~10% accuracy, corresponding to optimal recovery of the covert input from the public input given the way the protocol works. Kimi K3 is able to recover the covert computation exactly from Special Tokens traces with 100% accuracy.

The model-size scaling ... not informative.

We agree with these limitations. These analyses are about as far as we could go based on available models (with publicly known sizes) and given our budget. It could be interesting if researchers who have large-scale compute could revisit this in more detail in future work.

More importantly, it is not obvious ... less convincing.

Indeed, the depth enters Theorem 3.1 exponentially. We didn't mean to imply the scaling is linear. We have rephrased to simply claim that the transition point "increases with the model size" (line 368), without suggesting a specific functional form. Certainly, the data suggest the exponent is not far from 1, and thus far below the depths of these models, which is dozens or hundreds of layers, in line with evidence that current LMs do not use their depth very effectively. It would be exciting if future work could disentangle this, perhaps on the basis of future multi-scale model suites, or by directly training models across sizes (infeasible for us). Also, if future LMs could use their depth more efficiently, this would be expected to manifest in more clearly super-linear curves in Figure 3.

Q1: In Figure 3, is model size measured in total or active parameters? Several of the models are mixture-of-experts, where the two differ a lot (Table 12).

Figure 3 uses total parameters. Table 18 (previously, Table 12) separately reports active parameters. We made this distinction clear in the text (line 368).

Q2: The monitor is assumed to know the hidden task in advance. In practice it may not know what to look for. Do the results, for example the length tax, say anything about detecting that hidden reasoning is happening when the hidden task is unknown?

Yes. A monitor can look for extra reasoning length without knowing what the hidden task is. Under Corollary 3.2’s assumptions, a sufficiently large, independent hidden task can force longer traces on average than an efficient solution to the public task alone. Extra length is a clue and not a proof, meaning that public reasoning can be verbose, or public/covert tasks can share computation and reduce the length tax.

Q3: FAQ 6 explains qualitatively that deeper models can hide more. Can you give a rough sense of how large a hidden task would need to be before a model of realistic effective depth is forced to leak?

A hypothetical future LLM with an effective depth of dd could compute PARITY without any CoT up to length 2d2^d. E.g., at an effective depth of 1616, PARITY would be computable up to length 2162^{16} by such a future LLM, simply by implementing a circuit that aggregates parities along a binary tree. The fact that current LLMs fail, in the no-CoT protocol, already at just a few bits, is consistent with more mechanistic evidence [1] that they don't use their depth very efficiently.

Q4: [Suggestions for future work] Would be interesting to test whether optimisation pressure such as RL against a monitor ever pushes models toward encrypted or informative-but-unreadable CoTs without explicit supervision.

This would indeed be interesting. Given the end-to-end ablations and unsuccessful Qwen experiments, we speculate that discovering this specific cryptographic mechanism without intermediate supervision might be difficult.

[1] Csordás et al. Do language models use their depth efficiently? NeurIPS 2026.

Reviewer_RHxVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether Transformers can perform hidden computations without revealing them in their chain-of-thought (CoT). For several arithmetic and state-tracking tasks, the authors show that sufficiently complex hidden computation must leave an information-theoretic footprint in the CoT, under certain assumptions on model capacity and reasoning length. They also derive a lower bound on the additional CoT length needed for covert computation. Interestingly, they show that a single-layer Transformer can encrypt its reasoning under cryptographic assumptions, making the leaked information difficult for efficient monitors to recover. Experiments on synthetic reasoning tasks provide supporting evidence for these results.

Critical strengths
  1. The paper addresses a key question for CoT monitoring: when does hidden computation leave information in visible reasoning traces? Understanding this limitation is important for assessing the reliability of CoT-based safety monitoring.
  2. The information-theoretic lower bound and the covert computation length tax offer a useful perspective on the relationship between model capacity, reasoning length, and information leakage. The distinction between information leakage and efficient recoverability is particularly valuable.
  3. The authors show that, under cryptographic assumptions, a single-layer Transformer can encrypt its reasoning during generation. Implementing this mechanism within the Transformer itself, rather than through an external encryption procedure, is an interesting technical contribution.
  4. The experiments cover several arithmetic and state-tracking tasks, multiple model families, and different ways of hiding computation. The full-CoT controls and additional ablations help relate the empirical findings to the theoretical analysis.
Critical weaknesses
  1. The main lower bound assumes fixed-precision Transformers, uniformly distributed covert inputs, and exact computation. Its dependence on model depth also makes it difficult to assess how meaningful the bound is for current LLMs. The experiments show trends consistent with the theory, but it remains unclear how well these results extend to practical reasoning models.
  2. The experiments mainly measure covert-task accuracy and explicit leakage in visible reasoning traces. They do not directly test whether a monitor can detect hidden computation or recover information from the traces.
  3. The encryption experiments rely on explicit supervision of attention routing and intermediate computations, with separate models trained for different input lengths. The no-CoT and filler-only baselines also have limited retrieval supervision at longer lengths, as noted in the appendix. While the experiments demonstrate that the construction can be learned, they provide limited evidence that such behavior would emerge without carefully designed supervision.
  4. The experiments do not directly measure the mutual information between covert inputs and CoT traces. In addition, the piggybacking tasks are designed to share most of their computation, making it difficult to separate the benefit of information leakage from that of computation reuse.
Questions for authors
  1. Given the dependence of the lower bound on model depth, how informative is it for modern deep Transformers? Can the authors also clarify whether the result extends to nonzero error rates and standard stochastic decoding, beyond the setting discussed in Remark B.7?
  2. How well would monitors with access to raw token sequences perform at detecting covert computation or recovering hidden information across the different protocols? In particular, could encrypted CoTs still be flagged as suspicious even if their contents cannot be decoded?
  3. The ablations in Appendix E suggest that learning encrypted CoTs depends heavily on intermediate supervision. How should these findings be interpreted in terms of whether such behavior could emerge in pretrained reasoning models without explicit supervision?
3 comments

We thank the reviewer for their positive evaluation and would like to address their concerns (weakness, question numbered as W1-4, as Q1-3).

W1 (1st part): The main ... computation.

Indeed. We discuss the robustness of the lower bound in the Appendix lines 914-955. The core idea that complex reasoning must leave an information-theoretic footprint still holds if we consider infinite-precision transformers that are plausibly learnable (solution does not lie on sharp loss landscapes that are difficult to find during gradient descent).

Re: uniform covert inputs, this matches our experiments. The only place this is used in Thm 3.1 is in the definition of I[Xcovert,CoT]I[X_{covert}, CoT] (defined relative to the uniform distribution over XcovertX_{covert}). Other than that, we simply assume that the model performs the task on all inputs.

Re: exact computation, we now prove a lower bound for nonexact computation in the (new) Appendix B.10. In a nutshell, when the model has error rate ϵ\epsilon, the footprint bound is multiplied with 1−2ϵ1-2\epsilon and else stays the same.

W1 (2nd part): dependence ... models.

Q1 (1st part): dependence ... deep Transformers?

We had anticipated this question and have discussed this in the Appendix lines 1015-1032.

Essentially, our results do predict that deeper models can hide reasoning better, but this could only happen if models use their depth effectively. While current LMs have many layers and have in-principle ability to do so, empirical evidence suggests that the full depth is not effectively used [1]. There may be various reasons for this, eg. LMs learning some 1 step solutions that don't generalize to multi steps [2] or factual recall [3] happening at only one point in the model. All in all, an important take-away is that, if future innovations make LMs use their depth more effectively, undetectable reasoning in CoTs becomes exponentially more likely .

Q1 (2nd part). Can the authors .. Remark B.7?

The sketch in Remark B.8 addresses stochastic decoding in the case of 0 error rates.

Regarding non-0 error rates, we have newly added Appendix B.10. The footprint theorem extends to non-0 error rates, just with a factor depending on the accuracy modifying the footprint. We also added a pointer to it in the main paper.

W2: experiments ..the traces.

Q2. How .. decoded?

We added experiments (Appendix I) with LLM judges (Kimi K3, GPT-6.1 Terra).

1st, we judge whether there is hidden reasoning, by asking LLMs to discriminate between CoTs produced in the prompting experiments (covert task is given) and CoTs where only the public task was solved. Accuracy was near chance for Agnostic protocols but higher in Informative CoT mode (Table 20, Appendix I.2.1), [consistent with theory].

2nd, we checked if we can recover the covert input from only the CoT (Table 21, Appendix I.2.2). Accuracy was 0 in the Agnostic mode, (consistent with theory). In the Informative mode, accuracy was up to 100% for Special Tokens, and 10% for Piggybacking (random chance). This suggests that, from the generated CoTs, the only information about the hidden input that can be extracted by LLM judges is exactly determined by our protocol design.

Re encrypted CoTs: they could indeed be flagged as suspicious. Under its cryptographic assumption, our theorem prevents efficient recovery of the input, but it does not guarantee that ciphertext could be made to resemble ordinary reasoning. Section 5 explicitly discusses this limitation.

W3: While ... supervision. Q3: The ablations .. supervision?

We agree and with our result, we simply want to alert the community about the plausibility of such training, and thus motivate the potential need for whitebox certifications to ensure that model developers are not doing so. Whether or not such behavior can emerge without supervision is also an interesting question, but even if the answer turns out to be no, we hope our result helps increase the weight of arguments in favor of introducing mandatory white box certifications of LLMs.

W4: The experiments .. reuse.

For finetuning, I[Xcovert,CoT]I[X_{covert}, CoT] is fixed by design: small in agnostic and on the order of NN in the informative mode. We added Fig 37, in Appendix H, with exact numbers.

Measuring MI in prompting is more difficult as the CoTs are open-ended. Although the LLM judge experiments could be taken as a proxy, as they provide a lower bound by measuring what information can be extracted by LLMs.

We would be happy to make some of these discussion points appear more prominently in the main text, if the reviewers think that it would help in contextualizing the results better for readers.

[1] Csordás et al Do language models use their depth efficiently? NeurIPS 2025

[2] Greenblatt. Recent llms can do 2-hop and 3-hop latent (no cot) reasoning on natural facts. 2026.

[3] Biran et al Hopping too late: Exploring the limitations of large language models on multi-hop queries. EMNLP 2024

Thank you for the rebuttal. The additional theoretical analysis and experiments address some of my concerns. However, the connection between the theoretical and experimental tasks remains unclear, and the Piggybacking results still lack an answer-transformation baseline. Differences in supervision across LoRA finetuning protocols also remain unresolved. I therefore maintain my original rating.

Thank you for your response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.