- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a decision-theoretic framework for detecting encoded reasoning in LLMs without a known benign text distribution. It generalizes V-information to task utilities and defines the “steganographic gap” as the difference in utility gains a reasoning trace provides to a Receiver and a trusted Sentinel. Controlled experiments show that this metric tracks decoding asymmetries and detects encodings missed by the evaluated LLM judges and paraphrasing methods.
Critical strengths
- The proposed method enables auditing without a known distribution of benign text, addressing a key limitation of classical steganalysis for LLM reasoning.
- The experiments cover diverse encoding schemes, including semantic channels missed by the evaluated baselines.
Critical weaknesses
- The experiments do not sufficiently rule out that the measured gap arises from prompts failing to elicit the Sentinel’s decoding ability. This could make recoverable information appear hidden. Although Appendix I.5 examines alternative prompts and additional sampling, these checks cover only one Rot13 setting.
- The proposed metric may produce false positives when the Sentinel is less capable than the Receiver at understanding benign reasoning traces. Although the paper acknowledges this issue, it does not adequately quantify the resulting false-positive rates.
Questions for authors
- How should Sentinel capabilities be matched and false positives calibrated on benign reasoning traces?
- Do the elicitation robustness results extend beyond Rot13 to learned and semantic encodings?
- Can an audit-aware Receiver suppress the measured gap while retaining its ability to exploit hidden information?
3 comments
Response to Reviewer 8fKD (1/2): Audit-aware sandbagging
Audit-aware sandbagging
An audit-aware Receiver can suppress the empirical gap by deliberately underperforming. This is the capability-elicitation problem faced by behavioural evaluations, and we explicitly discuss it in Appendix A.4.
Our framework provides additional controls, which limit the detrimental effect of sandbagging. Simply ignoring the trace (or its encoded meaning) sacrifices its contribution to performance, which can be checked through complementary evaluations. If within this specific behavioural evaluation the performance of the Receiver model underperforms its earlier observed capabilities, such underperformance can be picked up as suspicious. Further, the Auditor can conceal the Sentinel’s identity and evaluate against multiple Sentinels, making it harder for the Receiver to tailor its behaviour to the comparison. These measures can be combined with stronger capability elicitation and consistency checks across evaluation contexts.
Overall, we note that while deliberate underperformance changes the observed estimate, it does not remove the Receiver’s underlying decoding capability. The relevant challenge (shared by all behavioural evaluations) is therefore to elicit that capability during the audit. We believe that future research into capability elicitation will thus also make our auditing framework more robust.
**We thank again for your positive review. We hope that our response was satisfactory. We have also provided extensive response to the other reviewers that you may also find informative. Please let us know if you have any further questions."
References
[1] Korbak et al. (2025). https://arxiv.org/html/2507.11473v2
[2] Pal et al. (2026). https://arxiv.org/html/2601.11517v1
[3] Cheng et al. (2026). https://arxiv.org/html/2605.28913v1
[4] Anthropic. https://alignment.anthropic.com/2025/distill-paraphrases/
Thank you for your response. Most of my concerns have been addressed, and I will maintain my positive score.
Response to Reviewer 8fKD (1/2): additional analysis of elicitation robustness and discussion on sentinel selection
Thank you for your positive review. For your information, we have now included an additional experiment that evaluates our method on naturally emerging encoded communication in a multi-agent setting. We describe this evalaution in the common response above.
Elicitation robustness (W1 and Q2)
We have extended the elicitation robustness analysis in Appendix I.5 (Tables 12–13) to the learned CoinFlip codebooks and all six loan-summary channels, including the semantic encodings. Substantial steganographic gaps persist in both settings, even under the additional prompts and ten-sample majority voting, extending the robustness results beyond Rot13. For loan summaries, majority voting changes Sentinel accuracy by at most 0.40 percentage points across all 30 conditions. Thank you for this suggestion, as we believe it helps to improve the robustness of our results.
Sentinel selection and false positives (W2 and Q1)
Sentinel capabilities matching: The Sentinel should be selected for its ability to use the public information relevant to the auditing task. For example, when studying encodings acquired during RL, the corresponding pre-RL model provides a natural reference (see, for example, our DeepSeek-R1 evaluation or Anthropic’s evaluation [4]). However, we strongly believe that this constraint can be reasonably fulfilled by a broader range of models. In Appendix D.3 (Figure 5), we show that DeepSeek-V3 and three Qwen models of different sizes extract essentially the same amount of task-relevant information from DeepSeek-R1’s AIME reasoning. The fact that unencoded CoTs can be readable and usable by reasonably capable external agents is precisely the property that makes CoT valuable for interpretability and monitoring (Korbak et al., 2025 [1]). While capability differences are a reasonable potential confound, our comparisons across Sentinel sizes and families show that they are not a practical obstacle here. Other works have also shown this transferability of CoTs: Pal et al. (2026) [2] demonstrate that CoTs transfer across models, and Cheng et al. (2026) [3] show that weaker models given a stronger model’s complete CoT approach its performance. Thus overall, we do not see finding a suitable capability-matched Sentinel as a significant practical challenge.
The reviewer may find our response to similar concerns of reviewer yffM also informative regarding this issue which further expands on this point that while this is indeed a resonable theoretical concern, we found it to be an easily addressable and a limited concern in practice. Across various settings, where the signal Z was not steganographic, our method was able to correctly conclude so while still detecting encoded reasoning where present. For example, in Table 1, our method detects all six planted encoding schemes while reporting zero gap for the unencoded control. Similarly, in the MONA-inspired loan experiment (Appendix J.4), it correctly finds no positive gap for the RL-learned policy.