- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces StegoBench, a benchmark evaluating 18 text steganography schemes across diverse language models and training methods. Results indicate that supervised fine-tuning effectively learns local token-level encoding, whereas more complex schemes frequently collapse. Additionally, steganographic performance relies more on training data quality than model scale.
Critical strengths
This paper systematically evaluates 18 established text steganography schemes across diverse language models and training paradigms, offering key empirical insights into their trainability, detectability, and failure modes.
Critical weaknesses
- Method Coverage is not sufficiently up to date. The 18 selected schemes rely almost entirely on algorithms from 2023 or earlier, overlooking substantial methodological advances from 2024–2026. Although the paper cites several recent studies on LLM steganography, failing to evaluate these newer encoding methods in the benchmark is a critical omission.
- StegoBench’s unified SFT framework may introduce evaluation bias. Several modern steganographic methods rely on inference-time mechanisms, such as token distribution access, sampling control, or shared randomness. Forcing models to internalize these runtime processes through SFT likely misrepresents the original algorithms and explains their poor benchmark performance. Consequently, more rigorous and equitable comparison methods are needed for these approaches.
Questions for authors
- Could you provide more results from recent methods about linguistic steganography, especially those LLM-based steganography?
- Also, it is unconvincing to me that all these modern steganography method collapses and fails. Is there more strict analysis on the benchmark settings?
3 comments
Response to Rebuttal from Reviewer VkGY
After reviewing the rebuttal, I am pleased to see additional recent work and have decided to raise my overall rating to 2. However, I remain unconvinced about using a benchmark on supervised steganography, which appears to have limited application.
I also noticed a strong indication of AI use in the revised paper, especially in Sections 4 and 5 (Results and Analysis). The content reads more like a data report and provides very little information. I suggest the authors and AC take this issue seriously.
Response to Reviewer VkGY on new revision
Thank you very much for your helpful feedback! During the course of editing the paper for the purpose of this rebuttal phase, we relied too much on AI assistance to incorporate updated results. This included using AI to identify all numerical values that needed to be updated, and then having it help update those sections. We have done another revision of the paper with this in mind.
Response to Reviewer VkGY
Q1
Since the original submission of this paper, we have studied TrojanStego (EMNLP 2025) and SparSamp (USENIX Security 2025).
We will add both of these schemes to our final paper.
Q2
The benchmark measures one well-defined training method, supervised fine-tuning, and our conclusions are scoped to that training method. Other training methods may produce model organisms of modern generative distribution-coupled stego schemes. Under strict SFT, we have observed the same pattern across nearly all distribution-coupled stego schemes. Not every modern scheme fails. TrojanStego (2025), which encodes bits through vocabulary choices rather than by sampling from a model’s distribution, is Trainable under the same SFT setup. Distribution-coupled schemes rely on sampling directly from a model's distribution. As SFT pressure is applied to the model, that distribution shifts. This means that if you create a sample right before you use it with the model's current distribution, by the time the model has been trained on that sample, the underlying distribution has shifted in a way that would not produce that same sample. It is constantly chasing a moving target. Drift is not the full explanation of what is happening, however: removing it through self-distillation does not prevent collapse.
We have run numerous experiments to try and resolve this problem. None produced a working organism.
1. EOS/Special-token masking
Mask EOS and special tokens out of the loss, so the model can’t fall back on them. Instead of spamming special tokens, the model falls back to repeating extremely common English words.
2. Iterative self-distillation
Regenerate targets from the current checkpoint, and train on them over multiple rounds. The model still collapses into repetitive phrases and special tokens.
3. RL with the scheme’s decoder in the reward loop
Trained with REINFORCE, using a reward of 1 - BER from the scheme’s decoder plus a KL penalty. The output didn’t collapse, but the model wrote the secret as plaintext instead of encoding it.
4. Cross-checkpoint decoding
This was not a training experiment, but instead an analysis of the resulting artifacts. We used the same checkpoint for both encoding and decoding a message using the scheme’s implementation, and found that the payload could be recovered 75-100% of the time. We then did the same experiment using the base model and the training artifact, and found that it degrades to chance: BER 0.49-0.52 for Discop, Sallee, RNN-Stega and Moraldo. Wayner kept a weak signal (0.36-0.40).