- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FragToken, a training-time resource-consumption attack that increases LLM inference costs by inducing noncanonical token generation. The main observation is that multiple token sequences can decode to the same text, allowing a model to generate substantially more tokens without proportionally increasing the visible response length. Unlike existing resource-consumption attacks that rely on malicious prompts, triggers, or excessively long outputs, FragToken aims to introduce persistent computational overhead across ordinary user requests.
The proposed method combines source model self-distillation, capacity-aware fragmentation budgeting, and BPE-Alignedmerging to construct fragmented training labels while preserving model utility. The authors evaluate the approach on four LLMs and three benchmarks, reporting average token inflation ratios of 1.99–2.46 with relatively small changes in average benchmark accuracy. They also report a 2.44 x increase in amortized generation time in their serving experiments and evaluate the method against existing resource-consumption attacks and several detection methods.
Critical strengths
1. Interesting and novel attack mechanism. The paper introduces an interesting approach to increasing LLM inference costs by exploiting noncanonical token generation rather than producing longer responses, wthout the need of triggers.
2. Well motivated method with useful ablations. The proposed components are motivated by the observation that full token atomization significantly hurts model utility. The ablations provide good evidence that self-distillation, capacity-aware budgeting, and BPE-Aligned merging help balance token inflation and accuracy.
3. Convincing evidence of inference overhead. The paper evaluates four different LLMs and consistently achieves around two times token inflation. Importantly, the authors also measure actual generation time, reporting a 2.44 x increase while the visible response length decreases. This supports the practical relevance of the underlying mechanism.
Critical weaknesses
1. The stealth evaluation misses an important detection baseline. The paper emphasizes that FragToken is difficult to detect, but the evaluated defenses mostly focus on suspicious inputs or visible output patterns. In the proposed threat model, the defender hosts the model and can observe generated token IDs. A natural defense would be to compare the generated token count with the canonical tokenization of the decoded output, essentially using the paper's own TIR metric. Without evaluating such a detector, I am not convinced that the attack is actually difficult for a model provider to detect.
2. Limited evaluation of realistic deployment settings and generalization. The main serving and detection experiments use only 50 Alpaca queries, which seems relatively small for supporting broad conclusions about deployment impact. It is unclear how consistently the attack would affect diverse real-world traffic, particularly under different serving loads and output-token limits.
3. Utility preservation is not fully convincing. Although the average accuracy degradation is relatively small, GSM8K accuracy consistently drops by approximately 3-6 percentage points across all four models. This seems significant given that preserving model utility is central to the attack. The evaluation is also limited to three benchmarks, making it difficult to assess whether the modified models retain their general capabilities.
Questions for authors
Token-aware detection: Have the authors considered detecting FragToken by comparing generated token counts with canonical reencoding lengths (i.e., using the proposed TIR)? How effective would such a simple detector be?
Utility preservation: Can the authors explain why GSM8K accuracy consistently drops across all four models? Also, what explains the large OpenBookQA improvement for OLMo-2?
Generalization and deployment: How well does the attack generalize to more diverse real-world queries and serving conditions, given the relatively small serving evaluation and the lower TIR on OOD benchmarks?
4 comments
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end, we wanted to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns.
If you have any remaining questions or feedback, we would greatly appreciate the opportunity to address them. Your feedback is valuable to us, and we hope to resolve any outstanding concerns before the discussion concludes.
Thank you again for your time and effort in reviewing our paper.
Best regards,
The Authors
Thank you for the new benchmarks and detection analysis. The TIR results show a simple detector flags many FragToken outputs at near zero FPR (69% TPR at threshold 1.1 for B=0.85), so the stealth claim holds only for the lower inflation settings. The GSM8K drop also remains (80.85% even with GSM8K-trained data vs. 84.46% original). My assessment is unchanged.
Dear Reviewer,
Thank you very much for your continued feedback and for acknowledging our additional experiments. We would like to offer two further clarifications.
TIR-based Detection. At β = 0.85, TIR-based detection achieves 69.08% TPR at a threshold of 1.1, leaving nearly one-third of FragToken outputs undetected. Reducing fragmentation further lowers TPR to 28.76% at β = 0.95. Importantly, β is an attacker-controlled hyperparameter that can be automatically tuned to favor stealthier configurations while retaining non-negligible token inflation. This demonstrates FragToken's flexibility for adaptive defense.
GSM8K Utility. We acknowledge the utility degradation on GSM8K, although the overall impact remains relatively modest (80.85% vs. 84.46%). We hypothesize that multi-step mathematical reasoning is particularly sensitive to accumulated generation errors. In-distribution training can partially mitigate this degradation, while FragToken maintains comparable performance across most other evaluated capabilities.
We sincerely appreciate your thoughtful feedback and the opportunity to clarify our work further.
Dear Reviewer,
Thank you for your constructive comments. We address the main concerns below. Unless otherwise specified, all experiments use Llama-3.1-8B and the same main setting.
W1. TIR-based Detection
Beyond the content-based defenses in Table 4, we evaluate adaptive TIR-based monitoring. First, our experiments reveal an inherent trade-off between detection sensitivity and false positives. Prior work [1] and our experiments demonstrate that even benign LLMs can naturally generate noncanonical token sequences without manipulation. Consequently, setting a low TIR threshold may incorrectly flag benign generations. Conversely, increasing the threshold will cause many FragToken-generated responses to evade detection. Second, FragToken allows attackers to flexibly reduce TIR to evade fixed detection thresholds while retaining non-negligible token inflation (6.8%–31.9%). Since this overhead persists across ordinary user requests, it can accumulate into substantial resource consumption at scale.
W2. Utility
Additional Benchmarks. We extend the evaluation to 8 additional benchmarks covering instruction following, code generation, and long-form text generation. The results show that FragToken maintains comparable performance.
GSM8K Degradation. We hypothesize that GSM8K is sensitive to accumulated reasoning errors and the Alpaca–mathematics distribution gap. Fine-tuning on self-generated responses to GSM8K training questions improves accuracy by 2.38% and raises TIR from 1.95 to 2.18, supporting training-data alignment as a way to mitigate utility loss.
OLMo-2 Improvement. Regarding the OBQA improvement on OLMo-2 (50.60% to 71.20%), we speculate that FragToken helps the model better elicit and express knowledge acquired during pretraining and SFT, improving its ability to leverage existing knowledge in answering domain-related questions.
W3. Generalization and Stability
Generalization to Real-World Queries. We clarify that the 50 Alpaca queries were used only for comparisons with existing attacks. We further evaluate FragToken on the GSM8K, PIQA, and OBQA in the paper. Additionally, we evaluate Llama-3.1-8B trained under our main FragToken setting on 1,000 real-world user requests from WildChat-1M, ShareLM, and PRISM. It demonstrates FragToken's generalization to real-world user inputs.
Robustness to Output Token Limits. With other settings fixed, TIR remains approximately 2.43 across output limits of 2,048–16,384 tokens, demonstrating its robustness.
Robustness to Serving Loads. Across three benchmarks, FragToken maintains a TIR of approximately 2.46 and increases latency while reducing throughput under all tested concurrency levels, demonstrating its robustness.
Reference
[1] Vieira T, Liu T, Pasti C, et al. Language models over canonical byte-pair encodings, Proceedings of ICML, 2025.