- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how test-time training (TTT) affects the safety alignment of large language models. The authors consider three attack settings, namely self-supervised, few-shot, and generation-phase TTT, and examine how attackers can exploit parameter updates during inference to bypass safety guardrails. Experiments across different models, finetuning configurations, and safety benchmarks show that TTT can substantially increase jailbreak success rates under certain conditions. The paper also introduces validity-aware evaluation to address misleading attack success estimates caused by degenerate generations and explores a preliminary defense based on perplexity shifts.
Critical strengths
- The research problem is practically relevant. As inference-time parameter adaptation receives increasing attention, understanding whether existing safety alignment mechanisms can withstand such updates is important. The paper provides a systematic investigation of three TTT attack settings.
- The experimental evaluation is extensive. The authors evaluate models of different scales under both LoRA and full finetuning, and further validate their findings across multiple safety benchmarks and through an external fine-tuning API. Overall, the experiments are fairly comprehensive.
- The analysis of evaluation reliability is valuable. The authors identify that TTT can produce degenerate generations that cause conventional safety judges to overestimate attack success rates. They address this issue through validity-aware evaluation, which improves the reliability of the reported results.
Critical weaknesses
- The distinction from existing harmful finetuning attacks remains unclear. The most effective few-shot and generation-phase attacks both rely on affirmative target prefixes associated with harmful requests to update model parameters, making them closely related to existing harmful finetuning attacks. The paper does not sufficiently clarify which aspects of the observed attack success are specific to TTT's dynamic adaptation mechanism and which reflect the already known vulnerabilities of safety alignment under harmful finetuning.
- The effectiveness of clean self-supervised TTT is inconsistent. Without harmful targets or additional adversarial prefixes, clean-prompt self-supervised TTT significantly affects only some models, while several larger models largely maintain their original safety performance. This limits the generality of the paper's claim that TTT undermines safety alignment.
- The practical relevance of the deployment setting remains somewhat limited. Since dedicated production TTT APIs are not currently available, the authors use an existing fine-tuning API to simulate transient parameter updates during inference. Although this evaluation is useful, the setting still differs from actual online adaptation mechanisms. The extent to which these findings generalize to real-world TTT deployments needs to be clarified.
- The emphasis on ASR@10 may give an overly favorable impression of attack effectiveness. ASR@10 allows ten generations per attack prompt and can therefore be substantially higher than the success rate of a single generation. Although the authors also report mean ASR, the main text should more clearly distinguish between single-generation success, repeated sampling, and results obtained under different adaptation-step budgets.
- The proposed defense has limited robustness. The perplexity-based detector performs well against non-adaptive attacks, but the appendix shows that adaptive attackers can substantially reduce detection rates while maintaining non-trivial attack success. This suggests that the method is better viewed as a preliminary detection approach rather than a robust practical defense. The additional inference cost associated with the private harmful holdout also deserves further discussion.
Questions for authors
- The relationship between TTT and harmful finetuning remains unclear. What is the fundamental difference between few-shot or generation-phase TTT and conventional harmful finetuning? Can the authors provide a more direct comparison under the same amount of training data and optimization budget to establish the security risks specific to TTT?
- Why does clean-prompt self-supervised TTT have substantially different effects across models? Are these differences mainly due to model scale, safety alignment methods, learning rates, or parameter update configurations?
- What specific capabilities does an attacker need when using a fine-tuning API to simulate TTT? If an online service restricts users' control over training data, update objectives, or model parameters, would all three attack settings still be feasible?
- Can the authors further discuss the difference between mean ASR and ASR@10, as well as the influence of adaptation-step selection on the final conclusions? In particular, would the attacks remain effective in practical settings where only a single generation or a limited number of queries is allowed?
- The adaptive attacks in the paper already show a substantial reduction in the detector's effectiveness. How do the authors view the practical scope of the proposed defense? Furthermore, would the size of the private harmful holdout and the additional inference cost affect its feasibility in online deployments?
4 comments
W5/Q5. We present the detector as “a first step toward defense” (abstract), and the introduction states that it alone is “unlikely to withstand future adaptive attacks”. It is not the main focus of the paper. To make this clearer, the revision calls it a detector rather than a defense, and the title of Section 5 is now “A first step toward defense: perplexity-based detection”. The adaptive attack in Appendix A.5 also changes the training loss, which our threat models assume the provider fixes, so we report it as an upper bound. Regarding the inference costs, the detector needs no generation and no gradients. The base model’s holdout perplexities are computed once. Each request then adds one batched forward pass of the adapted model over 200 short sequences (100 harmful, 100 clean). In our runs this takes about 2 s per request, 9-11% of the total time for TTT (5 steps), generation (10 samples) and detection, on all four models. For the holdout size, with 50 harmful items instead of 100, the detector separates harmful from benign requests with AUROC of at least 0.98 in all 20 random draws on every model. With 10 items, results vary between draws (down to 0.64). The threshold must be recalibrated for each size. Both results are in Appendix A.7 of the revision (Table 18 for the holdout size).
[1] Yuksekgonul et al., Learning to Discover at Test Time
[2] Wang et al., OpenClaw-RL: Train Any Agent Simply by Talking
[3] Beyer et al., LLM-Safety Evaluations Lack Robustness
[4] Beyer et al., Sampling-aware Adversarial Attacks Against Large Language Models
[5] Khattar et al., Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities
[6] Yang et al., Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
Official Comment by Reviewer rj3T
Thank you for the detailed response. I appreciate the authors' efforts to address my concerns, and I find some of the clarifications helpful. However, I still have concerns about what distinguishes the security risks specific to TTT from those of existing harmful finetuning attacks. While the response clarifies the differences in attack settings, it does not fully address my concerns about the novelty of the underlying vulnerability. In addition, the effectiveness of clean self-supervised TTT remains highly model-dependent. Overall, although the response has addressed some of my concerns, the main issue remains unresolved. I still maintain my original rating.
Thank you for the follow-up. The novelty of our work is the setting: the question we study is whether safety still degrades under TTT, where the attacker sends a single inference request, the update uses the query or a few short prefixes for at most ten steps, and the weights are reset afterwards (Table 1). This is a much more restricted setting than fine-tuning, and we believe that the answer does not follow from prior fine-tuning results, which train on full harmful answers for many more steps and serve a persistent model. Fine-tuning is just one of the techniques used to perform the weight update, not our proposed approach. We agree on the model-dependence, and the revision states it in the abstract, Section 4.2 and the limitations. We understand that you weigh the novelty differently, and we thank you for the discussion.
Thank you for your time in reviewing the paper. We appreciate that you found the paper’s topic relevant, the evaluation extensive and the validity-aware evaluation valuable. In the following we address your raised concerns:
W1/Q1. To make clearer the differences between harmful fine-tuning (jailbreak fine-tuning in the paper) and TTT, we have now included the following table in the revision (Table 1):
Specifically, in jailbreak fine-tuning, users typically need to provide the model with an example set of (prompt, full harmful answer) pairs and, after adapting the weights on this example set, provide the prompts to evaluate. On the other hand, TTT needs the user prompt, optionally with a few (prompt, prefix) pairs or a short affirmative prefix (like, “Sure, here is …”), and the model is adapted only on these data. After generating the reply to the user prompt, the model’s weights are reset to their initial state before the next user prompt. Another relevant detail is the number of steps. TTT requires few gradient updates (we study up to 10 steps), against 100 to 200 in [6]. Fine-tuning is just one of the techniques used to perform the weight update. The threat models we define come from how TTT is used, not from fine-tuning attacks. The same formulation covers other TTT instances, such as the RL setting of Khattar et al. [5], which corresponds to our generation-phase threat model.
W2/Q2. We agree that the effect is model-dependent, and we report its effects as modest (Section 4.2). We corrected the conclusion, which stated near-100% success under every threat model, and we are happy to more carefully revise remaining parts where wording points to a general success across threat models. Regarding why the effect of clean-prompt self-supervised TTT depends on the model, our experiments cannot isolate a single factor. We believe it depends on each model’s alignment procedure, which differs across model families, and also identifying the best learning rate matters to allow the model weights to move in a region where safety guardrails are avoided (far from the original aligned weights) without overfitting on the few tokens of the prompt. Separating these factors needs further study, and the revision states this in the limitations (Section 7).
W3/Q3. Fine-tuning APIs are already used to run TTT: [1] performs its test-time training through Tinker, the API we evaluate. Deployed systems also update weights from user interaction: [2] trains an agent online from live conversations and supports Tinker as a backend. Section 3 defines the different access each threat model needs:
None of the three requires control of the update objective (since the provider fixes the loss), or model weights (the provider runs the update and returns only the answer). The only additional surface the attacker can leverage are the hyperparameters exposed by the API (learning rate, number of steps, and LoRA rank).
W4/Q4. Mean ASR is the single-generation success rate: the fraction of individual samples judged harmful. We report it next to ASR@10 in every full table (Tables 8, 9, 11, 13). Under LoRA, a single generation succeeds 75% of the time for few-shot TTT and 74% for generation-phase TTT, averaged over seven models (Table 8). However, we put an emphasis on ASR@10 because sampling again costs an attacker little, so a defense should hold under repeated sampling [3,4]. Regarding the step selection, fixing 5 steps for every model gives 94.9% and 89.4% average ASR@10 under LoRA, against 95.1% and 93.4% with the best step (Table 8).