acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Training Undermines Safety Guardrails

Abstract

Test-Time Training (TTT) is an emerging paradigm that enables models to adapt their parameters during inference, improving performance on tasks such as few-shot learning, retrieval-augmented generation, and complex reasoning. However, this dynamic adaptation introduces new vulnerabilities that adversaries can exploit to jailbreak models. We identify three threat models for TTT and demonstrate how attackers can leverage them to bypass safety filters. Our results show that TTT can significantly increase the Attack Success Rate (ASR) and the ASR over 10 generation trials (ASR@10). For example, under LoRA, the few-shot and generation-phase threat models achieve an average ASR@10 of 95% and 93% respectively, across models from different families and scales. These vulnerabilities transfer to production fine-tuning APIs. We also show that TTT-induced overfitting can produce degenerate outputs that inflate ASR under standard judges, and propose a validity-aware evaluation to correct for this. Our findings suggest that TTT exposes a new attack surface, strengthens attacks, and undermines existing safety guardrails. As a first step toward defense, we propose a lightweight provider-side detector that flags TTT requests via the perplexity shift on a private harmful holdout, but robust deployment will ultimately require dynamic alignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak attacks, test-time training, safety alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rj3TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how test-time training (TTT) affects the safety alignment of large language models. The authors consider three attack settings, namely self-supervised, few-shot, and generation-phase TTT, and examine how attackers can exploit parameter updates during inference to bypass safety guardrails. Experiments across different models, finetuning configurations, and safety benchmarks show that TTT can substantially increase jailbreak success rates under certain conditions. The paper also introduces validity-aware evaluation to address misleading attack success estimates caused by degenerate generations and explores a preliminary defense based on perplexity shifts.

Critical strengths
  1. The research problem is practically relevant. As inference-time parameter adaptation receives increasing attention, understanding whether existing safety alignment mechanisms can withstand such updates is important. The paper provides a systematic investigation of three TTT attack settings.
  2. The experimental evaluation is extensive. The authors evaluate models of different scales under both LoRA and full finetuning, and further validate their findings across multiple safety benchmarks and through an external fine-tuning API. Overall, the experiments are fairly comprehensive.
  3. The analysis of evaluation reliability is valuable. The authors identify that TTT can produce degenerate generations that cause conventional safety judges to overestimate attack success rates. They address this issue through validity-aware evaluation, which improves the reliability of the reported results.
Critical weaknesses
  1. The distinction from existing harmful finetuning attacks remains unclear. The most effective few-shot and generation-phase attacks both rely on affirmative target prefixes associated with harmful requests to update model parameters, making them closely related to existing harmful finetuning attacks. The paper does not sufficiently clarify which aspects of the observed attack success are specific to TTT's dynamic adaptation mechanism and which reflect the already known vulnerabilities of safety alignment under harmful finetuning.
  2. The effectiveness of clean self-supervised TTT is inconsistent. Without harmful targets or additional adversarial prefixes, clean-prompt self-supervised TTT significantly affects only some models, while several larger models largely maintain their original safety performance. This limits the generality of the paper's claim that TTT undermines safety alignment.
  3. The practical relevance of the deployment setting remains somewhat limited. Since dedicated production TTT APIs are not currently available, the authors use an existing fine-tuning API to simulate transient parameter updates during inference. Although this evaluation is useful, the setting still differs from actual online adaptation mechanisms. The extent to which these findings generalize to real-world TTT deployments needs to be clarified.
  4. The emphasis on ASR@10 may give an overly favorable impression of attack effectiveness. ASR@10 allows ten generations per attack prompt and can therefore be substantially higher than the success rate of a single generation. Although the authors also report mean ASR, the main text should more clearly distinguish between single-generation success, repeated sampling, and results obtained under different adaptation-step budgets.
  5. The proposed defense has limited robustness. The perplexity-based detector performs well against non-adaptive attacks, but the appendix shows that adaptive attackers can substantially reduce detection rates while maintaining non-trivial attack success. This suggests that the method is better viewed as a preliminary detection approach rather than a robust practical defense. The additional inference cost associated with the private harmful holdout also deserves further discussion.
Questions for authors
  1. The relationship between TTT and harmful finetuning remains unclear. What is the fundamental difference between few-shot or generation-phase TTT and conventional harmful finetuning? Can the authors provide a more direct comparison under the same amount of training data and optimization budget to establish the security risks specific to TTT?
  2. Why does clean-prompt self-supervised TTT have substantially different effects across models? Are these differences mainly due to model scale, safety alignment methods, learning rates, or parameter update configurations?
  3. What specific capabilities does an attacker need when using a fine-tuning API to simulate TTT? If an online service restricts users' control over training data, update objectives, or model parameters, would all three attack settings still be feasible?
  4. Can the authors further discuss the difference between mean ASR and ASR@10, as well as the influence of adaptation-step selection on the final conclusions? In particular, would the attacks remain effective in practical settings where only a single generation or a limited number of queries is allowed?
  5. The adaptive attacks in the paper already show a substantial reduction in the detector's effectiveness. How do the authors view the practical scope of the proposed defense? Furthermore, would the size of the private harmful holdout and the additional inference cost affect its feasibility in online deployments?
4 comments

Thank you for your time in reviewing the paper. We appreciate that you found the paper’s topic relevant, the evaluation extensive and the validity-aware evaluation valuable. In the following we address your raised concerns:

W1/Q1. To make clearer the differences between harmful fine-tuning (jailbreak fine-tuning in the paper) and TTT, we have now included the following table in the revision (Table 1):

Jailbreak fine-tuningJailbreak TTT
Attacker’s accessUploads a dataset, runs a jobSends one inference request
Training dataExample set (100 pairs in Yang et al. [6])The query, or 1-5 pairs with a 10-20 token prefix
Link to the test querySet built in advance, independent of later queriesBuilt per request, can contain the test query itself (self-supervised, generation-phase)
Harmful answers neededUsually (benign-data variants exist)No, the model produces them
StepsUntil it works (100-200 in Yang et al. [6])1-10 per request, chosen by the attacker
Update servesAll later queriesOne query, then reset
Provider can inspectTraining set and finished modelRequest text, and any model test must run per request

Specifically, in jailbreak fine-tuning, users typically need to provide the model with an example set of (prompt, full harmful answer) pairs and, after adapting the weights on this example set, provide the prompts to evaluate. On the other hand, TTT needs the user prompt, optionally with a few (prompt, prefix) pairs or a short affirmative prefix (like, “Sure, here is …”), and the model is adapted only on these data. After generating the reply to the user prompt, the model’s weights are reset to their initial state before the next user prompt. Another relevant detail is the number of steps. TTT requires few gradient updates (we study up to 10 steps), against 100 to 200 in [6]. Fine-tuning is just one of the techniques used to perform the weight update. The threat models we define come from how TTT is used, not from fine-tuning attacks. The same formulation covers other TTT instances, such as the RL setting of Khattar et al. [5], which corresponds to our generation-phase threat model.

W2/Q2. We agree that the effect is model-dependent, and we report its effects as modest (Section 4.2). We corrected the conclusion, which stated near-100% success under every threat model, and we are happy to more carefully revise remaining parts where wording points to a general success across threat models. Regarding why the effect of clean-prompt self-supervised TTT depends on the model, our experiments cannot isolate a single factor. We believe it depends on each model’s alignment procedure, which differs across model families, and also identifying the best learning rate matters to allow the model weights to move in a region where safety guardrails are avoided (far from the original aligned weights) without overfitting on the few tokens of the prompt. Separating these factors needs further study, and the revision states this in the limitations (Section 7).

W3/Q3. Fine-tuning APIs are already used to run TTT: [1] performs its test-time training through Tinker, the API we evaluate. Deployed systems also update weights from user interaction: [2] trains an agent online from live conversations and supports Tinker as a backend. Section 3 defines the different access each threat model needs:

  • Self-supervised (Section 3.1) only needs the query.
  • Few-shot (Section 3.2) needs the support set (pairs of query and affirmative prefix).
  • Generation-phase (Section 3.3) needs the query and the affirmative prefix.

None of the three requires control of the update objective (since the provider fixes the loss), or model weights (the provider runs the update and returns only the answer). The only additional surface the attacker can leverage are the hyperparameters exposed by the API (learning rate, number of steps, and LoRA rank).

W4/Q4. Mean ASR is the single-generation success rate: the fraction of individual samples judged harmful. We report it next to ASR@10 in every full table (Tables 8, 9, 11, 13). Under LoRA, a single generation succeeds 75% of the time for few-shot TTT and 74% for generation-phase TTT, averaged over seven models (Table 8). However, we put an emphasis on ASR@10 because sampling again costs an attacker little, so a defense should hold under repeated sampling [3,4]. Regarding the step selection, fixing 5 steps for every model gives 94.9% and 89.4% average ASR@10 under LoRA, against 95.1% and 93.4% with the best step (Table 8).

W5/Q5. We present the detector as “a first step toward defense” (abstract), and the introduction states that it alone is “unlikely to withstand future adaptive attacks”. It is not the main focus of the paper. To make this clearer, the revision calls it a detector rather than a defense, and the title of Section 5 is now “A first step toward defense: perplexity-based detection”. The adaptive attack in Appendix A.5 also changes the training loss, which our threat models assume the provider fixes, so we report it as an upper bound. Regarding the inference costs, the detector needs no generation and no gradients. The base model’s holdout perplexities are computed once. Each request then adds one batched forward pass of the adapted model over 200 short sequences (100 harmful, 100 clean). In our runs this takes about 2 s per request, 9-11% of the total time for TTT (5 steps), generation (10 samples) and detection, on all four models. For the holdout size, with 50 harmful items instead of 100, the detector separates harmful from benign requests with AUROC of at least 0.98 in all 20 random draws on every model. With 10 items, results vary between draws (down to 0.64). The threshold must be recalibrated for each size. Both results are in Appendix A.7 of the revision (Table 18 for the holdout size).

[1] Yuksekgonul et al., Learning to Discover at Test Time

[2] Wang et al., OpenClaw-RL: Train Any Agent Simply by Talking

[3] Beyer et al., LLM-Safety Evaluations Lack Robustness

[4] Beyer et al., Sampling-aware Adversarial Attacks Against Large Language Models

[5] Khattar et al., Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

[6] Yang et al., Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models

Reviewer_rj3TOpenReview ↗

Official Comment by Reviewer rj3T

Thank you for the detailed response. I appreciate the authors' efforts to address my concerns, and I find some of the clarifications helpful. However, I still have concerns about what distinguishes the security risks specific to TTT from those of existing harmful finetuning attacks. While the response clarifies the differences in attack settings, it does not fully address my concerns about the novelty of the underlying vulnerability. In addition, the effectiveness of clean self-supervised TTT remains highly model-dependent. Overall, although the response has addressed some of my concerns, the main issue remains unresolved. I still maintain my original rating.

Thank you for the follow-up. The novelty of our work is the setting: the question we study is whether safety still degrades under TTT, where the attacker sends a single inference request, the update uses the query or a few short prefixes for at most ten steps, and the weights are reset afterwards (Table 1). This is a much more restricted setting than fine-tuning, and we believe that the answer does not follow from prior fine-tuning results, which train on full harmful answers for many more steps and serve a persistent model. Fine-tuning is just one of the techniques used to perform the weight update, not our proposed approach. We agree on the model-dependence, and the revision states it in the abstract, Section 4.2 and the limitations. We understand that you weigh the novelty differently, and we thank you for the discussion.

Reviewer_MPs4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests safety under temporary self-supervised, few-shot, and target-prefix weight updates. Attacks are evaluated on local models and a fine-tuning API. It also studies degenerate outputs that fool safety judges and proposes a perplexity-based detector.

Critical strengths
  1. The matched ICL control supports an effect of weight updates beyond demonstrations.

  2. The validity analysis identifies safety-judge false positives on degenerate outputs.

Critical weaknesses
  1. The few-shot/target settings overlap with harmful fine-tuning. The stronger restructured self-supervised variant equals single-example few-shot training on the test prompt, leaving the TTT-specific contribution unclear.

  2. Detector evaluation covers five-step few-shot LoRA, not all threats. Loss-modifying attacks can evade it, but exceed the stated attacker controls. The evidence does not yet establish broad practical protection.

Questions for authors
  1. How were hyperparameters selected, and what advantage remains over prefilling alone for target-prefix attacks and update settings chosen for benign TTT utility?

  2. Does the detector cover the other threats and held-out benign domains? Please separate permitted data/hyperparameter evasions from loss-modifying attacks.

2 comments

Thank you for your time in reviewing the paper. We appreciate that you found the ICL vs weight-update isolation and the validity-aware evaluation useful. In the following we address your raised concerns:

W1. We do not claim a stronger or cheaper attack than harmful fine-tuning (jailbreak fine-tuning in the paper). Our contribution is the setting: we provide a general way to define the attack vectors that TTT exposes, captured by three threat models. To clarify the differences between jailbreak fine-tuning and jailbreak TTT, we have now included the following table in the revision (Table 1):

Jailbreak fine-tuningJailbreak TTT
Attacker's accessUploads a dataset, runs a jobSends one inference request
Training dataExample set (100 pairs in Yang et al. [1])The query, or 1-5 pairs with a 10-20 token prefix
Link to the test querySet built in advance, independent of later queriesBuilt per request, can contain the test query itself (self-supervised, generation-phase)
Harmful answers neededUsually (benign-data variants exist)No, the model produces them
StepsUntil it works (100-200 in Yang et al. [1])1-10 per request, chosen by the attacker
Update servesAll later queriesOne query, then reset
Provider can inspectTraining set and finished modelRequest text, and any model test must run per request

In jailbreak fine-tuning, users typically provide an example set of (prompt, full harmful answer) pairs and, after the weights are adapted on it, send the prompts to evaluate. On the other hand, TTT needs the user prompt, optionally with a few (prompt, prefix) pairs or a short affirmative prefix, and the model is adapted only on these data. Moreover, after generating the reply to the user prompt, the model's weights are reset to their initial state before the next user prompt. Another relevant detail is the number of steps. TTT requires few gradient updates (we study up to 10 steps), against 100 to 200 in [1]. The restructured self-supervised variant has the same objective as one-example few-shot TTT on the test prompt, as we state in Appendix A.6. This follows from the self-supervised setting: the service trains on whatever the user sends, so a 10-20 token prefix in the query makes it a supervised update on the test query itself. The prefix is only the start of a compliant reply and adds no harmful content, so it is hard to flag from the request. We do not think that TTT attack success follows from prior fine-tuning results given those differences.

W2/Q2. We would like to highlight that we call the detector “a first step toward a defense”, and it is not the main focus of the paper. As in our reply to Reviewer x2XX, the revision shortens Section 5, calls it a detector rather than a defense (the section title is now “A first step toward defense: perplexity-based detection”), and replaces “can successfully defend” in the introduction. We evaluated the detector where TTT attacks succeed, since an update that does not change the model's behaviour leaves nothing to detect. Besides few-shot TTT, it flags 100% of generation-phase and restructured self-supervised requests at 5 steps, on all four models. Clean-prompt self-supervised updates change the model little and are rarely flagged (0-6%), although they succeed on up to 26% of prompts. On domains not used to set the threshold, it flags at most 2% of GSM8K and 10% of TriviaQA requests. Both results are in Appendix A.7 of the revision (Table 17). Our threat models assume the provider fixes the loss. The attacker controls only the query, the adaptation data and standard hyperparameters (Section 5). Within these controls, clean-prompt self-supervised TTT is the case that evades detection, as above. We did not search further over data or hyperparameters for evasions and the revision lists this as a limitation (Section 7). An attacker who can also change the loss is strictly stronger, so it is expected to evade the detector. We report this case in Appendix A.5 as an upper bound and list adaptive attacks as a limitation.

Q1. We tuned hyperparameters per model and step count under the generation-phase threat model, and reused them unchanged for the other two settings (Appendix C.2). Prefilling writes the prefix into the assistant turn at inference [2,3]. It requires control of the assistant turn, which our threat models do not give the attacker. In generation-phase (target-prefix) TTT the prefix enters only the update, and the adapted model answers the clean prompt. We do not claim a higher ASR than prefilling. The two attacks rely on different access. The revision adds this distinction in Appendix E. On benign utility, there is no single setting that makes TTT useful, because it depends on the task and the query. In our threat models the attacker chooses the hyperparameters, as on the fine-tuning API we evaluate.

[1] Yang et al., Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models

[2] Andriushchenko et al., Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

[3] Qi et al., Safety Alignment Should be Made More Than Just a Few Tokens Deep

Reviewer_x2XXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the safety threats of test-time training, where the use is allowed to fine-tune the model for few steps before getting the final answer. The paper considers three threat models and examines the ASR. They also propose a perplexity-based defense against such attacks.

Critical strengths

1- The paper is written very well and easy to follow. The main figure illustrates the paper very well.

2- The authors explain the considered threat model very well.

3- The experimental setting is sound. The 2 is very informative with showing the ASR at multiple iteration and when the model starts to overfit.

Critical weaknesses

1- There is limited novelty in the considered threat model. The paper already states that the previous work has studies fine-tuning on harmful content. In this regard, I see the second threat model the same as previous work, and the other two are basically more limited versions.

2- That said, the results are not surprising either. For instance, [1] already shows that alignment is eroded after few steps of fine-tuning. How is the results of the current paper compared to them? Are the authors claiming more successful attacks? or do they claim more efficiency?

3- On the previous point, I think the current threat model is only a reframed version with TTT of the existing ones.

4- I have concerns about the model outputs after fine-tuning. In particular, the model is only fine-tuned on one string in the first and the third threat models. The authors say they deploy tow additional filters to verify the validity, but they do not show if the responses are actually useful and the capabilities are not erased. Getting the affirmative responses on questions such as "How to build a bomb" does not carry a point, if the model has forgotten its cyber-security capabilities and it's useless.

5- The same concern applies to the perplexity defense. Even though they examine the perplexity on held out samples, it's only measured over the template affirmative response. This does not mean that the model actually generates coherent, harmful content.

[1] Yang et al, "SHADOW ALIGNMENT: THE EASE OF SUBVERTING SAFELY-ALIGNED LANGUAGE MODELS"

Questions for authors

Please read above.

4 comments

Thank you for your time in reviewing the paper. We appreciate that you found the paper well-written with a sound experimental setting. In the following we address your raised concerns:

W1. We believe that our novelty is formalizing and studying the new attack vectors exposed by test-time training as the three presented threat models. Fine-tuning is just one of the techniques used to perform the weight update, and we compare with jailbreak fine-tuning in the related work section (including [1]). The threat models come from how TTT is used, not from fine-tuning attacks. The same formulation covers other TTT instances, such as the RL setting of Khattar et al. [2], which corresponds to our generation-phase threat model. To make clearer the distinction between jailbreak TTT and jailbreak fine-tuning, the revision now includes a table listing the differences (Table 1):

Jailbreak fine-tuningJailbreak TTT
Attacker’s accessUploads a dataset, runs a jobSends one inference request
Training dataExample set (100 pairs in Yang et al. [1])The query, or 1-5 pairs with a 10-20 token prefix
Link to the test querySet built in advance, independent of later queriesBuilt per request, can contain the test query itself (self-supervised, generation-phase)
Harmful answers neededUsually (benign-data variants exist)No, the model produces them
StepsUntil it works (100-200 in Yang et al. [1])1-10 per request, chosen by the attacker
Update servesAll later queriesOne query, then reset
Provider can inspectTraining set and finished modelRequest text, and any model test must run per request

In summary, most relevant distinctions lie in the fact that in jailbreak fine-tuning, users typically need to provide the model with an example set of (prompt, full harmful answer) pairs and, after adapting the weights on this example set, provide the prompts to evaluate. On the other hand, TTT needs the user prompt, optionally with a few (prompt, prefix) pairs or a short affirmative prefix, and the model is adapted only on these data. Moreover, after generating the reply to the user prompt, the model’s weights are reset to their initial state before the next user prompt. Another relevant detail is the number of steps. TTT requires few gradient updates (we study up to 10 steps), against 100 to 200 in [1].

W2. The fine-tuning vs TTT table in W1 should now clarify how our threat models compare to [1]. The setting is more restricted than fine-tuning and regardless of that we achieve a concerning ASR. Our claim is that safety alignment does not survive TTT rather than a stronger or more efficient attack than fine-tuning. TTT has been shown to improve reasoning. We show that it also lets an attacker with far less data and access than in fine-tuning reach high ASR (93 to 95% average ASR@10 under LoRA for few-shot and generation-phase TTT). The revision states this message in the introduction (Section 1, with Table 1).

W3. We agree that our threat models use the same mechanism as fine-tuning attacks: gradient updates on data the attacker influences. What changes is the setting, and that is what we study: does safety still degrade under TTT when the attacker has far less data and access? We believe that the answer does not follow from prior fine-tuning results. [1] trains on 100 full question-answer pairs for 100 to 200 steps and serves a persistent model. TTT uses the query alone, or at most five pairs with a 10 to 20 token prefix, for at most 10 steps. Our results show that:

  • Self-supervised TTT on the clean prompt alone raises average ASR@10 from 4% to 17% (LoRA). The effect is modest and model-dependent, but it needs no harmful data.
  • Adaptation on a short affirmative prefix, which restates the request and contains no harmful answer, reaches 93 to 95% average ASR@10 (LoRA).
  • The few-shot support set placed in context, without a weight update, reaches 10.9%, against 94.9% after 5 steps of few-shot TTT.

How much safety degrades depends on what enters the TTT objective, which fine-tuning results do not predict.

W4. General capability matters less here than for fine-tuning. A fine-tuned model is kept and must stay useful, while TTT weights answer one query and are then reset to their initial state. Knowledge in the domain of the query is still relevant. We ran MMLU on its computer security and computer science subjects (300 questions) and its chemistry subjects (303 questions). For five models, we took the adapters from few-shot and generation-phase TTT (LoRA, 5 steps), one adapter per harmful prompt (50 prompts) as in the attack, and compared each with the base model on the same questions. Averaged over the 10 settings (5 models × 2 threat models), accuracy changes by -1.0 points on computer questions and 0.0 on chemistry. Llama3 8B is the only exception: it loses up to 5.5 points over all adapters, and 8.5 points when the adapter was trained on a hacking-related prompt. These results are in Appendix A.8 of the revision (Table 19), which also reports the few-shot adapters.

Table: Change in MMLU accuracy (percentage points) of generation-phase TTT adapters against the base model, with the 95% interval over questions. “All” uses the adapters of all 50 harmful prompts. “Matched” uses only the adapters of prompts on the same topic: 12 hacking-related prompts for computer questions, and 7 prompts on explosives, poisons or drugs for chemistry.

ModelComputer, all 50Computer, matched (12)Chemistry, all 50Chemistry, matched (7)
Gemma 7B-1.1 ±1.6-0.8 ±1.5+0.4 ±1.9+0.3 ±1.9
Llama3 8B-5.5 ±2.6-8.5 ±3.2-0.8 ±2.7+0.2 ±2.8
Qwen2.5 1.5B-0.2 ±1.5-0.5 ±1.9+1.6 ±2.2+1.4 ±2.4
Qwen2.5 7B+0.4 ±1.2+0.6 ±1.2-0.1 ±1.10.0 ±1.1
Qwen3 8B-0.1 ±1.3+0.1 ±1.5-0.9 ±2.0-0.9 ±2.2

W5. We agree that the detector does not inspect generated responses, it compares the perplexity of a private holdout of affirmative targets under the original and the adapted weights. As the detector is preliminary and not the main focus of the paper, we have shortened Section 5, which now says explicitly that the detector flags TTT updates that make the model more likely to begin an affirmative reply to harmful prompts rather than “successfully defend”. To make this clearer, the revision also calls it a detector rather than a defense, and the title of Section 5 is now “A first step toward defense: perplexity-based detection”. We leave content-level defenses to future work.

[2] Khattar et al., Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

I thank the authors for the clarifications and the new results. Part of my concerns have been addressed (especially re W4). As I understand the few-shot scheme fine-tunes on 5 random, harmful examples. So the new finding is that even training on 5 samples for 5 steps can erase the guardrails as opposed previously studies settings for harmful finetuning that were more expensive . Then, I think more investigation is needed regarding the other two with a similar Table for their capability. I disagree with the authors that general capabilities matter less here. I understand that the model is reset every time; what I instead tried to ask was that are the harmful answers by the model actually useful or not? Is it not the case that the model after self-supervised finetuning does not really give anything as helpful as before?

Also, can the authors explain the recipe for evaluating the general capabilities? specifically, do they evaluate each of the 50 adapters separately and then average over their accuracies, or do they average the LoRA weights and evaluate it?

Thank you for the follow-up.

On the other threat models: the table in our response is for generation-phase TTT, which adapts on a single string per request. Table 19 in the revision reports both few-shot and generation-phase TTT.

On whether the harmful answers are useful: we agree that the usefulness of the answer matters. Our point was only that the adapted weights serve one query, so the relevant capability is knowledge in the domain of that query, which is what the MMLU evaluation measures. A generation counts as a jailbreak only if it passes both the safety judge and our validity judge, which rejects prefix echoes, repetition, prompt regurgitation and unfilled templates (Section 4.4). We rely on the judge for whether a well-formed answer is harmful, as prior work does, and we did not rate answer quality separately.

On the evaluation recipe: we evaluate each of the 50 adapters separately on all questions and average their accuracies. We do not average LoRA weights. The interval comes from the per-question difference between the adapters and the base model.

One clarification on the few-shot setting: the five examples are prompts paired with a 10 to 20 token affirmative prefix, not harmful answers, and they exclude the test prompt.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.