acceptodds
Under review as a conference paper at ICLR 2027

Adaptation or Poisoning? Security Risks of Conversational Test-Time Learning in Large Language Models

Abstract

Test-time learning (TTL) enables large language models to adapt to unlabeled inputs during deployment. The model updates a lightweight adapter by minimizing the next-token prediction loss on the input it receives. When these updates persist across conversation turns, every user query serves both as the request answered in the current turn and as adaptation data for later turns. By choosing the queries used for adaptation, an attacker can influence parameter updates and weaken safety alignment through ordinary interactions with a TTL chat service. To demonstrate this risk, we introduce Progressive Test-Time Adaptation Poisoning (P-TAP), a query-only multi-turn attack that jailbreaks the model by poisoning its session adapter through the service’s own updates. A sequence of escalating adversarial queries shapes the adapter turn by turn; the final harmful request is then answered under the accumulated state without itself being used for adaptation. Across four open-weight instruction-tuned models, enabling updates raises attack success rates on average by 14.50 percentage points under greedy decoding and by 24.25 points under best-of-10 sampling, relative to matched sessions receiving the same queries without updates. Answering adversarial queries but replacing them with unrelated benign text for adaptation removes most of this increase. Among existing safeguards, screening incoming queries flags every attack stream before the final request, while a perplexity-based detector of unsafe adaptation flags none of the successful attacks. the optimizer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
test-time learning, test-time adaptation, LLM safety, jailbreak, multi-turn jailbreak, poisoning attack, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pcZWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces P-TAP, a multi-turn attack that exploits parameter updates in session-persistent test-time learning. Experiments on four instruction-tuned models show increased attack success compared with frozen-parameter baselines, supported by ablation studies and safeguard evaluations.

Critical strengths
  • The research topic is valuable.
  • This paper demonstrates that, within the defined session-based continual learning setting, user inputs can increase the attack success rate through parameter updates.
Critical weaknesses
  • The scenario setup is somewhat limited, and the practical implications are unclear. The paper focuses on a scenario involving continuous learning from user queries within a single session. In the main experiments, nearly all queries trigger the update process, yet there is no evaluation of how this strategy benefits normal tasks.

  • Perplexity is not a direct proxy for harmfulness. The evaluated PPL-based detector uses changes in the perplexity of harmful reference completions as an indirect indicator of security risk. Beyond the existing Alpaca controls, a control group using high-perplexity harmless text should be included to verify whether the detector can distinguish between normal adaptation and dangerous updates.

Questions for authors
  • The contribution of the progressive design has not been independently validated. Current experiments do not disentangle the respective roles of query content, sequential order, and multiple updates. Could the authors add control conditions such as randomized order, reverse order, and repeated single queries to verify whether the progressive design yields additional benefits?

  • The control conditions for update data do not distinguish between topical relevance and harmfulness. Replacing attack queries with unrelated Alpaca instructions alters both the topic and the harmfulness. Could the authors include controls using harmless text on the same topic and harmful text on different topics to clarify which factor primarily drives the attack’s effectiveness?

  • A comparison of thinking modes is missing. The experiments disabled the thinking modes of Qwen3 and Gemma. It remains unclear how the reasoning process influences this type of attack. Could the authors compare TTL activation and deactivation across both thinking and non-thinking modes?

  • A comparison with full-parameter fine-tuning is missing. The experiments utilized only LoRA, so it is unclear whether the results generalize to other parameter update methods. Could the authors add a full-parameter fine-tuning baseline to determine whether the attack’s effectiveness relies on LoRA?

  • The dependence of parameter effects on the original conversation history remains unclear. Table 2 validates the impact of parameter updates, but the evaluation retains the attack session history. Could the authors retain the adapter while clearing the history, then test both the original target and other targets to assess whether the security degradation extends beyond the current conversation?

  • Additional validation of attack scoring and statistical reliability is needed. The evaluation relies on a single automated judge and a fixed success threshold, and no systematic manual validation is reported. Could the authors provide further evidence of the reliability of the attack scores and the reported results?

4 comments

Thank you for your careful review and constructive suggestions. We have uploaded a revised PDF that includes the new experiments.

1. Scenario and benefit on normal tasks (W1)

On the scenario, updating parameters during inference and keeping the update for later inputs is an active research area. Recent work covers methods with theoretical analysis (T²PAM/ROSA, Wei et al., 2025; In-Place TTT, Feng et al., ICLR 2026), methods that improve performance (TLM, Hu et al., ICML 2025; dynamic evaluation, Rannen-Triki et al., NeurIPS 2023 Workshop; Temp-LoRA, Wang et al., COLM 2024; TTT-E2E, Tandon et al., 2025; aTTT, Wang et al., 2026), and serving systems that batch requests carrying their own test-time training state (RW-TTT, Yang et al., 2026). Several of them, such as T²PAM/ROSA and aTTT, keep the update within one conversation or episode, as our session-persistent setting does.

On the benefit for normal tasks, this paper does not measure it, and the evidence comes from the works above. TLM, whose update rule we use, reports gains on domain knowledge adaptation, and the others report gains on multi-turn reasoning, agent tasks and long-context language modeling.

2. High-perplexity harmless controls for the PPL detector (W2)

We agree that perplexity is not a direct proxy for harmfulness. We would like to clarify that our Alpaca control already has higher perplexity than the attack queries. Alpaca instructions have a median input log-perplexity of 5.75 on Qwen3-8B, above 97.9% of the attack queries, whose median is 4.16.

To reflect real usage, we use 400 benign sessions built from real user conversations in WildChat. As a high-perplexity harmless control, we keep the 251 WildChat sessions whose mean input log-perplexity exceeds attack queries’ median 4.16. The detector does not separate them from the attack sessions. With the original threshold of 0.770, the base-to-final score flags no attack session, and the adjacent-update score flags 87% of these benign sessions and 80% of the attack sessions. After recalibration on half of the sessions at a false-positive rate of at most 2%, the detector flags none of the 100 attack sessions (Section 5.4, Appendix A.4).

3. Contribution of the progressive design (Q1)

The original progressive order gives the highest success under both Frozen and Online on every model. Randomly permuting or reversing the eleven prefix queries lowers success, and Online still exceeds Frozen under both orders, with ASR@10 lifts of 16–26 and 3–15 points. With the final request sent at all eleven prefix turns, the ASR@10 lift is at most 2 points on three models and 22 points on Llama-3.1-8B-Instruct. In each control, the conversation and the updates both follow the rearranged stream. Cells show ASR@1 / ASR@10, and Lift is Online minus Frozen in percentage points.

ModelOriginalRandomReversedRepeated final request
Qwen3-8BFrozen.59 / .67.33 / .43.10 / .15.02 / .03
Online.72 / .85.48 / .60.13 / .25.01 / .04
Lift+13 / +18+15 / +17+3 / +10−1 / +1
Qwen2.5-7B-InstructFrozen.50 / .69.39 / .52.23 / .50.27 / .40
Online.63 / .85.49 / .78.27 / .59.26 / .42
Lift+13 / +16+10 / +26+4 / +9−1 / +2
Llama-3.1-8B-InstructFrozen.15 / .20.09 / .11.02 / .04.06 / .04
Online.31 / .51.17 / .27.07 / .19.13 / .26
Lift+16 / +31+8 / +16+5 / +15+7 / +22
Gemma-4-E4B-itFrozen.13 / .19.02 / .03.01 / .01.01 / .01
Online.23 / .43.04 / .20.02 / .04.02 / .02
Lift+10 / +24+2 / +17+1 / +3+1 / +1

Details are in Section 5.3 and Appendix A.6.

4. Topic and harmfulness of the update data (Q2)

Neither topical relevance nor harmful content alone reproduces the lift. We keep the original conversation and replace only the eleven update texts with benign queries on the same topic, harmful queries from a behavior in another JailbreakBench category, or Alpaca instructions. With any of the three, ASR@1 and ASR@10 differ from Frozen by at most 8 and 11 points, against lifts of 10–16 and 16–31 points with the original updates. Cells show ASR@1 / ASR@10, the four right columns are Online with the stated update text, and Lift is Online minus Frozen in percentage points.

ModelFrozenSame topic, harmful (original)Same topic, benignDifferent topic, harmfulDifferent topic, benign
Qwen3-8BASR.59 / .67.72 / .85.55 / .64.64 / .72.52 / .67
Lift+13 / +18−4 / −3+5 / +5−7 / 0
Qwen2.5-7B-InstructASR.50 / .69.63 / .85.49 / .66.51 / .74.42 / .58
Lift+13 / +16−1 / −3+1 / +5−8 / −11
Llama-3.1-8B-InstructASR.15 / .20.31 / .51.23 / .30.17 / .26.14 / .21
Lift+16 / +31+8 / +10+2 / +6−1 / +1
Gemma-4-E4B-itASR.13 / .19.23 / .43.08 / .14.06 / .21.19 / .28
Lift+10 / +24−5 / −5−7 / +2+6 / +9

Details are in Section 5.3 (Figure 4) and Appendix Table 3.

5. Thinking mode (Q3)

We repeated the Frozen and Online comparison with thinking enabled on Qwen3-8B and Gemma-4-E4B-it, the two models that support a thinking mode. On Qwen3-8B, Online still exceeds Frozen, by 26 and 22 points. On Gemma-4-E4B-it, the two conditions do not differ significantly. Cells show ASR@1 / ASR@10, and the rows with thinking off are from Table 1. With thinking on, ASR@1 uses each model's default sampling.

ModelThinkingFrozenOnlineLift
Qwen3-8Boff.59 / .67.72 / .85+13 / +18
Qwen3-8Bon.39 / .59.65 / .81+26 / +22
Gemma-4-E4B-itoff.13 / .19.23 / .43+10 / +24
Gemma-4-E4B-iton.25 / .36.20 / .34−5 / −2

6. Full-parameter fine-tuning (Q4)

We did not add a full-parameter baseline. Test-time updates must run quickly at deployment, and TLM, Temp-LoRA, T²PAM/ROSA and aTTT all update a LoRA module or a small subset of parameters. TLM adopts LoRA instead of full-parameter optimization to mitigate catastrophic forgetting and ensure adaptation stability, and we follow TLM in using LoRA.

7. Adapter state without the attack history (Q5)

The adapter raises success on its own session's target without the history and adds little on other targets. On Qwen3-8B with τ = 3, we answer the final request in a single-turn session with a fixed adapter and no history. The adapter after the eleven updates of the target's own session raises ASR@1 from 1% to 8% and ASR@10 from 5% to 35%, while the adapters of the other 99 sessions raise a target's ASR@1 only from 1.0% to 2.1% (Table 2, Section 5.3).

8. Reliability of the attack scores (Q6)

We manually labeled and double-checked 200 final responses, and MiMo-V2.5-Pro, the judge used in the submission, has relatively low agreement with them. We now use DeepSeek-V4.1-Flash as the judge.

We measured how well 13 candidate judges agree with the human labels in our setting. The general-purpose LLMs use the PAIR prompt with thinking enabled where available, and the safety guard models run in their officially recommended settings. Seven of them are listed below, and Appendix A.9 gives the full results.

JudgeAgreementCohen's κ
General-purpose LLMs
DeepSeek-V4.1-Flash0.9600.92
GPT-4o0.8710.73
MiMo-V2.5-Pro0.7600.52
Claude Haiku 5.50.7490.52
Safety guard models
gpt-oss-safeguard-20b0.8950.78
Nemotron 3.5 Content Safety0.8700.72
Llama Guard 4 12B0.6700.35

Considering agreement, speed, cost, and stability, we chose DeepSeek-V4.1-Flash. We have re-run the main experiments and re-scored all results with it, and all ASR numbers in the paper are updated.

Its κ with the human labels stays between 0.91 and 0.92 for any success threshold from 5 to 8, and the paper uses 7.

Online exceeds Frozen on all twelve model-metric pairs of Table 1, and most differences are significant at the 0.05 level. The table gives each Online − Frozen difference in percentage points, with its 95% bootstrap CI and the p-value of a paired exact test.

ModelASR@1Per-trial ASRASR@10
Qwen3-8B+13 [3, 23], p = .019+17.7 [10.2, 25.4], p < .001+18 [10, 27], p < .001
Qwen2.5-7B-Instruct+13 [2, 24], p = .029+15.0 [6.9, 23.2], p < .001+16 [7, 25], p = .002
Llama-3.1-8B-Instruct+16 [7, 26], p = .002+13.3 [6.4, 20.4], p < .001+31 [20, 42], p < .001
Gemma-4-E4B-it+10 [0, 20], p = .076+8.1 [0.3, 15.9], p = .047+24 [13, 35], p < .001

Dear reviewer, thank you for your effort during reviewing. Plz read the author's response to check whether they address your concerns.

Reviewer_cJyZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the security implications of session-persistent Test-Time Learning (TTL) in large language models. Under this scenario, a model updates a lightweight LoRA adapter turn-by-turn by minimizing next-token prediction loss on incoming user queries and retains this adapter across conversation turns within a session. The paper proposes Progressive Test-Time Adaptation Poisoning (P-TAP), a query-only attack in which an adversary submits a 12-turn Foot-In-The-Door (FITD) escalating query stream. The first 11 queries gradually steer the adapter weights while receiving ordinary responses; the 12th query submits the target harmful request, which is answered under the accumulated parameter state without itself being adapted upon. P-TAP is evaluated across four models on 100 progressive query sequences derived from JailbreakBench harmful behaviors using MiMo-V2.5-Pro as a judge. The results show that enabling updates increases unsafe compliance under the same user-query streams.

Critical strengths
  • The investigating problem of whether online deployment-time adaptation (TTL/TTT) creates a parameter-poisoning channel through standard user chat interfaces is necessary.
  • Comparing ONLINE directly against a matched FROZEN control receiving identical query sequences effectively isolates the causal effect of parameter updates from context-window accumulation.
  • The paper does not merely demonstrate a vulnerability; it evaluates practical defenses (input screening vs. parameter-level perplexity shifts), highlighting concrete failure modes in parameter-space monitoring.
Critical weaknesses
  • The results heavily rely on MiMo-V2.5-Pro as a judge, making the reliablity is low because there is no human-verified subset.
  • "Session-persistent TTL" updating adapters on arbitrary user inputs is a purely hypothetical deployment mode. The attack relies heavily on setting the selection threshold τ=0, forcing updates on every single input. When using TLM's ablated default threshold (τ=3), attack success drops substantially.
  • Claiming that the PPL detector fundamentally fails in multi-turn settings without calibrating γ\gamma directly on the distribution of max⁡tdh(t)\max_{t} d_h^{(t)} over benign multi-turn sessions is methodologically invalid.
Questions for authors
  • Please provide an agreement study (e.g., Cohen's κ) comparing MiMo-V2.5-Pro scores against human labels or an established safety judge (such as Llama-Guard-3-8B or GPT-4o) on a random sample of at least 100 final responses.
  • Please provide 95% confidence intervals or paired significance tests (e.g., McNemar's test) for the "ONLINE" vs. "FROZEN" comparisons across all four models in Table 1.
3 comments

Thank you for your careful review and constructive suggestions. We have uploaded a revised PDF that includes the new experiments.

1. Reliability of the LLM judge (W1, Q1)

We manually labeled and double-checked 200 final responses, and MiMo-V2.5-Pro, the judge used in the submission, has relatively low agreement with them. We now use DeepSeek-V4.1-Flash as the judge.

We measured how well 13 candidate judges agree with the human labels in our setting. The general-purpose LLMs use the PAIR prompt with thinking enabled where available, and the safety guard models run in their officially recommended settings. Seven of them are listed below, and Appendix A.9 gives the full results.

JudgeAgreementCohen's κ
General-purpose LLMs
DeepSeek-V4.1-Flash0.9600.92
GPT-4o0.8710.73
MiMo-V2.5-Pro0.7600.52
Claude Haiku 5.50.7490.52
Safety guard models
gpt-oss-safeguard-20b0.8950.78
Nemotron 3.5 Content Safety0.8700.72
Llama Guard 4 12B0.6700.35

Considering agreement, speed, cost, and stability, we chose DeepSeek-V4.1-Flash. We have re-run the main experiments and re-scored all results with it, and all ASR numbers in the paper are updated. The conclusions are unchanged, and Online still exceeds Frozen on all four models.

2. Selection threshold τ (W2)

We would like to clarify that the attack does not rely on τ = 0. The submission evaluated τ = 3, the TLM default, on two models, and the revision adds the other two with the conclusion unchanged. Online remains significantly higher than Frozen at τ = 3 on every model, with p < .05 on both metrics under the exact McNemar test, and attack success at τ = 3 stays within 4 points of that at τ = 0. The results are in Section 5.3 and copied below, with each cell giving ASR@1 / ASR@10.

ModelFrozenOnline, τ = 0Online, τ = 3Attack queries skipped at τ = 3
Qwen3-8B.59 / .67.72 / .85.73 / .852.0%
Qwen2.5-7B-Instruct.50 / .69.63 / .85.65 / .810.2%
Llama-3.1-8B-Instruct.15 / .20.31 / .51.29 / .470%
Gemma-4-E4B-it.13 / .19.23 / .43.23 / .440%

We also recorded the selection score of every attack query. Almost all of them exceed τ = 3, and at most 2.0% of the queries are skipped on any model (last column; Figure 5).

3. Calibration of the PPL detector (W3)

We have recalibrated γ\gamma on the session maximum max⁡tdh(t)\max_t d_h^{(t)} over benign multi-turn sessions, and the detector still does not separate attack sessions from benign ones. To reflect real usage, we use 400 benign sessions built from real user conversations in WildChat. The original threshold of 0.770 flags 80% of the attack sessions and 81% of the benign sessions. At a calibration false-positive rate of at most 2%, the recalibrated threshold is 1.363, which flags none of the 100 attack sessions. The attack and benign scores overlap, with an AUROC of 0.50. Details are in Section 5.4 and Appendix A.4.

4. Confidence intervals and significance tests (Q2)

All twelve differences are positive, and most are significant at the 0.05 level. The table gives the Online − Frozen difference for each comparison in Table 1, in percentage points, with its 95% bootstrap CI and paired-test p-value. The p-values come from the exact McNemar test for ASR@1 and ASR@10 and a paired permutation test for per-trial ASR.

ModelASR@1Per-trial ASRASR@10
Qwen3-8B+13 [3, 23], p = .019+17.7 [10.2, 25.4], p < .001+18 [10, 27], p < .001
Qwen2.5-7B-Instruct+13 [2, 24], p = .029+15.0 [6.9, 23.2], p < .001+16 [7, 25], p = .002
Llama-3.1-8B-Instruct+16 [7, 26], p = .002+13.3 [6.4, 20.4], p < .001+31 [20, 42], p < .001
Gemma-4-E4B-it+10 [0, 20], p = .076+8.1 [0.3, 15.9], p = .047+24 [13, 35], p < .001

I thank the author for the rebuttal. The answers resolved my concerns. Therefore, I adjust my rating.

Dear Reviewer cJyZ,

Thank you for reading our response and for updating your rating. Your comments on the judge, the threshold, and the detector calibration made our evaluation stronger.

Best regards,

The authors

Reviewer_wisDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TTL allows LLMs to adapt to new tasks using unlabeled data. An attacker can design or choose specific queries for adaptation. This paper introduces Progressive Test-Time Adaptation Poisoning, which can jailbreak a model by poisoning its session adapter through the service’s own updates using selected queries. Experiments across four models compare frozen and online adaptation settings and show higher attack success rates when updates are enabled.

Critical strengths
  1. Clear and easy-to-follow writing
  2. Strong attack performance with detail defense study
Critical weaknesses
  1. The evaluation only covers small models. I know larger model experiments may be costly, but providing results on at least one larger model would help to understand whether the attack remains effective.
  2. Novelty is limited. The query construction follows FITD, the updates use TLM, and Section 4.3 provides a straightforward first-order rationale.
  3. The attack evaluation uses fixed query sequences, and the input guard blocks all 100 tested streams. It is unclear how the results transfer to other attack strategies or guardrail.
Questions for authors

see weakness

2 comments

Thank you for your careful review and constructive suggestions. We have uploaded a revised PDF.

1. Larger models (W1)

Online still exceeds Frozen on ASR@1 and ASR@10 for both larger models, Qwen3-14B and Gemma-4-12B-it. We ran the main experiments on them with the settings of Section 5.1. Cells show ASR@1 / ASR@10, and Lift is Online minus Frozen in percentage points.

ModelFrozenOnlineLift
Qwen3-14B.46 / .60.58 / .66+12 / +6
Gemma-4-12B-it.03 / .05.07 / .14+4 / +9

2. Novelty (W2)

In session-persistent test-time learning (TTL), chat access alone is enough to poison the parameters that answer a later harmful request. Prompt-based jailbreaks leave the parameters fixed, and fine-tuning attacks need a training interface that accepts uploaded prompt and response pairs. Updating parameters during inference and keeping the update for later inputs is an active research area (see our response to Reviewer pcZW, point 1), and we study the safety risk of such systems.

P-TAP is designed for this setting. TLM is the TTL system we attack in our experiments, and the FITD prompt is our means of generating the queries. Running a multi-turn jailbreak on a TTL system is not enough to produce the Online–Frozen lift. Streams from the Analogy-based Multi-turn Attack (AMA; Wu et al., 2025) and ActorBreaker (Ren et al., 2025) show no consistent lift under the same update rule. P-TAP loses most of its lift when the updates are computed on unrelated text or on benign text on the same topic (point 3). Section 4.3 predicts this dependence on the update text, and these controls test it.

3. Other attack strategies and guardrails (W3)

On other attack strategies, a successful attack needs a context that can be poisoned, and the parameter updates then strengthen the jailbreak. First, the queries must escalate gradually toward the goal to poison the context. Second, explicit content of the goal must appear in the queries before the final request, as the optimization target that poisons the parameters (Section 4.3). The attack reaches its full effect only when both hold. P-TAP meets both conditions. AMA and ActorBreaker lack the second. AMA keeps every query before the final request free of harmful content, and ActorBreaker approaches the target through seemingly benign questions about a related actor. Neither shows a consistent lift.

Cells show Online minus Frozen in ASR@1 / ASR@10, in percentage points.

ModelP-TAPAMAActorBreaker
Qwen3-8B+13 / +18−2 / 0−2 / −2
Qwen2.5-7B-Instruct+13 / +16+3 / +4−2 / +0.5
Llama-3.1-8B-Instruct+16 / +31−12 / +1−1 / 0
Gemma-4-E4B-it+10 / +24−10 / −16−1 / −2.5

Within P-TAP sessions, shuffling the queries breaks the gradual escalation, and success drops under both Frozen and Online while Online still exceeds Frozen on every model. When we keep the conversation and replace only the update texts with unrelated text or with benign text on the same topic, the goal's content no longer enters the updates, and most of the lift is lost. Details are in Section 5.3 and Appendices A.5 and A.6.

On guardrails, Qwen3Guard-Gen-8B flags all 100 P-TAP streams before the final request, and Llama Guard 4 12B flags 97 of them. Each guard reads the user queries one at a time, and cells give the cumulative number of streams flagged by that turn.

Guard1234567891011Not flaggedMean first-flag turn
Qwen3Guard-Gen-8B0051530486784969910006.56
Llama Guard 4 12B01471529485972949737.61

Guard models stop most attacks that are not optimized against them, and P-TAP streams, fixed sequences with explicit harmful requests in their later queries, are flagged as expected. A guardrail is a separate layer whose scope and strictness the provider sets. We mainly focus on the safety of the model itself under updates, which the main experiments measure without any input guard (Table 1).

Dear reviewer, thank you for your effort during reviewing. Plz read the author's response to check whether they address your concerns.

Comments on the paper

1 comment

General response

Dear Area Chair and Reviewers,

Thank you for your time on our submission, and thank you to the Area Chair for reminding the reviewers to read our responses. Reviewer cJyZ has confirmed that our response resolved the concerns and has adjusted the rating. We have not yet heard from Reviewers wisD and pcZW. Below we summarize what the paper shows and what we added since the reviews. Details and tables are in our responses to each reviewer.

What the paper shows

1. In session-persistent test-time learning (TTL), chat access alone is enough to poison the parameters that answer a later harmful request. Prompt-based jailbreaks leave the parameters fixed, and fine-tuning attacks need a training interface that accepts uploaded prompt and response pairs. Updating parameters during inference and keeping the update for later inputs is an active research area, and we study the safety risk of such systems. Our attack, P-TAP, sends eleven escalating queries through the chat interface, and the service's own updates poison the adapter that answers the final harmful request.

2. Enabling updates raises the risk, and the added risk stays in the weights. Given the same queries, attack success with updates on (Online) exceeds that with updates off (Frozen) on all four open-weight models, by 13.00 percentage points on average under greedy decoding (ASR@1) and by 22.25 points under best-of-10 sampling (ASR@10). On Llama-3.1-8B-Instruct and Gemma-4-E4B-it, ASR@10 more than doubles. With the conversation history cleared and only the adapter kept, ASR@10 on the session's own harmful request still rises from 5% to 35% on Qwen3-8B.

3. An existing perplexity-based detector of unsafe test-time adaptation does not detect the attack. It flags none of the 100 attack sessions on Qwen3-8B.

What we added during the rebuttal

Online still exceeds Frozen on two larger models, Qwen3-14B and Gemma-4-12B-it. Reviewer wisD asked for results on at least one larger model. We ran the main experiments on both, and Online exceeds Frozen by 12 and 6 points in ASR@1 and ASR@10 on Qwen3-14B and by 4 and 9 points on Gemma-4-12B-it.

The increase depends on the text the adapter is updated on and on the order of the queries. Reviewer wisD asked about other attack strategies, and Reviewer pcZW asked which factors drive the attack. We kept the P-TAP conversation unchanged and replaced only the text used for the updates. With benign queries on the same topic, harmful queries on another topic, or unrelated benign instructions, ASR@1 and ASR@10 differ from Frozen by at most 8 and 11 points, against increases of 10–16 and 16–31 points with the original updates. Two other multi-turn jailbreaks, the Analogy-based Multi-turn Attack (AMA) and the newly added ActorBreaker, show no consistent increase under the same update rule. Random and reversed query orders lower success on every model.

The detector still misses the attack after recalibration on real user sessions. Reviewer cJyZ asked for calibration on benign multi-turn sessions, and Reviewer pcZW asked for a control with high-perplexity harmless text. We built 400 benign sessions from real WildChat conversations, including a high-perplexity subset, and recalibrated the detector on them. At a calibration false-positive rate of at most 2%, it flags none of the attack sessions, and it does not separate the high-perplexity subset from the attack sessions.

We chose a new LLM judge that agrees closely with human labels. Reviewers cJyZ and pcZW asked for human validation of the scores. We manually labeled 200 final responses, compared 13 candidate judges against these labels, and re-scored all results with DeepSeek-V4.1-Flash, which reaches a Cohen's κ of 0.92 with these labels. With the new judge, all twelve Online–Frozen differences in Table 1 of the revised paper are positive, and most are significant at the 0.05 level.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.