- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces P-TAP, a multi-turn attack that exploits parameter updates in session-persistent test-time learning. Experiments on four instruction-tuned models show increased attack success compared with frozen-parameter baselines, supported by ablation studies and safeguard evaluations.
Critical strengths
- The research topic is valuable.
- This paper demonstrates that, within the defined session-based continual learning setting, user inputs can increase the attack success rate through parameter updates.
Critical weaknesses
-
The scenario setup is somewhat limited, and the practical implications are unclear. The paper focuses on a scenario involving continuous learning from user queries within a single session. In the main experiments, nearly all queries trigger the update process, yet there is no evaluation of how this strategy benefits normal tasks.
-
Perplexity is not a direct proxy for harmfulness. The evaluated PPL-based detector uses changes in the perplexity of harmful reference completions as an indirect indicator of security risk. Beyond the existing Alpaca controls, a control group using high-perplexity harmless text should be included to verify whether the detector can distinguish between normal adaptation and dangerous updates.
Questions for authors
-
The contribution of the progressive design has not been independently validated. Current experiments do not disentangle the respective roles of query content, sequential order, and multiple updates. Could the authors add control conditions such as randomized order, reverse order, and repeated single queries to verify whether the progressive design yields additional benefits?
-
The control conditions for update data do not distinguish between topical relevance and harmfulness. Replacing attack queries with unrelated Alpaca instructions alters both the topic and the harmfulness. Could the authors include controls using harmless text on the same topic and harmful text on different topics to clarify which factor primarily drives the attack’s effectiveness?
-
A comparison of thinking modes is missing. The experiments disabled the thinking modes of Qwen3 and Gemma. It remains unclear how the reasoning process influences this type of attack. Could the authors compare TTL activation and deactivation across both thinking and non-thinking modes?
-
A comparison with full-parameter fine-tuning is missing. The experiments utilized only LoRA, so it is unclear whether the results generalize to other parameter update methods. Could the authors add a full-parameter fine-tuning baseline to determine whether the attack’s effectiveness relies on LoRA?
-
The dependence of parameter effects on the original conversation history remains unclear. Table 2 validates the impact of parameter updates, but the evaluation retains the attack session history. Could the authors retain the adapter while clearing the history, then test both the original target and other targets to assess whether the security degradation extends beyond the current conversation?
-
Additional validation of attack scoring and statistical reliability is needed. The evaluation relies on a single automated judge and a fixed success threshold, and no systematic manual validation is reported. Could the authors provide further evidence of the reliability of the attack scores and the reported results?
4 comments
4. Topic and harmfulness of the update data (Q2)
Neither topical relevance nor harmful content alone reproduces the lift. We keep the original conversation and replace only the eleven update texts with benign queries on the same topic, harmful queries from a behavior in another JailbreakBench category, or Alpaca instructions. With any of the three, ASR@1 and ASR@10 differ from Frozen by at most 8 and 11 points, against lifts of 10–16 and 16–31 points with the original updates. Cells show ASR@1 / ASR@10, the four right columns are Online with the stated update text, and Lift is Online minus Frozen in percentage points.
| Model | Frozen | Same topic, harmful (original) | Same topic, benign | Different topic, harmful | Different topic, benign | |
|---|---|---|---|---|---|---|
| Qwen3-8B | ASR | .59 / .67 | .72 / .85 | .55 / .64 | .64 / .72 | .52 / .67 |
| Lift | +13 / +18 | −4 / −3 | +5 / +5 | −7 / 0 | ||
| Qwen2.5-7B-Instruct | ASR | .50 / .69 | .63 / .85 | .49 / .66 | .51 / .74 | .42 / .58 |
| Lift | +13 / +16 | −1 / −3 | +1 / +5 | −8 / −11 | ||
| Llama-3.1-8B-Instruct | ASR | .15 / .20 | .31 / .51 | .23 / .30 | .17 / .26 | .14 / .21 |
| Lift | +16 / +31 | +8 / +10 | +2 / +6 | −1 / +1 | ||
| Gemma-4-E4B-it | ASR | .13 / .19 | .23 / .43 | .08 / .14 | .06 / .21 | .19 / .28 |
| Lift | +10 / +24 | −5 / −5 | −7 / +2 | +6 / +9 |
Details are in Section 5.3 (Figure 4) and Appendix Table 3.
5. Thinking mode (Q3)
We repeated the Frozen and Online comparison with thinking enabled on Qwen3-8B and Gemma-4-E4B-it, the two models that support a thinking mode. On Qwen3-8B, Online still exceeds Frozen, by 26 and 22 points. On Gemma-4-E4B-it, the two conditions do not differ significantly. Cells show ASR@1 / ASR@10, and the rows with thinking off are from Table 1. With thinking on, ASR@1 uses each model's default sampling.
| Model | Thinking | Frozen | Online | Lift |
|---|---|---|---|---|
| Qwen3-8B | off | .59 / .67 | .72 / .85 | +13 / +18 |
| Qwen3-8B | on | .39 / .59 | .65 / .81 | +26 / +22 |
| Gemma-4-E4B-it | off | .13 / .19 | .23 / .43 | +10 / +24 |
| Gemma-4-E4B-it | on | .25 / .36 | .20 / .34 | −5 / −2 |
6. Full-parameter fine-tuning (Q4)
We did not add a full-parameter baseline. Test-time updates must run quickly at deployment, and TLM, Temp-LoRA, T²PAM/ROSA and aTTT all update a LoRA module or a small subset of parameters. TLM adopts LoRA instead of full-parameter optimization to mitigate catastrophic forgetting and ensure adaptation stability, and we follow TLM in using LoRA.
7. Adapter state without the attack history (Q5)
The adapter raises success on its own session's target without the history and adds little on other targets. On Qwen3-8B with τ = 3, we answer the final request in a single-turn session with a fixed adapter and no history. The adapter after the eleven updates of the target's own session raises ASR@1 from 1% to 8% and ASR@10 from 5% to 35%, while the adapters of the other 99 sessions raise a target's ASR@1 only from 1.0% to 2.1% (Table 2, Section 5.3).
8. Reliability of the attack scores (Q6)
We manually labeled and double-checked 200 final responses, and MiMo-V2.5-Pro, the judge used in the submission, has relatively low agreement with them. We now use DeepSeek-V4.1-Flash as the judge.
We measured how well 13 candidate judges agree with the human labels in our setting. The general-purpose LLMs use the PAIR prompt with thinking enabled where available, and the safety guard models run in their officially recommended settings. Seven of them are listed below, and Appendix A.9 gives the full results.
| Judge | Agreement | Cohen's κ |
|---|---|---|
| General-purpose LLMs | ||
| DeepSeek-V4.1-Flash | 0.960 | 0.92 |
| GPT-4o | 0.871 | 0.73 |
| MiMo-V2.5-Pro | 0.760 | 0.52 |
| Claude Haiku 5.5 | 0.749 | 0.52 |
| Safety guard models | ||
| gpt-oss-safeguard-20b | 0.895 | 0.78 |
| Nemotron 3.5 Content Safety | 0.870 | 0.72 |
| Llama Guard 4 12B | 0.670 | 0.35 |
Considering agreement, speed, cost, and stability, we chose DeepSeek-V4.1-Flash. We have re-run the main experiments and re-scored all results with it, and all ASR numbers in the paper are updated.
Its κ with the human labels stays between 0.91 and 0.92 for any success threshold from 5 to 8, and the paper uses 7.
Online exceeds Frozen on all twelve model-metric pairs of Table 1, and most differences are significant at the 0.05 level. The table gives each Online − Frozen difference in percentage points, with its 95% bootstrap CI and the p-value of a paired exact test.
| Model | ASR@1 | Per-trial ASR | ASR@10 |
|---|---|---|---|
| Qwen3-8B | +13 [3, 23], p = .019 | +17.7 [10.2, 25.4], p < .001 | +18 [10, 27], p < .001 |
| Qwen2.5-7B-Instruct | +13 [2, 24], p = .029 | +15.0 [6.9, 23.2], p < .001 | +16 [7, 25], p = .002 |
| Llama-3.1-8B-Instruct | +16 [7, 26], p = .002 | +13.3 [6.4, 20.4], p < .001 | +31 [20, 42], p < .001 |
| Gemma-4-E4B-it | +10 [0, 20], p = .076 | +8.1 [0.3, 15.9], p = .047 | +24 [13, 35], p < .001 |
Dear reviewer, thank you for your effort during reviewing. Plz read the author's response to check whether they address your concerns.
Thank you for your careful review and constructive suggestions. We have uploaded a revised PDF that includes the new experiments.
1. Scenario and benefit on normal tasks (W1)
On the scenario, updating parameters during inference and keeping the update for later inputs is an active research area. Recent work covers methods with theoretical analysis (T²PAM/ROSA, Wei et al., 2025; In-Place TTT, Feng et al., ICLR 2026), methods that improve performance (TLM, Hu et al., ICML 2025; dynamic evaluation, Rannen-Triki et al., NeurIPS 2023 Workshop; Temp-LoRA, Wang et al., COLM 2024; TTT-E2E, Tandon et al., 2025; aTTT, Wang et al., 2026), and serving systems that batch requests carrying their own test-time training state (RW-TTT, Yang et al., 2026). Several of them, such as T²PAM/ROSA and aTTT, keep the update within one conversation or episode, as our session-persistent setting does.
On the benefit for normal tasks, this paper does not measure it, and the evidence comes from the works above. TLM, whose update rule we use, reports gains on domain knowledge adaptation, and the others report gains on multi-turn reasoning, agent tasks and long-context language modeling.
2. High-perplexity harmless controls for the PPL detector (W2)
We agree that perplexity is not a direct proxy for harmfulness. We would like to clarify that our Alpaca control already has higher perplexity than the attack queries. Alpaca instructions have a median input log-perplexity of 5.75 on Qwen3-8B, above 97.9% of the attack queries, whose median is 4.16.
To reflect real usage, we use 400 benign sessions built from real user conversations in WildChat. As a high-perplexity harmless control, we keep the 251 WildChat sessions whose mean input log-perplexity exceeds attack queries’ median 4.16. The detector does not separate them from the attack sessions. With the original threshold of 0.770, the base-to-final score flags no attack session, and the adjacent-update score flags 87% of these benign sessions and 80% of the attack sessions. After recalibration on half of the sessions at a false-positive rate of at most 2%, the detector flags none of the 100 attack sessions (Section 5.4, Appendix A.4).
3. Contribution of the progressive design (Q1)
The original progressive order gives the highest success under both Frozen and Online on every model. Randomly permuting or reversing the eleven prefix queries lowers success, and Online still exceeds Frozen under both orders, with ASR@10 lifts of 16–26 and 3–15 points. With the final request sent at all eleven prefix turns, the ASR@10 lift is at most 2 points on three models and 22 points on Llama-3.1-8B-Instruct. In each control, the conversation and the updates both follow the rearranged stream. Cells show ASR@1 / ASR@10, and Lift is Online minus Frozen in percentage points.
Details are in Section 5.3 and Appendix A.6.