- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Red-TTT, which builds on the simple Best-of-N attack and uses the victim's responses to the attack queries as a signal to train the attacker at test time (a per-behavior LoRA adapter on Qwen2.5-7B-Instruct, updated between rounds with a GRPO-style risk-seeking objective). The paper shows that such test-time training substantially outperforms the frozen Best-of-N baseline (average ASR 55.9% → 72.4% on HarmBench) under the same number of attack queries (N = 120 per behavior).
Critical strengths
- The idea is simple and well motivated. For LLM-based attackers that sample adversarial prompts, such as the Best-of-N setting in this paper, it is natural to expect that using feedback from the victim model to optimize the attacker helps elicit more harmful responses.
- Clear experimental results. In the Best-of-N control experiments with matched attack queries, the ASR gain is consistent across all 12 configurations (+9.0 to +23.0 points), including on top of the RL-trained Jailbreak-R1 attacker.
Critical weaknesses
- The reward signal directly uses the same classifier as the evaluation. The main reward term is the continuous probability from the same HarmBench Llama-2-13B classifier (same model and prompt), whose thresholded output forms the classifier component of the final success criterion. Some reward signal for harmful output is clearly needed, and the paper acknowledges the shared classifier and introduces a relevance guard ρ against reward exploitation. However, ρ gates only the StrongREJECT term, not the dominant classifier term, so directly optimizing the evaluation metric still raises a concern about reward hacking. The classifier term has weight 2.0 out of a maximum reward of 3.3, and in Table 2 (ZeroShot × Llama-3-8B) removing it drops ASR to 46.0, roughly the Best-of-N level (49.0). The gain therefore appears to depend heavily on optimizing the evaluation classifier itself.
- The main evaluation relies on a single judge model, which is also used in training. The independent-judge check in Table 5 (Gemini-3-Flash) supports the concern raised in Weakness 1. Although the gain over Best-of-N remains positive for all four victims, the average gain on the ZeroShot substrate falls from +19.1 points under the HarmBench classifier to +9.25 under Gemini with the HarmBench prompt and +7.5 under the stricter rubric, so roughly half of the gain does not transfer to an independent judge. Moreover, on Llama-3-8B, gpt-4.1-mini, and DeepSeek-V4-Pro, Gemini confirms a smaller fraction of Red-TTT's successes than of Best-of-N's (69%, 69%, and 81% vs. 77%, 74%, and 87%), which is consistent with optimization against the training classifier. In addition, this independent check covers only the ZeroShot substrate. It also re-scores only one delivered completion per behavior, whose selection rule is not specified and presumably relies on the HarmBench classifier, instead of letting the independent judge select among all 120 responses.
Questions for authors
- If an independent judge is used both to select and to score the responses (e.g., best of 120 under Gemini), how large is the gain over Best-of-N for all 12 configurations?
- Given that removing the classifier term in Table 2 eliminates most of the gain, if the reward uses a judge different from the evaluation classifier (e.g., train with another classifier or LLM judge, evaluate with HarmBench), does the gain persist?
- For Red-TTT vs. Best-of-N, what are the wall-clock time per behavior, the GPU type and GPU-hours, and the total number of victim calls including StrongREJECT grading (which re-prompts the victim, i.e., roughly 120 attack queries plus 120 grading calls per behavior)? Fig. 3(a) compares against larger Best-of-N budgets only for ZeroShot × Llama-3-8B. How does Best-of-N compare at a matched number of victim calls (e.g., N = 240) in the other configurations?
I'll consider raising score if my questions are addressed.
8 comments
Response to Q3
Q3. Computational cost and victim-call accounting
We agree that attack-query matching alone does not establish computational efficiency. We therefore measured end-to-end runtime, victim-call usage, and estimated FLOPs.
Runtime and computational overhead
Both methods were profiled on the same 12 behaviors using one NVIDIA RTX PRO 6000 Blackwell (98 GiB), with the attacker, victim, and 13B classifier co-resident.
| Component | Red-TTT | Frozen Best-of-N |
|---|---|---|
| Attacker generation | 28.9 s | 0.9 s |
| Victim attack calls | 40.3 s | 3.2 s |
| Reward scoring | 48.8 s | 43.4 s |
| LoRA updates | 11.8 s | 0 s |
| Total runtime | 132.5 s | 48.5 s |
Red-TTT requires 2.73× wall-clock time, corresponding to approximately 7.4 versus 2.7 GPU-hours for 200 behaviors. LoRA updates account for only 8.9% of Red-TTT's total runtime, with the remaining overhead primarily arising from sequential generation and reward evaluation.
Victim-call accounting
Both methods use 120 victim attack calls. Red-TTT additionally performs 120 StrongREJECT grading calls, yielding 240 total calls under this accounting. Frozen Best-of-N performs additional grading only for candidates passing the classifier gate, requiring approximately 1.4–41.7 additional calls per behavior, depending on the configuration. We will clarify that the original comparison matches the number of attack queries rather than total victim-side evaluation calls.
FLOPs and compute-matched comparison
Using measured token counts and analytical FLOPs estimates, Red-TTT requires 4.11 PFLOPs per behavior, compared with 2.15 PFLOPs for frozen Best-of-N (or 1.88 PFLOPs with shared-prefix caching), corresponding to approximately 1.9–2.2× the computation. We further evaluated frozen Best-of-N under larger sampling budgets on ZeroShot × Llama-3-8B:
| Frozen budget N | Frozen Best-of-N ASR |
|---|---|
| 120 | 49.0% |
| 240 | 57.0% |
| 480 | 60.0% |
| 1,000 | 60.5% |
| 2,048 | 63.5% |
| Estimated equal wall-clock (~328) | 59.0% |
Red-TTT achieves 66.0% ASR at N=120. At approximately equal wall-clock time, frozen Best-of-N reaches 59.0%, leaving a 7.0-percentage-point advantage for Red-TTT. We also evaluated the estimated FLOPs-matched budget, corresponding to approximately 230–263 frozen candidates, where frozen Best-of-N achieves 57.0–57.8% ASR. These results show that Red-TTT incurs additional computational overhead but retains an advantage under the estimated compute- and wall-clock-matched comparisons. We will explicitly distinguish attack-query efficiency from overall computational efficiency in the revised paper.
We will incorporate the additional cross-evaluator experiments, clarify the evaluation protocol, and report computational costs more transparently. We thank the reviewer again, and hope these clarifications address the concerns. We are happy to answer any further questions.
Thank you for clarifying the reporting errors and providing the full-benchmark coefficient-sensitivity results. The rerun provides useful evidence of robustness across several nonzero coefficient settings. I also appreciate the additional runtime measurements and clarification of victim-call accounting.
My main concern about evaluator dependence remains insufficiently resolved:
-
Independent-judge evaluation. For the four configurations where Gemini has evaluated all 120 candidate responses, please report behavior-level ASR@120 for both Red-TTT and frozen Best-of-N. The reported fraction of harmful responses in the first and last adaptation rounds measures a different quantity and does not directly establish how much of the original ASR improvement survives independent evaluation.
-
Alternative reward judges. The ShieldGemma and Llama-Guard experiments would be more informative with explicit attacker/victim configurations, evaluation populations, and corresponding frozen controls under the same candidate-selection procedure and attack budget. Nonzero ASR after replacing the reward judge does not, by itself, establish an advantage from parameter adaptation.
The new sensitivity results address part of the experimental uncertainty, but they do not yet establish the magnitude of the improvement when reward computation and evaluation are separated. I therefore maintain my current rating of 2 (Weak rejection), while remaining open to reconsideration if these comparisons are provided.
Response to W1, W2 and Q1, Q2
We thank the reviewer for the detailed and constructive feedback. We conducted additional experiments to investigate evaluator dependence, independently judged attack effectiveness, and computational efficiency.
W1 / Q2.
We appreciate the reviewer's concern that using the evaluation classifier as part of the training reward may introduce evaluator-specific optimization. We first clarify the reward ablation in Table 2. The original comparison inadvertently mixed results from different evaluation subsets and protocols. We will correct the table to ensure consistent evaluation and explicitly report the matched frozen control. Removing the classifier reward achieves 46% ASR, compared with 30% for frozen Best-of-N, retaining a 16-percentage-point advantage.
Cross-evaluator experiments
To investigate whether Red-TTT's effectiveness depends on optimizing the evaluation classifier, we conducted additional experiments using alternative reward judges. We ran the following experiments on ZeroShot × Llama-3-8B hard set:
When using ShieldGemma-9B or Llama-Guard-3-8B, the alternative judge replaces the original classifier in both reward computation and candidate selection. The HarmBench classifier is used only for final evaluation. Switching to independent judges, Red-TTT continues to generate successful attacks. These experiments demonstrate that its effectiveness is not solely attributable to optimizing the evaluation classifier.
Additional benchmark
We additionally evaluated Red-TTT on AdvBench, using the same attacker and 120-query budget. On ZeroShot × Llama-3-8B settings,Red-TTT achieves 59.0% ASR, compared with 48.0% for frozen Best-of-N, a 11.0 percentage-point improvement. This further demonstrates the generalization.
W2 / Q1.
We thank the reviewer for highlighting that independently judging only the classifier-selected final response does not fully eliminate selection bias. To address this concern, we conducted a more comprehensive evaluation using Gemini-3-Flash as an independent judge. Rather than evaluating only the final selected response, Gemini-3-Flash judges all 120 stored candidate responses per behavior for both Red-TTT and frozen Best-of-N, under the same evaluation procedure. Gemini is never used during training or candidate selection. This analysis covers four configurations for which all candidate responses were available.
Independently judged attack effectiveness across adaptation rounds
We measured the fraction of candidate responses judged harmful by Gemini during the first and last three adaptation rounds. For frozen Best-of-N, its 120 candidates were divided into 15 groups of eight for comparison.
The clearest improvement occurs on ZeroShot × DeepSeek-V4-Pro, where the fraction of independently judged harmful responses increases from 4.31% to 10.65% during adaptation, while the frozen control remains approximately unchanged. This provides direct evidence that Red-TTT can improve attack effectiveness under a judge that participates in neither reward computation nor candidate selection. Together with the cross-evaluator experiments in W1, these results indicate that Red-TTT's gains are not exclusively classifier-specific, while also motivating more careful evaluation of reward-model dependence.