- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper constructs MTID with automatically annotated intervention boundaries and trains a response-aware monitor through weighted supervised fine-tuning and offline reinforcement learning.
Critical strengths
-
Intervention timing is a relevant problem, and considering candidate responses alongside dialogue history is well motivated.
-
Matched benign examples, seed-level splits, and adaptive online evaluations are useful experimental choices.
Critical weaknesses
- Limited novelty. The contribution largely combines existing attack generation, automated judging, supervised learning, and policy optimization. Its engineering emphasis and limited insights weaken its suitability for ICLR.
- Limited attacker diversity. Training and primary evaluations rely on CKA. Beyond MAJ, independent multi-turn attacks, including ActorAttack, are needed to assess robustness beyond CKA-specific patterns.
- Missing targeted baseline. Since the annotation pipeline evaluates sufficiency given a known harmful goal, a natural deployment baseline would first infer plausible harmful goals from the dialogue history, then assess whether the accumulated dialogue and candidate response provide sufficient information to achieve them. The paper does not clearly evaluate this explicit two-stage alternative.
Questions for authors
How does TurnGate compare with an explicit goal-inference-then-sufficiency monitor under identical information constraints?
5 comments
Response to Reviewer ST9z (Part 2)
W3 & Q1. Goal inference followed by sufficiency evaluation
Thank you for proposing this baseline. The judge uses the ground-truth harmful objective during annotation and evaluation. TurnGate has no access to it at inference time and sees only the dialogue history, current query, and candidate response. Under the same information constraints, TurnGate achieves an overall score of 0.667, compared with 0.162 and 0.154 for the two-stage Qwen3-4B and GPT-5.2 baselines.
In the first stage, the monitor infers up to three plausible harmful objectives from the observable dialogue. In the second stage, it checks whether the accumulated dialogue, including the candidate response, contains sufficient information to achieve each objective. For example, given a series of individually benign cybersecurity questions, the first stage may infer an unauthorized-access objective. The second stage then checks whether the information accumulated so far is sufficient for unauthorized access.
We use the same A/B/C/D sufficiency rubric as in our annotation pipeline. The monitor outputs BLOCK if any inferred objective is judged as Full Success (D). The implementation also triggers BLOCK upon an explicit provider refusal. Intention Analysis [7] assesses inferred intent. This baseline evaluates the sufficiency of the accumulated information for the inferred objective.
| Supplementary evaluation | Intercepted by boundary | Exact-boundary block | Benign all-pass | Overall |
|---|---|---|---|---|
| Goal sufficiency, Qwen3-4B | 3/34 (8.8%) | 3/34 (8.8%) | 59/60 (98.3%) | 0.162 |
| Goal sufficiency, GPT-5.2, medium reasoning | 4/34 (11.8%) | 2/34 (5.9%) | 37/60 (61.7%) | 0.154 |
| TurnGate | 22/34 (64.7%) | 15/34 (44.1%) | 53/60 (88.3%) | 0.667 |
The evaluation panel contains 34 Full-Success harmful traces and 60 benign traces selected by an outcome-blind per-seed rule. Both stages receive the complete observable context. Qwen3-4B-Instruct-2507 uses greedy decoding and 512 output tokens per stage. GPT-5.2 uses medium reasoning and up to 4,096 tokens per stage, including reasoning. We report each baseline's highest overall score among ordinary-goal and capability-oriented prompt variants.
TurnGate intercepts more harmful trajectories than either two-stage baseline while preserving 88.3% of benign dialogues. Even with GPT-5.2, explicitly prompting the model to infer goals and assess sufficiency performs substantially worse than TurnGate.
References
[1] Wei et al. The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search. arXiv preprint, 2025.
[7] Zhang et al. Intention Analysis Makes LLMs A Good Jailbreak Defender. COLING, 2025.
[8] Ren et al. Derail Yourself: Multi-turn LLM Jailbreak Attack through Self-discovered Clues. arXiv preprint, 2024, v1.
[9] Srivastav and Zhang. Safe in Isolation, Dangerous Together: Agent-Driven Multi-Turn Decomposition Jailbreaks on LLMs. REALM Workshop, 2025.
We thank the reviewer for recognizing the motivation behind our response-aware design and the value of our experimental evaluations. We appreciate the valuable feedback and would be happy to provide further clarification if any concerns remain.
Response to Reviewer ST9z (Part 1)
We thank Reviewer ST9z for the insightful review and for finding the response-aware design well motivated. Below, we respond to the questions on novelty, attacker diversity, and the goal-inference-then-sufficiency baseline.
To our knowledge, TurnGate is the first work to formulate multi-turn defense as an information-sufficiency boundary localization problem. We provide explicit supervision for this boundary and optimize intervention timing. The intervention target is the first turn at which releasing a candidate response makes the accumulated dialogue sufficient to realize a harmful objective.
Identifying harmful intent alone does not tell us when to intervene. Harmful intent may be present from the beginning of a dialogue, while the information needed to realize it accumulates across turns. Blocking too early sacrifices dialogue utility. Blocking too late allows sufficient harmful information to be delivered. With MTID and a sequential stopping objective, we make this safety–utility trade-off an explicit learning problem.
Table 1 shows why intervention timing matters. Trajectory-level Naive-SFT has few misses (8.9%), yet it blocks early on 62.9% of harmful trajectories and preserves only 60.9% of benign sessions. Uniform turn-level SFT preserves 93.0% of benign sessions but misses 73.8% of harmful trajectories. TurnGate achieves the highest exact-turn accuracy (41.4%) and overall score (0.699).
Thank you for suggesting ActorAttack [8]. We added an evaluation on 50 objectives to further assess TurnGate's robustness beyond CKA, complementing the existing independent MAJ [9] evaluation in Fig. 4d.
For MAJ, ASR falls from 0.780 to 0.631 for Chemistry → Cybersecurity and from 0.715 to 0.593 in the reverse direction.
We evaluate ActorAttack on 50 objectives fixed before observing outcomes, with shared initial plans, a GPT-5.2 target, three actors per objective, up to three rewrites after refusals, a Qwen3-8B-abliterated controller, and Gemini-2.5-Flash judging. Candidates are gated before delivery, and follow-ups depend on delivered responses.
TurnGate reduces ActorAttack ASR from 34% to 10%, a relative reduction of approximately 71%. Together with the consistent improvements against MAJ, these results provide evidence that TurnGate generalizes beyond the CKA attack patterns used in training.
CKA remains our primary stress test because The Trojan Knowledge [1] reports higher HarmBench Full-Success rates than MAJ and ActorBreaker on GPT-oss-120B and Claude-Haiku-4.5. ActorAttack [8] refers to the v1 paper. The prior comparison uses the name ActorBreaker.