- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that past methods developed for safety tasks (e.g., jailbreak detection, agent action authorization) start from the same base conversation (natural-language) input. Instead, this paper motivates representing conversations with a universal, semantic state. This state can take the form of a graph-like structure, where entities and cross-entity relationships are established as a conversation proceeds. The work proposes a learned Markovian updater to continually revise the state (turn-by-turn). Once the state representation is trained, it is frozen and task-specific readers accept this state representation for use in some downstream task.
Critical strengths
- (S1) Compelling motivation. The motivation behind a universal, Markovian state representation of a conversation is compelling and has many applications (e.g., addressing the long-context problem in LLMs). The graph-like representation introduced in the method also seems compelling.
Critical weaknesses
- (W1) Largely unreadable. I feel pessimistic that any human reader will gain benefit from this paper. While I feel that the motivation for the paper is strong, as a reader, I am left with many doubts and unanswered questions on the method and the experimental results. Even after spending time manually reading the main paper, I am unable to understand how the method works (specifically, the training), and comprehending the experimental setup and results is difficult. I put a non-exhaustive list of examples below to illustrate:
- Reliance-versus-value decomposition in line 151 is introduced without context. Not sure what this means.
- Is line 152 grammatically correct? I’m unable to understand what this means.
- Matched occlusions (line 154) is not defined.
- It is unclear which parts in 2.1 and 2.2 are M1 and M2. How do M1+M2 differ from the complete inducer F_theta (referenced in line 219)
- Topical shortcuts are never defined. It makes the experimental setup in lines 240-250 hard to understand. What are topic-matched negatives? Please define AUROC, TPR, FPR.
- Table 1 is introduced on page 2, and only referenced in page 9 (related works). Table 2 is never referenced in the main text. It will be difficult for readers to understand these tables without any context.
- What are cross-paper defense numbers (line 256)?
- How are the primary metrics defined in line 263 defined> Specifically, benign utility & utility under attack.
I am not trying to be annoying. I merely want to provide evidence to convey my point. I strongly encourage authors to consider a substantial re-write of the paper so that readers can benefit from the main ideas and intuitions presented in this work.
- (W2) Training objectives & loss functions are fully unspecified in the main text. I deem this to be a crucial methodological component. It's wholly unclear how training is done on M1+M2 (e.g., loss function, source of supervision, etc) and there are similarly no details on how fine-tuning of the reader is done. There seems to be some text about this in the Appendix, but these sections are never referenced in the main text (as far as I was able to tell), and the Appendix is ~35 pages long, so it is unreasonable to expect the reader to go hunting for important methodological details there.
Questions for authors
No questions.
6 comments
Part 2/2
W2. Training objectives and loss functions
We agree that the training procedure is central and should not have to be reconstructed from the appendices. A new “Training” paragraph in Section 2 summarizes the supervision, joint objective, optimization, and reader training, with implementation detail in Appendices C (M1/M2) and E (readers).
Supervision (upstream training without safety labels). M1+M2 is distilled from a structured teacher that labels typed mentions, persistent-node assignments, and typed relations (Appendix C.1). The 64,888-conversation corpus combines Chatbot Arena dialogue and Claude-generated synthetic conversations (Appendix C.2). No jailbreak, injection, or topic-shift labels are used to train M1+M2. After the inducer is frozen, the three readers are trained separately on their respective task labels. Teacher-label noise is analyzed in Appendix C.7.
Joint objective. The heads are jointly optimized, with gradients flowing into the shared ModernBERT-large encoder:
- (M1): focal token classification (γ = 2) over BIO tags for the four mention types; class weights O = 1, B = 2, I = 1.5, with inverse-square-root frequency weighting across types.
- (M2, identity): cross-entropy over legal existing-node candidates, NEW, and UNRESOLVED, supervised by the teacher's persistent graph.
- (M2, relations): multi-label binary loss over nine relation types, using a learned adaptive threshold per pair (following ATLOP) and type-conditioned expert scoring.
- : an auxiliary loss encouraging agreement between attachment and relation predictions.
Optimization. AdamW; learning rates 2×10⁻⁵ (encoder) and 1×10⁻⁴ (heads); six epochs; checkpoint selected by validation mention-plus-relation F1 (Appendix C.4). M1+M2 is then frozen before training any task reader.
Reader training (Appendix E). Each reader uses the frozen induced state. A node's 1,034-dimensional input comprises its 1,024-dimensional span embedding and 10 structural features, followed by an MLP to 128 dimensions and two R-GCN layers over nine relation types. Readers use focal cross-entropy (γ = 2) on task labels and a learning rate of 3×10⁻⁴, with task-specific readouts and schedules:
- Jailbreak (929K parameters; Appendix E.2): three-view pooling, causal GRU, a Pre head for user turns and a Screen head for assistant turns; four epochs, validation-F1 checkpoint; trained only on Crescendo.
- Action Guard (24.8K parameters; Appendix E.3): action node, mean goal node, and mean constraint node passed to a two-layer head; six epochs, trained only on InjecAgent. The checkpoint and 3% false-block target threshold are chosen on InjecAgent validation, without using AgentDojo.
- Topic shift (929K parameters; Appendix E.4): jailbreak-style pooling and GRU; six epochs, trained on SuperDialseg, selected on DialSeg711, and evaluated on TIAGE.
Inference. For each new turn, M1 extracts typed mentions; M2 links each to an existing persistent node, creates a new node, or marks the mention UNRESOLVED, and predicts typed relations. The task-specific reader makes its prediction from the updated state without modifying the frozen inducer.
Revisions. Section 2 now includes notation and training summaries; Section 3 defines the metrics and points to task-specific appendices; and every table is cited at first mention. We thank the reviewer again and hope these changes make the method easier to understand and evaluate. All revisions added in the latest uploaded version.
I thank the authors for the efforts.
The introduction has certainly improved. However, I’m still concerned about the paper's readability. It feels like the paper conducts a large array of experiments, but seldom connects / curates / condenses the findings. The result is that a reader needs to continuously jump across many sections when reading the paper to understand what's going on. Even if the method is strong, this makes it hard for readers to parse the paper and take away insights for their own problems.
At this point, I have independently read the paper 4 times, and while the re-written manuscript helps clarifies previously raised concerns, it has also surfaced new issues:
Significant
-
Reproducability concern: It is not known which frontier-LLM interface is used to generate the teacher labels. The prompts used to generate teacher labels is not provided. No link to the training dataset is provided. I’m doubtful that any independent researcher would be able to reproduce this training dataset, which seems to be a critical part of the training.
-
Concerns about Table 2: it seems possible that the fine-tune baseline is overfitting, which may explain the lower results in Table 2. Especially since all weights in the model are trained. The paper itself has stronger backbone-controlled baselines in Table 43 of the Appendix (K3) which would be better to place in Table 2 (frozen backbone + trained “reader”-head. I am confused why the authors did not use these results in Table 2.
-
Presentation: Section 2.1 should probably explain that per-token BIO heads for each token type help identify the span over each extracted mention m. The BIO has are mentioned only in the training text (line 235).
-
Presentation: An image showing architecture + training would be helpful in clarifying the method described in Section 2.
Minor
-
What are the “10 structural features” in line 246? That’s the first time the term “structural features” is used.
-
The role of Pre/Screen heads in line 248 are not clearly defined.
-
“InjecAgent” not defined when it is first introduced in line 249
-
Table 1: The AUROC label is in the wrong task
-
The in conversation prefix (line 166) are not defined. Is it different than used in Fig. 1? If they are the same, please unify the notation.
-
Unclear what is, even after checking Appendix.
At this point in the review phase, I still recommend the paper for rejection, and I strongly suggest the authors to rewrite their paper. The readability issues continue to interfere with my ability to assess the methodological & technical strengths of the work. The authors may benefit from using an outline like so:
-
Introduction (this section is largely fine, though Table 1 and 2 should probably be moved to results)
- Motivate a persistent state
- Introduce and name the three tasks this paper focuses on (multi-turn jailbreaks, agent action authorization, topic-shift detection).
- Fig. 1
- Contributions
-
Method
- Notation
- Mention Encoder (M1)
- Graph Inducer (M2)
- Training Dataset Construction
- Training Loss Functions
- Design of Readers (probably better to defer per-task specific details)
-
Main Benchmark
- Task Description + Metrics
- Main Results (Table 1 and 2)
-
Transfer Evaluation
-
Ablations & other analysis
- intrinsic quality of M1+M2 inducer (Table 3)
- Occlusion vs. matched retrains
- deployment cost (can probably be reduced to one paragraph)
-
Related Work + Limitations + Conclusion
If the authors are able to respond in the remainder of the discussion period, I will make my best effort to review their response and revise my assessment, but time is limited and re-reviewing the paper takes significant effort on my part. Rather than rushing, it may be better for the authors to take the time to work on the manuscript and re-submit at a future venue.
Part 1/2
We thank the reviewer for the careful reading and for finding the motivation compelling. We agree that the main text assumed some familiarity with both the terminology and the experimental setup. We have added definitions, clarified the methodological components, and made the connections between the experiments and their motivation explicit. Below we address each readability concern. Part 2 summarizes the training recipe now included in the main text, with implementation details in the appendices.
W1. Readability: point by point
“Reliance-versus-value decomposition.” Reliance measures how much a trained reader's performance changes when a state channel (e.g., edges) is removed at inference. Value measures its contribution by comparing matched readers trained with and without that channel. A reader may rely on edges at inference even if a reader retrained without them performs similarly. Both terms are now defined at first use, with results reported side by side in Section 4.
The contributions sentence. We replaced it with: “We separate what transfers from what inflates transfer using three controls: reader-level versus representation-level exposure, shortcut probes, and position-matched evaluation.”
“Matched occlusions” and “matched retrains.” An occlusion removes a channel (edges, node semantics, or node types) at inference from an already trained reader. A matched retrain trains a reader of the same size and with the same procedure, but without that channel from the start. These are now defined where first mentioned.
M1, M2, and the inducer. M1 is the mention encoder (Section 2.1); M2 is the graph inducer (Section 2.2). Together they form the jointly trained 419M-parameter inducer, F_θ, which is subsequently frozen. A notation paragraph at the start of Section 2 and the description of Eq. (2) now make this explicit.
Topical shortcut, topic-matched negatives, AUROC, TPR, and FPR. A topical shortcut is a feature that distinguishes the evaluated classes without capturing the intended attack behavior. Here the first user turn's topic is predictive because attack conversations concern harmful topics while MT-Bench-101 benign conversations often concern homework or coding. A first-turn TF-IDF probe scores 0.87 AUROC on that generic split but only 0.53 against topic-matched benign twins: conversations on similar topics without the harmful objective. AUROC measures how often a random positive is ranked above a random negative (0.5 is chance). TPR at 1% FPR is the fraction of attacks detected when 1% of benign conversations are flagged. A new “Metrics” paragraph in Section 3 defines these measures.
Tables 1 and 2. Table 1 (positioning against published systems) is now cited and introduced in Section 1. Table 2 (same-backbone comparison) is cited there and discussed in Section 4. Every main-text table now has an explicit in-text reference.
“Cross-paper defense numbers.” These refer to results reported in other papers under potentially different AgentDojo versions, task and injection sets, agent snapshots, and thresholds; they are therefore not directly comparable. The revised text explains why we instead rerun competing defenses in the same live harness (Table 20).
Benign utility, utility under attack, and targeted ASR. Benign utility is the fraction of legitimate tasks completed with the defense enabled and no attack present; utility under attack is the fraction completed when injected instructions are present. Targeted ASR is the fraction of runs in which the attacker's injected objective succeeds. These definitions now appear in the Section 3 “Metrics” paragraph rather than only in Appendix G.
Each task-setup paragraph in Section 3 now points to the appendix sections covering its data, reader, and evaluation protocol. We have revised Sections 1–3 accordingly.