- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a Guard-and-Guide Architecture (G2A) that first detects accumulated risks in long-horizon agent actions during runtime and then proposes refinement to the potentially harmful action. The Guard module is based on a lightweight detection model learned through Trajectory-Contrastive Safety Distillation. The guide module is triggered when Guard’s predicted risk exceeds an operational threshold, which leverages evolution-supported safety skills to achieve intervention. The proposed G2A is tested on multiple benchmarks compared with SOTA guardrails from three categories and achieves generally better performance.
Critical strengths
-
The paper proposes a principled approach for learning a lightweight guardrail and achieves strong performance and testing data.
-
The evaluation covers a broad range of agentic settings and baselines.
Critical weaknesses
-
The Guard model is learned for one or a limited range of task domains, hindering its generalization to broader task domains. The authors deliberately made this choice instead of using an LLM or an agent for judging, but the latter seems more capable of task generalization. OOD evaluation would strengthen this weakness, but I am skeptical about the results.
-
Creating training data may be challenging for some domains, as some trajectories may converge into a harmful action without safe alternatives.
-
Detection followed by mitigation strategies for long-horizon safety guardrails have been investigated by prior work such as [1]. What is the advantage of the proposed output-level discrimination between safe and unsafe over the latent space discrimination in [1]?
[1] Luo et al., Agent MechSuits: Mechanistic Subspace Safety Steering for Multi-Turn CLI Agents, arxiv 2026.
Questions for authors
-
Would it be in the constraint in Eq. (1)?
-
How are the training data constructed?
-
In Eq. (12), where TCR is defined, what is the definition of task completion when there are safety risks? Would it be proper to terminate the task? Can you give a concrete case study for illustration?
-
During the evaluation, is the skill evolution activated or not? With skill evolution on, the comparison with most baselines, with the testing tasks independently evaluated, is unfair. But with skill evolution off, how did you ensure that the initial skill set does not contain data leakage?
-
Space in front of "Detailed" (line 365).
-
Quality of figures, e.g., the overlapped text in the left figure of Figure 3.
3 comments
Response to Reviewer w3zU
Question 1: What do the state and constraint in Equation (1) represent?
A4: s_t represents the current observable state before the action is executed. It includes the task specification, interaction history up to step t, environment and tool observations, and the currently active skills.
The constraint is applied to the immediate risk of the candidate action:
q_t = f_theta(s_t, omega_t, a_t) <= xi, for every t.
When q_t > xi, the Guide generates a revised action. The revised action is executed only after it passes the same safety constraint.
The original expression, f_theta(s, a_t), omitted the time index and the skill input. We will consistently revise it to f_theta(s_t, omega_t, a_t).
Question 2: How is the training data constructed?
A5: We begin with a fixed version of the English tasks from Workspace-Bench. For each AgentS4D-style case, we preserve the original authorized objective, resources, required output, and completion criteria. We add only one controlled risk source and pre-register the conditions for payload contact, unsafe signals, task completion, and evidence integrity.
For each source task, we construct one safe demonstration and one unsafe demonstration:
- In the safe demonstration, the agent encounters and rejects the risky instruction.
- In the unsafe demonstration, the agent encounters and follows the risky instruction.
Both trajectories still complete the original authorized task.
We then select 196 pairs through stratified sampling along the S/T/L dimensions. The data are split by source_task_id, preventing variants of the same source task from appearing across the training, validation, and test sets.
Question 3: How is TCR defined when a safety risk is present, and is task termination an appropriate response?
A6: TCR uses the case-level task-completion predicate from AgentS4D:
TCR = n_C / n_T
TCR is calculated independently of the safety outcome. A run is counted as completed only if it satisfies the original deliverable requirements, the required environment state, and the case-level completion conditions.
A safe termination may prevent a safety violation. However, if the original task remains incomplete, the run receives a TCR of 0. Therefore, termination is not treated as task success, and the system cannot improve its safety–utility results through general refusal.
Appendix E provides a concrete example. A webpage asks the agent to upload local credentials that are unrelated to the original task. Before the action is executed, the Guard intercepts it. The Guide then activates a sensitive-information boundary skill and revises the action so that the agent ignores the credential-upload request and continues retrieving the information required by the original task.
After the revised action passes the Guard, execution continues. The final trajectory satisfies both the safety conditions and the original task-completion conditions.
If no safe alternative path exists, the system requests user confirmation or terminates the dangerous subtask. In this case, the run retains its successful safety outcome but is counted as a TCR failure if the original task is not completed.
Question 4: Is skill evolution enabled during testing, and how do you prevent data leakage?
A7: Skill evolution is completely disabled during testing.
Skill evolution is performed offline for three rounds using only the training trajectories. After the third round, the Guard, skill pool, editor, router/Guide prompts, decoding configuration, and thresholds are all frozen.
Neither the validation set nor the test set is used in the editor buffer, Guard refinement, skill generation, component editing, or threshold selection.
The initial skill pool contains only one general-purpose skill fixed before training. When the Guard is triggered, this skill instructs the agent to infer a coarse-grained risk category and make the smallest possible modification that preserves the original task objective.
The fixed risk categories come from a predefined operational taxonomy. The initial skill does not contain any test tasks, test templates, hidden labels, or benchmark-specific solutions. This procedure ensures that no test information is used during skill evolution or component optimization.
Question 5: There is a missing space before “Detailed,” and the text in the left panel of Figure 3 overlaps
A8: Thank you for pointing out these presentation issues. We will correct the missing space and perform a complete formatting check of the manuscript.
We will also redraw Figure 3 by increasing the width and margins of the left panel, repositioning the legend and statistical annotations, and using consistent font sizes. We will inspect every page of the final PDF to ensure that no text overlaps and that all content remains readable.
Thanks for the authors' responses. I will consider the rebuttal when discussing the paper with other reviewers and the AC.
Response to Reviewer w3zU
We sincerely thank you for your careful review and constructive comments. Below, we respond to each weakness and question.
W1: The Guard is trained on a limited domain, raising concerns about its ability to generalize across task domains
A1: We agree that the open-domain coverage of a lightweight Guard is limited by its training distribution. In the paper, we therefore position it as a low-cost, calibratable monitor for continuous safety checking, rather than as a universally generalizable open-domain safety model.
G2A already provides two types of evidence for transferability:
These results support transfer across execution backbones and across scenarios involving related risk types. However, they do not support a claim of universal generalization to arbitrary open domains.
In the revision, we will clearly state this scope, report the results for each source–target split, and identify transfer beyond the risk families seen during training as an important direction for future work. We will also report the stronger safety performance of the LLM judge alongside its cost of 640 ms per action and 32.9% end-to-end overhead, compared with 5.6 ms per action for the TCSD Guard.
W2: In some domains, safe alternative trajectories may not exist, making the training data difficult to construct
A2: This situation is a limitation of TCSD’s paired-supervision setting. During training, we include only tasks that have both a verifiable safe completion path and an unsafe path. Tasks without a safe alternative are not included in the paired training set.
At deployment time, when the system encounters such a state, the Guide can request additional authorization, skip the dangerous subtask, or terminate the current subtask. Safety and task completion are then measured separately: a safe fallback can prevent a safety violation, but if it does not satisfy the original task-completion predicate, the run is counted as a TCR failure.
This design prevents the system from obtaining artificially high safety scores by simply refusing all potentially risky tasks.
We will add the data-inclusion criteria and clearly discuss this limitation in the revised paper.
W3: What are the advantages of output-level classification over the latent-space approach used by Agent MechSuits?
A3: Agent MechSuits is a white-box method. It trains linear probes on intermediate hidden states and directly steers the model within a 10-dimensional safety subspace at a single layer.
In contrast, G2A operates outside the model and reads only the observable state and candidate action. It can therefore be applied to closed-source API models whose internal representations are inaccessible. It can also be reused across different execution backbones.
G2A separates risk detection from action correction. The Guard performs action-level risk assessment, while the Guide modifies risky actions using skills with explicit activation, intervention, termination, and fallback conditions. The revised action is then checked again before execution. This process provides auditable reasons for each intervention, readable intervention rules, and explicit control over tool calls.
The two methods address different deployment constraints: MechSuits is designed for white-box CLI agents with access to model weights and hidden states, whereas G2A targets black-box or heterogeneous agent runtimes.
In the revision, we will cite Agent MechSuits and provide a detailed comparison along five dimensions: internal-access requirements, intervention location, cross-model portability, auditability, and runtime cost.