- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HILT turns unsafe agent trajectories into localized corrective supervision. Positive changes in a verifier jury's risk scores estimate hazard onset and commitment boundaries, which determine the student's starting prefix and the evidence supplied to a privileged teacher. On-policy self-distillation transfers this guidance without an inference-time verifier. Across five main model configurations, HILT reduces over-refusal relative to whole-trajectory OPSD while maintaining low observed attack success rates. The main unresolved questions concern the contribution of hazard localization and its effectiveness in interactive agent environments.
Critical strengths
-
The paper addresses a meaningful safety–utility problem. Separating where correction begins from the evidence guiding it provides an intuitive way to learn from failed trajectories while preserving legitimate task progress.
-
The reduction in over-refusal is consistent across the main model configurations. Relative to whole-trajectory OPSD, HILT reduces ORR by 6.09–23.60 percentage points across three model families, with significant paired differences reported in Table 8. Reporting task success and reasoning accuracy alongside safety also makes the remaining utility costs visible.
Critical weaknesses
-
The experiments do not establish the added value of hazard localization over simpler local correction. The main ablation changes the continuation prefix, privileged evidence, and teacher output instructions together: HILT requests a direct replacement, while the outcome-record teacher requests reasoning before continuing. Each change can affect the supervision independently of boundary quality. The comparison therefore supports the combined training recipe but does not identify what the trigger–commit formulation contributes. Matched teacher instructions and controls using trigger-only evidence or simpler localization are needed to substantiate the central methodological contribution.
-
The boundary evaluation does not establish that correction begins while recovery is still possible. Equation (6) estimates commitment using the last risk increase after the trigger, although commitment is defined as the first loss of recoverability. It also excludes coincident boundaries allowed by the annotation protocol. Since the trigger determines the correction prefix, the more direct test is whether that prefix precedes annotated commitment. The reported within-one-step accuracy does not answer this question. Pre-adjudication annotator agreement is also unreported, leaving uncertainty in the reference labels. These gaps limit the safety interpretation of the localization results (Section 3.3; Appendix I).
-
The baseline implementations are too underspecified to support a convincing comparison. Section 4.1 and Appendix D give shared training settings but leave essential method-specific choices unclear, particularly reward construction and the generation, filtering, and verification of corrective supervision. Matching optimizer steps does not establish that competing methods were faithfully adapted or adequately tuned. All policy training also uses a single seed, while the reported confidence intervals measure evaluation-item uncertainty. Consequently, the comparisons leave both implementation fidelity and sensitivity to training randomness unresolved.
-
The evaluation protocol is insufficiently specified to assess interactive agent safety. Harmful tasks are adapted to a common scaffold and graded from TASK and RESPONSE, but tool execution, subsequent observations, and environment-level success criteria are not adequately described. This prevents assessment of whether the reported metrics capture harmful changes to the environment and successful completion of legitimate tasks. Those behaviors are central to the paper's motivation of intervening before irreversible harm.
Questions for authors
-
What does hazard localization add under controlled supervision? Please compare trigger-only evidence and a simpler localization rule while matching teacher output instructions, training records, and optimization settings.
-
How reliable are the boundary labels and selected correction locations? On the annotated set, please report pre-adjudication trigger/commit agreement and the fraction of selected prefixes preceding annotated commitment, including cases with coincident boundaries.
-
Are generated tool calls executed during evaluation? If so, how are subsequent observations produced and harmful outcomes and legitimate task completion verified in the environment?
-
Are the main gains stable across training seeds? Could the authors repeat the HILT–OPSD comparison on one representative backbone with additional seeds and report variation in ASR, ORR, and TSR?
9 comments
Response to Reviewer duH9 (2/4):
W2 / Q2. Selected correction locations. We thank the reviewer for proposing the direct test. We agree that the within-one-step accuracy in Section 4.4 does not answer this question, so we ran the test.
Additional analysis: does the correction prefix precede the annotated commit? The trigger alone sets the correction prefix: the student starts from the prefix before the estimated trigger (Section 3.3). We evaluated the prefixes selected on the 350 annotated trajectories by the estimator of the paper (Eqs. 3–5) with the five-member jury used to build the training records.
| Outcome on the 350 annotated trajectories | Count | Share |
|---|---|---|
| Correction prefix before the commit | 340 | 97.1% |
| – estimated trigger earlier the annotated trigger | 249 | 71.1% |
| – estimated trigger after the annotated trigger | 91 | 26.0% |
| Correction prefix starts at or after the annotated commit | 4 | 1.1% |
| No trigger estimate | 6 | 1.7% |
In 97.1% of the trajectories the correction starts before the annotated commit, and in 1.1% it starts too late. When the estimate is earlier than the annotated trigger, the correction starts earlier and is still in time. A trajectory without a trigger estimate receives no localized correction and is used as an outcome record (Section 3.3).
Why does Eq. (6) take the last risk increase and not the first? Because the first increase after the trigger is usually too early. The assessed risk often rises again at later steps where recovery is still possible, so a first-increase rule stops before the commit. We tested both rules on the annotated set with the same estimator and jury (additional analysis). The comparison covers the 275 trajectories with a commit estimate; of the 350, 6 have no trigger estimate and 69 have no commit estimate.
| Commit rule | Same position as the annotated commit | Before it | After it |
|---|---|---|---|
| Last increase after the trigger (Eq. 6) | 45.8% | 27.3% | 26.9% |
| First increase after the trigger | 26.5% | 65.1% | 8.4% |
- The last increase matches the annotated commit more often: 45.8% against 26.5%.
- The first increase is too early in 65.1% of the cases.
- Eq. (6) does not systematically overshoot. Its errors are balanced: 27.3% early and 26.9% late.
- The commit rule does not decide where the correction starts. That depends only on the trigger. The commit only selects the evidence the teacher sees.
Does Eq. (6) exclude coincident boundaries? Yes. Eq. (6) searches only after the estimated trigger, so it cannot return a commit at the trigger itself. For such a trajectory the commit stays unresolved, and the teacher receives trigger evidence only (Section 3.3). This does not change where the correction starts, which depends only on the trigger.
Are coincident cases included in the 97.1%? No. The annotation protocol allows coincident boundaries (Appendix I), but the 350 annotated trajectories contain none: the annotated commit always comes after the annotated trigger. The 97.1% is therefore measured on trajectories with distinct boundaries. We will state both points in Section 3.3 and Appendix I.
Response to Reviewer duH9 (3/4): annotation reliability, baselines, and seeds
W2 / Q2 (continued). Inter-annotator agreement (additional analysis). We thank the reviewer for asking. The table below reports the agreement between the two annotators on their independent labels, before adjudication (Appendix I). Exact agreement means that both annotators mark the same position. Agreement within one entry means that their positions differ by at most one trajectory entry. The mean absolute difference is the average distance between their positions, in entries.
| Boundary | Annotators | Trajectories | Exact agreement | Agreement within one entry | Mean absolute difference |
|---|---|---|---|---|---|
| Trigger | 2 | 350 | 78.83% | 89.78% | 0.41 |
| Commit | 2 | 350 | 76.33% | 88.70% | 0.67 |
The two annotators mark the same position on more than three-quarters of the trajectories for both boundaries, and they are within one entry of each other on about 89%. The remaining cases were resolved by joint review. We will add this table to Appendix I.
W3. Baseline specification. We thank the reviewer for raising this, and we apologize that the method-specific choices are not described. All trained methods share one recipe (Section 4.1; Appendix D): the 26,881 trajectories with binary safety outcomes, rank-16 LoRA adapters, AdamW at learning rate 1 × 10⁻⁵ with linear decay, an effective batch size of 128, 211 optimizer steps (one epoch). No method is tuned per model, HILT included. No method executes tools during training, so the baselines build their supervision from the logged trajectories or from the policy's own text.
| Method | Reward or supervision | How the targets are generated and checked |
|---|---|---|
| SFT | Cross-entropy on a target response | Unsafe trajectory: a refusal message. Safe trajectory: the recorded final response. |
| GRPO | +1 for refusing on an unsafe trajectory or answering on a safe one, −1 otherwise | The policy samples a group of responses. A response counts as a refusal when it contains one of a fixed list of refusal phrases. |
| CSO | Preference pairs at a critical step, trained with DPO | Critical step: the earliest consequential action of the unsafe trajectory. Rejected: the logged unsafe action. Preferred: a constructed safe, task-preserving action. The alternatives are not verified by execution. |
| FATE | Supervised training on repairs, followed by preference optimization | The current policy generates new trajectories; its failures are collected and repaired by the same policy. Candidate repairs are scored by a rule-based check in place of an external verifier. |
| OPSD | Eq. (8) | The teacher additionally receives the binary safety outcome (Appendix C). |
We will add a per-method appendix with these descriptions and the full settings of each baseline, so that the comparison can be assessed and reproduced.
W3 / Q4. Stability across training seeds. We thank the reviewer for this point. Each model in the paper is trained with one seed, and as the reviewer notes, the intervals in Appendix H measure uncertainty over evaluation items.
- What we have: the same result on five separately trained models. HILT and OPSD are trained separately on each of the five base models, which span three families. HILT has lower ORR in all five, by 6.09%–23.60% (p < 0.001 each). Seed noise would not be expected to produce a difference of the same sign and this size in five independent pairs of runs.
- What is running: the comparison the reviewer asks for (additional experiment). We are training three seeds each of HILT and OPSD on Qwen3.5 2B and will report the variation in ASR, ORR, and TSR in the follow-up comment.
Response to Reviewer duH9 (1/4):
We sincerely thank the reviewer for the precise and constructive review. We address the four weaknesses (W1–W4) and the four questions (Q1–Q4) together, in the order of the review. Results that are not in the paper are marked as additional experiments. Three are complete and reported here: training with only the localized unsafe trajectories (W1), the placement of the correction start and the commit rule (W2, Q2), and the inter-annotator agreement (Q2). Two are still running and will follow in a separate comment: the controlled comparison (Q1) and the additional training seeds (Q4).
W1 / Q1. What does hazard localization add under controlled supervision? We thank the reviewer for this precise point, and we agree with the description of the ablation. We first give the controlled comparison the reviewer asks for and the evidence the paper already contains, and then the supporting results.
(a) Additional experiments in progress: the controlled comparison of Q1. We are running this comparison on Qwen3.5 2B. All four settings below use the same teacher output instruction (direct replacement), the same 26,881 training trajectories, and the same optimization hyperparameters.
Training is in progress, and we will post the results in a follow-up comment as soon as they are complete.
(b) What the paper already shows about simpler local correction.
(c) What the paper's ablation shows. Appendix P compares HILT with OPSD under the same distillation objective and optimization schedule. As Section 4.4 states, it tests the joint contribution of the student's starting prefix and the teacher's boundary evidence. The reviewer is right that the teacher instruction also differs: the localized teacher is asked for a direct replacement, and the outcome teacher is asked to reason before continuing (Appendix N). The comparison shows that the localized recipe as a whole reduces ORR by 6.09%–23.60% on every base model (p < 0.001, Appendix H). The first setting in (a) removes the instruction difference.
(d) Additional experiment: training with only the localized unsafe trajectories. HILT's training set is constructed from localized unsafe trajectories and from safe trajectories with their safe outcome label, which are handled as in OPSD (Appendix C). The variant below keeps only the localized unsafe trajectories, with the same localized records and objective.
The result is close to HILT. HILT's advantage over OPSD therefore comes from the localized records, not from the records it shares with OPSD. This narrows the source of the gain; the runs in (a) then separate what the localized records contain.