- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RISA for long-horizon safety in LLM agents. It predicts a distribution over the first future safety violation, derives cumulative risk estimates across multiple horizons, and uses a policy trained by GRPO to control the strength of a safety activation direction. Experiments across three benchmarks and four backbones report improvements in unsafe completion, task utility, and over-refusal relative to the evaluated safeguards. Additional analyses examine forecasting accuracy, steering directions, and intervention rewards.
Critical strengths
-
The framework connects future-risk prediction to graded intervention in a clear and useful way. Deriving cumulative risks from a shared first-violation distribution guarantees consistency across horizons (Sec. 3.1). Training supervision and inference inputs are explicitly separated, and trajectories are partitioned before prefix extraction to avoid leakage between closely related prefixes.
-
The evaluation considers safety and utility jointly and includes informative controls. For example, removing the utility reward improves UCR but substantially worsens utility and over-refusal (Table 10), supporting the need for a balanced objective. The matched-magnitude positive, reversed, and random steering controls also provide behavioral evidence that the learned direction captures safety-relevant structure (Table 8).
Critical weaknesses
-
The timing evaluation is inconsistent with the stated policy. Appendix I.4 uses a strictly positive Beta mean for intervention strength, while Eq. 33 defines onset as the first nonzero intervention. This would trigger intervention at the first eligible step, leaving Table 3’s lead-time difference of 3.29 versus 1.94 unexplained. An effective-intervention threshold or a revised definition is needed to support the timing claim.
-
The benefit of adaptive control is not consistently established. On Qwen3-4B/BLINDSPOT, fixed positive steering in Table 8 outperforms full RISA in Table 2 on UCR/Utility/ORR: 0.276/0.817/0.089 versus 0.300/0.775/0.120. If the protocols are comparable, this qualifies the claimed advantage of adaptation; otherwise, the differences need clarification. Matched safety–utility curves would provide a more conclusive comparison.
-
The no-intervention reference does not affect the stated policy update. Its contribution to Eq. 24 is constant across candidates and therefore cancels from both the centered rewards and their standard deviation in Eq. 26. Since Appendix I.5 excludes the reference candidate from normalization, the matched rollout supplies no intervention-relative learning signal. This does not invalidate the remaining objective, but its methodological role is overstated.
-
Risk reduction is not independently validated on intervention-generated trajectories. The frozen predictor supplies the risk-reduction reward and also measures the improvement in Figure 3. Although Appendix F acknowledges this limitation, Appendix D does not separately assess calibration after intervention. Such evaluation is needed to establish whether the controller reliably reduces downstream risk rather than exploiting prediction errors; the end-to-end results alone do not establish this mechanism.
Questions for authors
-
Is an activation gate or effective-intervention threshold used in practice? How should Eq. 33 be interpreted under Beta-mean evaluation, and does the lead-time advantage remain under a consistent activation definition?
-
Are Tables 2 and 8 directly comparable? What explains the positive-direction control outperforming full RISA on Qwen3-4B/BLINDSPOT, and how do their safety–utility curves compare under matched tuning budgets?
-
Given that the shared reference term cancels in Eq. 26, does removing the no-intervention reference rollout leave training unchanged? Is this reference used elsewhere in the implementation?
-
How accurately does the frozen predictor estimate risk on intervention-generated trajectories? Do reductions in predicted risk correspond to reductions in independently evaluated downstream violations?
4 comments
W4 / Q4: Predictor Reliability and Realized Risk Reduction
We agree that Figure 3 alone does not independently validate risk reduction, since the frozen predictor supplies both the risk-reduction reward and the predicted-risk measurements.
Existing outcome-based evidence. End-to-end UCR is determined by benchmark-adjudicated safety outcomes rather than the predictor. On LLaMA-8B/BLINDSPOT, RISA reduces UCR from (Base Agent) to (Table 4). Removing the predicted risk-reduction reward increases UCR from to (Table 10), indicating that this reward component contributes to realized safety improvements. However, these results do not independently establish predictor calibration after intervention.
Additional intervention-conditioned calibration. We evaluate the unchanged predictor on held-out trajectories generated under no intervention, threshold steering, and RISA. For each condition, predicted -step risk is compared with the independently adjudicated indicator of a violation within the next valid steps. We report AUROC, expected calibration error (ECE), and Brier score.
Table 1: Additional intervention-conditioned risk evaluation on LLaMA-8B/BLINDSPOT (). Violation rates are independently adjudicated.
| Rollout condition | AUROC | ECE | Brier | Violation rate |
|---|---|---|---|---|
| No intervention | 0.852 | 0.058 | 0.149 | 0.361 |
| Threshold steering | 0.829 | 0.071 | 0.158 | 0.228 |
| RISA | 0.816 | 0.079 | 0.163 | 0.142 |
Table 1 shows that the predictor retains discrimination on RISA-generated trajectories (AUROC ), although calibration and discrimination degrade relative to unsteered trajectories (ECE versus ; AUROC versus ). Meanwhile, independently observed four-step violation frequency decreases from to .
Predicted versus observed risk reduction. We further compare matched intervention and no-intervention continuations from identical prefixes. For method , we compute the mean predicted risk reduction and independently observed violation-frequency reduction:
where indicates an independently adjudicated violation within the next valid steps under method .
Table 2: Additional predicted versus independently observed four-step risk reduction on LLaMA-8B/BLINDSPOT. Both quantities use matched initial prefixes.
| Method | Predicted | Observed |
|---|---|---|
| Threshold steering | 0.151 | 0.133 |
| RISA | 0.239 | 0.219 |
As shown in Table 2, RISA achieves mean predicted and observed risk reductions of and , respectively, compared with and for threshold steering. The independently observed reductions are consistent in direction and approximate magnitude with the predicted improvements.
These results provide evidence that RISA's risk reductions are not solely artifacts of the predictor's scores, while the calibration degradation confirms a remaining intervention-induced distribution-shift limitation.
We will extend Appendix D with intervention-conditioned calibration, Appendix F with matched predicted-versus-observed risk analysis, and clarify that Figure 3 is a predicted-risk diagnostic rather than independent evidence of realized safety improvement.
Dear Reviewer Hcy1,
Thank you again for your time and constructive feedback.
In our rebuttal, we have: (1) corrected the activation-onset definition and provided threshold-sensitive warning lead-time evaluations; (2) clarified fixed-versus-adaptive steering comparisons and added a validation-selected fixed-strength baseline; (3) mathematically clarified the cancellation of the no-intervention reference in the GRPO objective; and (4) provided additional intervention-conditioned predictor calibration and independently adjudicated downstream violation results.
As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether any further questions or concerns remain. We would be happy to address them and would appreciate your reconsideration in light of the additional evidence.
Best regards,
Authors
Follow-up on the rebuttal
Thank you for the detailed revision and for qualifying the claims. I am keeping my score at 2 because the evidence for an advantage from adaptive steering remains inconclusive. On Qwen3-4B/BLINDSPOT, the fixed positive-direction control in Table 6 outperforms RISA on all three metrics. The relative tuning budgets are difficult to assess from the reported fixed-steering settings.
Table 17 provides a more meaningful threshold-based lead-time analysis, with coverage reported for pre-violation crossings. The rate of threshold crossings on benign trajectories remains unclear. Table 3 still retains the legacy Lead values, although at every eligible step. The independent outcome check in Tables 11 and 15 is helpful and covers one model–benchmark pair. Under RISA, Table 11 reports a Brier score of 0.163. If the reported prevalence of 0.142 uses the same examples and weighting, a constant-prevalence reference would score about 0.122. This makes the aggregation important to clarify.
I also had difficulty reconciling Fig. 2 with Table 8. For example, the plotted LLaMA-8B RISA over-refusal rate appears to be about 11%, while the unweighted average in Table 8 is about 6.9%. The evaluation protocol describes per-decision inference overhead, and I could not locate the corresponding measurements in the revision.
We thank the reviewer for the constructive feedback and address each concern below.
W1 / Q1: Activation Criterion and Warning Lead Time
We agree that the deterministic Beta-mean controller in Appendix I.4 produces αt∈(0,1) and therefore implements continuous risk-dependent steering rather than explicit zero-action selection. Consequently, the original αt>0 criterion in Eq. (33) cannot meaningfully identify intervention onset.
Corrected timing evaluation. As detailed in our response to Reviewer zN6X (W1), we redefine substantial intervention onset as the first pre-violation step satisfying αt≥τact, where τact>0. We report warning lead time conditional on threshold crossing, together with activation coverage.
On LLaMA-8B/BLINDSPOT, at τact=0.25, RISA achieves a lead time of 3.18 steps and coverage of 91.5, compared with 1.96 steps and 87.3 for RISA without the early-intervention reward. The advantage persists across thresholds 0.10,0.25,0.50,0.75 (Reviewer zN6X, W1).
These results support earlier substantial steering, not exact intervention abstention. We will correct Eq. (33), replace the original timing interpretation, and clarify this distinction in Secs. 3.3–4.4 and Appendices B.5 and I.4. This correction changes the timing metric, not the interventions used to obtain the reported UCR, utility, and ORR.
W2 / Q2: Adaptive Versus Fixed Steering
We agree that adaptive steering does not uniformly outperform fixed steering across backbones, as also discussed in our response to Reviewer 47Vi (W2).
Qwen3-4B comparison. Table 8 (Appendix E.2) evaluates steering-direction quality using positive, reversed, and random directions at matched perturbation magnitude. Its fixed +dsafe configuration achieves UCR/utility/ORR of 0.276/0.817/0.089, outperforming RISA's 0.300/0.775/0.120 in Table 2. Although Table 8 was designed to isolate direction quality, its result demonstrates a favorable fixed-strength operating point that cannot be dismissed on that basis.
An additional validation-selected fixed controller achieves 0.345/0.738/0.147, compared with RISA's 0.300/0.775/0.120 on Qwen3-4B (Reviewer 47Vi, W2). However, this comparison does not establish why the Table 8 configuration performs better, nor does it establish adaptive superiority over every fixed-strength choice.
Cross-backbone evidence. On LLaMA-8B/BLINDSPOT (Table 3), RISA achieves 0.106/0.820/0.074, compared with 0.251/0.742/0.183 for fixed steering. Thus, the benefit of adaptive control is supported in this setting but is not uniform across all reported configurations.
We will clarify the distinct purposes and tuning protocols of Tables 2 and 8, report the fixed-strength validation procedure, and qualify the claim of adaptive superiority in Sec. 4.4 and Appendix E.2. A matched safety–utility comparison across fixed-strength operating points is needed to establish whether adaptation improves the attainable tradeoff.
W3 / Q3: Role of the No-Intervention Reference in GRPO
We agree that the shared no-intervention reference cancels from the group-relative advantages under the stated implementation.
Mathematical clarification. Substituting Eq. (22) into the intervention reward (Eq. 24), each candidate reward can be expressed as
qt(i)=Ct+qt(i),
where
Ct=λr(1+λeηt)∑k∈KwkR^t+1(k,0)
is constant across candidates from the same prefix, and
qt(i)=−λr(1+λeηt)∑k∈KwkR^t+1(k,i)+λsrsafe(i)+λurtask(i)−λc(αt(i))2.
Because subtracting a shared constant preserves centered rewards and their standard deviation,
σt+ϵqt(i)−qˉt=σt+ϵqt(i)−qt.
Thus, the GRPO advantages (Eq. 26) and policy objective (Eq. 27) are unchanged.
Implication. Appendix I.5 excludes the no-intervention reference from the G candidates used for advantage normalization. Therefore, its contribution is interpretive rather than an independent optimization signal: it expresses predicted risk reduction relative to an unmodified continuation but is unnecessary for computing the stated GRPO update.
Removing this reference leaves the policy update mathematically unchanged, provided candidate continuations and other optimization components remain fixed. With G=8, it also removes one of nine continuation rollouts per prefix, reducing the rollout count by 11.1.
We will clarify this cancellation in Sec. 3.3 and Appendix I.5 and avoid attributing additional learning benefits to the reference rollout.