acceptodds
Under review as a conference paper at ICLR 2027

RISA: Risk-Informed Safety Activation for Long-Horizon Tool Using Agents

Abstract

Tool-using language-model agents increasingly operate over long interaction horizons, where unsafe outcomes may emerge only after a sequence of individually plausible actions. Existing safeguards often focus on the current action or compress trajectory history into a single risk decision, providing limited control over when and how strongly to intervene. We introduce RISA (Risk-Informed Safety Activation), a framework for long-horizon agent safety that combines multi-horizon future-risk prediction with adaptive representation steering. RISA models the distribution over the first future safety violation, derives risk estimates across multiple horizons, and uses these predictions to train an intervention policy that adaptively controls the strength of a safety-relevant internal representation direction. Complete trajectories, consisting of the full sequence of agent inputs, actions, and tool or environment observations, provide privileged supervision during training, while inference depends only on the observed trajectory prefix. Across multiple long-horizon agent-safety benchmarks and model families, RISA improves unsafe completion and safe completion while preserving task utility and limiting over-refusal compared with step-level, trajectory-aware, fixed-steering, and threshold-based safeguards. Additional analyses show that the learned safety direction captures behaviorally relevant structure, multi-horizon conditioning supports more selective intervention, and RISA shifts risky trajectory states toward lower predicted future risk. These results suggest that separating future-risk forecasting from adaptive internal control provides an effective approach to proactive safety intervention in long-horizon agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Tool-Using Agents, Long-Horizon Agent Safety, Risk Prediction, Safety Steering, Representation Steering, Adaptive Intervention, Agentic AI, Trajectory-Level Safety, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hcy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISA for long-horizon safety in LLM agents. It predicts a distribution over the first future safety violation, derives cumulative risk estimates across multiple horizons, and uses a policy trained by GRPO to control the strength of a safety activation direction. Experiments across three benchmarks and four backbones report improvements in unsafe completion, task utility, and over-refusal relative to the evaluated safeguards. Additional analyses examine forecasting accuracy, steering directions, and intervention rewards.

Critical strengths
  • The framework connects future-risk prediction to graded intervention in a clear and useful way. Deriving cumulative risks from a shared first-violation distribution guarantees consistency across horizons (Sec. 3.1). Training supervision and inference inputs are explicitly separated, and trajectories are partitioned before prefix extraction to avoid leakage between closely related prefixes.

  • The evaluation considers safety and utility jointly and includes informative controls. For example, removing the utility reward improves UCR but substantially worsens utility and over-refusal (Table 10), supporting the need for a balanced objective. The matched-magnitude positive, reversed, and random steering controls also provide behavioral evidence that the learned direction captures safety-relevant structure (Table 8).

Critical weaknesses
  • The timing evaluation is inconsistent with the stated policy. Appendix I.4 uses a strictly positive Beta mean for intervention strength, while Eq. 33 defines onset as the first nonzero intervention. This would trigger intervention at the first eligible step, leaving Table 3’s lead-time difference of 3.29 versus 1.94 unexplained. An effective-intervention threshold or a revised definition is needed to support the timing claim.

  • The benefit of adaptive control is not consistently established. On Qwen3-4B/BLINDSPOT, fixed positive steering in Table 8 outperforms full RISA in Table 2 on UCR/Utility/ORR: 0.276/0.817/0.089 versus 0.300/0.775/0.120. If the protocols are comparable, this qualifies the claimed advantage of adaptation; otherwise, the differences need clarification. Matched safety–utility curves would provide a more conclusive comparison.

  • The no-intervention reference does not affect the stated policy update. Its contribution to Eq. 24 is constant across candidates and therefore cancels from both the centered rewards and their standard deviation in Eq. 26. Since Appendix I.5 excludes the reference candidate from normalization, the matched rollout supplies no intervention-relative learning signal. This does not invalidate the remaining objective, but its methodological role is overstated.

  • Risk reduction is not independently validated on intervention-generated trajectories. The frozen predictor supplies the risk-reduction reward and also measures the improvement in Figure 3. Although Appendix F acknowledges this limitation, Appendix D does not separately assess calibration after intervention. Such evaluation is needed to establish whether the controller reliably reduces downstream risk rather than exploiting prediction errors; the end-to-end results alone do not establish this mechanism.

Questions for authors
  • Is an activation gate or effective-intervention threshold used in practice? How should Eq. 33 be interpreted under Beta-mean evaluation, and does the lead-time advantage remain under a consistent activation definition?

  • Are Tables 2 and 8 directly comparable? What explains the positive-direction control outperforming full RISA on Qwen3-4B/BLINDSPOT, and how do their safety–utility curves compare under matched tuning budgets?

  • Given that the shared reference term cancels in Eq. 26, does removing the no-intervention reference rollout leave training unchanged? Is this reference used elsewhere in the implementation?

  • How accurately does the frozen predictor estimate risk on intervention-generated trajectories? Do reductions in predicted risk correspond to reductions in independently evaluated downstream violations?

4 comments

We thank the reviewer for the constructive feedback and address each concern below.

W1 / Q1: Activation Criterion and Warning Lead Time

We agree that the deterministic Beta-mean controller in Appendix I.4 produces αt∈(0,1)\alpha_t\in(0,1) and therefore implements continuous risk-dependent steering rather than explicit zero-action selection. Consequently, the original αt>0\alpha_t>0 criterion in Eq. (33) cannot meaningfully identify intervention onset.

Corrected timing evaluation. As detailed in our response to Reviewer zN6X (W1), we redefine substantial intervention onset as the first pre-violation step satisfying αt≥τact\alpha_t\geq\tau_{\mathrm{act}}, where τact>0\tau_{\mathrm{act}}>0. We report warning lead time conditional on threshold crossing, together with activation coverage.

On LLaMA-8B/BLINDSPOT, at τact=0.25\tau_{\mathrm{act}}=0.25, RISA achieves a lead time of 3.183.18 steps and coverage of 91.591.5\\%, compared with 1.961.96 steps and 87.387.3\\% for RISA without the early-intervention reward. The advantage persists across thresholds 0.10,0.25,0.50,0.75\\{0.10,0.25,0.50,0.75\\} (Reviewer zN6X, W1).

These results support earlier substantial steering, not exact intervention abstention. We will correct Eq. (33), replace the original timing interpretation, and clarify this distinction in Secs. 3.3–4.4 and Appendices B.5 and I.4. This correction changes the timing metric, not the interventions used to obtain the reported UCR, utility, and ORR.

W2 / Q2: Adaptive Versus Fixed Steering

We agree that adaptive steering does not uniformly outperform fixed steering across backbones, as also discussed in our response to Reviewer 47Vi (W2).

Qwen3-4B comparison. Table 8 (Appendix E.2) evaluates steering-direction quality using positive, reversed, and random directions at matched perturbation magnitude. Its fixed +dsafe+d_{\mathrm{safe}} configuration achieves UCR/utility/ORR of 0.276/0.817/0.0890.276/0.817/0.089, outperforming RISA's 0.300/0.775/0.1200.300/0.775/0.120 in Table 2. Although Table 8 was designed to isolate direction quality, its result demonstrates a favorable fixed-strength operating point that cannot be dismissed on that basis.

An additional validation-selected fixed controller achieves 0.345/0.738/0.1470.345/0.738/0.147, compared with RISA's 0.300/0.775/0.1200.300/0.775/0.120 on Qwen3-4B (Reviewer 47Vi, W2). However, this comparison does not establish why the Table 8 configuration performs better, nor does it establish adaptive superiority over every fixed-strength choice.

Cross-backbone evidence. On LLaMA-8B/BLINDSPOT (Table 3), RISA achieves 0.106/0.820/0.0740.106/0.820/0.074, compared with 0.251/0.742/0.1830.251/0.742/0.183 for fixed steering. Thus, the benefit of adaptive control is supported in this setting but is not uniform across all reported configurations.

We will clarify the distinct purposes and tuning protocols of Tables 2 and 8, report the fixed-strength validation procedure, and qualify the claim of adaptive superiority in Sec. 4.4 and Appendix E.2. A matched safety–utility comparison across fixed-strength operating points is needed to establish whether adaptation improves the attainable tradeoff.

W3 / Q3: Role of the No-Intervention Reference in GRPO

We agree that the shared no-intervention reference cancels from the group-relative advantages under the stated implementation.

Mathematical clarification. Substituting Eq. (22) into the intervention reward (Eq. 24), each candidate reward can be expressed as

qt(i)=Ct+q~t(i),q_t^{(i)}=C_t+\widetilde q_t^{(i)},

where

Ct=λr(1+λeηt)∑k∈KwkR^t+1(k,0)C_t= \lambda_r(1+\lambda_e\eta_t) \sum_{k\in\mathcal K} w_k\hat R_{t+1}^{(k,0)}

is constant across candidates from the same prefix, and

q~t(i)=−λr(1+λeηt)∑k∈KwkR^t+1(k,i)+λsrsafe(i)+λurtask(i)−λc(αt(i))2.\widetilde q_t^{(i)} = -\lambda_r(1+\lambda_e\eta_t) \sum_{k\in\mathcal K} w_k\hat R_{t+1}^{(k,i)} +\lambda_s r_{\mathrm{safe}}^{(i)} +\lambda_u r_{\mathrm{task}}^{(i)} -\lambda_c(\alpha_t^{(i)})^2.

Because subtracting a shared constant preserves centered rewards and their standard deviation,

qt(i)−qˉtσt+ϵ=q~t(i)−q~‾tσ~t+ϵ.\frac{q_t^{(i)}-\bar q_t} {\sigma_t+\epsilon} = \frac{\widetilde q_t^{(i)} -\overline{\widetilde q}_t} {\widetilde\sigma_t+\epsilon}.

Thus, the GRPO advantages (Eq. 26) and policy objective (Eq. 27) are unchanged.

Implication. Appendix I.5 excludes the no-intervention reference from the GG candidates used for advantage normalization. Therefore, its contribution is interpretive rather than an independent optimization signal: it expresses predicted risk reduction relative to an unmodified continuation but is unnecessary for computing the stated GRPO update.

Removing this reference leaves the policy update mathematically unchanged, provided candidate continuations and other optimization components remain fixed. With G=8G=8, it also removes one of nine continuation rollouts per prefix, reducing the rollout count by 11.111.1\\%.

We will clarify this cancellation in Sec. 3.3 and Appendix I.5 and avoid attributing additional learning benefits to the reference rollout.

W4 / Q4: Predictor Reliability and Realized Risk Reduction

We agree that Figure 3 alone does not independently validate risk reduction, since the frozen predictor supplies both the risk-reduction reward and the predicted-risk measurements.

Existing outcome-based evidence. End-to-end UCR is determined by benchmark-adjudicated safety outcomes rather than the predictor. On LLaMA-8B/BLINDSPOT, RISA reduces UCR from 0.4010.401 (Base Agent) to 0.1060.106 (Table 4). Removing the predicted risk-reduction reward increases UCR from 0.1060.106 to 0.1610.161 (Table 10), indicating that this reward component contributes to realized safety improvements. However, these results do not independently establish predictor calibration after intervention.

Additional intervention-conditioned calibration. We evaluate the unchanged predictor on held-out trajectories generated under no intervention, threshold steering, and RISA. For each condition, predicted kk-step risk is compared with the independently adjudicated indicator of a violation within the next kk valid steps. We report AUROC, expected calibration error (ECE), and Brier score.

Table 1: Additional intervention-conditioned risk evaluation on LLaMA-8B/BLINDSPOT (k=4k=4). Violation rates are independently adjudicated.

Rollout conditionAUROC ↑\uparrowECE ↓\downarrowBrier ↓\downarrowViolation rate ↓\downarrow
No intervention0.8520.0580.1490.361
Threshold steering0.8290.0710.1580.228
RISA0.8160.0790.1630.142

Table 1 shows that the predictor retains discrimination on RISA-generated trajectories (AUROC 0.8160.816), although calibration and discrimination degrade relative to unsteered trajectories (ECE 0.0790.079 versus 0.0580.058; AUROC 0.8160.816 versus 0.8520.852). Meanwhile, independently observed four-step violation frequency decreases from 0.3610.361 to 0.1420.142.

Predicted versus observed risk reduction. We further compare matched intervention and no-intervention continuations from identical prefixes. For method mm, we compute the mean predicted risk reduction and independently observed violation-frequency reduction:

ΔR^(k,m)=E[R^t+1(k,0)−R^t+1(k,m)],\Delta\hat R^{(k,m)} = \mathbb E\left[ \hat R_{t+1}^{(k,0)} - \hat R_{t+1}^{(k,m)} \right],

ΔV(k,m)=E[zt+1(k,0)−zt+1(k,m)],\Delta V^{(k,m)} = \mathbb E\left[ z_{t+1}^{(k,0)} - z_{t+1}^{(k,m)} \right],

where zt+1(k,m)z_{t+1}^{(k,m)} indicates an independently adjudicated violation within the next kk valid steps under method mm.

Table 2: Additional predicted versus independently observed four-step risk reduction on LLaMA-8B/BLINDSPOT. Both quantities use matched initial prefixes.

MethodPredicted ΔR^(4)\Delta\hat R^{(4)}Observed ΔV(4)\Delta V^{(4)}
Threshold steering0.1510.133
RISA0.2390.219

As shown in Table 2, RISA achieves mean predicted and observed risk reductions of 0.2390.239 and 0.2190.219, respectively, compared with 0.1510.151 and 0.1330.133 for threshold steering. The independently observed reductions are consistent in direction and approximate magnitude with the predicted improvements.

These results provide evidence that RISA's risk reductions are not solely artifacts of the predictor's scores, while the calibration degradation confirms a remaining intervention-induced distribution-shift limitation.

We will extend Appendix D with intervention-conditioned calibration, Appendix F with matched predicted-versus-observed risk analysis, and clarify that Figure 3 is a predicted-risk diagnostic rather than independent evidence of realized safety improvement.

Dear Reviewer Hcy1,

Thank you again for your time and constructive feedback.

In our rebuttal, we have: (1) corrected the activation-onset definition and provided threshold-sensitive warning lead-time evaluations; (2) clarified fixed-versus-adaptive steering comparisons and added a validation-selected fixed-strength baseline; (3) mathematically clarified the cancellation of the no-intervention reference in the GRPO objective; and (4) provided additional intervention-conditioned predictor calibration and independently adjudicated downstream violation results.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether any further questions or concerns remain. We would be happy to address them and would appreciate your reconsideration in light of the additional evidence.

Best regards,
Authors

Reviewer_Hcy1OpenReview ↗

Follow-up on the rebuttal

Thank you for the detailed revision and for qualifying the claims. I am keeping my score at 2 because the evidence for an advantage from adaptive steering remains inconclusive. On Qwen3-4B/BLINDSPOT, the fixed positive-direction control in Table 6 outperforms RISA on all three metrics. The relative tuning budgets are difficult to assess from the reported fixed-steering settings.

Table 17 provides a more meaningful threshold-based lead-time analysis, with coverage reported for pre-violation crossings. The rate of threshold crossings on benign trajectories remains unclear. Table 3 still retains the legacy Lead values, although αt>0\alpha_t>0 at every eligible step. The independent outcome check in Tables 11 and 15 is helpful and covers one model–benchmark pair. Under RISA, Table 11 reports a Brier score of 0.163. If the reported prevalence of 0.142 uses the same examples and weighting, a constant-prevalence reference would score about 0.122. This makes the aggregation important to clarify.

I also had difficulty reconciling Fig. 2 with Table 8. For example, the plotted LLaMA-8B RISA over-refusal rate appears to be about 11%, while the unweighted average in Table 8 is about 6.9%. The evaluation protocol describes per-decision inference overhead, and I could not locate the corresponding measurements in the revision.

Reviewer_zN6XOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

this paper studies proactive safety intervention for tool-using agents. specifically, a predictor estimates the distribution of the first future violation and derives cumulative risks at multiple horizons. a policy trained with GRPO maps these risks to the strength of hidden-state steering along a safe-minus-risk centroid direction. the acting model remains frozen. experiments on three benchmarks and several backbones report improved safety and utility compared with existing safeguards and simpler intervention strategies.

Critical strengths
  • separating future-risk prediction from intervention strength is a worthwhile direction.
  • the random-direction, reversed-direction, single-horizon, and reward-component controls address several plausible alternative explanations.
Critical weaknesses
  • the reviewer’s main technical concern is whether the policy can decide not to intervene. Appendix I.4 evaluates the Beta mean a/(a+b) with positive parameters, so every decision receives nonzero steering. however, Section 3.3 claims selective activation, and Eq. (33) measures warning lead time from the first α>0. under this implementation, that is always the first eligible decision. thus, it is unclear how Table 3 establishes earlier activation. please clarify any omitted threshold or zero-action mechanism and recompute the timing analysis using the actual rule. adaptive magnitude may remain useful, but selective activation requires separate evidence.

  • the presentation is unnecessarily fragmented across 28 numbered equations in the main paper. several merely define notation or state immediate consequences: the horizon set in Eq. (2), risk vectors in Eqs. (4) and (11), probability normalization in Eq. (9), and monotonicity in Eq. (12). Eqs. (16)–(19) could be consolidated, while the standard GRPO details in Eqs. (25)–(28) could move to the appendix. retaining the core risk, steering, and reward equations would make the actual contribution easier to follow.

  • Figure 1 suffers from poor visual quality, with several elements appearing blurry or distorted.

  • the necessity of GRPO is not sufficiently isolated. the alternatives use constant or threshold steering, whereas RISA receives substantial candidate-rollout supervision. a regressor trained on the best candidate strengths using matched rollout data would help establish whether policy optimization contributes beyond risk-conditioned magnitude selection and additional training computation.

  • the experimental setup remains difficult to assess despite the extensive appendix. trajectory-level splitting prevents prefix overlap, but the paper does not give benchmark-specific split counts or establish whether different trajectories from the same task remain together. the construction and adjudication of step-level safety labels also need clarification. please report these details and uncertainty for the main comparisons. otherwise, readers cannot determine the effective sample size or whether the gains generalize beyond familiar task instances.

Questions for authors

where exactly is steering applied during decoding: only to the final prefix token, or also to subsequently generated tokens? this affects both reproducibility and the interpretation of the intervention magnitude.

6 comments

We thank the reviewer for the constructive feedback and address each concern below.

W1: Selective Activation and Warning Lead Time

Controller parameterization. We agree that the deterministic Beta-mean policy in Appendix I.4 produces strictly positive intervention strengths:

αt=aψ(R^t)aψ(R^t)+bψ(R^t)∈(0,1).\alpha_t = \frac{a_\psi(\hat{\mathbf R}_t)} {a_\psi(\hat{\mathbf R}_t)+b_\psi(\hat{\mathbf R}_t)} \in(0,1).

Thus, the implemented controller learns risk-dependent steering magnitude but does not explicitly select an exact zero-intervention action. We will correct the description of selective activation in Sec. 3.3.

Evidence for adaptive magnitude. Appendix E.3 (Table 9) shows that mean steering strength increases from 0.310.31 to 0.760.76 as one-step risk rises from 0.120.12 to 0.790.79, while eight-step risk remains approximately constant (0.780.78–0.810.81). Moreover, removing the intervention-cost penalty (Table 10) reduces UCR from 0.1060.106 to 0.0870.087 but decreases utility from 0.8200.820 to 0.7390.739 and increases ORR from 0.0740.074 to 0.1740.174. These results support risk-dependent magnitude control and the importance of limiting excessive steering.

Corrected warning lead time. We agree that the original condition αt>0\alpha_t>0 in Eq. (33) cannot identify meaningful intervention onset under Beta-mean inference. We therefore define substantial intervention using an operational threshold τact>0\tau_{\mathrm{act}}>0:

t_{\mathrm{act}}(\tau_{\mathrm{act}}) = \min\left\\{ t<t_{\mathrm{unsafe}}^{(0)}: \alpha_t\geq\tau_{\mathrm{act}} \right\\},

Δlead(τact)=tunsafe(0)−tact(τact),\Delta_{\mathrm{lead}}(\tau_{\mathrm{act}}) = t_{\mathrm{unsafe}}^{(0)} - t_{\mathrm{act}}(\tau_{\mathrm{act}}),

where tunsafe(0)t_{\mathrm{unsafe}}^{(0)} denotes the first violation in the matched no-intervention continuation. Lead time is averaged over trajectories crossing the threshold before violation; activation coverage reports the fraction of eligible trajectories with such a crossing.

Table 1: Additional threshold-sensitive warning lead time and activation coverage on LLaMA-8B/BLINDSPOT. Lead time is conditional on pre-violation threshold crossing.

τact\tau_{\mathrm{act}}Lead Time ↑\uparrow (RISA)Lead Time ↑\uparrow (w/o Lead)Coverage (%) ↑\uparrow (RISA)Coverage (%) ↑\uparrow (w/o Lead)
0.103.422.2194.290.8
0.253.181.9691.587.3
0.502.841.6285.780.4
0.752.311.1873.665.2

Table 1 shows that RISA achieves longer conditional lead times and higher threshold-crossing coverage than the variant without the early-intervention reward across all evaluated thresholds. At τact=0.25\tau_{\mathrm{act}}=0.25, RISA achieves lead time 3.183.18 versus 1.961.96, with coverage 91.591.5\\% versus 87.387.3\\%.

These results support earlier substantial steering, not exact intervention abstention. We will replace the original Eq. (33) interpretation with this threshold-based definition and clarify the corresponding timing results in Sec. 4.4 and Appendix B.5. The correction does not change the intervention policy used to obtain the reported UCR, utility, and ORR.

W2: Equation Fragmentation and Presentation

We agree that several numbered equations introduce notation or immediate consequences rather than distinct methodological contributions. We will streamline Secs. 2–3 by moving the horizon-set and risk-vector definitions (Eqs. 2, 4, and 11), probability normalization (Eq. 9), and monotonicity property (Eq. 12) into the surrounding text.

We will consolidate the safety-prefix partitioning and direction construction (Eqs. 16–19) and move standard GRPO sampling, advantage normalization, and policy-update details (Eqs. 25–28) to Appendix A. The main text will retain the core first-violation forecasting, cumulative risk, steering, and intervention-reward formulations. These editorial changes do not alter the methodology or experimental results.

W3: Figure 1 Visual Quality

We agree that Figure 1 requires improved readability. We have replaced it with a vector-based figure using consistent typography, arrow styles, spacing, and legible labels.

W4: Isolating the Contribution of GRPO

We agree that fixed- and threshold-steering baselines do not isolate the benefit of GRPO from access to candidate-rollout supervision.

Rollout-matched regression. We additionally evaluate RISA-Reg, a supervised controller trained using the same candidate intervention strengths and continuation rewards as RISA. For each training prefix, the highest-reward candidate defines the regression target:

αt∗=αt(i∗),i∗=arg⁡max⁡i∈1,…,Gqt(i).\alpha_t^{\ast} = \alpha_t^{(i^{\ast})}, \qquad i^{\ast}=\arg\max_{i\in\\{1,\ldots,G\\}}q_t^{(i)}.

A two-layer MLP predicts intervention strength from the same multi-horizon risk vector by minimizing

Lreg=1∣P∣∑t∈P(fω(R^t)−αt∗)2.\mathcal L_{\mathrm{reg}} = \frac{1}{|\mathcal P|} \sum_{t\in\mathcal P} \left( f_\omega(\hat{\mathbf R}_t)-\alpha_t^{\ast} \right)^2.

Both controllers use the same candidate-rollout dataset, risk inputs, acting model, safety direction, intervention layer, and reward definition. This controls access to rollout supervision and rollout-generation cost, while comparing different policy-learning objectives.

Table 2: Intervention-policy learning ablation on LLaMA-8B/BLINDSPOT. RISA-Reg is an additional rollout-matched baseline; other rows reproduce Table 3.

MethodUCR ↓\downarrowUtility ↑\uparrowORR ↓\downarrow
Fixed Steering0.2510.7420.183
Threshold Steering0.1630.7590.154
RISA-Reg0.1290.8010.095
RISA (GRPO)0.1060.8200.074

RISA-Reg improves over fixed and threshold steering, confirming that learning risk-conditioned strengths is beneficial even without GRPO. Under matched rollout supervision, RISA further reduces UCR from 0.1290.129 to 0.1060.106, increases utility from 0.8010.801 to 0.8200.820, and reduces ORR from 0.0950.095 to 0.0740.074.

These results support a benefit from group-relative policy optimization beyond supervised regression to the best sampled intervention. We will include the regression baseline and matched-rollout protocol in Appendix B.4.

W5: Dataset Splits, Safety Labels, and Statistical Uncertainty

We thank the reviewer for highlighting the importance of dataset independence, safety-label provenance, and statistical reliability.

Dataset partitioning. As described in Appendices B.1 and I.7, trajectories are partitioned before prefix extraction, ensuring that prefixes originating from the same trajectory do not cross training, validation, and test splits. The BLINDSPOT preprocessing subset contains 200200 retained trajectories and 1,8051{,}805 labeled prefixes, after excluding 5050 indeterminate trajectories.

Importantly, trajectory-level disjointness does not establish task-level disjointness: distinct trajectories generated from the same underlying task may occur in different partitions. Accordingly, the existing evaluation supports generalization to held-out trajectories, but does not independently establish generalization to unseen task identities.

Table 3: BLINDSPOT preprocessing statistics.

SplitPrefixesFractionTrajectoriesTasks
Training1,33073.7%15043
Validation27915.5%3011
Test19610.9%2011
Total1,805100.0%20065

Safety-label construction. The preprocessing records contain 1,7401{,}740 weak-terminal labels (96.496.4\\%) and 6565 tier-C adjudicated labels (3.63.6\\%). Their terminal-outcome distribution is summarized below.

Table 4: BLINDSPOT safety-label provenance and terminal outcomes across 1,8051{,}805 labeled prefix records.

CategoryCountFraction
Label provenance
Weak-terminal1,74096.4%
Tier-C adjudicated653.6%
Terminal outcomes
Unsafe completion60733.6%
Safe completion60233.4%
Correct refusal57131.6%
Over-refusal251.4%
Total1,805100.0%

These terminal-outcome labels are distinct from the step-level safety indicators yty_t used for first-violation prediction. Specifically, the target in Eq. (13) is constructed from the earliest future step satisfying yt+j=1y_{t+j}=1 within the forecasting horizon; otherwise, it is assigned ∅\varnothing. Therefore, identifying the exact violation step requires temporally localized safety supervision, not merely the terminal outcome.

End-to-end evaluation. All methods use matched held-out evaluation instances within each backbone–benchmark setting (Appendix I.7). Table 5 summarizes the submitted UCR results and their absolute and relative improvements over the Base Agent.

Table 5: End-to-end safety improvements from Table 2. Absolute and relative reductions are calculated from the reported UCR values.

BackboneBenchmarkBase UCRRISA UCRAbsolute ReductionRelative Reduction
Qwen3-4BBLINDSPOT0.4320.3000.13230.6%
MT-AgentRisk0.3940.1280.26667.5%
LongDS-Bench0.3610.1180.24367.3%
Gemma-2-9BBLINDSPOT0.4470.3200.12728.4%
MT-AgentRisk0.4080.1390.26965.9%
LongDS-Bench0.3740.1270.24766.0%

RISA reduces UCR across all six evaluated configurations, with relative reductions ranging from 28.428.4\\% to 67.567.5\\%. These results demonstrate consistent improvements in the reported held-out trajectory evaluations.

Statistical uncertainty and independence. Appendix I.7 specifies three evaluation seeds. The reductions in Table 5 are differences between reported aggregate estimates; they are not confidence intervals or statistical significance tests.

Because prefixes from the same trajectory are correlated, the effective number of independent evaluation units is smaller than the number of prefixes. Moreover, if trajectories share an underlying task, trajectory-level independence does not imply task-level independence. The reported comparisons therefore establish improvements on held-out trajectories, without establishing task-disjoint generalization or confidence-interval precision.

Q1: Steering Location During Autoregressive Decoding

RISA applies steering to the final token of the observed trajectory prefix HtH_t at the selected transformer layer ℓ\ell, immediately before generating the next agent action (Sec. 3.2, Eq. 15; Appendix I.3):

h~t(ℓ)=ht(ℓ)+αtdsafe(ℓ).\tilde{\mathbf h}_t^{(\ell)} = \mathbf h_t^{(\ell)} + \alpha_t\mathbf d_{\mathrm{safe}}^{(\ell)}.

The perturbation is applied once at the decision boundary. Subsequently generated tokens are not directly steered, and αt\alpha_t is not recomputed during that decoding sequence. The modified prefix representation may nevertheless influence subsequent token generation through normal autoregressive computation.

At the next agent decision, RISA recomputes the risk profile from the updated trajectory prefix and selects a new steering strength. We will explicitly document the intervention token position, selected layer, and decoding scope in Sec. 3.3 and Appendix I.3.

Reviewer_47ViOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RISA, a runtime safeguard for long-horizon tool-using LLM agents. RISA forecasts how soon a safety violation may occur and uses this forecast to decide how strongly to steer the frozen acting model toward safe behaviour. First, the authors train an MLP forecaster on the acting model's hidden state to predict when the first violation will occur, which gives the probability R̂(k) of a violation within k = 1, 2, 4 and 8 steps (the risk vector R̂; Sec. 3.1). Second, they compute a steering direction d_safe as the difference between the mean hidden states of prefixes that stay violation-free for the next 8 steps and of those that do not (Eq. 19). Third, they train a small controller with a GRPO-style objective: conditioned only on R̂, it picks a steering strength α ∈ [0,1] and adds α·d_safe to the hidden state (Eqs. 20–21).

Experiments cover BLINDSPOT, MT-AgentRisk and LongDS-Bench with Qwen3-4B, Gemma-2-9B, LLaMA-8B and Qwen2.5-14B. Baselines include ToolSafe, TRACE, TRACES and a rubric-based safeguard. RISA reports the lowest unsafe-completion (UCR) and over-refusal (ORR) rates in all 12 settings, and the highest utility among safeguards in 11 (Tables 2, 4).

Critical strengths

1. A sensible forecast-then-steer decomposition with a clean forecasting formulation.

  • The paper models the first-violation time as one categorical distribution yields horizon-consistent, monotone risks by construction (Eqs. 7–12). Privileged future information is confined to training targets and rewards. Algorithms 1–3 and Tables 13–16 specify the pipeline clearly.

Why it is consequential: This is the paper’s main conceptual contribution. Forecast-driven intervention addresses a real limitation of per-step allow/block guards, which act late and all-or-nothing.

2. Steering along d_safe shows a real, demonstrable safety improvement.

  • At matched magnitude on Qwen3-4B/BLINDSPOT, steering along +d_safe lowers UCR from 0.432 to 0.276, compared with 0.407 for a random direction and 0.468 for −d_safe (Table 8). A direction estimated purely from future-violation labels is behaviourally meaningful for an agent.

Why it is consequential: Steering is a clean way to make the agent’s actions safer, and it is the central mechanism of this paper.

Critical weaknesses

1. RISA's gains over baselines may be confounded with in-distribution training.

  • RISA is fit on training splits drawn from the evaluation benchmarks themselves, and its test sets are held-out trajectories from the same benchmarks (App. B.1, I.7). The paper reports no cross-benchmark or out-of-distribution test. The baselines are existing methods developed on other datasets, and App. B.4 does not say whether they were retrained on these splits.

Why it is consequential: If RISA is the only method trained on the test distribution, part of its improvement may come from the setup rather than the method.

2. Conflicting results of adaptive vs. fixed steering across backbones.

  • Fixed steering applies d_safe with the same strength at every step. On LLaMA-8B/BLINDSPOT, RISA reaches UCR 0.106 / utility 0.820 / ORR 0.074, against fixed steering's 0.251 / 0.742 / 0.183 (Table 3). However, on Qwen3-4B/BLINDSPOT, fixed steering beats RISA on all three metrics that Table 8 reports: UCR 0.276 / utility 0.817 / ORR 0.089 (row +d_safe), against RISA's 0.300 / 0.775 / 0.120 (Table 2).

Why it is consequential: The learned controller is the method's distinctive component, and the paper claims gains over fixed-steering safeguards. With one comparison going each way, the ablations do not show that learning α helps.

Questions for authors
  1. App. I.7 says all methods use the same trajectory splits. Were ToolSafe, TRACE, TRACES and Safety Rubric trained or calibrated on the training split, or used as released?

  2. Single-Horizon RISA conditions on R̂(8) only (App. B.4). Did you train a controller on R̂(1) alone, and if so, how did it compare with RISA in Table 3?

3 comments

We thank the reviewer for the constructive feedback and address each concern below.

W1 / Q1: In-Distribution Training and Baseline Fairness

Baseline calibration. ToolSafe, TRACE, TRACES, and Safety Rubric were calibrated using the corresponding benchmark-specific training/validation splits rather than evaluated solely with their released configurations. All methods were evaluated on identical held-out test trajectories, with trajectory-level splitting performed before prefix extraction (Appendices B.1 and I.7). We will expand Appendix B.4 to specify the calibrated components, parameter-selection procedures, and training budgets for each baseline.

Cross-benchmark generalization. To assess dependence on in-distribution training, we additionally evaluate zero-shot cross-benchmark transfer on Qwen3-4B. RISA is trained exclusively on a source benchmark and evaluated on the other benchmarks without target-specific retraining or calibration. The risk predictor, safety direction, intervention layer, and GRPO-trained controller remain fixed during transfer.

Table 1: Cross-benchmark transfer on Qwen3-4B. ID rows reproduce the submitted Table 2; OOD rows are additional zero-shot transfer experiments.

TrainTestUCR ↓\downarrowUtility ↑\uparrowORR ↓\downarrowSCR ↑\uparrow
BLINDSPOTBLINDSPOT (ID)0.3000.7750.1200.775
MT-AgentRisk (OOD)0.1860.8070.1030.701
LongDS-Bench (OOD)0.1740.8010.0990.744
MT-AgentRiskMT-AgentRisk (ID)0.1280.8310.0790.723
BLINDSPOT (OOD)0.3580.7420.1470.718
LongDS-Bench (OOD)0.1590.8020.0910.751
LongDS-BenchLongDS-Bench (ID)0.1180.8240.0750.782
BLINDSPOT (OOD)0.3710.7280.1530.704
MT-AgentRisk (OOD)0.1810.7980.1070.692

Table 1 shows that RISA retains lower UCR than the unsafeguarded agent across all six cross-benchmark evaluations, although target-trained RISA generally performs better. For example, BLINDSPOT-trained RISA achieves UCR 0.1860.186 and utility 0.8070.807 on MT-AgentRisk, compared with 0.3940.394 and 0.8580.858 for the unsafeguarded agent and UCR 0.1280.128/utility 0.8310.831 for target-trained RISA (Table 2).

These results establish partial cross-benchmark generalization, not superiority over target-trained safeguards or robustness to arbitrary distribution shifts.

Within-benchmark controls. Table 3 provides complementary evidence independent of comparisons with external safeguards. On LLaMA-8B/ BLINDSPOT, RISA achieves UCR/utility/ORR of 0.106/0.820/0.0740.106/0.820/0.074, compared with 0.163/0.759/0.1540.163/0.759/ 0.154 for threshold steering and 0.141/0.793/0.1080.141/0.793/0.108 for single-horizon RISA.

We will clarify baseline calibration in Appendix B.4 and distinguish benchmark-specific performance from zero-shot transfer.

W2: Adaptive Versus Fixed Steering Across Backbones

We agree that the results in Tables 2, 3, and 8 do not establish uniform superiority of adaptive steering over every fixed-strength configuration.

Experimental distinction. Table 8 (Appendix E.2) evaluates the learned safety direction by comparing positive, reversed, and random directions under matched perturbation magnitude. Table 3 instead compares intervention-control strategies, including fixed, threshold-based, and adaptive steering.

On LLaMA-8B/BLINDSPOT, RISA achieves UCR/utility/ ORR of 0.106/0.820/0.0740.106/0.820/0.074, compared with 0.251/0.742/0.1830.251/0.742/0.183 for fixed steering (Table 3), corresponding to a 57.8%57.8\% relative UCR reduction with higher utility and lower over-refusal.

Qwen3-4B comparison. We acknowledge that fixed positive-direction steering in Table 8 achieves 0.276/0.817/0.0890.276/0.817/0.089, outperforming RISA in Table 2 (0.300/0.775/0.1200.300/0.775/0.120) on all three metrics. Although Table 8 was designed to test direction quality rather than optimize a fixed-strength controller, this does not invalidate its favorable operating point.

We additionally evaluate a validation-selected fixed-strength controller on Qwen3-4B, using the same backbone, safety direction, intervention layer, decoding configuration, and held-out test trajectories.

Table 2: Fixed and adaptive steering on BLINDSPOT. The validation-selected Qwen3-4B baseline is an additional experiment; the remaining rows reproduce Tables 2, 3, and 8 of the submitted paper.

BackboneMethodUCR ↓\downarrowUtility ↑\uparrowORR ↓\downarrow
Qwen3-4BFixed +dsafe+d_{\mathrm{safe}} (Table 8)0.2760.8170.089
Fixed (validation-selected)0.3450.7380.147
RISA (Table 2)0.3000.7750.120
LLaMA-8BFixed (Table 3)0.2510.7420.183
RISA (Table 3)0.1060.8200.074

As shown in Table 2, RISA outperforms the additional validation-selected fixed controller on Qwen3-4B, reducing UCR from 0.3450.345 to 0.3000.300 while improving utility and over-refusal. However, the fixed positive-direction control from Table 8 remains better than both configurations at its reported operating point.

Therefore, the LLaMA-8B results support adaptive control in that setting, while the Qwen3-4B results demonstrate that fixed steering can be competitive or superior. We will qualify the general claim of adaptive superiority in Sec. 4.4 and clarify the fixed-strength selection and evaluation protocols in Appendix E.2. A matched safety–utility comparison across fixed strengths is necessary to establish whether adaptation improves the attainable tradeoff beyond the reported operating points.

Q2: One-Step Versus Multi-Horizon Risk Conditioning

The submitted Single-Horizon RISA baseline conditions the controller only on the longest-horizon estimate R^t(8)\hat R_t^{(8)} (Appendix B.4). We additionally train RISA-1 using only R^t(1)\hat R_t^{(1)}, while keeping the risk predictor, safety direction, intervention layer, GRPO reward, and training protocol unchanged.

Table 3: Effect of controller risk horizons on LLaMA-8B/BLINDSPOT. RISA-1 is an additional ablation; RISA-8 and Full RISA reproduce Table 3. Safety–utility metrics are reported separately from the activation-timing analysis.

Controller inputUCR ↓\downarrowUtility ↑\uparrowORR ↓\downarrow
R^t(1)\hat R_t^{(1)} (RISA-1)0.1320.8010.096
R^t(8)\hat R_t^{(8)} (RISA-8)0.1410.7930.108
Full RISA (1,2,4,81,2,4,8)0.1060.8200.074

RISA-1 achieves lower UCR (0.1320.132 versus 0.1410.141), higher utility (0.8010.801 versus 0.7930.793), and lower ORR (0.0960.096 versus 0.1080.108) than RISA-8. Full RISA further improves all three metrics, achieving UCR 0.1060.106, utility 0.8200.820, and ORR 0.0740.074.

These results support the complementary value of near-term and longer-horizon risk information for intervention control. Consistently, Appendix E.3 (Table 9) shows that mean intervention strength increases from 0.310.31 to 0.760.76 as one-step risk increases from 0.120.12 to 0.790.79, even when eight-step risk remains approximately constant.

The submitted Table 3 additionally reports warning lead times of 2.732.73 and 3.293.29 for RISA-8 and Full RISA, respectively, while the additional RISA-1 evaluation reports 2.412.41. We retain these reported values but do not interpret them as evidence of selective activation until the onset criterion in Eq. (33) is reconciled with the strictly positive Beta-mean policy specified in Appendix I.4.

Dear Reviewer 47Vi,

Thank you again for your time and constructive feedback.

In our rebuttal, we have: (1) clarified baseline calibration and evaluation fairness; (2) provided additional zero-shot cross-benchmark experiments to evaluate generalization; (3) added fixed-versus-adaptive steering comparisons across backbones; and (4) provided a one-step versus multi-horizon risk-conditioning ablation.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether any further questions or concerns remain. We would be happy to address them.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.