- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper combines internal monitoring with corrective self-advice to mitigate harmful agent actions. A small temporal classifier reads late residual states and Jacobian-derived probe features during reasoning. When it triggers, the same language model generates a brief reminder from the interrupted reasoning and a behavioral specification; several variants then restart or continue the actor. Experiments on Qwen3-14B and Qwen3-32B examine detection, held-out scenario factors, and correction strategies. The best tested variants reduce i.i.d. population harm from 45.2% to 11.1% and from 47.0% to 5.9%, respectively.
Critical strengths
- The paper evaluates an intervention rather than stopping at prediction of harmful behavior. It reports population harm, accounts for untriggered runs, and separates rescued from newly induced harmful outcomes. The reductions are substantial, and Appendix F includes a genuine held-out-restriction mitigation test in addition to the i.i.d. results.
- The correction controls are informative. Comparing restart alone, advice with a restart, and advice while retaining the reasoning helps identify what contributes to recovery. The fact that the best variant differs between 14B and 32B is useful evidence about the limits of a single recipe. Reporting the large false-positive rates under some distribution shifts also makes the detector’s limitations visible.
Critical weaknesses
- The current detector comparison does not isolate the value of the Jacobian construction. The residual-only baseline reads one late layer, whereas the hybrid adds supervised cosine features from roughly 21 layers. A comparison with matched multilayer mean-difference directions or learned probes, using the same classifier and feature budget, is needed to attribute the improvement to Jacobian weighting. All correction experiments use the hybrid, so its added value under an otherwise identical restart-and-advice policy is also unknown. This is compounded by missing construction details. Appendix B leaves the Jacobian estimation corpus size, token alignment, and derivative approximation undocumented. Those details matter for reproducing the central contribution.
- The safety and utility tradeoff of the actual online procedure has not been established. Deployment triggers on a threshold crossing anywhere during generation, whereas Table 13 reports only end-of-trace false positives on ordinary workloads. Appendix H explicitly says the corresponding online comparison was not remeasured. End-of-trace results cannot establish the rate of unnecessary interruptions under continuous monitoring.
- The paper also does not report ordinary task success after intervention or the total measured cost of monitoring, advice, and regeneration. Together with the elevated false-positive rates under shift, this leaves the practical benefit of the full safeguard uncertain.
Questions for authors
- Does the hybrid retain its advantage over matched multilayer probes constructed without the Jacobian? Please report both detection quality and final harm when the detectors use the same correction policy.
- What data, token alignment, and approximation are used to estimate the Jacobians? Are the label-dependent contrast directions rebuilt using only the training portion of each held-out split? Please also provide the advisor prompt and behavioral-specification version needed to reproduce the interventions.
- For one fixed detector and correction configuration, what are the full-trajectory benign alarm rate, task-success rate, and end-to-end runtime or token overhead? These should use the same threshold-crossing rule as the safety experiments.
2 comments
Response to Reviewer s9Vc (2/2)
W2 / W3 / Q3. One fixed configuration: benign alarm rate, task success and overhead under the threshold-crossing rule. The revision adds a deployment table with one fixed configuration per model. The monitor is residual-only, with a single-window trigger, stride 32 and calibrated on online scores of benign training runs (), and the intervention is wipe-and-advise. An alarm counts if any window crosses , exactly as in the safety experiments.
| Qwen3-32B, fixed configuration | Value |
|---|---|
| Harm on held-out AM runs | 46.1% 15.5% (always-advise 16.2%) |
| Induced harm (benign runs turned harmful) | 4.1% (always-advise 13.7%) |
| Benign interruption rate, held-out, any-time rule | utility 0.008; SALAD 0.015; benign AM runs 0.137; DarkBench 0.267 |
| Utility task success | 93.3% 93.3% (always-advise 85.8%) |
| Token multiple (all tokens counted) | (always-advise ) |
| Cost, benign runs / all runs (, ) | / (always-advise ) |
| Cost, all runs (, ) | (always-advise ) |
| Monitor latency | at most 0.024 ms per token on GPU |
For Qwen3-14B, harm falls from 46.3% to 18.0% with 4.8% induced harm, utility 0.008, tokens and cost on benign runs and across all runs. The same table covers Llama, Gemma and gpt-oss: cost on originally benign runs is , and , and across all runs it is , and . On detected runs we count tokens up to the first action, the same point at which advice is written for the reported harm rates; most of the added cost falls on originally harmful runs. Task success is measured on Qwen3-32B utility runs only, and runtime is reported as the full token count plus the monitor's own latency. The end-of-trace numbers of the submission's Table 13 are replaced throughout, and Appendix H's "not remeasured" caveat is resolved, because every false-alarm number now uses the deployment rule.
The trade-off under shift (W2) remains, and the revision reports it as a limitation. With a scenario held out entirely, benign alarms rise for every feature set, and with murder held out, online reaches 0.32 on Qwen3-32B. Benign agentic misalignment runs are the hardest in-domain case, because they deliberate about the harmful option before declining it. For deployments that prefer fewer interruptions, calibrating on benign agentic misalignment training runs gives 32B 0.082 at 17.3% harm and 1.4% induced harm.
On the choice of a single configuration, the reviewer noted that the best variant differed between 14B and 32B. Under the rebuilt evaluation (temperature 0.7, rebuilt split) it does not: wiping outperforms keeping the reasoning on both Qwen sizes, by 25.0 points (95% CI 14.6 to 35.4) and 36.7 points (24.5 to 48.0), and it is never significantly worse on the three new families, so one configuration serves all five models.
We thank the reviewer again: the deployment table was added in response to Q3 and is now the paper's main summary table.
Response to Reviewer s9Vc (1/2)
We thank the reviewer for a careful review focused on deployment. The three questions map directly onto three new analyses: a matched-probe comparison reported in final harm, a complete Appendix B, and a single-configuration deployment table measured under the threshold-crossing rule. We answer them in order.
W1 / Q1. Does the hybrid keep its advantage over matched multilayer probes, including in final harm? It does not keep a consistent advantage. We built matched multilayer probes as described, with per-layer cosines to residual mean-difference directions and to random directions, holding the layers, the feature budget and the MiniMamba fixed, and every detector is used with the same correction procedure (wipe-and-advise at the online τ). Final agentic misalignment harm on held-out runs with the multidomain monitors (mean over three seeds; Qwen3-14B seed 0):
Detection quality follows the same pattern. Online held-out AUROCtest ranges across the six feature sets are 0.927–0.948 (32B), 0.924–0.947 (14B), 0.984–0.989 (Llama), 0.977–0.995 (Gemma) and 0.968–0.982 (gpt-oss), and the residual stream alone lies within that range everywhere. The monitoring signal is therefore already present in the residual stream, and the method is now presented as a residual-stream monitor. We did not train learned per-layer probes; the mean-difference directions are the closed-form version of that control, and the random-direction set shows how much any added features help.
The Jacobian rows are variable rather than uniformly null. The band J-hybrid ties for the lowest harm on Qwen3-32B (14.4%), but on Qwen3-14B it is above the residual alone (21.3% against 18.0%), and with the AM-only 14B monitor it is among the highest (26.1%, close to the random-direction control). Holding out a scenario, J cosines alone are usually the weakest detector (four of six folds). Where the J-hybrid helps, the md-hybrid usually helps as much, so we do not claim a Jacobian-specific effect. The difference that remains is that the residual stream is fixed by the model, whereas the lens is fit, and its corpus and size were generic in this paper (WikiText; an 80-prompt partial fit for 32B). We therefore present the lens as an analysis tool that can be tuned rather than as the central source of the gains, and leave a lens tuned to agentic trajectories as an open direction.
The control also exposed a train–test leak, which answers part of Q2. The contrast directions had been fit on all agentic misalignment runs, including runs later held out; they are now refit per training fold, and every result is rebuilt. This is why the submission's held-out gains and its headline harm numbers (5.9% / 11.1%) do not reappear. The final harm reduction for Qwen models is 46.1% → 15.5% (32B) and 46.3% → 18.0% (14B), and we extend it to three more families. The held-out-restriction mitigation test the reviewer credited (the submission's Appendix F) used the old pipeline and is withdrawn; its rebuilt counterpart reports end harm with each scenario held out.
Q2. Jacobian data, token alignment, approximation, advisor prompt and specification. All of these are in the revised Appendix B. The Jacobians are pre-fitted Neuronpedia lenses on WikiText-103 train (128-token passages; 615 prompts for 14B, and an 80-prompt partial fit for 32B, which we disclose; 125, 861 and 457 prompts for Llama, Gemma and gpt-oss, with the Gemma lens fit on the base checkpoint). The first 16 positions and the final position are excluded, with a one-hot cotangent at every valid target, and the corrected Eq. (1) is Jℓ=EpromptEt[∑t′≥t∂hfinal,t′/∂hℓ,t]. Derivatives use exact reverse-mode autodiff, not finite differences. Contrast directions and class means are rebuilt on each training fold only, while J itself is label-free. The full advisor system and user prompts are included verbatim: the advisor sees the specification excerpt (the compact Model Spec Midtraining Safety Principles/General Principles excerpt) and the reasoning truncated at the detection point, and returns one first-person bullet, which is inserted into the restarted reasoning followed by the fixed phrase "With that in mind,". The deployment protocol under all variants is documented.