acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Agentic Misalignment via Jacobian Workspace Self-Supervision

Abstract

We show that using an AI agent's Jacobian workspace can mitigate agentic misalignment even after the agent begins rationalizing harmful behavior, substantially reducing harmful actions without retraining the underlying model. The Jacobian lens measures how changes in internal states propagate toward output representations, providing a basis for detecting when reasoning is heading toward a harmful action. We combine measurements derived from this lens with ordinary internal states in a small trained detector that monitors the agent as it reasons. When the detector's score crosses a calibrated threshold, we interrupt generation, discard the current reasoning, and ask the same model to write a brief corrective reminder. The agent then restarts from the task and reminder. An optional intervention steers the advisor's internal states while it writes the reminder. On a held-out i.i.d. evaluation set, our best interventions reduce agentic misalignment from to on Qwen3-32B and from to on Qwen3-14B. These results establish internal monitoring and guided self-correction as a promising complement to training-time safety interventions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Misalignment, Jacobian Workspace, Artificial Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s9VcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines internal monitoring with corrective self-advice to mitigate harmful agent actions. A small temporal classifier reads late residual states and Jacobian-derived probe features during reasoning. When it triggers, the same language model generates a brief reminder from the interrupted reasoning and a behavioral specification; several variants then restart or continue the actor. Experiments on Qwen3-14B and Qwen3-32B examine detection, held-out scenario factors, and correction strategies. The best tested variants reduce i.i.d. population harm from 45.2% to 11.1% and from 47.0% to 5.9%, respectively.

Critical strengths
  1. The paper evaluates an intervention rather than stopping at prediction of harmful behavior. It reports population harm, accounts for untriggered runs, and separates rescued from newly induced harmful outcomes. The reductions are substantial, and Appendix F includes a genuine held-out-restriction mitigation test in addition to the i.i.d. results.
  2. The correction controls are informative. Comparing restart alone, advice with a restart, and advice while retaining the reasoning helps identify what contributes to recovery. The fact that the best variant differs between 14B and 32B is useful evidence about the limits of a single recipe. Reporting the large false-positive rates under some distribution shifts also makes the detector’s limitations visible.
Critical weaknesses
  • The current detector comparison does not isolate the value of the Jacobian construction. The residual-only baseline reads one late layer, whereas the hybrid adds supervised cosine features from roughly 21 layers. A comparison with matched multilayer mean-difference directions or learned probes, using the same classifier and feature budget, is needed to attribute the improvement to Jacobian weighting. All correction experiments use the hybrid, so its added value under an otherwise identical restart-and-advice policy is also unknown. This is compounded by missing construction details. Appendix B leaves the Jacobian estimation corpus size, token alignment, and derivative approximation undocumented. Those details matter for reproducing the central contribution.
  • The safety and utility tradeoff of the actual online procedure has not been established. Deployment triggers on a threshold crossing anywhere during generation, whereas Table 13 reports only end-of-trace false positives on ordinary workloads. Appendix H explicitly says the corresponding online comparison was not remeasured. End-of-trace results cannot establish the rate of unnecessary interruptions under continuous monitoring.
  • The paper also does not report ordinary task success after intervention or the total measured cost of monitoring, advice, and regeneration. Together with the elevated false-positive rates under shift, this leaves the practical benefit of the full safeguard uncertain.
Questions for authors
  1. Does the hybrid retain its advantage over matched multilayer probes constructed without the Jacobian? Please report both detection quality and final harm when the detectors use the same correction policy.
  2. What data, token alignment, and approximation are used to estimate the Jacobians? Are the label-dependent contrast directions rebuilt using only the training portion of each held-out split? Please also provide the advisor prompt and behavioral-specification version needed to reproduce the interventions.
  3. For one fixed detector and correction configuration, what are the full-trajectory benign alarm rate, task-success rate, and end-to-end runtime or token overhead? These should use the same threshold-crossing rule as the safety experiments.
2 comments
AuthorsOpenReview ↗

Response to Reviewer s9Vc (1/2)

We thank the reviewer for a careful review focused on deployment. The three questions map directly onto three new analyses: a matched-probe comparison reported in final harm, a complete Appendix B, and a single-configuration deployment table measured under the threshold-crossing rule. We answer them in order.

W1 / Q1. Does the hybrid keep its advantage over matched multilayer probes, including in final harm? It does not keep a consistent advantage. We built matched multilayer probes as described, with per-layer cosines to residual mean-difference directions and to random directions, holding the layers, the feature budget and the MiniMamba fixed, and every detector is used with the same correction procedure (wipe-and-advise at the online τ\tau). Final agentic misalignment harm on held-out runs with the multidomain monitors (mean over three seeds; Qwen3-14B seed 0):

Feature setQwen3-32BQwen3-14B (seed 0)Llama-3.3-70BGemma-4-31Bgpt-oss-20b
No intervention46.1%46.3%42.6%78.9%4.8%
Residual only15.0%18.0%5.3%54.4%3.2%
1L J-hybrid15.6%18.8%5.8%54.6%3.3%
1L md-hybrid15.5%23.9%5.4%54.6%3.1%
Band J-hybrid14.4%21.3%5.6%54.4%2.7%
Band md-hybrid14.4%21.0%5.2%54.6%2.6%
Band random-hybrid14.6%18.8%5.7%54.8%2.7%

Detection quality follows the same pattern. Online held-out AUROCtest\text{AUROC}_\text{test} ranges across the six feature sets are 0.927–0.948 (32B), 0.924–0.947 (14B), 0.984–0.989 (Llama), 0.977–0.995 (Gemma) and 0.968–0.982 (gpt-oss), and the residual stream alone lies within that range everywhere. The monitoring signal is therefore already present in the residual stream, and the method is now presented as a residual-stream monitor. We did not train learned per-layer probes; the mean-difference directions are the closed-form version of that control, and the random-direction set shows how much any added features help.

The Jacobian rows are variable rather than uniformly null. The band J-hybrid ties for the lowest harm on Qwen3-32B (14.4%), but on Qwen3-14B it is above the residual alone (21.3% against 18.0%), and with the AM-only 14B monitor it is among the highest (26.1%, close to the random-direction control). Holding out a scenario, J cosines alone are usually the weakest detector (four of six folds). Where the J-hybrid helps, the md-hybrid usually helps as much, so we do not claim a Jacobian-specific effect. The difference that remains is that the residual stream is fixed by the model, whereas the lens is fit, and its corpus and size were generic in this paper (WikiText; an 80-prompt partial fit for 32B). We therefore present the lens as an analysis tool that can be tuned rather than as the central source of the gains, and leave a lens tuned to agentic trajectories as an open direction.

The control also exposed a train–test leak, which answers part of Q2. The contrast directions had been fit on all agentic misalignment runs, including runs later held out; they are now refit per training fold, and every result is rebuilt. This is why the submission's held-out gains and its headline harm numbers (5.9% / 11.1%) do not reappear. The final harm reduction for Qwen models is 46.1% →\to 15.5% (32B) and 46.3% →\to 18.0% (14B), and we extend it to three more families. The held-out-restriction mitigation test the reviewer credited (the submission's Appendix F) used the old pipeline and is withdrawn; its rebuilt counterpart reports end harm with each scenario held out.

Q2. Jacobian data, token alignment, approximation, advisor prompt and specification. All of these are in the revised Appendix B. The Jacobians are pre-fitted Neuronpedia lenses on WikiText-103 train (128-token passages; 615 prompts for 14B, and an 80-prompt partial fit for 32B, which we disclose; 125, 861 and 457 prompts for Llama, Gemma and gpt-oss, with the Gemma lens fit on the base checkpoint). The first 16 positions and the final position are excluded, with a one-hot cotangent at every valid target, and the corrected Eq. (1) is Jℓ=EpromptEt[∑t′≥t∂hfinal,t′/∂hℓ,t]J_\ell = \mathbb{E}_{\text{prompt}} \mathbb{E}_t\big[\sum_{t'\geq t} \partial h_{\text{final},t'} / \partial h_{\ell,t}\big]. Derivatives use exact reverse-mode autodiff, not finite differences. Contrast directions and class means are rebuilt on each training fold only, while J itself is label-free. The full advisor system and user prompts are included verbatim: the advisor sees the specification excerpt (the compact Model Spec Midtraining Safety Principles/General Principles excerpt) and the reasoning truncated at the detection point, and returns one first-person bullet, which is inserted into the restarted reasoning followed by the fixed phrase "With that in mind,". The deployment protocol under all variants is documented.

AuthorsOpenReview ↗

Response to Reviewer s9Vc (2/2)

W2 / W3 / Q3. One fixed configuration: benign alarm rate, task success and overhead under the threshold-crossing rule. The revision adds a deployment table with one fixed configuration per model. The monitor is residual-only, with a single-window trigger, stride 32 and τ\tau calibrated on online scores of benign training runs (FPRtrain≈0.05\text{FPR}_\text{train} \approx 0.05), and the intervention is wipe-and-advise. An alarm counts if any window crosses τ\tau, exactly as in the safety experiments.

Qwen3-32B, fixed configurationValue
Harm on held-out AM runs46.1% →\to 15.5% (always-advise 16.2%)
Induced harm (benign runs turned harmful)4.1% (always-advise 13.7%)
Benign interruption rate, held-out, any-time ruleutility 0.008; SALAD 0.015; benign AM runs 0.137; DarkBench 0.267
Utility task success93.3% →\to 93.3% (always-advise 85.8%)
Token multiple (all tokens counted)1.60×1.60\times (always-advise 2.16×2.16\times)
Cost, benign runs / all runs (c=0.1c=0.1, r=8r=8)1.05×1.05\times / 1.18×1.18\times (always-advise 1.35×1.35\times)
Cost, all runs (c=0.25c=0.25, r=4r=4)1.24×1.24\times (always-advise 1.46×1.46\times)
Monitor latencyat most 0.024 ms per token on GPU

For Qwen3-14B, harm falls from 46.3% to 18.0% with 4.8% induced harm, utility FPRtest\text{FPR}_\text{test} 0.008, 1.59×1.59\times tokens and cost 1.05×1.05\times on benign runs and 1.18×1.18\times across all runs. The same table covers Llama, Gemma and gpt-oss: cost on originally benign runs is 1.08×1.08\times, 1.23×1.23\times and 1.04×1.04\times, and across all runs it is 1.26×1.26\times, 1.70×1.70\times and 1.07×1.07\times. On detected runs we count tokens up to the first action, the same point at which advice is written for the reported harm rates; most of the added cost falls on originally harmful runs. Task success is measured on Qwen3-32B utility runs only, and runtime is reported as the full token count plus the monitor's own latency. The end-of-trace numbers of the submission's Table 13 are replaced throughout, and Appendix H's "not remeasured" caveat is resolved, because every false-alarm number now uses the deployment rule.

The trade-off under shift (W2) remains, and the revision reports it as a limitation. With a scenario held out entirely, benign alarms rise for every feature set, and with murder held out, online FPRtest\text{FPR}_\text{test} reaches 0.32 on Qwen3-32B. Benign agentic misalignment runs are the hardest in-domain case, because they deliberate about the harmful option before declining it. For deployments that prefer fewer interruptions, calibrating τ\tau on benign agentic misalignment training runs gives 32B FPRtest\text{FPR}_\text{test} 0.082 at 17.3% harm and 1.4% induced harm.

On the choice of a single configuration, the reviewer noted that the best variant differed between 14B and 32B. Under the rebuilt evaluation (temperature 0.7, rebuilt split) it does not: wiping outperforms keeping the reasoning on both Qwen sizes, by 25.0 points (95% CI 14.6 to 35.4) and 36.7 points (24.5 to 48.0), and it is never significantly worse on the three new families, so one configuration serves all five models.

We thank the reviewer again: the deployment table was added in response to Q3 and is now the paper's main summary table.

Reviewer_ctXZOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the issue with agentic misalignment - where tool-augmented LLM agents with extended CoT may falter with severe policy violations (such as data exfiltration, sabotage etc). The paper proposes continuous probe mechanisms and also steering interventions for two core failure modes in existing inference-time defenses:

  • zeroth order trap : where an agent deliberating (to reject) is in the same space as an agent reject a request
  • transcript intertia : once already in multi-step unsafe rationalization in CoT, making late safer activations steer gets tougher

The paper proposes a three stage inference time monitoring (probing/steering):

  • first order jacobian probes using jlens which maps intermediate residuals int output coordinates
  • online hybrid gate : probe monitor using 2-layer minimamba with 8-token sliding window. this helps trigger an alarm if ti exceeds the threshold
  • wipe and advise restart : when probe gate is triggered, generation halts before tool execution and the poisoned CoT prefix is wiped from acting agent's context and then rectified either through an "advisor" behavioral spec aware prompting or through residual steering.
Critical strengths
  • The proposed monitoring probe (through jlens) is neat that can intervene during reasoning even before an harmful tool call has been triggered.
  • The two proposed correction, through advisor steering and behavioral spec prompting is novel that explores wiping and retrain reasoning with possible fixes.
  • The paper also effectively uses an elegant computational efficient jlens operation per layer per token using adjoint identity for storing the pullback vector thus reducing the dot products per token - this makes a practical continuous inference time solution.
  • The empirical validation for zeroth-order trap w/ LOGO, LOTO and LOTaskO is quite effective to motivate the work and to show how raw residual probing fails under distribution shift.
  • Clean ablation for self-correction controls and the cross-scale insights provides strong evidence for support of the paper
Critical weaknesses
  • Incomplete reproducibility details left as placeholder. It is missing details on estimating Jl=E[∂hfinal,t′/∂hl,t]J_l = \mathbb{E}[\partial h_{\text{final}, t'} / \partial h_{l,t}] is central theme of the paper, also the understanding of how JlJ_l was estimated (prompt distribution, sample size, token offset, etc.) is a significant gap.
  • compare jacobian mapping and multi-layer contrast pooling : in sec 3.2, the 0th order baseline feeds minimamba the noramlized residual from a single layer (41), while 1st order concatenates with 21 layers of supervised contrast cosines. There is no empirical ablation between then without which it is slightly unclear how much OOD gain comes from the first-order jacobian projection vs simply providing mean-diff cosine features instead of a single raw layer.
  • missing empirical latency and token-overhead computations? while Appendix J1 notest taht precomputing makes per-token feature extraction O(d)O(d), there is no discussion or analysis on end-to-end latency or total tokens overhead. Reasoning discards signficant tokens and prefill/generation pass and restarting from scratch may be expensive and it is important metric to quantify.
Questions for authors
  • in the wiped steered intervention (sec 4.1 and Appendix B), the advisor's residual are shifted by h←h−α unit(vdefect)h \leftarrow h - \alpha\,\text{unit}(v_{\text{defect}}) rather than along the pullback direction unit unit(Jl⊤vdefect)\text{unit}(J_l^\top v_{\text{defect}}) - this mixes the native residual space. is there a particular theoretical or empirical reason for this?
  • could you please provide all missing details in Appendix B/open-details? Furthermore, for every LOGO/LOTO/LOTaskO split were JlJ_l, the class means, and the contrast vectors computed strictly on the training fold without heldout trajectories?
  • over the eval set, what was the total token overhead in terms of wasted prefix tokens + advisor tokens + regenerated tokens compared to unmonitored generation? Also, what fraction of runs in each intervention type still successfully completes the authorized benign task (to account for over-refusal).
  • (related to 2nd bullet in weaknesses) : to isolate the contribution of jacobian linear map from multi-layer contrast pooling, what is the detection AUROC and FPR if you compute 21 per layer contrast direction directly from raw residual and feed those 21 cosines into the exact same miniMamba architecture?
  • Cognition-of-thought Zhang et al. 2025 uses a cognitive perceiver sharing the generator's backbone with a different prompt to monitor generated text and uses attention maps to pick a rollback point before doing guided regeneration which is similar to wipe-and-advise loop bit (on text rather than hidden states). Is this not a more appropriate baseline for the paper?
  • Throughout the main paper, the trigger happens the moment any token crosses the threshold. however, when defending against false alarm on benign utility benchmarks (appendix H), the metric is switch to "end-FPR" . what's the actual online run-level FPR on these utility benchmarks when used the standard trigger rule?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ctXZ (1/2)

We thank the reviewer for a detailed review. Q4 asked for 21 per-layer contrast directions computed directly from raw residuals and fed to the same MiniMamba. That experiment changed the framing of the paper; we report it below, together with the controls it motivated.

W1 / Q2. How J is estimated, and train–test separation. Appendix B is now complete. We use the pre-fitted Jacobian lenses released by Neuronpedia, fitted with the open-source jacobian-lens following Gurnee et al.; for each block ℓ\ell, Jℓ∈Rd×dJ_\ell \in \mathbb{R}^{d\times d} maps the block-ℓ\ell residual to the final residual. The corpus is WikiText-103 train, re-chunked into passages of about 2,000 characters and truncated to 128 tokens, and per-prompt Jacobians are averaged with early stopping (615 prompts for Qwen3-14B). For Qwen3-32B we used the file at that path when the experiments ran, an 80-prompt partial fit that stopped before the early-stopping criterion, and the revision states this explicitly. The Llama, Gemma and gpt-oss lenses come from the same release (125, 861 and 457 prompts); the Gemma lens was fit on the base checkpoint and is applied to the instruction-tuned model, which affects only Gemma's J feature sets and J-steered advice. The first 16 positions and the last position of each prompt are excluded, and a one-hot cotangent at every valid target gives Jℓ=EpromptEt[∑t′≥t∂hfinal,t′/∂hℓ,t]J_\ell = \mathbb{E}_{\text{prompt}} \mathbb{E}_t \big[\sum_{t'\geq t} \partial h_{\text{final},t'} / \partial h_{\ell,t}\big], which corrects Eq. (1). Derivatives use exact reverse-mode autodiff through every downstream block, not finite differences.

On train–test separation (Q2), J is fit on unlabeled WikiText and so contains none of our trajectories. The label-dependent quantities, the class means and contrast directions, had been estimated once on all AM runs in the submission, including runs later held out by LOGO, LOTO and LOTaskO, while detectors and thresholds were already training-only. In the revision every direction is re-estimated on each training fold. The submission's goal and threat holdouts used the leaked directions and are withdrawn rather than rerun; the revision holds out whole scenarios instead, for all five models, with every direction refit on the training scenarios (Section 2). This correction accounts for the change in the held-out numbers.

W2 / Q4. Jacobian vs multi-layer contrast pooling. We ran the proposed experiment as stated, adding a random-direction control and J-only and md-only detectors, on five models over three seeds (Section 2). On the i.i.d. split (Qwen3-32B multidomain monitor, held-out AUROC), the residual alone reaches 0.953, the md-hybrid 0.955–0.958, the J-hybrid 0.955–0.957 and the random-direction hybrid 0.953 (AM-only monitor: 0.932, 0.931–0.940, 0.936–0.941 and 0.943). With murder held out, the residual reaches 0.786 offline, md alone 0.806 and J alone 0.644, and online under the deployment rule md alone reaches an AUROC of 0.868 on 32B and 0.881 on 14B, against 0.652 and 0.579 for J alone. The tables below give AUROC and TPR / FPR on held-out test runs for Qwen3-32B (τ\tau calibrated to a training FPR of 0.05; the band is the same 21 layers, L21–41).

I.i.d. split, multidomain monitor (mean of three seeds):

DetectorAUROCTPR / FPR
Residual only0.9530.91 / 0.153
Residual + band md cosines0.9580.93 / 0.164
Residual + band J cosines0.9570.88 / 0.122
Residual + band random cosines0.9530.87 / 0.125

Murder scenario held out, AM-only monitor (seed 0):

DetectorOffline AUROCOffline TPR / FPROnline AUROCOnline TPR / FPR
Residual only0.7860.92 / 0.6180.7770.72 / 0.317
md cosines alone0.8060.82 / 0.4520.8680.89 / 0.302
J cosines alone0.6440.51 / 0.3220.6520.23 / 0.070

The out-of-distribution gain in the submission therefore came from pooling contrast features across layers, and in part from the fold leak above, rather than from the Jacobian projection, and under scenario shift mean-difference cosines alone are stronger than J cosines alone. The Jacobian results are variable rather than uniformly null, however: under one correction policy the band J-hybrid gives the lowest final harm on Qwen3-32B and among the highest on Qwen3-14B. We read this as a property of the lens we used rather than of the idea. The residual stream and the mean-difference direction are fixed once the layer and labels are chosen, whereas the lens depends on its fitting corpus and size, which here were generic WikiText and, for 32B, a partial fit. The revision presents the method as a residual-stream monitor and the lens as an analysis tool that can be tuned, and names a lens fit on agentic trajectories as future work.

AuthorsOpenReview ↗

Response to Reviewer ctXZ (2/2)

W3 / Q3. Latency, tokens, total overhead and task success. We count every token: on detected runs, the reasoning up to the first action (the same point at which advice is written for the reported harm rates), the advisor prompt and output, and the restarted generation, with input split into new and cached (Section 4). On Qwen3-32B the monitored system uses 1.60×1.60\times the tokens of an unmonitored run, against 2.16×2.16\times for advising every run. At GPT-5-style prices it costs 1.05×1.05\times as much as an unmonitored run on originally benign runs and 1.18×1.18\times across all runs, against 1.35×1.35\times for always-advise; across models, cost on originally benign runs is 1.041.04–1.08×1.08\times (1.23×1.23\times on Gemma) and the cost across all runs ranges from 1.07×1.07\times to 1.70×1.70\times, because most of the added cost falls on the harmful runs the monitor detects. The monitor itself adds at most 0.024 ms per token on GPU (0.249 ms on CPU); we count every token but did not separately time end-to-end wall-clock latency. On Qwen3-32B utility tasks (120 GSM8K, MATH-500, MMLU and ARC questions), task success per intervention type is 93.3% with no intervention, 93.3% with sampling on every run, 85.8% with always-advise (it refuses some benign questions), and 93.3% with each of the six monitors, which trigger on 0–1 of the 120. We measured utility success on Qwen3-32B only. Induced harm, the share of benign AM runs that the intervention turns harmful, is 4.1% for the monitor and 13.7% for always-advise.

Q1. Steering along unit(v_defect) instead of the pullback. There was no theoretical justification, and the reviewer's reading is correct: v_defect lives in the lens's final-layer coordinates, so adding it to the layer-ℓ\ell residual implicitly treats JℓJ_\ell as close to the identity, which the lenses do not support. The question prompted a direct test. With directions fit on the training fold only, we steer the advisor along both the Jacobian direction pulled back to the residual stream (J⊤vJ^\top v) and the residual mean-difference direction, at the same α\alpha scale. Neither reliably improves on unsteered advice. The only near-significant gains are J-steering with a wipe on Llama (−8.5-8.5 points, p=0.064p = 0.064) and on gpt-oss (−20-20 points on 25 detected runs, p=0.062p = 0.062), and J-steering hurts on Gemma (the one model whose lens was fit on a different checkpoint, as noted above), the same variability we see in detection. We therefore report steering as an analysis rather than a contribution. We also test steering the acting model in place, by subtraction or by activation capping, and find that it does not undo harm once the transcript is kept (88% →\to 96–100% on Llama), whereas a fresh start gives 56%. This supports the reviewer's transcript-inertia reading and the choice to wipe (Section 5).

Q5. Cognition-of-Thought (CooT, Zhang et al. 2025). We thank the reviewer for the pointer; the work is close in spirit and is now discussed in related work. CooT monitors generated text with a prompted copy of the generator. The closest controlled comparison we can run is the same idea at the same point in generation, in which a judge LLM reads the reasoning so far and scores harm by log-probability.

MonitorQwen3-32B online AUROCQwen3-14B online AUROC
Residual monitor (ours)0.9380.948
Gemma-4-31B text judge on reasoning0.8680.836
Qwen3-32B text judge on reasoning0.8840.812
Text judge that also sees the final action (ceiling)0.9790.980

At a fixed operating point the stronger text judge is competitive on 32B, catching 0.78 of harmful runs at FPR 0.05 against our 0.76 at 0.082, but it costs a 31B-parameter pass per check, whereas our monitor costs a small recurrent update per token. A text judge reaches 0.979 only once it also sees the action, which is too late to prevent it. CooT's rollback-and-regenerate loop is analogous to our wipe-and-advise step and could be paired with either monitor, and we include this in our revision. However, due to our host compute (Appendix B) running CooT on all models was time prohibitive.

Q6. Online run-level FPR on utility tasks. Under the deployment trigger, in which any window crossing τ\tau counts and τ\tau is calibrated on online scores of benign training runs, the held-out utility FPRtest\text{FPR}_\text{test} is 0.008 (1 of 120) for both Qwen3-32B and Qwen3-14B. The end-of-trace metric is removed from the paper, every false-alarm number now uses the deployment rule, and Section 3 gives the per-domain breakdown and the harder case of benign agentic runs.

We thank the reviewer again; Q4 in particular sharpened the paper's claims and clarified where further gains may lie.

Thank you for the detailed and substantial revision.

Since some of the main claims have change/reversed/demoted, I am updating my rating to reflect that - especially new reading on jacobian-specific probe, pullback scoring, goal & threat holdouts, advisor steering, reported 14B-vs-32B difference etc - which have been withdrawn or revised. The central method can be seen as a residual-stream monitor that gates a restart. Also, the value of using internal activations rather than a text gate remains untested in final harm. The new revision have brought up some questions, that i would appreciate if authors could address:

  • what does monitoring add over a text gate? Table 4 compares detection but not final harm after the same restart. On qwen3-32b, the gemma judge reading reasoning has numerically higher TPR at lower FPR at the reported operating points. These are not matched operating points. The paper calls an action-aware judge too late yet it says tool calls execute after the full reply, and the headline policy waits for the first unexecuted tool call before advising. Could you explain why that judge cannot gate the restart and compare text and activation gates using the same alarm budget and correction procedure?
  • Do corrected agents finish their authorized tasks? The utility benchmark shows that monitor rarely fires on math & QA tasks. It does not measure task completion after intervention on agentic-misalignment runs. About 14% of benign Qwen runs are interrupted, 6 out of 20 on qwen3-32b and 7 out of 21 on qwen3-14b end harmful after intervention. These samples are small, but false alarms cannot be treated solely as a token cost.
Reviewer_RJrCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an inference-time method for detecting and mitigating agentic misalignment without changing the base model’s weights. It combines Jacobian-derived features from roughly 21 layers with residual activations and trains a small MiniMamba detector to monitor the model’s reasoning. When the detector triggers, the same language model generates corrective advice. The intervention variants differ in whether they remove the original reasoning and steer the advisor’s activations. On the held-out i.i.d. evaluation, the best variants reduce harmful behavior from 47.0% to 5.9% on Qwen3-32B and from 45.2% to 11.1% on Qwen3-14B.

Critical strengths
  1. The paper addresses an important problem: detecting and correcting harmful behavior before an agent executes an action. It evaluates the final behavioral outcomes, and the reported reductions in harmful actions are substantial within the tested settings. The method also leaves the base model’s weights unchanged.

  2. The detection experiments include held-out goals, threats, and task types in addition to an i.i.d. split. Adding the Jacobian-derived features improves detection across several settings. For example, on the Qwen3-32B murder-task holdout, AUROC increases from 0.730 to 0.996, while the false-positive rate decreases from 68.3% to 26.6%.

  3. The correction experiments compare resampling, safety advice, removal of prior reasoning, and advisor activation steering. These comparisons help identify which steps contribute to the results. They also reveal a useful difference between the models: removing prior reasoning works better for 32B, whereas retaining it works better for 14B.

Critical weaknesses
  1. The current comparison does not establish how much of the detection gain comes from the Jacobian. The residual-only baseline uses one late layer, while the proposed method adds features from roughly 21 layers. The improvement could therefore come from access to additional layers rather than the Jacobian transformation itself. A necessary control would use the same layers, feature dimensions, and detector size without the Jacobian—for example, cosine features constructed directly from the harmful-versus-benign residual mean difference at each layer.

  2. The model evaluation is limited to two sizes from the same family. The experiments cover only Qwen3-14B and Qwen3-32B, with no validation on other open-weight model families. Since even these two models benefit from different correction strategies, further experiments are needed to establish whether the detection gains and intervention choices generalize.

  3. False-positive rates remain high in some held-out settings. The rate reaches 48.4% for the 14B murder-task holdout and 68.4% for the DarkBench holdout, meaning that many benign runs would also be interrupted. The utility experiments report false positives only at the end of generation, which does not measure the probability of a false alarm anywhere during generation. Normal task completion, additional latency, and total computational cost are also insufficiently evaluated.

  4. Some essential implementation details are missing, and several reported results need clarification. The appendix states that the Jacobian estimation corpus size, token alignment, derivative approximation, full advisor prompt, and explicit KV-cache reset verification remain to be documented. Figure 1 contains results that differ from the main text, and the first-alarm analysis has inconsistent sample-size and threshold annotations. These details should be clarified so that the experiments can be interpreted and reproduced reliably.

Questions for authors

please ref to the weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer RJrC (1/2)

Response to Reviewer RJrC

We thank the reviewer for a precise review. We ran the matched-layer control proposed in W1 as described; it is the central new experiment of the revision and changed how the paper presents the method; we answer each weakness below. Quantities with the subscript test\text{test} are measured on held-out test runs, and τ\tau is always calibrated on training runs.

W1. How much of the gain comes from the Jacobian? We built the proposed control, holding the layers, the feature dimension and the MiniMamba fixed, and replacing the Jacobian features with per-layer cosines to the harmful-vs-benign residual mean difference. We added a random-direction version and J-only and md-only detectors, over three seeds and on five models. On the i.i.d. AM test, every feature set lies within a narrow band (32B AUROCtest\text{AUROC}_\text{test} 0.931–0.943, residual only 0.932), and holding out murder, J cosines alone are the weakest feature set (offline AUROCtest\text{AUROC}_\text{test} 0.644, against 0.786 for the residual and 0.806 for md alone on 32B). The Jacobian is therefore not what produces the detection gain, and the signal is already present in the residual stream.

The Jacobian results are nonetheless variable rather than uniformly null: under the same correction policy, the band J-hybrid gives the lowest final harm on Qwen3-32B and among the highest on Qwen3-14B. Because the residual stream is fixed by the model, whereas the lens is fit (here on a generic corpus with a modest number of prompts), we present the lens as an analysis tool that can be tuned and leave a task-tuned lens as an open direction rather than a claim.

The submission's large holdout gain (0.730 →\to 0.996) did not survive the rebuild. It came from two pipeline issues that the control exposed: the contrast directions had been fit on runs that included the held-out fold, and the residual baseline read a different layer from the hybrid. Both are fixed and every reported number is recomputed. The goal and threat holdouts are replaced by scenario holdouts on all five models. The method is now presented as a residual-stream monitor, with the matched comparison as the evidence for that choice, and the title changes to Mitigating Agentic Misalignment via Self-Supervision.

W2. Only one model family. We add three families, Llama-3.3-70B, Gemma-4-31B and gpt-oss-20b, under the same pipeline, grader and train/test split. Each family gets the same AM-only and multidomain monitors as Qwen. We see similar detection performance: the multidomain residual monitor reaches online AUROCtest\text{AUROC}_\text{test} of 0.989 on Llama, 0.997 on Gemma and 0.974 on gpt-oss (AM-only, offline: 0.988, 0.999 and 0.968). Mitigation carries over as well: the monitor with wipe-and-advise takes harm from 42.6% to 5.2% on Llama and from 78.9% to 54.4% on Gemma (3.3% when the monitor is added to model-spec prompting, which alone reaches 20.4%), and from 4.8% to 3.3% on gpt-oss, which has only 13 harmful test runs. The reviewer noted that 32B and 14B preferred different correction strategies. Under the final evaluation (temperature 0.7, final split) that difference disappears: wiping is better on both Qwen sizes, with significant paired differences of 25.0 and 36.7 points, and it is never significantly worse on the new families, so one configuration serves all five models.

AuthorsOpenReview ↗

Response to Reviewer RJrC (2/2)

W3. False alarms, any-time FPR, task completion, latency and cost. Every false-alarm number in the revision now uses the online deployment rule: an alarm anywhere during generation counts, and τ\tau is calibrated on those online scores. On held-out utility tasks, the 32B and 14B monitors interrupt 1 of 120 runs (FPRtest\text{FPR}_\text{test} 0.008). Benign agentic runs are the most difficult case (32B 0.137, 14B 0.144).

The 14B murder holdout (the reviewer's 48.4%) remains high offline after the rebuild (residual FPRtest\text{FPR}_\text{test} 0.484). Under the deployment rule with an online τ\tau it falls to 0.160, but only 39% of held-out harmful murder runs are then caught. Murder is hard to transfer to for every feature set and every family, so we report it as the main limitation rather than claim a fix. With DarkBench held out entirely (the reviewer's 68.4%), the agentic misalignment part of detection is unaffected (32B AUROCtest\text{AUROC}_\text{test} 0.901, TPRtest\text{TPR}_\text{test}/FPRtest\text{FPR}_\text{test} 0.768/0.123), and across all test examples the monitor's FPRtest\text{FPR}_\text{test} is 0.086 (32B) and 0.039 (14B), though it misses much of the unseen DarkBench harm. DarkBench has only 15–22 benign test examples, so its own FPR moves in steps of about 0.05–0.07.

On task completion, Qwen3-32B utility success is 93.3% with the monitor, identical to no monitoring, against 85.8% when advice is given on every run (we measured utility success on Qwen3-32B only). Scoring costs 0.024 ms per token on GPU, measured for the largest feature set (band J-hybrid), and the residual-only monitor requires strictly less computation. Counting all tokens (the reasoning up to the first action on detected runs, the advisor, and the restarted generation), the monitored system costs 1.05×1.05\times as much as an unmonitored run on originally benign runs and 1.18×1.18\times across all runs, on both Qwen3-32B and Qwen3-14B, against 1.35×1.35\times / 1.34×1.34\times for always-advise.

W4. Missing details, Figure 1 and the first-alarm analysis. Appendix B is complete in the revision. It gives the Jacobian corpus, size and token alignment; states that the Jacobians use exact autodiff rather than a finite-difference approximation; reproduces the full advisor prompt and specification version; documents the KV-cache reset check; and adds a statement on train–test separation: every direction, class mean and threshold is fit on the training fold only. Figure 1 is redrawn for the residual-stream monitor, and all results use these final numbers.

The first-alarm analysis is also completed for all models. The median first alarm comes 61% of the way through the original output on Qwen3-32B, 54% on Qwen3-14B, 44% on Llama-3.3-70B, 27% on Gemma-4-31B and 39% on gpt-oss-20b, and 78%, 74%, 62%, 50% and 25% of these alarms, respectively, occur before reasoning ends.

We thank the reviewer for their constructive review, and believe that the matched control and the four-family results address the reviewer's concerns about soundness and generality.

Thank you for the additional experiments and clarifications. The matched-layer comparison shows no consistent benefit from the Jacobian features. The revised method uses a classifier trained on residual activations to detect risk. When it raises an alarm, the same language model writes a short corrective reminder, and the agent restarts from the original task and the reminder after its earlier reasoning is cleared. I see limited methodological novelty in this approach. The results are also mixed. For example, Llama reaches the same final harm rate when advice is given on every run, and on the Qwen3-14B murder holdout the detector catches only 39% of harmful runs while interrupting 16% of benign runs. I think the method still needs refinement and further analysis, so I am maintaining my weak reject rating.

We thank the reviewer for the follow-up and answer the three points below.

Novelty. The monitor is deliberately simple; the contribution is the finding that a plan written into the context resists correction in activation space but not a restart. On 25 Llama-3.3-70B runs flagged by the monitor, 22 (88%) end harmful when the transcript is kept, and steering the acting model in place raises this to 96–100%, whereas restarting without the transcript, even with no reminder, lowers it to 14 of 25 (56%). The monitor makes this restart practical by applying it only where needed.

Llama and always-advise. Equal final harm is the intended outcome rather than a weakness. Always-advise intervenes on every run; on Llama the monitor reaches the same 5.2% while lowering induced harm on benign runs from 1.9% to 0.6% and the cost over all runs from 1.45× to 1.26× (1.08× on benign runs). On Qwen3-32B, advising every run also lowers utility task success from 93.3% to 85.8%, while the monitored system leaves it at 93.3%. On Qwen3-14B the monitor is also better than always-advise in final harm (18.0% against 22.1%).

Murder holdout. This number comes from a stress test, not from the deployed monitor. In that experiment the entire murder scenario is removed from training, so the monitor has never seen a run in which the agent considers letting someone die, and it is then asked to detect such runs. The deployed monitor is trained on all three scenarios; on held-out test runs of Qwen3-14B it detects 112 of 126 harmful runs (89%) overall and 26 of 39 (67%) in the murder scenario, while interrupting 16% of benign murder runs. An interruption of a benign run rarely changes its outcome (7 of 146 benign test runs, 4.8%, become harmful after the restart), so false alarms cost tokens rather than safety. Murder is also the scenario in which inference-time interventions struggle most: on the same 90 murder test runs, harm is 43% without intervention, 54% with model-spec prompting, 52% when every run is resampled and 43% when every run is advised, whereas the monitor lowers it to 32%. We agree that murder remains the hardest scenario (leaking falls from 66% to 14% and exfiltration from 30% to 8%), and transfer to an entirely unseen scenario is the main limitation we report.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers (1/3)

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. The reviewers collectively asked for a control that isolates the contribution of the Jacobian from the contribution of reading additional layers. We ran this control, holding the layers, the feature dimension, and the detector fixed, and compared Jacobian features with the residual stream alone and with per-layer directions given by the mean difference between harmful and benign residuals, reporting both detection and final harm under one self-correction policy. This was repeated on five models across four families, yielding a central claim that is stronger and simpler.

The matched controls show that the monitoring signal is already present in the residual stream, so the Jacobian lens is not uniquely needed for detection. We now present it as an analysis tool that can be tuned, rather than as the central source of the gains. The Jacobian lenses used in this work were fit on a generic corpus of prompts, and their results under the matched control are variable, best in some settings and worst in others. We interpret that variability along with the tunability of the lens as factors that may enable further detector gains (section 2). We leave this for future work.

1. Mitigating Agentic Misalignment Across Five Models

The monitor in this table reads the residual stream alone, scoring every 32 tokens online. Detection is triggered the first time any window crosses τ\tau, which is calibrated on the online scores of benign training runs at FPRtrain≈0.05\text{FPR}_\text{train} \approx 0.05. Upon detection, the interrupted reasoning is wiped and the model's own one-line advice is inserted before it continues. All runs are held-out agentic misalignment (AM) test runs (an i.i.d. split stratified by scenario combination and label), generated at temperature 0.7 with up to 8,192 tokens and graded by the same prompt using Claude Sonnet 4.6.

Model (AM test runs, harmful)No interventionModel-spec promptingAlways-adviseMonitor + wipe-and-adviseInduced harm: always-advise →\to monitorBenign AM runs interrupted (FPRtest\text{FPR}_\text{test})Cost: benign runs / all runs (always-advise, all runs)
Qwen3-32B (271, 125)46.1%26.9%16.2%15.5%13.7% →\to 4.1%0.1371.05×1.05\times / 1.18×1.18\times (1.35×1.35\times)
Qwen3-14B (272, 126)46.3%37.1%22.1%18.0%18.5% →\to 4.8%0.1441.05×1.05\times / 1.18×1.18\times (1.34×1.34\times)
Llama-3.3-70B (270, 115)42.6%25.6%5.2%5.2%1.9% →\to 0.6%0.1291.08×1.08\times / 1.26×1.26\times (1.45×1.45\times)
Gemma-4-31B (270, 213)78.9%20.4%54.4%54.4%1.8% →\to 0.0%0.3161.23×1.23\times / 1.70×1.70\times (1.75×1.75\times)
gpt-oss-20b (270, 13)4.8%3.3%3.3%3.3%1.6% →\to 1.2%0.0541.04×1.04\times / 1.07×1.07\times (1.49×1.49\times)

The monitored system matches the harm reduction of advising on every run while inducing far less harm in benign runs and costing less. Induced harm is the share of originally benign runs that end harmful. Cost weights tokens as new+0.1⋅cached+8⋅output\text{new} + 0.1\cdot\text{cached} + 8\cdot\text{output} (GPT-5-style prices) and is reported relative to an unmonitored run. On detected runs we count the tokens up to the first action, the same point at which the advice behind the reported harm rates is written. Most of the added cost comes from originally harmful runs, so originally benign runs cost 1.041.04–1.08×1.08\times as much as an unmonitored run on four models and 1.23×1.23\times on Gemma, whose monitor interrupts more benign AM runs. Every detector is now trained on all four domains (AM, DarkBench, SALAD, utility), with the same non-AM prompts and splits for every model. The result is stable across detector seeds (32B 14.4–15.5%, Llama 5.2–5.6%, Gemma 54.4%, gpt-oss 2.6–3.7%).

These figures replace the submission's 47.0% →\to 5.9% (32B) and 45.2% →\to 11.1% (14B), which came from a detector whose contrast directions had seen held-out runs (section 2), combined with steered advice under greedy decoding. The rebuilt numbers are smaller but robust, holding across seeds and families without any fitted direction.

Two limitations of this evaluation concern Gemma and gpt-oss. On Gemma, the monitor catches 212 of the 213 harmful test runs, but the advice rescues only about 30% of them (end harm 54.4%), suggesting that the inertia towards harm is not always responsive to self correction. When prompting Gemma with safety-focused model specification, base harm drops to 20.4%, and adding a monitor trained on spec-prompted runs reduces harm to 3.3% (9.6%, 21.3% and 10.0% on 32B, 14B and Llama). We have added a new appendix table for these experiments. On gpt-oss, there are only 13 harmful test runs, so its differences are within noise. The revised paper adds figures with every advisor option for every model, with bootstrap 95% intervals.

AuthorsOpenReview ↗

General response to all reviewers (2/3)

2. The Signal is Already in the Residual Stream (RJrC W1, ctXZ W2/Q4, s9Vc W1/Q1)

We ran the control the reviewers proposed, with the same layers, feature dimensions and MiniMamba throughout: the residual stream alone; the residual with per-layer J cosines over a band of layers (21 on Qwen3-32B); the residual with per-layer mean-difference (md) cosines fit on raw residuals; the residual with random-direction cosines; and J or md cosines alone. Each is trained in a single-layer and a band variant over three seeds. The table gives held-out AUROCtest\text{AUROC}_\text{test} of the band variants (offline; mean ±\pm 95% interval over three seeds).

ModelResidual only+ J cosines+ md cosines+ random cosines
Qwen3-32B (AM-only monitor)0.932±0.0320.932 \pm 0.0320.941±0.0190.941 \pm 0.0190.931±0.0480.931 \pm 0.0480.943±0.0160.943 \pm 0.016
Qwen3-32B (multidomain monitor)0.953±0.0090.953 \pm 0.0090.957±0.0200.957 \pm 0.0200.958±0.0010.958 \pm 0.0010.953±0.0290.953 \pm 0.029
Qwen3-14B (AM-only monitor)0.930±0.0200.930 \pm 0.0200.886±0.0360.886 \pm 0.0360.908±0.0190.908 \pm 0.0190.893±0.0270.893 \pm 0.027
Qwen3-14B (multidomain monitor)0.934±0.0260.934 \pm 0.0260.930±0.0340.930 \pm 0.0340.944±0.0330.944 \pm 0.0330.938±0.0270.938 \pm 0.027
Llama-3.3-70B0.988±0.0090.988 \pm 0.0090.991±0.0080.991 \pm 0.0080.986±0.0240.986 \pm 0.0240.990±0.0120.990 \pm 0.012
Gemma-4-31B0.999±0.0010.999 \pm 0.0010.998±0.0010.998 \pm 0.0010.998±0.0040.998 \pm 0.0040.997±0.0060.997 \pm 0.006
gpt-oss-20b0.968±0.0510.968 \pm 0.0510.982±0.0070.982 \pm 0.0070.976±0.0390.976 \pm 0.0390.978±0.0110.978 \pm 0.011

On every model, the interval of the residual alone overlaps those of the richer feature sets, and the same holds online under the deployment rule. The matched comparison therefore does not support a consistent Jacobian advantage.

The Jacobian results are, however, variable rather than uniformly null. With the same correction policy (three-seed means), the band J-hybrid gives the lowest final harm on Qwen3-32B (14.4%, against 15.0% for the residual) and among the highest on Qwen3-14B (26.1%, against 20.6%). Holding out a scenario, J cosines alone are the weakest feature set (32B with murder held out: residual 0.786, md alone 0.806, J alone 0.644), while J-steered advice gives the only near-significant steering gains we observe (with a wipe: Llama −8.5-8.5 points (p=0.064p = 0.064); gpt-oss −20-20 points on 25 detected runs (p=0.062p = 0.062)). In contrast, J-steered advice hurts Gemma, whose lens was fit on the base model rather than the instruction-tuned checkpoint. Where the lens helps, the mean-difference direction usually helps as much, suggesting the curation of a training corpus of harmful and benign examples is the source of the lift. Unlike the residual stream, the lens can still be tuned: all lenses used in our experiments were fit on generic WikiText passages (for Qwen3-32B, an 80-prompt partial fit). A lens fit on agentic trajectories or tuned to the detection target is a potential area for future work.

This control surfaced two pipeline issues in the submission, which we corrected before running all experiments. First, the label-dependent contrast directions (J and md) had been fit once on all 1,350 AM runs, including runs held out in the leave-one-out folds; detectors and τ\tau were already training-only. Every direction is now refit on each training fold. For leave-one-out experiments for all five models, the revision holds out scenarios (exfiltration, leaking, murder). Second, the residual-only baseline read a different layer (L41) from the hybrid; every comparison now reads the same layer. Together these produced the submission's large leave-one-out gaps (for example, murder holdout 0.730 →\to 0.996). The mitigation result in section 1 uses neither the Jacobian nor any fitted direction.

AuthorsOpenReview ↗

General response to all reviewers (3/3)

3. False Alarms Under the Deployment Trigger (RJrC W3, ctXZ Q6, s9Vc W2/Q3)

We now measure false alarms as deployed: any window crossing τ\tau during generation counts as a detection, and τ\tau is calibrated on those online scores rather than on end-of-trace scores. The held-out benign rates for the Qwen3-32B monitor are 0.008 on utility tasks (1 of 120), 0.015 on SALAD, 0.137 on benign AM runs and 0.267 on DarkBench (15 benign test examples); for Qwen3-14B they are 0.008 on utility and 0.144 on benign AM runs; for Llama, 0.000 on utility and 0.129 on benign AM runs; for Gemma, 0.000 on utility and 0.316 on benign AM runs; and for gpt-oss, 0.008 on utility and 0.054 on benign AM runs. Benign agentic runs are the most difficult case, because they deliberate about the harmful option before declining it, and a threshold calibrated on utility tasks alone would miss this; τ\tau is therefore calibrated on benign training runs from every domain. A deployment that prefers fewer interruptions can calibrate τ\tau on benign AM training runs instead, which gives 32B FPRtest\text{FPR}_\text{test} 0.082 at 17.3% harm and 1.4% induced harm.

Under scenario shift, false alarms remain elevated for every feature set: with murder held out, the residual monitor's online FPRtest\text{FPR}_\text{test} is 0.32 on Qwen3-32B, 0.34 on Llama and 0.70 on gpt-oss, and other feature sets reach 0.59 (Llama) and 0.89 (gpt-oss). We report this as a limitation shared by all trained monitors.

4. Cost and Task Success (RJrC W3, ctXZ W3/Q3, s9Vc W3/Q3)

We measure cost from cumulative token usage (discarded reasoning, advisor prompt/output, and restarted continuation), weighted relative to an unmonitored run as new+c⋅cached+r⋅output\text{new} + c\cdot\text{cached} + r\cdot\text{output} (c=0.1,r=8c=0.1, r=8 or c=0.25,r=4c=0.25, r=4). We compare wiping at the alarm with wiping at the first action (or at the alarm if it comes later). The two give similar harm, and we report the first-action cutoff, the more expensive of the two, here; Appendix G gives the alarm-point costs and the per-condition token breakdowns (for example, Qwen3-14B costs 1.06×1.06\times instead of 1.18×1.18\times over all runs when wiping at the alarm).

Token multipleCost, benign runs (c=0.1c=0.1, r=8r=8)Cost, all runs (c=0.1c=0.1, r=8r=8)Cost, all runs (c=0.25c=0.25, r=4r=4)
Always-advise, Qwen3-32B / 14B2.16×2.16\times / 2.16×2.16\times—1.35×1.35\times / 1.34×1.34\times1.46×1.46\times / 1.45×1.45\times
Monitor, Qwen3-32B1.60×1.60\times1.05×1.05\times1.18×1.18\times1.24×1.24\times
Monitor, Qwen3-14B1.59×1.59\times1.05×1.05\times1.18×1.18\times1.24×1.24\times
Monitor, Llama-3.3-70B1.60×1.60\times1.08×1.08\times1.26×1.26\times1.29×1.29\times
Monitor, Gemma-4-31B2.13×2.13\times1.23×1.23\times1.70×1.70\times1.73×1.73\times
Monitor, gpt-oss-20b1.12×1.12\times1.04×1.04\times1.07×1.07\times1.07×1.07\times

Scoring the monitor costs at most 0.024 ms per token on GPU, measured for the largest feature set. On Qwen3-32B with the multidomain monitor, utility task success stays at 93.3%, the same as without monitoring (only 1 of 120 utility runs classified as harmful), whereas advising every run lowers it to 85.8%. The revision includes a deployment table with one fixed configuration per model, reporting cost on originally benign runs, on originally harmful runs, and on all runs, next to the final harm rate (s9Vc Q3).

5. Motivation for the Wipe Intervention (Transcript Inertia)

We test the impact of wiping reasoning in two ways. First, at the same detection points and with the same advice, wiping reduces harm by 25.0 points more than keeping the reasoning on Qwen3-32B (95% CI 14.6 to 35.4) and by 36.7 points more on Qwen3-14B (24.5 to 48.0); on Llama the advantage (+8.5) is not significant, and on Gemma there is none. Second, on 25 detected Llama-3.3-70B runs, we keep the transcript and steer the model in place, either subtracting the J or md direction at every generated token or capping its projection at the clean-run 75th percentile (activation capping, Lu et al. 2026). Neither undoes the harm (88% unsteered, 96–100% steered or capped), while restarting from the prompt gives 56%, and the output stays coherent. The caps rarely bind, which suggests that the kept text, rather than an elevated internal direction, carries the decision and supports discarding the transcript.

This also corrects one observation in the submission. Under the rebuilt evaluation, 14B no longer prefers retaining the reasoning: the submission used greedy decoding and the old labels, and at temperature 0.7 with the rebuilt split, wiping is better on both Qwen sizes, so a single configuration serves all models. Steering the advice itself, along the J or md direction, is now an analysis rather than a contribution, for the reasons given in section 2.

We thank the reviewers again for their careful review, which has substantially strengthened the work.

AuthorsOpenReview ↗

Summary of Revisions

Revisions

  • Title: Mitigating Agentic Misalignment via Self-Supervision.
  • Contributions: (i) an online internal monitor, evaluated as deployed (online τ\tau, held-out scenarios, per-domain false alarms) and with a matched analysis showing that the residual stream already carries the signal, and (ii) guided self-correction under transcript inertia (wipe-and-advise), reported with induced harm, cost and task success.
  • Strengthened Experiments: Experiments repeated with per-fold directions and matched layers; goal and threat holdouts replaced by scenario holdouts; five models, each with AM-only and multidomain monitors (per-domain false alarms, leave-one-domain-out, three seeds); model-spec prompting combined with the monitor.
  • Code and demo (supplementary material): anonymized code, per-model reproduction scripts, and an interactive demo in which a trained monitor triggers wipe-and-advise on user-written scenarios.
  • Completed Appendices for Reproducibility: Jacobian corpus and size, token alignment, exact autodiff, advisor prompt, specification version, KV-cache reset, and a statement on train–test separation.
  • Related Work: Cognition-of-Thought (Zhang et al. 2025), shallow safety alignment (Qi et al. 2025) and activation capping (Lu et al. 2026).
  • Future Work: a Jacobian lens fit on agentic trajectories or tuned to the detection target.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.