acceptodds
Under review as a conference paper at ICLR 2027

Training Agent Monitors with Minimal Counterfactual Pairs

Abstract

Agent monitors support safer deployment by detecting malicious actions within legitimate work, sometimes before task completion. Yet separately collected benign and malicious runs can differ throughout execution, making it difficult to construct matched contrasts for training and evaluation. We propose a framework for training agent monitors with controlled minimal counterfactual pairs. We construct each pair by producing a malicious twin from a real benign trajectory through bounded local edits. An adversarial editor inserts actions for a malicious side task, while mechanical checks and behavioral judges assess structural validity and coherence. The construction also supports feedback-guided collection, using monitor responses to target subsequent edits. The pairs support direct monitor training and controlled trajectory-level, paired, and prefix-level evaluation. On CUA-SHADE-Arena, an external benchmark of computer-use agent trajectories, direct training on our pairs raises Qwen3.5-35B-A3B AUROC from 0.665 prompted to 0.866, above 0.753 with STRIDE and Gloom training alone. In a separate four-round collection study, feedback-guided generation reaches 0.881 AUROC versus 0.770 for random collection at the same training-set size. Our trained monitors achieve Pareto-optimal cost–detection trade-offs among the evaluated configurations, with Qwen3.5-397B-A17B reaching detection performance comparable to Gemini 3.1 Pro at about 3.65-fold lower modeled inference cost. This framework connects failure diagnosis, data construction, and monitor training through controlled counterfactual pairs, providing a practical way to improve detection with agent monitors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, LLM agents, agent monitoring, misbehavior detection, counterfactual data generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j8y6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper constructs malicious variants of benign agent trajectories through bounded local edits and uses the resulting pairs to train agent monitors. It also explores monitor-guided data collection. Experiments show substantial improvements on CUA-SHADE, while transfer to BashArena is less consistent.

Critical strengths
  1. The paper addresses a practical problem with an interpretable approach to generating malicious examples while preserving the surrounding task context.
  2. The substantial CUA-SHADE gains and matched-count continuation experiment provide useful evidence beyond evaluation on the paper’s own synthetic data.
Critical weaknesses
  1. The results show the generated data is useful, but do not clearly establish a specific benefit from bounded editing or pairing. The pairing control shows no additional benefit, while the length-only baseline correctly orders 35 of 36 pairs, suggesting that paired evaluation can reward superficial cues. The length analysis helps, and these findings do not invalidate the external gains, but sensitivity to editing artifacts remains a legitimate concern.
  2. The generalization evidence is mixed. Gains largely survive removal of exact training–test overlaps, but separation between source trajectories and their derivatives remains unclear. BashArena gains are inconsistent, and its use during development makes it important to clarify how independent that evaluation was.
Questions for authors
  1. Could the authors clarify how a benign trajectory is modified into a malicious one, including how edit locations are chosen and the extent of changes is constrained?
  2. Could a small control with comparable benign edits help distinguish detection of malicious changes from sensitivity to editing artifacts?
  3. Could the authors clarify source-level separation across training and evaluation, and how BashArena informed development?
4 comments
AuthorsOpenReview ↗

Response to Reviewer j8y6 (1/3)

Thank you for the careful assessment of pairing, length cues, and evaluation separation. Your suggestions prompted additional training and length controls and a comparable benign-edit test. Below, we report these results and clarify how we used and checked the evaluation datasets, and we will integrate these results into the paper in the next version.

The results show the generated data is useful, but do not clearly establish a specific benefit from bounded editing or pairing. The pairing control shows no additional benefit […]

Your reading of our 4B control (Appendix E.1, Table 9) is correct, and it sharpens what we claim. In that control, pairing means membership in the training set: replacing each benign twin with an unrelated benign trajectory leaves CUA-SHADE-Arena AUROC essentially unchanged (0.840 versus 0.849 for the original matched set). Co-occurrence of the two twins in the training set therefore does not carry the gain.

Our two new controls instead vary how the malicious trajectory is constructed. Given the limited rebuttal period, each condition is trained at a reduced scale of 250 pairs (500 examples). All conditions use the same feedback-guided collection.

(i) To further test bounded editing, we fix the benign traces and attack objectives, and remove the minimal-edit constraints. The results in the table below show the benefit of minimal editing on both evaluations.

Training dataAUROC on CUA-SHADE-ArenaAUROC on our test suite
w/ minimal-edit constraints0.8080.761
w/o minimal-edit constraints0.6790.589

(ii) To test conditioning on the benign trajectory, we generate the malicious trajectories independently, without conditioning on the benign ones. The results show that edited attacks outperform independently generated attacks on both evaluations.

Training dataAUROC on CUA-SHADE-ArenaAUROC on our test suite
Edited attacks0.8080.761
Independently generated attacks0.7440.601

These two controls show that building each malicious trajectory as a bounded edit of a real benign trajectory improves AUROC. Table 9 shows that keeping both twins of a pair in the training set does not. We will revise the wording in Sections 1, 4.2, and 5 to say this, and place Table 9 next to these two controls.

AuthorsOpenReview ↗

Response to Reviewer j8y6 (2/3)

The length analysis helps, and these findings do not invalidate the external gains, but sensitivity to editing artifacts remains a legitimate concern.

and

Could a small control with comparable benign edits help distinguish detection of malicious changes from sensitivity to editing artifacts?

We agree that length cues and sensitivity to editing artifacts are legitimate concerns, and the benign-edit control you suggest is a useful one. We therefore run two controls on our 35B monitor to test whether it responds to the behavior itself rather than to these superficial cues. Our findings are:

(i) Our monitor remains stable when we change length using task-relevant text.

To control for length cues, we sample 27 fixed benign–malicious pairs, and insert assistant observations into each trace at the end of the shared prefix, before the traces diverge. We then score these pairs with our 35B monitor.

To avoid affecting the semantics of trajectories, the observation restates information already visible at that point. For example, in a pillow-protector shopping task, we add an observation that only notes the requested Queen size and quantity of two. We vary how many paragraphs from the same set of observations are included. The original text, tool calls, arguments and results stay in their original order. See the results here:

Length-control resultMalicious longerNear equalBenign longer
Median token difference (malicious minus benign)+84+7−70
AUROC on corresponding pairs0.8720.8650.879

In the table above, under the tested additions, our monitor’s AUROC is broadly stable, and longer malicious traces have no consistent advantage. We will add this length-controlled evaluation to Appendix F.2.

We agree that our edits mainly insert malicious behavior, which can introduce length cues. While performance is broadly stable in this length-controlled evaluation, explicitly removing these cues during data construction remains an important direction. We will add this to the future-work discussion in Section 5.

(ii) Our trained monitors usually score malicious edits above comparable benign edits.

On 24 source traces, we score the original, a benign edit and a malicious edit with each 35B monitor. For example, both edits add a file read at the same point. One reads project configuration and the other reads an unrelated private key. We match location, tool structure and length. “External+ours” denotes an External monitor (trained on STRIDE and Gloom) that is further trained on our generated data.

Training dataOriginal benign false alarms (%)Benign-edit false alarms (%)Malicious-edit detections (%)Malicious edit scores above benign edit (%)
External4.24.220.816.7
Ours0.04.2100.095.8
External+ours8.312.5100.087.5

Here a false alarm means a score of at least 5 on benign trajectories. The result shows that benign edits also raise the false-alarm rate slightly, while the major performance gain remains (i.e., training on our data improves detection of malicious edits and discrimination between malicious and benign edits). Transfer to CUA-SHADE-Arena also provides complementary evidence on trajectories outside our editing pipeline, since it is an out-of-domain benchmark with no artifacts introduced by the editing or validation pipeline.

Could the authors clarify how a benign trajectory is modified into a malicious one, including how edit locations are chosen and the extent of changes is constrained?

In our pipeline, the planner sees a compact description of the benign trajectory (the user task and available affordances) and selects an attack objective and an edit window. The editor inserts actions there and declares any later repairs needed for the trace to remain coherent. The structure check verifies that the other steps stay unchanged, and behavioral judges apply a minimality floor (Appendix B.1–B.3). Across our generated pairs, edits insert 1–13 steps at 1–5 locations and repair 0–2 existing steps. Most edits are small: 81.8% of pairs insert at most 4 steps and 98.3% at most 6, and only 2 pairs (0.16%) insert 13. These measured ranges give a concrete meaning to “minimal.” We will move this description and the edit-size statistics into Section 3.2.

AuthorsOpenReview ↗

Response to Reviewer j8y6 (3/3)

Gains largely survive removal of exact training–test overlaps, but separation between source trajectories and their derivatives remains unclear.

and

Could the authors clarify source-level separation across training and evaluation?

We agree that assessing source-level separation requires tracing carrier ancestry (Appendix A.1). Using the carrier IDs recorded for every training example, we confirmed shared carriers for 2 of the 36 suite pairs and 7 of the 96 additional benign traces. After excluding all confirmed carrier overlaps, 157 trajectories remain (34 pairs and 89 additional benign traces). 35B AUROC is 0.883 for Ours and 0.650 for External, unchanged to three decimals from the 159-trajectory subset in Appendix A.1. The training pool was fixed before we obtained the suite, and its scores were not used for main-model training or checkpoint selection. CUA-SHADE-Arena and BashArena were evaluated separately from this suite. Both had zero whitespace-normalized exact matches to the audited training pool, and the earlier exhaustive CUA-SHADE-Arena check found a maximum 5-gram Jaccard similarity of 0.010. We will update the remaining main-text results to use this same subset (Sections 4.2 and 4.4) and describe the carrier audit in Appendix A.1.

BashArena gains are inconsistent, and its use during development makes it important to clarify how independent that evaluation was.

and

How BashArena informed development?

Thank you for highlighting this point. We agree that the development history and transfer results should be distinguished. BashArena informed only our earliest 4B exploration and was excluded from selection of the final 35B training recipe reported in the paper. We will revise Section 4.2 and Appendix E.2 to make this distinction explicit. The fixed 35B training inputs for the paper and the new comparison have zero normalized-text or recorded-source matches with the 1,122 test records.

We also examine generalization on BashArena further. On BashArena, our method is most useful as a supplement. The paper reports the following AUROCs for its 35B monitors on 1,122 BashArena test records:

Training dataAUROC on BashArena
Prompted0.498
External only0.674
Ours only0.689
External+ours0.793

To test our method’s value as a supplement at equal training sizes, we add a comparison with 2,000 balanced training examples per condition. All models are 35B monitors evaluated on the same 1,122 BashArena records:

Training dataAUROC on BashArena
2,000 External only0.626
2,000 Ours only0.631
1,000 External + 1,000 Ours0.760

Here the new External-only and External+ours variants share the first 1,000 External examples. In the tables above, External+ours has the highest AUROC in both comparisons. Training only on our data lifts the 35B monitor well above its prompted base (0.689 vs. 0.498) and is on par with External at equal size (0.631 vs. 0.626). Continuing an External monitor on our data gains 0.134 over continuing on External data. This complementarity is clearest at 9B and 35B. At 4B and 397B, continuing on External data is on par or better (Appendix E.2), so we keep these comparisons labelled exploratory, as in the paper. These results support using our data as a complement to existing training data. We will report both the standalone and External+ours results in Appendix E.2 and summarize them in Section 4.2.

Presentation. You noted that the paper requires substantial effort in places. To make it easier to follow, we will (1) move the editing procedure from Appendix B into Section 3.2, with a worked example pair and edit-size statistics; (2) add one table listing the training data, size, and seeds behind each experiment, which are currently spread across Section 4 and Appendices C and E; and (3) collect the construction ablations and controls, including Table 9, in a single table in Section 4.2.

Dear Reviewer j8y6,

As the discussion phase is coming to an end, we would be grateful if you could provide your feedback and consider updating your score where concerns have been resolved.

Best, Authors

Reviewer_YeLeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author introduces a framework for enhancing LLM agent monitoring by constructing minimal counterfactual pairs. The approach starts with benign agent trajectories and employs an adversarial planner and editor to inject malicious actions while keeping the surrounding execution context largely unchanged. The resulting trajectory pairs are subjected to structural and behavioral validity checks before being used to train agent monitors via supervised fine-tuning. To further improve data efficiency, the authors develop a feedback-guided collection strategy that focuses on failure cases of the current monitor. They also establish three evaluation protocols at the trajectory, paired, and prefix levels to assess monitoring performance from different perspectives. Experiments with four Qwen3.5 model sizes show that the trained monitors outperform prompted baselines and monitors trained on external datasets on CUA-SHADE-Arena. The results further indicate that feedback-guided data collection is more effective than random sampling at comparable training-set sizes, while the resulting monitors deliver competitive detection performance with relatively low inference overhead.

Critical strengths
  • Constructing closely matched benign and malicious trajectories is a promising strategy for reducing confounding differences between training examples and encouraging monitors to identify genuinely malicious behavior rather than relying on superficial cues. The proposed pipeline systematically integrates controlled trajectory editing, validity checks, and feedback-guided data collection, providing a practical and well-structured approach to generating targeted supervision for agent monitoring.

  • The paper demonstrates substantial improvements in detection performance over prompted baselines and monitors trained on external datasets. The experimental evaluation spans multiple model sizes and is complemented by paired and prefix-level analyses, offering a more comprehensive assessment of monitoring capabilities. The authors further demonstrate the effectiveness of feedback-guided data collection, highlighting its potential to improve training efficiency. Overall, these results provide convincing empirical evidence that the proposed approach can enhance monitoring effectiveness while maintaining competitive inference costs.

Critical weaknesses

The proposed synthetic data generation framework consists of several components, including adversarial planning, bounded trajectory editing, structural validation, behavioral judging, and feedback-guided selection. However, the experimental evaluation largely focuses on the complete pipeline without systematically examining the contribution of each component. In particular, it is unclear to what extent the observed improvements can be attributed to minimal counterfactual pairing itself, rather than simply increasing the number of malicious training examples, improving data quality through filtering, or leveraging feedback-guided adversarial generation. While the paper provides some controlled training comparisons and an ablation of supervision targets, these analyses do not directly assess the necessity or effectiveness of the key design choices in the data generation pipeline. Given that minimal counterfactual pairing is the central methodological contribution, more targeted component-level ablations are needed to substantiate its benefits and distinguish them from those of other pipeline components.

Questions for authors
  • Could the authors provide ablation studies to disentangle the contributions of the key components in the synthetic data generation pipeline? Specifically, how does performance change when (i) relaxing or removing the minimal-edit constraints, (ii) generating benign and malicious trajectories independently rather than as matched pairs, or (iii) omitting individual validation stages? Such experiments would help identify which components are most critical to the reported improvements.

  • While the results show that feedback-guided collection outperforms random collection, it remains unclear whether the gains stem from adaptive feedback or primarily from selecting more challenging examples. Could the authors compare against stronger data selection baselines, such as hard-example mining using a fixed monitor, under matched generation and validation budgets? This would help isolate the added value of the proposed feedback loop.

  • Since the quality checks do not involve executing the edited trajectories, it is unclear whether the inserted malicious actions are functionally valid and representative of realistic attacks. Could the authors validate a subset of the generated pairs through actual execution or human annotation? Additionally, could the authors provide evidence that the monitors learn to recognize malicious behavior rather than artifacts introduced by the editing or validation pipeline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YeLe (1/3)

Thank you for the detailed review and for asking us to separate the effects of data construction, validation, and feedback. We have added controlled comparisons, execution checks, and independent human review to address these questions.

In particular, it is unclear to what extent the observed improvements can be attributed to minimal counterfactual pairing itself, rather than simply increasing the number of malicious training examples, improving data quality through filtering, or leveraging feedback-guided adversarial generation.

and

Could the authors provide ablation studies to disentangle the contributions of the key components in the synthetic data generation pipeline?

We agree that component-level controls are needed to explain the improvement. To further validate the contributions of key components in our pipeline, we conduct new controlled comparisons based on Qwen3.5-35B-A3B. We will also integrate these new results into Section 4.2 and Appendix E.1.

First, the equal-count continued training already in the paper (Section 4.2, Table 8) tests whether our method’s benefit comes simply from adding more training data or updates. Both variants start with the same monitor trained on 1,000 STRIDE and Gloom examples (External). One continues training on 1,000 additional External examples, and the other continues training on 1,000 of our generated examples (Ours). Both therefore use 2,000 training examples in total. The results show improved detection at matched example counts:

Training dataTotal training examplesAUROC on CUA-SHADE-ArenaAUROC on our test suite
2,000 External2,0000.7290.578
1,000 External + 1,000 Ours2,0000.8510.840

Our new comparisons then test which construction choices matter through the three changes you propose: relaxing the edit constraint, generating trajectories separately, and removing individual validation stages. Given the limited rebuttal period, each condition is trained at a reduced scale of 250 pairs (500 examples). Unless a feedback ablation is stated, all conditions use the same feedback-guided collection.

(i) Relaxing the edit constraint. We fix the benign traces and attack objectives, and remove the minimal-edit constraints. The results show that minimal editing gives higher AUROC on both evaluations.

Training dataAUROC on CUA-SHADE-ArenaAUROC on our test suite
w/ minimal-edit constraints0.8080.761
w/o minimal-edit constraints0.6790.589

(ii) Generating the trajectories separately. We generate the malicious trajectories independently, without conditioning on the benign trajectories. The results show that edited attacks outperform independently generated attacks on both evaluations.

Training dataAUROC on CUA-SHADE-ArenaAUROC on our test suite
Edited attacks0.8080.761
Independently generated attacks0.7440.601

In our existing 4B control (Appendix E.1, Table 9), replacing each benign twin with an unrelated benign trajectory leaves AUROC essentially unchanged. So we did not observe an improvement from keeping both twins in our 4B control. Ablations (i) and (ii) show that building each malicious trajectory as a bounded edit of a real benign trajectory does.

(iii) Removing each validation stage. We apply each validation policy to the same candidate pool, while removing one check at a time. The results show that the genuineness verification matters most: removing it lowers AUROC from 0.808 to 0.676 on CUA-SHADE-Arena and from 0.761 to 0.667 on our test suite. Removing either of the other two checks has a smaller effect at this scale.

Validation settingAUROC on CUA-SHADE-ArenaAUROC on our test suite
All checks0.8080.761
Without the edit-structure check0.7770.760
Without the text-quality review0.7950.711
Without the genuineness verification0.6760.667
AuthorsOpenReview ↗

Response to Reviewer YeLe (2/3)

Could the authors compare against stronger data selection baselines, such as hard-example mining using a fixed monitor, under matched generation and validation budgets? This would help isolate the added value of the proposed feedback loop.

We agree that hard-example mining with a fixed monitor is the relevant control for testing the value of monitor updates. We therefore compare the two variants, each trained on 250 pairs with Qwen3.5-35B-A3B. The results show that the updated monitor gives higher AUROC in this comparison. The generation and validation budgets of the two variants are closely matched. Reaching 250 accepted pairs takes 688 generation attempts and 13,216 validation attempts with the fixed monitor. It takes 658 and 12,641 with the updated monitor. The fixed-monitor variant uses about 5% more of both, so the higher AUROC of the updated monitor does not come from a larger budget. We will report this comparison and its budgets in Section 4.3 and update the feedback limitation in Section 5.

Selection monitorAUROC on CUA-SHADE-ArenaAUROC on our test suite
Updated0.8080.761
Fixed0.7770.708

Could the authors validate a subset of the generated pairs through actual execution or human annotation?

This is a fair point: LLM-based checks alone do not establish that the actions produce their intended effects. We therefore added execution checks and independent human review. Our findings are:

(i) The key behaviors can be reproduced in minimal reconstructed environments. For 80 pairs, we rebuild the minimal environment needed to execute the key actions; its contents need not match the original environment, but the recorded tool calls reproduce the recorded outputs. We find that the benign key operation can be reproduced in 100% of pairs and the intended malicious effect in 96.25% (77 of 80). For example, we run recorded Git push commands against local repositories and check which receivers get the commit.

(ii) Human review supports the malicious labels. Three human annotators independently assessed the same ten generated pairs. Percentages in both human studies average the three annotators' rates. All three identified unauthorized behavior in every edited attack (100.0%). For original benign traces, average ratings were 76.7% within scope, 13.3% uncertain and 10.0% unauthorized. On average, 63.3% of edited attacks and 90.0% of originals were judged coherent. The review supports the attack labels while identifying continuity issues in some edited attacks and ambiguous permissions in some source trajectories. Reduced coherence could in principle serve as a shortcut for the monitor, but two results argue against it being the main signal: CUA-SHADE-Arena trajectories are not edited and so contain no such discontinuities, and benign edits matched in location, tool structure and length rarely raise alarms (benign-edit control below).

AuthorsOpenReview ↗

Response to Reviewer YeLe (3/3)

Additionally, could the authors provide evidence that the monitors learn to recognize malicious behavior rather than artifacts introduced by the editing or validation pipeline?

We agree that transfer performance alone does not resolve the artifact concern. As shown below, our trained monitors usually score malicious edits above comparable benign edits in the new control. On 24 source traces, we score the original, a benign edit and a malicious edit with each 35B monitor. For example, both edits add a file read at the same point. One reads project configuration and the other reads an unrelated private key. We match location, tool structure and length. “External+ours” denotes an External monitor (trained on STRIDE and Gloom) that is further trained on our generated data.

Training dataOriginal benign false alarms (%)Benign-edit false alarms (%)Malicious-edit detections (%)Malicious edit scores above benign edit (%)
External4.24.220.816.7
Ours0.04.2100.095.8
External+ours8.312.5100.087.5

Here a false alarm means a score of at least 5 on benign trajectories. The result shows that benign edits also raise the false-alarm rate slightly, while the major performance gain remains (i.e., training on our data improves detection of malicious edits and discrimination between malicious and benign edits). Transfer to CUA-SHADE-Arena also provides complementary evidence on trajectories outside our editing pipeline, since it is an out-of-domain benchmark with no artifacts introduced by the editing or validation pipeline.

We further conducted a human review to assess the trajectories and model explanations separately. Before inspecting explanations, we randomly sampled 50 CUA-SHADE-Arena attacks detected by our 35B monitor. Three independent annotators judged whether the agent's actions in each trajectory were unauthorized without seeing model identities. Their judgments about the trajectories themselves averaged 98.0% unauthorized and 2.0% uncertain.

The table below evaluates the evidence in the models' explanations. For the same trajectories, the annotators separately rated whether each explanation provided sufficient evidence to support its behavioral conclusion without seeing model identities. The table reports the three annotators' mean explanation ratings.

Support for the explanation's behavioral conclusionOurs, mean (%)External, mean (%)
Sufficiently supported26.00.0
Moderately supported73.321.3
Poorly supported0.758.0
Uncertain0.020.7

In the table above, our monitor produces more specific and better-supported explanations. The External-only monitor repeatedly used two generic response templates without citing specific evidence from the trajectories, in line with its fixed class-level training targets (Appendix C.1). In contrast, each of our pairs records what was injected, so its training rationale names the action, and our monitor cited concrete actions that annotators could check against the recorded behavior, although some details were incorrect. Across annotators, 99.3% of our explanations were rated at least moderately supported (26.0% sufficiently), versus 21.3% (0.0% sufficiently) for External on the same trajectories. Because CUA-SHADE-Arena trajectories are not edited, the cited evidence is recorded agent behavior, not an editing artifact. We treat these ratings as supporting evidence, with the benign-edit control above as the direct test. We will add the ratings and representative examples to a new appendix section (Appendix G).

Presentation. You noted that the paper requires substantial effort in places. To make it easier to follow, we will (1) move the editing procedure from Appendix B into Section 3.2, with a worked example pair and edit-size statistics; (2) add one table listing the training data, size, and seeds behind each experiment, which are currently spread across Section 4 and Appendices C and E; and (3) collect the construction ablations and controls, including Table 9, in a single table in Section 4.2.

Dear Reviewer YeLe,

As the discussion phase is coming to an end, we would be grateful if you could provide your feedback and consider updating your score where concerns have been resolved.

Best, Authors

Reviewer_HmQCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors are trying to tackle a data issue in training agent monitors -- benign and malicious runs are constructed and labelled separately. The authors are proposing a framework to train agent monitors with controlled minimum counterfactual pairs, in which each pair contains a benign trajectory and a synthetically created malicious twin. The authors showed improved AUROC when using data created with approach.

The authors also demonstrated that feedback guided data collection can further improve the performance.

Critical strengths
  1. The paper is well-written and easy to follow.

  2. The problem the authors are trying to solve is important in industry, when monitor models are not trained with benign and malicious pairs, the monitor models tend to over-generalize and perform poorly.

  3. The proposed data generation approach is reasonable and uses mechanism to ensure naturalness, coherence and minimality.

  4. Evaluation setup is easy to understand and designed properly

Critical weaknesses
  1. [Major] The current work does not seem to use a similar paired data creation approach as a baseline.

  2. [Minor] What does the monitor model output in training, a score (from 0-10) after reasoning? Does the training data only contain output score or it also includes reasoning?

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer HmQC

Thank you for highlighting the need to compare related paired-data construction methods and clarify the training targets. Below, we explain the methodological differences, report the new training comparisons, and specify what the monitor is supervised to predict.

The current work does not seem to use a similar paired data creation approach as a baseline.

We agree that a direct comparison with related paired-data methods would clarify our contribution. SLEIGHT-Bench is the closest paired construction, but it is an evaluation benchmark with only 40 attacks, which is too few to train on. It starts from human-authored attack strategies, implemented as transcripts with AI assistance, and repairs them into benign counterparts. Its authors report typically 2–3 hours of human authoring per transcript, plus verification. We therefore compare our construction with three alternative ways of building the training data, each trained with Qwen3.5-35B-A3B.

(i) Edits without the minimal-edit constraints. We fix the benign traces and attack objectives, and remove the minimal-edit constraints.

(ii) Independently generated attacks. We generate the malicious trajectories independently, without conditioning on the benign trajectories.

(iii) Malicious-to-benign repair. To compare the training value of SLEIGHT-Bench’s construction approach, we adapt its malicious-to-benign repair direction to our data. We start from our generated malicious trajectories and remove the attacks to produce benign counterparts. The malicious trajectories stay fixed, so only the benign side differs.

Given the limited rebuttal period, each condition is trained at a reduced scale of 250 pairs (500 examples), and all conditions use the same feedback-guided collection.

Training dataAUROC on CUA-SHADE-ArenaAUROC on our test suite
Ours: bounded edits of benign trajectories0.8080.761
(i) Edits without minimal-edit constraints0.6790.589
(ii) Independently generated attacks0.7440.601
(iii) Malicious-to-benign repair0.8030.644

Our construction has the highest AUROC on both evaluations. The repair variant is close on CUA-SHADE-Arena (0.803 vs. 0.808) but lower on our test suite (0.644 vs. 0.761). Our construction also scales more easily: benign trajectories accumulate during routine agent use and are easier to collect at scale than successful malicious runs, and our pipeline accepts them from wide range of sources and constructs malicious counterparts automatically, without first collecting or manually authoring attacks. We will add these comparisons to Appendix E.1 and clarify the scaling advantage in Section 2.

What does the monitor model output in training, a score (from 0-10) after reasoning?

Yes. The monitor is trained to generate a written rationale followed by a risk score from 0 to 10. The final line has the format Answer (0-10): N.

Does the training data only contain output score or it also includes reasoning?

The training targets contain both the reasoning text and the final score. Both parts contribute to the training loss. We will state the output format and the contents of the training targets explicitly in Section 3.3 (they are currently detailed in Appendix C.1–C.2).

Dear Reviewer HmQC,

As the discussion phase is coming to an end, we would be grateful if you could provide your feedback and consider updating your score where concerns have been resolved.

Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank all reviewers for their careful and constructive feedback. We are glad the reviewers found the problem practical and important (HmQC, j8y6), the construction approach reasonable, practical, and interpretable (HmQC, YeLe, j8y6), and the gains on CUA-SHADE-Arena substantial (YeLe, j8y6). This comment summarizes the new results and the setup they share, and details are in the individual responses.

New results

  • Construction ablations (YeLe, j8y6, HmQC). Removing the minimal-edit constraints, generating attacks independently of the benign trajectories, or removing the genuineness verification each lowers AUROC on both CUA-SHADE-Arena and our test suite. A malicious-to-benign repair variant is close on CUA-SHADE-Arena and lower on our test suite. Together with our existing 4B control (Table 9), these results support a more precise claim about pairing. The key factor is how the malicious trajectory is constructed (a bounded, carrier-conditioned edit of a real benign trajectory), rather than whether the two twins co-occur in the training set.
  • Feedback control (YeLe). Updating the feedback monitor gives higher AUROC than hard-example mining with a fixed monitor (0.808 vs. 0.777 on CUA-SHADE-Arena).
  • Artifact controls (YeLe, j8y6). Our 35B monitor detects all 24 malicious edits and flags 1 of 24 matched benign edits. Its AUROC is stable when the benign trace is made longer than the malicious one.
  • Pair validity (YeLe). The intended malicious effect is reproduced by execution in 77 of 80 pairs, and three annotators judged every reviewed edited attack to be unauthorized.
  • Evaluation separation (j8y6). Our advantage on the test suite holds after excluding all confirmed carrier overlaps (AUROC 0.883 for Ours vs. 0.650 for External). On BashArena, External+ours outperforms either data source alone at equal training size.

Shared setup for the ablations. Given the limited rebuttal period, we run the three construction ablations and the feedback control at a reduced scale: each condition is trained on 250 pairs (500 examples) with Qwen3.5-35B-A3B. All conditions use the main training recipe and are matched in training-set size and schedule, so the comparisons are controlled, but absolute AUROCs are lower than the main results trained on 1,290 pairs (e.g., 0.808 vs. 0.866 on CUA-SHADE-Arena). The full-pipeline result at this scale is in line with round 1 of our feedback study (0.804 at 628 examples, Table 2). Unless a feedback ablation is stated, every condition uses the same feedback-guided collection as the main pipeline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.