- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper constructs malicious variants of benign agent trajectories through bounded local edits and uses the resulting pairs to train agent monitors. It also explores monitor-guided data collection. Experiments show substantial improvements on CUA-SHADE, while transfer to BashArena is less consistent.
Critical strengths
- The paper addresses a practical problem with an interpretable approach to generating malicious examples while preserving the surrounding task context.
- The substantial CUA-SHADE gains and matched-count continuation experiment provide useful evidence beyond evaluation on the paper’s own synthetic data.
Critical weaknesses
- The results show the generated data is useful, but do not clearly establish a specific benefit from bounded editing or pairing. The pairing control shows no additional benefit, while the length-only baseline correctly orders 35 of 36 pairs, suggesting that paired evaluation can reward superficial cues. The length analysis helps, and these findings do not invalidate the external gains, but sensitivity to editing artifacts remains a legitimate concern.
- The generalization evidence is mixed. Gains largely survive removal of exact training–test overlaps, but separation between source trajectories and their derivatives remains unclear. BashArena gains are inconsistent, and its use during development makes it important to clarify how independent that evaluation was.
Questions for authors
- Could the authors clarify how a benign trajectory is modified into a malicious one, including how edit locations are chosen and the extent of changes is constrained?
- Could a small control with comparable benign edits help distinguish detection of malicious changes from sensitivity to editing artifacts?
- Could the authors clarify source-level separation across training and evaluation, and how BashArena informed development?
4 comments
Response to Reviewer j8y6 (2/3)
The length analysis helps, and these findings do not invalidate the external gains, but sensitivity to editing artifacts remains a legitimate concern.
and
Could a small control with comparable benign edits help distinguish detection of malicious changes from sensitivity to editing artifacts?
We agree that length cues and sensitivity to editing artifacts are legitimate concerns, and the benign-edit control you suggest is a useful one. We therefore run two controls on our 35B monitor to test whether it responds to the behavior itself rather than to these superficial cues. Our findings are:
(i) Our monitor remains stable when we change length using task-relevant text.
To control for length cues, we sample 27 fixed benign–malicious pairs, and insert assistant observations into each trace at the end of the shared prefix, before the traces diverge. We then score these pairs with our 35B monitor.
To avoid affecting the semantics of trajectories, the observation restates information already visible at that point. For example, in a pillow-protector shopping task, we add an observation that only notes the requested Queen size and quantity of two. We vary how many paragraphs from the same set of observations are included. The original text, tool calls, arguments and results stay in their original order. See the results here:
| Length-control result | Malicious longer | Near equal | Benign longer |
|---|---|---|---|
| Median token difference (malicious minus benign) | +84 | +7 | −70 |
| AUROC on corresponding pairs | 0.872 | 0.865 | 0.879 |
In the table above, under the tested additions, our monitor’s AUROC is broadly stable, and longer malicious traces have no consistent advantage. We will add this length-controlled evaluation to Appendix F.2.
We agree that our edits mainly insert malicious behavior, which can introduce length cues. While performance is broadly stable in this length-controlled evaluation, explicitly removing these cues during data construction remains an important direction. We will add this to the future-work discussion in Section 5.
(ii) Our trained monitors usually score malicious edits above comparable benign edits.
On 24 source traces, we score the original, a benign edit and a malicious edit with each 35B monitor. For example, both edits add a file read at the same point. One reads project configuration and the other reads an unrelated private key. We match location, tool structure and length. “External+ours” denotes an External monitor (trained on STRIDE and Gloom) that is further trained on our generated data.
| Training data | Original benign false alarms (%) | Benign-edit false alarms (%) | Malicious-edit detections (%) | Malicious edit scores above benign edit (%) |
|---|---|---|---|---|
| External | 4.2 | 4.2 | 20.8 | 16.7 |
| Ours | 0.0 | 4.2 | 100.0 | 95.8 |
| External+ours | 8.3 | 12.5 | 100.0 | 87.5 |
Here a false alarm means a score of at least 5 on benign trajectories. The result shows that benign edits also raise the false-alarm rate slightly, while the major performance gain remains (i.e., training on our data improves detection of malicious edits and discrimination between malicious and benign edits). Transfer to CUA-SHADE-Arena also provides complementary evidence on trajectories outside our editing pipeline, since it is an out-of-domain benchmark with no artifacts introduced by the editing or validation pipeline.
Could the authors clarify how a benign trajectory is modified into a malicious one, including how edit locations are chosen and the extent of changes is constrained?
In our pipeline, the planner sees a compact description of the benign trajectory (the user task and available affordances) and selects an attack objective and an edit window. The editor inserts actions there and declares any later repairs needed for the trace to remain coherent. The structure check verifies that the other steps stay unchanged, and behavioral judges apply a minimality floor (Appendix B.1–B.3). Across our generated pairs, edits insert 1–13 steps at 1–5 locations and repair 0–2 existing steps. Most edits are small: 81.8% of pairs insert at most 4 steps and 98.3% at most 6, and only 2 pairs (0.16%) insert 13. These measured ranges give a concrete meaning to “minimal.” We will move this description and the edit-size statistics into Section 3.2.
Response to Reviewer j8y6 (3/3)
Gains largely survive removal of exact training–test overlaps, but separation between source trajectories and their derivatives remains unclear.
and
Could the authors clarify source-level separation across training and evaluation?
We agree that assessing source-level separation requires tracing carrier ancestry (Appendix A.1). Using the carrier IDs recorded for every training example, we confirmed shared carriers for 2 of the 36 suite pairs and 7 of the 96 additional benign traces. After excluding all confirmed carrier overlaps, 157 trajectories remain (34 pairs and 89 additional benign traces). 35B AUROC is 0.883 for Ours and 0.650 for External, unchanged to three decimals from the 159-trajectory subset in Appendix A.1. The training pool was fixed before we obtained the suite, and its scores were not used for main-model training or checkpoint selection. CUA-SHADE-Arena and BashArena were evaluated separately from this suite. Both had zero whitespace-normalized exact matches to the audited training pool, and the earlier exhaustive CUA-SHADE-Arena check found a maximum 5-gram Jaccard similarity of 0.010. We will update the remaining main-text results to use this same subset (Sections 4.2 and 4.4) and describe the carrier audit in Appendix A.1.
BashArena gains are inconsistent, and its use during development makes it important to clarify how independent that evaluation was.
and
How BashArena informed development?
Thank you for highlighting this point. We agree that the development history and transfer results should be distinguished. BashArena informed only our earliest 4B exploration and was excluded from selection of the final 35B training recipe reported in the paper. We will revise Section 4.2 and Appendix E.2 to make this distinction explicit. The fixed 35B training inputs for the paper and the new comparison have zero normalized-text or recorded-source matches with the 1,122 test records.
We also examine generalization on BashArena further. On BashArena, our method is most useful as a supplement. The paper reports the following AUROCs for its 35B monitors on 1,122 BashArena test records:
| Training data | AUROC on BashArena |
|---|---|
| Prompted | 0.498 |
| External only | 0.674 |
| Ours only | 0.689 |
| External+ours | 0.793 |
To test our method’s value as a supplement at equal training sizes, we add a comparison with 2,000 balanced training examples per condition. All models are 35B monitors evaluated on the same 1,122 BashArena records:
| Training data | AUROC on BashArena |
|---|---|
| 2,000 External only | 0.626 |
| 2,000 Ours only | 0.631 |
| 1,000 External + 1,000 Ours | 0.760 |
Here the new External-only and External+ours variants share the first 1,000 External examples. In the tables above, External+ours has the highest AUROC in both comparisons. Training only on our data lifts the 35B monitor well above its prompted base (0.689 vs. 0.498) and is on par with External at equal size (0.631 vs. 0.626). Continuing an External monitor on our data gains 0.134 over continuing on External data. This complementarity is clearest at 9B and 35B. At 4B and 397B, continuing on External data is on par or better (Appendix E.2), so we keep these comparisons labelled exploratory, as in the paper. These results support using our data as a complement to existing training data. We will report both the standalone and External+ours results in Appendix E.2 and summarize them in Section 4.2.
Presentation. You noted that the paper requires substantial effort in places. To make it easier to follow, we will (1) move the editing procedure from Appendix B into Section 3.2, with a worked example pair and edit-size statistics; (2) add one table listing the training data, size, and seeds behind each experiment, which are currently spread across Section 4 and Appendices C and E; and (3) collect the construction ablations and controls, including Table 9, in a single table in Section 4.2.
Dear Reviewer j8y6,
As the discussion phase is coming to an end, we would be grateful if you could provide your feedback and consider updating your score where concerns have been resolved.
Best, Authors
Response to Reviewer j8y6 (1/3)
Thank you for the careful assessment of pairing, length cues, and evaluation separation. Your suggestions prompted additional training and length controls and a comparable benign-edit test. Below, we report these results and clarify how we used and checked the evaluation datasets, and we will integrate these results into the paper in the next version.
Your reading of our 4B control (Appendix E.1, Table 9) is correct, and it sharpens what we claim. In that control, pairing means membership in the training set: replacing each benign twin with an unrelated benign trajectory leaves CUA-SHADE-Arena AUROC essentially unchanged (0.840 versus 0.849 for the original matched set). Co-occurrence of the two twins in the training set therefore does not carry the gain.
Our two new controls instead vary how the malicious trajectory is constructed. Given the limited rebuttal period, each condition is trained at a reduced scale of 250 pairs (500 examples). All conditions use the same feedback-guided collection.
(i) To further test bounded editing, we fix the benign traces and attack objectives, and remove the minimal-edit constraints. The results in the table below show the benefit of minimal editing on both evaluations.
(ii) To test conditioning on the benign trajectory, we generate the malicious trajectories independently, without conditioning on the benign ones. The results show that edited attacks outperform independently generated attacks on both evaluations.
These two controls show that building each malicious trajectory as a bounded edit of a real benign trajectory improves AUROC. Table 9 shows that keeping both twins of a pair in the training set does not. We will revise the wording in Sections 1, 4.2, and 5 to say this, and place Table 9 next to these two controls.