- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies when spreading a hidden side task over several PRs helps an untrusted coding agent evade a per-PR diff monitor. The authors define criteria for decomposable side tasks. They build five multi-tenant Flask services with a cache-poisoning side task. Two credential-exfiltration CLI tasks from Hills et al. serve as the contrast. Over 11,050 trajectories, gradual attacks beat the best single-PR placement on cache poisoning. 43 of 252 comparisons are significant after Holm correction, and none go the other way. On the credential tasks, single-PR attacks appear more evasive. The mechanism analysis points to task scope, requirement checking, the honest-score tail, and definitions missing from the review input. Probes on the attacker reach AUC around 0.87 under both strategies.
Critical strengths
The evaluation protocol is careful. Single-PR attacks are compared at their most evasive placement, chosen per monitor and threshold. False-positive rates are checked on held-out honest runs. Missing outcomes are kept as intervals, not imputed. I found this convincing. It matters because ordering claims between attack strategies are easy to get wrong.
I liked the five-input study in Section 5. Excerpts without the operative rule (R) keep the gap. Excerpts without an unrelated helper (U) close it. This is clean causal evidence for what the reviewer misses. In my view, it is the most useful finding for monitor design. The data, prompts and full monitor database are also released.
Critical weaknesses
I am not entirely sure the “why” part says much beyond the task design. The PR 2 and PR 3 requests say “do NOT edit” the earlier modules (Appendix A.1). A last-position single-PR attacker has to break that sentence. Appendix E.3 confirms that 100% of successful last-position single-PR attacks edit an earlier module, against 0% of honest PRs. These edits provide an explicit out-of-scope cue for the monitor. The “best placement” is also, in practice, the last position. Earlier positions yield very few successes because later honest PRs overwrite the planted modules (Appendix E.4). I therefore cannot tell how much of the gradual advantage is driven by this explicit restriction. The answer “scope fit decides the direction” is then close to the construction itself. I suggest an ablation that removes the “do NOT edit” sentences and keeps everything else.
The “when” part rests on one strong side and one rather weak side. On the credential side, the best-placement comparison in Table 1 covers Expense Tracker only, with 22 successful gradual attacks and 7 successful placement sets; the two-sided Fisher p is 0.55. The second experiment uses a different attacker (GLM-4.5-Air) and monitor (GPT-4o), and has no complete placement sets. Yet Figure 1(b) sits next to Figure 1(a), although the attacker, monitor and calibration protocol differ. This limits what the contrast tells us about the effect of the side-task family. I also wonder how this fits with Hills et al., who find gradual attacks more evasive under the GPT-4o task-description diff monitor, including on triggered-exfil. The cache side is narrow too. All 43 Holm-significant comparisons in the main experiment come from one attacker, Qwen3.8-27B. The Sonnet 5 extension adds one significant comparison, but evidence across attackers remains limited. The gap is also most pronounced under Mistral-24B, GPT-OSS-120B and GLM-4.7-Flash. Without stronger evidence for the credential contrast, the remaining empirical claim overlaps substantially with Hills et al.
Questions for authors
Clarifications:
-
Table 1 uses thresholds calibrated on 20 honest Expense Tracker trajectories, while Appendix G uses 53 pooled histories from both tools. Why do these analyses use different calibration populations, and how sensitive are the credential results to this choice?
-
Could the authors explain the difference from Hills et al. on triggered-exfil under the GPT-4o task-description diff monitor? How much comes from the attacker, attack prompt or evaluation protocol?
-
The gradual advantage is most pronounced under Mistral-24B, GPT-OSS-120B and GLM-4.7-Flash, while Sonnet 5 catches almost all successful attacks of both kinds in the reported cells. How should we reconcile this with Hills et al.’s finding that the gap grows for stronger monitors?
-
How were the 42 undefined Sol-attacker cells handled in the Holm correction? Were they retained in the family size of 252?
Suggestions:
-
Does the gradual advantage survive without the “do NOT edit” sentences?
-
Could the authors run the credential tasks under the main protocol, with the same attacker configurations, 200 honest calibration runs, complete placement sets, and nominal 1% and 2% FPR thresholds?
-
Could the authors evaluate the Section 8 recommendations on the main attack trajectories? Section 5 tests access to definitions on constructed bugs, and Appendix F tests a link-tracker, but its honest calibration is incomplete. A full comparison could include explicit requirement checking, access to relevant definitions, and stateful monitors such as:
- link-tracker [Hills et al., COLM 2026]
- the sequential monitor [Chen et al., ICLR 2026]
- the hybrid hierarchical-sequential monitor [Kale et al., ICLR 2026]
These could be evaluated using the saved trajectories, with suitable adaptations and fresh honest calibration, to test whether the gradual advantage survives.