-
The monitor receives the malicious objective in advance, substantially limiting external validity.
Page 7 explicitly discloses that the monitor input contains the environment's hidden system message, including the attacker's side-task specification. The monitor may therefore already know what the attacker intends to accomplish when judging whether the current action advances that objective.
This differs substantially from detecting unknown attacks in real deployments. Giving both methods the same information ensures input consistency, but different prompting strategies may exploit that information to different degrees. Its effect on the relative advantage therefore cannot be ruled out. The paper leaves prefix removal to future work, although this should be a central ablation supporting its main conclusions. (pp. 7, 9)
-
Online interception is evaluated through log replay, without validating actual blocking capability.
Page 6 explicitly states that the authors do not run or gate a live agent. Instead, they replay existing logs prefix by prefix. The evidence therefore establishes that the monitor could potentially raise an alarm earlier in the logged action sequence.
The same page reports that the causal-cone monitor takes approximately one minute to process a five-step chunk and is slower than the baseline. Synchronous monitoring requires evaluating the resulting task latency and throughput; asynchronous execution requires assessing whether harm has already occurred by the time the alarm arrives. Log replay is a reasonable evaluation approach, but it does not establish real-time deployment feasibility or end-to-end safety benefits. (p. 6)
-
The mechanistic interpretation of the causal cone exceeds the evidence, and the algorithmic contribution has not been sufficiently isolated.
The paper explicitly states that the method neither constructs a state-transition model nor enumerates a reachable set. Instead, it asks an LLM to score the potential consequences of an action. The method is therefore closer to reachability-inspired prompting and does not provide formal causal analysis or safety guarantees.
Both monitors actually read the same complete transcript. The causal-cone monitor additionally generates structured evidence, while output format and token expenditure are not matched. The current results cannot clearly distinguish gains from reachability reasoning, evidence generation, or prompt wording.
Furthermore, injection effectiveness is defined using baseline scores and then related to performance gains over that same baseline, so the two quantities share the baseline signal. With only two attack conditions, this contrast is insufficient to establish the proposed causal mechanism. Factorial experiments matching output format and computational expenditure, together with adaptive attacks specifically optimized against the causal-cone prompt, are needed. (pp. 5, 8, 17, 20)
-
Statistical support for the main gains is limited, and generalization across independent tasks remains weak.
The main average improvement on SHADE deceptive reasoning is approximately 17 percentage points, but its 95% confidence interval is [−0.00,+0.23], placing it at the boundary of significance. At a false-alarm budget of 0.10, the improvement is approximately six percentage points, with a confidence interval of [−0.19,+0.29].
The SHADE monitor-hijacking evaluation contains only 15 valid trajectories, and the confidence intervals for the gains span zero at every budget. Although the computer-use experiments contain more trajectories, they cover only approximately 13 task clusters; the trajectories cannot all be treated as independent evidence of generalization.
These results support the possibility that offline performance and timely detection can diverge, but provide weaker support for consistent superiority across environments and models. More independent tasks, attack strategies, and external test sets are needed. (pp. 8–9, 18–20)
-
False-alarm budgets are relatively high, and equal budgets do not imply equal realized false-alarm costs.
The main experiments primarily use trajectory-level false-alarm budgets of 10%–25%. This is sufficient to expose an evaluation gap, but provides limited support for deployment under stringent low-false-alarm requirements. The authors also acknowledge that the benign samples are too small to evaluate lower false-alarm regimes reliably.
In the five-fold validation, the realized false-alarm rates at a budget of 0.25 are approximately 0.22 and 0.14 for the two monitors; at a budget of 0.20, they are approximately 0.21 and 0.10. Thus, the methods satisfy the same budget constraint but incur different realized false-alarm costs.
The finer-grained baseline wins at near-zero false-alarm rates, demonstrating that the method's advantage depends strongly on the operating budget. Further evaluation should compare the methods at the same realized false-alarm rate and report legitimate-task completion rates and human review costs. (pp. 5, 19–21)
-
Deterministic harm horizons do not guarantee semantic validity, and filtering affects the scope of the evaluation.
Rule-based detectors provide reproducible and auditable values of tI, which is a strength. However, determining when actions such as file writes or changes to numerical fields in documents constitute the first irreversible harm still requires an independent definition and validation. Detecting completion of an attacker objective is not necessarily equivalent to locating the earliest point of irreversible risk.
The paper also excludes one SHADE task because benign and malicious behavior share the same action class, and computes PREH only for malicious trajectories with finite tI. The metric therefore primarily measures whether attacks that would otherwise cause identifiable harm can be detected in advance. It should not be interpreted directly as the overall safety rate across all malicious attempts. Additional evidence is needed on horizon-annotation agreement, sensitivity to the chosen time point, and performance over the complete task set. (pp. 6, 21)
I thank the author for their response. The author addresses part of my concerns, but remains the weakness considering the novelty and adaptive attack. I will keep my score.