- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies sabotage detection when attacker-controlled code, comments or execution text can influence both a monitor's judgment and its subsequent evidence acquisition. TAEA places scoring and acquisition behind a shared interface of preregistered static facts and finite categorical probe receipts. It combines route-specific logistic models with interactions and a learned policy that obtains one mandatory source measurement and at most one further source or behavioral measurement. For a frozen deterministic monitor, the paper proves transcript invariance for submissions that agree on all registered measurements under shared trusted randomness, and gives a total-variation stability bound. Experiments on BigCodeBench-Sabotage and APPS report the highest pAUC@10% among the evaluated monitors. Measurement-preserving text edits leave TAEA unchanged; a separate APPS evaluation permits feedback-guided program rewrites and reports 20% evasion among TAEA's initial detections after up to 20 attempts. Acquisition ablations and source-score fusion across five model sizes provide additional evidence.
Critical strengths
- A precise and relevant security boundary. Applying the same registered evidence interface to both scoring and action selection addresses an important failure mode of adaptive oversight. The treatment of categorical failures and fresh-state execution makes the assumptions inspectable. Although the proof is straightforward, it usefully specifies what is protected and when the entire inspection transcript must remain unchanged.
- Substantial empirical evidence of practical utility. The evaluation includes two benchmarks, multiple monitor families, paired task-bootstrap comparisons, text manipulation, feedback-guided rewrites, and acquisition ablations. The APPS adaptive evaluation is particularly valuable because the attacker may change the measurements: TAEA retains 144/218 detections after attack, compared with 75/218 for Full-Evidence and 73/218 for the injection-augmented classifier (Table 6). The shared-initial-detection comparison in Appendix G.4 further strengthens this evidence beyond comparing conditional ASRs alone.
Critical weaknesses
- Data construction and operating-point details leave important validity questions unresolved. Appendix D.1 says BCB sabotage submissions are admitted through the frozen typed-measurement pipeline, without specifying the admission criteria or candidate attrition. In addition, the calibration split and its separation from model/policy selection are unclear. Table 6 reports a 5% calibration-FPR target without the realized benign test FPR; TAEA's clean TPR there is 82.57%, whereas Table 1 gives 80.28% as the maximum attainable TPR at test FPR <= 5%. This needs explanation before fixed-threshold robustness comparisons can be fully assessed.
- The experiments do not sufficiently isolate the proposed mechanisms. Table 7 compares useful policy variants, but does not separately ablate interaction features, the continuation gate and the source/behavior switch, or compare scoring-only isolation with isolation of both scoring and acquisition in the same adaptive architecture. On BCB, the pAUC gain over random continuation is only 1.04 points and over the uncertainty band 0.76 points, without uncertainty estimates for these differences. Measurement-preserving text attacks validate the interface invariant rather than general sabotage robustness; the broader result relies on one APPS attack generator with 20 attempts and validity checks on three public tests plus the original backdoor input. The evidence supports a promising system under the evaluated conditions, but leaves the distinct contribution and broader robustness uncertain. 3.The method is introduced abstractly; readers must piece together from the appendices . A compact end-to-end example, along with a table linking each registered field and probe to its possible receipt and downstream decision, would make the method easier to audit and reproduce. 4.Some terminology and examples also need sharper alignment. Table 3 defines capability change in terms of newly introduced capability categories, while Appendix J.1 describes an additional file-write example; the relationship between those descriptions should be clarified. The paper should also make more prominent that the theorem’s interface-level invariance covers a broader formal condition than the narrower text-comment intervention evaluated in Section 4.3. Finally, small labels in multi-panel Figures 4, 5, 7, and 9 make some results harder to inspect at paper scale.
Questions for authors
- Data and reference clarification: What exactly admits or rejects a generated BCB submission, how many candidates were excluded, and do the criteria depend on registered-probe differences or detectability?How does TAEA behave on correct independent solutions or substantial benign refactorings?
- Calibration clarification: Please specify calibration tasks, sample counts and separation from route fitting, alignment, policy selection and final evaluation; report thresholds and realized benign test FPRs for Tables 2 and 6. How should the APPS clean TPRs of 80.28% in Table 1 and 82.57% in Table 6 be reconciled?
- Mechanism validation: Under the same registered features and route-model family, how much remains after removing interaction terms, fixing the second action, or learning only the continuation gate? Can the authors separate scoring-only protection from the shared scoring/acquisition boundary?
- Attack-protocol clarification: What precise validity criteria explain the low APPS text-edit coverage (43/218 benign and 144/218 sabotage)?