- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AEGIS is a verifier for tool-using agent trajectories. It combines evidence-grounded safety predicates, diffusion-based scoring, an Ising readout, and symbolic rules with abstention. Experiments report competitive precision and lower service time, with coverage and recall trade-offs.
Critical strengths
The focus on efficient and auditable safety oversight is practically relevant. Explicit predicates and abstention make the verifier easier to inspect than a single unstructured verdict. The component ablations provide useful evidence for evidence contrast and learned predicate interactions, helping explain the design choices.
Critical weaknesses
The evidence-extraction procedure, parameter fitting, and training/validation splits need more detail. These details are important for reproducing the verifier and assessing the separation between development and evaluation data. Detection metrics exclude UNKNOWN outputs while coverage differs across methods, so the reported precision gains would benefit from matched-coverage comparisons. The annotation-agreement percentages also need reconciliation with the underlying counts.
Questions for authors
Clarification: Could you specify the extraction procedure, the fitting of calibration and interaction parameters, and the data splits? Please also provide reconciled annotation-agreement counts. Suggested additional evidence: Could you provide TP/FP/FN/TN/UNKNOWN counts and matched-coverage comparisons with uncertainty estimates? These would help establish whether the reported precision advantage is robust.
2 comments
Response to Reviewer vrRX - Part 2
4. UNKNOWN counts, matched coverage, and uncertainty
The documented UNKNOWN counts are 22/229, 18/514, and 7/256, giving 90.39%, 96.50%, and 97.27% coverage. Precision, recall, and FPR exclude UNKNOWN. These are not unconditional detection rates: baseline subsets differ, and FormalJudge–Opus, for example, has 32.68% coverage on Mixed-SafetyClaw and 51.56% on Upward Deception. We will bring these coverage qualifications into the main comparison in future versions.
The current aggregate record is insufficient to reconstruct TP/FP/FN/TN, establish the pooling/averaging convention, or produce paired confidence intervals and matched-coverage curves. In particular, the reported 72.21% Agent-SafetyBench recall cannot be reconciled as a single integer-count rate merely by excluding UNKNOWN. We acknowledge this inconsistency and do not resolve it by inventing run averages or counts.
Accordingly, the small precision differences should not be read as a robust statistical advantage. The present evidence supports only the reported operating-point and service-time observations, subject to these aggregation and coverage limitations. Matched-coverage evaluation and uncertainty-aware comparisons remain future work rather than promised additions to this final-stage response.
We thank the reviewer again for recognizing the practical and explanatory value of AEGIS and for the constructive suggestions. We will incorporate these explanations and reporting limitations into future versions, while keeping existing observations separate from future improvements.
Response to Reviewer vrRX - Part 1
We thank the reviewer for recognizing the practical importance of efficient, auditable oversight, the inspectability provided by explicit predicates and abstention, and the useful evidence from the contrast and interaction studies. We appreciate the emphasis on reproducibility and on interpreting precision together with coverage.
1. Evidence extraction and the runtime model
The reported runtime uses frozen LLaDA-8B as its only language model; GPT-5.6-sol, Grok 4.6, and Gemini-3.7-flash provide offline reference annotations. Evidence extraction is specified as rubric-specific construction of views from the trusted goal, source-typed messages, tool calls, and observable outcomes. The shared interface preserves source, role, span, and event identity, while the predicates have different evidence requirements: an anchored witness for existential claims, domain coverage for universal claims, and jointly identified endpoints for relational claims.
For example, a failed transfer attempt supports an attempt and a failure; a separate successful authorized action does not turn that attempt into a completed unauthorized transfer. Missing evidence is insufficient evidence, not a negative finding or proof of safety. This is the distinction the contracts are designed to preserve.
These semantics explain the interface, but do not replace executable provenance. The available benchmark record does not identify the exact extractor revision bound to the reported H100 results. We will distinguish the contract specification from this implementation-provenance limitation in future versions, rather than presenting an unverified development implementation as the evaluated one.
2. Calibration, interaction fitting, and data splits
The unary interface is ηh=λhgh+dh, followed by symmetric pairwise interactions and exact enumeration of the 212=4,096 predicate states. The reported Ising ablation holds inputs, data partitions, calibration, and decision thresholds common across variants. These controls explain what changes within that comparison; they do not specify the fitting optimizer or establish disjoint fitting, calibration, selection, and test sets.
The main-table settings are α=0.5,nmc=16. The α/nmc sweep uses all 229 Agent-SafetyBench trajectories; its 79.9% maximum at 32 draws is descriptive, not an independently selected test result. We cannot supply exact split identities or a verified fitting recipe from the available records and do not infer them from earlier development experiments. This is a reproducibility limitation of the current evidence. Future versions will explicitly separate the documented interfaces and settings from the unverified fitting/split provenance, without using the sweep as a generalization guarantee.
3. Annotation agreement and the underlying counts
The intended statistic is unanimous headwise agreement among the three annotators over the 514 Mixed-SafetyClaw trajectories. The reviewer correctly identifies a reporting problem: the supplied percentages for F02, F03, F04, F05, F10, and F12 are incompatible with rounded integer unanimous counts under that common denominator. We cannot provide reconciled counts from the aggregate plot, and rounding to the nearest compatible count would fabricate measurements. We therefore do not use those percentages as verified evidence of annotation reliability.
This agreement plot is also distinct from the rubric-accuracy study on a 192-trajectory subset, which evaluates only unanimous, admissible references for each head. Those references are model-generated, not human gold, and unanimity does not establish semantic correctness. Future versions will explicitly flag the unreconciled agreement claims; improving annotation reliability and its reporting remains a future direction, not new evidence asserted here.