acceptodds
Under review as a conference paper at ICLR 2027

AEGIS: Evidence-Grounded Agent Safety Verification with Diffusion Scoring and Ising-Structured Risk Inference

Abstract

Reliable oversight of tool-using language agents requires contextual reasoning, auditable decisions, and low runtime overhead. We present AEGIS, a sidecar neuro-symbolic verifier that separates offline policy development from runtime detection. Rubric-specific extraction grounds twelve safety predicates in execution evidence. MaGiC uses frozen LLaDA-8B to score complete semantic verbalizers under bidirectional masked contexts and paired real/null contrasts. Ising-structured free-energy inference integrates these measurements into global risk; Policy As Code combines that score with hard constraints and abstention. We characterize when evidence contrasts and learned interactions improve their respective losses, and derive cascade score and threshold-decision bounds under explicit assumptions. On Agent-SafetyBench, Mixed-SafetyClaw, and Upward Deception, AEGIS achieves the highest precision on two benchmarks and competitive precision on the third among valid predictions, with – service-time speedups over direct and reasoning-augmented GPT-5.5 and Opus-4.8 judges. Component studies support moderate evidence correction and learned interactions, while recall gaps remain. The results establish a practical precision–latency trade-off with an 8B runtime model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent safety, Safety Oversight, Neuro-symbolic system, Diffusion language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vrRXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AEGIS is a verifier for tool-using agent trajectories. It combines evidence-grounded safety predicates, diffusion-based scoring, an Ising readout, and symbolic rules with abstention. Experiments report competitive precision and lower service time, with coverage and recall trade-offs.

Critical strengths

The focus on efficient and auditable safety oversight is practically relevant. Explicit predicates and abstention make the verifier easier to inspect than a single unstructured verdict. The component ablations provide useful evidence for evidence contrast and learned predicate interactions, helping explain the design choices.

Critical weaknesses

The evidence-extraction procedure, parameter fitting, and training/validation splits need more detail. These details are important for reproducing the verifier and assessing the separation between development and evaluation data. Detection metrics exclude UNKNOWN outputs while coverage differs across methods, so the reported precision gains would benefit from matched-coverage comparisons. The annotation-agreement percentages also need reconciliation with the underlying counts.

Questions for authors

Clarification: Could you specify the extraction procedure, the fitting of calibration and interaction parameters, and the data splits? Please also provide reconciled annotation-agreement counts. Suggested additional evidence: Could you provide TP/FP/FN/TN/UNKNOWN counts and matched-coverage comparisons with uncertainty estimates? These would help establish whether the reported precision advantage is robust.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vrRX - Part 1

We thank the reviewer for recognizing the practical importance of efficient, auditable oversight, the inspectability provided by explicit predicates and abstention, and the useful evidence from the contrast and interaction studies. We appreciate the emphasis on reproducibility and on interpreting precision together with coverage.

1. Evidence extraction and the runtime model

The reported runtime uses frozen LLaDA-8B as its only language model; GPT-5.6-sol, Grok 4.6, and Gemini-3.7-flash provide offline reference annotations. Evidence extraction is specified as rubric-specific construction of views from the trusted goal, source-typed messages, tool calls, and observable outcomes. The shared interface preserves source, role, span, and event identity, while the predicates have different evidence requirements: an anchored witness for existential claims, domain coverage for universal claims, and jointly identified endpoints for relational claims.

For example, a failed transfer attempt supports an attempt and a failure; a separate successful authorized action does not turn that attempt into a completed unauthorized transfer. Missing evidence is insufficient evidence, not a negative finding or proof of safety. This is the distinction the contracts are designed to preserve.

These semantics explain the interface, but do not replace executable provenance. The available benchmark record does not identify the exact extractor revision bound to the reported H100 results. We will distinguish the contract specification from this implementation-provenance limitation in future versions, rather than presenting an unverified development implementation as the evaluated one.

2. Calibration, interaction fitting, and data splits

The unary interface is ηh=λhgh+dh\eta_h=\lambda_h g_h+d_h, followed by symmetric pairwise interactions and exact enumeration of the 212=4,0962^{12}=4,096 predicate states. The reported Ising ablation holds inputs, data partitions, calibration, and decision thresholds common across variants. These controls explain what changes within that comparison; they do not specify the fitting optimizer or establish disjoint fitting, calibration, selection, and test sets.

The main-table settings are α=0.5,nmc=16\alpha=0.5,n_{\rm mc}=16. The α/nmc\alpha/n_{\rm mc} sweep uses all 229 Agent-SafetyBench trajectories; its 79.9% maximum at 32 draws is descriptive, not an independently selected test result. We cannot supply exact split identities or a verified fitting recipe from the available records and do not infer them from earlier development experiments. This is a reproducibility limitation of the current evidence. Future versions will explicitly separate the documented interfaces and settings from the unverified fitting/split provenance, without using the sweep as a generalization guarantee.

3. Annotation agreement and the underlying counts

The intended statistic is unanimous headwise agreement among the three annotators over the 514 Mixed-SafetyClaw trajectories. The reviewer correctly identifies a reporting problem: the supplied percentages for F02, F03, F04, F05, F10, and F12 are incompatible with rounded integer unanimous counts under that common denominator. We cannot provide reconciled counts from the aggregate plot, and rounding to the nearest compatible count would fabricate measurements. We therefore do not use those percentages as verified evidence of annotation reliability.

This agreement plot is also distinct from the rubric-accuracy study on a 192-trajectory subset, which evaluates only unanimous, admissible references for each head. Those references are model-generated, not human gold, and unanimity does not establish semantic correctness. Future versions will explicitly flag the unreconciled agreement claims; improving annotation reliability and its reporting remains a future direction, not new evidence asserted here.

AuthorsOpenReview ↗

Response to Reviewer vrRX - Part 2

4. UNKNOWN counts, matched coverage, and uncertainty

The documented UNKNOWN counts are 22/229, 18/514, and 7/256, giving 90.39%, 96.50%, and 97.27% coverage. Precision, recall, and FPR exclude UNKNOWN. These are not unconditional detection rates: baseline subsets differ, and FormalJudge–Opus, for example, has 32.68% coverage on Mixed-SafetyClaw and 51.56% on Upward Deception. We will bring these coverage qualifications into the main comparison in future versions.

The current aggregate record is insufficient to reconstruct TP/FP/FN/TN, establish the pooling/averaging convention, or produce paired confidence intervals and matched-coverage curves. In particular, the reported 72.21% Agent-SafetyBench recall cannot be reconciled as a single integer-count rate merely by excluding UNKNOWN. We acknowledge this inconsistency and do not resolve it by inventing run averages or counts.

Accordingly, the small precision differences should not be read as a robust statistical advantage. The present evidence supports only the reported operating-point and service-time observations, subject to these aggregation and coverage limitations. Matched-coverage evaluation and uncertainty-aware comparisons remain future work rather than promised additions to this final-stage response.

We thank the reviewer again for recognizing the practical and explanatory value of AEGIS and for the constructive suggestions. We will incorporate these explanations and reporting limitations into future versions, while keeping existing observations separate from future improvements.

Reviewer_WkeSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AEGIS is a sidecar verifier for tool-using language agents. It extracts evidence from user goals, source-typed messages, tool calls, and observable outcomes, maps the trajectory to 12 safety predicates, scores predicate verbalizers with a frozen LLaDA-8B masked-diffusion model using real/null evidence contrasts (MAGIC), aggregates predicate states with an Ising model, and applies Policy As Code (PAC) hard rules, thresholds, and abstention. On Agent-SafetyBench, Mixed-SafetyClaw, and Upward Deception, it reports high precision and 2.10–2.70 s mean service time with an 8B runtime model. The paper also gives finite-sample KL and threshold-decision bounds intended to characterize the scoring and structured-risk cascade. The empirical system is plausible and the evidence contracts are useful. However, the advertised certificates are not numerically instantiated, key calibration and dependence quantities are not estimated, and the experiments do not establish that masked diffusion scoring or Ising inference is necessary compared with simpler calibrated aggregation. The main precision comparisons are conditional on non-UNKNOWN predictions and are confounded by bespoke evidence handling, decision logic, and abstention.

Critical strengths
  1. Evidence contracts preserve safety-relevant event distinctions. The 12 predicates distinguish external instructions, instruction adoption, user authorization, attempted actions, realized effects, sensitive-data flow, and blocked or reverted actions. The explicit treatment of NA and insufficient evidence reduces a common failure mode in trajectory judging: joining evidence from different events or treating missing evidence as proof of safety.

  2. The verifier has an auditable selective-decision interface. PAC gives hard-rule violations precedence, routes missing evidence to approval, and records rule identifiers, evidence receipts, model versions, and policy versions. The controlled Ising ablation also shows that learned interactions outperform the tested random interactions, which is evidence that the structured readout is doing more than adding

Critical weaknesses
  1. The theoretical certificate is not instantiated and may be vacuous. The cascade bound depends on per-predicate semantic calibration radii κh\kappa_h, total correlation TCT_C, and downstream quantities Breg\mathcal{B}_{reg} and RpR_p. The paper reports none of these numerically. It explicitly states that the benchmark does not instantiate or validate the optional certificate (p. 6), that TCT_C is not identified by the available data and that a distribution-free bound can be uninformative (p. 25), and that the benchmark does not establish the prerequisites for an empirical certificate (p. 26). Thus the substantial mathematical apparatus does not currently provide a quantitative safety guarantee for the deployed verifier.

  2. The headline comparisons do not isolate the claimed architectural contribution. AEGIS changes evidence extraction, prompts, scoring, Ising aggregation, hard rules, and abstention relative to the LLM and FORMALJUDGE baselines. The authors acknowledge that the comparison is pipeline-level and not an isolated test of diffusion architecture (pp. 2, 7). The paper therefore does not show whether the gains come from LLaDA masked-diffusion scoring, the evidence contract, the risk aggregator, or selective abstention. The mathematical and architectural framing is stronger than the current controlled evidence.

Questions for authors
  1. Can you report numerical estimates for κh\kappa_h, TCT_C, Breg(n)\mathcal{B}_{reg}(n), and RpR_p, together with the observed decision margins?

  2. Can you prove that Ising model is better than other simple models, such as pairwise logisitic model or small MLP?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WkeS - Part 1

We thank the reviewer for recognizing the value of the evidence contracts, their preservation of event-level distinctions, and the auditable selective-decision interface. We also appreciate the recognition that the learned-versus-random interaction control provides useful component evidence. We agree that these contributions should be distinguished from numerical certification and from a claim that the chosen components are universally necessary.

1. Numerical budgets, decision margins, and the certificate: Q1

The reviewer is correct that the experiments do not instantiate κh\kappa_h, a valid bound on conditional total correlation, Breg(n)\mathcal B_{\rm reg}(n), or RpR_p. We cannot provide defensible numerical estimates of these quantities or a validated distribution of true decision margins from the available aggregate results. In particular, confidence in a model prediction is not a measurement of its error against the true conditional risk.

The analytical contribution is to identify how distinct error sources and improvements compose. Under the stated assumptions, candidate-score bias, sampling variance, semantic reference mismatch, and predicate dependence contribute to the regional KL budget, while aligned evidence correction and interactions can reduce it. Increasing nmcn_{\rm mc} reduces the explicit variance term; it does not remove semantic bias or identify the dependence term.

The operational condition is explicit. For B=Breg(n)B=\mathcal B_{\rm reg}(n),

\Pr(\widehat a\ne a^\star) \le \Pr\!\left(|p_U^\star-\tau|\le R_p\right)+\delta.$$ Here $p_U^\star$ is the true conditional dangerous-region probability, $\tau$ is the frozen threshold, and $\widehat a$ and $a^\star$ threshold $\widehat p_U$ and $p_U^\star$, respectively. The probability is over fresh admitted inputs and MC draws, for $\delta\in(0,1)$; it is not a per-input or conditional-on-acceptance guarantee. The condition $B<2\delta m^2$ makes $R_p<m$, sufficient to protect a true margin $m>0$ on the coverage event. Even $R_p<1$ alone does not imply a useful decision guarantee. Without independently justified budgets and margin information, we do not claim that the bound is nonvacuous on the evaluated benchmarks. The complete PAC policy additionally includes admission, hard rules, abstention, and approval outcomes. Its independent calibration proposition in the appendix specifies sufficient conditions for evaluating those operational losses; it is not an empirical certification of the current system. We will make this distinction more prominent in future versions. Numerical certification remains a future research direction, rather than an experiment promised or assumed in this final response.
AuthorsOpenReview ↗

Response to Reviewer WkeS - Part 2

2. Is Ising provably better than pairwise logistic regression or an MLP? Q2

No universal superiority result is claimed. Proposition 3.3 compares a fixed interaction direction uu, normalized by ∑h<h′∣uhh′∣=1\sum_{h<h'}\lvert u_{hh'}\rvert=1, with the same frozen uncoupled baseline. Suppressing the fixed evidence-contrast parameter, define regional alignment as

aI=−(∂KC(ρu)∂ρ)ρ=0.a_{\mathrm{I}}=-\left(\frac{\partial\mathcal K_C(\rho u)}{\partial\rho}\right)_{\rho=0}.

The bound is

KC(ρu)−KC(0)≤−ρaI+ρ22.\mathcal K_C(\rho u)-\mathcal K_C(0)\le-\rho a_{\mathrm{I}}+\frac{\rho^2}{2}.

The displayed bound guarantees a reduction when aI>0a_{\mathrm{I}}>0 and 0<ρ<2aI0<\rho<2a_{\mathrm{I}}. Failure of this sufficient condition does not rule out improvement. This is neither a comparison against an optimized logistic/MLP baseline nor a theorem about precision or recall.

The reason for the Ising formulation is its joint-state interpretation. It assigns probabilities to all twelve-predicate states and obtains risk by summing the mass of policy-defined dangerous states. Writing Ω={−1,+1}12\Omega=\lbrace-1,+1\rbrace^{12} for the state space and U⊂ΩU\subset\Omega for the dangerous region, its risk logit is

ℓ(g)=log⁡∑s∈UeAg(s)−log⁡∑s∈Ω∖UeAg(s).\ell(g)=\log\sum_{s\in U}e^{A_g(s)}-\log\sum_{s\in\Omega\setminus U}e^{A_g(s)}.

Here the joint-state score is

Ag(s)=∑h(λhgh+dh)sh+∑h<h′Khh′shsh′.A_g(s)=\sum_h(\lambda_hg_h+d_h)s_h+\sum_{h<h'}K_{hh'}s_hs_{h'}.

The sampled score uses ℓ^=ℓ(g^)\widehat\ell=\ell(\widehat g). A quadratic energy over predicate states is therefore not generally a quadratic logistic readout over the input unary scores. Nevertheless, Ising node conditionals themselves have logistic form, and this distinction does not establish greater predictive power than a suitably chosen discriminative model. The benefit we motivate is a coherent joint representation with an explicit region-based policy interface.

Empirically, the existing no-Ising, random-sparse, and learned-Ising controls report precision/recall/FPR of 73.43/70.15/11.31%, 54.14/32.19/48.57%, and 76.64/72.21/8.35%, respectively, under the reported common inputs, partitions, calibration, and thresholds. These are point estimates on valid predictions, not a demonstration of necessity, statistical significance, or superiority over an untested MLP. Simpler calibrated readouts remain a valuable future comparison. We will incorporate this precise theoretical scope and modeling rationale into future versions.

3. Separating system evidence from architectural attribution

We agree that the main benchmark changes multiple parts of the pipeline and uses method-specific accepted subsets. Its evidence is about the resulting system's operating point: the reported 8B verifier has 2.10–2.70-second mean service time, with competitive conditional precision, lower recall than some baselines, and coverage of 90.39%, 96.50%, and 97.27%. It is not a controlled attribution of every gain to diffusion or Ising.

The existing component study offers a narrower observation: at fixed nmc=16n_{\rm mc}=16, evidence correction changes MaGiC accuracy from 68.3% to 74.4%; the learned-interaction control gives the changes above. These support the tested mechanisms within the reported setup. They do not resolve backbone/interface confounding, establish an independently selected optimum, or remove the need for matched-coverage comparisons. We will make these distinctions explicit in future versions; causal attribution across backbones and aggregation families remains future work.

We thank the reviewer again for the recognition of AEGIS's evidence-grounded and auditable design and for the constructive recommendations. Future versions will incorporate these clarifications and present the empirical trade-offs, conditional guarantees, and remaining research directions with appropriately distinct claims.

Reviewer_GqMkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AEGIS is a sidecar verifier for tool-using agents. It extracts evidence for 12 safety predicates, scores them with frozen LLaDA-8B using real/null contrasts, merges them with a learned Ising model, and decides with executable policy code. It reports higher precision on two of three benchmarks and 3.5 to 10.8× lower service time than frontier API judges, plus conditional risk bounds.

Critical strengths

The design goal is practical, and the paper is open about abstention, conditional metrics, and what the theory does not cover. There are real component studies, and the random-coupling control honestly shows that random structure hurts.

Critical weaknesses

FormalJudge comparison needs the authors’ released code, the original configuration, and accuracy alongside precision.

The “highest precision” claim is within noise and selective. The margins are 1.21 and 2.28 points, about one trajectory on ~38 positives. There are no seeds or CIs. UNKNOWN outputs (9.6%, 3.5%, 2.7%) are excluded, and baseline coverage is not in the main text. AEGIS has a lower F1 than the best baseline on all three benchmarks (the paper also claims precision-latency). On Mixed-SafetyClaw, direct Opus has about the same FPR (8.05 vs 8.09) and higher recall. AEGIS’s ASB recall of 72.21% is not a ratio of at most 38 integers, unlike the baselines, which suggests averaging over runs that should be reported.

The diffusion claim is not isolated. The only autoregressive reference is Qwen3-8B prompting against AEGIS, and it changes both the model and the interface. A same-pipeline ablation is missing, and the authors concede this.

The Ising gain is small and not controlled. The gains are 3.2, 2.1 and 3.0 points, which is a handful of trajectories. There is no logistic or GBDT baseline over the 12 logits, and the training data and splits for the couplings are unspecified. This confounds supervision with structure.

Data and protocol problems. Mixed-SafetyClaw takes unsafe cases from an adversarial benchmark and safe cases from a capability benchmark, which invites source shortcuts. The 12 predicates map onto injection and exfiltration, and it is unclear how they detect concealed tool failure in Upward Deception.

The theory is vacuous as instantiated. The bounds are sums of unmeasured quantities (b, v, κ, T_C), and the experiments never estimate them. Corollary 4.1 is informative only if B_reg < 2δ. The PAC layer is outside the theorem, so “finite-sample guarantees” overstates the contribution.

No adaptive-attack evaluation, although the verifier reads attacker-controlled text. “Weak-to-strong” is only system-level, since agent strength is never varied and the rubrics come from stronger models.

Questions for authors

Q1 Which model does evidence extraction, and what are the data splits for the couplings and for α and n_mc? Include the confusion counts, the seeds and CIs, all-trajectory recall, and a risk–coverage curve. Q2 What do the estimated budgets give for R_p on a held-out set?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GqMk - Part 1

We thank the reviewer for recognizing the practical motivation of AEGIS, our explicit treatment of abstention and theoretical limitations, and the informative component controls. We agree that the contribution must be assessed as an efficient, auditable verification pipeline, rather than as uniformly superior detection or an empirically certified safety guarantee. We address the concerns below using the existing results.

1. Precision, recall, F1, latency, and selective prediction

The 2.28- and 1.21-percentage-point precision advantages are reported point estimates, not statistically established improvements. The intended claim is a precision–recall–service-time trade-off. On Mixed-SafetyClaw, the reviewer correctly notes that direct Opus-4.8 has slightly lower FPR (8.05% versus 8.09%) and higher recall (95.52% versus 82.59%). AEGIS instead reports 65.21% versus 64.00% precision and 2.70 versus 10.44 seconds mean service time, approximately 3.87× faster. We do not claim statistical equivalence: the FPR point estimate favors direct Opus. Nor do we claim detection dominance.

The F1 observation is also correct at the level of the displayed aggregates. Applying 2PR/(P+R)2PR/(P+R) to the reported precision and recall gives approximately 74.36%, 72.88%, and 87.59% for AEGIS, versus 75.33%, 76.65%, and 88.64% for the strongest corresponding entries. These are arithmetic summaries of rounded aggregate values, not newly measured or independently validated F1 scores. AEGIS does not lead this summary metric.

Abstention must remain visible: 22/229, 18/514, and 7/256 UNKNOWN outputs correspond to 90.39%, 96.50%, and 97.27% coverage. Baseline coverage also varies: the recorded FormalJudge–Opus coverage is 32.68% on Mixed-SafetyClaw and 51.56% on Upward Deception. These differences preclude a matched-coverage interpretation. The 3.53–10.80× service-time ratios describe the reported H100-versus-API deployments, not hardware-normalized compute, monetary savings, or statistically paired timing results. We will make this qualified comparison, including baseline coverage, more prominent in future versions.

2. Confusion counts, aggregation, and uncertainty: Q1

The concern about the 72.21% Agent-SafetyBench recall is valid: it cannot be explained simply by excluding UNKNOWN from a single integer-count evaluation on 229 trajectories. The available aggregate record does not establish its pooling or averaging convention. We therefore cannot attribute the discrepancy to repeated seeds, reconstruct TP/FP/FN/TN, or derive reliable confidence intervals from these percentages. This is an unresolved reporting inconsistency, not evidence of a verified multi-run evaluation.

Likewise, total UNKNOWN counts alone do not determine all-trajectory recall. If UNKNOWN is counted as failure to issue an autonomous unsafe verdict, that recall is TPTP+FN+U+,\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}+U_+}, where U+U_+ is the number of unsafe trajectories receiving UNKNOWN. The reported total UNKNOWN does not identify U+U_+. Nor does a single operating point determine a risk–coverage curve. We consequently do not claim robust precision superiority, provide guessed counts, or present synthetic uncertainty estimates. Future versions will explicitly delimit these reporting deficiencies; matched-coverage and uncertainty-aware evaluation remain directions for subsequent work, not additional evidence claimed in this response.

AuthorsOpenReview ↗

Response to Reviewer GqMk - Part 2

3. Extraction, fitting, splits, and FormalJudge provenance: Q1

The manuscript specifies frozen LLaDA-8B as the sole runtime language model; the three commercial annotators supply offline references. The available benchmark record does not independently identify the exact extraction implementation, so this specification should not be read as verification that LLaDA itself performs extraction. The extraction interface preserves source, role, span, and event identity, with rubric-specific witness and coverage requirements. For example, a denied transfer attempt cannot become evidence of a completed transfer by borrowing an outcome from another action.

The model interfaces are a frozen affine unary calibration and learned pairwise couplings; exact enumeration of the 4,096 predicate states specifies inference, not how those parameters were fitted. The available benchmark records do not establish the complete extractor revision, fitting configuration, or train/calibration/test identities. We cannot substitute a historical development protocol for that missing provenance. The reported Ising ablation holds inputs, partitions, calibration, and thresholds common across variants, but this alone does not establish independence between fitting and evaluation.

The main configuration is α=0.5,nmc=16\alpha=0.5,n_{\rm mc}=16. The correction/sampling sweep uses all 229 Agent-SafetyBench trajectories; its maximum is descriptive, not an independently selected held-out result. For FormalJudge, the documented commercial-model settings are thinking-off, and the Mixed-SafetyClaw/Upward Deception 80B entries use Qwen3-Next-80B-A3B-Instruct. These records do not establish exact reproduction of the original released configuration. The comparison is therefore restricted to the reported implementations and settings. We also cannot add a trustworthy end-to-end accuracy value without reconciled confusion counts; the MaGiC sweep accuracy is a different measurement and is not a substitute. Future versions will state these provenance and reproducibility limits explicitly.

4. What the existing component studies support

We agree that the Qwen3-8B prompting comparison changes both backbone and interface and therefore does not isolate diffusion architecture. A more focused existing comparison is the within-MaGiC evidence-contrast sweep: at nmc=16n_{\rm mc}=16, changing α\alpha from 0 to 0.5 raises reported accuracy from 68.3% to 74.4%; at 32 draws, it raises accuracy from 71.1% to 79.9%. This supports the usefulness of evidence correction within the tested scorer, not a general diffusion-over-autoregression claim. A same-pipeline backbone comparison remains future work.

For Ising, the reported no-Ising → learned-Ising precision/recall/FPR changes are 73.43/70.15/11.31% → 76.64/72.21/8.35%; the random-sparse variant reports 54.14/32.19/48.57%. Thus, the tested learned interactions outperform the two reported controls; adding arbitrary interactions is not sufficient. These point estimates do not establish statistical significance, separate the value of supervision from the chosen parameterization, or demonstrate superiority to logistic regression or GBDT. Such comparisons remain future research directions. We will distinguish these narrower component observations from pipeline-level claims in future versions.

5. Dataset scope, concealed failures, and adaptive attacks

Mixed-SafetyClaw does confound collection source with safety label. Evidence normalization cannot by itself rule out source-correlated shortcuts, so its result supports performance on that fixed mixture, not source-invariant generalization. Source-balanced evaluation is a future direction.

For Upward Deception, F10 and F12 represent a relevant attempted action and its observable failure, block, or reversal. This distinguishes attempts from outcomes, but detecting failure is not equivalent to detecting concealment. The twelve contracts do not explicitly test consistency between the final report and the observed tool outcome. The benchmark result therefore does not establish deception-complete semantics; report–outcome consistency is a direction for extending the predicate set.

We have not evaluated attacks adapted to the frozen verifier. Auditable policy execution is not proof of adversarial robustness. Similarly, the present small-model-versus-frontier-judge comparison is system-level; it does not demonstrate oversight across increasing agent capability. We will replace any broader reading of “weak-to-strong” with this precise scope. Adaptive robustness and capability-scaling studies remain future work.

AuthorsOpenReview ↗

Response to Reviewer GqMk - Part 3

6. Numerical certificates and their scope: Q2

We cannot provide a validated held-out RpR_p: the required semantic calibration budgets, conditional dependence bound, and simultaneous gain bounds have not been independently established. Corollary 4.1 is a conditional error-propagation result, not an instantiated certificate. Writing B=Breg(n)B=\mathcal B_{\rm reg}(n), its radius is Rp=B/(2δ)R_p=\sqrt{B/(2\delta)}. For a deployment failure level δ∈(0,1)\delta\in(0,1), B<2δB<2\delta gives Rp<1R_p<1, while B<2δm2B<2\delta m^2 gives Rp<mR_p<m. The latter is sufficient for this certificate to protect a true probability margin mm on its coverage event; an informative population decision bound also needs control of the mass near the threshold.

The cascade theorem concerns the admitted score branch, not the entire PAC procedure. The separate complete-policy calibration result in the appendix states additional requirements; the experiments do not instantiate it. PAC means Policy As Code. Empirical certification is future work, and future versions will explicitly describe the theoretical contribution as conditional finite-sample analysis, without suggesting an established deployment guarantee.

We thank the reviewer again for recognizing the practical value and transparency of AEGIS and for the constructive scrutiny. We will incorporate these explanations and limitations into future versions, with clearer separation between reported empirical observations, conditional theory, and future optimization directions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.