acceptodds
Under review as a conference paper at ICLR 2027

EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

Abstract

Large Language Model (LLM) agents turn language into real-world effects and must stay safe against both indirect prompt injections and direct harmful requests. System-level safety harnesses add an enforcement layer around the model, but existing harnesses are built once by experts and applied across heterogeneous models and domains. The effective defense is deployment-dependent: models differ in how much external enforcement they tolerate before utility drops, and domains differ in which effects, state, and action sequences must be governed. A harness strict enough for one model over-blocks another, and a policy general enough to transfer across domains misses the safety relations of the application. We present **EvoSafeHarness**, a safety-specific harness optimization framework that synthesizes a deployable harness for a frozen model in a target domain. Unlike harness generation frameworks that target utility alone, it jointly searches a natural-language policy and executable code logic, guided by the target model's behavioral feedback and a domain specification, and screens each candidate with a fresh-context adversarial review that rejects rules keyed to benchmark artifacts. Across four agent benchmark families, EvoSafeHarness establishes a stronger safety–utility frontier than fixed expert-designed defenses. On DecodingTrust-Agent, across fifteen independently searched model-by-domain deployments, it reduces attack success rate (ASR) from 50.9% to 12.6% for direct attacks and from 40.4% to 7.4% for indirect attacks at a 3.3-point utility cost. On AgentDojo it reaches 82.8% utility at 0.0% ASR, twice the utility of CaMeL at the same zero-ASR operating point, and the same harness transfers unchanged to unseen AgentDyn suites. It also attains the best score on Agent-SafetyBench for every victim and keeps ASR below 18% against an adaptive PAIR-style attacker with 16 attempts per task. Analysis of the synthesized harnesses shows that domain semantics determine which safety relations and trajectory state are required, while model and runtime behavior determine how and where those relations are enforced.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent safety, Prompt injection defense, Harness optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BGdiOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EvoSafeHarness, a framework that automatically generates safety harnesses tailored to a frozen language model and its deployment domain. It jointly optimizes natural language policies and executable enforcement code using feedback from benign and adversarial trajectories. A Criticizer operating in a fresh context reviews candidate harnesses for rules that depend on benchmark artifacts, while a cascaded evaluation procedure reduces search costs. The paper evaluates the approach on DTAP, Agent-SafetyBench, AgentDojo, and AgentDyn, with additional experiments on transfer, adaptive attacks, and component ablations.

Critical strengths

I find the question of how harnesses can enforce safety constraints on LLMs both interesting and valuable. The paper is also well written. Jointly optimizing policies and runtime controls is a meaningful systems contribution. The independent Criticizer addresses benchmark overfitting, with ablations supporting its effectiveness. The experiments demonstrate substantial safety improvements while preserving utility across multiple settings. Transfer to unseen tools and workflows provides encouraging evidence of generalization.

Critical weaknesses

I believe there is considerable room to strengthen the experimental evaluation, as the current evidence leaves some concerns unresolved.

  1. The same baselines are used across evaluations, except that SafeHarness is included only in the comparison on Agent-SafetyBench. The differences between EvoSafeHarness and SafeHarness should also be clarified. The comparisons do not isolate the benefit of deployment specific design from additional search and implementation differences. Baselines remain frozen, while the Meta-Harness comparison covers only one setting and lacks clear budget matching.
  2. Robustness evidence remains limited. Adaptive testing covers one attacker, victim, and injection surface. Some legitimate workflows that could trigger excessive blocking are also absent from evaluation, as acknowledged in Table A7. The authors do not provide code, and I also have concerns about how the baselines were configured.

Overall, I have some confidence in the paper’s claims, but my remaining concerns prevent a stronger recommendation. I would be willing to raise my score if the authors satisfactorily address these concerns.

Questions for authors
  1. Were search budgets and feedback matched for Meta-Harness? How would a shared harness perform under the same optimization procedure?
  2. What task and threat metadata are visible at runtime, and would all these fields be available in deployment?
  3. How many tasks were used for adaptive attacks, how were they selected, and what are the confidence intervals?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer BGdi — Part 1/4

We thank the reviewer for the thoughtful assessment and for recognizing the value of harness-based safety enforcement, the contribution of jointly optimizing policies and runtime controls, and the Criticizer’s role in addressing benchmark overfitting. We have carefully addressed the concerns below with additional experiments and clarifications on baseline comparisons, adaptive robustness, and evaluation protocols.

W1. Comparison with SafeHarness: empirical results and design differences

(a) Empirical results. We report results for Sonnet 4.6 and Kimi-K2.5 on os-filesystem, finance and telecom. Each cell uses 30 benign, 35 direct and 35 indirect tasks. Rates are percentages; Score = Util − overall ASR. The final two rows average the six deployments.

VictimDomainHarnessUtil ↑D-ASR ↓I-ASR ↓ASR ↓Score ↑
Sonnet 4.6os-filesystemSafeHarness66.75.70.02.963.8
Sonnet 4.6os-filesystemEvoSafeHarness80.00.00.00.080.0
Sonnet 4.6financeSafeHarness83.30.05.72.980.5
Sonnet 4.6financeEvoSafeHarness80.00.02.91.478.6
Sonnet 4.6telecomSafeHarness76.75.72.94.372.4
Sonnet 4.6telecomEvoSafeHarness83.32.90.01.481.9
Kimi-K2.5os-filesystemSafeHarness73.354.311.432.940.5
Kimi-K2.5os-filesystemEvoSafeHarness70.031.45.718.651.4
Kimi-K2.5financeSafeHarness83.311.414.312.970.5
Kimi-K2.5financeEvoSafeHarness83.38.611.410.073.3
Kimi-K2.5telecomSafeHarness80.037.15.721.458.6
Kimi-K2.5telecomEvoSafeHarness80.02.92.92.977.1
Mean6 cellsSafeHarness77.219.06.712.964.4
Mean6 cellsEvoSafeHarness79.47.63.85.773.7

Across these six deployments, EvoSafeHarness achieves lower ASR and higher mean utility: mean ASR falls from 12.9% to 5.7% (a 55.6% relative reduction), while mean utility rises from 77.2% to 79.4%. SafeHarness’s mean ASR is approximately 2.25 times EvoSafeHarness’s. The improvement covers both direct attacks (19.0% vs. 7.6%) and indirect attacks (6.7% vs. 3.8%). EvoSafeHarness has lower overall ASR in all six cells and higher Score in five, raising mean Score from 64.4 to 73.7. The largest direct-ASR reductions occur on Kimi/filesystem (54.3% to 31.4%) and Kimi/telecom (37.1% to 2.9%); SafeHarness retains a higher Score on Sonnet/finance.

(b) How EvoSafeHarness differs from SafeHarness. SafeHarness is an expert-designed harness with a fixed defense architecture: its layers, rule tables and thresholds are specified in advance and applied unchanged across the evaluated models and domains. Its design is therefore static across deployments. EvoSafeHarness makes the policy and enforcement logic themselves the objects of optimization: before deployment, it uses the target model’s benign and adversarial trajectories to search natural-language policies and executable code, including which checks to invoke, how to enforce their decisions, and what trajectory state to maintain. The key distinction is how the defense is constructed: a fixed, manually specified design versus an automatically searched design tailored to the target deployment. We will revise the paper to make this distinction explicit in the method description and related-work discussion. For reproducibility, we will provide the baseline implementations and configurations.

AuthorsOpenReview ↗

Reply to Reviewer BGdi — Part 2/4

W2. Adaptive robustness

Adaptive attacks: two methods, three victims. We expanded the single-setting evaluation with a paired study. The attacker model is DeepSeek-V4-Flash for both attack methods: PAIR and TAP. The study covers three victim models (Kimi-K2.5, DeepSeek-V4-Flash and Sonnet 4.6), two domains (finance and os-filesystem), and both direct and indirect injection surfaces. Each victim has 60 frozen tasks. Under each method, the attacker revises its candidates using the victim trajectory and exact task verdict as feedback. No-Defense, fixed DRIFT and the unedited EvoSafeHarness harness are evaluated using the same attacker model, attack methods and task sets. Successful attacks after four adaptive attempts are:

Victim / attack methodNo-DefenseFixed DRIFTEvoSafeHarness
Kimi-K2.5 / PAIR39/6040/609/60
Kimi-K2.5 / TAP45/6041/6012/60
DeepSeek-V4-Flash / PAIR31/6030/609/60
DeepSeek-V4-Flash / TAP34/6029/6010/60
Sonnet 4.6 / PAIR5/605/600/60
Sonnet 4.6 / TAP6/605/602/60

Across the Kimi and DeepSeek evaluations, EvoSafeHarness reduces successful attacks to 40/240 (16.7%), compared with 149/240 (62.1%) for No-Defense and 140/240 (58.3%) for DRIFT. TAP achieves more successes against EvoSafeHarness than PAIR on both victims (12 vs. 9 on Kimi; 10 vs. 9 on DeepSeek), but success counts remain below both comparison arms. On Sonnet, No-Defense already has low ASR (5/60 under PAIR and 6/60 under TAP), while EvoSafeHarness records 0/60 and 2/60, respectively. These results show reduced attack success across both attack methods and all three victims within the evaluated four-attempt budget.

AuthorsOpenReview ↗

Reply to Reviewer BGdi — Part 3/4

Q1. Budget-matched search and shared-harness controls

Meta-Harness comparison under a matched update limit. We added a controlled comparison on Kimi-K2.5 across finance, os-filesystem and telecom. Both methods used Claude Opus 4.8 as the outer model, the same training data within each domain, and a maximum of ten harness-update rounds per search. Both received candidate code, scores and raw training trajectories as feedback. Candidate selection used training results only, with held-out tasks reserved for final evaluation. The methods retain their respective search designs: EvoSafeHarness includes its security warm start, Criticizer and cascade, whereas Meta-Harness uses its generic search procedure.

Util and ASR are reported as percentages. Pooled ASR combines both attack channels. Score is the difference between unrounded Util and pooled ASR, in percentage points.

DomainArmUtil ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
FinanceNo-Defense60.0%42.9%34.3%38.6%21.4
FinanceMeta-Harness80.0%31.4%28.6%30.0%50.0
FinanceEvoSafeHarness83.3%8.6%11.4%10.0%73.3
os-filesystemNo-Defense80.0%94.3%51.4%72.9%7.1
os-filesystemMeta-Harness86.7%57.1%25.7%41.4%45.2
os-filesystemEvoSafeHarness70.0%31.4%5.7%18.6%51.4
TelecomNo-Defense73.3%77.1%31.4%54.3%19.0
TelecomMeta-Harness76.7%8.6%5.7%7.1%69.5
TelecomEvoSafeHarness80.0%2.9%2.9%2.9%77.1
MeanNo-Defense71.1%71.4%39.0%55.2%15.9
MeanMeta-Harness81.1%32.4%20.0%26.2%54.9
MeanEvoSafeHarness77.8%14.3%6.7%10.5%67.3

Under the matched budget, EvoSafeHarness achieves lower pooled ASR and higher Score than Meta-Harness in all three domains. Mean ASR falls from 26.2% to 10.5%, and mean Score rises from 54.9 to 67.3. We will add this comparison and clarify the budget-matching protocol in the revised paper.

Shared harness. We searched one os-filesystem harness using pooled training feedback from three victims, using Claude Opus 4.8 as the outer model and the same maximum of ten harness-update rounds per search as the per-victim searches. The pooled training data combine the corresponding per-victim training sets. The shared harness was evaluated on 300 episodes (three victims × 100 tasks). Entries below are per-victim → shared, in percent; denominators are 30 benign, 35 direct and 35 indirect tasks per arm.

VictimUtil ↑D-ASR ↓I-ASR ↓ASR ↓Score ↑
Sonnet 4.680.0 → 60.00.0 → 2.90.0 → 5.70.0 → 4.380.0 → 55.7
Kimi-K2.570.0 → 73.331.4 → 45.75.7 → 17.118.6 → 31.451.4 → 41.9
GLM-580.0 → 70.017.1 → 14.35.7 → 8.611.4 → 11.468.6 → 58.6

Per-victim harnesses achieve higher Score on all three victims. On Sonnet, they provide both higher utility and lower overall ASR. On Kimi, the shared harness retains slightly higher utility (73.3% vs. 70.0%), but its ASR rises from 18.6% to 31.4%. On GLM-5, overall ASR is unchanged at 11.4%, while per-victim utility is 10 points higher (80.0% vs. 70.0%), yielding a Score of 68.6 vs. 58.6.

Reviewer_9Dc9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoSafeHarness searches for a deployment-specific safety harness around a frozen agent model. It jointly modifies natural-language policies and executable enforcement logic using a Designer, an Analyzer, a fresh-context Criticizer, and a cascaded evaluation procedure. The method is evaluated across DecodingTrust-Agent, Agent-SafetyBench, AgentDojo, and AgentDyn. The manuscript reports large reductions in attack success while preserving utility, transfer from AgentDojo to AgentDyn, and an adaptive-attack experiment on Agent-SafetyBench.

Critical strengths

The problem formulation is practically relevant. Optimizing both policy text and model-external enforcement allows the system to modify state tracking, routing, and action checks rather than relying entirely on compliance with a system prompt. The empirical coverage is substantial. Table 1 evaluates fifteen model–domain deployments, and Table 2 includes both additional safety settings and unchanged-harness transfer. The average DTAP reductions from 50.9% to 12.6% direct ASR and from 40.4% to 7.4% indirect ASR, at a 3.3-point utility decrease, are meaningful. The fresh-context Criticizer addresses a concrete optimization failure mode: fitting recurring benchmark artifacts. The on/off comparison, code audit, and discussion of the finance exception in Appendix A provide useful evidence and appropriately show that token-keying does not explain every gain. The appendices contain valuable supporting analysis, including paired significance tests, confidence intervals, mechanism inventories, and separate outer-search and evaluator resource ledgers. These strengthen the submission beyond aggregate headline scores.

Critical weaknesses
  1. The experimental design does not sufficiently isolate the contribution of the proposed safety-specific optimization framework from the benefits of deployment-specific search. EvoSafeHarness is optimized independently for each model–domain pair, whereas the main defense baselines are ported once and subsequently frozen. The Meta-Harness comparison partially addresses this concern, but is limited to one victim–benchmark configuration. Budget-matched comparisons with a generic harness optimizer across multiple deployments, together with a shared searched harness or cross-model harness transfer, would clarify which gains require the proposed specialization and search components. This distinction is important because deployment-specific harness optimization is a central contribution of the paper.
  2. The adaptive evaluation feels too narrow for the robustness claims. It uses one victim and one injection surface. Maybe can introduce stronger attack methods.
Questions for authors

refer to weakness.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer 9Dc9 — Part 1/2

W1. Isolating per-deployment search from the framework

We thank the reviewer for highlighting this distinction. We added a generic-optimizer comparison across three deployments and a shared-harness control to assess the search procedure and per-victim specialization separately.

Matched comparison with Meta-Harness. We optimized the baseline method, metaharness, per model and domain. Here, given the victim model as Kimi-K2.5, we optimizecompared both our methods and metaharness on Kimi-K2.5 across different domains, in finance, os-filesystem and telecom, respectively. Within each domain, both methods used Claude Opus 4.8 as the outer model, identical training data and feedback (candidate code, scores and raw training trajectories), and the same limit of ten harness-update rounds per search. Both selected candidates using training results and reserved held-out tasks for final evaluation.

We report Mean Utility, ASR and Scores across the three domains and the results are shown below; ASR combines direct and indirect attacks, and Score = Util − ASR.

DomainArmUtil ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
FinanceNo-Defense60.0%42.9%34.3%38.6%21.4
FinanceMeta-Harness80.0%31.4%28.6%30.0%50.0
FinanceEvoSafeHarness83.3%8.6%11.4%10.0%73.3
os-filesystemNo-Defense80.0%94.3%51.4%72.9%7.1
os-filesystemMeta-Harness86.7%57.1%25.7%41.4%45.2
os-filesystemEvoSafeHarness70.0%31.4%5.7%18.6%51.4
TelecomNo-Defense73.3%77.1%31.4%54.3%19.0
TelecomMeta-Harness76.7%8.6%5.7%7.1%69.5
TelecomEvoSafeHarness80.0%2.9%2.9%2.9%77.1
MeanNo-Defense71.1%71.4%39.0%55.2%15.9
MeanMeta-Harness81.1%32.4%20.0%26.2%54.9
MeanEvoSafeHarness77.8%14.3%6.7%10.5%67.3

Under the matched budget, EvoSafeHarness achieves lower pooled ASR and higher Score than Meta-Harness in all three domains. Mean ASR falls from 26.2% to 10.5%, and mean Score rises from 54.9 to 67.3.

Shared versus per-victim search. We searched one os-filesystem harness on the pooled training sets of Sonnet 4.6, Kimi-K2.5 and GLM-5, using the same outer model and maximum of ten update rounds per search as the per-victim runs. Shared-harness evaluation covers 300 episodes (three victims × 100 tasks), with 30 benign, 35 direct and 35 indirect tasks per victim. Entries are per-victim → shared, in percent.

VictimUtil ↑ASR ↓Score ↑
Sonnet 4.680.0 → 60.00.0 → 4.380.0 → 55.7
Kimi-K2.570.0 → 73.318.6 → 31.451.4 → 41.9
GLM-580.0 → 70.011.4 → 11.468.6 → 58.6

Per-victim search achieves higher Score on all three victims. Sonnet gains both utility and safety; Kimi trades slightly lower utility for lower ASR; GLM-5 gains utility at unchanged ASR. These results support per-victim specialization in the tested settings, without establishing that separate searches are always necessary.

The paper also illustrates why the enforcement mechanism can depend on the model (Figure 1C; Appendix Figure A1). For the same file-moving task and injected request for an unauthorized copy, the searched Sonnet 4.6 harness uses lightweight semantic checks, whereas the GLM-5 harness adds deterministic safeguards and data-flow tracking. The safety requirement remains the same, but differing model behavior motivates adapting how it is enforced. We will add the controlled comparisons and clarify their scope and budget limits in the revision.

AuthorsOpenReview ↗

Reply to Reviewer 9Dc9 — Part 2/2

W2. Broader adaptive testing and a stronger attack method

Following the reviewer's suggestion, we expanded the adaptive evaluation to three victims (Kimi-K2.5, DeepSeek-V4-Flash and Sonnet 4.6), two domains (finance and os-filesystem), and both direct and indirect injection surfaces.

We additionally evaluated Tree of Attacks with Pruning (TAP) [1], which extends PAIR-style iterative refinement with branching over alternative attack candidates and pruning of unpromising branches. This explores a wider range of attack strategies; its original evaluation reports higher jailbreak success than PAIR on most tested models with fewer target queries. Both methods use DeepSeek-V4-Flash as the attacker model in our study.

The paired protocol uses 60 held-out tasks per victim, selected with a fixed seed and stratified by risk family, with 15 tasks per domain × surface combination. Both methods and all three defense arms use the same frozen task sets. The attacker receives the victim trajectory and exact task verdict as feedback. A task counts as a successful attack if any of four adaptive attempts succeeds. Results for No-Defense, fixed DRIFT and the unedited EvoSafeHarness harness are:

Victim / attack methodNo-DefenseDRIFTEvoSafeHarness
Kimi-K2.5 / PAIR39/6040/609/60
Kimi-K2.5 / TAP45/6041/6012/60
DeepSeek-V4-Flash / PAIR31/6030/609/60
DeepSeek-V4-Flash / TAP34/6029/6010/60
Sonnet 4.6 / PAIR5/605/600/60
Sonnet 4.6 / TAP6/605/602/60

Across Kimi and DeepSeek, EvoSafeHarness records 40/240 successful attacks (16.7%), versus 149/240 (62.1%) for No-Defense and 140/240 (58.3%) for DRIFT. TAP achieves more successes against EvoSafeHarness than PAIR on all three victims, providing a more demanding test, while EvoSafeHarness remains below both comparison arms in every setting. On Sonnet, where No-Defense already has low ASR, EvoSafeHarness records 0/60 under PAIR and 2/60 under TAP.

Attacker access to the harness policy and code. We additionally repeated PAIR on all three victims, giving the attacker the frozen harness's natural-language policy and executable enforcement code (P+C). The task sets, attacker model, trajectory and verdict feedback, and four-attempt budget remained the same. The table reports tasks with at least one successful attack.

Victim / methodNo-DefenseEvoSafeHarness without P+C accessEvoSafeHarness with P+C access
Kimi-K2.5 / PAIR39/60 (65.0%)9/60 (15.0%)11/60 (18.3%)
DeepSeek-V4-Flash / PAIR31/60 (51.7%)9/60 (15.0%)11/60 (18.3%)
Sonnet 4.6 / PAIR5/60 (8.3%)0/60 (0.0%)0/60 (0.0%)

Access to P+C increased observed attack success by two tasks each on Kimi-K2.5 and DeepSeek-V4-Flash; Sonnet 4.6 remained at 0/60. With access to the harness policy and code, EvoSafeHarness retained lower ASR than the corresponding No-Defense baseline on all three victims under the four-attempt budget.

These results extend the adaptive evidence across attack methods, victims, injection surfaces, and attacker access to the harness. We will include the task-selection protocol and confidence intervals in the revision.

[1] Mehrotra et al. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. NeurIPS, 2024.

Thank you for the response and new experiments. I will keep my score.

Reviewer_zvcpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a single, expert-designed safety harness cannot give the best safety–utility trade-off across heterogeneous victim models and application domains. It proposes EvoSafeHarness, an LLM-driven search that, for a frozen victim model and a target domain, jointly evolves a natural-language policy bundle P and an executable runtime enforcement program C (pre/post tool-call hooks, deterministic filters, quarantined victim-copy judges, trajectory state). The search loop has three safety-specific components: (i) balanced feedback that keeps benign, direct-attack and indirect-attack traces separate, analyzed by an Analyzer into failure-pattern / design-lesson logs; (ii) a fresh-context Criticizer that never sees search data and flags rules keyed to literal benchmark artifacts (file names, tool names, phrases); and (iii) a nested four-stage evaluation cascade with a bootstrap non-inferiority gate to cut evaluation cost. On DecodingTrust-Agent (DTAP; 5 victims × 3 domains, one search per cell), the method reduces direct ASR from 50.9% to 12.6% and indirect ASR from 40.4% to 7.4% at a 3.3-point utility cost; it reports 82.8% utility at 0.0% ASR on AgentDojo with zero-shot transfer to AgentDyn (75.0% / 0.0%), the best Score on Agent-SafetyBench for all three victims, and ≤17.9% ASR under a 16-attempt PAIR-style attacker. A mechanism inventory (Table A6) and search-lineage evidence (Table A7) are used to argue that the domain determines which safety relations are protected while the model determines how they are enforced.

Critical strengths
  1. Well-motivated problem and a sensible design space. The argument that model-only enforcement is probabilistic, while a fixed enforcement program limits expressiveness, cleanly motivates H = (P, C). Covering both direct (user-channel) and indirect (data-channel) attacks in one framework is valuable.
  2. Safety-specific search components address real failure modes of generic harness optimizers. Separating benign/direct/indirect evidence prevents "refuse everything" from scoring well; the Criticizer targets the specific risk that an optimizer reading its own evaluation traces learns literal-token rules. The Criticizer ablation (Table A3: −18.6/−9.1/−31.0 held-out score without it, despite equal or higher search score) and the concrete examples (tool-name–keyed PII gate, authorized_keys/ProxyCommand literals) are convincing qualitative evidence of the mechanism.
  3. Large, statistically supported reductions on DTAP. 13/15 cells show significant paired reductions; the two exceptions are near-floor Sonnet cells. Utility is reported alongside ASR, and the paper honestly notes that per-cell utility intervals overlap.
  4. Honest and useful analysis. Appendix G identifies risk types with no measured reduction (client-targeted scams, options solicitation, telecom finance fraud) and explains why (harm lies in message content of an in-scope action). Appendix A states that ablations are single searches, not variance estimates. The cost ledgers (Appendix H) are more transparent than is typical.
Critical weaknesses

W1. The comparison does not isolate the contribution of the search method, and parts of it evaluate baselines outside their threat model.

  • EvoSafeHarness is searched per (model, domain) cell on that domain's training split, whereas every baseline is "ported once and used unchanged" (L313–314). The table therefore compares tuned vs. untuned deployments; it cannot separate the value of the proposed optimizer from the value of any per-deployment adaptation (e.g., an expert or a simple LLM spending the same budget to configure CaMeL/Progent policies per domain).
  • CaMeL, DRIFT and Progent are indirect-prompt-injection defenses: they treat the user query as trusted. Reporting that they "leave direct ASR above 44%" (L159, L323) is expected by design and should not be presented as evidence that fixed harnesses fail; likewise, the observation that CaMeL slightly increases direct ASR for Qwen/DeepSeek mostly reflects noise in a defense that does not act on that channel. A fair direct-attack comparison needs a user-channel defense (e.g., a guard model such as Llama Guard / LlamaFirewall, or SafeHarness, which is only run on ASB).
  • Baseline port fidelity is hard to assess from the short descriptions (Appendix D). CaMeL obtains 0.0% utility on all three AgentDyn models; that may be faithful (CaMeL is known to be restrictive) or a port artifact, and either way a 0-utility/0-ASR point carries no security information, so the abstract's "twice the utility of CaMeL at the same zero-ASR operating point" is a weak comparison.
  • The only comparison against another search method (Meta-Harness, Fig. 3c) is a single cell (GLM-5.2 on ASB), without reported search budgets, candidate counts, stopping rules, or separate UA/ASR.

W2. No estimate of search variance, while the reported effects of design choices are of the same order as the observed spread. Each cell is one search followed by one evaluation. Table A1 shows that merely swapping the outer Designer with a fixed Kimi-K2.5 victim moves the three-domain mean score from 65.7 to 81.4 (and the default Opus 4.8 used in the main grid scores 67.3). The Criticizer (≈9–31 points) and warm-start (≈9.5–11.9 points) ablations are single runs per domain of comparable or smaller magnitude; in finance the paper itself concedes the Criticizer gap is "not attributable to token keying" (L769–771). Without repeated searches (even 3 seeds on a few cells), it is unclear which component effects exceed search-to-search variance. Relatedly, the Kimi/telecom lineage mentions a "wider train confirmation" step (L1080–1081) that does not appear in Algorithm 1.

W3. The central "model-specific" claim is supported by inventory, not by a controlled test. §4 and Figure 1C argue that a well-aligned victim needs light judges while weaker ones need deterministic gates. But there is no cross-deployment matrix: e.g., run the GLM-5/os-fs harness on Sonnet 4.6 and vice versa, or run each domain's best harness across all five victims. If a single "hardened" harness per domain performed similarly on all victims, the model-specific thesis (and the title) would be substantially weakened; if it over-blocks strong victims or under-protects weak ones, the thesis would be directly demonstrated. In Table A6, the model-specific-patch stratum contains only a handful of entries, which makes this control more, not less, important. The domain-specific claim is better supported (each domain has a signature mechanism shared by all five victims), but an analogous cross-domain transfer test would also help. Ideally the paper would report a small factorial comparison at matched search budget: a universal harness, a model-specific harness (pooled over domains), a domain-specific harness (pooled over victims), and the joint model×domain harness. The interaction term would directly quantify the value of joint specialization.

Two further design claims lack direct ablations:

  • Joint search over P and C. §2 argues that both components are necessary ("Optimizing P alone leaves enforcement probabilistic … optimizing only a fixed enforcement program … restricts the harness"), but no policy-only or code-only search is reported. A comparison of P-only, C-only, and joint search at the same budget on a strong and a weak victim would show whether joint search is needed, or whether, for example, P-only already suffices for Sonnet 4.6.
  • What the Criticizer rejects. The Criticizer ablation shows a held-out benefit, but its precision is not characterized. Some legitimate domain rules necessarily mention concrete objects (e.g., "destination must be in the approved-account list"), while others are benchmark-keyed. Please report what fraction of flagged rules were benchmark-specific versus legitimate domain invariants, its false-rejection rate, and whether it catches overfitting forms beyond renaming/relocation/paraphrase (the three it is prompted to test).

W4. Robustness evidence is limited in scope and size.

  • Adaptive attack. The reported cumulative ASRs 10.7/14.3/17.9% correspond to 3/4/5 out of 28 tasks; the number of tasks is not stated, and with n≈28 the 95% interval at 17.9% is roughly [8%, 36%]. Only one victim, one harness, one injection surface, and a black-box attacker that does not see the harness policy/code are used; there is also no undefended adaptive baseline under the same protocol. Since the harness is itself LLM-designed code whose rules are easy to read, an attacker with harness knowledge (the standard in adaptive-attack evaluation, cf. Zhan et al., 2025) is the relevant setting. "Stays robust" (L377) overstates what this shows. An ASR-versus-budget curve beyond 16 attempts, under black-box, gray-box (architecture known), and white-box (P and C readable) attackers, would make the robustness claim checkable.
  • AgentDojo/AgentDyn. Appendix D states a 70/30 split and a 285-pair held-out matrix, but many Table 2b values are not attainable on n=285 (e.g., 76.7, 73.6, 75.0, 89.7, 13.8, 3.4). The DeepSeek-V4-Pro AgentDojo row (58.6, 62.1, 37.9, 3.4, 75.9, 13.8, 89.7) is exactly consistent with n=29 (17, 18, 11, 1, 22, 4, 26 of 29). If utility (or some rows) is computed on ~30 episodes, the "0.0% ASR" and "82.8% utility" claims carry much more uncertainty than implied. Please state the population behind each column. Also, if the 70/30 split is over task×injection pairs, the held-out pairs share user tasks and injection goals with the search pairs, making AgentDojo only weakly held-out (AgentDyn transfer partly addresses this).
  • Generalization on DTAP is in-distribution: train and held-out tasks share domains and risk types. A held-out-risk-type split would better support the claim that search learns general relations rather than per-category patches.

W5. Safety and utility are entangled in ways that need clarification.

  • In several cells the harness raises benign utility substantially (e.g., Kimi-K2.5 finance 60.0→83.3; UA of DeepSeek-V3.2 and GLM-5.2 on ASB). The search objective (U − ASR) thus also rewards capability-improving prompts/hooks, which can mask the safety cost of enforcement. Please report how much of the utility comes from non-safety guidance (e.g., an ablation removing task-help text from P).
  • The AgentDyn transfer example (L908–912) credits the harness for completing a checkout where the undefended agent "stopped to ask for confirmation." Removing human confirmation before a purchase is arguably a safety regression in deployment, not a utility win; this highlights that the benchmark utility metric can reward behaviors a safety harness should not encourage.
  • Benign evaluation is 30 tasks per cell, and none of them are hard negatives that resemble attacks (the paper notes that a legitimate sell-to-fund-payment composition is absent, Table A7). Over-blocking risk of deterministic gates (e.g., "hidden/tmp location" or wash/drain ledgers) is therefore largely unmeasured.
  • Appendix B states the runtime context exposes "task and threat metadata." Please enumerate exactly which fields candidate code can read at inference time. Any field that reveals whether a task is an attack, its risk category, or the injection location would not be available in deployment and would invalidate the comparison.

W6. Presentation and reporting issues.

  • Figure 1C/D labels do not match the plotted data. Panel C is labeled "Domain: OS-system," but its numbers (Sonnet CaMeL 63.3/2.9, ours 81.1/0.9; GLM-5 68.5/29.3 vs 78.9/8.1) are each victim's three-domain averages from Table 1. Panel D is labeled "Model: GLM-5," but its numbers (OS-system 61.3/32.9 vs 76.0/16.3; finance 69.7/35.2 vs 84.6/11.7) are the five-victim averages per domain. The actual GLM-5/os-fs cell is CaMeL 63.3 utility vs ours 80.0. Since these panels illustrate the paper's main thesis, they should be recomputed from the cells their labels name.
  • Figure 3b is described relative to "full evaluation," whereas Tables A3–A4 compare against the staged 3+12+30 schedule; please clarify the baseline. Table A4's caption claims "zero score regret" but the table contains no score columns.
  • Undeclared denominators. Seven Table 1 entries are not attainable with 30/35/35 tasks (e.g., No-Defense GLM-5/finance Util 79.3 = 23/29; CaMeL telecom D-ASR 61.8 = 21/34; DRIFT telecom I-ASR 44.0 = 11/25). This is consistent with the stated exclusion of tasks lost to provider outages (L902–903), but the per-cell n is never reported, and exclusions differ by arm, so arms are not evaluated on identical task sets (which also matters for the paired tests in Appendix I). On ASB, where the paper states that a cell is reportable only at 1,440/1,440 rows, the GLM-5.2 EvoSafeHarness UBR of 12.2% is not attainable on 240 clean episodes. Please add a k/n table for Tables 1, 2a, 2b and A5.
  • Cost. Appendix H aggregates search, baselines and held-out runs. Please report the cost of one representative search (outer + inner) and the per-episode runtime overhead (extra judge calls, latency) of the deployed harness, since the approach requires a new search for every (model, domain) pair. The cascade replay (Fig. 3b) compares against the staged schedule only; a random-subset evaluation at the same token budget is the natural baseline for the claim that the cascade "preserves complete coverage" more efficiently.
  • Wording of the headline claims. EvoSafeHarness is not best on every metric: on DTAP, Progent has lower average indirect ASR (6.3% vs. 7.4%), at much lower utility (56.4% vs. 79.8%). Each method is also reported at a single operating point, so "a stronger safety–utility frontier" (abstract) is not shown; "a better observed trade-off on the evaluated deployments" would be accurate. Similarly, "Evolving" in the title suggests continual online adaptation, whereas the method is an offline per-deployment search followed by a frozen harness.
  • Minor: Appendix I applies 15 per-cell tests without multiplicity correction (with Holm, Sonnet/os-fs and GLM-5/finance would no longer be significant at 0.05; the pooled conclusion is unaffected).

W7. No released artifacts and an incompletely specified search protocol limit reproducibility. The submission contains no code, data, or anonymized repository link, no commitment to release, and no reproducibility statement. For most papers this would be a minor point, but here the method's output is itself code and prompts: the 15 searched DTAP harnesses, the AgentDojo and ASB harnesses, and the four baseline ports are described only at the level of mechanism families (Table A6) and schematics (Figures A1–A2). Without them, readers cannot check the central claims: that the selected harnesses contain no benchmark-keyed rules, that the "model-specific" mechanisms differ as described, that the baseline ports are faithful, or that the runtime context exposes no attack-revealing metadata (W5). The search itself also cannot be re-run from the paper, because the following are missing:

  • the Designer/Analyzer/Criticizer prompts;
  • the domain specifications Σd and the content of the warm-start distillation;
  • the search budget B, stopping criterion, and number of candidates per cell;
  • the estimation rule for the gate reference s0_q;
  • sampling temperatures and the number of rollouts per episode;
  • the judges used for AgentDojo/AgentDyn.

The protocol descriptions also conflict in places. Appendix D says one outer model drives the Designer, Analyzer and Criticizer (L940–941), whereas Appendix A says the warm-start/Criticizer arms pair an Opus 4.8 Designer with a Sonnet 4.6 Criticizer (L689). The search-sample size is also described both as a 30-task sample (App. B) and as 60 training tasks per domain (App. D). Given that the outer optimizer is a proprietary LLM and runs are expensive (Appendix H), exact re-runs may be infeasible even with code. That makes releasing the final harnesses, prompts, and search archives the minimum needed for verification. I ask the authors to provide them, at least as anonymized supplementary material, during the discussion period.

Questions for authors
  1. Cross-deployment control (most important): For a fixed domain, how does each searched harness perform on the other four victims, and for a fixed victim, how does each domain's harness transfer to other domains? Alternatively, how does a single harness searched on pooled data from all victims compare to per-victim harnesses at equal budget?
  2. Can you run at least 3 independent searches on 2–3 DTAP cells (and for the Criticizer-on/off arms) to quantify search variance?
  3. Can you compare against (a) a user-channel defense on DTAP direct attacks, and (b) baselines given a comparable per-deployment adaptation budget? For Meta-Harness, please report budgets, candidate counts and separate UA/ASR.
  4. Exactly which runtime-context fields ("task and threat metadata") can candidate harness code read during held-out evaluation?
  5. What are the denominators behind every column of Table 2b and Table A5? Is the AgentDojo 70/30 split over pairs or over user/injection tasks? How many tasks were used in the adaptive study, how were they chosen, and what is the undefended ASR under the same adaptive protocol? Can you add a white-box adaptive attacker that is given P and C?
  6. How many tasks were excluded per cell/arm in Table 1, and are the paired tests in Table A10 restricted to tasks evaluated under both arms?
  7. Please reconcile the outer-model description (App. A vs. App. D) and state the search budget, stopping rule and candidate count used for the main grid.
  8. Can you report utility on benign hard negatives (legitimate actions that resemble attack patterns, e.g., authorized sell-then-pay, legitimate /tmp usage), and on AgentDyn tasks that require following benign third-party instructions?
  9. How do policy-only and code-only search compare with joint P+C search at the same budget, and what fraction of Criticizer-flagged rules were genuinely benchmark-keyed versus legitimate domain invariants?
  10. Can you provide, as anonymized supplementary material, the searched harnesses (P and C) for all reported cells, the Designer/Analyzer/Criticizer prompts, Σd for each domain, and the baseline ports, and will they be publicly released?
8 comments
AuthorsOpenReview ↗

Reply to Reviewer zvcp — Part 1/7

We thank the reviewer for recognizing the motivation, the safety-specific search components, and the empirical reductions in attack success. We have added experiments addressing the central comparison, specialization and adaptive-attack questions. Below, we present these additional results and clarify the evaluation and search settings relevant to the questions.

W1 / Q3. Search-method comparison and user-channel defense

Matched comparison with Meta-Harness. We tuned the meta-harness per model and per domain. Here, we tune metaharness on added a Meta-Harness comparison on Kimi-K2.5 across different domains including finance, os-filesystem and telecom respectively. Both metaharness and our method used Claude Opus 4.8 as the outer model and identical training data within each domain. Both received candidate code, scores, and raw training trajectories as feedback and could propose up to 10 new harness versions per search, one per update round. We selected candidates using training results and reserved held-out tasks for final evaluation.

The table reports results for each domain and averages across the three domains. Pooled ASR combines direct and indirect attacks, and Score is utility minus pooled ASR.

DomainArmUtil ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
FinanceNo-Defense60.0%42.9%34.3%38.6%21.4
FinanceMeta-Harness80.0%31.4%28.6%30.0%50.0
FinanceEvoSafeHarness83.3%8.6%11.4%10.0%73.3
os-filesystemNo-Defense80.0%94.3%51.4%72.9%7.1
os-filesystemMeta-Harness86.7%57.1%25.7%41.4%45.2
os-filesystemEvoSafeHarness70.0%31.4%5.7%18.6%51.4
TelecomNo-Defense73.3%77.1%31.4%54.3%19.0
TelecomMeta-Harness76.7%8.6%5.7%7.1%69.5
TelecomEvoSafeHarness80.0%2.9%2.9%2.9%77.1
MeanNo-Defense71.1%71.4%39.0%55.2%15.9
MeanMeta-Harness81.1%32.4%20.0%26.2%54.9
MeanEvoSafeHarness77.8%14.3%6.7%10.5%67.3

Under the matched budget, EvoSafeHarness achieves lower ASR and higher Score than Meta-Harness in all three domains. Mean ASR falls from 26.2% to 10.5%, and mean Score rises from 54.9 to 67.3. Searching separately for each model–domain pair also yields a higher mean Score than the fixed baselines in the main DTAP evaluation.

Direct-attack comparisons with SafeHarness and LlamaFirewall. We evaluated both defenses on Sonnet 4.6 and Kimi-K2.5 across the three DTAP domains. Each deployment uses 30 benign, 35 direct and 35 indirect tasks per arm. We report both attack channels to show the trade-off with benign utility.

Victim / domainHarnessUtil ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
Sonnet / os-filesystemSafeHarness66.75.70.02.963.8
Sonnet / os-filesystemEvoSafeHarness80.00.00.00.080.0
Sonnet / os-filesystemLlamaFirewall66.72.90.01.465.2
Sonnet / financeSafeHarness83.30.05.72.980.5
Sonnet / financeEvoSafeHarness80.00.02.91.478.6
Sonnet / financeLlamaFirewall43.30.00.00.043.3
Sonnet / telecomSafeHarness76.75.72.94.372.4
Sonnet / telecomEvoSafeHarness83.32.90.01.481.9
Sonnet / telecomLlamaFirewall76.70.02.91.475.2
Kimi / os-filesystemSafeHarness73.354.311.432.940.5
Kimi / os-filesystemEvoSafeHarness70.031.45.718.651.4
Kimi / os-filesystemLlamaFirewall76.748.65.727.149.5
Kimi / financeSafeHarness83.311.414.312.970.5
Kimi / financeEvoSafeHarness83.38.611.410.073.3
Kimi / financeLlamaFirewall90.022.98.615.774.3
Kimi / telecomSafeHarness80.037.15.721.458.6
Kimi / telecomEvoSafeHarness80.02.92.92.977.1
Kimi / telecomLlamaFirewall70.020.017.118.651.4
MeanSafeHarness77.219.06.712.964.4
MeanEvoSafeHarness79.47.63.85.773.7
MeanLlamaFirewall70.615.75.710.759.8

Across the six deployments, EvoSafeHarness achieves a mean utility of 79.4%, a mean direct ASR of 7.6%, achieving the best on both safety and utility.

AuthorsOpenReview ↗

Reply to Reviewer zvcp — Part 2/7

W3 / Q1. Controlled evidence for model-specific specialization

A searched shared-harness control. We searched one os-filesystem harness using pooled training feedback from Sonnet 4.6, Kimi-K2.5 and GLM-5, with Claude Opus 4.8 as the outer model and the same maximum of ten update rounds per search as the per-victim runs. The shared harness was evaluated on 300 episodes: 30 benign, 35 direct and 35 indirect tasks for each victim. Entries are per-victim → shared, in percent.

VictimUtil ↑D-ASR ↓I-ASR ↓ASR ↓Score ↑
Sonnet 4.680.0 → 60.00.0 → 2.90.0 → 5.70.0 → 4.380.0 → 55.7
Kimi-K2.570.0 → 73.331.4 → 45.75.7 → 17.118.6 → 31.451.4 → 41.9
GLM-580.0 → 70.017.1 → 14.35.7 → 8.611.4 → 11.468.6 → 58.6

Per-victim harnesses achieve higher Score on all three victims, compared with a shared searched harness. On Sonnet, specialization improves both utility and safety. On Kimi, the shared harness has slightly higher utility, but ASR rises from 18.6% to 31.4%. On GLM-5, overall ASR remains 11.4%, while per-victim utility is 10 points higher. These results directly complement the mechanism inventory: using the same search procedure to produce a shared harness does not recover the per-victim trade-offs in this domain.

Transfer of the GLM-5 harness. We applied the frozen GLM-5/os-filesystem harness to Sonnet 4.6 and Kimi-K2.5. The table compares transfer with the harness independently searched for each victim in the original paper's main DTAP table. All rates are percentages. For Kimi transfer, indirect ASR uses 34 valid episodes, so pooled ASR is 19/69. The anonymous supplement includes the final os-filesystem implementations for GLM-5, Sonnet 4.6, and Kimi-K2.5 under harnesses/DTAP/os-filesystem/, so readers can inspect the source and per-victim harnesses side by side.

VictimHarnessUtil ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
Sonnet 4.6Per-victim search (original main table)80.00.00.00.080.0
Sonnet 4.6GLM-5 harness transfer46.70.02.91.445.2
Kimi-K2.5Per-victim search (original main table)70.031.45.718.651.4
Kimi-K2.5GLM-5 harness transfer66.745.78.827.539.1

EvoSafeHarness's per-victim search achieves both higher benign utility and lower pooled ASR than transferring the frozen GLM-5 harness to either victim. On Sonnet, the 34.8-point Score gain comes mainly from higher benign task completion (80.0% vs. 46.7%). On Kimi, the 12.3-point Score gain comes mainly from lower pooled ASR (18.6% vs. 27.5%).

Why model-specific design matters. A central intuition of EvoSafeHarness is that the domain defines the safety requirement, while the victim's behavior guides how the harness enforces it. Figure 1C and Appendix Figure A1 illustrate this with the same filesystem task and an injected request to make an unauthorized copy. The Sonnet 4.6 harness uses a provenance reminder and two semantic checks. The GLM-5 harness adds deterministic filters and data-flow tracking to address GLM-5's greater tendency to follow the injected request. The transfer and pooled-harness comparisons show that a harness built for the same domain can produce different safety–utility trade-offs across victims, motivating search against each victim's own behavior.

AuthorsOpenReview ↗

Reply to Reviewer zvcp — Part 3/7

W2 / Q2. Outer-model sensitivity and repeated-search variance

Table A1 changes the outer Designer with the victim held fixed, so its spread reflects sensitivity to optimizer choice. To measure run-to-run variation with a fixed Designer, we now report three independent searches per Criticizer arm on GLM-5/finance. The On arm includes the original main-table search and two additional searches. The Off arm consists of three new searches. We kept the search configuration, training tasks, and budget fixed across arms and toggled the Criticizer. We selected each harness using training results and evaluated it on the same held-out tasks.

Counts below are benign tasks completed or attacks succeeded, followed by the corresponding rate. ASR pools direct and indirect attacks, and Score = Utility − ASR. We compute Score and differences from the unrounded rates.

RunCriticizerUtility ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
R1On (main table)25/30 (83.33%)3/35 (8.57%)4/35 (11.43%)7/70 (10.00%)73.33
R1Off19/30 (63.33%)5/35 (14.29%)8/35 (22.86%)13/70 (18.57%)44.76
R2On25/30 (83.33%)3/35 (8.57%)5/35 (14.29%)8/70 (11.43%)71.90
R2Off17/30 (56.67%)4/35 (11.43%)7/35 (20.00%)11/70 (15.71%)40.95
R3On24/30 (80.00%)3/35 (8.57%)3/35 (8.57%)6/70 (8.57%)71.43
R3Off17/30 (56.67%)5/35 (14.29%)7/35 (20.00%)12/70 (17.14%)39.52
On − OffΔUtility (pp)ΔD-ASR (pp)ΔI-ASR (pp)ΔPooled ASR (pp)ΔScore (points)
R1+20.00−5.71−11.43−8.57+28.57
R2+26.67−2.86−5.71−4.29+30.95
R3+23.33−5.71−11.43−8.57+31.90
Mean+23.33−4.76−9.52−7.14+30.48

Across searches, the held-out Score was 72.22 ± 0.99 with the Criticizer and 41.75 ± 2.71 without it (mean ± sample SD). The sample Score variances were 0.98 and 7.33 points², respectively. The three On−Off Score differences averaged 30.48 ± 1.72 points, with a sample variance of 2.95 points². All three Criticizer-on searches achieved higher utility and lower pooled ASR than every Criticizer-off search. The Score gap was large relative to the observed run-to-run spread on GLM-5/finance. With three searches per arm, this is a descriptive estimate for this setting. The warm-start results retain the scope of the original single-search ablations.

For the original Criticizer ablation, the code and trajectory analyses explain specific artifact-dependent rules and their behavior under intent-preserving changes. The finance results measure the Criticizer's downstream performance effect. The original finance analysis did not attribute its gap to token keying.

W3 / Q9. Joint policy/code design and Criticizer behavior

Policy-only, code-only, and joint search. On DTAP with GLM-5 in os-filesystem, we compare search restricted to the natural-language policy (P-only), search restricted to the executable enforcement code (C-only), and the original EvoSafeHarness joint P+C result from Table 1. All three arms use the same 100 held-out tasks (30 benign, 35 direct attacks, and 35 indirect attacks). Score is Utility − pooled ASR.

Search armUtility ↑D-ASR ↓I-ASR ↓Pooled ASR ↓Score ↑
P-only24/30 (80.0%)19/35 (54.3%)12/35 (34.3%)31/70 (44.3%)35.71
C-only23/30 (76.7%)10/35 (28.6%)2/35 (5.7%)12/70 (17.1%)59.52
Joint P+C (Table 1)24/30 (80.0%)6/35 (17.1%)2/35 (5.7%)8/70 (11.4%)68.57

The policy and code play complementary roles. The natural-language policy guides the model’s judgments about user intent and authorization. Executable hooks track state across tool calls and enforce constraints before execution. In this setting, joint P+C achieved the highest Score (68.57, compared with 35.71 for P-only and 59.52 for C-only). It reduced pooled ASR to 11.4%, compared with 44.3% for P-only and 17.1% for C-only.

Audit of Criticizer flags. To assess the accuracy of the Criticizer's flags, we audited 27 issues it raised during harness search. A human reviewer, Claude Fable 5.2, and GPT-6-Astra examined each flagged issue to determine whether the rule relied on an incidental feature of the benchmark to enforce a security constraint. All three agreed that the Criticizer correctly identified benchmark-specific shortcuts in 25 of the 27 flagged cases (92.6%). Consistent with the held-out benefit the reviewer noted in W3, our repeated GLM-5/finance ablation found that enabling the Criticizer raised mean held-out Score from 41.75 to 72.22 across three searches per arm.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.