- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EvoSafeHarness, a framework that automatically generates safety harnesses tailored to a frozen language model and its deployment domain. It jointly optimizes natural language policies and executable enforcement code using feedback from benign and adversarial trajectories. A Criticizer operating in a fresh context reviews candidate harnesses for rules that depend on benchmark artifacts, while a cascaded evaluation procedure reduces search costs. The paper evaluates the approach on DTAP, Agent-SafetyBench, AgentDojo, and AgentDyn, with additional experiments on transfer, adaptive attacks, and component ablations.
Critical strengths
I find the question of how harnesses can enforce safety constraints on LLMs both interesting and valuable. The paper is also well written. Jointly optimizing policies and runtime controls is a meaningful systems contribution. The independent Criticizer addresses benchmark overfitting, with ablations supporting its effectiveness. The experiments demonstrate substantial safety improvements while preserving utility across multiple settings. Transfer to unseen tools and workflows provides encouraging evidence of generalization.
Critical weaknesses
I believe there is considerable room to strengthen the experimental evaluation, as the current evidence leaves some concerns unresolved.
- The same baselines are used across evaluations, except that SafeHarness is included only in the comparison on Agent-SafetyBench. The differences between EvoSafeHarness and SafeHarness should also be clarified. The comparisons do not isolate the benefit of deployment specific design from additional search and implementation differences. Baselines remain frozen, while the Meta-Harness comparison covers only one setting and lacks clear budget matching.
- Robustness evidence remains limited. Adaptive testing covers one attacker, victim, and injection surface. Some legitimate workflows that could trigger excessive blocking are also absent from evaluation, as acknowledged in Table A7. The authors do not provide code, and I also have concerns about how the baselines were configured.
Overall, I have some confidence in the paper’s claims, but my remaining concerns prevent a stronger recommendation. I would be willing to raise my score if the authors satisfactorily address these concerns.
Questions for authors
- Were search budgets and feedback matched for Meta-Harness? How would a shared harness perform under the same optimization procedure?
- What task and threat metadata are visible at runtime, and would all these fields be available in deployment?
- How many tasks were used for adaptive attacks, how were they selected, and what are the confidence intervals?
6 comments
Reply to Reviewer BGdi — Part 2/4
W2. Adaptive robustness
Adaptive attacks: two methods, three victims. We expanded the single-setting evaluation with a paired study. The attacker model is DeepSeek-V4-Flash for both attack methods: PAIR and TAP. The study covers three victim models (Kimi-K2.5, DeepSeek-V4-Flash and Sonnet 4.6), two domains (finance and os-filesystem), and both direct and indirect injection surfaces. Each victim has 60 frozen tasks. Under each method, the attacker revises its candidates using the victim trajectory and exact task verdict as feedback. No-Defense, fixed DRIFT and the unedited EvoSafeHarness harness are evaluated using the same attacker model, attack methods and task sets. Successful attacks after four adaptive attempts are:
| Victim / attack method | No-Defense | Fixed DRIFT | EvoSafeHarness |
|---|---|---|---|
| Kimi-K2.5 / PAIR | 39/60 | 40/60 | 9/60 |
| Kimi-K2.5 / TAP | 45/60 | 41/60 | 12/60 |
| DeepSeek-V4-Flash / PAIR | 31/60 | 30/60 | 9/60 |
| DeepSeek-V4-Flash / TAP | 34/60 | 29/60 | 10/60 |
| Sonnet 4.6 / PAIR | 5/60 | 5/60 | 0/60 |
| Sonnet 4.6 / TAP | 6/60 | 5/60 | 2/60 |
Across the Kimi and DeepSeek evaluations, EvoSafeHarness reduces successful attacks to 40/240 (16.7%), compared with 149/240 (62.1%) for No-Defense and 140/240 (58.3%) for DRIFT. TAP achieves more successes against EvoSafeHarness than PAIR on both victims (12 vs. 9 on Kimi; 10 vs. 9 on DeepSeek), but success counts remain below both comparison arms. On Sonnet, No-Defense already has low ASR (5/60 under PAIR and 6/60 under TAP), while EvoSafeHarness records 0/60 and 2/60, respectively. These results show reduced attack success across both attack methods and all three victims within the evaluated four-attempt budget.
Reply to Reviewer BGdi — Part 3/4
Q1. Budget-matched search and shared-harness controls
Meta-Harness comparison under a matched update limit. We added a controlled comparison on Kimi-K2.5 across finance, os-filesystem and telecom. Both methods used Claude Opus 4.8 as the outer model, the same training data within each domain, and a maximum of ten harness-update rounds per search. Both received candidate code, scores and raw training trajectories as feedback. Candidate selection used training results only, with held-out tasks reserved for final evaluation. The methods retain their respective search designs: EvoSafeHarness includes its security warm start, Criticizer and cascade, whereas Meta-Harness uses its generic search procedure.
Util and ASR are reported as percentages. Pooled ASR combines both attack channels. Score is the difference between unrounded Util and pooled ASR, in percentage points.
| Domain | Arm | Util ↑ | D-ASR ↓ | I-ASR ↓ | Pooled ASR ↓ | Score ↑ |
|---|---|---|---|---|---|---|
| Finance | No-Defense | 60.0% | 42.9% | 34.3% | 38.6% | 21.4 |
| Finance | Meta-Harness | 80.0% | 31.4% | 28.6% | 30.0% | 50.0 |
| Finance | EvoSafeHarness | 83.3% | 8.6% | 11.4% | 10.0% | 73.3 |
| os-filesystem | No-Defense | 80.0% | 94.3% | 51.4% | 72.9% | 7.1 |
| os-filesystem | Meta-Harness | 86.7% | 57.1% | 25.7% | 41.4% | 45.2 |
| os-filesystem | EvoSafeHarness | 70.0% | 31.4% | 5.7% | 18.6% | 51.4 |
| Telecom | No-Defense | 73.3% | 77.1% | 31.4% | 54.3% | 19.0 |
| Telecom | Meta-Harness | 76.7% | 8.6% | 5.7% | 7.1% | 69.5 |
| Telecom | EvoSafeHarness | 80.0% | 2.9% | 2.9% | 2.9% | 77.1 |
| Mean | No-Defense | 71.1% | 71.4% | 39.0% | 55.2% | 15.9 |
| Mean | Meta-Harness | 81.1% | 32.4% | 20.0% | 26.2% | 54.9 |
| Mean | EvoSafeHarness | 77.8% | 14.3% | 6.7% | 10.5% | 67.3 |
Under the matched budget, EvoSafeHarness achieves lower pooled ASR and higher Score than Meta-Harness in all three domains. Mean ASR falls from 26.2% to 10.5%, and mean Score rises from 54.9 to 67.3. We will add this comparison and clarify the budget-matching protocol in the revised paper.
Shared harness. We searched one os-filesystem harness using pooled training feedback from three victims, using Claude Opus 4.8 as the outer model and the same maximum of ten harness-update rounds per search as the per-victim searches. The pooled training data combine the corresponding per-victim training sets. The shared harness was evaluated on 300 episodes (three victims × 100 tasks). Entries below are per-victim → shared, in percent; denominators are 30 benign, 35 direct and 35 indirect tasks per arm.
| Victim | Util ↑ | D-ASR ↓ | I-ASR ↓ | ASR ↓ | Score ↑ |
|---|---|---|---|---|---|
| Sonnet 4.6 | 80.0 → 60.0 | 0.0 → 2.9 | 0.0 → 5.7 | 0.0 → 4.3 | 80.0 → 55.7 |
| Kimi-K2.5 | 70.0 → 73.3 | 31.4 → 45.7 | 5.7 → 17.1 | 18.6 → 31.4 | 51.4 → 41.9 |
| GLM-5 | 80.0 → 70.0 | 17.1 → 14.3 | 5.7 → 8.6 | 11.4 → 11.4 | 68.6 → 58.6 |
Per-victim harnesses achieve higher Score on all three victims. On Sonnet, they provide both higher utility and lower overall ASR. On Kimi, the shared harness retains slightly higher utility (73.3% vs. 70.0%), but its ASR rises from 18.6% to 31.4%. On GLM-5, overall ASR is unchanged at 11.4%, while per-victim utility is 10 points higher (80.0% vs. 70.0%), yielding a Score of 68.6 vs. 58.6.
Reply to Reviewer BGdi — Part 1/4
We thank the reviewer for the thoughtful assessment and for recognizing the value of harness-based safety enforcement, the contribution of jointly optimizing policies and runtime controls, and the Criticizer’s role in addressing benchmark overfitting. We have carefully addressed the concerns below with additional experiments and clarifications on baseline comparisons, adaptive robustness, and evaluation protocols.
W1. Comparison with SafeHarness: empirical results and design differences
(a) Empirical results. We report results for Sonnet 4.6 and Kimi-K2.5 on os-filesystem, finance and telecom. Each cell uses 30 benign, 35 direct and 35 indirect tasks. Rates are percentages; Score = Util − overall ASR. The final two rows average the six deployments.
Across these six deployments, EvoSafeHarness achieves lower ASR and higher mean utility: mean ASR falls from 12.9% to 5.7% (a 55.6% relative reduction), while mean utility rises from 77.2% to 79.4%. SafeHarness’s mean ASR is approximately 2.25 times EvoSafeHarness’s. The improvement covers both direct attacks (19.0% vs. 7.6%) and indirect attacks (6.7% vs. 3.8%). EvoSafeHarness has lower overall ASR in all six cells and higher Score in five, raising mean Score from 64.4 to 73.7. The largest direct-ASR reductions occur on Kimi/filesystem (54.3% to 31.4%) and Kimi/telecom (37.1% to 2.9%); SafeHarness retains a higher Score on Sonnet/finance.
(b) How EvoSafeHarness differs from SafeHarness. SafeHarness is an expert-designed harness with a fixed defense architecture: its layers, rule tables and thresholds are specified in advance and applied unchanged across the evaluated models and domains. Its design is therefore static across deployments. EvoSafeHarness makes the policy and enforcement logic themselves the objects of optimization: before deployment, it uses the target model’s benign and adversarial trajectories to search natural-language policies and executable code, including which checks to invoke, how to enforce their decisions, and what trajectory state to maintain. The key distinction is how the defense is constructed: a fixed, manually specified design versus an automatically searched design tailored to the target deployment. We will revise the paper to make this distinction explicit in the method description and related-work discussion. For reproducibility, we will provide the baseline implementations and configurations.