- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Existing agent-safety methods typically rely on static guards or harnesses, or on self-evolution with repeatedly accessible fixed datasets, whereas deployment more closely resembles continual test-time adaptation in which tasks arrive sequentially and only past trajectories and feedback are available. The authors address this setting with a dual-timescale Harness–Guard co-evolution framework: S-Harness updates prompts, validated memory, safety skills, permission policies and guard rules after each task, while GuardVPO periodically internalizes accumulated experience into the Guard through event-level safety feedback, asymmetric rewards, PPO-style clipping and KL regularization, with the task-execution model kept frozen.
Critical strengths
- Interesting dual-timescale design. GuardVPO addresses an important dimension of agent safety, while SafeCoEvo cleanly separates short-term adaptation through S-Harness from long-term parametric consolidation through GuardVPO.
- Practical continual test-time formulation. The paper studies a realistic deployment setting in which tasks arrive sequentially and adaptation can use only trajectories and feedback from previously observed tasks.
- Strong generalization. The method maintains strong performance on the held-out test set and yields consistent improvements when either the task-performing model or the Guard backbone is replaced.
Critical weaknesses
- Limited evaluation on frontier models. The evaluation is restricted to relatively small or mid-scale models. It would be valuable to assess whether SafeCoEvo remains effective on stronger frontier models, such as GPT-5.5 and Opus 5.5.
- Limited baseline coverage. The comparison set is relatively narrow. The authors should include one or two additional competitive agent-safety baselines.
- Missing AgentDojo comparison. Given its relevance to agent safety and prompt-injection robustness, AgentDojo should be included as either an evaluation benchmark or comparison point.
- Notation and presentation. A dedicated notation table would improve readability and make the framework easier to follow, particularly given the interaction between S-Harness, GuardVPO, memory, and event-level feedback.
- Statistical robustness. The paper should report standard deviations or confidence intervals across multiple runs to better establish the reproducibility and robustness of the reported gains.
- Reliance on reliable risk feedback. GuardVPO depends on event-level risk feedback, an assumption that may be difficult to satisfy in realistic deployments where safety feedback can be delayed, incomplete, noisy, or entirely unavailable.
- Long-term memory stability. Because S-Harness continually accumulates explicit safety knowledge, redundancy and conflicts may increase over time. The current evaluation provides limited analysis of memory growth, rule conflicts, catastrophic adaptation, or performance degradation over substantially longer task streams.
Questions for authors
- Table formatting. Tables should follow standard LaTeX conventions, using
\topruleand\bottomrulefor cleaner presentation. - Equation formatting. Unnumbered equations that are not referenced in the text should use
\begin{align*}...\end{align*}rather than numbered environments. - Lack of reproducibility and ethical considerations
11 comments
Response to Weaknesses 2 and 3: Additional Baseline Comparisons and Cross-Benchmark Evaluation
Thank you for these valuable suggestions. To broaden the baseline coverage, we introduce ShieldAgent as an additional Guard baseline and evaluate the contributions of S-Harness evolution and GuardVPO with this Guard, using Qwen3.7-Flash as the task-performing model. We also expand the external evaluation on AgentDojo and add results on AgentDyn.
Additional Guard baseline. Table A compares SHE, our static configuration with ShieldAgent, and online S-Harness evolution with ShieldAgent. “+ ShieldAgent” denotes replacing the original AgentDoG 1.5 Guard with ShieldAgent. In the two ShieldAgent configurations, the Guard parameters remain fixed, and GuardVPO is disabled. Compared with the static configuration, S-Harness evolution reduces UOR by 9.70 percentage points and increases TSR and SUCR by 12.08 and 11.68 percentage points, respectively. Compared with SHE, it reduces UOR by 13.15 percentage points and increases TSR and SUCR by 18.17 and 13.42 percentage points, respectively. These results supplement the baseline comparisons and support the effectiveness of S-Harness evolution with an additional Guard model.
Table A. Performance on the continual evolution stream using Qwen3.7-Flash as the task-performing model.
| Method | UOR ↓ | TSR ↑ | SUCR ↑ |
|---|---|---|---|
| SHE | 39.88% | 22.82% | 20.83% |
| SafeCoEvo (Static) + ShieldAgent | 36.44% | 28.91% | 22.57% |
| SafeCoEvo w/o GuardVPO + ShieldAgent | 26.73% | 40.99% | 34.26% |
We further evaluate the contributions of S-Harness evolution and GuardVPO with ShieldAgent on the held-out test set. As shown in Table B, replacing the initial S-Harness with its evolved version reduces UOR by 10.67 percentage points and increases TSR and SUCR by 12.33 and 10.67 percentage points, respectively. With the evolved S-Harness held fixed for this controlled comparison, updating ShieldAgent through GuardVPO further reduces UOR by 5.00 percentage points and increases TSR and SUCR by 3.67 and 5.00 percentage points, respectively. These results support GuardVPO’s additional contribution beyond S-Harness evolution and show that its benefits are not limited to the original Guard model.
Table B. Held-out performance with ShieldAgent before and after S-Harness evolution and GuardVPO.
| Setting | UOR ↓ | TSR ↑ | SUCR ↑ |
|---|---|---|---|
| 32.33% | 22.67% | 16.33% | |
| 21.67% | 35.00% | 27.00% | |
| 16.67% | 38.67% | 32.00% |
Here, and denote the initial S-Harness and the version obtained after 512 episodes, respectively. and denote ShieldAgent before and after the GuardVPO update.
AgentDojo and AgentDyn evaluation. Table 11 in Appendix C of the original manuscript already reports results on a subset of AgentDojo. We expand this evaluation to 249 cases across the Slack and Banking scenarios and additionally evaluate 200 cases from the DailyLife scenario in AgentDyn. In both benchmarks, cases are constructed by combining selected user tasks with injection tasks. We use Qwen3.7-Flash as the task-performing model and AgentDoG 1.5 as the Guard backbone. SafeCoEvo is evaluated after evolution on the original adaptation stream, with both S-Harness and the Guard parameters frozen throughout the cross-benchmark evaluation. No interaction trajectories or feedback from these external benchmarks are used for updates. The results are presented in Table C.
Table C. Cross-benchmark evaluation on AgentDojo and AgentDyn with Qwen3.7-Flash.
| Benchmark | Configuration | UOR ↓ | TSR ↑ | SUCR ↑ |
|---|---|---|---|---|
| AgentDojo | Qwen3.7-Flash | 45.38% | 75.50% | 43.37% |
| AgentDojo | SHE | 6.83% | 75.10% | 69.48% |
| AgentDojo | SafeCoEvo | 0.00% | 75.50% | 75.50% |
| AgentDyn | Qwen3.7-Flash | 24.00% | 74.00% | 56.50% |
| AgentDyn | SHE | 4.00% | 77.00% | 73.50% |
| AgentDyn | SafeCoEvo | 0.00% | 76.00% | 76.00% |
As shown in Table C, no unsafe outcomes are observed among SafeCoEvo’s valid outputs on either benchmark. Compared with SHE, SafeCoEvo reduces UOR by 6.83 percentage points on AgentDojo and 4.00 percentage points on AgentDyn. On AgentDojo, TSR increases by 0.40 percentage points and SUCR increases by 6.02 percentage points. On AgentDyn, TSR decreases by 1.00 percentage point, while SUCR increases by 2.50 percentage points. These results support improved safe task completion in the evaluated external prompt-injection scenarios, with task success rates close to those of SHE.
We will include these additional comparisons, results, and experimental protocols in the revised manuscript and appendix.
Response to Weakness 4 and Presentation Comments: Notation, Formatting, Reproducibility, and Ethics
Thank you for your suggestions regarding notation, formatting, reproducibility, and ethical considerations. We have made the following changes in the revised manuscript.
-
Notation. We have added Table 1, “Notation for SafeCoEvo,” to Section 3.1 (Overview). It summarizes the notation for tasks, execution trajectories, episode-level feedback, accumulated runtime experience, S-Harness states, Guard states, and event-level safety experience. The table also clarifies that Validated Memory is part of the S-Harness state and lists the event inputs, verdicts, and rewards used by GuardVPO. This provides a reference for the task-by-task updates in Section 3.2 and the event-level optimization in Section 3.3.
-
Table formatting. We have standardized the experimental results tables in Section 4 and Appendices A–D using the three-rule style provided by the LaTeX
booktabspackage. Specifically, we use\toprule,\midrule, and\bottomruleat the top of each table, below the header, and at the bottom, respectively. Metric groups within multilevel headers are marked using\cmidrule. -
Equation formatting. Following the reviewer’s suggestion, we have changed the displayed equations in Sections 2–3 that require no numbering and are not referenced in the text to use
\begin{align*}...\end{align*}. -
Reproducibility and ethical considerations. We have added a Reproducibility Statement and an Ethics Statement after the AI Use Statement and before the references.
The Reproducibility Statement points to the S-Harness update procedure and GuardVPO optimization objective in Sections 3.2–3.3, the data splits and evaluation protocol in Section 4.1, the GuardVPO optimization curves and validation results in Appendix I, and the Target Agent and Evolution Agent prompt templates in Appendix L.
The Ethics Statement explains that the experiments are conducted in benchmark environments and that SafeCoEvo is intended to provide runtime safety protection for tool-using language model agents. It also discusses the uses of prompt-injection examples and clarifies that the experimental results are reported and interpreted within the scope of the benchmark tasks used.
Response to Weakness 1: Effectiveness with a Frontier Task-Performing Model
Thank you for this valuable suggestion. To further evaluate the effectiveness of SafeCoEvo on frontier models, we conduct additional experiments using GPT-6.1-terra as the task-performing model, comparing SafeCoEvo, SafeCoEvo (Static), and SHE over 512 episodes.
As shown in Table A, compared with SafeCoEvo (Static), online evolution reduces UOR by 9.18 percentage points while increasing TSR and SUCR by 0.39 and 0.20 percentage points, respectively. Compared with SHE, SafeCoEvo reduces UOR by 14.65 percentage points and increases TSR and SUCR by 6.25 and 4.88 percentage points, respectively. Together with the results for DeepSeek-v4.1-Flash and Qwen3.7-Flash reported in the main text, these findings show that SafeCoEvo’s benefits extend across different task-performing models and that its external safety adaptation mechanism does not depend on a particular backbone. We will include these additional results and experimental details in the revised manuscript and appendix.
Table A. Performance comparison with GPT-6.1-terra as the task-performing model.