- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
SkillGovernor governs risky skills at their source before execution. A Risk Audit builds the skill’s dependency graph and reasons over it in a bounded cascade to produce a risk-connected view; a Dynamic Governor then applies graded operations (edit/rewrite/delete/alert) under dual verification. On 78 cases across five configurations, ASR drops 0.32→0.06 with task completion held, and injection-point detection rises 0.26→0.76.
Critical strengths
- Intervene-at-source framing. Rather than blocking actions at runtime or merely flagging, SkillGovernor removes risk-bearing content while preserving legitimate function. The least-intervention grading (edit vs rewrite vs delete) is a thoughtful middle ground that the delete-only and alert-only ablations justify.
- Strong, well-ablated audit. Graph structure plus bounded cascade lifts IDR to 0.76, clearly above SkillAudit (0.50) and a naive judge (0.26). The w/o-Cascade and w/o-Bound ablations show each mechanism contributes, and the contextual/obvious split is a genuinely informative cut.
- Honest, careful evaluation. A clean-skill control measures false positives (1.8%), Table 7 transparently lists both recovered and broken tasks, and the authors repeatedly flag that one case equals 1.3 points. Manual injection-point annotation and released prompts add credibility and reproducibility.
Critical weaknesses
- Margins sit at benchmark resolution. With 78 cases, one case moves rates 1.3 points. SkillGovernor’s 0.06 ASR versus ARGUS’s 0.09 is a 3-point gap — barely above noise — and the w/o-Scheduler ablation (0.06 vs 0.07) suggests one of the two core roles is nearly inconsequential.
- Governance both heals and harms. “Maintained TCR” masks Table 7: several done→failed cases are offset by failed→done ones. Net-zero is presented favorably, but it means source editing sometimes damages legitimate function, and the balance may not hold on other skill distributions.
- Coverage ceiling and residual risk. IDR of 0.76 means 24% of injection points are never surfaced, so a quarter of risk is never governed and still needs the runtime layer the paper doesn’t provide. Pre-execution governance is therefore a complement, not the standalone defense the framing suggests.
- Cost and model dependence. Audit, Scheduler, and Editor run on strong proprietary models (GPT-5.4, DeepSeek-V4), with median 150s and p90 917s per skill. Heavy reliance on frontier auxiliary models plus curated case selection (“most realistic”) raises cost and selection-bias concerns.
Questions for authors
see weakness. Moreover, I think there is a problem with the rendering of fig1.
2 comments
Thank you for the response, which addresses my concern. I have updated the rating to 3.
Thank you for recognizing the value of source-level intervention, the audit ablations, and the transparent reporting of task outcomes. We address the interpretation of the performance margins, individual task changes, residual risk, and deployment cost below.
W1 | Margins at benchmark resolution
We agree that the small ASR margins alone should not carry the argument. The two comparisons concern different contributions:
ARGUS. Our contribution is supported by the joint security-and-utility result. SkillGovernor and ARGUS achieved pooled ASRs of 5.6% and 8.7%, respectively. The 95% case-level interval for ARGUS minus SkillGovernor is [−0.8, +7.2] points, so we do not treat this as a definitive ASR ranking. Task completion separates the two more clearly: the average TCRs are 0.41 and 0.33, and the paired difference is 8.5 points higher for SkillGovernor, with a 95% case-level interval of [+4.6, +12.6]. In every configuration, SkillGovernor achieved the same or lower observed ASR and the same or higher observed TCR.
Scheduler. The ASR difference is small: 7% without the Scheduler and 6% with it. Its distinct function is to independently check the Editor's proposals and record the decisions. It accepted 96% of proposals and returned 8 nodes for revision, making these interventions traceable. The contribution of the full governance mechanism is demonstrated separately by the fixed-audit ablation: 6% ASR with the full Governor, compared with 11% for delete-only and 14% for alert-only. Thus, the evidence for governance does not rest on the Scheduler's one-point margin.
W2 | TCR changes in both directions
Our claim concerns aggregate task completion, and Table 8 explicitly reports both gains and losses. We agree that the average does not guarantee preservation of every task. Among the case-configuration pairs where governance blocked an otherwise successful attack, Table 8 records 9 done-to-failed transitions across 7 cases and 16 failed-to-done transitions across 8 cases.
The clean-skill control helps interpret these changes. Even among untouched cases, three tasks moved in each direction. Task outcomes can change between executions without any source edits, so a changed outcome alone cannot establish damage from editing.
The evidence supports maintaining aggregate task completion on the evaluated cases while substantially reducing ASR. We report the individual losses alongside the gains, and do not assume that their balance holds for every skill distribution.
W3 | Coverage ceiling
We agree that pre-execution governance is one layer, with runtime safeguards at a different intervention point. The audit missed 63 of 267 annotated injection points, which the Governor could not directly address. This 23.6% is a proportion of injection points, not an attack success rate. End-to-end evaluation separately measured an average ASR of approximately 6%, down from 32% without defense.
Runtime safeguards can additionally inspect attempted actions during execution. We evaluated pre-execution governance and runtime defenses separately, so the effectiveness of a combined pipeline has not been measured.
W4 | Cost, model dependence, and case selection
We agree that auxiliary-model cost and case selection affect deployment and generalization. Our evaluation establishes the feasibility of governing useful skills with the specified models and cases. The relevant costs and selection criteria are explicit:
Cost. The median governance time is 150.6 seconds per case and p90 is 917.4 seconds, over the 50 cases the undefended run attacked successfully. The full audit, graph construction plus judging, has a median of 226.9 seconds per case over the 78-case set. The work is performed offline, so its cost is amortized across repeated uses under unchanged skill content and task requirements.
Model dependence. We used GPT-5.4 for the audit and Scheduler based on preliminary safety checks, and DeepSeek-V4 for the Editor based on its task completion in the base runs. All compared methods used the same judge model. Whether smaller auxiliary models retain the same performance is not established.
Case selection. Appendix A requires each case to retain a meaningful task after injection removal and provide the assets needed to complete it. This criterion is necessary to evaluate functionality preservation. We also favored realistic workflows and reduced repetition among similar skill variants, yielding 52 SkillSafetyBench and 26 OpenSkillRisk cases. All methods were evaluated on the same curated set. Comparison holds within this setting, and generalization depends on the skill distribution.
Additional question | Figure 1 rendering
We have checked Figure 1 on multiple devices and with different viewers and have not reproduced the rendering issue. PDF and PNG versions are included in the supplementary materials.