- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that refusal-based evaluations confound safety with inability to act. Using AgentHarm’s matched harmful/benign tasks, it proposes a capability-adjusted DiD that subtracts the benign refusal change from the harmful refusal change, together with conversion rates conditioned on demonstrated capability. Across 24 configurations, the average hosted-model effect decreases from −0.319 to −0.188 after adjustment; several estimates lose significance. The paper also studies where the interface effect occurs, whether harmfulness remains linearly decodable, and two diagnostic interventions.
Critical strengths
-
The paper identifies an important confound in agent-safety evaluation. The matched controls and capability-conditioned conversion rates are more informative than a simple comparison of refusal rates.
-
The evaluation includes broad model coverage, clustered inference, multiple-testing correction, judge validation, parser and cell-health audits, and several sensitivity analyses.
Critical weaknesses
The central interpretation depends on a strong parallel-response assumption: interface changes must affect harmful and benign twins similarly except through safety. This is not directly validated. Indeed, many benign refusals are themselves safety-framed, and the conservative sensitivity materially changes some estimates. The evidence supports the narrower claim that naive effects are overstated, but not necessarily that the residual identifies the interface’s safety cost.
The contribution is incremental. Capability confounding, matched benign controls, text/action divergence, and tool-interface effects are already established. The main novelty is their combination in one estimator, which is evaluated primarily on AgentHarm. The probing and intervention results are limited or mixed and do not establish a general mechanism.
Questions for authors
Can the authors provide stronger empirical support or placebo tests for the parallel-response assumption?
How do the headline results change if all records with conflicting reruns are excluded, rather than resolved using the last-write rule?