- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether the benefits of structured LLM runtimes persist across changes in evaluation criteria. Motivated by “agent carcinization,” it demonstrates that different definitions of task success can lead to opposing conclusions about the value of runtime controls.
The main contribution is an audit of how evaluation choices affect comparisons between runtimes. By rescoring the same execution traces, the authors show that the apparent advantage of capability gating can reverse when evaluation focuses on business completion. However, this reversal is substantially attenuated when compliance is also required.
The authors analyze 200 replay-verified episodes across four models, four tasks, and five configurations. Compared with the schema placebo, gating increases historical success by 27.08 percentage points but decreases business completion by 29.86 percentage points. Requiring completion without recorded violations narrows the difference to −2.08 percentage points.
Critical strengths
The paper provides a valuable evaluation audit by demonstrating how rescoring the same execution traces can reverse conclusions about runtime benefits. This helps clarify what reported improvements actually measure.
Critical weaknesses
The reversal is demonstrated on small synthetic tasks using success criteria that lack independent validation. The external experiments neither reproduce the reversal nor sufficiently exercise the gate, leaving its generalizability unsupported.
Questions for authors
- Gating performs slightly worse than the placebo even when completion requires no recorded violations. Could this reflect limitations in the specific gate design used in the experiments?