- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes the EP–Path–EF framework, which connects nine risk entry categories to five technical effect categories through agent-mediated risk paths, together with an automated workflow that iteratively constructs each entry-effect pair into executable cases, yielding a frozen set of 450 tasks across six scenarios. The authors evaluate nine model-harness configurations spanning three models and three harnesses and find that ASR varies more across models than across harnesses, with harness effects depending on the model.
Critical strengths
- The risk representation covers the full span from entry point to effect, with nine risk entry categories, agent-mediated risk paths, and five technical effects forming an operational taxonomy. A human expert study supports the interpretability and categorical consistency of these categories on representative scenarios.
- Case construction and outcome verification are integrated into an end-to-end automated pipeline, which has practical potential for extending the case set as models and harnesses evolve.
Critical weaknesses
- Case construction and test cases are generated entirely by LLMs. Quality control rests on the generator's own rules and replay-based trigger verification, with no human review and no validated LLM judge, so the data quality lacks a credible basis.
- The evolving claim in the title lacks empirical support. The authors only state that the workflow can extend the case set and provide no analysis of the evolution process or its results, so whether evolution actually occurs and what it changes cannot be verified.
Questions for authors
- What does the case generator take as input, such as seed environments, hand-written examples, or existing benchmarks, and how is diversity ensured within each EP–EF cell?
- Claude Opus 5's pass rate is 73.33 on Claude Code and 26.89 and 30.00 on Codex and OpenClaw, while the cross-harness gap for the other two models is only 9 to 14 points. Could the authors explain what causes this gap and whether it also affects the reported ASR?