- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents an evaluation framework for irrecoverable turning points in agent trajectories.
Critical strengths
- The paper studies poses important problem of distinguishing recoverable from irrecoverable failures, and also those recoverable under a fixed budget.
- The evaluation framework, such as the independent checker evaluation, is clean and nicely designed.
- Interesting finding that qwen3-4B is able to recover from recoverable failures but is not able to identify irrecoverable failures.
Critical weaknesses
- The paper is hard to follow because many terms and methods are used without proper definition, e.g., "certificate", "witness" etc. Please use consistent terminology and improve the flow of the paper.
- There aren't many useful findings, except the qwen behavior. Perhaps more smaller models need to be evaluated to show the benefits of the evaluation framework.
- It is unclear why model self-reported runtime is being compared. No one would plausibly use self-reports as resource certificates.
- The evaluation framework is useful but seems synthetic and does not analyze real agent trajectories. While reading the earlier sections, such as Equation 3, the paper gives the impression that the "turning point" from real agent trajectories would be evaluated but that is not the case.
- Only 24 task clusters are evaluated.
- Please highlight novelty compared to recent work around measuring irrecoverability in LLM agents, such as, [1].
[1] Ruan, K., Huang, Z., Zhou, Z., Wei, Q., Lin, J., Wang, X., & Sun, H. (2026). Doomed from the start: Early abort of llm agent episodes via a recall-controlled probe cascade. arXiv preprint arXiv:2607.06503.
Questions for authors
- How is the independent checker and certificate verification implemented?
- How were the irrecoverable environments created, i.e., what makes each c2 irrecoverable?
- Please explain the setup of experiments done in Section 5.4 clearly.