- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that set agreement, the frequency with which independent runs select the same best recovery action at a failed agent checkpoint, can be high even when recovery rarely succeeds, because universal failure produces ties. It proves a set-path symmetry for two equal-cost Bernoulli actions, together with results on certifying ties and best actions, and evaluates agreement alongside held-out success on frozen RecoveryBench episodes, two planning cohorts with three GPT-5.6 models, and a Qwen3-8B panel. It recommends reporting agreement together with the all-zero fraction, held-out success and pooled success.
Critical strengths
- The paper raises a practical caution that is worth making: stability numbers such as set agreement are easy to over-read on hard tasks where most attempts fail, and the two-action coupling in Section 4.1 illustrates this concisely.
Critical weaknesses
- I noticed that the main text appears to run beyond the ICLR 2027 page limit. The Author Guidelines state that at submission "the main text should be 9 pages or fewer" and that "Papers with main text beyond the page limit will be desk-rejected", and here the end of the Limitations paragraph in Section 6 continues onto page 10 (lines 486-489). As I understand the guidelines, only the references, appendices and the reproducibility, ethics and AI-use statements are exempt, so I believe this would fall under the desk-rejection policy. I also had some concerns about the scope of the theory and the interpretation of the experiments, but since this issue appears decisive, I have kept my review focused on it.
Questions for authors
- [Clarification] I may have misjudged where the main text ends, so I would be grateful if the authors could confirm whether the text on page 10 was intended to fall within the 9-page limit.