- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces RegretBench, 205 executable tasks testing whether coding agents preserve state before modifications that users may later regret. Across six models, task completion substantially exceeds backup coverage. The proposed BackUpper intercepts tool calls and combines command rewriting, backup commands, snapshots, and a lightweight subagent informed by tool-specific backup experiences. It reports 100% judge-assessed backup coverage and 92.2% character-level similarity after restoration, alongside efficiency and personalization results.
Critical strengths
The benchmark addresses a meaningful gap in agent evaluation. Successful task execution can still destroy state that users subsequently need. Grounding tasks in documented losses and covering five tool categories makes this a useful evaluation target. Its focus on preservation before modification complements BackBench’s evaluation of recovery from already-harmful states.
The system is practical and the evaluation is reasonably broad. The study includes six models, a crossed model/harness comparison, human validation of 700 actions, executed restoration tests, and evaluation on external tasks. Separating backup execution from the main agent is sensible, and the reported improvements in coverage and backup overhead suggest practical value.
Critical weaknesses
Backup coverage and successful recovery remain insufficiently distinguished. The 100% coverage result is based on an LLM judge; humans confirm 188/200 sampled BackUpper-positive cases. The actual restoration experiment is valuable, but character-level similarity between serialized states does not establish whether each documented recovery need is satisfied. Differences involving object IDs or SQL sequence values can affect references or subsequent operations, even when most content is recovered. The acknowledged restoration limitations should therefore be reflected more carefully in the paper’s reversibility claims.
The source and novelty of the improvement need sharper isolation. Removing tool-specific experiences reduces BackUpper’s backup rate to 77.6%, below Skill-Based prompting at 81.4%. It is unclear whether Skill-Based receives the same experiences, leaving uncertainty about the contribution of the architecture versus additional recovery knowledge. The positioning should also address proactive systems more directly: YoloFS preserves filesystem state through staging and snapshots, while Cordon mediates local mutations and stages external effects before commit. BackUpper’s contribution should be distinguished through its lightweight backup mechanism and heterogeneous tool coverage.
Questions for authors
From the existing restoration runs, can you distinguish tasks where the documented recovery need is fully satisfied from those with incomplete recovery? How many human--judge disagreements concern harmless metadata rather than necessary state?
Does Skill-Based receive the same tool-specific backup experiences as BackUpper? Please clarify its prompt and available recovery information.
What are the task pass rates with BackUpper on the existing 205-task RegretBench runs? The reported no-degradation comparison currently covers 40 external tasks.
4 comments
Dear Reviewer fYEC,
Thank you again for your detailed review. We have responded to each of your concerns above, including functional recovery tests, the comparison with Skill-Based under the same experiences, and a discussion of YoloFS and Cordon. As the discussion period is coming to an end, we would like to check whether anything remains unclear that we can address. If your concerns have been addressed, we would appreciate it if you could consider updating your assessment.
Thank you for your time.
I appreciate the time spent by the authors in preparing the rebuttal and addressing the concerns raised. I do not have further questions, my issues have been resolved. I will retain my original positive score of 3 for this work.
Thanks
Thank you for the follow-up and for your positive assessment. We will include all the additional experiments and clarifications in the next version of our manuscript.
Response to Reviewer fYEC
We thank Reviewer fYEC for the thoughtful review and for recognizing the practical value of our benchmark and BackUpper.
W1/Q1: Judge-Based Evaluation
We will revise the claim to "100% judge-assessed coverage" and distinguish coverage from recovery as follows.
Coverage. To quantify the human–judge disagreements, we expanded the human evaluation to 300 judge-positive BackUpper actions. Annotators agree with the judge on 95.0% (285/300), and all 15 disagreements concern only additional metadata, such as read/unread flags, rather than necessary state (General Response W3). We then re-read all 205 regret-moment reports and found that no documented recovery need involves such metadata.
Recovery. To test directly whether each documented recovery need is satisfied, we constructed 468 functional tests on the 60-task subset, with at least one test per need (General Response W2). Restoring from BackUpper's backups passes 94.7% of the tests. None of the failures is caused by missing metadata: all come from state that the service itself cannot reproduce, such as a shared link that changes once a document is re-uploaded.
Together, these results demonstrate the quality of BackUpper's backups: the disagreements in coverage concern only metadata that no recovery need involves, and the failures in functional recovery come only from state that the service itself cannot reproduce.
W2/Q2: Skill-Based with Tool Experiences and Missing Related Work
In our main experiment, the Skill-Based prompt only instructs the main agent to back up every out-of-workspace action before executing it. To compare under equivalent information, we additionally give Skill-Based the same experiences as BackUpper's subagent and run it on Claude Opus-5 and GPT 5.6-Sol over the 60-task subset:
The experiences raise Skill-Based from 90.0% to 95.8% RMBR. Yet with equivalent information, BackUpper still achieves higher coverage (+4.2 RMBR, +6.0 OWBR) and a higher functional pass rate (94.7% vs. 91.5%), at 22× lower cost. BackUpper’s advantage therefore also comes from its architecture: each backup subagent applies the experiences in a fresh context rather than amid the main agent’s long task context. Experiment details can be found in General Response W1 and W2.
Related work. We thank the reviewer for pointing us to YoloFS [1] and Cordon [2], and we will discuss both in the related work. Both methods hold an agent's action until a check at commit time: YoloFS stages filesystem mutations for user review, and Cordon stages local mutations and external effects for policy validation. Such checks catch problems that are recognizable at that moment. However, regret moments are different: the user often requested the action or would approve it at commit time, and the regret comes only later when they still need something back. BackUpper therefore does not rely on a check at commit time. It lets the action run and preserves the affected state beforehand, so the state remains recoverable afterwards. BackUpper is also lightweight, requiring no kernel module or transactional runtime, and it covers state beyond the filesystem, such as cloud documents and databases. We see the approaches as complementary: YoloFS and Cordon block actions that should not happen, while BackUpper keeps the actions that do happen recoverable.
Q3: Task Pass Rate with BackUpper
We evaluate the pass rate on all 205 RegretBench tasks with and without BackUpper:
The average pass rate is 86.0% with BackUpper and 86.1% without it, with small per-model differences in both directions. We manually inspected every task whose outcome changed and found none caused by BackUpper. This further confirms that BackUpper improves recoverability without degrading task performance. Experiment details can be found in General Response W4.
[1] Don’t Let AI Agents YOLO Your Files: Shifting Information and Control to Filesystems for Agent Safety and Autonomy, arXiv, 2026
[2] Cordon: Semantic Transactions for Tool-Using LLM Agents, arXiv, 2026