- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper looks at the problem of knowing when an agent task is actually done - determining the completion of an agentic is a nuanced problem. The paper focuses on a model that uses a set of assertions that must be satisfied to declare a task done. The paper runs experiments using AutomationBench, STATE-bench, and AppWorld. The paper’s experiment show that the inclusion of difficult “unmet” conditions (assertions that are not satisfied in a baseline run) largely determine if a task will be completed at a higher percentage rate. The results determined from having only a partial list of assertions is just an arithmetic exercise because it will eventually include the unmet conditions as you go from 0-100%. The paper further demonstrates that the runtime is a more effective gate than the agent, which cannot reliably judge its own completion, and that using a model to generate completion criteria is not as effective. Having a baseline run to classify unmet conditions is relatively successful in deterring which assertions are important.
Critical strengths
- Current techniques like spec-driven development rely on human evaluators to judge completion, this paper takes a steps in better understanding how to automate a system could determine task completion.
- Experimental data shows that determining the critical assertions using a baseline run may be effective
- Paper experiments are well thought out, measurements seem sound
Critical weaknesses
- Scope layer seems out of place in the description of the runtime as it does not have any conditions, bur rather serves as a baseline.
- Paper assumes (for practical reasons) that completion assertions come from an outside source and leaves open how the system could help determine completion criteria
- Paper may be difficult to read for ICLR audience; language is written is more like clinical trial.
Questions for authors
As the paper points out, the most difficult part is how to come up with a specification. If LLMs have not been effective at generating the assertions, are there any thoughts on how could this work be expanded to help here?
Scope, Completion Checks, and Readability
Thank you for reviewing our paper and for asking how a system could establish its own completion checks. Your comments also helped us see where readers had to work too hard to follow the experiments. Since submission, we have continued this research and developed Self-Improving Runtime (SIR), which lets an agent revise its plan and checks as new evidence arrives and verify what it delivered before closing a task.
SIR tracks the plan, checks, and evidence on a dynamic blackboard. A process verifier checks the next action against the latest requirements. A completion verifier checks tool receipts, the resulting state, and what the user received. If work remains unfinished, SIR can diagnose the issue and resume from a checkpoint. An independent scorer determines whether the whole task passed.
In the AutomationBench historical comparison reported in our Official Comment, the baseline passed 33/318 tasks (10.4%) and the full SIR execution and recovery workflow passed 310/318 (97.5%), an 839.4% relative increase calculated from the pass counts. Both workflows were independently scored on the same 318 tasks and used DeepSeek-V4-Pro as the execution model.
1. What role does Scope play?
Scope tells the agent which services are available, where to find tools, and how to identify the objects it needs to work with. Every group in the coverage experiment received the same guidance. The Scope-only group received zero completion checks, providing a baseline for measuring what changes when checks are added. In the revision, we will introduce Scope-only in the experimental setup and then explain when the stop gate checks completion and when the action guard blocks a tool call.
2. How can the system build and revise completion checks without a ready-made specification?
SIR starts with explicit requirements from the user request, available documents, and applicable policies. It turns them into checks that can be revised as the agent sees what evidence the tools actually provide. Prior runs help determine what to check first. In Figure 4d, we kept the task, model, and number of checks fixed and replaced just one check for a repeatedly missed requirement. With DeepSeek-V4-Pro, strict passes fell from 42/62 to 24/62. SIR uses this result to prioritize checks for requirements that agents often miss.
In an AutomationBench 1099 summary task, the user asked the agent to email an accountant and exclude one supplier. The draft correctly excluded that supplier, but the sending component appended an internal W-9 reminder that put the name back into the email. The delivered email scored 8/9. After reviewing the run, we moved the check to the final email just before sending and kept the reminder on the internal blackboard. The rerun sent a corrected email and scored 9/9. This development example shows how reviewing an actual delivery can reveal what a check should inspect and when it should run. We made this revision during author review.
Appendix I shows how hard it is to generate reliable checks. Even with task documents, the generated checks held back 14 of 27 completed states in an archived-state evaluation. After submission, we ran six full tasks using previously generated checks. The request-only setup passed 3/6, and the setup with documents passed 4/6. Both the source material and the generation prompt differed. Our next test will hold the prompt, tools, and budget fixed on tasks held out from rule development. It will measure whether the system can turn available materials into usable checks and repair a check when execution evidence shows it is wrong.
3. How will we make the paper easier to read?
We spent time deciding how to describe the experiments so readers could verify what each group received, what stayed fixed, and how pass rates were counted. Your feedback helped us see that in several places we introduced those details before the question they were meant to answer.
We will use plainer language and start with a concrete task. The email example shows how a correct draft can still produce an incorrect delivery. From there, Figure 4d asks what happens when two groups run the same number of checks and one check for a common omission is replaced. The caption will say what changed, what stayed fixed, and how passes were counted. We will put the detailed group definitions and recomputation instructions in the experiments and appendix.