acceptodds
Under review as a conference paper at ICLR 2027

How Much Specification Does External Supervision of an Agent Need?

Abstract

Runtimes increasingly decide from outside the model whether a language model agent's task is done. We ask how much of a task's executable specification such supervision needs, which part, and where that part can come from. The need is real: every model we tested, production and frontier, declared unfinished business workflows complete in nearly every sampled failure, and no instruction repaired it; once “done” is the stop criterion, the report is a target, not a measurement. With a dose design that holds the agent, the tasks and the step budget fixed and varies only the share of success conditions the runtime holds, we find that external supervision is specification-limited: it is as reliable as the conditions it holds, and which conditions it holds matters more than how many. At the same budget, masks that cover the conditions the agent fails on its own pass more than twice as often as masks that miss one, in four models from four vendors; conditions chosen from another model's failures beat random ones, and a learned verifier in place of the conditions did not help in the configuration tested. Holding the conditions, even silently, removes false release; telling them is what gets tasks done. The conditions that matter carry values and prohibitions found in the data and policy documents, not in the request: checks a model writes from the request alone name the right events but not their bindings, and held online they help no more than random checks. A writer that first reads the data and documents recovers more of them exactly (18% against 7%) and releases falsely less often (11–21% of incomplete stops against 24–28%), at the cost of holding about half of complete tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, external supervision, runtime verification, task specification, false completion, agent evaluation, specification coverage, tool-use agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xnTkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at the problem of knowing when an agent task is actually done - determining the completion of an agentic is a nuanced problem. The paper focuses on a model that uses a set of assertions that must be satisfied to declare a task done. The paper runs experiments using AutomationBench, STATE-bench, and AppWorld. The paper’s experiment show that the inclusion of difficult “unmet” conditions (assertions that are not satisfied in a baseline run) largely determine if a task will be completed at a higher percentage rate. The results determined from having only a partial list of assertions is just an arithmetic exercise because it will eventually include the unmet conditions as you go from 0-100%. The paper further demonstrates that the runtime is a more effective gate than the agent, which cannot reliably judge its own completion, and that using a model to generate completion criteria is not as effective. Having a baseline run to classify unmet conditions is relatively successful in deterring which assertions are important.

Critical strengths
  • Current techniques like spec-driven development rely on human evaluators to judge completion, this paper takes a steps in better understanding how to automate a system could determine task completion.
  • Experimental data shows that determining the critical assertions using a baseline run may be effective
  • Paper experiments are well thought out, measurements seem sound
Critical weaknesses
  • Scope layer seems out of place in the description of the runtime as it does not have any conditions, bur rather serves as a baseline.
  • Paper assumes (for practical reasons) that completion assertions come from an outside source and leaves open how the system could help determine completion criteria
  • Paper may be difficult to read for ICLR audience; language is written is more like clinical trial.
Questions for authors

As the paper points out, the most difficult part is how to come up with a specification. If LLMs have not been effective at generating the assertions, are there any thoughts on how could this work be expanded to help here?

1 comment
AuthorsOpenReview ↗

Scope, Completion Checks, and Readability

Thank you for reviewing our paper and for asking how a system could establish its own completion checks. Your comments also helped us see where readers had to work too hard to follow the experiments. Since submission, we have continued this research and developed Self-Improving Runtime (SIR), which lets an agent revise its plan and checks as new evidence arrives and verify what it delivered before closing a task.

SIR tracks the plan, checks, and evidence on a dynamic blackboard. A process verifier checks the next action against the latest requirements. A completion verifier checks tool receipts, the resulting state, and what the user received. If work remains unfinished, SIR can diagnose the issue and resume from a checkpoint. An independent scorer determines whether the whole task passed.

In the AutomationBench historical comparison reported in our Official Comment, the baseline passed 33/318 tasks (10.4%) and the full SIR execution and recovery workflow passed 310/318 (97.5%), an 839.4% relative increase calculated from the pass counts. Both workflows were independently scored on the same 318 tasks and used DeepSeek-V4-Pro as the execution model.

1. What role does Scope play?

Scope tells the agent which services are available, where to find tools, and how to identify the objects it needs to work with. Every group in the coverage experiment received the same guidance. The Scope-only group received zero completion checks, providing a baseline for measuring what changes when checks are added. In the revision, we will introduce Scope-only in the experimental setup and then explain when the stop gate checks completion and when the action guard blocks a tool call.

2. How can the system build and revise completion checks without a ready-made specification?

SIR starts with explicit requirements from the user request, available documents, and applicable policies. It turns them into checks that can be revised as the agent sees what evidence the tools actually provide. Prior runs help determine what to check first. In Figure 4d, we kept the task, model, and number of checks fixed and replaced just one check for a repeatedly missed requirement. With DeepSeek-V4-Pro, strict passes fell from 42/62 to 24/62. SIR uses this result to prioritize checks for requirements that agents often miss.

In an AutomationBench 1099 summary task, the user asked the agent to email an accountant and exclude one supplier. The draft correctly excluded that supplier, but the sending component appended an internal W-9 reminder that put the name back into the email. The delivered email scored 8/9. After reviewing the run, we moved the check to the final email just before sending and kept the reminder on the internal blackboard. The rerun sent a corrected email and scored 9/9. This development example shows how reviewing an actual delivery can reveal what a check should inspect and when it should run. We made this revision during author review.

Appendix I shows how hard it is to generate reliable checks. Even with task documents, the generated checks held back 14 of 27 completed states in an archived-state evaluation. After submission, we ran six full tasks using previously generated checks. The request-only setup passed 3/6, and the setup with documents passed 4/6. Both the source material and the generation prompt differed. Our next test will hold the prompt, tools, and budget fixed on tasks held out from rule development. It will measure whether the system can turn available materials into usable checks and repair a check when execution evidence shows it is wrong.

3. How will we make the paper easier to read?

We spent time deciding how to describe the experiments so readers could verify what each group received, what stayed fixed, and how pass rates were counted. Your feedback helped us see that in several places we introduced those details before the question they were meant to answer.

We will use plainer language and start with a concrete task. The email example shows how a correct draft can still produce an incorrect delivery. From there, Figure 4d asks what happens when two groups run the same number of checks and one check for a common omission is replaced. The caption will say what changed, what stayed fixed, and how passes were counted. We will put the detailed group definitions and recomputation instructions in the experiments and appendix.

Reviewer_pQP2OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

AI agents often say they have finished a task when they actually have not. This paper asks how we can build an external checker to make sure the task is really done, without having to write down every detail of what a correct result should look like.

The paper's main contribution is a controlled evaluation of partial specifications. Experiments show that supervising conditions agents otherwise miss is substantially more effective than supervising arbitrary conditions at the same coverage, and that another model’s failures can help select useful checks. The paper further separates enforcement from corrective feedback and examines generating specifications from requests and environment data. Its central conclusion is that external supervision can greatly improve reliability, but its effectiveness depends on obtaining accurate task specifications, a problem that the tested automatic methods only partially solve.

Critical strengths
  1. The paper provides convincing evidence that which conditions a supervisor checks matters more than how many it checks. Figure 4d directly tests this by replacing one consistently unmet condition with an already-satisfied condition, reducing success from 68% to 39% for DeepSeek and from 56% to 22% for GLM. Figure 4c further shows that another model’s failures can guide useful selection, giving the finding practical relevance.
  2. The experiments distinguish the benefits of corrective feedback from those of enforcement. The silent-supervisor experiment in Section 5.6 preserves zero false release at full coverage but substantially reduces completion, while Tables 2 and 3 compare the same warning with and without action blocking. These controls clarify how supervision works, rather than attributing every improvement to the supervisor as a whole.
Critical weaknesses
  1. It is unclear whether the method works well without access to the benchmark’s answers. In Table 1, the supervisor receives exact success conditions, including required values, and tells the agent what remains unfinished. In practice, someone would first need to write those conditions. Appendix I tries generating them automatically, but the resulting checks often reject correctly completed tasks. Testing the full agent workflow with checks generated from the request and available documents would help show how much of the reported improvement is achievable in practice.
  2. The comparison with an LLM supervisor may disadvantage that baseline. Section 5.4 finds that an LLM checking the agent’s work performs poorly. However, Appendix P reports that its input was cut off in an estimated 42% of cases where it incorrectly approved completion. It may therefore have failed because relevant information was missing. Giving it enough context to inspect the work would make the comparison more convincing.
Questions for authors
  1. Could you address weakness 1?
  2. Could you address weakness 2?
  3. I suspect significant use of AI tools in presenting the results, including quite obvious patterns in table making. Can you justify that level of involvement?
5 comments
AuthorsOpenReview ↗

Completion Checks, Verifier Evidence, and AI Use

Thank you for reading our paper. Your review raised a practical question we have continued to work on since submission: how can an agent check its own work when the benchmark’s answers aren’t available? We developed Self-Improving Runtime (SIR) to let the agent keep track of its plan, checks, and evidence on a blackboard and revise them as the task unfolds. If a check shows there is still work to do, SIR can diagnose the issue and resume from a checkpoint. An independent scorer evaluates the task after the run.

1 — Building checks without benchmark answers

In Table 1, we gave the supervisor success conditions and target values from the benchmark. That let us study which checks were worth running. In Appendix I, we tried having a model write checks from the user’s request. We still used the benchmark’s assertion categories to decide what was in scope and past final states to identify the data fields. Even with task documents, the resulting checks held back 14 of 27 completed tasks when we tested them on archived states.

With SIR, the user’s request, available documents, and applicable policies tell the agent what to check. The agent links each check to a record it can look up. Earlier runs help it decide which requirements to check first. If the user changes a requirement or the records show that a check is looking at the wrong thing, the agent can update it during the task.

As reported in our Official Comment, we independently scored both workflows on the same 318 AutomationBench tasks. The historical baseline passed 33/318 (10.4%), and the full SIR execution and recovery workflow passed 310/318 (97.5%), an 839.4% relative increase based on the pass counts. Both used DeepSeek-V4-Pro as the execution model.

In one 1099 summary task, the user asked for an email to their accountant that left out a particular supplier. That gives us a simple check: does the email contain the supplier’s name? The draft didn’t, but the sending component added an internal W-9 reminder that put the name back in. The independent score was 8/9. When we reviewed the sent email, we moved the check to the final version just before sending and kept the reminder on the internal blackboard. The corrected email scored 9/9 on the rerun. The user’s request told us what to check. Seeing the email that actually went out told us when to check it.

2 — How the verifier gets its evidence

The verifier in Section 5.4 saw a length-limited summary of state changes. In SIR, the blackboard tracks which checks still need evidence, and the verifier can look up the relevant records with authorized read-only tools.

In the email example, the process verifier can check the entire message before it goes out. Once the agent says it is done, the completion verifier can use the message ID from the send receipt to look up the sent email and verify the recipient and body. If it still can’t confirm a requirement, that check stays pending and the task remains open.

3 — How we used AI

We used AI to help carry out experiments we designed, edit and translate text, and find and organize references. Page 10 of the submitted paper describes these uses and explains that the authors reviewed the AI-assisted work.

For the tables you asked about, we went through the preparation records table by table. AI helped prepare the tables. We designed them, and the numbers came from the experiments.

AuthorsOpenReview ↗

Correction to the Verifier Input Truncation Estimate

Thank you for drawing our attention to the verifier’s input limit. We rechecked the 42% estimate in Section 5.4 and Appendix P. Using the initial state saved during each run, we found that the 3,000-character limit would omit at least one full line from the reconstructed state-change summary in 11 of the 108 cases where the verifier let an unfinished task close. We will revise the estimate from 42% (45/108) to 10.2% (11/108) and explain how we reconstructed the summaries.

The estimate changed because we used a different starting point. The summary lists what changed between the start and end of a task. For the original estimate, we regenerated the starting states after the runs. If a regenerated state differs from the actual starting state, the comparison can count pre-existing content as a new change and make the summary longer. For the revised estimate, we used the original renderer with the starting state saved during each run.

The original results are unchanged: the verifier let 108 unfinished tasks close, and 32 of 144 evaluable tasks passed under the strict criterion. The revised estimate counts full lines omitted because of the total length limit.

Reviewer_pQP2OpenReview ↗

Response to rebuttal

I appreciate the additional work the authors put in during the rebuttal. With that said, I am still having some adverse feeling of the paper, as I have the impression that a significant portion of the paper (possibly including the research question itself) is vibe-coded. I am not questioning the validity or faithfulness of the experimental results, but rather I am questioning what value this paper provides to the community especially based on how it is written and presented. I would appreciate the authors' further elaboration on this, and potentially a revised abstract and other relevant text.

Reviewer_ixKgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper discusses the effects of a tasks' executable success specification. The supervisor has two parts. One is a stop gate, scoring the live world whenever the agent reports completion. Another one is pre-action guard, running each tool call on a copy of the world. The authors mainly tested it on automationbench. Strict pass goes 30.2% without conditions to 34/47/61% → 98.7%, while false release falls from 70% to zero.

Critical strengths

1.the design is rigorous. It considers different factors, and ablate the effects of different supervision mechanisms.

2.it shows negative results which are valuable too. The paper documents which methods have limited impact or do not work, it could help future researchers avoid dead ends.

Critical weaknesses

1.no code is available, which is not possible to reproduce for readers.

2.the LLM verifier is a deepseek-v4-flash model. Its input diff was truncated in 42% of false completions.

But overall, it's a rigorous study. I don't see major reasons to reject.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Reproduction Code and Evidence Retrieval in SIR

Thank you for reviewing our paper and for raising the question of what evidence the verifier can access. Since submission, we have continued this work and run additional experiments. We developed Self-Improving Runtime (SIR), which lets an agent revise its plan and checks as new evidence comes in and verify what it actually delivered before closing a task.

SIR builds checks from the user request, task documents, and applicable policies. It uses past execution traces to prioritize requirements the agent often misses. During execution, a dynamic blackboard tracks the current plan, checks, and evidence. The process verifier checks the proposed next action against the latest requirements. If a user changes which ticket to update, for example, it can ask the agent to reread the ticket list and update the target and checks. The completion verifier checks tool receipts, the resulting state, and what the user received. If work remains unfinished, SIR can diagnose the issue and resume from a saved checkpoint. After the run, an independent scorer determines whether the whole task passed based on the final state and the response delivered to the user.

In the AutomationBench comparison reported in our Official Comment, both workflows were independently scored on the same 318 tasks. The historical baseline passed 33/318 (10.4%), and SIR passed 310/318 (97.5%). Both used DeepSeek-V4-Pro as the execution model.

1. Where is the reproduction code?

We included the Supplementary Material with our initial submission. It contains the recomputation script and 5,051 archived run records. To reproduce the results covered by the archive, extract it and run python3 -I code/recompute.py from the supplementary/ directory. The script requires only the Python standard library. We reran it in a clean directory and reproduced all 111 values it checks. Supplementary Material from the initial submission

2. How does the verifier get the evidence it needs?

When a check lacks evidence, SIR’s completion verifier can query the underlying records using authorized read-only tools. For example, if the summary says “Email sent” and the task specifies a recipient, the verifier can look up the sent message and check the actual recipient. It records the evidence on the blackboard for the completion decision. If the evidence is still insufficient, the check stays pending.

The historical comparison above reports the performance of SIR as a whole. Measuring the effect of on-demand evidence retrieval requires a paired comparison with the same execution model, tool permissions, and budget.

AuthorsOpenReview ↗

Correction to the Verifier Input Truncation Estimate

Thank you for drawing our attention to the verifier’s input limit. We rechecked the 42% estimate in Section 5.4 and Appendix P. Using the initial state saved during each run, we found that the 3,000-character limit would omit at least one full line from the reconstructed state-change summary in 11 of the 108 cases where the verifier let an unfinished task close. We will revise the estimate from 42% (45/108) to 10.2% (11/108) and explain how we reconstructed the summaries.

The estimate changed because we used a different starting point. The summary lists what changed between the start and end of a task. For the original estimate, we regenerated the starting states after the runs. If a regenerated state differs from the actual starting state, the comparison can count pre-existing content as a new change and make the summary longer. For the revised estimate, we used the original renderer with the starting state saved during each run.

The original results are unchanged: the verifier let 108 unfinished tasks close, and 32 of 144 evaluable tasks passed under the strict criterion. The revised estimate counts full lines omitted because of the total length limit.

Reviewer_VvPpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how the availability and selection of executable success conditions affect external supervision of LLM agents. Using AutomationBench, the authors vary specification coverage and show that supervising conditions an agent previously failed is more effective than selecting conditions randomly. They further investigate cross-model transfer, learned verifiers, and the effects of feedback versus runtime enforcement.

Critical strengths
  1. The controlled specification-coverage experiments and matched condition-swap interventions are well designed.

  2. The cross-model transfer results are interesting: targeting conditions identified from another model's failures improves success by approximately 19 percentage points over random selection (Section 5.3).

  3. The ablations distinguishing completion verification, informative feedback, and action enforcement provide useful empirical insights

Critical weaknesses

1. Limited novelty and unclear practical contribution Execution feedback, test-based verification, and iterative correction have been extensively studied in prior work, including Self-Debugging [1], CodeT [2], and SWE-agent [3]. While the paper provides controlled empirical evidence, its central observation that verifying previously failed conditions is more effective than verifying randomly selected conditions is relatively intuitive. The authors should clarify what new actionable insight emerges beyond established verification-based approaches and how their findings meaningfully inform the design of agent systems.

2. Dependence on oracle specifications The main experiments assume access to ground-truth executable assertions and simulate incomplete supervision by withholding subsets. Even the targeted condition-selection experiments rely on failures identified using these oracle specifications, limiting their practical applicability. Although the appendix investigates automatically generated checks, the results remain limited. A more compelling direction would be to investigate how verification signals can be acquired and refined at inference time using execution traces, tool outputs, or dynamically generated tests, without relying on ground-truth specifications.

3. Restrictive LLM-verifier baseline Section 5.4 evaluates a DeepSeek Flash verifier using an environment diff capped at 3,000 characters, with individual values truncated to 80 characters. The verifier has no tool access to investigate the environment, and the experimental configuration also changes the action guard. These restrictions make the negative result difficult to interpret, particularly when the authors report that reconstructed diffs exceed the context cap in 42% of false-completion cases. A stronger comparison would evaluate an agentic verifier with access to relevant environment state and tools, ideally across different verifier capacities and inference budgets.

References

[1] Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching Large Language Models to Self-Debug. ICLR 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/2460396f2d0d421885997dd1612ac56b-Abstract-Conference.html

[2] Chen, B., Zhang, F., Nguyen, A., et al. CodeT: Code Generation with Generated Tests. ICLR 2023. https://openreview.net/forum?id=ktrw68Cmu9c

[3] Yang, J., Jimenez, C. E., Wettig, A., et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html

[4] Madaan, A., Tandon, N., Gupta, P., et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html

Questions for authors
  1. What is the main actionable contribution beyond existing execution-feedback and verification-based approaches [1–3]? In particular, what new design principle follows from the condition-selection experiments?

  2. How can the proposed condition-selection strategy operate when ground-truth specifications are unavailable? Could useful verification conditions be acquired or refined from execution trajectories or task-specific rubrics?

  3. How robust are the findings when verification conditions are noisy or incorrect rather than merely incomplete?

5 comments
AuthorsOpenReview ↗

Choosing, Updating, and Verifying Completion Checks

Thank you for reviewing our paper. Your comments helped us sharpen four questions about which checks to prioritize, how to create them without benchmark answers, what to do when a check is wrong, and how a verifier can get the evidence it needs. The study compares several supervision settings, so in the revision we will use plainer language and start with a concrete task before explaining the design and results of Figure 4d.

Since submission, we have continued this work and developed Self-Improving Runtime (SIR). The agent keeps its plan, checks, and evidence on a blackboard and updates them as the task unfolds. If the checks show that work remains, SIR can diagnose the issue and resume from a saved checkpoint. An independent scorer evaluates the whole task after the run.

1 — What new design guidance does the paper offer?

Self-Debugging, CodeT, and SWE-agent show how execution feedback, tests, and guards can help models correct errors. We wanted to measure a more specific choice: on the same task, with the same model and the same number of checks, how much does replacing one check change the outcome?

In Figure 4d, we used baseline runs to find requirements the model repeatedly missed. We replaced a check for one of those requirements with a check outside that set and kept everything else fixed. Whole-task passes fell from 42/62 to 24/62 for DeepSeek-V4-Pro and from 36/64 to 14/64 for GLM-5.1. This gives a practical way to choose what to check first. Run the agent on representative tasks, see what it repeatedly misses, and prioritize checks for those requirements. SIR uses this approach.

2 — Where do the checks come from without benchmark answers?

In the main experiments, the benchmark supplied the success conditions and target values. In Appendix I, we had a model write checks from the user request, using the benchmark’s assertion categories to define the scope and past final states to identify the data fields. Even with task documents, the generated checks held back 14 of 27 completed states in an archived evaluation.

In SIR, the user’s request, available documents, and applicable policies tell the agent what to check. Tool records provide the evidence, and past runs help the agent decide which requirements to check first. For example, if the user asks for a summary email to go to a particular accountant, the agent can turn that instruction into a recipient check and verify it against the send record. If the user changes the recipient, the check on the blackboard changes too.

As reported in our Official Comment, both workflows were independently scored on the same 318 AutomationBench tasks. The historical baseline passed 33/318 (10.4%), and the full SIR execution and recovery workflow passed 310/318 (97.5%), an 839.4% relative increase based on the pass counts. Both used DeepSeek-V4-Pro as the execution model.

3 — What if the check itself is wrong?

SIR can trace a check back to the requirement it came from and see which object and version it was checking. If new evidence shows that the check points to the wrong object or version, the agent can revise it on the blackboard. If the evidence is still insufficient, the check stays pending.

In an AutomationBench 1099 summary task, the user asked the agent to leave a supplier out of an email to an accountant. The draft left the supplier out, but the sending component appended an internal W-9 reminder that put the name back into the sent email. The independent score was 8/9. After reviewing the run, we moved the check to the entire final email just before sending and kept the reminder on the internal blackboard. The rerun sent the corrected email and scored 9/9. Our review changed which version the check inspected and when it ran.

We also tested six deliberately incorrect checks on fixed task states. Evidence contradicted two, one stayed pending for lack of evidence, and three yielded no usable result because of technical issues.

4 — How does SIR get evidence the original verifier could not query?

The DeepSeek-V4-Flash verifier in Section 5.4 received a length-limited summary of state changes, could not query the environment, and used a different action guard from the comparison group.

In SIR, the blackboard records what each check needs and which evidence is missing. The verifiers can look up the relevant records with authorized read-only tools. In the 1099 example, the process verifier can consult the task requirements and supplier records before sending, then check the entire outgoing email for the supplier that should be excluded.

When the agent reports completion, the completion verifier can use the message ID from the send receipt to retrieve the sent email and check its actual recipient and body. It looks up the specific evidence each check needs. If it still cannot confirm a requirement, the check stays pending and the task remains open.

AuthorsOpenReview ↗

Correction to the Verifier Input Truncation Estimate

Thank you for drawing our attention to the verifier’s input limit. We rechecked the 42% estimate in Section 5.4 and Appendix P. Using the initial state saved during each run, we found that the 3,000-character limit would omit at least one full line from the reconstructed state-change summary in 11 of the 108 cases where the verifier let an unfinished task close. We will revise the estimate from 42% (45/108) to 10.2% (11/108) and explain how we reconstructed the summaries.

The estimate changed because we used a different starting point. The summary lists what changed between the start and end of a task. For the original estimate, we regenerated the starting states after the runs. If a regenerated state differs from the actual starting state, the comparison can count pre-existing content as a new change and make the summary longer. For the revised estimate, we used the original renderer with the starting state saved during each run.

The original results are unchanged: the verifier let 108 unfinished tasks close, and 32 of 144 evaluable tasks passed under the strict criterion. The revised estimate counts full lines omitted because of the total length limit.

AuthorsOpenReview ↗

Matched Verifier Comparisons and Runtime Check Revision

Thank you for suggesting a fairer verifier comparison. We ran an additional online comparison following your suggestion and examined a case in which the model actually revised a check.

  1. How do the verifiers compare with the same evidence access and action guard? We reused the original 12 tasks, branching from the same checkpoint at each task’s first completion declaration. We held the execution model, action guard, and read-only evidence tools fixed. We first compared DeepSeek V4.1 Flash with GPT-6.1 Sol under the same nominal generation-token limits, then increased only Flash’s limits. We also retained a Scope-only baseline with the same action guard.

Among the 10 tasks jointly evaluable for Flash and Sol, each passed 2 whole tasks. In the separate set of 10 tasks jointly evaluable for standard-budget and higher-budget Flash, each also passed 2. Relative to their paired Scope-only results, none of the three verifier arms rescued an initially incomplete task, and each regressed on one initially successful task.

This adds the matched comparison missing from the submission. On these tasks and under the tested configurations, changing the verifier or increasing Flash’s budget did not improve whole-task completion. In the revision, we will report completion decisions, unresolved judgments, technical failures, and whole-task outcomes separately, and limit the conclusion of Section 5.4 to the configurations tested.

  1. Why did the original experiments use benchmark-provided checks? We first held check correctness fixed, then separately varied the number of checks, which checks were selected, and the feedback provided. This lets us distinguish the effects of checking more conditions, covering a critical omission, and giving the agent specific corrective feedback.

Figure 4d therefore asks which checks to prioritize when reliable checks are available and their number is fixed. How to derive checks from requests and documents, and how to revise them during execution, are questions we have continued to study.

  1. Has the model actually revised a check? Yes. In an access-request validation task, the model first created a check that a record’s status was Pending. It then revised the same check to require both Pending status and membership in the specified worksheet. Both versions appeared in subsequent model requests and were actually invoked. Independent counterexample tests confirmed that the revised check rejected processed records and records from the wrong worksheet, and returned unknown when the record was unavailable. This case shows the model refining a check during execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.