acceptodds
Under review as a conference paper at ICLR 2027

Success Is Not Completion: Goal-Closure Monitoring and Corroborated Contracts for Tool-Using Agents

Abstract

An API can report success even when an agent has not achieved the user’s goal. Repeating a check does not resolve this ambiguity if the check shares the original failure cause. We study post-action goal-closure identification: when does the available evidence justify declaring a task complete? We introduce GoalSeal-CG, a training-free monitor that represents ways the goal may still be open, selects observations that distinguish them from completion, and admits only recovery actions that are safe across the re- maining alternatives. The resulting rule is sound relative to an explicit goal contract and fault model; indistinguishable omitted worlds remain an impossibility boundary. We introduce GoalCommitBench to test safety and useful completion jointly under tool, evidence, contract, and recovery faults. Across 3,456 episodes spanning 1,152 matched conditions from nine model configurations, GoalSeal-CG has zero observed dangerous declara- tions at 70.6% declaration coverage, versus 9.1% danger for prompting. On held-out ToolBench-X workflows it reduces faulty danger from 33.9% to 1.7%. Four of six preregistered criteria pass; the two that do not are clean- execution autonomy gains, where the safety-first gate trades added coverage for zero danger rather than missing recoverable closure. Matched mecha- nism and stateful cross-node studies attribute the safety gain to closure admission and show successful recovery or containment outside a deliber- ately indistinguishable common-mode control. QuorumContract recovers required-clause recall to 100% at zero added danger, but automatic contract exactness remains a bottleneck we isolate rather than mask. A prospective 45-task supportive retest of one targeted gate refinement then reaches 0% faulty danger while improving clean safe closure by 26.7 points and passes six of six pre-specified decision criteria across all three model families. The evidence therefore supports model-relative safety and recovery, not a pro- duction or end-to-end goal-understanding guarantee

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
tool-using agents, goal-closure monitoring, failure recovery, verification, agent benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LSAdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a post action verification formalization and a greedy algorithm to select probes and utilize evidences to determine whether a goal is actually met when an agent declares success. The authors propose GoalSeal-CG which is a greedy method for selecting counter factuals that selects counterfactuals with the least cost of probe set first and reviews evidences to see if a goal is met. If all counter factuals are exhausted without gola closure condition violation then the agent success is ascertained. Else a danger or failure is reported.

Critical strengths

I really liked the theoretical formulation of the problem. It sets up a very useful framework where properties of agents can be proven.

It addresses an important problem about ascertaining goal success for an agent when probes and evidences are uncertain.

THe counterexample guided recovery algorithm is sound greedy approach with bounded cost. Although it is NP hard but for practical cases the combinatorial problem is not compute heavy and hence can be solved through enumeration.

The GoalCommitBench is especially useful contribution it has explicit construction of ambiguity. For example successful and unsuccessful worlds share the same initial transcript, registered probes distinguish them, and recovery actions can introduce additional harm.

Critical weaknesses

While the theoretical exposition is interesting and useful and makes the problem precise, it is not a central novelty. The problem has been discussed by many papers as highlighted in the related work section. Also Propositions 1 is by definition. Proposition 2 is known apriori for greedy approach. Proposition 3 is by construction. Proposition 4 is also by definition. Isnt completion the rule the only rule available and should be satisfied? Then it has to be the most permissive because it is the only rule. So I would ask the authors exactly what are the claimed novelty of the paper?

If the probe has a deterministic outcome, then is your problem definition same as verifier grounded replanning?

Also from my reading of the paper it seems like your formulation is only non-trivial when the verifier is wrong or incomplete. So if a verifier is complete then there is only a single trivial counter example. Is my reading correct?

Questions for authors

Look at weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LSAd: Method and Benchmark Contributions, Deterministic Probes, and Complete Verifiers

Thank you for recognizing the value of the formalization and GoalCommitBench's explicit construction of ambiguity. We address your three concerns below.

Response to W1: Methodological novelty and benchmark contribution

Among the methods compared in this paper, GoalSeal is the first to explicitly incorporate common-cause evidence failures into compatible execution states and use unresolved failure states to drive evidence acquisition, safe recovery, and completion decisions. It prioritizes queries that distinguish completed from incomplete states, admits only recovery actions satisfying the harm budget in every compatible state, and declares completion after excluding all compatible incomplete states. After recovery changes external state, the relevant old evidence is invalidated and completion conditions are checked again.

The key innovation is to make unresolved failure states actionable: they determine which evidence to acquire, which corrective actions are safe, and when there is sufficient justification to finish. The four propositions specify the method's identifiability, query-cost, and sound-admission conditions.

GoalCommitBench explicitly constructs tasks with identical initial transcripts but different true completion states. Distinguishing probes, recovery actions with different side effects, and controlled faults separately test whether a method can identify incomplete goals, avoid false declarations, and recover safely. Actual completion, dangerous false success, and realized harm are measured separately, distinguishing genuine task completion from merely withholding success declarations.

In the nine-configuration matched evaluation, GoalSeal-CG achieves 85.0% actual completion, exceeding VTC and RAC by 2.4 and 4.3 percentage points, respectively. Its observed dangerous false success rate is 0.0%, compared with 5.8% and 6.2%. These results demonstrate its ability to improve actual completion while controlling erroneous declarations in the evaluated conditions. VTC and RAC refer to the benchmark implementations of their verification and compensation mechanisms, with adaptation scope documented in the appendix.

Response to W2: Do deterministic probes reduce the problem to verifier-grounded replanning?

Determinism does not imply that an observation distinguishes the relevant states. For example, both “replacement ticket issued” and “replacement ticket not issued” can produce the same cached response. GoalSeal selects evidence that distinguishes the remaining states and checks whether a retry could cause duplicate charging in any of them.

When a verifier reliably determines whole-task completion and supplies the information needed for safe replanning, verifier-grounded replanning can realize this special case. GoalSeal addresses situations in which observation ambiguity remains and recovery must be constrained across compatible execution states.

Response to W3: Does a complete verifier leave only one trivial counterexample?

Thank you for this careful reading. Your interpretation is close; the key distinction is between completion assessment and recovery decisions.

If a verifier soundly and completely determines whether the entire user goal has been achieved, completion can indeed be assessed directly from its output, without further resolving completion ambiguity. This is a simplified special case of the framework.

However, an “incomplete” verdict does not imply a unique failure state. A refund that has not been initiated, is still being processed, or was sent to the wrong account may receive the same verdict but require different corrective actions. Even with an accurate completion verdict, distinguishing these states can therefore remain necessary for choosing safe recovery.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Reviewer_hZYwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies goal completion verification for tool-using agents. It proposes a training-free monitoring framework that represents possible goal-incomplete worlds, selects observations to distinguish them from completed states, and restricts recovery actions using worst-case harm constraints. The paper additionally introduces a synthetic benchmark for evaluating false success declarations and recovery under injected faults, along with automatically generated goal specifications.

Critical strengths
  • The paper addresses an important problem between successful tool execution and successful completion of a user's objective; it is observed quite ubiquitously in today's world.

  • The evaluation distinguishes false success declarations from actual goal closure and realized harm, which is useful.

  • The authors investigate correlated observation failures, contract omissions, and the limitations of witness corroboration.

  • The paper includes extensive controlled experiments and acknowledges several limitations of its assumptions and evaluation environments.

Critical weaknesses
  • The paper is difficult to read not in terms of understanding but in terms of the authors have not defined terms correctly. It is more AI-generated and not edited suitably to make it human readable -- many terms are introduced without formal definitions and claims are just stated without sufficient evidence.

  • The main theoretical results largely follow from established principles in formal verification and reasoning under partial observability. The novelty is unclear to me.

  • The framework depends critically on accurately translating user goals into formal contracts. How faithful is the translation?

  • The paper contains an unusually large number of experimental tracks, variants, controls, and protocol revisions, making it difficult to identify which experiments provide the primary evidence for the contribution. The presentation would benefit from a much clearer separation.

  • The distinction between detecting incomplete goals and recovering from them is not sufficiently clear throughout the paper. Much of the proposed benefit appears to come from refusing to declare success when evidence is insufficient.

  • The paper repeatedly emphasizes zero dangerous success declarations, but this does not necessarily translate into improved task completion or overall safety. ?

  • The comparisons with baselines are not convincing, were they run with the same original implementations or too many modifications?

Questions for authors

As above in the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hZYw: Method Contribution, Evidence, and Baseline Implementation

Thank you for recognizing the importance of task-level completion verification and our evaluation.

Response to W1: Definitions and readability

Thank you for highlighting the need for precise terminology and explicit support for claims. Section 3 formally defines goal closure, compatible execution states, and closure counterexamples. In particular, a closure counterexample is a state consistent with the model and observations in which the goal remains incomplete—not an arbitrary hypothetical failure. Propositions 3–4 and their proofs establish conditional soundness and maximal permissiveness under the stated assumptions; empirical improvements are assessed separately through paired experiments.

We revised four aspects to make these connections easier to follow:

  • Definitions and exposition: Connected formal terms and algorithmic decisions through one flight-change example, distinguishing evidence acquisition from state-changing recovery.
  • Figure: Redesigned Figure 1 and aligned its terminology and branches with the algorithm.
  • Claims and evidence: Clarified the question and metrics of each comparison, separating theoretical guarantees from measured completion, false declarations, and realized harm.
  • Organization: Added an evidence guide and condensed repetitive appendix material to make supporting results easier to locate.

Response to W2: Method and benchmark contributions

Among the compared methods, GoalSeal is the first to explicitly model common-cause evidence failures in compatible execution states to guide evidence acquisition, safe recovery, and completion. Queries distinguish states; recovery must meet the harm budget in every state; completion requires excluding all compatible incomplete states. Recovery invalidates relevant old evidence.

The innovation makes unresolved failure states actionable. The propositions characterize identifiability, query cost, and sound admission. GoalCommitBench pairs identical initial transcripts with different completion states, distinguishing probes, and recovery side effects, testing detection, recovery, and false declarations separately. Results: 85.0% completion and 0.0% observed dangerous false success (W6).

Response to W3: Contract faithfulness

The main comparison uses reference contracts; automatic generation is evaluated separately:

MetricSingle modelQuorumContract
Required-clause precision ↑80.7%93.5%
Required-clause recall ↑70.3%100.0%
Completion gap to reference ↓16.0 pp6.2 pp

QuorumContract improves clause coverage and completion, not full-contract equivalence. The guarantee requires a complete contract and a model covering the actual execution.

Response to W4: Primary experimental evidence

The nine-configuration matched comparison (1,152 conditions per policy) is primary evidence. The 45-task ToolBench-X retest tests transfer; 80 recovery-conformance cases test decisions; contract and correlated-fault tests examine assumptions. We condensed appendix repetition and separated historical contrasts from current results. The evidence guide maps questions to experiments.

Response to W5: Detection versus recovery

Detection acquires evidence; recovery changes external state through constrained actions. Strict certification never declares success yet achieves 77.9% completion. GoalSeal-CG declares success in 82.4% of episodes, with 85.0% completion and no observed dangerous false success. Completion is measured from environment state, not declarations. These whole-policy results do not isolate recovery; that requires an on/off comparison with the same gate and budget.

Response to W6: Completion and overall safety

Nine-configuration results:

MethodDangerous false success ↓Realized harm ↓Actual completion ↑
VTC5.8%9.5%82.6%
RAC6.2%11.5%80.7%
GoalSeal-CG0.0%4.7%85.0%

The completion gains over RAC/VTC are 4.3/2.4 percentage points, with paired template-cluster 95% CIs [0.1, 8.9]/[−2.2, 8.2]; the latter includes zero. All 54 harmful GoalSeal-CG episodes involve external state revocation after an irreversible effect; none receives a success declaration. Zero observed dangerous declarations is not equated with zero harm.

Response to W7: Baseline implementations

Our task/tool adaptations preserve the original methods' core verification and compensation logic.

VTC: Without a public implementation from the original paper, we reimplemented its published algorithm for the common task/tool interfaces.

RAC: We tested the authors' RecoveryManager separately. The main comparison adapts effect logging and reverse-order compensation to our benchmark. The appendix maps implementations to sources and algorithms.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Thanks for the comments, I will keep my score.

Reviewer_1GBHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests a new framework and monitor for udnerstanding when AI models have successfully completed a task, beyond APIs. It introduces a synthetic benchmark and a contract-voting scheme.

Critical strengths
  • Nice observation that repeated checks are not independent evidence
  • Boundaries and limitations argued
Critical weaknesses
  • Somewhat difficult to read
  • Engineering effort is appreciated but not entirely sure what the unique scientific insight is
Questions for authors

Why is the headline eight models when nine configurations were run?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1GBH: Readability, Scientific Insight, and Nine-Configuration Evaluation

Thank you for recognizing the importance of distinguishing repeated checks from independent evidence and of specifying the framework's boundaries. We address your concerns point by point below.

Response to W1: Readability

The revision uses a consistent flight-change example to connect the intuition, formal definitions, and Algorithm 1, while Figure 1 presents the general architecture. We made four improvements:

  • Figure: We redesigned Figure 1 and aligned its evidence-acquisition, recovery, and completion branches with Algorithm 1.
  • Method: We connect the running example to the definitions and distinguish gathering evidence from changing external state.
  • Experiments: We clarify the question addressed by each comparison and distinguish actual completion, dangerous false success, and realized harm.
  • Appendix: We added an evidence guide, reorganized related subsections, and condensed repetitive explanations to make proofs and supporting experiments easier to locate.

Response to W2: Unique scientific insight

The key insight is that a safe completion declaration requires excluding every incomplete execution compatible with the available evidence, under the stated contract and model assumptions. A shared fault can make repeated checks report success without excluding the same incomplete execution. For a fixed contract and model, Propositions 3–4 specify the conditions for sound admission and its maximal admissible region under the same observation record.

GoalSeal uses the unresolved execution states to determine what to query, which recovery actions are safe, and when to declare completion. GoalCommitBench makes this insight testable by constructing completed and incomplete executions with identical initial transcripts. Across nine configurations, GoalSeal-CG reduces observed dangerous false success from 9.1% to 0.0% and increases actual completion from 74.8% to 85.0% relative to the prompted-only agent, while declaring success in 82.4% of episodes.

Response to Q1: Why did the original main comparison report eight configurations when nine were run?

The original matched comparison included eight configurations because Claude Sonnet's inference API was unavailable during the baseline extension, preventing completion of its evaluation before submission. We have now completed the missing Claude Sonnet evaluations for three policies on the same 128 task–fault conditions and included them in the nine-configuration main comparison. The updated results are:

MethodDangerous false success ↓Actual completion ↑
VTC5.8%82.6%
RAC6.2%80.7%
GoalSeal-CG0.0%85.0%
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.

As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.

If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.

Thank you for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response: Practical Motivation, Methodological Contribution, and Revisions

We thank the reviewers for their careful evaluations and constructive feedback. This work was motivated by failure patterns observed in products serving hundreds of millions of users: successful tool responses do not necessarily imply completion of the user’s objective. Our rebuttal and revised manuscript clarify GoalSeal’s contribution and strengthen the evidence through controlled benchmarks and service experiments.

  • Methodological and benchmark contributions. Among the methods compared in this paper, GoalSeal is the first to explicitly incorporate common-cause evidence failures into compatible execution states and use unresolved failure states to drive evidence acquisition, safe recovery, and completion decisions. This allows the monitor to distinguish repeated checks from genuinely informative evidence and to assess whether a repair is safe across the remaining possible states. GoalCommitBench makes these decisions experimentally testable through identical initial transcripts with different true completion states.

  • Improved completion and safety. The complete nine-configuration comparison covers 1,152 matched conditions per policy. GoalSeal-CG achieves 85.0% actual completion and 0.0% observed dangerous false success, compared with 82.6% / 5.8% for VTC and 80.7% / 6.2% for RAC. Realized harm is also lower: 4.7%, versus 9.5% and 11.5%, respectively. These results evaluate completion and harm separately from the decision to declare success.

  • Recovery and transfer beyond the main benchmark. A disjoint 45-task ToolBench-X evaluation shows zero observed dangerous declarations under injected faults and a 26.7-percentage-point improvement in clean safe completion over the strict fixed gate. Stateful service experiments exercise the action–verification–recovery loop, while recovery-selector ablations examine the roles of evidence selection and harm constraints. These supporting studies are reported separately from the primary matched comparison.

  • Contract faithfulness and verification assumptions. We clarify how QuorumContract supports goal-specification construction, while GoalSeal-CG performs execution-time monitoring. Contract-quality and correlated-evidence tests examine the assumptions underlying completion guarantees. Our responses also explain why deterministic probes need not resolve task-level ambiguity and why a complete completion verifier need not identify a safe recovery action.

  • Baseline comparability. We specify implementation sources, task and tool-interface adaptations, and correspondence with the published algorithms. The matched evaluation holds task conditions, permissions, recovery primitives, and budgets fixed, clarifying both the comparison scope and the mechanisms being evaluated.

  • Readability and presentation. We redesigned Figure 1, clarified terminology and metric definitions, and connected the method to a consistent flight-change example. We also removed historical development narratives and redundant tables, focusing the appendix on the reported method, supporting experiments, and statistical analysis.

We hope these responses and revisions support a reassessment of the paper’s methodological contribution, practical relevance, and empirical evidence. We welcome further questions concerning any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.