- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a post action verification formalization and a greedy algorithm to select probes and utilize evidences to determine whether a goal is actually met when an agent declares success. The authors propose GoalSeal-CG which is a greedy method for selecting counter factuals that selects counterfactuals with the least cost of probe set first and reviews evidences to see if a goal is met. If all counter factuals are exhausted without gola closure condition violation then the agent success is ascertained. Else a danger or failure is reported.
Critical strengths
I really liked the theoretical formulation of the problem. It sets up a very useful framework where properties of agents can be proven.
It addresses an important problem about ascertaining goal success for an agent when probes and evidences are uncertain.
THe counterexample guided recovery algorithm is sound greedy approach with bounded cost. Although it is NP hard but for practical cases the combinatorial problem is not compute heavy and hence can be solved through enumeration.
The GoalCommitBench is especially useful contribution it has explicit construction of ambiguity. For example successful and unsuccessful worlds share the same initial transcript, registered probes distinguish them, and recovery actions can introduce additional harm.
Critical weaknesses
While the theoretical exposition is interesting and useful and makes the problem precise, it is not a central novelty. The problem has been discussed by many papers as highlighted in the related work section. Also Propositions 1 is by definition. Proposition 2 is known apriori for greedy approach. Proposition 3 is by construction. Proposition 4 is also by definition. Isnt completion the rule the only rule available and should be satisfied? Then it has to be the most permissive because it is the only rule. So I would ask the authors exactly what are the claimed novelty of the paper?
If the probe has a deterministic outcome, then is your problem definition same as verifier grounded replanning?
Also from my reading of the paper it seems like your formulation is only non-trivial when the verifier is wrong or incomplete. So if a verifier is complete then there is only a single trivial counter example. Is my reading correct?
Questions for authors
Look at weaknesses.
2 comments
Follow-up on our rebuttal
Dear Reviewer,
Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns.
As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification.
If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate.
Thank you for your consideration.
Response to Reviewer LSAd: Method and Benchmark Contributions, Deterministic Probes, and Complete Verifiers
Thank you for recognizing the value of the formalization and GoalCommitBench's explicit construction of ambiguity. We address your three concerns below.
Response to W1: Methodological novelty and benchmark contribution
Among the methods compared in this paper, GoalSeal is the first to explicitly incorporate common-cause evidence failures into compatible execution states and use unresolved failure states to drive evidence acquisition, safe recovery, and completion decisions. It prioritizes queries that distinguish completed from incomplete states, admits only recovery actions satisfying the harm budget in every compatible state, and declares completion after excluding all compatible incomplete states. After recovery changes external state, the relevant old evidence is invalidated and completion conditions are checked again.
The key innovation is to make unresolved failure states actionable: they determine which evidence to acquire, which corrective actions are safe, and when there is sufficient justification to finish. The four propositions specify the method's identifiability, query-cost, and sound-admission conditions.
GoalCommitBench explicitly constructs tasks with identical initial transcripts but different true completion states. Distinguishing probes, recovery actions with different side effects, and controlled faults separately test whether a method can identify incomplete goals, avoid false declarations, and recover safely. Actual completion, dangerous false success, and realized harm are measured separately, distinguishing genuine task completion from merely withholding success declarations.
In the nine-configuration matched evaluation, GoalSeal-CG achieves 85.0% actual completion, exceeding VTC and RAC by 2.4 and 4.3 percentage points, respectively. Its observed dangerous false success rate is 0.0%, compared with 5.8% and 6.2%. These results demonstrate its ability to improve actual completion while controlling erroneous declarations in the evaluated conditions. VTC and RAC refer to the benchmark implementations of their verification and compensation mechanisms, with adaptation scope documented in the appendix.
Response to W2: Do deterministic probes reduce the problem to verifier-grounded replanning?
Determinism does not imply that an observation distinguishes the relevant states. For example, both “replacement ticket issued” and “replacement ticket not issued” can produce the same cached response. GoalSeal selects evidence that distinguishes the remaining states and checks whether a retry could cause duplicate charging in any of them.
When a verifier reliably determines whole-task completion and supplies the information needed for safe replanning, verifier-grounded replanning can realize this special case. GoalSeal addresses situations in which observation ambiguity remains and recovery must be constrained across compatible execution states.
Response to W3: Does a complete verifier leave only one trivial counterexample?
Thank you for this careful reading. Your interpretation is close; the key distinction is between completion assessment and recovery decisions.
If a verifier soundly and completely determines whether the entire user goal has been achieved, completion can indeed be assessed directly from its output, without further resolving completion ambiguity. This is a simplified special case of the framework.
However, an “incomplete” verdict does not imply a unique failure state. A refund that has not been initiated, is still being processed, or was sent to the wrong account may receive the same verdict but require different corrective actions. Even with an accurate completion verdict, distinguishing these states can therefore remain necessary for choosing safe recovery.