- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how an agent should verify its own actions before they take effect. The authors argue that current verifiers are confirmation oriented, meaning they look for evidence that an action is right, and so they approve plausible but wrong actions. They propose BLACKSWAN, a refutation oriented protocol. Before a side effecting code cell runs, the agent writes down what the cell should achieve, lists up to five preregistered suspicions about how it could be wrong, and then tests each suspicion with read-only probes of the environment. Each suspicion gets a verdict of confirmed, dismissed, or deferred, and confirmed suspicions trigger a redraft. On AppWorld with a Qwen3.5-27B backbone, BLACKSWAN reaches 84.7 TGC and 72.4 SGC on the challenge split, about 15 to 19 points above the best baseline, and the gains transfer to WebShop and ALFWorld. Ablations show that refutation and preregistered suspicions only help when combined.
Critical strengths
- A clear and well motivated idea. The shift from confirming an action to actively trying to falsify it is simple, well framed, and the expect-suspect-test loop is easy to reuse. The qualitative examples in Appendix F make the difference from confirmation based verification very concrete.
- Strong and consistent results. Gains hold across three benchmarks and five backbones, with and without thinking, and the ablation cleanly shows the two key components only work together. The reporting is unusually honest, including a careful reimplementation of the closest baseline, per run standard deviations, and a deferred verdict category.
- Cost is measured and disclosed rather than hidden, with a common token price so the comparison reflects harness behavior rather than model pricing.
Critical weaknesses
- The compute overhead is large, about 6.4 times the tokens and 9.4 times the cost of REACT on the AppWorld challenge split. Part of the gain may come from simply spending more compute, and a budget matched baseline such as REACT with retries or best of n at equal cost is missing.
- The main comparison uses the authors' own reimplementation of ACCORD, which scores below the numbers reported in the original paper, so the margin over ACCORD may be somewhat overstated. Only open or mid size backbones are tested, so it is unclear how the method behaves with frontier closed models.
Questions for authors
- At an equal token or dollar budget, how does BLACKSWAN compare to simpler ways of spending the same compute, such as REACT with best of n sampling or self consistency?
- How often are suspicions deferred, and do deferred suspicions predict downstream failures? Could the agent re examine deferred suspicions after the action runs?
- The protocol assumes cheap read-only probes of the environment. How would the method apply in settings where probes are slow, costly, or unavailable, such as real web actions?
4 comments
Generalization to frontier closed models [W2]
We agree that evaluation with frontier closed models would strengthen our findings. Given their substantial API costs, however, it is hard to include them in our evaluation. We have acknowledged this limitation in the revised manuscript (Section 6).
Deferred suspicions [Q2]
How often are suspicions deferred?
The model judges 13.5% of suspicions as deferred. Among tested actions, 28.8% carry at least one deferred suspicion, and 9.3% are executed with one.
Do deferred suspicions predict downstream failures?
Among actions executed with deferred suspicions, 1.5% and 59.4% lead to immediate regressions and progress in task-level unit tests, respectively. We measured these effects by evaluating the benchmark unit tests immediately before and after each executed action.
Table T4. Effect of deferred suspicions.
| Effect | Rate |
|---|---|
| Progression only | 59.1% |
| Regression only | 1.2% |
| Both progression and regression | 0.3% |
| Neither | 39.4% |
We acknowledge that this analysis does not directly establish the relationship between deferred suspicions and "downstream" failures at task completion. While the unit tests provide reliable, fine-grained criteria for analysis, they assess only the resulting environment state. Directly identifying such relationship would require identifying the actions responsible for a failure, but the effects of actions within a trajectory are highly intertwined. As one of many possible scenarios, an incorrect order could result from an overlooked cart item, a mistaken product selection, an earlier misinterpretation of retrieved information, or a combination of these errors. We therefore report directly measurable action-level effects rather than relying on subjective failure attribution.
Could the agent re-examine deferred suspicions after the action runs?
We opt for implicit rather than explicit re-examination. Both the deferred suspicion and the execution result remain in the agent's context, allowing it to consider them when proposing subsequent actions. For example, when placing an order, the verifier raises a suspicion that the payment will fail due to insufficient funds but defers it because no API is available to check the balance. After observing that the execution fails for insufficient balance, the agent retries with another card. We favor this implicit behavior, as explicit post-execution re-examination would incur additional LLM calls. We have added a case study illustrating this behavior to Appendix B.
Practical constraints on observations [Q3]
Slow and costly. When probes are slow or costly (e.g., high API latency, per-request fees), BlackSwan remains applicable with several natural extensions to reduce probe cost. For example, the agent can prioritize suspicions likely to be confirmed (stopping upon the first suspicion confirmation), parallelize probes, or escalate to users. Each involves trade-offs between latency, API fees, and user burden. We deliberately keep these mechanisms simple in this work to isolate our central contribution: refutation-oriented verification for identifying and correcting misaligned actions.
Unavailable. In real web applications, agents can often obtain observations through either backend APIs (when permitted) or user interfaces (e.g., by searching or opening pages). Even if the means to observe the information needed to verify particular suspicions are unavailable, BlackSwan can still check prior observations for overlooked evidence and defer the suspicion if it remains unresolved. In the limiting case where all suspicions are deferred due to unavailable observation means, BlackSwan reduces to ReAct in terms of action execution, permitting all actions (as does ACCORD under its default-approval policy). However, we do not expect this limiting case to be typical, as the observation interfaces used for task execution and verification often overlap.
We have added extending BlackSwan to such settings as future work in Section 6.
References
- Popper (1963). Conjectures and Refutations: The Growth of Scientific Knowledge.
I thank the authors for the detailed rebuttal.
[W1, Q1]: The oracle pass@k results and the matched probe budget experiment largely resolve my main concern. [W2]: The 404 explanation is fair. Frontier closed models remains unevaluated, but I understand the cost constraint and appreciate the explicit acknowledgment in Section 6. [Q2]: The numbers are clear. Deferral happens 13.5% of the time, and these actions rarely cause problems. I am fine with this design now. [Q3]: The degradation argument make sense and the proposed extensions remain future work, which is acceptable.
Overall, the rebuttal addresses my concerns. I will maintain my score.
We thank the reviewer for their thoughtful response. We are pleased that our revisions, additional experiments, and analyses have addressed your concerns, particularly regarding the role of additional computation, the fidelity of the baseline reimplementation, and deferred suspicions. We also appreciate your understanding of the scope of our evaluation and the proposed future directions. Thank you again for your constructive feedback and careful consideration.
We thank the reviewer for their thoughtful and detailed evaluation. We appreciate the recognition of the conceptual framing, the consistent empirical improvements, and the transparent reporting of results. We hope the following responses address your concerns, and we welcome any further questions. Changes in the revised manuscript are marked in green.
Compute-matched comparisons [W1, Q1]
We conducted additional experiments. We allow agents to use an unbounded number of LLM calls until they explicitly decide to end the task by proposing an action containing
complete_task. Thus, the costs reported below, measured as in Figure 5, reflect each agent's own behavior rather than imposed compute budgets.Best of k trials with an oracle. Implementing ReAct with retries requires arbitrarily deciding when to retry, how to determine failure, etc. To avoid concerns arising from such arbitrary decisions, we instead evaluate oracle Pass@K, counting a task as successful if any of K runs succeeds. We note that this uses ground-truth labels from the benchmark evaluator and allows cost-free environment resets between retries. At comparable costs, neither baseline reaches BlackSwan's Pass@1.
Table T1. Oracle Pass@K.
Best-of-N with an LLM judge. At each step, ReAct samples N candidate actions from the same history. A listwise LLM judge selects one for execution. We use temperature 1.0 because candidates must be diverse enough for BoN.
Table T2. Best-of-N with an LLM judge. A single ReAct run at temperature 1.0 reaches 52.5 TGC (58.4 at temperature 0).
Matched probe budget. We gave ACCORD's grounding agent and BlackSwan's Test stage the same read-only probe allowance per verification, which neither method exhausted. This allowance limits probes per verification, while total LLM calls per task remain unbounded. Under the same allowance, BlackSwan gathers more observations and achieves substantially higher TGC.
Table T3. Probes per verification.
Takeaway. Under the same probe allowance or comparable compute, BlackSwan achieves substantially higher TGC. The gap thus lies not in how much the agent spends but in what it looks for. As Popper (1963) argues, observation is always selective: it needs a chosen object, an interest, a point of view, and a problem. BlackSwan's refutation-oriented suspicions supply these through preregistration.
Fidelity of ACCORD reimplementation [W2]
We agree that the fidelity of ACCORD's reproduction matters for a fair comparison. Unfortunately, ACCORD's code has not been released: although the paper links to its GitHub repository, it remains inaccessible (404) to date. Reimplementation was therefore unavoidable.
A consequential unspecified design choice is the number of in-context demonstrations. The ACCORD paper does not report this number, leaving two plausible settings:
We therefore evaluated both:
While an exact reproduction cannot be guaranteed, we have made every reasonable effort to reproduce ACCORD under the available specifications. Importantly, BlackSwan outperforms ACCORD in all evaluated settings (thinking on/off and 1/3 demonstrations).