acceptodds
Under review as a conference paper at ICLR 2027

THE PHYSICAL SEARCH FALLACY: EXACT-EFFECT AUTHORIZATION FOR IRREVERSIBLE AGENTS

Abstract

An agent must create a new output fle. It sees an existing delivery.tsv,overwrites it, and only then learns that the fle was a bystander. We call this the Physical Search Fallacy: using a state-changing action to discover whether that action was right. Reading frst is insuffcient when the read still permits two different physical changes. Our rule is simple: do not authorize an irreversible action until trusted evidence leaves only one exact physical effect compatible with the task. Not all uncertainty must be removed. Two possible environments may differ yet require the same physical change; only uncertainty that crosses different effects blocks authority. HypoProbe therefore separates three questions: identify whether remaining uncertainty changes the effect, resolve such ambiguity with safe observa-tions, and preserve the certifed effect through commit. Resolving ambiguity means separating every pair of compatible worlds that would require different physical effects; minimum-cost non-adaptive resolution is a weighted set-cover problem. On a frozen 252-instance controlled test expanded from 126 base tasks and spanning flesystem, PostgreSQL, system settings, composed terminal operations, browser profles, and controlled online accounts, the previous rule causes 84 destructive out-comes. VBE+, a strengthened verify-before-execute baseline sharing HypoProbe’sΣ, dependency closure, safe reads, and execution safeguards, commits all 346/346 required effects but causes 47 destructive outcomes. HypoProbe commits the same effects with none observed; without evidence acquisition, lawful commit-task cov-erage falls to 46/74 (62.2%), while overall terminal correctness is 88.9%. In live evaluations, each of two models has 0/42 observed destructive actions and recov-ers all 9/9 intercepted destructive proposals. On WebArena-Verifed, HypoProbe attains 75.7% commit precision (340/449) and 89.9% commit recall (340/378),compared with 43% precision for VBE+; screening fags 19 of HypoProbe’s 109 false-positive commits as C3 high-impact candidates, of which 17 are realized wrong writes and two are evaluator artifacts. In the narrower 374-case method-valid consequential cohort, its 302 authorized mutations have no observed destructive outcome.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Physical Search Fallacy, Exact-Effect Authorization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_r17nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The core problem which the paper is trying to address is the fact that AI agents nowadays assume something which is actually not true in reality and end up performing an action which should not have been performed. The example which the authors take is that of an AI agent overwriting an existing file which should not have been overwritten. In order to counter this problem, the authors propose a framework which consists of three layers: failure, identify + resolve, and preserve. Evaluation is performed on two datasets: one which the authors build themselves, and the second one is the existing Webarena verified benchmark.

Critical strengths

The identified problem is timely, relevant, and interesting.

Critical weaknesses

First of all, the paper is very, very difficult to read. There is hardly anything understandable in the paper, probably because the authors have used too much AI to write the paper (authors can confirm this). If thats the case, request the authors to write the first draft themselves, and then, of course, they can use AI to improve the grammar and presentation of the paper.

Also, I'm not really positive about this paper. Although the problem identified is relevant and interesting, the proposed solution does not seem to be general enough to be applied in any agent setting. At the end of the day, the ultimate problem is to identify what is a safe action and what is not a safe action, and it ultimately depends on the instructions given by the user, etc. An easier solution might just be to check with the user if something is safe or not before actually executing that action. That's why we see that in today's agents, Claude Code, Codex, etc., they have different kinds of permissioning modes. Depending on the criticality of the task and the environment, the users can choose the permissioning mode.

Is it possible to ground the proposed solution or idea more firmly in existing work? Was not able to understand from the related work what the state of the art is in this area.

Questions for authors

See weaknesses.

Reviewer_ZDVXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HypoProbe authorizes a mutation when all worlds compatible with trusted evidence require the same task- correct effect. Resolve acquires safe observations separating different-effect worlds. Preserve maintains the supporting dependencies through execution. The central idea is to resolve uncertainty that changes the justified effect, without identifying every feature of the world.

Critical strengths

The decomposition is clear. It distinguishes unsupported semantics, unresolved evidence, justified no-mutation, and execution obligations. The conditional separation and fixed-cover arguments appear ok in their static deterministic setting.

Critical weaknesses

The principal comparative result does not adequately explain the specific advance over a strong conventional gate. VBE+ shares configuration, reads, dependencies, and execution machinery, yet records 47 destructive outcomes.

The practical contribution depends on semantic and execution obligations that remain largely supplied. Truthful observations, complete effect dependencies, and preventive execution appear to be guesses.

Questions for authors
  1. Can one complete VBE+ failure show every passed check, the remaining alternative effect, and the decisive observation or validation condition that HypoProbe adds?
  2. What reusable procedure constructs and validates adapters with complete dependencies and preventive effect enforcement, including observations with side effects and tasks with multiple valid outputs?
Reviewer_52rjOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HypoProbe, a framework for safely authorizing irreversible actions by LLM agents. It formalizes authorization through exact physical effects and introduces a three-stage process: identifying effect-relevant ambiguity, resolving it through safe observations, and preserving authorization until execution. Experiments across controlled environments, live agents, and WebArena-Verified demonstrate improved safety over verification-based baselines, although some incorrect authorizations remain in external evaluations.

Critical strengths

1. Meaningful problem formulation. The paper identifies an important safety issue in irreversible agent actions and provides a principled exact-effect authorization framework.

Critical weaknesses

1. Limited methodological novelty.
Although exact-effect authorization provides an interesting perspective, the core technical components largely build on established ideas. The singleton-effect authorization condition follows directly from the definition of compatible worlds, while effect-separating observations and their reduction to weighted set cover are closely related to existing information acquisition and combinatorial optimization techniques. Dependency tracking, revalidation, and restricted execution are also well-established mechanisms. The paper does not sufficiently demonstrate what fundamentally new algorithmic or theoretical insight is introduced beyond integrating these existing concepts into a unified framework.

2. The trusted configuration Σ may encode much of the solution.
HypoProbe assumes a trusted configuration specifying authoritative sources, completeness, identities, dependencies, side effects, and safe observations. Accurately identifying this information is arguably one of the hardest parts of safe agent execution. Much of the domain knowledge needed to determine the correct physical effect may therefore already be embedded in Σ. This raises a fundamental question about whether HypoProbe solves the core reasoning problem or shifts it to a manually constructed configuration. The paper also lacks a convincing assessment of the effort required to construct and maintain Σ in realistic environments.

3. Strong assumptions and insufficient robustness analysis.
The safety guarantees depend on correct configurations, truthful observations, complete dependency specifications, and sound execution enforcement. The deployment preflight only checks structural declarations and cannot establish their semantic correctness. However, the evaluation provides limited evidence about how HypoProbe behaves under incomplete or incorrect configurations. Without such analysis, it is difficult to assess whether the proposed framework remains useful outside carefully controlled environments.

4. Limited real-world validation and remaining authorization errors.
The zero-destructive-outcome result on 252 controlled instances is encouraging, but these experiments rely on predefined task semantics and trusted configurations. On WebArena-Verified, HypoProbe still produces 109 false-positive commits, including 17 realized wrong writes in fixtures classified as invalid. The narrower method-valid cohort reports no observed destructive outcomes, but the relationship between this filtered result and the broader evaluation requires stronger justification. Overall, the evidence does not yet establish robust safety in realistic, open-ended agent environments.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.