acceptodds
Withdrawn from ICLR 2027

Monitoring Through a Noisy Sensor: Soundness Transfer and Net-Harm Thresholds for GUI-Agent Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Runtime verification for LLM agents reports compliance at or near one hundred percent, and every such result assumes readable agent state: structured events, tool arguments, logs. A GUI agent has none. Its predicates must be inferred from pixels by an operator that is wrong at rate epsilon and silent on a 1-gamma share of atoms, so the object analysed is a monitor plus a noisy sensor. Verification under noisy observation exists but parameterises error against a model a pixel-driven agent does not have; and no deployed monitor formulation can represent the state "the sensor could not resolve this atom", so it is recorded as a clean verdict. We give the transfer theorem in the parameters that survive — soundness and completeness as closed-form functions of precision, recall and coverage, a hole no operating point closes — together with a threshold past which monitoring is net harmful, a quantity that depends on the base rate and the cost ratio and on nothing else: not on coverage, not on formula shape, not on the dependence among sensor errors. On oracle atoms our reproduction gate returns soundness 1.000, completeness 1.000 and a false-block rate of 0.000, reproducing the compliance figure the field quotes; substituting pixels for the oracle drives two published monitors and one of ours into the net-harm regime. The threshold gets the sign right in 9 of 9 rows over 240 episodes, and we report what that does and does not establish: one negative row separates at p<0.05, while the decisive row sits at an effective error of 0.246 against a threshold of 0.238 and cannot be distinguished from it (p=0.76), so calibrated simulation at n=3000 per cell resolves the crossing instead, bracketing it in [0.220, 0.250] against a predicted 0.2375. The sharpest result needed no new experiment: 76 of 240 episodes contain an atom the operator would not resolve, and both published two-valued monitors record every one as a clean verdict — a guarantee that is vacuous rather than violated, and invisible to any compliance rate. One preregistered premise of ours failed: look-alike widgets were predicted to cost coverage, and coverage is 1.000.

Primary area:
datasets and benchmarks
Keywords:
runtime verification, GUI agents, LLM agents, agent safety, monitoring, perception error, soundness, completeness, cost-sensitive evaluation, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mZxjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a framework to analyze runtime formal verification of agents with fallible observation of the atomic propositions, with potential application to GUI agents. It derives the soundness and completeness probabilities of monitors (modeled as conjunctions of atomic propositions) under assumptions of independent observation of atomic propositions. The framework includes modeling of the utility of including the monitor and derives a critical threshold for net positive utility. It applies this framework on existing monitoring mechanisms and on the dataset CON-150.

Critical strengths
  • The problem (runtime verification of GUI agents) is very interesting and of high value if properly solved. The proofs seem correct to me.
  • I'm not very familiar with this field, the underlying theory and proposed monitor architecture might be very important despite being somewhat buried in the poor writing.
Critical weaknesses
  • The presentation and writing make the paper very difficult to read as it is. Many non-standard jargons and terms are used with little context or proper definition. The experiment setups are not clearly presented and the relevant appendix sections are not pointed to in the main text.
  • The case under consideration seems very simple, namely the monitor being a conjunction of atomic propositions that can be read off from the input by the fallible sensor (grounding operator). This makes much of the theoretical derivation almost tautological. That being said, I do appreciate the value of the framework itself.
  • The connection to GUI agent seems rather indirect to me. The whole framework seems to be just condering a fallible sensor (grounding operator). The empirical part directly compares with Kamath and Xavier, none of which seems to have been applied on agents with vision modality verification. I cannot identify the dataset used, CON-150, throughout the paper, which seems to be the only real dataset mentioned in the main text. I believe this is the proposed dataset because of the primary area choice. But there is very little description about this dataset itself.
Questions for authors
  • What is CON-150? Does it invovle visual modality?
  • Is figure 2 measured or predicted/simulated?
  • Is it fair to assume monitors being conjunction of atomic propositions, with no logical dependency between them? I'd imagine there can be atomic propositions with nontrivial logical dependency while one cannot reduce any completely to the rest.
  • Could you please clarify where the detail of each experiment presented in the main text can be found in appendix, if at all? Could you please clarify how the content of each appendix contributes to the whole paper?
Reviewer_r2QbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how visual perception errors affect GUI-agent monitors. It separates incorrect readings from unresolved observations and identifies when false blocks can outweigh the benefits of monitoring. The idea is useful, but the theory and experiments do not fully support the main claims.

Critical strengths
  1. Reporting incorrect readings and unresolved observations separately helps explain failures that a single compliance score would hide.
  2. The experiments give all monitor variants the same sensor outputs, making the comparison easier to interpret.
Critical weaknesses
  1. The proofs use the same error rate for false alarms and missed violations, and the same coverage for violating and compliant states. These simplifications are not clearly justified.
  2. The paper does not consistently follow standard academic writing conventions. The figure captions are overly long, and headings such as “What we do not claim” feel too informal.
Questions for authors
  1. Could you justify the shared error and coverage rates used in the proofs, and explain how the results change when these rates differ?
  2. Can you validate the threshold on new episodes using sensor statistics estimated separately from the outcomes used to calculate net benefit?
Reviewer_FfUrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies runtime monitors whose inputs are inferred from screenshots rather than read from structured state. It separates perception error from abstention, derives guarantee-transfer formulas and cost-based monitoring thresholds, and evaluates three monitor variants on 240 episodes under different rendering conditions. The practical message is important: a logically correct monitor can still make harmful decisions when its observations are unreliable. However, the main mathematical guarantees and the interpretation of the audit need substantial revision.

Overall assessment: The motivation is worthwhile, but the central guarantees and empirical validation do not currently support the paper's conclusions.

Critical strengths
  • The problem formulation is useful. Separating unresolved observations from incorrect ones, and pricing false blocks alongside missed violations, exposes weaknesses that a single compliance score hides. This is relevant to practical GUI-agent monitoring.
  • The paper reports important negative evidence. It acknowledges the failed coverage prediction and that the key near-threshold empirical row is inconclusive, rather than treating every point estimate as significant (Secs. 6.2 and 6.5).
Critical weaknesses
  • The transfer theorem is incorrect as written. Theorem 1 defines the coefficients in its false-positive probability as counts of non-violating states, but they must be conditional probability masses. For two atoms with uniformly distributed non-violating states, the probability is [2ε(1−ε)+ε2]/3[2\varepsilon(1-\varepsilon)+\varepsilon^2]/3, not the stated unweighted sum. Moreover, independent reads alone do not imply equal false-positive/false-negative rates or coverage independent of the true state. Those additional assumptions are needed for the claimed cancellation of coverage and the single-error-rate threshold. These are central guarantees, not peripheral notation issues.
  • The audit does not establish the claimed transfer to published monitors. Secs. 2 and 5 say published implementations are reused unchanged, whereas Sec. 6.1 calls them re-implementations and explicitly disclaims reproduction of the published compliance results. Also, Sec. 6.2 obtains exact agreement from an episode-level accounting identity using observed outcomes. That checks arithmetic, but is not independent validation of a predictive threshold. The simulation tests the assumed model, not whether the real sensor satisfies it.
Questions for authors
  1. Clarification/correction: What distribution over non-violating states and class-conditional error/coverage assumptions are intended? Please give corrected theorems and show which conclusions survive without symmetric errors.
  2. Clarification: Which original monitor code was actually executed, and how is effective error estimated independently of the net-benefit calculation? Which empirical results test a threshold fixed before observing those outcomes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.