Monitoring Through a Noisy Sensor: Soundness Transfer and Net-Harm Thresholds for GUI-Agent Verification
Abstract
Runtime verification for LLM agents reports compliance at or near one hundred percent, and every such result assumes readable agent state: structured events, tool arguments, logs. A GUI agent has none. Its predicates must be inferred from pixels by an operator that is wrong at rate epsilon and silent on a 1-gamma share of atoms, so the object analysed is a monitor plus a noisy sensor. Verification under noisy observation exists but parameterises error against a model a pixel-driven agent does not have; and no deployed monitor formulation can represent the state "the sensor could not resolve this atom", so it is recorded as a clean verdict. We give the transfer theorem in the parameters that survive — soundness and completeness as closed-form functions of precision, recall and coverage, a hole no operating point closes — together with a threshold past which monitoring is net harmful, a quantity that depends on the base rate and the cost ratio and on nothing else: not on coverage, not on formula shape, not on the dependence among sensor errors. On oracle atoms our reproduction gate returns soundness 1.000, completeness 1.000 and a false-block rate of 0.000, reproducing the compliance figure the field quotes; substituting pixels for the oracle drives two published monitors and one of ours into the net-harm regime. The threshold gets the sign right in 9 of 9 rows over 240 episodes, and we report what that does and does not establish: one negative row separates at p<0.05, while the decisive row sits at an effective error of 0.246 against a threshold of 0.238 and cannot be distinguished from it (p=0.76), so calibrated simulation at n=3000 per cell resolves the crossing instead, bracketing it in [0.220, 0.250] against a predicted 0.2375. The sharpest result needed no new experiment: 76 of 240 episodes contain an atom the operator would not resolve, and both published two-valued monitors record every one as a clean verdict — a guarantee that is vacuous rather than violated, and invisible to any compliance rate. One preregistered premise of ours failed: look-alike widgets were predicted to cost coverage, and coverage is 1.000.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.