acceptodds
Withdrawn from ICLR 2027

Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world websites and applications often deploy CAPTCHAs and other anti-automation mechanisms that use behavioral signals, such as interaction trajectories and timing, rather than outcomes alone, to distinguish human operators from automated agents. We refer to such settings as *protected GUI environments*, where correctly solving a local verification task is insufficient for end-to-end task completion. An agent must also convert solver outputs into interactions accepted by the environment and adapt its execution based on runtime feedback under time constraints. For these environments, we introduce **SolveBind**, a GUI-agent framework that combines *hierarchical solving with runtime supervision*. A hierarchical solving tree decomposes verification into interdependent *Perceive*, *Ground*, *Solve*, *Act*, and *Verify* capabilities. A runtime supervisor grounds action commitments in tool-derived evidence to allow them to be revised as new evidence arrives, and mediates execution under explicit recovery budgets. For verification, across nine CAPTCHA task families and nine backbone models, **SolveBind** achieves the highest success rate on eight backbones, outperforming the strongest baseline by 13.3–34.5 percentage points. For end-to-end task completion, it achieves the highest success rate on all six backbones evaluated on AndroidWorld. In a paired GPT-5.5 analysis, MobileAgent produces correct answers but fails dynamic verification on 13 of 90 tasks; **SolveBind** succeeds on seven of these tasks. With automatic submission retained, removing binding reduces success from 82.22% to 71.56% and increases latency and token usage. Recovery caps reduce execution cost with a small observed success trade-off: removing them raises success from 82.22% to 83.56%, while increasing latency and token usage to 3.7× and 2.4× their full-supervision values, respectively. These results support evidence-grounded execution control while identifying a success–cost trade-off in recovery and failure boundaries of heuristic submission rules.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, protected GUI environments, CAPTCHA solving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dUpJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SolveBind combines hierarchical GUI capabilities with evidence-bound action commitments, supervisory dispatch, progress checks, and bounded recovery. It is evaluated on HLL and AndroidWorld across multiple backbones and harnesses.

Critical strengths

The paper addresses a meaningful failure mode: correct solutions can still fail environment-level interaction validation.

Critical weaknesses
  1. Baseline credibility is insufficient. Qwen3.7-Plus has an official 81.0% AndroidWorld reference, versus 16.38–25.0% across the competing harnesses here; GLM-5V-Turbo reports 75.7%, versus 6.03–25.0%. Although these configurations are unmatched, the paper does not reconcile the gaps. Its appendix also documents frequent parser failures and unreliable symbol entry without quantifying their aggregate impact. Thus, some gains may reflect weak integrations rather than the proposed control mechanism.
  2. The AndroidWorld mechanism evidence is under-controlled. SolveBind and competing harnesses differ in tools and execution interfaces. The useful HLL ablation does not establish the cause of AndroidWorld gains. Missing executable configurations, model snapshots, and inference settings further prevent assessing these confounds. Validated baselines and matched-tools/executor experiments are needed.
Questions for authors
  1. Which model snapshots, inference settings, observations, task seeds, and harness configurations explain the Qwen/GLM gaps? Can you reproduce a reference configuration?
  2. How many failures arise from parsing, text entry, executor errors, and infrastructure exceptions? How do validated integrations change Table 2?
  3. On AndroidWorld, what is the effect of removing binding/supervision while holding tools, observations, executor, and budgets fixed?
Reviewer_ywkWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies GUI agents in protected environments, where websites use CAPTCHAs and behavior checks, so a correct answer can still be rejected. The authors propose SolveBind, a training-free harness with two parts, a hierarchical solving tree that splits the task into perceive, ground, solve, act and verify steps, and a runtime supervisor that keeps tool results as evidence and can override the model's proposal with an evidence-supported action. Recovery is also limited by explicit budgets. Experiments on HLL and AndroidWorld with several foundation models show that SolveBind gets higher success rates than MobileAgent, AppAgent and the UI-TARS harness in most settings, often with lower latency.

Critical strengths
  • The research question is important and practical. Real-world websites often use CAPTCHAs and behavioral checks, so even if a GUI agent provides the correct answer, it may still be blocked. This gap is crucial for deploying agents in real-world environments.

  • The evaluation covered multiple backbone models and reported costs and success rates. SolveBind is tested on HLL with nine base models and on AndroidWorld with six base models, achieving the best success rates on 8/9 models and 6/6 models, respectively. Latency and token usage for all runs (including failed runs) were reported, providing more comprehensive information than simply reporting success rates.

Critical weaknesses
  • Some citations feel not very necessary. The paper cites quite a few older works, especially in the runtime supervision and metareasoning part, which reads more like framing than real support for the method. It does not help too much in understanding the technical details or the motivation.
  • The experiments are not convincing enough. The main baselines are earlier general-purpose harnesses (AppAgent, MobileAgent v1), and closer harnesses that also do verification and recovery are not compared (like MAPLE, VLAA-GUI and StructAgent). In addition, HLL is the only CAPTCHA benchmark, and the main table covers just 90 of its 900 instances with a single run.
  • The technical contribution is relatively limited. Verification, bounded recovery and execution state are already common in existing harnesses, and the main new part is making the runtime override the model’s proposal. The paper has many ablations, but the difference from prior work is mainly described by using words and is not well supported by direct empirical results.
Questions for authors

SolveBind has extra tools (detector, OCR, solver) and element-indexed actions that the baselines do not have. How much of the gain comes from these tools rather than the runtime supervisor, especially on AndroidWorld?

Reviewer_uZ46OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies GUI execution in settings where solving a verification challenge does not guarantee that the resulting interaction will be accepted. SolveBind organizes tool use through a hierarchy of capabilities and a runtime supervisor that records tool-supported actions as revisable commitments. When a commitment remains admissible, the supervisor gives it precedence over conflicting model proposals, checks subsequent progress, and bounds recovery attempts. Evaluation covers HLL verification tasks and AndroidWorld application workflows across several model backbones. The ablations examine dispatch precedence and the trade-off between recovery cost and success.

Critical strengths
  1. I find binding a useful contribution to GUI execution control. With tools, model-visible evidence, and automatic submission held fixed, enforcing pending commitments raises success from 161/225 to 185/225 (Table 3).

  2. The complete framework shows practical value across both models and task settings. SolveBind achieves the highest success rate on eight of nine HLL backbones and all six AndroidWorld backbones in the reported comparisons (Tables 1 and 2). The application results extend the system's demonstrated usefulness beyond CAPTCHA verification, even though they do not isolate binding's contribution.

  3. The recovery ablation supports bounded recovery as a practical design choice. Removing the caps increases success from 185/225 to 188/225, while mean runtime rises from 112.13 to 417.45 seconds (Table 3). The capped configuration thus achieves nearly the same completion rate at substantially lower execution cost, providing

Critical weaknesses
  1. The methodological advance is somewhat limited. Hierarchical planning with specialist grounding already appears in Agent S2, while MAPLE uses structured state verification and recovery. The clearest addition here is dispatch precedence for pending tool-supported actions. The binding ablation demonstrates its utility, but the contribution of the hierarchical modeling beyond these established ideas is less apparent.

  2. The operational role of the AND/OR hierarchy is unclear. The paper defines readiness predicates but allows their implementation to combine runtime checks and model guidance, without specifying which dependencies are enforced at dispatch. This makes it difficult to reconstruct how the hierarchy constrains the permitted operations in an actual run.

  3. The experiments do not isolate the contribution of the hierarchical organization. The runtime ablations retain this structure throughout, so they support the binding and recovery rules without establishing whether the hierarchy improves on the same tools and dispatch policy operating in a simpler execution loop.

  4. Evidence that the binding benefit transfers across settings remains not sufficiently strong. Table 3 provides a controlled comparison for GPT-5.5 on HLL, while the broader results compare complete harnesses with different tools and execution procedures. Those results support the complete system, but do not establish how much of its advantage comes from the proposed control mechanism.

Questions for authors

Q1. Which concrete checks retain or invalidate a pending commitment after new feedback, and which are implemented by runtime rules rather than model judgments? One recorded example of a commitment being released would clarify this distinction.

Q2. In the full and no-binding conditions of Table 3, how many failed episodes submit a correct answer that is rejected by the dynamic validator? This breakdown would help determine whether binding reduces the specific execution failure motivating the paper.

Q3. Does the binding benefit persist in a matched AndroidWorld comparison with the same tools, executor, and other controls? A result for one backbone would help assess transfer beyond HLL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.