- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces ConGUIBench, a controlled benchmark of 118 shopping and travel tasks on four real platforms (Amazon, JD, Booking, XieCheng) in two languages, with 688 non-procedural (NPG) and 688 matched explicit-procedure (EP) instructions, 2-4 constraints per task and a validated graph of 901 GUI states and 1,148 transitions. It asks a question that terminal success alone cannot answer: whether GUI agents are invariant to equivalent goal wording (goal invariance), follow an explicitly requested order (procedural controllability), and adapt to realized task state (state adaptivity). Across nine open-weight models, the authors find cross-order disagreement in outcomes across semantically equivalent instructions; that agents largely follow a requested order when they reach constraint, so failures stem mainly from execution rather than procedure following, with grounding misses named as the largest error type. The studies conducted by authors also find useful insights into agent performance using SLMs, for example Agents respond to visible progress i.e. when some constraints are already satisfied at the start, the agent's first action more often targets a pending constraint. Overall, the benchmark's dual-annotator validation is commendable and instills confidence in the quality and fidelity of the benchmark. Furthermore, there is a lot to learn from the studies and ablations done in this paper.
However, some claims might have been overstated in the paper and certain evaluation protocol have some inconsistencies which raises concerns about the validity of some results. For example, the benchmark has setup an offline graph-based execution of agents which has a strict range of valid click actions. Any click outside the annotated graph ends the episode, due to which valid but unannotated actions count as failures, so reported success rates may underestimate agent ability by an unknown amount (Detailed in W2). Further, several ordering ablations rely on the fact that all permutations have equal difficulty. However, that may not be the case as ordering of contraints in physical layout matters and some orders might be simply easier to execute on the page (W4). Finally, the scope of the benchmark seems quite small and limited due to length and difficulty of tasks (discussed at length in W2) and the eval experiments only cover open-weight models up to 32B, so the frontier on ConGUIBench is unknown, and the paper's own scale trends suggest several of the reported deficits, including order sensitivity, may shrink in larger models (W1).
Due to these reasons, and many more highlighted in the weakness and questions sections, the review rating is a weak reject.
Critical strengths
Overall, useful study of GUI capability gaps for Small-to-medium SLMs
S1 Dual-annotator quality control of every instruction and transition Each instruction realization is independently reviewed by two annotators, who check that it contains the intended constraint set in the specified mention order while preserving the goal semantics and instruction is retained only when both annotators approve. Every state transition is also inspected by both annotators (source state, assigned constraint, click region, and resulting state), and disputed regions or transitions are corrected and independently reviewed again. The authors further verify that every nonterminal state has an outgoing transition. This level of vetting across all 688 instruction pairs and 1,148 transitions gives confidence in the benchmark's labels.
S2 Well-thought ablations The study varies one factor at a time along many axes, while holding the constraint set fixed: the wording of the instruction (every permutation of the mention order), its format (non-procedural NPG vs. explicit-procedure EP), the starting state of the task (no constraint satisfied vs. some already satisfied), the outcome of an action (normal vs. a valid click that fails to take effect), the state information given to the agent (instruction only vs. Oracle satisfied/pending labels vs. an auditor-maintained checklist), and the source of the constraint list (ground truth vs. parsed). Results are further broken down by platform, language, and model family and size. Along these axes, the authors examine several distinct kinds of trends: how stable success is across equivalent wordings, whether agents follow a requested order, which route agents actually take and how consistent that route is across runs, whether agents redirect their actions after visible progress, how much completion they lose when an action fails, which types of errors end an episode, and how much outcomes vary across repeated runs. Covering this many angles gives a far more complete picture than a single success rate. The error-type study (Table 6) was also quite insightful as shows that grounding near misses are the largest named error type (27.6–28.7%) and are not reduced by EP, while EP does reduce completed-target repetition and premature selections. This breaks failures that ISR lumps together into distinct causes and makes the result actionable: click-level grounding, rather than procedure following, is a key bottleneck to address.
S3 Well-Designed metrics The metrics are also well chosen: CSR vs. ISR separates partial from full completion, and COCR vs. SOCR separates order adherence from success. Δ_order metric is unbiased and principled as the authors designed a noise-corrected estimator to study mention order sensitivity. By subtracting the disagreement between repeated runs of the same instruction from disagreement between runs of different order, it's expectation equals mean squared difference in success probability between orders, so it is unbiased and has expectation exactly zero when order has no effect.
Critical weaknesses
W1 - No frontier or larger models are evaluated, so the frontier (i.e. ISR) on ConGUIBench is unknown; The applicability of trends like cross-order disagreement, sharpening of route-alignment, constraint-state tracking reduces variability, etc on larger models is also unknown.
- The evaluated set stops at 32B. All nine models are open-weight and range from 2B to 32B (Qwen3-VL 2B/4B/8B/32B, GLM-4.6V-Flash, UI-Mate-9B/27B, UI-Venus-1.5-2B/8B; §4.2.1). No proprietary frontier model and no larger open-weight model is evaluated. §4.2.1 also states that “proprietary models are accessed through their official APIs from the same evaluation pipeline,” yet no proprietary results appear anywhere in the paper. Were such models run and their results left out, or is this sentence left over from an earlier version?
- The conclusions are stated about GUI agents in general but the model size also matters. The abstract and conclusion attribute “unstable goal execution, conditional procedural control, and brittle recovery” to GUI agents, and name constraint-state tracking as “an important reasoning bottleneck.” The paper's own scale trend suggests some of these deficits shrink with size. Within Qwen3-VL, going from 2B to 32B, SAG rises from +8.8 to +18.9, the paired B−A completion loss shrinks from −12.2 to −4.4 (Table 5a), and the checklist gain falls from +9.5 to +2.1 under EP and from +14.9 to +7.9 under NPG (Table 7). Without stronger models, it is unclear whether these are lasting properties of GUI agents or limitations of small models that disappear at scale.
- Order sensitivity itself appears to shrink with scale. Reading Fig. 2 (values approximate), the largest model in each family has the lowest Δ_order within that family: Qwen3-VL 2B ≈ 8.7, 4B ≈ 9, 8B ≈ 14, 32B ≈ 6; UI-Mate 9B ≈ 4.7, 27B ≈ 2.8; UI-Venus-1.5 2B ≈ 6.1, 8B ≈ 4.9. If this trend continues, larger or frontier models may be largely insensitive to mention order, which would turn the paper's headline finding into a property of smaller models. Testing at least one larger model would show whether general capability gained through scale or pretraining makes agents robust to these instruction variations.
- Headroom is not established. The best ISR is 56.7% under NPG (Table 2) and 66.8% under EP (Table 3), both from an 8B model. Without frontier results, readers cannot tell how much headroom the benchmark leaves for the strongest current agents, which matters for its usefulness going forward.
- A missed opportunity. The benchmark's separation of goal invariance, procedural controllability, and state adaptivity is well suited to show which capabilities emerge with scale and which do not, e.g. whether larger models close the recovery gap but remain sensitive to mention order. This would be one of the most informative results the benchmark could produce, and the paper does not attempt it.
- Suggested resolution. Evaluate at least one or two frontier proprietary models and the largest available open-weight model (e.g. Qwen3-VL-235B-A22B). If cost is a concern, a task subset with fewer samples per instruction would suffice. Then report, for each of Q1–Q3 and the checklist probe, which deficits narrow with scale and which persist. Otherwise, limit the claims to open-weight models of 32B parameters or fewer.
W2 - The benchmark's scope is narrow and its tasks are very short, and the offline evaluation protocol limits what the results can say about real GUI execution.
- ConGUIBench has 118 tasks on four sites (Table 8), and 279 of its 307 constraints (90.9%) are filters. 93 tasks contain only filters. The benchmark therefore mainly measures multi-filter selection.
- Extremely short horizons. Tasks average only 2.60 constraints (Table 8; 59 tasks have two, 47 three, and 12 four), and each constraint corresponds to a single annotated transition. A complete episode is thus two to four correct clicks. Findings about recovery, state tracking, and procedure following may not transfer to the long, multi-page workflows where these capabilities matter most.
- An offline graph with strict termination. Agents do not interact with a live or replayed website but move through recorded screenshots; a click outside the candidate region of an annotated transition ends the episode. The potentially undestimates ISR (discussed in W3 in more detail)
- The action space is unspecified. Only clicks are described. It is unclear whether agents can scroll or type, and this matters because “requires scrolling” is the largest subtype within Other errors (44.2% EP, 51.8% NPG; Table 6b): if scrolling is unavailable, these are limitations of the benchmark rather than agent failures.
W3 - Off-graph termination marks valid but unannotated actions as errors, and this directly lowers Success Rate!
- The graph is admittedly incomplete, yet any click outside it ends the episode. The graph is “a validated representation of annotated feasible paths, not a claim that every possible interaction trajectory has been exhaustively enumerated” (App. B.2), but a click outside every annotated region terminates the episode with zero credit for the remaining constraints (§4.2; App. B.3). Exploratory clicks (e.g. opening a dropdown or expanding a filter panel) and equivalent but unannotated controls (e.g. the same filter in another panel) are therefore scored as failures.
- This directly lowers ISR. ISR requires every constraint to be satisfied, so a single valid but unannotated click (e.g. opening a dropdown) ends the run and counts as a complete failure. Reported ISR may therefore underestimate what agents can actually do, by an unknown amount.
W4 - The Q1 order-sensitivity result cannot be attributed to mention order: reordering also changes the words in instruction because of unavoidable paraphrasing, and there is no paraphrase control to separate the two.
- Reordering changes more than the order. The paper's own examples (Fig. 3) show it: “Find black Versace sunglasses for under 200”, “Find Versace sunglasses under 200 in black” and “Find black sunglasses, made by Versace, under 200” differ not only in constraint order but also in connective words (“for,” “in,” “made by”).
- The baseline removes noise, not wording effects. Δ_order subtracts the disagreement between repeated runs of the identical instruction text (Eq. 9). This correctly removes sampling noise, but a positive Δ_order shows only that changing the text changes outcomes. It cannot say whether the change comes from the new order or from the new words. A model that is sensitive to any rewording would show the same result even if order played no role.
- The claims attribute the effect to order. The abstract states that “reordering equivalent NPG requirements produces 2.8–14.0% of excess cross-order success disagreement,” and §4.3.1 speaks of “changes in success or satisfied constraints caused by reordering the same requirements.” The caveat in App. A does not reach these statements.
- Suggested resolution: a paraphrase control. For each order, add two or three vetted paraphrases that keep the constraint order fixed and change only the surrounding connective words, exactly the kind of change reordering induces. The constraint phrases themselves should stay verbatim, as they do under reordering (e.g. keep “Versace sunglasses” and “under 200”; vary only connectives such as “for,” “in,” or “made by”: “Find sunglasses made by Versace, in black, under 200”). Compute Δ_paraphrase with the same estimator, using pairs of runs from different paraphrases of the same order as the between term. Δ_order − Δ_paraphrase then estimates the order-specific effect (assuming order and wording effects add), with the same task bootstrap. A full orders × paraphrases grid analyzed with a mixed-effects model would separate the two effects without that assumption. A subset of tasks, e.g. the 3–4 constraint ones, would suffice. Alternatively, rephrase the claims as sensitivity to rewording rather than to order only.
W5 - Q1's order sensitivity may reflect layout-driven route difficulty rather than unstable goal execution.
- The layout imposes its own ordering. Constraints are physical controls on the page and since all orders take the same number of clicks, some orders are easier purely because of the layout (e.g. the next control on the layout matches the next constraint or stays visible without scrolling). An agent that follows the text will therefore succeed more often when the mention order happens to match the actual layout order, so Δ_order may measure route difficulty rather than “unstable goal execution” (abstract, §5).
- Suggested resolution. Identify the easiest order per task from the recorded pages, report SR(mention order = easiest) − SR(otherwise), and test whether Δ_order remains positive after conditioning on the executed route.
W6 - The checklist experiment shows some gains, but those gains might be happening due to more than just checklist items
- The checklist gives the agent more than progress marks. Besides the DONE/TODO labels, it also provides the goal already split into clean constraints, the auditor's notes on each constraint, and instructions such as "Do NOT repeat actions" (App. J.3). Any of these could explain the gain. Adding them one at a time would show which part actually helps.
- The agent is never asked to keep its own checklist. To show that agents cannot track their progress, the authors would need to prompt the agent to write and update its own checklist. If that works as well as the auditor's checklist, the agents can track state; they simply weren't asked to.
- The gains are small and have no error bars. Under EP, four of nine models gain at most 1.4 points or get worse, and the stability gain is just 1.1–2.0 points.
- The auditor is never checked. A small 4B model fills in the DONE/TODO labels, but its accuracy is never measured, even though the benchmark has the true states to compare against.
Questions for authors
Q1 What actions are available to the agent? Only clicks are described. Can agents scroll or type, and how are the three form-value constraints (Table 10) entered? This matters for interpreting the error profile: "requires scrolling" is the largest subtype within Other errors (44.2% EP, 51.8% NPG; Table 6b). If scrolling is unavailable, these are artifacts of the benchmark rather than agent failures.
Q2 Is the Q1 order sensitivity explained by route difficulty? On a real web page, the order of the controls is fixed by the layout. Some completion orders are easier to execute than others, for example when the next filter stays visible without scrolling or the order of filters on the layout itself matches with the order in instruction. Since agents also tend to follow the order in which constraints are mentioned (Table 4), success may change across orderings simply because some orderings happen to match the actual layout order.
8 comments
Response to Reviewer fgPk (Part 2)
W4 Summary: Does Q1 sensitivity arise from constraint mention order or accompanying wording changes, and does it persist when connective wording is controlled?
Order sensitivity persists after controlling for connective wording. We evaluate nine models on 40 NPG tasks, fully crossing two mention orders and three connective templates with three repeats (6,480 trajectories). Constraint phrases remain unchanged, isolating order and connective-template variations. Mean order-minus-paraphrase contrasts are positive for both outcomes, though model-level effects vary. This supports mention-order sensitivity beyond connective wording changes. The design and aggregate results are now reported in Appendix D.4, Table 15, of the revised manuscript.
Table Q5. Nine-model mean controlled disagreement contrasts (pp; 95% CIs). Each Δ subtracts same-text repeat disagreement; terminal-state disagreement is normalized Hamming distance. Intervals use 10,000 website-stratified physical-task bootstrap draws shared across models, with equal task/model weights and no multiplicity adjustment.
| Outcome | Δorder (pp) | Δparaphrase (pp) | Δorder − Δparaphrase (pp) |
|---|---|---|---|
| Task success | +3.26 [+1.88, +4.77] | +0.22 [-0.44, +0.90] | +3.05 [+1.66, +4.54] |
| Terminal constraint state | +6.67 [+4.82, +8.65] | +0.48 [-0.04, +1.02] | +6.19 [+4.26, +8.28] |
W5/Q2 Summary: Does mention order affect success mainly because some orders follow easier layout-dependent routes, and does excess disagreement remain when the executed route is held fixed?
Layout alignment does not consistently improve success, and instruction sensitivity persists within shared execution prefixes. We define layout order by control positions and low-cost routes by annotated scrolls, actions, and backtracking, without using model outcomes. All nine layout-match 95% CIs include zero; low-cost matching shows opposite nonzero effects for Qwen3-VL-8B (+8.19 pp) and GLM-4.6V-Flash (−5.48 pp) (Table Q6-A).
Conditioning on the same first one or two accepted transitions, excess cross-instruction success disagreement remains positive with 95% CIs excluding zero for 7/9 and 6/9 models, respectively (Table Q6-B). Thus, layout alignment alone does not consistently explain order sensitivity, which persists within shared initial paths. However, prefix conditioning does not control subsequent routes or selection effects.
Table Q6-A. NPG success-rate difference: matching order − other orders (pp; 95% CIs). Layout order is top-to-bottom, then left-to-right for vertically overlapping controls (68 tasks); the low-cost comparison includes only graph-feasible orders and tasks with both groups (86 tasks). Low cost is a predefined proxy, not a verified easiest route.
| Model | Layout match (68 tasks) | Low-cost order match (86 tasks) |
|---|---|---|
| Qwen3-VL-2B | -0.23 [-9.00, +8.94] | +0.23 [-6.87, +7.55] |
| Qwen3-VL-4B | +4.93 [-3.07, +13.05] | +1.92 [-4.34, +8.32] |
| Qwen3-VL-8B | +5.08 [-4.85, +15.23] | +8.19 [+0.39, +16.17] |
| Qwen3-VL-32B | +4.02 [-2.72, +10.97] | +2.44 [-2.99, +7.85] |
| GLM-4.6V-Flash | -2.59 [-8.16, +2.41] | -5.48 [-10.92, -0.55] |
| UI-Mate-9B | -2.03 [-9.67, +5.19] | -3.32 [-9.55, +2.17] |
| UI-Mate-27B | -0.71 [-7.27, +5.16] | -0.84 [-5.86, +3.81] |
| UI-Venus-1.5-2B | -4.20 [-11.75, +3.47] | +3.37 [-2.26, +8.91] |
| UI-Venus-1.5-8B | -0.17 [-5.69, +5.28] | -0.20 [-5.13, +4.72] |
Table Q6-B. NPG excess success disagreement within shared execution prefixes (pp; 95% CIs). Δ subtracts same-instruction repeat disagreement from cross-instruction disagreement; coverage columns show eligible tasks and retained runs as a percentage of all runs for that model.
| Model | Δ: same first transition | Tasks; runs retained | Δ: same first two transitions | Tasks; runs retained |
|---|---|---|---|---|
| Qwen3-VL-2B | +11.10 [+6.25, +16.73] | 59; 34.4% | +11.58 [+5.40, +18.64] | 29; 15.6% |
| Qwen3-VL-4B | +7.50 [+2.69, +13.25] | 62; 37.8% | +11.57 [+4.69, +19.79] | 36; 21.2% |
| Qwen3-VL-8B | +11.89 [+7.29, +17.25] | 69; 58.2% | +13.15 [+7.60, +19.05] | 40; 34.4% |
| Qwen3-VL-32B | +1.17 [-0.12, +2.77] | 58; 38.9% | +1.85 [+0.29, +3.78] | 38; 22.3% |
| GLM-4.6V-Flash | +3.09 [+0.35, +6.62] | 75; 43.0% | +4.64 [+0.65, +10.67] | 39; 23.1% |
| UI-Mate-9B | +7.89 [+1.96, +14.56] | 57; 26.9% | +1.66 [-0.71, +4.58] | 30; 13.0% |
| UI-Mate-27B | +2.18 [-1.00, +5.65] | 48; 29.3% | +3.20 [-0.47, +8.24] | 30; 13.7% |
| UI-Venus-1.5-2B | +10.12 [+5.63, +15.38] | 65; 40.9% | +9.58 [+3.57, +16.50] | 29; 16.6% |
| UI-Venus-1.5-8B | +6.87 [+2.25, +12.05] | 75; 34.2% | +4.30 [-0.40, +11.16] | 37; 11.7% |
Response to Reviewer fgPk (Part 3)
W6 Summary: Which components—constraint decomposition, progress labels, auditor notes, or behavioral guidance—account for the checklist gains, and can incremental ablations isolate their contributions?
The component ablations show that the checklist benefit is not attributable to progress labels alone. We compare A: NoCL; B: the static constraint list; C: the same list with externally predicted DONE/TODO/UNSURE labels; and D: the full checklist, including notes and behavioral guidance. The static list helps under NPG, while dynamic status and the full checklist provide additional average gains across the evaluated models. These results support contributions from multiple components. Full per-model ablations appear in Appendix F.3, Tables 23 and 24, of the revised manuscript. Table 27 in Appendix F.5 (“Constraint Sources and Auditor Label Reliability”) further shows that replacing ground-truth constraints with auxiliary-parser-extracted constraints, while retaining the 4B auditor, changes the nine-model mean EP ISR from 50.0% to 49.0%. This suggests limited average sensitivity to constraint source in this setting, despite model-level variation.
Table Q7. Mean ISR changes (pp; 95% CIs where available). Component contrasts follow the shared paired evaluation protocol, average physical tasks and then models, and use 10,000 paired task-bootstrap draws. The Model-Parsed row is the separate EP greedy comparison reported in Appendix F.5, Table 27; its CI was not reported (—).
| Format | Comparison | ΔISR [95% CI] |
|---|---|---|
| EP | B − A: static list | -0.9 [-2.1, +0.2] |
| EP | C − B: dynamic status | +2.6 [+1.8, +3.5] |
| EP | D − C: notes + guidance | +1.9 [+1.0, +2.9] |
| NPG | B − A: static list | +2.8 [+0.8, +5.0] |
| NPG | C − B: dynamic status | +4.4 [+3.3, +5.6] |
| NPG | D − C: notes + guidance | +1.9 [+0.8, +3.1] |
| EP | Model-Parsed − Ground-Truth | -1.0 [—] |
W6. The agent is never asked to keep its own checklist. To show that agents cannot track their progress, the authors would need to prompt the agent to write and update its own checklist. If that works as well as the auditor's checklist, the agents can track state; they simply weren't asked to.
Self-checklist prompting yields lower mean ISR than the external status-only checklist. Both use the same status-only action-prompt format.
Self-checklist prompting has an average ISR deficit of 1.6 pp under EP and 2.3 pp under NPG, with heterogeneous model-level differences. The ISR-difference intervals include zero for Qwen3-VL-4B/8B/32B and UI-Venus-1.5-8B under both formats. These results support an average advantage for the external updater under this protocol, rather than an intrinsic inability of agents to maintain their own state.
Table Q8. Self-checklist − external status-only: paired differences (pp; 95% CIs). Δ = Self − External; positive values favor self-checklist. Mean External/Self ISR is 48.6%/47.0% under EP and 48.9%/46.6% under NPG. Confidence intervals use 10,000 paired physical-task bootstrap draws shared across models.
| Model | EP ΔISR [95% CI] | EP ΔCSR [95% CI] | NPG ΔISR [95% CI] | NPG ΔCSR [95% CI] |
|---|---|---|---|---|
| Qwen3-VL-2B | -3.67 [-6.46, -0.95] | -2.37 [-4.66, -0.09] | -7.87 [-11.58, -4.68] | -6.17 [-8.68, -3.92] |
| Qwen3-VL-4B | +0.32 [-0.01, +0.76] | +0.27 [+0.03, +0.65] | +0.16 [-0.49, +0.89] | +0.01 [-0.37, +0.38] |
| Qwen3-VL-8B | -0.20 [-1.71, +0.93] | -0.40 [-1.61, +0.54] | +0.55 [-0.08, +1.39] | +0.23 [-0.43, +0.93] |
| Qwen3-VL-32B | +0.28 [-0.49, +1.00] | +0.11 [-0.27, +0.45] | +0.39 [-0.75, +1.34] | -0.20 [-1.20, +0.49] |
| UI-Mate-9B | -3.72 [-5.94, -1.53] | -2.03 [-3.44, -0.60] | -3.79 [-6.09, -1.63] | -2.41 [-3.96, -0.91] |
| UI-Mate-27B | -3.09 [-4.93, -1.43] | -1.95 [-2.99, -1.01] | -3.20 [-5.01, -1.55] | -1.88 [-3.00, -0.84] |
| UI-Venus-1.5-2B | -3.36 [-5.50, -1.32] | -2.15 [-3.59, -0.75] | -4.22 [-6.74, -1.91] | -1.57 [-3.30, -0.01] |
| UI-Venus-1.5-8B | -0.13 [-0.95, +0.44] | -0.03 [-0.44, +0.30] | -0.53 [-1.69, +0.48] | -0.22 [-0.87, +0.37] |
| GLM-4.6V-Flash | -0.84 [-1.53, -0.22] | -0.59 [-1.15, -0.16] | -1.84 [-3.21, -0.64] | -0.78 [-1.51, -0.13] |
| Nine-model mean | -1.60 [-2.28, -0.97] | -1.02 [-1.46, -0.58] | -2.26 [-2.97, -1.61] | -1.44 [-1.85, -1.05] |
Response to Reviewer fgPk
We thank the reviewer for the careful reading of our paper and the constructive feedback.
R: We extend ConGUIBench to three API models: Doubao-Seed-2.0-Pro/Lite and Qwen3.7-Plus. On a fixed 36-task subset, the best ISR reaches 75.1% EP / 73.2% NPG, leaving substantial completion headroom.
Order and state sensitivity persist in API models. All three show positive Δstate with 95% CIs excluding zero; Δorder is also positive with CIs excluding zero for Lite and Qwen3.7-Plus. EP−NPG ISR intervals include zero for all three. These results extend the original diagnostics.
State-intervention results reveal model-dependent recovery. Across all starts, including inapplicable cases, the paired ISR loss is 9.4–20.9 pp for Pro/Lite (all four 95% CIs exclude zero), but only 2.8–2.9 pp for Qwen3.7-Plus (CIs include zero; revised manuscript, Appendix E.2, Table 18, all-start column). All six state-adaptation-gap intervals include zero. In the Oracle follow-up, gold state labels show no consistent mitigation: five of six mitigation CIs include zero, while Lite/EP shows a negative effect (−13.9 pp).
Overall, stronger API models still exhibit completion gaps, instruction sensitivity, and adaptation failures. These findings apply to the tested 36-task subset. Intervals use 10,000 physical-task bootstrap draws.
R: ConGUIBench complements existing GUI benchmarks by diagnosing goal satisfaction, procedural adherence, and state adaptation under matched conditions for the same constraint set. Its 118 tasks contain two to four constraints, predominantly filters (279/307), whose explicit states allow us to verify what has been completed and what remains; a constraint may require multiple primitive actions, so constraint count is not click count. The compact tasks and validated offline graphs support reproducible comparisons across instruction orders and execution states, separating intermediate progress from final completion. This provides a controlled diagnostic complement to end-to-end evaluation; our findings concern short multi-constraint workflows along validated paths and do not establish performance on long, open-ended web tasks.
R: Scrolling is supported, and the form-value constraints are annotated as click–type–confirm sequences. “Requires scrolling” refers to rejected clicks at states where no eligible click candidate exists and the correct action is to scroll, as indicated by an annotated scroll transition. Its 44.2%/51.8% EP/NPG figures represent shares of the Other error category. The form-value constraints enter price values: agents focus the relevant field, type the value, and confirm it. Their legacy annotations encode the numeric value in the action-type field.
We control equivalent-action ambiguity while allowing multiple valid execution paths. We exclude tasks with multiple semantically valid actions for the same constraint, but retain different constraint-completion orders and annotated preparatory actions. Since instructions explicitly require filtering, searching or selecting items prematurely violates the specified workflow.
Off-graph actions terminate execution due to unvalidated successors, while CSR credits satisfied constraints. Error categories provide behavioral diagnostics, not exhaustive semantic validation. Thus, the protocol measures execution reliability within validated paths.