acceptodds
Under review as a conference paper at ICLR 2027

DragForge: Procedural Evaluation of Drag for GUI Agents

Abstract

Drag actions combine object selection, a cursor trajectory, and release in a valid state. Endpoint grounding alone cannot evaluate interactions whose outcome depends on the intervening path. We introduce DragForge, a procedural benchmark with 18 task families and 60 configurations spanning placement, path-constrained manipulation, shape production, selection, and dynamic control. Policies observe screenshots and execute bounded waypoint sequences; evaluators measure the resulting task state. Privileged references solve all 180 instances used in the policy evaluation. In paired experiments on 117 development and 117 evaluation instances, replacing successful paths with straight segments while preserving their endpoints retains all 45 placement successes but only one of 72 successes in path-dependent families, in both splits. We evaluate five screenshot-conditioned vision-language policies on the same 180 instances under one protocol: four open-weight models and Claude Opus 5.5. Family-balanced task quality ranges from 6.8 to 70.1 on a 0–100 scale. Among the four open-weight models, quality rises descriptively with size, from 6.8 for a dense 27B model that fails even placement to 60.8 for a 2.8T-parameter model, although size is not isolated from training. The strongest policy, Claude Opus 5.5, solves every placement family but scores 78.4 on geometry and 32.0 on dynamic control. Recorded static prefixes improve with early calls and saturate as episodes terminate under native limits. In track, maze and curve tracing, replayed failures of the stronger policies usually press and release at the correct locations but violate the path. DragForge supplies executable tasks, state-based scoring, and controlled trajectory interventions for studying drag behavior beyond endpoint prediction.

Primary area:
datasets and benchmarks
Keywords:
llm, vlm, benchmark, gui, drag, control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aHJ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DragForge with 18 task families and 60 configurations spanning placement, path-constrained manipulation, shape production, selection, and dynamic control. Policies observe screenshots and execute bounded waypoint sequences

Critical strengths
  1. The writing is clear.
Critical weaknesses
  1. The task settings in this paper rarely appear in real GUI operations. In the introduction, the authors use dragging operating-system files as an example, but agents such as Claude Code can accomplish this operation simply through bash commands. The paper lacks a justification of its motivation, especially why are drag operations worth evaluating?

  2. Lack of benchmark calibratio. Specifically, the authors should add a condition that removes the visual input and includes only the text brief, asking the model to respond, to test whether the model’s performance truly comes from visual understanding of the screenshot.

  3. The author lack the analysis of Outliers. Such as, why Kimi K3 is 0 on ping-pong, but Qwen 3,8-27B got 9.3, also why claude opus 5.5 only got the 10.5 but Kimi K3 is 37.4 on flappy. The author should provide the case study of this outliers. not just show how the task looks like.

Questions for authors

1: Could the author provide the motivation of drag bench? I don't understand why this little problem have any constribution to the Agents development.

2: Based on the paper’s current presentation, I am confident that this benchmark has no potential to become a mainstream benchmark for LLMs or GUI agents. I hope the authors can rebut my view, and I am happy to discuss it with them.

Reviewer_qtsrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DragForge, a procedural benchmark for evaluating drag interactions by executing cursor trajectories and measuring resulting task states. It contain 18 task families and 60 configurations covering placement, constrained transport, drawing, selection, puzzles, and dynamic control. The authors validate feasibility using privileged reference controllers and evaluate five screenshot-conditioned vision-language policies on 180 instances. Endpoint-preserving interventions demonstrate that replacing successful trajectories with straight segments preserves placement success but largely eliminates success on path-dependent tasks. The paper then suggests evaluating intermediate trajectory geometry and task outcomes alongside endpoint grounding, and provides an environment suite, scoring framework, and diagnostic analyses for this purpose.

Critical strengths
  1. The evaluation target is relevant and well motivated. Correct source and destination predictions do not guarantee successful dragging. Execution-based scoring also accommodates multiple valid trajectories without requiring agreement with one demonstration.

  2. The reporting is transparent and appropriately qualified. The paper distinguishes privileged feasibility from screenshot-based solvability, acknowledges that model size is confounded with training, and correctly limits the interpretation of recorded-prefix budget curves. Exact prompts, family-level results, and terminal-status counts make the evaluation easier to assess.

Critical weaknesses
  1. The baseline coverage is insufficient for establishing the benchmark’s diagnostic value. All evaluated policies are general-purpose models using a common textual waypoint interface. Excluding endpoint-only agents is understandable, but it does not address trajectory-capable approaches such as ShowUI-π\pi, which explicitly generates continuous drag actions. A comparison on a compatible subset would help establish whether DragForge reveals limitations that persist with an action representation designed for this problem. A screenshot-based human or nonprivileged planner baseline would also provide a more informative difficulty reference than a controller with simulator access.

  2. The observation and action protocol limits the interpretation of failures. Each request omits conversational history, static drags execute without intermediate visual feedback, and screenshots are downscaled. These choices can affect geometry perception, recovery from previous mistakes, and inference of dynamic state. For example, fruit slicing provides no projectile-position trail, while previous observations are unavailable. Correct endpoints therefore do not establish that the intervening geometry was perceived correctly. The authors acknowledge this limitation, but targeted history, resolution, or intermediate-feedback ablations are needed to make the benchmark more diagnostic.

  3. Output failures substantially complicate the model comparisons. Table 15 reports 73 truncated episodes for MiniMax M3, 34 invalid plus 12 truncated episodes for Kimi K2.6, and 22 invalid episodes for Claude Opus 5.5, each out of 180. Retaining partial task quality is sensible, but early termination still removes opportunities to act. These results measure performance of the complete model–decoder–budget combination; they provide weaker evidence about trajectory capability itself. The main text should foreground these rates and include a controlled decoding or output-budget sensitivity analysis, with actual token consumption reported.

  4. The connection to practical GUI capability remains underdeveloped. Synthetic tasks are valuable for controlled diagnosis, but six game-like dynamic families receive one-third of the aggregate weight, and some easy configurations include solved action guides. The paper does not yet show that this weighting or these challenges predict performance on application-based dragging. A small transfer study, or sensitivity analysis excluding guided configurations and varying family-group weights, would clarify what a high aggregate score represents.

Questions for authors
  1. Can the authors evaluate a trajectory-capable agent or screenshot-based planning baseline on a representative subset? Would this change the interpretation of the observed performance gap?
  2. How much do dynamic and geometry scores improve when policies receive a short observation/action history or higher-resolution screenshots?
  3. What causes the high truncation and invalid-action rates, and do the principal rankings persist under a decoding configuration that reduces these failures at a documented compute budget?
Reviewer_DEnrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DragForge, a procedural benchmark for evaluating drag interactions in GUI agents. It includes 18 task families and 60 configurations covering placement, path-constrained manipulation, shape production, selection, and dynamic control. Unlike endpoint-based evaluation, DragForge executes waypoint trajectories and measures resulting task states. The paper validates path dependence through endpoint-preserving interventions and evaluates five general-purpose VLMs under a common protocol.

Critical strengths

The benchmark addresses a meaningful limitation of existing GUI grounding evaluation. Correct drag endpoints do not necessarily imply successful interactions when intermediate trajectories matter. State-based evaluation provides a more appropriate measure for such tasks.

The controlled trajectory interventions provide convincing validation. Replacing successful paths with straight segments preserves placement success while almost eliminating success in path-dependent families. This directly demonstrates the importance of evaluating trajectory geometry.

The benchmark covers diverse interaction types with a relatively careful protocol. Reference controllers, procedural splits, strict completion measures, and family-level analyses provide useful diagnostic information beyond aggregate task success.

Critical weaknesses

howUI-π already studies continuous GUI drag trajectories and provides a dedicated drag evaluation benchmark. DragForge mainly differs through procedural environments, state-based scoring, and endpoint-preserving interventions. These are useful additions, but the paper does not demonstrate how its conclusions differ from those of existing trajectory-based benchmarks.

The evaluation lacks specialized GUI-agent baselines. All five evaluated policies are general-purpose VLMs, rather than models specifically trained for GUI interaction or continuous drag generation. Consequently, the results establish limitations of general-purpose models but do not clearly characterize the capabilities or limitations of modern specialized GUI agents.

The synthetic environments limit practical generalization. Each configuration has only three evaluation instances, and all tasks use procedurally generated browser interfaces. The benchmark does not establish whether the observed failures transfer to realistic desktop applications or long-horizon computer-use workflows.

Questions for authors

How does DragForge complement existing drag benchmarks in terms of failure modes and model rankings?

How stable are the results with more evaluation seeds and realistic drag interactions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.