acceptodds
Under review as a conference paper at ICLR 2027

FORGE: Feedback-driven Optimization and Refinement for GUI Evolution

Abstract

GUI agents based on vision-language models have shown strong potential for general-purpose desktop automation, but their post-training remains constrained by the lack of scalable and reliably verifiable training data. Existing synthesis pipelines are largely static, making it difficult to adapt to new data quality issues and the evolving capabilities of trained models. We present FORGE, an end-to-end data flywheel that produces both high-quality supervised fine-tuning trajectories and verifiable reinforcement learning tasks. FORGE constructs diverse GUI tasks through scenario-behavior matching and pairs each task with an executable evaluator. Crucially, the pipeline evolves through its own feedback: recurring generation errors are distilled into reusable validation skills, while model failures on exploratory tasks inform subsequent data production. This closed loop progressively improves both data quality and task coverage. Using FORGE, we construct over 10,000 verifiable GUI tasks and evaluate the resulting models on OSWorld across two backbones. FORGE-8B and FORGE-4B achieve 51.2% and 40.4%, improving over their base models by 17.3% and 9.0%, respectively. We will release the dataset, code, and trained models upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, Computer-Use Agent, Data Synthesis, Self-Evolving Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DckXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FORGE, a feedback-driven data generation framework for GUI agent post-training. It pairs synthesized tasks with executable evaluators and uses generation errors and model failures to refine validation skills and subsequent task production. The resulting tasks and trajectories support both SFT and RL, yielding substantial improvements on OSWorld across two Qwen3-VL model sizes.

Critical strengths
  1. The paper presents a well-designed feedback-driven data evolution framework that continuously refines data quality and task coverage based on observed failures. The experiments demonstrate consistent performance improvements across iterations, supporting the effectiveness and scaling of this approach.
  2. Instead of directly relying on VLM judges for task evaluation, the framework employs executable evaluators with a multi-stage validation process to mitigate potential judging biases and provide more consistent and reliable supervision for training.
Critical weaknesses
  1. The framework relies heavily on LLMs for task generation, validation, repair, and feedback analysis. The paper lacks a systematic analysis of the associated costs and whether the improvements justify the additional overhead compared to simpler data generation strategies.
Questions for authors
  1. Can the authors provide a cost-efficiency comparison with simpler static or adaptive data generation strategies under comparable budgets?
  2. Since both the generation and refinement of executable evaluators rely on LLMs, could biases or errors propagate into the reward signals? Did the authors observe any reward hacking or training instability during RL, and how were these issues monitored or mitigated?
  3. Free-composition exploration primarily recombines existing behaviors, how can FORGE discover capabilities that are entirely absent from the initial taxonomy?
2 comments
AuthorsOpenReview ↗

Response to Reviewer DckX: Cost-Efficiency, Evaluator Reliability, and Behavior Discovery

Thank you for recognizing the value of FORGE’s feedback-driven data evolution, the consistent improvements across iterations, and the use of executable evaluators with multi-stage validation. We address your questions below.

First, our initial version, v0, uses a static data generation pipeline without self-evolution and achieves limited improvements. It therefore provides a direct reference for assessing the benefits of introducing feedback. Section 4.3 compares data from different iterations while keeping the backbone, SFT dataset size, and training configuration fixed. Table 12 further examines the contributions of the two self-evolution components. Our cost measurements show that introducing dynamic feedback increases total data production costs by approximately 10%–20% relative to the static pipeline. Data synthesis remains the dominant cost. Together with the iteration results and component ablations, these measurements indicate that self-evolution improves the training utility of the generated data at a moderate additional cost. We will include the cost breakdown in the revision and clearly distinguish data production costs from downstream training costs.

Second, we agree that executable evaluators do not automatically guarantee correct rewards. LLM-generated evaluators may omit task requirements, impose overly permissive or restrictive conditions, or contain implementation errors. FORGE therefore subjects generated tasks and evaluators to programmatic checks, semantic checks, and real-environment validation, with repair where necessary. Recurring generation errors are also distilled into reusable validation skills. For SFT data collection, we additionally filter trajectories using a reward threshold. The human audit reported in the appendix independently examines task instructions, execution trajectories, and final states, and compares human judgments against evaluator outputs. We sample trajectories that evaluators label as both successful and unsuccessful to identify false positives and false negatives. The audit found a low proportion of false positives, providing direct empirical evidence that the evaluator validation pipeline reduces the risk of incorrect reward signals. In earlier training runs, we did encounter reward hacking or training instability, which motivated targeted constraints in our reward design. Specifically, F_t checks action-format validity and penalizes noncompliant outputs, while L_t penalizes repeated actions. The latter reduces unproductive repetition and indirectly limits unnecessary trajectory growth caused by such behavior. The appendix provides the definitions and related ablations.

Third, it is useful to distinguish free-composition task generation from subsequent failure diagnosis. Free composition initially samples from the existing behavior library to construct task combinations that existing templates do not cover. Expansion of the behavior library occurs during the subsequent feedback analysis. A task composed of known behaviors may require an intermediate operation that is not yet represented in the library. Failure analysis distinguishes among failures to execute an existing behavior, missing composition templates, and missing atomic behaviors. For missing atomic behaviors, the pipeline adds new behavior entries; for gaps in how existing behaviors are combined, it updates the composition templates. Section 3.3 and Algorithm 1 describe these updates. Tasks generated using the new entries must still pass the task validation pipeline. Thus, FORGE’s exploration is not permanently restricted to its initial taxonomy. Execution feedback can reveal missing operations and guide the expansion of the behavior library. This discovery depends on the tasks explored and the failures encountered; we do not claim that FORGE can discover every unknown capability unrelated to its exploration.

Thank you again for these constructive questions. We will add the cost analysis and clarify evaluator reliability and the scope of behavior discovery in the revised manuscript.

Thank you for the rebuttal. I looking forward to your revised manuscript and confim my original score.

Reviewer_Kgq1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FORGE, a self-evolving pipeline for constructing GUI training data. It combines task synthesis and evaluator validation, and uses failures during data generation and agent execution to update validation rules and task coverage. The resulting data are used with SFT and RL, improving performance of Qwen3-VL-4B/8B on OSWorld.

Critical strengths
  1. The paper makes a substantial data collection effort, producing over 10K verifiable GUI tasks. Its self-evolving pipeline updates both validation rules and task generation based on feedback, providing a useful mechanism for improving data quality and coverage.
  2. The OSWorld improvements of 17.3 and 9 percentage points are non-trivial and suggest that the collected data are useful for the evaluated models. (For this point, more concerns in the weaknesses part)
Critical weaknesses
  1. The core flaw of this study is the training data are synthesized within the applications and environments covered by OSWorld, while the end-to-end task evaluation is also conducted on OSWorld. Additional in-domain training is expected to improve performance, so these gains alone provide limited evidence of the dataset’s broader usefulness. Evaluation on OOD task environments is needed.
  2. The training experiments use only the Qwen3-VL-4B/8B, which belong to the same model family. The authors do not examine whether the collected data further improve an existing GUI-specialized model (e.g., ComputerRL and Mobile-Agent-v3.5). Evidence from another model family and further training of a GUI-specialized model would help establish the dataset’s value as a reusable contribution.
  3. Some crucial information about the dataset is deferred to the appendix. For example, the scenario and task counts per application in Table 5 are important for assessing the dataset’s coverage and representativeness, and should be in the main text.
  4. Writing can be improved. Section 3.1.2 uses dense notation for simple sampling rules, without sufficient intuitive explanations and concrete examples. Key procedural details, such as how scenario compatibility is maintained after behavior resampling, are also unclear. Abbreviation usage is unpolished, with late redefinitions of common terms (e.g., SFT, RL) and defined but unused acronyms (e.g., CoT).
Questions for authors

Please refer to the weaknesses

Reviewer_R1rLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FORGE, an end-to-end data flywheel framework designed for the post-training of vision-language model based GUI agents. Addressing the limitations of static data synthesis pipelines and unreliable LLM-based judges, FORGE jointly produces supervised fine-tuning trajectories and verifiable reinforcement learning tasks. The framework constructs diverse GUI tasks via scenario-behavior matching across three synthesis modes (atomic, template composition, and free composition) and pairs each task with an executable, deterministic evaluator. Its core contribution is a closed-loop self-evolution mechanism: recurring generation errors are distilled into reusable validation skills, while model failures on free-composition tasks guide subsequent data production to adapt to the model's evolving capabilities.

Critical strengths
  1. FORGE’s self-evolving data flywheel responds to shifts in the model’s capability bottlenecks. Generation errors become validation skills, and exploration failures become new task coverage. This is a concrete way to keep static pipelines from going stale as agents improve.

  2. The framework requires RL rewards to be verifiable through executable evaluators, and the L0–L4 validation and repair pipeline makes this practical. This is a useful design choice for keeping reward signals grounded.

Critical weaknesses
  1. The system-level engineering is strong, but the methodological novelty is limited: the core pipeline mainly repurposes GRPO, sliding-window SFT, and DAgger-style failure-prefix recovery.
  2. Only two models from the same Qwen3-VL family are tested. No experiments on different architectures are conducted to demonstrate architecture-agnostic applicability.
Questions for authors
  1. The self-evolving loop is presented as a key contribution, but the feedback signals (clustering errors into rules and probing failures to adjust sampling) resemble standard active/curriculum learning. It is unclear how much the full flywheel adds over a strong curriculum baseline such as difficulty-aware sampling without the loop.
  2. Both backbones are Qwen3-VL variants and thus share architecture, tokenizer, and vision encoder. The gains may therefore partly reflect Qwen3-VL-specific coupling rather than FORGE itself. A small-scale test on a structurally different VLM would make the generality claim more convincing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.