- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents FORGE, a feedback-driven data generation framework for GUI agent post-training. It pairs synthesized tasks with executable evaluators and uses generation errors and model failures to refine validation skills and subsequent task production. The resulting tasks and trajectories support both SFT and RL, yielding substantial improvements on OSWorld across two Qwen3-VL model sizes.
Critical strengths
- The paper presents a well-designed feedback-driven data evolution framework that continuously refines data quality and task coverage based on observed failures. The experiments demonstrate consistent performance improvements across iterations, supporting the effectiveness and scaling of this approach.
- Instead of directly relying on VLM judges for task evaluation, the framework employs executable evaluators with a multi-stage validation process to mitigate potential judging biases and provide more consistent and reliable supervision for training.
Critical weaknesses
- The framework relies heavily on LLMs for task generation, validation, repair, and feedback analysis. The paper lacks a systematic analysis of the associated costs and whether the improvements justify the additional overhead compared to simpler data generation strategies.
Questions for authors
- Can the authors provide a cost-efficiency comparison with simpler static or adaptive data generation strategies under comparable budgets?
- Since both the generation and refinement of executable evaluators rely on LLMs, could biases or errors propagate into the reward signals? Did the authors observe any reward hacking or training instability during RL, and how were these issues monitored or mitigated?
- Free-composition exploration primarily recombines existing behaviors, how can FORGE discover capabilities that are entirely absent from the initial taxonomy?
2 comments
Thank you for the rebuttal. I looking forward to your revised manuscript and confim my original score.
Response to Reviewer DckX: Cost-Efficiency, Evaluator Reliability, and Behavior Discovery
Thank you for recognizing the value of FORGE’s feedback-driven data evolution, the consistent improvements across iterations, and the use of executable evaluators with multi-stage validation. We address your questions below.
First, our initial version, v0, uses a static data generation pipeline without self-evolution and achieves limited improvements. It therefore provides a direct reference for assessing the benefits of introducing feedback. Section 4.3 compares data from different iterations while keeping the backbone, SFT dataset size, and training configuration fixed. Table 12 further examines the contributions of the two self-evolution components. Our cost measurements show that introducing dynamic feedback increases total data production costs by approximately 10%–20% relative to the static pipeline. Data synthesis remains the dominant cost. Together with the iteration results and component ablations, these measurements indicate that self-evolution improves the training utility of the generated data at a moderate additional cost. We will include the cost breakdown in the revision and clearly distinguish data production costs from downstream training costs.
Second, we agree that executable evaluators do not automatically guarantee correct rewards. LLM-generated evaluators may omit task requirements, impose overly permissive or restrictive conditions, or contain implementation errors. FORGE therefore subjects generated tasks and evaluators to programmatic checks, semantic checks, and real-environment validation, with repair where necessary. Recurring generation errors are also distilled into reusable validation skills. For SFT data collection, we additionally filter trajectories using a reward threshold. The human audit reported in the appendix independently examines task instructions, execution trajectories, and final states, and compares human judgments against evaluator outputs. We sample trajectories that evaluators label as both successful and unsuccessful to identify false positives and false negatives. The audit found a low proportion of false positives, providing direct empirical evidence that the evaluator validation pipeline reduces the risk of incorrect reward signals. In earlier training runs, we did encounter reward hacking or training instability, which motivated targeted constraints in our reward design. Specifically, F_t checks action-format validity and penalizes noncompliant outputs, while L_t penalizes repeated actions. The latter reduces unproductive repetition and indirectly limits unnecessary trajectory growth caused by such behavior. The appendix provides the definitions and related ablations.
Third, it is useful to distinguish free-composition task generation from subsequent failure diagnosis. Free composition initially samples from the existing behavior library to construct task combinations that existing templates do not cover. Expansion of the behavior library occurs during the subsequent feedback analysis. A task composed of known behaviors may require an intermediate operation that is not yet represented in the library. Failure analysis distinguishes among failures to execute an existing behavior, missing composition templates, and missing atomic behaviors. For missing atomic behaviors, the pipeline adds new behavior entries; for gaps in how existing behaviors are combined, it updates the composition templates. Section 3.3 and Algorithm 1 describe these updates. Tasks generated using the new entries must still pass the task validation pipeline. Thus, FORGE’s exploration is not permanently restricted to its initial taxonomy. Execution feedback can reveal missing operations and guide the expansion of the behavior library. This discovery depends on the tasks explored and the failures encountered; we do not claim that FORGE can discover every unknown capability unrelated to its exploration.
Thank you again for these constructive questions. We will add the cost analysis and clarify evaluator reliability and the scope of behavior discovery in the revised manuscript.