acceptodds
Withdrawn from ICLR 2027

RSI-CUA: Towards Recursive Self-Improvement of Computer-Use Agents in an Endless Training Ground

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

23 views

Abstract

Computer-use agents (CUAs) can improve through environment interaction, but simply accumulating more experience does not determine which experiences best drive progress. As an agent's mastered capabilities and their compositions continually evolve, the central question for sustained improvement is: what should the agent learn next? We present RSI-CUA, a framework that drives recursive self-improvement through an explicit, evolving capability state. In each stage, trajectory-grounded capability profiling analyzes task semantics, interaction traces, and execution outcomes to distinguish capabilities that were successfully exercised, attempted but failed, or unreached. Guided by these identified gaps, RSI-CUA synthesizes new executable tasks calibrated to the learner's empirical capability boundary by balancing complexity, horizon, and composition. Validated tasks accumulate into an expanding Endless Training Ground for policy optimization via rejection-sampling SFT and online RL with GRPO, while a fixed 400-task profiling benchmark, GeneralCUABench, tracks stage-consistent capability progression. Evaluated on Qwen3-VL-8B-Instruct across three successive stages, pass@1 success on GeneralCUABench rises from 3.25% to 8.50%, 11.25%, and 17.75%, with the capability-guided curriculum outperforming an unguided counterpart under a matched training budget. On OSWorld, held out from profiling, synthesis, and policy optimization, the score improves from 18.06% to 25.28%. Capability analysis confirms that these gains stem from better action composition, robust state tracking, artifact persistence, and reliable task termination.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
recursive self-improvement, computer-use agents, reinforcement learning, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SPJMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RSI-CUA studies how to select training tasks as a computer-use agent improves. The framework profiles trajectories on a fixed 400-task benchmark, distinguishes successful, failed, and unreached capabilities, and generates executable tasks targeting the identified gaps. Each stage trains the same base model on the accumulated curriculum using rejection-sampling SFT followed by GRPO. Across three stages, Qwen3-VL-8B-Instruct improves from 3.25% to 17.75% on the profiling benchmark and from 18.06% to 25.28% on held-out OSWorld. The paper also introduces a capability ontology and ablates curriculum selection, optimization components, and cumulative versus sequential training.

Critical strengths

The held-out transfer result is meaningful. OSWorld is excluded from profiling, task construction, and optimization, and performance increases from 18.06% to 25.28%. The paper explicitly acknowledges that GeneralCUABench is part of the training loop and should not be treated as an independent generalization benchmark. This makes OSWorld an important complementary evaluation.

The task-construction pipeline includes substantive evaluation safeguards. Generated tasks specify reproducible initial states and artifact-based assertions, with checks that reject initial states and semantic near misses while accepting valid target artifacts. Combined with trajectory-grounded diagnosis, this is a useful approach to producing targeted training experience with verifiable outcomes.

Critical weaknesses

The experiments do not sufficiently isolate the contribution of recursive capability guidance. The only random-curriculum comparison is at Stage 1, where guidance improves GeneralCUABench from 7.50% to 8.50%, equivalent to four additional solved tasks, and OSWorld from 21.11% to 22.50%. No uncertainty estimates are reported. Later stages expand the training corpus and change the training schedule, but lack corresponding unguided or fixed-profile controls. Consequently, the larger final gains establish that the overall pipeline helps, without establishing how much comes from repeatedly updating the capability diagnosis.

The capability analysis does not directly validate the proposed mechanism. Appendix A.3 defines each capability score as whole-task pass@8 averaged over tasks associated with that capability. A task can contribute to multiple capability nodes, so improving a shared bottleneck such as saving or termination can increase several scores without demonstrating improvement in each named skill. The trajectory-level successful/failed/unreached labels are separate qualitative evidence, and their accuracy is not quantitatively validated. Figure 6 and the case study are suggestive, but do not establish that the gains arise from the specific capability changes claimed.

Questions for authors

What exactly is matched in the random-curriculum comparison: generated task count, successful SFT samples, optimization steps, environment interactions, or total construction and training cost? Does the advantage persist across repeated runs, and is there a later-stage comparison with an unguided or frozen-profile curriculum?

How are trajectory diagnoses aggregated into curriculum specifications, and what concrete execution-feedback rules determine whether a task is simplified, expanded, retained, or discarded? How reliable are the successful/failed/unreached labels against human judgments on a sample of trajectories?

For OSWorld, please specify the evaluated task set, action budget, decoding settings, handling of invalid runs, and whether all Table 1 baselines were rerun under the same protocol. These details matter for interpreting the relatively small differences between methods.

Reviewer_GHFDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes RSI-CUA, a recursive self-improvement system for computer use agents. Within the system, another agent profiles the learner's capabilities and suggests a new curriculum, and the learner is later trained on the synthesized tasks. In particular, RSI-CUA features a ‘capability state’, a graph of semantic skills representing the learner's current abilities and their relationships, to guide better curriculum generation. The trained learner performs better on both the validation and held-out task sets.

Critical strengths
  • Timely and effective framework for recursively improving agents in an interesting domain.
  • Having separate validation/profiling tasks is valuable for both integrating ‘capability state’ seamlessly and for rigorous assessment in experiments.
  • The experiments, including ablation studies, show that many of the design components are important and carefully selected.
Critical weaknesses

1. GeneralCUABench construction and evaluation

1.1. Why are only four domains selected for GeneralCUABench?

1.2. How is the validity of generated tasks verified?

1.3. If not all the generated tasks are verifiable (or usable), what is the proportion of them?

1.4. How is the automated evaluator defined?

1.5. Please include a separate analysis on the validity of defined evaluators.

2. Capability profiling and capability state

2.1. In Figure 6, how is the taxonomy in the right radar defined?

2.2. Elaborate more on the claim that the necessity of capability state is observed in the heterogeneity (stated in line 424). I argue that a comparison with and without capability state is required to validate this claim, in addition to the success-rate comparison in Table 3.

2.3. How do we ensure the validity of profiled capability? In particular, how can we confirm that the capability gap extracted precisely reflects the agents' failure modes?

3. Curriculum synthesis and task generation

3.1. Leakage of profiling benchmark information via profiles while generating new curriculum tasks?

3.2. Add details on the synthesized curriculum tasks. How many tasks are created? How do they differ from the profiling benchmark tasks in quality? Are the agents trained with the accumulated synthesized tasks across multiple stages?

4. Experiments

4.1. If we are using different models for the trainee and curriculum synthesizer, why do you call the framework ‘self-improvement’?

4.2. While the information from OSWorld should not have been used for creating the profiling benchmark, the overlap of tasks between the training/validation tasks (curriculum tasks and the profiling benchmark tasks) vs. test tasks (OSWorld) may and would exist. Please provide this analysis to better understand the gaps between the tasks.

4.3. Explain why RSI-CUA benefited on Thunderbird and Chrome tasks, although the profiling benchmark does not feature those apps.

4.4. Explain why the trained agents performed particularly well on Calc and Impress tasks, compared to other tasks.

5. Ablation studies

5.1. Why did performing both SFT and RL at each stage perform better?

5.2. Why does Table 3 feature inconsistent baselines?

5.3. How do the authors conjecture the superior performance of cumulative training over sequential training?

5.4. Can we also perform an ablation study on the task difficulty calibration?

6. Minor clarifications

I notice that some of these are already presented in the Appendix, but not clarified in the main text. Please clarify them in the main text, for example by referring to the appropriate Appendix section.

6.1. What is j in line 235?

6.2. What are the prompts used for the profiling, aggregation, and synthesis?

6.3. Clarify that S3 in line 328 means stage 3.

6.4. In line 377, what do you mean by ‘measurable gain’?

7. Format

7.1. Citations in the related work and experiment section are wrong, unlike in the intro section.

Questions for authors

See weaknesses.

Reviewer_Bv6yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RSI-CUA presents an agentic pipeline for iterative data construction conditioned on the learner's capabilities and constructs a fixed 400-task diagnostic benchmark across four desktop applications. An external profiling agent analyzes learner trajectories to distinguish successful operations, failed attempts, and operations not reached. An external synthesis agent uses these diagnoses and learner execution feedback to construct and calibrate new verifiable training tasks. Both orchestration agents use GPT-5.6 Luna with the Codex harness; the learner is Qwen3-VL-8B-Instruct. At each of three stages, training restarts from the same base policy and applies rejection-sampling SFT followed by online GRPO on the accumulated curriculum. GeneralCUABench pass@1 increases from 3.25% to 17.75%, and held-out OSWorld performance increases from 18.06% to 25.28%. The paper reports stage-wise training ablations and capability analyses.

Critical strengths
  1. The distinction between failed and unreached operations is sensible. An early navigation failure should not automatically label a downstream editing skill as deficient. The GIMP example illustrates how completing an edit but failing to export calls for a different training intervention (§3.3; Appendix G).

  2. The reported transfer evaluation is outside the diagnostic loop. The authors explicitly exclude OSWorld from profiling, task synthesis, and optimization (§4.1). This makes its improvement more informative about transfer than improvement on GeneralCUABench, which guides the curriculum. These strengths do not resolve the central framing and attribution concerns below.

Critical weaknesses
  1. The central recursive self-improvement claim is not demonstrated, and the actual contribution is insufficiently differentiated from adaptive curriculum generation. Under the interpretation relevant to my assessment, recursive self-improvement requires the improved system to take an active role in improving itself again. Here, the learner executes GUI tasks, while fixed external GPT-5.6 Luna agents perform capability diagnosis and curriculum synthesis (§4.1.3; Appendix C.2). No experiment shows an improved learner acquiring or improving those functions, replacing its external teacher, or becoming a more capable designer of its subsequent improvement. Providing trajectories and success/failure feedback to a fixed external curriculum designer establishes learner conditioning; it does not by itself establish the claimed form of self-improvement.

    Moreover, each stage discards direct continuation of the preceding learner's parameters and retrains the original base on an enlarged curriculum (§3.4; Appendix E.1). The outer data-construction loop is iterative, and information does pass between stages, but this is not evidence of successive generations recursively improving their own improvement capability. The demonstrated mechanism is externally orchestrated adaptive data construction followed by ordinary SFT and RL. Calling this recursive self-improvement materially overstates what the experiments establish.

    Reframed as curriculum generation, the contribution also needs a sharper technical justification. The paper does cite close work: MobileGen / Learning with Challenges already constructs learner capability profiles and uses them to control subsequent training-data difficulty; CUA-Gym already jointly constructs task instructions, executable states, and verifiable rewards. RSI-CUA's more specific proposal is its fixed diagnostic space and trajectory-grounded capability representation. However, the paper does not establish that this representation improves curriculum decisions enough to justify the larger framing. The generated task space is also substantially constrained by preimplemented artifact renderers and assertion catalogs (Appendix C.2): four applications and prescribed output formats, flattened GIMP pixels without layer-state support, no embedded images in the Writer renderer, and at most two dependent semantic changes for the bounded-composition variant. These constraints may help verification, but increasing the number of instances within them does not demonstrate an indefinitely expanding capability space. The evidence supports a bounded synthesis pipeline more directly than the broader “Endless Training Ground” narrative.

  2. The experiments do not separate the proposed mechanism from data scaling and conventional training, and the isolated gain is too limited to substantiate the claimed significance. The only budget-matched Random Curriculum comparison is at Stage 1 (Table 3). Capability guidance improves OSWorld from 21.11% to 22.50%, a difference of 1.39 percentage points; on GeneralCUABench, the gain is 7.50% to 8.50%, only four of 400 tasks. These are limited observed margins, with no repeated training runs or uncertainty estimates. I do not infer statistical insignificance from the margins alone, but their stability and practical value are unestablished. The complete 18.06% to 25.28% gain cannot be attributed to capability guidance, since that comparison also adds the whole SFT/GRPO training pipeline.

    Across stages, the data itself changes substantially: SFT serialized samples increase from 1,858 to 10,565 to 13,037, while RL source tasks increase from 815 to 1,773 to 2,173 (Tables 8–9). RL schedules also change: 4,096, 4,096, and 1,600 slots, respectively. Thus, I am not assuming monotonically increasing RL compute; the problem is that data quantity, data selection, and training exposure are not isolated. Restarting from a common base controls initialization, not these factors. There is no multi-stage budget-matched random or outcome-only curriculum, nor a one-shot generation control matched to the final data and total construction/training budget. The increasing stage scores therefore cannot distinguish a benefit of the claimed recursive mechanism from learning on a larger accumulated synthetic dataset.

    The significance of the result is further limited by testing only one relatively weak starting learner. The final OSWorld score is 25.28%, whereas the EvoCUA project reports 46.1% for its 8B model at 50 steps. Those raw numbers are contextual and cannot establish a fair performance gap without matched protocols. Nevertheless, they make it important to test whether this method adds value to an already capable GUI learner. EvoCUA-8B is included on GeneralCUABench but omitted from the OSWorld main table. The paper does not establish either an advantage over sufficiently trained, budget-matched alternatives or an ability to extend a strong learner's ceiling. Finally, the capability plots do not independently validate the proposed explanation: their node scores average task-level pass@8 over overlapping supporting task sets (Appendix A.3), rather than measuring conditional correctness of the particular operation. Improvements in upstream execution or termination can therefore raise multiple node scores without demonstrating the claimed capability-specific mechanism.

Questions for authors
  1. Which component of the improved learner actively performs or improves its own next-stage diagnosis, curriculum design, or learning process? Is there evidence for that role beyond supplying execution traces to fixed external models? If not, why is recursive self-improvement the appropriate central claim?
  2. Is there an existing full three-stage random or outcome-only curriculum control? These comparisons would materially affect my attribution and significance assessment.
  3. How are capability hypotheses checked for correctness and used to change task selection? Please give the operational calibration rules, success-rate targets or acceptance criteria, candidate rejection rates, and the checkpoint sources of successful SFT trajectories. What evidence shows added value over simpler difficulty- or failure-based conditioning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.