- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RSI-CUA studies how to select training tasks as a computer-use agent improves. The framework profiles trajectories on a fixed 400-task benchmark, distinguishes successful, failed, and unreached capabilities, and generates executable tasks targeting the identified gaps. Each stage trains the same base model on the accumulated curriculum using rejection-sampling SFT followed by GRPO. Across three stages, Qwen3-VL-8B-Instruct improves from 3.25% to 17.75% on the profiling benchmark and from 18.06% to 25.28% on held-out OSWorld. The paper also introduces a capability ontology and ablates curriculum selection, optimization components, and cumulative versus sequential training.
Critical strengths
The held-out transfer result is meaningful. OSWorld is excluded from profiling, task construction, and optimization, and performance increases from 18.06% to 25.28%. The paper explicitly acknowledges that GeneralCUABench is part of the training loop and should not be treated as an independent generalization benchmark. This makes OSWorld an important complementary evaluation.
The task-construction pipeline includes substantive evaluation safeguards. Generated tasks specify reproducible initial states and artifact-based assertions, with checks that reject initial states and semantic near misses while accepting valid target artifacts. Combined with trajectory-grounded diagnosis, this is a useful approach to producing targeted training experience with verifiable outcomes.
Critical weaknesses
The experiments do not sufficiently isolate the contribution of recursive capability guidance. The only random-curriculum comparison is at Stage 1, where guidance improves GeneralCUABench from 7.50% to 8.50%, equivalent to four additional solved tasks, and OSWorld from 21.11% to 22.50%. No uncertainty estimates are reported. Later stages expand the training corpus and change the training schedule, but lack corresponding unguided or fixed-profile controls. Consequently, the larger final gains establish that the overall pipeline helps, without establishing how much comes from repeatedly updating the capability diagnosis.
The capability analysis does not directly validate the proposed mechanism. Appendix A.3 defines each capability score as whole-task pass@8 averaged over tasks associated with that capability. A task can contribute to multiple capability nodes, so improving a shared bottleneck such as saving or termination can increase several scores without demonstrating improvement in each named skill. The trajectory-level successful/failed/unreached labels are separate qualitative evidence, and their accuracy is not quantitatively validated. Figure 6 and the case study are suggestive, but do not establish that the gains arise from the specific capability changes claimed.
Questions for authors
What exactly is matched in the random-curriculum comparison: generated task count, successful SFT samples, optimization steps, environment interactions, or total construction and training cost? Does the advantage persist across repeated runs, and is there a later-stage comparison with an unguided or frozen-profile curriculum?
How are trajectory diagnoses aggregated into curriculum specifications, and what concrete execution-feedback rules determine whether a task is simplified, expanded, retained, or discarded? How reliable are the successful/failed/unreached labels against human judgments on a sample of trajectories?
For OSWorld, please specify the evaluated task set, action budget, decoding settings, handling of invalid runs, and whether all Table 1 baselines were rerun under the same protocol. These details matter for interpreting the relatively small differences between methods.