- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes GUI-Evolver, an iterative data–model loop for mobile GUI agents. A State Model Graph (SMG), built by visual BFS over 20 AndroidWorld apps and continually updated from training interactions, feeds two cumulative data streams: graph-derived GUI knowledge QA (IGKS) and failure-directed remedial subtasks. Each round continues the same LoRA adapter on all accumulated data, and a Validation set selects the checkpoint. Smaller models receive teacher-assisted intervention, while the 27B model uses diagnosis-guided self-rerollouts, which the authors motivate as “supervision compatibility” with its long CoT. Across five Qwen models, the method reports gains of 9.9–25.0 points on AndroidWorld and 4.8–13.1 on MobileWorld over self-measured baselines. Controlled experiments on 9B and 27B are offered to attribute these gains to knowledge supervision, targeted practice, persistent memory, and the choice of supervision source.
Critical strengths
The paper does more than report end-to-end gains. It includes a 2×2 factorial on knowledge vs. task increments, an acquisition comparison (reruns vs. ordinary synthesis vs. failure-directed subtasks), and a repair cohort that reports operation reach and conditional operation success. The last one is a sensible way to tell genuine bottleneck repair apart from “easier practice.” If the numbers were reliable, this design would make the mechanism claims testable rather than anecdotal.
Improvements appear across five model sizes on both benchmarks. Checkpoints are selected by Validation rather than benchmark maxima, and all rounds, including regressions, are reported. This lowers the risk of cherry-picking at the checkpoint level.
Critical weaknesses
The results are hard to trust. Several reported means cannot come out of 116 or 117 tasks averaged over three runs. Every Validation score is a round integer, and the per-round knowledge increments shrink by the exact same ratios for all five models, which is not what independent runs look like. There is no variance and no training seeds, and many ablation gaps amount to one or two tasks, so the controlled comparisons can’t support the mechanism claims. Training tasks, the graph and the AndroidWorld test set also share the same 20 apps. The paper never says whether they share task templates or where the Validation set comes from, so the AndroidWorld gains may just be in-distribution fitting. Since each component already exists, the paper needs a plain iterative self-training baseline to show the added machinery matters, and it doesn’t have one.
The paper defends itself instead of explaining. Almost every sentence ends with a disclaimer of what the result does not show, which reads more like generated text tuned to preempt reviewers than authors describing their work. The method is scattered across the appendix, the only worked example is hypothetical, and no real trajectory or failure case appears anywhere. The overview figure is AI-generated, as the AI usage statement admits: it is cluttered, nearly illegible, and shows an “illustrative” phone screen rather than the actual system. For a systems paper whose credibility depends on showing the pipeline at work, this is a serious gap.
Questions for authors
Clarifications:
Do the training tasks or Validation100 share task templates with AW116? How was Validation100 built?
Please provide raw per-task, per-run results that reproduce Tables 1, 2, 14 and 15.
Please also explain why all Validation means are integers and why the IGKS increments decay identically across models.
Suggestion: Add a matched-budget iterative RFT baseline, and report results over multiple training seeds.