- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Existing computer-use agent (CUA) benchmarks evaluate isolated tasks and largely ignore what sustained assistance requires: reusing a user's preferences across tasks and revising deliverables in response to feedback. This paper introduces LifelongCUAWorld, a benchmark of 180 professional desktop tasks across 19 applications with GUI/CLI/code interfaces and a structured CLI knowledge base.
The tasks form three suites: Expert (72 independent tasks with verifiable deliverables), Profile (nine-task sequences per user with hidden preferences, a user history and an ask_user tool), and Revision (54 episodes with pre-authored staged feedback and preservation requirements).
On 36-task subsets, ten systems are evaluated with a rule-plus-LLM-judge evaluator: no system strictly completes any Expert task, six systems never ask for clarification, and feedback compliance reaches at most 17.2%.
Critical strengths
- The paper targets an important and under-evaluated setting: professional multi-application work (92% of Expert tasks span several applications) combined with cross-task preference reuse and feedback-driven revision. Existing benchmarks cover only parts of this (Table 8).
- The evaluator is carefully designed. Rule checks gate the rubric-based content judgments, and Revision scores feedback compliance and preservation of previously correct content separately on saved snapshots (Eq. 2).
- The ablations and diagnostics are informative: interfaces and CLI knowledge (Table 5), supplied user information (Table 6), the revision protocol (Table 7), and judge stability and veto sensitivity (Tables 22, 24, 25).
Critical weaknesses
- Collaboration metrics are dominated by execution failure. Nearly all drafts fail the initial-stage check (DFR –, Table 2), and even an oracle draft reaches only FCR (Table 7). PCR is non-zero only in the Research track for eight of the ten systems (Table 15). The paper itself states that PCR "does not isolate preference understanding from professional delivery" (App. E.4), so the headline numbers (PCR , FCR ) cannot be attributed to preference learning or revision ability.
- No evidence that tasks are solvable under the strict criterion. No system passes any Expert or Profile task (Table 2; App. C.5). Yet the paper reports no human baseline and does not show that reference solutions reach under the full evaluator; App. H.2 only tests rule checkpoints on reference artifacts. Without this, a 0% pass rate cannot be distinguished from overly strict thresholds or evaluator errors.
- The content judge is not validated against human judgments, and judge choice matters. Alternative judges agree with the recorded Gemini-3.1-Pro verdicts on only 46.9–75.6% of checkpoints (–) and pass 47–74% of them versus 25% (Table 24). Profile rankings also shift (Kendall's as low as , Table 25). The main text reports only same-judge repeatability (Sec. 3.4), and human agreement is left to future work (App. A).
- The "lifelong learning" and "computer-use" framings are overstated. Cross-task memory is supplied by the protocol, which appends prior exchanges to the agent's context (Algorithm 1). The evaluated sequences have only six to nine tasks, and the three suites are disjoint, so the benchmark tests in-context reuse rather than lifelong learning. In addition, GUI-only execution scores , and CLI-only execution without the knowledge base () already matches the default hybrid setting () (Table 5).
- Comparisons across systems are confounded. The scaffold for the model baselines (action and observation spaces) is not described, and settings differ across systems: 3–20 screenshots of history, temperature –, and different reasoning efforts (Table 23). DeepSeek Harness also uses native context management and more than the input tokens (Table 13), so the claim that "a suitable agent harness" helps (Sec. 3.2) is not isolated from these differences. Finally, FCR, PS and CPR have system-dependent denominators (e.g., CPR is computed over 3 to 45 units, Table 17), so they are not comparable across systems.
Questions for authors
- Do the reference solutions for Expert and Profile tasks reach under the full evaluator, including the content judge? How long do human experts take to complete these tasks?
- How were the task-specific thresholds and the critical checks chosen? How many Expert tasks would pass under the threshold alone?
- Why do the main experiments use only 108 of the 180 tasks, and will results on the full suites be reported?
2 comments
Thank you for the additional experiments and revisions. The human baseline and expert validation of the content judge substantially address my concerns about solvability and evaluation reliability. The revised PCR, FCR, and PS definitions also improve the evaluation. My remaining concerns are about what the benchmark measures and what the comparisons establish.
The history-retention experiment supports preference reuse within short sequences, but does not establish lifelong learning. The protocol supplies the retained history, and the evaluation does not test long-term memory maintenance or adaptation to changing preferences. The paper’s framing should be narrowed accordingly.
The same-model harness comparisons are useful, but the Execution gains come with higher token use. They establish differences between complete systems under their respective configurations, rather than isolating the contribution of harness design.
CPR also remains difficult to compare across systems. Each system is evaluated on the content it previously got right, so both the amount and difficulty of that content can differ. Expressing preservation as a percentage does not resolve this selection effect.
The revisions strengthen the paper, but the gap between the lifelong-learning framing and the capabilities actually evaluated remains central to my assessment. Together with the remaining limitations of the system comparisons, this keeps me at 2 (Weak rejection).
Response to Reviewer UDiw
Thank you for your feedback.
W1. Execution and collaboration metrics. PCR now scores each annotated preference from artifact evidence, removing dependence on complete-delivery checkpoint scores (Section 2.5). Eleven of 12 systems have nonzero PCR across all six roles (Appendix E.3, Table 19). In the DeepSeek Harness ablation, prior answers raise PCR from 0.875 to 0.926 while PTS falls from 0.268 to 0.193 (Table 7). An accepted initial draft raises Revision FRS from 0.113 to 0.278, while FCR stays at 0.224 (Table 8). These results distinguish preference application and feedback satisfaction from overall delivery quality. Fixed denominators are discussed in W5 below.
W2, Q1. Task solvability and human completion time. Six graduate students familiar with the software attempted 48 sampled tasks once each (18 Execution, 12 Profile, and 18 complete Revision episodes). Of these, 37 passed the full evaluator, including content checks (77.1%; Appendix D.2.1). Completion times were approximately 75, 90, and 125 minutes for the three suites, respectively (Section 3.1). Agents also achieve strict completion and earn partial credit on many tasks (Table 2, Appendix E.2).
W3. Content evaluation and judge stability. Gemini-3.1-Pro agrees with 98.61% of human expert labels on 216 artifact packages (PASS F1 97.09%; Section 3.5, Table 5; sampling in Appendix D.2.2). In a separate experiment on 160 checkpoints, repeated evaluation gives 95.6% agreement and suite-score changes below 0.005. With deterministic rules and prerequisite gates fixed, alternative judges produce maximum score changes of approximately 0.022 in Execution/Profile and at most 0.003 in Revision FRS. Some nearby rankings change, but the leading system in each evaluated suite remains unchanged (Appendix D.3, Table 14). Rule checks cover required counts, dimensions, and fields (Appendix D.1).
W4. Lifelong learning and computer use. The benchmark evaluates capabilities needed for lifelong assistance through finite task sequences covering professional execution, preference learning, and feedback-driven revision (Sections 2.1–2.4). Profile retains answers to agents' questions for use in later tasks. GUI, CLI, and code support professional desktop workflows. Both GUI-only configurations score zero. The added DeepSeek Harness ablation shows that hybrid GUI/CLI execution scores higher than CLI-only execution, with further gains from task-relevant CLI knowledge (Table 6).
W5. System comparisons and metric denominators. Following OSWorld 2.0, we evaluate complete agent systems on shared tasks and scoring rules. Each system includes its model, tools, and harness logic, including context management. We retain system-specific context settings and assess the combined performance. Model baselines use a shared framework with model-specific adapters (Appendices E.1 and H). Using the same DeepSeek-V4.1-Flash model, DeepSeek Harness achieves higher Profile PTS than Claude Code and Codex while using fewer input tokens per task on average (Tables 2 and 24).
We updated FCR/PS definitions and results to use fixed denominators covering all applicable requirements, with unreached requirements scoring zero (Section 2.5, Tables 2 and 8). CPR measures how reliably agents preserve previously correct content, expressed as a percentage to account for differing amounts available to preserve.
Q2. Thresholds and critical checks. Task-level thresholds θτ, aggregation weights, and critical checks are fixed before evaluation (Section 2.5). Critical checks follow explicit mandatory requirements, such as required text, dimensions, fields, and application states. At the content-checkpoint level, manual artifact inspection informed the rubric anchors (0.25/0.50/0.75/1.00, with zero for missing or invalid content). Acceptance requires a weighted score ≥0.80 and every critical dimension ≥0.75, preventing high scores from masking critical weaknesses (Appendix D.1.1). Content-judge stability is reported in W3 above.
Under the task-score threshold S(τ)≥θτ alone (Eq. 1), DeepSeek Harness passes 2/36 Execution tasks and all other systems pass 0/36, matching strict completion (2/432 overall).
Q3. Evaluated subsets. Execution time and evaluation costs limited the main comparison to 108 tasks. Appendices E.2–E.4 describe task selection and sampling. We will report results on all 180 tasks in future work.
Thank you again for your detailed suggestions. We hope these responses address your concerns and would be grateful if you would consider raising your score.