- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MyPCBench evaluates computer-use agents in a personalized desktop environment populated with one coherent user’s cross-application history. Its main challenge is long-horizon navigation and reasoning over personal data across applications, with tasks requiring inference of user habits or preferences. However, because the benchmark uses only one persona, it evaluates personalized-environment reasoning may more strongly than generalizable personalization across different users.
Critical strengths
- The benchmark provides a highly detailed and internally consistent personal desktop environment.
Critical weaknesses
- It is unclear whether MyPCBench truly evaluates personalization, rather than long-horizon reasoning over personal data. The benchmark uses only one persona, Michael Scott, so it does not test whether agents can adapt across users with different preferences, routines, and social structures. Moreover, many tasks focus on lookup, aggregation, reconciliation, and multi-app orchestration, which weakens the central claim that MyPCBench is specifically a benchmark for “personally intelligent” agents.
- The benchmark-construction novelty appears more incremental than the framing suggests. The main distinctive design is a dense, cross-application personal history generated from one persona specification. This is a substantial engineering effort, but the paper itself acknowledges related benchmarks that already seed coherent user identities or personalized device states. Therefore, the main novelty seems to lie in scale, density, and cross-app consistency rather than in a new personalization formulation.
Questions for authors
- How do the authors distinguish the capability measured by MyPCBench from long-horizon, cross-application reasoning over stateful personal data? In particular, what evidence shows that benchmark performance reflects personalization rather than retrieval, aggregation, and orchestration difficulty? Please compare your paper with OSWorld2 etc paper to explain.
- What about the cost to build the benchmark and how much and when human involve?
- Since the benchmark uses only one persona, how should readers interpret claims about “personally intelligent” agents? Have the authors tested whether model rankings or failure modes remain stable across additional personas with substantially different routines, preferences, and social structures?
3 comments
Thank you for you detailed clarification
Thank you for you detailed clarification. However, I still don't understand how inferred habits is related to the task in the bench as your bench seems to only provide few metrics(prefect-task rate, rubric score). Can you dive into real persona evaluation with more metrics? My overall assessment of the paper remains unchanged, and I will maintain my original rating.
Reply to Reviewer DUn
Thank you for your response. We answer both points directly.
How inferred habits relate to the tasks. Habits are evaluated in MyPCBench in two key ways. First, 11 tasks are pattern-inference tasks (§4, task-type table): the agent must infer a habit or preference that is never stored anywhere and exists only as a pattern in the user's records. For example, "What do I usually tip on food delivery, in dollars and as a percent?" requires looking through patterns that exist in the user data. The agent must read a user's full order history (over 300 HangryDash orders) and compute the habitual tipping pattern. Second, habit evaluations also appear inside other tasks, for example when a request depends on the restaurant the user orders from most. Across MyPCBench, 27 rubric items grade a habit or preference directly, so these items measures the habit-inference accuracy of personal agents.
Persona-specific metrics. Beyond perfect rate and rubric score, the same leaderboard runs give per-model metrics that isolate personalization:
| Model | Pattern-inference tasks perfect | Habit-item accuracy (27 items) | Other-item accuracy | Persona-data hallucination (failed tasks) | Skipped-required-app failures |
|---|---|---|---|---|---|
| Opus 4.6 | 9/11 | 88.9% | 83.4% | 6 | 39 |
| GPT-5.6-sol | 7/11 | 92.6% | 82.3% | 8 | 47 |
| GPT-5.6-luna | 10/11 | 92.6% | 80.2% | 8 | 40 |
| Sonnet 4.6 | 8/11 | 76.9% | 75.8% | 3 | 56 |
| Qwen 3.5 35B | 0/11 | 33.3% | 41.3% | 13 | 83 |
| Qwen 3.5 9B | 0/11 | 14.8% | 22.1% | 8 | 128 |
Frontier agents infer habits as accurately as much as they complete other items, while weaker agents fall further behind on habit items (14.8–33.3%) than on the rest. Persona-data hallucination counts failed tasks in which the agent asserted something not in the user's records, and skipped-required-app failures count tasks that end before every app holding the user's relevant records was visited. Both are personalization failures beyond the single metric of task success.
Persona-level evaluation. The same habit question was asked for each of the three personas we generated (concerning their average tip), and the correct answer differs by person: about USD 3.04 and 20.2% for Jim Halpert, and about USD 3.55 and 20% for Dwight Schrute, each computed from that person's own order history. GPT-5.6-sol, Opus 4.6 and Muse Spark 1.3 answer it correctly for all three people. Because the tip pattern changes with each persona, a model can only succeed by inferring the habit from that user's records, not by relying on a fixed prior or memorized information.
Thank you for your feedback! Let us know if this addresses your concerns.
Response to Reviewer DUno
We thank the reviewer for their thoughtful review and appreciation of MyPCBench's highly detailed and consistent personal desktop environment.
We are happy to address your questions below:
W1 and Q1. Personalization versus long-horizon reasoning over personal data
We measure personal-context grounding, whhere the agent must act correctly from one specific user's state and history inside their own logged-in apps. Retrieving and aggregating records from that history is part of this skill, since benchmarks that put the user profile in the prompt remove that step. Preference inference is one slice of this capability (11 tasks).
OSWorld 2.0 draws the same distinction. It names MyPCBench as one of its two closest points of comparison and writes that MyPCBench "centers evaluation on a coherent personalized desktop state, whereas OSWorld 2.0 places greater stress on much longer workflow execution." OSWorld 2.0 seeds a task-specific workspace for each task, and Opus 4.7 averages 318 tool calls per task there. MyPCBench seeds one persistent history shared by all tasks; Opus 4.6 averages 46.5 steps per task. MyPCBench holds different depth in personalization tasks due to the volume of seeded data and interconnectivity between apps.
We tested this in two ways. First, at a fixed count of 2–3 apps, tasks that require recovering facts from the user's history separate frontier from weaker agents (48–80% vs. 3–13% perfect), while bounded actions separate them less (56% vs. 24%). App count and horizon are held fixed here, so the difference comes from what the task asks of the user's history. Second, when the same requests are rewritten for two other people's histories, every model scores lowest on the same persona (Q3).
W2. Novelty
We respectfully disagree that these properties are incremental, because they make a new kind of task possible: requests whose correct answer depends on a year of one user's history spread across a full desktop. Inferring a tipping habit needs hundreds of delivery orders, checking whether two trips are affordable needs bank, airline, rental and email records that agree, and planted contradictions only test cross-checking if everything else is consistent. An environment seeded sparsely or per task cannot pose these requests, and desktop benchmarks that start from a default state do not include them.
Q2. Cost and human involvement
Building MyPCBench took two Claude Max subscriptions for three months plus two authors. Humans selected the requests from the 2,749 candidates, specified the persona, reviewed the app clones, validated every task and rubric, and did the blind judge grading. Evaluating the new persona subset cost US$79 for GPT-5.6-sol and US$112 for Opus 4.6. Extending MyPCBench is cheap as we release an open-source generator which ships 16 possible persona specifications and creates new ones with one command, which we used to build Jim and Dwight for this rebuttal.
Q3. One persona, ranking and failure-mode stability
We tested cross-user transfer directly (table in the general response). On 9 matched tasks across Michael, Jim and Dwight, every model scores between 0.89 and 1.00 rubric on every persona, and no task scores below 0.5 for any model. Persona difficulty is consistent: GPT-5.6-sol, Opus 4.6 and Muse Spark 1.3 all have their lowest perfect count on Dwight. This subset is near ceiling for current frontier models (66 of 80 runs perfect), so it tests transfer across users.