acceptodds
Under review as a conference paper at ICLR 2027

MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

Abstract

Current benchmarks for computer-use agents evaluate models in impersonal environments. This leaves a gap between evaluation and deployment where personal assistants are expected to work across a user’s whole digital life, including their context, historical data, and logged-in accounts. This gap is widest on web tasks, where live web evaluations cannot exercise sites that require logging in or personal information, the kind of site a real personal assistant has to drive. We introduce MYPCBENCH, which tests computer-use agents as personal assistants on a Linux desktop populated with 17 simulated real-world web applications and a full desktop stack, all seeded for one canonical persona, Michael Scott from The Office. We define 184 tasks in this environment, each inspired by a real request drawn from the OpenClaw community, and benchmark ten closed- and open-weight models, each given the same computer and shell tools. The highest perfect rate is 58.2% (Claude Opus 4.6), with GPT-5.6-sol at 55.4% and a slightly higher rubric score. We use LLM-as-a-Judge to verify tasks, where our rubric judge agrees with human grading on 94.5% of sampled rubric items. Model failures cluster on tasks that span many applications and on long trajectories, where personalization stresses an assistant the most. We release the environment, task set, agent harness, and judge anonymously at https://anonymous.4open.science/r/MyPCBench-21FF/.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Personal Assistants, Computer-Use Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DUnoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MyPCBench evaluates computer-use agents in a personalized desktop environment populated with one coherent user’s cross-application history. Its main challenge is long-horizon navigation and reasoning over personal data across applications, with tasks requiring inference of user habits or preferences. However, because the benchmark uses only one persona, it evaluates personalized-environment reasoning may more strongly than generalizable personalization across different users.

Critical strengths
  1. The benchmark provides a highly detailed and internally consistent personal desktop environment.
Critical weaknesses
  1. It is unclear whether MyPCBench truly evaluates personalization, rather than long-horizon reasoning over personal data. The benchmark uses only one persona, Michael Scott, so it does not test whether agents can adapt across users with different preferences, routines, and social structures. Moreover, many tasks focus on lookup, aggregation, reconciliation, and multi-app orchestration, which weakens the central claim that MyPCBench is specifically a benchmark for “personally intelligent” agents.
  2. The benchmark-construction novelty appears more incremental than the framing suggests. The main distinctive design is a dense, cross-application personal history generated from one persona specification. This is a substantial engineering effort, but the paper itself acknowledges related benchmarks that already seed coherent user identities or personalized device states. Therefore, the main novelty seems to lie in scale, density, and cross-app consistency rather than in a new personalization formulation.
Questions for authors
  1. How do the authors distinguish the capability measured by MyPCBench from long-horizon, cross-application reasoning over stateful personal data? In particular, what evidence shows that benchmark performance reflects personalization rather than retrieval, aggregation, and orchestration difficulty? Please compare your paper with OSWorld2 etc paper to explain.
  2. What about the cost to build the benchmark and how much and when human involve?
  3. Since the benchmark uses only one persona, how should readers interpret claims about “personally intelligent” agents? Have the authors tested whether model rankings or failure modes remain stable across additional personas with substantially different routines, preferences, and social structures?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DUno

We thank the reviewer for their thoughtful review and appreciation of MyPCBench's highly detailed and consistent personal desktop environment.

We are happy to address your questions below:

W1 and Q1. Personalization versus long-horizon reasoning over personal data

It is unclear whether MyPCBench truly evaluates personalization, rather than long-horizon reasoning over personal data.

We measure personal-context grounding, whhere the agent must act correctly from one specific user's state and history inside their own logged-in apps. Retrieving and aggregating records from that history is part of this skill, since benchmarks that put the user profile in the prompt remove that step. Preference inference is one slice of this capability (11 tasks).

Please compare your paper with OSWorld2 etc paper to explain.

OSWorld 2.0 draws the same distinction. It names MyPCBench as one of its two closest points of comparison and writes that MyPCBench "centers evaluation on a coherent personalized desktop state, whereas OSWorld 2.0 places greater stress on much longer workflow execution." OSWorld 2.0 seeds a task-specific workspace for each task, and Opus 4.7 averages 318 tool calls per task there. MyPCBench seeds one persistent history shared by all tasks; Opus 4.6 averages 46.5 steps per task. MyPCBench holds different depth in personalization tasks due to the volume of seeded data and interconnectivity between apps.

What evidence shows that benchmark performance reflects personalization rather than retrieval, aggregation, and orchestration difficulty?

We tested this in two ways. First, at a fixed count of 2–3 apps, tasks that require recovering facts from the user's history separate frontier from weaker agents (48–80% vs. 3–13% perfect), while bounded actions separate them less (56% vs. 24%). App count and horizon are held fixed here, so the difference comes from what the task asks of the user's history. Second, when the same requests are rewritten for two other people's histories, every model scores lowest on the same persona (Q3).

W2. Novelty

The main novelty seems to lie in scale, density, and cross-app consistency rather than in a new personalization formulation.

We respectfully disagree that these properties are incremental, because they make a new kind of task possible: requests whose correct answer depends on a year of one user's history spread across a full desktop. Inferring a tipping habit needs hundreds of delivery orders, checking whether two trips are affordable needs bank, airline, rental and email records that agree, and planted contradictions only test cross-checking if everything else is consistent. An environment seeded sparsely or per task cannot pose these requests, and desktop benchmarks that start from a default state do not include them.

Q2. Cost and human involvement

What about the cost to build the benchmark and how much and when human involve?

Building MyPCBench took two Claude Max subscriptions for three months plus two authors. Humans selected the requests from the 2,749 candidates, specified the persona, reviewed the app clones, validated every task and rubric, and did the blind judge grading. Evaluating the new persona subset cost US$79 for GPT-5.6-sol and US$112 for Opus 4.6. Extending MyPCBench is cheap as we release an open-source generator which ships 16 possible persona specifications and creates new ones with one command, which we used to build Jim and Dwight for this rebuttal.

Q3. One persona, ranking and failure-mode stability

Have the authors tested whether model rankings or failure modes remain stable across additional personas with substantially different routines, preferences, and social structures?

We tested cross-user transfer directly (table in the general response). On 9 matched tasks across Michael, Jim and Dwight, every model scores between 0.89 and 1.00 rubric on every persona, and no task scores below 0.5 for any model. Persona difficulty is consistent: GPT-5.6-sol, Opus 4.6 and Muse Spark 1.3 all have their lowest perfect count on Dwight. This subset is near ceiling for current frontier models (66 of 80 runs perfect), so it tests transfer across users.

Reviewer_DUnoOpenReview ↗

Thank you for you detailed clarification

Thank you for you detailed clarification. However, I still don't understand how inferred habits is related to the task in the bench as your bench seems to only provide few metrics(prefect-task rate, rubric score). Can you dive into real persona evaluation with more metrics? My overall assessment of the paper remains unchanged, and I will maintain my original rating.

AuthorsOpenReview ↗

Reply to Reviewer DUn

Thank you for your response. We answer both points directly.

How inferred habits relate to the tasks. Habits are evaluated in MyPCBench in two key ways. First, 11 tasks are pattern-inference tasks (§4, task-type table): the agent must infer a habit or preference that is never stored anywhere and exists only as a pattern in the user's records. For example, "What do I usually tip on food delivery, in dollars and as a percent?" requires looking through patterns that exist in the user data. The agent must read a user's full order history (over 300 HangryDash orders) and compute the habitual tipping pattern. Second, habit evaluations also appear inside other tasks, for example when a request depends on the restaurant the user orders from most. Across MyPCBench, 27 rubric items grade a habit or preference directly, so these items measures the habit-inference accuracy of personal agents.

Persona-specific metrics. Beyond perfect rate and rubric score, the same leaderboard runs give per-model metrics that isolate personalization:

ModelPattern-inference tasks perfectHabit-item accuracy (27 items)Other-item accuracyPersona-data hallucination (failed tasks)Skipped-required-app failures
Opus 4.69/1188.9%83.4%639
GPT-5.6-sol7/1192.6%82.3%847
GPT-5.6-luna10/1192.6%80.2%840
Sonnet 4.68/1176.9%75.8%356
Qwen 3.5 35B0/1133.3%41.3%1383
Qwen 3.5 9B0/1114.8%22.1%8128

Frontier agents infer habits as accurately as much as they complete other items, while weaker agents fall further behind on habit items (14.8–33.3%) than on the rest. Persona-data hallucination counts failed tasks in which the agent asserted something not in the user's records, and skipped-required-app failures count tasks that end before every app holding the user's relevant records was visited. Both are personalization failures beyond the single metric of task success.

Persona-level evaluation. The same habit question was asked for each of the three personas we generated (concerning their average tip), and the correct answer differs by person: about USD 3.04 and 20.2% for Jim Halpert, and about USD 3.55 and 20% for Dwight Schrute, each computed from that person's own order history. GPT-5.6-sol, Opus 4.6 and Muse Spark 1.3 answer it correctly for all three people. Because the tip pattern changes with each persona, a model can only succeed by inferring the habit from that user's records, not by relying on a fixed prior or memorized information.

Thank you for your feedback! Let us know if this addresses your concerns.

Reviewer_7HKROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MyPCBench evaluates personally intelligent computer-use agents in a reproducible Linux desktop containing 17 simulated consumer applications, desktop tools, and cross-linked records for one fictional persona. The benchmark contains 184 tasks inspired by community requests, covering lookup, actions, historical inference, reconciliation, reporting, and multi-application workflows. Ten provider-specific agent configurations are evaluated with computer and shell tools using a trajectory-based rubric judge. The paper also reports tool-access ablations, human–judge validation, paired comparisons, and run-variance analyses. I inspected the released 184-task rubric file and sampled tasks across its nine source categories.

Critical strengths
  1. The coherent, densely populated cross-application environment is a useful contribution beyond isolated task fixtures. The inspected tasks include genuine dependencies on historical records and coordinated side effects, such as computing delivery-tip patterns, reconciling flight history against financial and email records, and organizing a multi-application event. The release contains all 184 task definitions and 1,191 rubric items, making their design inspectable.

  2. The validation is more substantive than a leaderboard alone. The authors report end-to-end completion of every task with independent verification, document the common tools and provider-specific harnesses, and distinguish task-sampling uncertainty from execution variance. Human–judge checks, repeated judging, tool-access controls, and explicit paired tests appropriately temper claims about the leading agents. On balance, I find the environment and task collection a useful resource and lean toward acceptance.

Critical weaknesses
  1. I would appreciate broader calibration evidence for task quality and rubric accuracy. The end-to-end author review is valuable, but the paper does not provide timed human statistics or inter-annotator agreement. The judge validation covers 36 trajectories from one agent, with 83.3% agreement on perfect-task labels; those labels also vary across judge and agent reruns. Some inspected rubric items specify interaction or presentation details beyond the request’s explicit outcome, such as a particular calendar view or intermediate UI inspection. These may be reasonable process requirements, but clarification and a systematic audit would help establish that valid alternative solutions are treated consistently under the shared GUI-plus-shell setting. This is a calibration concern, rather than a claim that the task set is generally defective.

  2. The evidence for personalization is narrower than the breadth of the environment. All reported results use one public fictional persona, and many tasks combine general retrieval or orchestration with personal records. The persona-prior audit is useful but does not isolate the benefit of coherent history from generic task difficulty. A controlled change of persona or history, or a matched ablation of relevant historical context, would better establish what additional capability this benchmark measures. The current results are informative for this environment, but generalization across users remains open.

Questions for authors
  1. Could you elaborate on task-review coverage, human completion time and success statistics, and how task/rubric disagreements were adjudicated? A stratified audit across task types and model families, with overlapping human annotations, would help quantify grader accuracy beyond the current single-agent sample.

  2. How are outcome requirements distinguished from mandatory interaction steps in the rubrics? Can you report calibration on valid GUI, API, and shell alternatives, and clarify when a correct persistent result should receive credit despite a different path?

  3. How reproducible are tasks requiring delayed LLM-generated contact replies? Please specify replay controls, fixed configuration, and whether response variability changes solvability or graded outcomes.

  4. Could you provide a controlled persona/history intervention or a small second-persona evaluation to distinguish personal-context use from general cross-app competence?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7HKR (1/2)

We thank the reviewer for their thoughtful review. We appreciate that they found our densely populated, cross-app personal desktop environment useful, our tasks genuinely dependent on the user's past records, our release of all 184 tasks easy to inspect, and our thorough quality control: end-to-end completion of every task with independent verification, human–judge comparisons, repeated judging and paired tests.

We are happy to address your questions and suggestions below:

W1 and Q1. Review coverage, human completion and adjudication

Could you elaborate on task-review coverage, human completion time and success statistics, and how task/rubric disagreements were adjudicated?

Two authors independently reviewed all 200 first-round entries (198 tasks). Their approve-versus-change decisions agreed on 74.5% of entries (Cohen's κ=0.42). Both approved 113 tasks as-is, and 14 were deleted. Most disagreements (40 of 51) were one reviewer requesting a rewrite the other did not, and in every such case the rewrite was adopted, so the stricter judgment won. Rewritten tasks were re-checked by a per-task LLM audit and a final author pass, and GPT-5.6-sol audited the alignment between each instruction and its rubric to make sure nothing is asked for that the task did not request. For human completion, authors with expert knowledge of the environment completed all 184 tasks end-to-end on the live VM, with a second author independently verifying every required action and rubric (§4.1). These sessions doubled as QA and were not timed.

A stratified audit across task types and model families, with overlapping human annotations, would help quantify grader accuracy beyond the current single-agent sample.

We ran the stratified, overlapping study you suggested. Two annotators, blind to the judge, independently graded the same 24 trajectories from GPT-5.6-sol, Opus 4.6 and Muse Spark 1.3, across three personas and eight of the nine task categories (150 items). Half of the trajectories were chosen for containing judge-failed items, so failures make up 28% of items:

PairAgreementκAgreement on fails
Annotator 1 vs. judge80.7%0.550.69
Annotator 2 vs. judge81.3%0.550.68
Annotator 1 vs. annotator 280.7%0.560.70
Annotator consensus vs. judge (121 items)88.4%0.710.79

The judge agrees with each annotator as often as the annotators agree with each other. On the randomly sampled half, its perfect-trajectory labels match each annotator on 9 of 12 trajectories, against 8 of 12 between the annotators. This extends our earlier blind study (36 GPT-5.6-sol trajectories, κ=0.73) and Odysseys' validation of the same judge (κ=0.79). On rerun variance, re-judging the Opus 4.6 leaderboard trajectories three more times gives rubric scores within 0.9 points of each other (appendix, judge repeatability). These results are in the new appendix section of the revised PDF.

W1 and Q2. Outcome items, process items and alternative paths

How are outcome requirements distinguished from mandatory interaction steps in the rubrics?

Our intended policy is that any path to the correct persistent outcome earns credit, whether through the GUI, an app's own HTTP endpoint or the shell, and that process items appear only where the instruction asks for the step. As the paper notes in its judge-design appendix, some items do grade how the answer was reached, so we measured how often the released rubrics depart from this policy. An LLM from a third model family labeled all 1,191 graded items: 67.4% grade outcomes, 15.8% grade a step the instruction asks for, and 16.8% grade a step it does not name. Of those 200, 70 are steps any correct solution must take and 24 reward a final persistence check, which leaves 106 items (8.9%) that could fail a valid alternative path.

Can you report calibration on valid GUI, API, and shell alternatives, and clarify when a correct persistent result should receive credit despite a different path?

We rescored the paper's runs from the stored per-item verdicts without these path items (106 to 128 per model, depending on each model's graded rubric version):

ModelPerfect (paper)Without path itemsRubric (paper)Without path items
Opus 4.658.264.785.186.3
GPT-5.6-sol55.463.685.386.9
GPT-5.6-luna55.463.083.284.5
Sonnet 4.650.559.278.580.8
Qwen 3.5 35B7.612.542.443.3
Qwen 3.5 9B6.58.724.124.6

The model ordering is unchanged, and it also stays unchanged when all 200 flagged items are removed.

AuthorsOpenReview ↗

Response to Reviewer 7HKR (2/2)

Q3. Contact replies

How reproducible are tasks requiring delayed LLM-generated contact replies?

Contacts reply through gpt-5.4-mini, and no rubric item grades the content of a reply. We reran 8 tasks in which the agent messages a contact, across all three personas with two runs per condition, once with replies on and once with the reply model's key removed. Mean rubric score was 0.94 with replies and 0.95 without; 5 of the 8 tasks scored identically and the other 3 moved in both directions, so we find no evidence that replies change solvability or graded outcomes. The release will add a flag to disable replies or replay recorded ones.

W2 and Q4. Persona and history intervention

Could you provide a controlled persona/history intervention or a small second-persona evaluation to distinguish personal-context use from general cross-app competence?

We ran the second-persona evaluation. We built two new environments, Jim Halpert and Dwight Schrute, with the open-source generator and rewrote the same requests for them (table in the general response). Every model keeps high scores (rubric 0.89 to 1.00 on every persona), each model's perfect count changes by one to three tasks across personas, and every model scores lowest on Dwight. This points to the tasks' structure, which the three people share, as the main source of difficulty, though 9 tasks per persona is a small sample. A matched ablation of historical context on the full suite is listed as future work in the revised Limitations.

Thanks to the authors for the detailed feedback, I will maintain my original score at this stage

Reviewer_iZXROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark called MyPCBench where agents are tasked to 184 personal and day-to-day tasks. The persona is Michael Scott like in the office, and the environment contains personalized information about the user attempting to solve the task such as browser history, documents, emails and so on.

They compare 10 models, Claude Opus 4.6 with GPT-5.6-sol and analyze performance accross task types, longer trajectories and tasks involving more applications. They also document failure cases and success cases.

Some tasks require specific factual answers or actions while others involve creating outputs such as spreadsheets, documents or presentations. All tasks are evaluated using task specific rubrics and an llm-judge.

Critical strengths

The paper is easy to follow and is well written.

The paper proposes a benchmark that has over 150 tasks spanning multiple domains for personal and day-to-day work tasks. They include personalization properties such as browser history and emails and so on.

The authors create tasks with an ubuntu environment that it is solvable by humans.

The authors show how different Agentic Systems like GPT-6 and Claude Opus 4.6 succeed and fail.

The authors used human experts to validate some of the LLM-based evaluations.

Critical weaknesses

The findings in this paper are not particularly interesting. For example, in the abstract we read "Model failures cluster on tasks that span many applications and on long trajectories,". It's very obvious that more applications and longer trajectories leads to more model failures, and we do not need a paper to make that claim.

I don't see a clear difference between this work and iOSWorld except that the environment is different. they both have personalized workflows, What new insights does this benchmark give that iOSWorld doesn't? that is not clear.

There are many benchmark papers in this area where similar success and failure cases of the models are present such as more applications and longer trajectories lead to performance degradation. How does this benchmark help us get better agents that others don't? the failure cases of the models on this benchmark is similar to many of the existing works presented in the related works section.

There needs to be some evidence that the results obtained in this benchmark reflects what would be obtained in real usecases in the real world. In order to do so there needs to be good held-out actual test examples obtained from a real company. Ultimately we want a benchmark that can indicate whether agents that perform well on them would actually lead to actual productivity gains or KPIs like time saved on performing a task with an AI Agent in the real world(otherwise there isn't much use of such a benchmark). It is not clear that the results that we get here in this benchmark actually reflects reality, especially because the tasks are obtained from an OpenClaw Discord community, how do we know those are realistic.

The authors also claimed that they paraphrased the discord-posted usecases. But even small shifts like paraphrases like that in use case descriptions can potentially lead to big discrepancy in how much the use case represents the real world.

The usecases that measure the quality of the output have groundtruth generated by an LLM and are evaluated based on rubrics, how do we make sure that the llm judge based on the selected rubrics actually properly evaluate the quality of the agent's output (an LLM evaluating an LLM on data generated by an LLM is circular)? For this we also need real held out usecases from real enterprise and have a quality comparison of the outputs between an AI agent and human-generated output and some kind of preference evaluation between human answers and AI Agent answers.

Plus most of the tasks are LLM-synthesized how do you make sure that these task descriptions and information actually represent the complexity of the real world workflos. The benchmark is also not diverse enough as there is only one persona that is Michael Scott.

Also the main claim of the paper is that this benchmark extends others by having a personalization aspect in the usecases. However, the authors haven't isolated how much personalization affects the model performance on the tasks, there needs to be some kind of experiments that compare with and without personalization in terms of how much the results are being affected when personalized information is included.

Questions for authors

Please address the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer iZXR (1/2)

We thank the reviewer for their thoughtful review and appreciation of MyPCBench. Namely, the personal day-to-day tasks built on browser history and email, our Ubuntu environment whose tasks are solvable by humans, our analysis of how different agents succeed and fail, and human validation of the LLM-based evaluation.

We are happy to address the reviewer's concerns below:

W1. Are the findings obvious?

It's very obvious that more applications and longer trajectories leads to more model failures, and we do not need a paper to make that claim.

We agree and believe that that our abstract sentence about app count undersells the paper, and the revised PDF replaces it. Four findings are specific to MyPCBench. First, personalization tasks that require recovering facts from the user's history separate frontier from weaker agents even further when the number of apps is held fixed (48–80% vs. 3–13% perfect at 2–3 apps). Second, rankings change as tasks span more apps: GPT-5.6-sol ties Opus 4.6 overall but perfects 18% vs. 36% of tasks spanning 7 or more apps. Third, agents finish most of a personal request and miss one piece, with rubric scores of 78–85 against perfect rates of 44–58, and skipped required apps are the largest failure mode. Fourth, failure modes on personalization tasks are family-specific and vary across model releases: premature stopping fell from 104 failed items for GPT-5.5 to 16 for GPT-5.6-sol.

W2. Difference from iOSWorld

I don't see a clear difference between this work and iOSWorld except that the environment is different.

On a desktop the agent faces different problems than on a phone. Agents work across web apps, a file system, LibreOffice and a shell (iOSWorld has no terminal) over the same logged-in accounts, and must choose between GUI and programmatic access. Shell use ranges from 1% to 52% of actions across models, and Claude agents' console-script shortcuts, a failure mode we observe only for Claude, exist because a shell is available. Many tasks also require producing desktop artifacts such as spreadsheets together with coordinated side effects across apps, over a history shared by 17 apps and the file system.

W3. How does this help build better agents?

How does this benchmark help us get better agents that others don't?

Agent developers get failure modes they can act on and track across releases: Claude agents take console-script shortcuts, and premature stopping in GPT agents fell from 104 to 16 failed items between GPT-5.5 and GPT-5.6-sol. Because the open-source generator rebuilds the environment for any persona, developers can also check that an improvement holds for a user the agent was never tuned on.

W4. Real-world validity

There needs to be some evidence that the results obtained in this benchmark reflects what would be obtained in real usecases in the real world.

Validation against productivity KPIs is an open problem for the whole field: no computer-use benchmark has it, including OSWorld, WebArena, AppWorld and TheAgentCompany. Every task from MyPCBench comes from a real request seed by users of a consumer assistant agent (OpenClaw) and and the authors solved every task end-to-end on the live VM. Many data points in MyPCBench without synthetic data would require real bank records and private messages that can't be released, which is why a coherent synthetic persona is needed.

AuthorsOpenReview ↗

Response to Reviewer iZXR (2/2)

W5. Paraphrasing

Even small shifts like paraphrases like that in use case descriptions can potentially lead to big discrepancy in how much the use case represents the real world.

The source pool was 2,749 anonymized and paraphrased OpenClaw use cases. The authors selected the requests, and drafted each task with LLMs by mapping the requests onto Michael Scott's data. The authors then manually reviewed every task against its source request, checking that only named entities changed and that the intent and required actions were preserved.

W6. An LLM judging an LLM on LLM-generated data

An LLM evaluating an LLM on data generated by an LLM is circular.

The rubrics were drafted by an LLM, but every graded value is a record read from the seeded databases, and two authors validated every task and rubric on the live VM (§4.1). The builder and the judge also come from different model families. Further, we ran a new double-annotated human study across three model families: the judge agrees with each human grader as often as the two graders agree with each other (81% vs. 81%), and with their consensus on 88% of items (κ=0.71). On comparing agent outputs with human-written outputs: every rubric item in MyPCBench checks a verifiable record or end state, so we compare the judge's verdicts directly with blind human graders' verdicts on the same trajectories. Please see our response to Reviewer 7HKR for the full table.

W7. LLM-synthesized tasks and one persona

The benchmark is also not diverse enough as there is only one persona that is Michael Scott.

Every task starts from a real, human-selected request; 68% are multi-app, they average 6.5 rubric items, and the best agent scores 58%. We also built two new personas, Jim Halpert and Dwight Schrute, with the open-source generator and rewrote the same requests for their data. Every model scores between 0.89 and 1.00 rubric on all three personas (table in the general response). We provide an open-source persona generator to expand the benchmark as desired.

W8. With and without personalization

The authors haven't isolated how much personalization affects the model performance on the tasks.

We provide two pieces of evidence. First, holding the number of apps fixed, tasks that require recovering facts from the user's own history separate frontier from weaker agents by 35 to 77 points, while bounded actions separate them by 32 (general response). Second, when the same requests are rewritten for two other people's histories, every model keeps high scores (0.89 to 1.00 rubric) and every model scores lowest on the same persona.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal Response to All Reviewers (1/2)

We thank all three reviewers for their detailed feedback. We appreciate that they found MyPCBench's desktop environment detailed and internally consistent (DUno), its tasks grounded in a user's real cross-app history (7HKR), and its thorough creation process, from end-to-end manual completion of every task to human–judge comparisons and paired tests (7HKR). All three reviewers asked whether MyPCBench measures personalization or long-horizon multi-app work, and whether results from one persona generalize. We answer each concern individually and summarize the main new evidence here.

What we mean by personalization (Reviewers iZXR, 7HKR, and DUno). We call an agent personally intelligent when it is grounded in a user's state and history, such as a user's logged-in applications. Every MyPCBench task runs against one persistent, year-long history of about 42K records across 17 logged-in apps, so the agent must find the few relevant records among a whole life's worth of data. Live-web benchmarks cannot use logged-in personal accounts, empty desktops have no user history, and profile-based benchmarks such as PersonalWAB and Persona2Web hand the model its context explicitly. The revised PDF adds this definition to §1.

Comparison with OSWorld 2.0 (Reviewers DUno and iZXR). OSWorld 2.0 names MyPCBench as one of its two closest points of comparison and contrasts the two directly: MyPCBench "centers evaluation on a coherent personalized desktop state, whereas OSWorld 2.0 places greater stress on much longer workflow execution." OSWorld 2.0 seeds a task-specific workspace and averages 318 tool calls per task for Opus 4.7 with maximum thinking. MyPCBench tasks are much shorter (46.5 steps for Opus 4.6), yet the best agent perfects only 58.2%, and the most common failures are skipped required apps (659 failed items) and premature stopping.

History-dependent tasks separate agents at a fixed app count (Reviewers iZXR, 7HKR, and DUno). We held the number of apps fixed at 2–3 (72 tasks) and compared task types.

Task typeFrontier agents (perfect)Weaker agents (perfect)
Bounded actions56%24%
Recovering facts from the user's history (reconciliation, aggregation, pattern inference, lookup)48–80%3–13%

We define frontier agents as Opus 4.6, Sonnet 4.6, GPT-5.6-sol, GPT-5.6-luna and GPT-5.5. Weaker agents are GPT-5.4 mini and both Qwen 3.5 models. GPT-5.4 and Muse Spark 1.1 fall between the two groups and are not included. The gap holds in every history-dependent type (5 to 12 tasks each; full table in the revised appendix), and the persona experiment below tests whether it depends on one person's data.

AuthorsOpenReview ↗

Rebuttal Response to All Reviewers (2/2)

Two new personas built with the open-source generator (Reviewers iZXR, 7HKR, and DUno). MyPCBench's generator is open source and builds every application's records from a single persona specification. It ships 16 complete persona specifications, and a new persona can be created from a template with one command. For the rebuttal we built complete environments for Jim Halpert and Dwight Schrute.We then rewrote a subset of tasks for each persona (the same request, answered from that person's records) and also wrote and validated new tasks not in the original MyPCBench set using our framework in the paper.

PersonaGPT-5.6-sol (perfect, rubric)Opus 4.6Muse Spark 1.3
Michael (6 released + 3 new tasks)7/9, 0.939/9, 1.009/9, 1.00
Jim7/8, 0.987/9, 0.938/9, 0.94
Dwight6/9, 0.896/9, 0.937/9, 0.94

We find that scores transfer and stay high on the new personas. Every model scores between 0.89 and 1.00 rubric on every persona, and no task scores below 0.5 for any model. This matched subset is near ceiling for current frontier models (66 of 80 runs perfect), so it tests transfer across users, and the full leaderboard remains the basis for ranking models.

Rubric calibration (Reviewers 7HKR and iZXR). An LLM from a third model family (Muse Spark 1.3, as the tasks were built with Claude and the judge is Gemini) labeled all 1,191 graded rubric items. Muse classified the rubric breakdown as 67.4% grading outcomes, 15.8% grading a step the instruction asks for, and 16.8% grading an implictly required step. Of those, 106 items (8.9%) could fail a valid alternative path. Removing them from the stored verdicts raises perfect rates by 2.2 to 8.7 points and leaves the model ordering unchanged. The judge itself was validated in two blind human studies: on 36 MyPCBench trajectories stratified across all task categories (220 items; κ=0.73 on items and 0.64 on perfect labels), and with the same protocol on 120 trajectories in Odysseys (κ=0.79 on items and 0.85 on task success). A new study had two annotators independently grade the same 24 trajectories from three model families and three personas: the judge agrees with each annotator as often as they agree with each other (81% vs. 81%), and with their consensus on 88% of items (κ=0.71).

Revisions. The revised PDF keeps the main text as submitted, within the same nine pages, apart from two changes: a definition of "personally intelligent" in §1 and a replacement for the abstract's app-count sentence. All new material is collected in one appendix section, "Additional Analyses Requested During Review". Changes are marked in blue.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear Reviewers,

Thank you again for your feedback and detailed reviews. We wanted to follow up on our rebuttal and check whether we have addressed your concerns, given that the discussion period will end soon. If any points remain unclear, please let us know, and we would be happy to respond.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.