acceptodds
Under review as a conference paper at ICLR 2027

Deep Personalization: Continuous Context Observation for Local Proactive Agents

Abstract

As AI agents become increasingly capable, users must still recognize when and how to ask for help, leaving many opportunities for useful support unrealized. Proactive agents can close this gap by recognizing when support may be useful and initiating interaction. However, effective proactivity requires understanding users’ real-time activities and personalizing intervention timing so that assistance is offered when it is likely to be helpful rather than disruptive. We introduce **deep personalization**, a closed-loop process that connects continuous context observation with continual agent adaptation: an agent observes user activity, selectively retains long-term context, and adapts intervention decisions from implicit feedback. We initiate deep personalization in **Coco**, a proactive desktop agent. At Coco's core is an Observer Agent that monitors desktop activity, maintains compact long-term user modeling context, and learns when assistance is desirable from implicit signals such as engagement, dismissal, explicit requests, and repeated work patterns. We further develop efficient on-device strategies including change-aware frame selection and observation-first scheduling, enabling the Observer Agent to run on local AI compute platforms without streaming raw activity data off-device. To study deep personalization at scale without collecting sensitive real-user activity, we introduce a data-synthesis pipeline that transforms existing computer-use trajectories into **OSWorld-Persona**, a benchmark with persona-conditioned intervention labels. On OSWorld-Persona, deep personalization improves macro F1 across three model backbones from 0.12-0.19 under direct prompting to 0.35-0.48 after adaptation, a 2.4-3.0 relative gain. In a week-long user study, we observe that Coco improves its macro-averaged F1 score from 0.199 to 0.515. These results suggest that on-device deep personalization can make proactive agents more timely and increasingly aligned with each user’s everyday work.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personalization, agentic systems, implicit feedback learning, on-device deployment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qDGAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces "deep personalization," a closed-loop process in which a proactive desktop agent (Coco) continuously observes user activity, maintains memory, decides when to intervene, and adapts from feedback. The authors build OSWorld-Persona by labeling OSWorld 2.0 trajectories with Kimi-K2.6 under four personas, report 2.4–3.0× macro-F1 gains over direct prompting, compare their memory module against Mem0, and run a one-week study with 10 users where an F1 proxy rises from 0.199 to 0.515.

Critical strengths
  • Timely and important problem: Deciding when to intervene is a genuinely subjective, under-studied bottleneck for proactive agents.
  • System contributions: The paper provides useful implementation and real-world deployment details.
  • Reproducibility: The data-synthesis pipeline is transparent and well-documented.
Critical weaknesses
  • The test set is very small and highly imbalanced, making several of the per-persona claims potentially unreliable.
  • Flaws in the benchmark setup and user-study design leave the paper's central claims under-supported.
Questions for authors
  • Disconnect between claims and benchmark evaluation: The paper's central claim is that Coco learns from implicit feedback. However, in the OSWorld-Persona experiments, the Reflector prompt is given the <ground truth label> directly (Appendix B.4). The authors need to justify how learning from implicit feedback is properly evaluated when explicit ground-truth labels are provided during reflection.
  • Small and imbalanced test set: The test set is too small and skewed to reliably support the per-persona claims. Table 4 shows only 10 positive test examples for Learning-oriented and 30 for Reserved Drafts. With so few positive examples, a single flipped prediction changes the F1 score by several points, making it difficult to assess the true robustness of the reported gains.
  • Lack of a control condition and detailed metrics in the user study: The user study lacks a control condition (such as a frozen Day-1 prompt or a fixed-frequency trigger), meaning the reported improvement may simply reflect the agent "suggesting more often" over time. On Day 1, 6 out of 10 users have an F1 of 0 because Coco almost never suggests anything. Consequently, any subsequent increase in suggestion frequency would naturally raise the F1 proxy, regardless of whether intervention timing actually improved. Furthermore, the paper does not report daily raw counts of suggestions, acceptances, or user help requests, making it impossible to disentangle improved precision from increased trigger frequency.
6 comments

We thank the reviewer for the detailed and constructive review, and for recognizing that the work addresses a timely and important problem, provides system contributions, and that the data-synthesis pipeline is transparent and well documented. We address each question below.

Disconnect between claims and benchmark evaluation: The paper's central claim is that Coco learns from implicit feedback. However, in the OSWorld-Persona experiments, the Reflector prompt is given the <ground truth label> directly (Appendix B.4). The authors need to justify how learning from implicit feedback is properly evaluated when explicit ground-truth labels are provided during reflection.

Thanks for brining up this point. We apologize that the naming in Appendix B.4 caused confusion. The "ground truth label" in the Reflector prompt is not an oracle supplied at test time. It is the training target ot’o_t’ produced by the implicit-feedback step in Section 3.3. We renamed the placeholder field name as “<label>” in the prompt and clarified in the text.

Here, we appreciate a chance to walk through the self-adaptation pipeline in Deep Personalization:

  1. Observe and decide. At each timestep, the Observer outputs a structured observation and a decision yty_t (Section 3.1).
  2. Derive implicit feedback rtr_t. If Coco intervenes, rt=1r_t = 1 when the user engages and rt=0r_t = 0 when the user dismisses. If Coco stays silent, a retrospective scan of the accumulated context flags a missed opportunity when the user later repeats operations, gets stuck, or explicitly invokes AI (Section 3.3).
  3. Build the adaptation set D\mathcal{D}. When rt=0r_t = 0, the on-device model self-reflects to produce a revised output ot’o_t’; when rt=1r_t = 1, ot’=oto_t’ = o_t. This ot’o_t’ is what fills the "label" field in the Reflector prompt.
  4. Adapt. Coco-PE or Coco-LoRA is trained on D\mathcal{D}.

On OSWorld-Persona, step 2 cannot come from a real person, so the persona-conditioned simulated user plays that role: its decision stands in for whether the user would have engaged with, dismissed, or later sought the help. Importantly, labels are used only on the training split; the test split is chronologically later and its labels are never seen during adaptation.

The test set is very small and highly imbalanced, making several of the per-persona claims potentially unreliable.

We thank the reviewer for the careful reading. The full test set contains 1,000 examples with 332 positives; each persona has 237–285 test examples (Table 4). The imbalance is intentional: different people prefer very different levels of support. A user who rarely wants help is also important to include.

Note that the results in Table 1 are averaged across three random seeds and we further report the 95% confidence interval under N=10,000 resamples below. The clearest improvements occur for “Automation-first” and “Eager Hands-on”. For Automation-first, Coco-PE’s intervals lie entirely above Vanilla’s for all three backbones: Qwen-3.5-9B (0.5233 [0.4598, 0.5872] versus 0.0371 [0.0000, 0.0930]), Qwen-3.5-35B-A3B (0.4514 [0.4007, 0.5104] versus 0.1020 [0.0324, 0.1793]), and Nemotron-3-Nano-Omni (0.3229 [0.2879, 0.3761] versus 0.0362 [0.0000, 0.0889]). For Eager Hands-on, both adaptation methods have intervals entirely above Vanilla’s across all three backbones.

ModelMethodF1: Automation FirstF1: Eager Hands-onF1: Learning-orientedF1: Reserved Draft
Qwen-3.5-9BVanilla0.0371 [0.0000, 0.0930]0.1387 [0.0518, 0.2451]0.3919 [0.3000, 0.4224]0.1765 [0.1432, 0.2283]
Qwen-3.5-9BInduced Persona0.0785 [0.0220, 0.1479]0.1993 [0.1374, 0.2603]0.2583 [0.2379, 0.2905]0.3014 [0.2050, 0.3812]
Qwen-3.5-9BOracle Persona0.2096 [0.1875, 0.2341]0.7291 [0.6766, 0.7777]0.3099 [0.2500, 0.4008]0.3243 [0.2356, 0.4015]
Qwen-3.5-9BCoco-PE0.5233 [0.4598, 0.5872]0.8088 [0.7681, 0.8500]0.3888 [0.3275, 0.4507]0.2121 [0.1712, 0.2642]
Qwen-3.5-9BCoco-LoRA0.3126 [0.2293, 0.3852]0.8408 [0.8024, 0.8739]0.4264 [0.3830, 0.4738]0.2404 [0.1563, 0.3229]
Qwen-3.5-35B-A3BVanilla0.1020 [0.0324, 0.1793]0.1507 [0.0843, 0.2201]0.1900 [0.0500, 0.3070]0.2878 [0.2458, 0.3191]
Qwen-3.5-35B-A3BInduced Persona0.1184 [0.0561, 0.1879]0.2158 [0.1392, 0.2935]0.3440 [0.3000, 0.3837]0.2850 [0.1813, 0.3740]
Qwen-3.5-35B-A3BOracle Persona0.2671 [0.1563, 0.3724]0.3810 [0.3041, 0.4447]0.1769 [0.1002, 0.2692]0.2602 [0.1882, 0.3264]
Qwen-3.5-35B-A3BCoco-PE0.4514 [0.4007, 0.5104]0.7369 [0.6597, 0.8009]0.2603 [0.2106, 0.3200]0.2850 [0.2475, 0.3300]
Qwen-3.5-35B-A3BCoco-LoRA0.1911 [0.1068, 0.2790]0.5385 [0.4452, 0.6201]0.2563 [0.2031, 0.2738]0.2682 [0.1935, 0.3382]
Nemotron-3-Nano-OmniVanilla0.0362 [0.0000, 0.0889]0.0951 [0.0376, 0.1641]0.1525 [0.0840, 0.2751]0.1863 [0.1238, 0.2486]
Nemotron-3-Nano-OmniInduced Persona0.0657 [0.0102, 0.1501]0.1991 [0.1194, 0.2850]0.3043 [0.1733, 0.4511]0.3075 [0.1920, 0.4014]
Nemotron-3-Nano-OmniOracle Persona0.1721 [0.0950, 0.2584]0.5247 [0.4393, 0.6033]0.1515 [0.0300, 0.2308]0.2759 [0.2483, 0.3934]
Nemotron-3-Nano-OmniCoco-PE0.3229 [0.2879, 0.3761]0.4737 [0.3548, 0.5903]0.1928 [0.0957, 0.3012]0.2949 [0.1774, 0.4003]
Nemotron-3-Nano-OmniCoco-LoRA0.1771 [0.1314, 0.2253]0.8609 [0.8204, 0.8948]0.1635 [0.0901, 0.2904]0.1923 [0.1215, 0.2554]

Additional experiment: broader persona coverage. To assess whether the gains extend beyond the original four personas, we applied the same data-synthesis pipeline to three additional personas sampled from Nemotron-Personas-USA (https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA), augmented with AI-use habits and assistance preferences. These personas describe an aspiring electrician, a heavy-equipment operator and logistics coordinator, and a retired maintenance mechanic.

Joseph Giordano, a diligent aspiring master electrician, channels his methodical mindset and love for routine into residential wiring and emerging solar PV projects, ensuring NEC compliance while planning to eventually open a modest family‑owned contracting shop where they can mentor apprentices in a low‑pressure setting. … AI experience: Often accepts the first answer and may save drafts without reviewing them. Habits: Focused, reliable, tackles difficult tasks early, and frequently skips breaks. Welcomes specific help even during steady progress when it saves effort, resolves uncertainty, catches errors, or explains unfamiliar steps. Preferred help: Concise next steps, editable drafts, targeted explanations, concrete checks, and step-by-step guidance.

Bernadette Gangaram, a 28‑year‑old heavy‑equipment operator and logistics coordinator, leverages their methodical mindset and innate curiosity to excel in safety compliance, precision route planning, and mentoring new operators while quietly championing emerging autonomous technologies on the job site. … AI experience: Often accepts the first answer and may overlook requirements or mistake fluent explanations for understanding. Habits: Focused, reliable, takes regular breaks, and tackles difficult tasks early. Welcomes occasional targeted help for visible uncertainty, repeated effort, unfamiliar steps, or meaningful improvements in accuracy or efficiency. Preferred help: Concise next steps, editable drafts, targeted explanations, concrete checks, and step-by-step guidance.

Antonio Gonzalez, a retired maintenance mechanic with four decades of plant repair experience, they now channel their practical curiosity into woodworking, home repair mentorship, and bilingual storytelling, balancing flexible, sometimes disorganized project planning with a competitive edge that drives them to perfect every bench they build. … AI experience: Works manually and may rewrite routine notes or assume requirements without checking the source. Habits: Focused, sometimes forgets commitments, takes regular breaks, and tackles difficult tasks early. Welcomes targeted help for visible uncertainty, repeated effort, unfamiliar steps, or meaningful improvements in accuracy or efficiency. Preferred help: Concise next steps, step-by-step guidance, explanations of unfamiliar features, and reminders to record commitments, save work, and check unfinished steps.

PersonaTraining examples (positive)Test examples (positive)
Joseph Giordano1,345 (156)725 (71)
Bernadette Gangaram1,351 (78)731 (35)
Antonio Gonzalez1,349 (102)730 (53)

We evaluated Qwen-3.5-9B using the same five methods and the results are as follows. Coco-PE and Coco-LoRA achieve macro F1 scores of 0.5993 and 0.4965, respectively, compared with 0.1871 for Vanilla and 0.2455 for Oracle Persona. Both methods also improve precision and recall over Vanilla for all three personas, although false-alarm rates increase slightly. These results provide additional evidence that the gains extend beyond the original persona set in Table 1.

MethodJosephBernadetteAntonioMacro F1
Vanilla0.15690.20690.19750.1871
Induced Persona0.21180.34380.17860.2447
Oracle Persona0.34180.19610.19850.2455
Coco-PE0.68540.43640.67630.5993
Coco-LoRA0.48730.48650.51580.4965
Reviewer_8k9nOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Coco, a proactive desktop agent designed to provide personalized assistance through continuous observation of user activities. The system integrates an Observer Agent, long-term memory, and continual adaptation based on implicit user feedback to learn when proactive interventions are appropriate. The authors also propose on-device optimizations for efficient deployment and construct OSWorld-Persona, a synthetic benchmark for evaluating personalized intervention decisions. Experiments on three multimodal models and a week-long user study with 10 participants demonstrate improvements in intervention prediction performance after personalization.

Critical strengths
  1. Interesting and practically relevant problem. The paper addresses an important problem of enabling proactive agents to learn when to offer assistance based on user activities and preferences. The integration of continuous observation, long-term memory, and implicit feedback provides a practical framework for personalized proactive assistance.

  2. Comprehensive system implementation and evaluation. The authors implement a functional on-device proactive agent and evaluate it using both a synthetic benchmark (OSWorld-Persona) and a week-long user study. The experiments demonstrate improvements in intervention prediction performance through personalization.

Critical weaknesses
  1. Limited methodological novelty and unclear research contributions. The proposed Coco framework primarily integrates existing techniques, including multimodal observation, long-term memory, LoRA fine-tuning, and prompt evolution. While the overall system is reasonably comprehensive, the paper does not introduce a clearly identifiable novel algorithm or learning mechanism. As a result, the work reads more like a system demonstration than a research paper with substantial methodological contributions.

  2. Questionable validity of the synthetic benchmark. OSWorld-Persona relies on a VLM to generate intervention labels conditioned on four predefined synthetic personas. It is unclear whether these labels accurately reflect real user preferences and intervention needs. Consequently, improvements on this benchmark may primarily demonstrate the ability to fit synthetic preference patterns rather than genuine personalization. The limited diversity of personas further raises concerns about the generalizability of the findings.

  3. I think the paper is more suitable for Demo paper rather than research track.

Questions for authors

N/A

4 comments

Thank you so much for recognizing that the paper provides “a practical framework for personalized proactive assistance” and contributes comprehensive system implementation and evaluation including “functional on-device proactive agent,” “synthetic benchmark,” and “week-long user study”. We address each concern below.

Limited methodological novelty and unclear research contributions.

While the overall system is reasonably comprehensive, the paper does not introduce a clearly identifiable novel algorithm or learning mechanism. As a result, the work reads more like a system demonstration than a research paper with substantial methodological contributions.

We respectfully disagree, and we appreciate the opportunity to clarify. We see three research contributions beyond a system demonstration: a timely problem formulation, a method designed around that problem's constraints, and evidence that it works both in controlled settings and in the wild.

Why the problem matters (Section 1). Agents are moving from on-demand tools that wait for an instruction to always-on collaborators that work alongside the user. In this setting, the main question is no longer only "how to complete the task" but "when to step in for this particular user." (Line 39-49)

In fact, after we submitted the paper to ICLR 2027, OpenAI and Anthropic released proactive agents products Dot [1, OpenAI, 2026] [2, Anthropic, 2026]. Always-on operation also creates a tension between personalization and data exposure (Line 066). For example, recent agent-infrastructure work names running persistent, context-rich agents under local compute and privacy constraints as one of the main open challenges for AI agents [3, DeepSeek Harness, 2026]. Our paper provides a framework to personalize proactive agents in a privacy-preserving way.

A new formulation of personalization (Section 2). Existing work personalizes LLMs and agents mainly by (i) conditioning on a user profile or persona, (ii) retrieving from the user's conversation history or memory, (iii) parameter-efficient fine-tuning on user data, or (iv) aligning to explicitly elicited preferences such as pairwise comparisons, demonstrations, or edits [4, Zhang et al., 2025; 5, Xu et al., 2026]. In these paradigms, personalization is applied after the user issues a request, and the supervision comes from explicit feedback or a fixed profile. Deep Personalization asks a different question: when should an agent help a particular user, and how can it learn this without asking the user to label anything? We formulate it as a closed loop in which continuous observation supplies both the agent's state and its supervision.

Our experiments show why existing approaches fall short here. In Table 1:

  • The Induced Persona baseline reaches 0.21–0.24 macro F1.
  • The Oracle Persona baseline reaches only 0.27–0.39, even though it receives the exact persona description used to generate the labels.
  • Our approach reaches 0.35–0.48 and outperforms Oracle Persona on all three backbones.

A solid method designed around the closed loop (Sections 3–4). Making this loop work requires answering three questions together: what the agent should see, what it should remember, and how it should learn. All of this must hold within the compute budget of a single personal device. We introduce a practical system to achieve these goals (Lines 057–069) :

  • The Observer Agent is the unit of personalization. The Observer Agent is an always-on component that tracks the user's ongoing work and decides whether assistance would be valuable. The heavier Execution Agent is called only when it is needed. We further introduce on-device deployment strategy to ensure local Observer Agent can fit its multi-second observation window. To our knowledge, the personalization literature reviewed in [4, 5] has not proposed this decoupling in AI agents.
  • The memory is designed for the Observer's needs. Desktop activity is long-horizon and interleaves many tasks, so the Observer needs a state that is compact enough to run on every observation yet rich enough to learn from. The proposition–observation memory serves both purposes. It conditions each decision, and it gives the adaptation step the context it needs to explain why an intervention succeeded or failed. The memory outperforms Mem0 on all four memory benchmarks (Table 3), and removing it lowers F1 on every backbone (Table 2).
  • Implicit feedback supplies the supervision. We deliberately use standard optimizers (ACE-style prompt evolution and LoRA), because the open problem is not the optimizer but where the supervision signal comes from. Coco derives that signal from the user's own behavior, including acceptances, rejections, explicit requests, and missed opportunities recovered by retrospectively scanning memory. This is what turns continuous observation into personalization without any labeling.

We believe that a timely problem formulation, a method designed for on-device deep personalization, and evidence from both controlled and real-world evaluation are well within the scope of ICLR's research track.

More broadly, as AI systems move beyond a single model or algorithm, a key research question is how different parts must be designed together to make a new capability possible, and we hope the paper is judged as a contribution to this kind of system-level machine learning research to align proactive AI agents to be truly useful.

[1] Introducing dots, OpenAI, https://openai.com/index/introducing-dots/

[2] Introducing Claude Duet, Anthropic, https://claude.com/resources/articles/claude-duet

[3] Exploring Super-agent Design: Towards a More Autonomous Agent System, DeepSeek Harness, https://github.com/deepseek-ai/dsh-superagent-prototype

[4] Personalization of large language models: A survey, Zhang et al., TMLR 2025, https://openreview.net/pdf?id=tf6A9EYMo6

[5] Toward personalized llm-powered agents: Foundations, evaluation, and future direction, Xu et al., 2026, https://arxiv.org/abs/2602.22680

OSWorld-Persona relies on a VLM to generate intervention labels conditioned on four predefined synthetic personas. It is unclear whether these labels accurately reflect real user preferences and intervention needs. Consequently, improvements on this benchmark may primarily demonstrate the ability to fit synthetic preference patterns rather than genuine personalization. The limited diversity of personas further raises concerns about the generalizability of the findings.

We thank the reviewer for raising this important point. We used synthetic personas deliberately: they let us study personalization at scale and reproducibly without collecting sensitive real-user activity.

We address label validity in three ways.

  1. Label quality control (Section 5.1, Appendix C). Each decision stage is re-run twice at temperature 0.4, and an example is kept only if both runs reproduce the original label. Every positive ("need support") label is then reviewed by a human annotator, who must identify the on-screen context that supports proactive help; examples without such context are removed.
  2. What the benchmark measures. Each persona is trained and evaluated separately, with a chronological 70/30 split per persona (Table 4). The benchmark therefore tests whether deep personalization can learn a user's decision boundaries from earlier sessions and generalize them to later, unseen sessions. Moreover, our method outperforms the Oracle Persona baseline even though that baseline receives the persona description directly. Thus, the gains cannot be explained merely by reconstructing persona text; they instead reflect the model’s ability to use interaction history to predict when intervention is helpful.
  3. Real-user validation. The week-long user study involves no synthetic labels: 10 participants used Coco during their own work, and feedback came only from their thumbs-up, thumbs-down, dismissals, and help requests. The macro F1 proxy rose from 0.199 to 0.515 and pooled precision from 0.12 to 0.53 (we’ve revised the manuscript to report daily raw counts).

On persona diversity: While we agree that the personas we include in experiment cannot cover the full population due to compute limit, they were chosen to span the range of desired support, from "Automation First" to "Learning-oriented", with positive rates from about 6% to 77% (Table 4). Because the data-synthesis pipeline takes any persona description as input, adding personas is straightforward. As future work, we are collecting longer trajectories paired with more diverse to build larger datasets for training and evaluating the Observer Agent. We revised the manuscript to explicitly mention this.

Hi Reviewer 8k9n,

Thank you again for your thoughtful feedback! We have posted our rebuttal addressing the concerns raised in the review and revised the manuscript (changes marked in blue) to make the contribution clearer.

We would greatly appreciate it if you could take a look at our rebuttal and reconsider your assessment. Please let us know if any concerns remain or if further clarification would be helpful.

Best regards,

Submission13869 Authors

Reviewer_RHHvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper studies an interesting topic, personalized ai assistant on device. specifically, the main contributions:

  1. developed an end to end personalized ai assistant on device that observes the user's context and proactively suggest actions to the users using a learned model.

  2. introduced a new benchmark based on osworld, called osworld-persona, to evaluate personalization of desktop ai assistant.

the model is continuously learned on device with lora type of fine tuning methods as the users keep using the ai assistant.

Critical strengths
  1. the paper is well motivated, the problem is imprtant as well.

  2. the system is end-to-end and it looks like the authors have a prototype that's distributable already.

  3. the architecture is privacy-centered. the models are deployed and learned on device.

  4. the proposed benchmark can contribute to more research on personalization of desktop agent.

Critical weaknesses
  1. the paper seems to me more of an end-to-end system integration. it's not a bad thing, but as a research paper then i would suggest the authors focus more on the design of each components and how these components are tuned to achieve greater outcomes. for example, it is known the fine-tuning can lead to overfitting and the discussino around how to prevent forgetting would be an important topic for this paper.

  2. the experimental validation is relatively weak as a system paper. for on device models, we also have to evaluate their power utilization and resource utilization because those can affect usability of the desktops for daily purposes.

  3. it is unclear whether the system is reproducible. will the code be open sourced?

  4. I'm not sure if osworld is the best benchmark to build upon because osworld tasks usually don't involve interruptions in the middle of the tasks. fewer frequency of proactiveness is required for these kind of tasks.

Questions for authors

please respond to my feedbacks above.

2 comments

We thank the reviewer for the careful reading and for recognizing that the problem is important and well motivated, that Coco provides an end-to-end, privacy-centered system, and that the benchmark can support further research on personalized desktop agents. We address each concern below.

i would suggest the authors focus more on the design of each components and how these components are tuned to achieve greater outcomes. for example, it is known the fine-tuning can lead to overfitting and the discussion around how to prevent forgetting would be an important topic for this paper.

In our work, we test different design choices around the Observer Agent to evaluate how to achieve greater outcomes in deep personalization:

  • Accumulated context. Removing the activity context reduces Coco-PE macro F1 from 0.483 to 0.388 on Qwen-3.5-9B, from 0.433 to 0.398 on Qwen-3.5-35B-A3B, and from 0.321 to 0.304 (Table 2).
  • Context management strategy. The proposition–observation memory outperforms Mem0 on LoCoMo, LongMemEval, BEAM-100K, and BEAM-1M across three answer models (Table 3).
  • Structured observation. Asking the Observer to output only the intervention decision, rather than also describing the observation and intent, reduces Coco-PE macro F1 from 0.483 to 0.305 on Qwen-3.5-9B, from 0.433 to 0.199 on Qwen-3.5-35B-A3B, and from 0.321 to 0.253, demonstrating the value of having the Observer Agent to provide structured observation.

We do agree that having more analysis on each of the components such as understanding forgetting in deep personalization would be useful. We leave them as future work as this paper focuses on system-level contribution of leveraging on-device compute to improve personalized proactive agents. We revise the Limitations and Future Work section to explicitly discuss this.

for on device models, we also have to evaluate their power utilization and resource utilization because those can affect usability of the desktops for daily purposes.

Thank you for the suggestion. We assess resource convention and responsiveness and observe limited CPU contention, indicating that on-device Observer Agent can coexist with everyday desktop workloads.

Specifically, we profiled three metrics while running the Observer Agent:

  • CPU utilization: The percentage of CPU capacity in use, indicating contention for compute resources shared with everyday applications.
  • Memory usage: The percentage of system memory in use, indicating the memory available to other applications.
  • Scheduling delay: The additional time a background thread takes to resume, indicating the OS responsiveness. We measure it by asking a background thread to sleep for 20 milliseconds, then measure how much later it actually resumes.

We profiled two hardware configurations:

  • an RTX 5090 workstation with an Intel Core i9-14900KF CPU (24 cores, 32 threads) ; and
  • an NVIDIA DGX Spark with an NVIDIA GB10 GPU, a 20-core ARM CPU (10 Cortex-X925 and 10 Cortex-A725 cores).

The results are as follows:

DeviceModelCPU UtilizationMemory UsedScheduling Delay
DGX SparkQwen-9B5.48%65.81%0.55 ms
DGX SparkQwen-35B-A3B FP85.60%64.83%0.58 ms
RTX 5090 workstationQwen-9B6.08%13.35%0.19 ms

CPU utilization remains low (5.48–6.08%), and mean scheduling delay is below 0.6 ms across all configurations. These measurements suggest limited CPU contention.

On DGX Spark, system memory usage rises to approximately 65% because DGX Spark uses a unified memory architecture for CPU and GPU. On the RTX 5090 workstation, system-memory usage is 13.35%, because our deployment does not require system-memory offloading.

We revise the manuscript to add these measurements in Appendix D “Profiling On-device Deployment”.

it is unclear whether the system is reproducible. will the code be open sourced?

Yes. The code and OSWorld-Persona data are available at https://anonymous.4open.science/r/ coco-iclr-annonymous-repo-7F7E. We will replace the current link with a public link after the anonymity period. We will also record videos to help users to install coco on different types of AI PCs.

osworld tasks usually don't involve interruptions in the middle of the tasks. fewer frequency of proactiveness is required for these kind of tasks.

Thank you for this important point! We build on OSWorld 2.0 ([1, Yuan et al., 2026]) rather than the original OSWorld precisely because it is designed for realistic, long-horizon computer use with app switching and mid-task state changes. For example, the representative data point shown on the OSWorld 2.0 website (https://osworld-v2.xlang.ai/) covers reading a tutorial PDF, operating a legacy reimbursement portal, extracting the correct amount from noisy receipts, tracing order evidence across Gmail and a bank website, reacting to a new email that changes the task state, recovering hidden employee information from a prior report, identifying an inconsistency that requires user clarification, and completing the final review and submission.

Moreover, we want to highlight that our data-synthesis pipeline is not tied to OSWorld 2.0: it applies to any computer-use recording (Section 5.1). We recognize there is a lack of public day-scale computer-use data, and we are collecting such trajectories to build larger datasets for training and evaluating the Observer Agent. Besides the evaluation on OSWorld-Persona, our week-long user study (Section 6.2) complements the benchmark with real, naturally interrupted everyday work.

[1] OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks, Yuan et al., 2026, https://arxiv.org/abs/2606.29537

Comments on the paper

1 comment

We thank the reviewers for their thoughtful and constructive feedback, and for recognizing the following strengths and contributions of our work:

  1. Timely and important problem. “Deciding when to intervene is a genuinely subjective, under-studied bottleneck for proactive agents.” (RHHv, 8k9n, qDGA)
  2. Comprehensive system contribution. The approach is “end-to-end” and provides a “privacy centered architecture”, providing “a practical framework for personalized proactive assistance.” (RHHv, 8k9n, qDGA)
  3. Comprehensive evaluation and useful benchmark. “The proposed benchmark can contribute to more research on personalization of desktop agents”. (RHHv, 8k9n)

In response to the reviewer’s concern, we revise the manuscript (the updated parts are marked with blue color) in three key areas:

  1. Research contribution. We improve Section 1 Introduction and Section 8 Conclusion to clarify Deep Personalization as a new approach to personalization: learning when to offer help through continuous observation and adaptation from user behavior. Our contribution focuses on the system design that enables this loop on device, opening opportunities for personalized agents as AI agents start to move from session-based to always-on as exemplified by recent releases from OpenAI, Anthropic, and DeepSeek.
  2. User-study interpretation. We provide daily per-participant interaction counts. The data supports improved acceptance beyond increased frequency.
  3. Deployment feasibility. We add CPU, memory, and scheduling-delay measurements on two hardware platforms.

The code and OSWorld-Persona data are available at https://anonymous.4open.science/r/ coco-iclr-annonymous-repo-7F7E and we will change to a public link after the anonymity period. We will also record videos to help users to install coco on different types of AI PCs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.