- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces "deep personalization," a closed-loop process in which a proactive desktop agent (Coco) continuously observes user activity, maintains memory, decides when to intervene, and adapts from feedback. The authors build OSWorld-Persona by labeling OSWorld 2.0 trajectories with Kimi-K2.6 under four personas, report 2.4–3.0× macro-F1 gains over direct prompting, compare their memory module against Mem0, and run a one-week study with 10 users where an F1 proxy rises from 0.199 to 0.515.
Critical strengths
- Timely and important problem: Deciding when to intervene is a genuinely subjective, under-studied bottleneck for proactive agents.
- System contributions: The paper provides useful implementation and real-world deployment details.
- Reproducibility: The data-synthesis pipeline is transparent and well-documented.
Critical weaknesses
- The test set is very small and highly imbalanced, making several of the per-persona claims potentially unreliable.
- Flaws in the benchmark setup and user-study design leave the paper's central claims under-supported.
Questions for authors
- Disconnect between claims and benchmark evaluation: The paper's central claim is that Coco learns from implicit feedback. However, in the OSWorld-Persona experiments, the Reflector prompt is given the <ground truth label> directly (Appendix B.4). The authors need to justify how learning from implicit feedback is properly evaluated when explicit ground-truth labels are provided during reflection.
- Small and imbalanced test set: The test set is too small and skewed to reliably support the per-persona claims. Table 4 shows only 10 positive test examples for Learning-oriented and 30 for Reserved Drafts. With so few positive examples, a single flipped prediction changes the F1 score by several points, making it difficult to assess the true robustness of the reported gains.
- Lack of a control condition and detailed metrics in the user study: The user study lacks a control condition (such as a frozen Day-1 prompt or a fixed-frequency trigger), meaning the reported improvement may simply reflect the agent "suggesting more often" over time. On Day 1, 6 out of 10 users have an F1 of 0 because Coco almost never suggests anything. Consequently, any subsequent increase in suggestion frequency would naturally raise the F1 proxy, regardless of whether intervention timing actually improved. Furthermore, the paper does not report daily raw counts of suggestions, acceptances, or user help requests, making it impossible to disentangle improved precision from increased trigger frequency.
6 comments
The test set is very small and highly imbalanced, making several of the per-persona claims potentially unreliable.
We thank the reviewer for the careful reading. The full test set contains 1,000 examples with 332 positives; each persona has 237–285 test examples (Table 4). The imbalance is intentional: different people prefer very different levels of support. A user who rarely wants help is also important to include.
Note that the results in Table 1 are averaged across three random seeds and we further report the 95% confidence interval under N=10,000 resamples below. The clearest improvements occur for “Automation-first” and “Eager Hands-on”. For Automation-first, Coco-PE’s intervals lie entirely above Vanilla’s for all three backbones: Qwen-3.5-9B (0.5233 [0.4598, 0.5872] versus 0.0371 [0.0000, 0.0930]), Qwen-3.5-35B-A3B (0.4514 [0.4007, 0.5104] versus 0.1020 [0.0324, 0.1793]), and Nemotron-3-Nano-Omni (0.3229 [0.2879, 0.3761] versus 0.0362 [0.0000, 0.0889]). For Eager Hands-on, both adaptation methods have intervals entirely above Vanilla’s across all three backbones.
| Model | Method | F1: Automation First | F1: Eager Hands-on | F1: Learning-oriented | F1: Reserved Draft |
|---|---|---|---|---|---|
| Qwen-3.5-9B | Vanilla | 0.0371 [0.0000, 0.0930] | 0.1387 [0.0518, 0.2451] | 0.3919 [0.3000, 0.4224] | 0.1765 [0.1432, 0.2283] |
| Qwen-3.5-9B | Induced Persona | 0.0785 [0.0220, 0.1479] | 0.1993 [0.1374, 0.2603] | 0.2583 [0.2379, 0.2905] | 0.3014 [0.2050, 0.3812] |
| Qwen-3.5-9B | Oracle Persona | 0.2096 [0.1875, 0.2341] | 0.7291 [0.6766, 0.7777] | 0.3099 [0.2500, 0.4008] | 0.3243 [0.2356, 0.4015] |
| Qwen-3.5-9B | Coco-PE | 0.5233 [0.4598, 0.5872] | 0.8088 [0.7681, 0.8500] | 0.3888 [0.3275, 0.4507] | 0.2121 [0.1712, 0.2642] |
| Qwen-3.5-9B | Coco-LoRA | 0.3126 [0.2293, 0.3852] | 0.8408 [0.8024, 0.8739] | 0.4264 [0.3830, 0.4738] | 0.2404 [0.1563, 0.3229] |
| Qwen-3.5-35B-A3B | Vanilla | 0.1020 [0.0324, 0.1793] | 0.1507 [0.0843, 0.2201] | 0.1900 [0.0500, 0.3070] | 0.2878 [0.2458, 0.3191] |
| Qwen-3.5-35B-A3B | Induced Persona | 0.1184 [0.0561, 0.1879] | 0.2158 [0.1392, 0.2935] | 0.3440 [0.3000, 0.3837] | 0.2850 [0.1813, 0.3740] |
| Qwen-3.5-35B-A3B | Oracle Persona | 0.2671 [0.1563, 0.3724] | 0.3810 [0.3041, 0.4447] | 0.1769 [0.1002, 0.2692] | 0.2602 [0.1882, 0.3264] |
| Qwen-3.5-35B-A3B | Coco-PE | 0.4514 [0.4007, 0.5104] | 0.7369 [0.6597, 0.8009] | 0.2603 [0.2106, 0.3200] | 0.2850 [0.2475, 0.3300] |
| Qwen-3.5-35B-A3B | Coco-LoRA | 0.1911 [0.1068, 0.2790] | 0.5385 [0.4452, 0.6201] | 0.2563 [0.2031, 0.2738] | 0.2682 [0.1935, 0.3382] |
| Nemotron-3-Nano-Omni | Vanilla | 0.0362 [0.0000, 0.0889] | 0.0951 [0.0376, 0.1641] | 0.1525 [0.0840, 0.2751] | 0.1863 [0.1238, 0.2486] |
| Nemotron-3-Nano-Omni | Induced Persona | 0.0657 [0.0102, 0.1501] | 0.1991 [0.1194, 0.2850] | 0.3043 [0.1733, 0.4511] | 0.3075 [0.1920, 0.4014] |
| Nemotron-3-Nano-Omni | Oracle Persona | 0.1721 [0.0950, 0.2584] | 0.5247 [0.4393, 0.6033] | 0.1515 [0.0300, 0.2308] | 0.2759 [0.2483, 0.3934] |
| Nemotron-3-Nano-Omni | Coco-PE | 0.3229 [0.2879, 0.3761] | 0.4737 [0.3548, 0.5903] | 0.1928 [0.0957, 0.3012] | 0.2949 [0.1774, 0.4003] |
| Nemotron-3-Nano-Omni | Coco-LoRA | 0.1771 [0.1314, 0.2253] | 0.8609 [0.8204, 0.8948] | 0.1635 [0.0901, 0.2904] | 0.1923 [0.1215, 0.2554] |
Additional experiment: broader persona coverage. To assess whether the gains extend beyond the original four personas, we applied the same data-synthesis pipeline to three additional personas sampled from Nemotron-Personas-USA (https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA), augmented with AI-use habits and assistance preferences. These personas describe an aspiring electrician, a heavy-equipment operator and logistics coordinator, and a retired maintenance mechanic.
Joseph Giordano, a diligent aspiring master electrician, channels his methodical mindset and love for routine into residential wiring and emerging solar PV projects, ensuring NEC compliance while planning to eventually open a modest family‑owned contracting shop where they can mentor apprentices in a low‑pressure setting. … AI experience: Often accepts the first answer and may save drafts without reviewing them. Habits: Focused, reliable, tackles difficult tasks early, and frequently skips breaks. Welcomes specific help even during steady progress when it saves effort, resolves uncertainty, catches errors, or explains unfamiliar steps. Preferred help: Concise next steps, editable drafts, targeted explanations, concrete checks, and step-by-step guidance.
Bernadette Gangaram, a 28‑year‑old heavy‑equipment operator and logistics coordinator, leverages their methodical mindset and innate curiosity to excel in safety compliance, precision route planning, and mentoring new operators while quietly championing emerging autonomous technologies on the job site. … AI experience: Often accepts the first answer and may overlook requirements or mistake fluent explanations for understanding. Habits: Focused, reliable, takes regular breaks, and tackles difficult tasks early. Welcomes occasional targeted help for visible uncertainty, repeated effort, unfamiliar steps, or meaningful improvements in accuracy or efficiency. Preferred help: Concise next steps, editable drafts, targeted explanations, concrete checks, and step-by-step guidance.
Antonio Gonzalez, a retired maintenance mechanic with four decades of plant repair experience, they now channel their practical curiosity into woodworking, home repair mentorship, and bilingual storytelling, balancing flexible, sometimes disorganized project planning with a competitive edge that drives them to perfect every bench they build. … AI experience: Works manually and may rewrite routine notes or assume requirements without checking the source. Habits: Focused, sometimes forgets commitments, takes regular breaks, and tackles difficult tasks early. Welcomes targeted help for visible uncertainty, repeated effort, unfamiliar steps, or meaningful improvements in accuracy or efficiency. Preferred help: Concise next steps, step-by-step guidance, explanations of unfamiliar features, and reminders to record commitments, save work, and check unfinished steps.
| Persona | Training examples (positive) | Test examples (positive) |
|---|---|---|
| Joseph Giordano | 1,345 (156) | 725 (71) |
| Bernadette Gangaram | 1,351 (78) | 731 (35) |
| Antonio Gonzalez | 1,349 (102) | 730 (53) |
We evaluated Qwen-3.5-9B using the same five methods and the results are as follows. Coco-PE and Coco-LoRA achieve macro F1 scores of 0.5993 and 0.4965, respectively, compared with 0.1871 for Vanilla and 0.2455 for Oracle Persona. Both methods also improve precision and recall over Vanilla for all three personas, although false-alarm rates increase slightly. These results provide additional evidence that the gains extend beyond the original persona set in Table 1.
| Method | Joseph | Bernadette | Antonio | Macro F1 |
|---|---|---|---|---|
| Vanilla | 0.1569 | 0.2069 | 0.1975 | 0.1871 |
| Induced Persona | 0.2118 | 0.3438 | 0.1786 | 0.2447 |
| Oracle Persona | 0.3418 | 0.1961 | 0.1985 | 0.2455 |
| Coco-PE | 0.6854 | 0.4364 | 0.6763 | 0.5993 |
| Coco-LoRA | 0.4873 | 0.4865 | 0.5158 | 0.4965 |
We thank the reviewer for the detailed and constructive review, and for recognizing that the work addresses a timely and important problem, provides system contributions, and that the data-synthesis pipeline is transparent and well documented. We address each question below.
Thanks for brining up this point. We apologize that the naming in Appendix B.4 caused confusion. The "ground truth label" in the Reflector prompt is not an oracle supplied at test time. It is the training target ot’ produced by the implicit-feedback step in Section 3.3. We renamed the placeholder field name as “<label>” in the prompt and clarified in the text.
Here, we appreciate a chance to walk through the self-adaptation pipeline in Deep Personalization:
On OSWorld-Persona, step 2 cannot come from a real person, so the persona-conditioned simulated user plays that role: its decision stands in for whether the user would have engaged with, dismissed, or later sought the help. Importantly, labels are used only on the training split; the test split is chronologically later and its labels are never seen during adaptation.