- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PersonalWorkEnv, a synthetic benchmark and training framework for evaluating personalized AI assistants in multi-day work environments. The benchmark focuses on whether agents can remember and apply user-specific work norms revealed in previous interactions, including rules that may be updated or replaced over time.
The authors construct 50 synthetic environments spanning seven work sectors, comprising 1,126 tasks and 500 personalization profiles. They introduce the personalization gap to distinguish successful task completion from compliance with user-specific constraints, using both deterministic validators and LLM-based evaluations.
Experiments across seven language models reveal substantial personalization gaps, including cases where agents fail to apply rules already available in their memory. The authors also demonstrate that the environments can support reinforcement learning through a mixed-curriculum training strategy. While RL improves overall task success, additional analyses show that these improvements do not necessarily translate into better personalization compliance.
Critical strengths
1. Well-designed and carefully validated synthetic work environments.
The paper presents a systematic pipeline for generating executable work environments, including user personas, multi-day tasks, stateful CLI tools, personalization profiles, and corresponding validators. I particularly appreciate the effort devoted to environment quality, including automated consistency checks, golden solutions, and human auditing. The resulting benchmark appears carefully constructed rather than being a simple collection of LLM-generated instructions. I believe this provides a useful evaluation and training resource for studying personalized work assistants.
2. Meaningful evaluation framework and insightful empirical findings.
The distinction between surface task success and personalization compliance is useful, as conventional agent benchmarks may overlook failures to respect previously established user-specific constraints. The experiments demonstrate that such failures remain common even among strong models. I also find the conditional-compliance analysis valuable: it shows that improvements in overall task success through RL do not necessarily imply improvements in personalization compliance. These findings provide useful insights for future research on personalized agents.
Critical weaknesses
1. The operationalization of personalization is somewhat narrow.
The benchmark primarily evaluates whether agents follow explicitly introduced user-specific rules across multiple tasks. While this is a meaningful aspect of personalization, it does not fully capture the broader challenge of learning user preferences from interaction histories, particularly when preferences are implicit or uncertain. Moreover, since the harness loads the entire MEMORY.md into the agent's context before each task, some of the observed failures may reflect general instruction-following or multi-constraint satisfaction difficulties rather than personalization-specific limitations. A comparison with an oracle setting where all active rules are explicitly provided in the current task would help clarify this distinction.
2. Limited evidence of generalization to realistic work environments.
Although the synthetic environments are carefully constructed, several design choices simplify real-world workflows. For example, the fixed schedule for introducing profiles and the resetting of task workspaces reduce the complexity of long-term interactions and dependencies between tasks. Furthermore, the RL experiments use held-out environments generated by the same pipeline, with checkpoint selection performed on the validation split. The external transfer evaluation is also relatively limited and relies on adapted tools and validators. These factors make it difficult to determine how well the reported improvements would generalize to more realistic personalized work settings.
Questions for authors
Q1. Distinguishing personalization from general instruction following.
Since the benchmark primarily evaluates compliance with explicitly revealed user-specific rules, and the full MEMORY.md is provided in context, how much of the observed personalization gap can be attributed to personalization-specific difficulties rather than general multi-constraint instruction following? Have the authors considered an oracle-profile baseline in which all active user rules are explicitly included in each task prompt? Such a comparison would help establish what additional challenges arise from longitudinal personalization.
Q2. Generalization and environmental realism.
The environments use a fixed profile introduction schedule and reset the workspace between tasks, while the held-out environments share the same generation pipeline as the training environments. Could the authors provide additional evidence that the benchmark captures meaningful challenges beyond these controlled synthetic settings? In particular, how was the realism of the generated work scenarios assessed, and to what extent do the reported RL improvements generalize beyond the shared environment-generation and validation conventions?
3 comments
Response to Reviewer AzuL (Part 2/2)
Q2. How realistic is the setting, and how far do the results generalize?
We agree these should be stated more carefully. The revision separates four kinds of evidence and what each does and does not establish (new table below), and corrects one description of the environment.
| Evidence | What it establishes | What it does not |
|---|---|---|
| Scoring validity (tool tests, golden end states, auditor review) | Checks grade what they claim to grade | How realistic scenarios are |
| Scenario coverage (50 environments, explicit rules on a fixed schedule) | Controlled, comparable rule exposure across models | How people actually disclose preferences |
| Within-generator learning (held-out environments) | Learning signal under shared conventions | Anything beyond this generator |
| External transfer (one adapted suite) | A limited portability check | Broad transfer |
What is controlled. Rules are stated explicitly, two per day on a fixed schedule. This is an experimental control so that every model gets the same opportunity to learn each rule, not a model of how people disclose preferences. 82% of rule checks fall on tasks where the rule is not restated, so most of the benchmark tests use of previously stated rules.
What persists between tasks (corrected). Thank you for raising this; it made us correct our description.
-
Conversation: each task starts a new conversation; the memory file is the only channel carrying conversational context forward.
-
Workspace: tool state and files persist across the tasks of one simulated day and are re-seeded at each day boundary, as described in Appendix A. All reported evaluation and training runs use this per-day reset.
-
Correction: the sentence in Section 2.1 saying the workspace is reset before every task was inaccurate; we have corrected it there and in Sections 3 and 4.1.
-
Grading: validators scope their checks to records created by the current task; an artifact written earlier the same day remains visible to the assistant, and we now state this.
How scenarios and graders were checked. Every environment ships tool tests and a golden end state; each environment's checks are verified to fail on the untouched seed workspace and pass on the golden state, and task-check pairs were reviewed by two independent auditors (Section 3, Appendix J.1). This establishes scoring validity, not realism: we have no real-user or workplace data, and we now say so directly.
How far the training result carries. The held-out environments come from the same generator, so they show learning signal under shared conventions and nothing broader. The external environment is a single adapted suite with remapped tools and re-implemented validators; we keep its result in the appendix (Table 15) but describe it in the main text as a limited portability check, not as evidence of transfer.
Thank you for the detailed and thoughtful rebuttal. I appreciate the additional oracle-profile experiments and the authors' careful clarification of the benchmark's scope.
The oracle results help distinguish longitudinal personalization challenges from general multi-constraint instruction-following difficulties, although the relative contribution of each remains somewhat unclear. I also appreciate the clarification regarding workspace persistence and the more appropriately qualified claims about realism and generalization.
My remaining concern is that the evidence for generalization beyond the controlled synthetic environments remains limited. Nevertheless, I continue to find the benchmark construction and empirical analyses valuable contributions.
Overall, the rebuttal has strengthened my confidence in the paper, but I do not see sufficient new evidence to warrant a higher rating. I will maintain my score of 3.
Response to Reviewer AzuL (Part 1/2)
We thank the reviewer for the supportive and precise review. We answer the two questions in turn.
Q1. Does the benchmark measure personalization or general instruction following?
Both are involved, and the oracle comparison you suggested gives a first measure of the instruction-following part. With all active rules stated in the request, 29% (Claude Haiku 4.5) and 19% (Qwen3.8-27B) of surface-successful tasks still violate at least one rule. A substantial part of the gap is therefore the multi-constraint instruction-following component you describe, and an explicit oracle recovers a modest part of it.
Setup. We ran the oracle comparison together with a corrected-memory condition requested by another reviewer, under the paper's evaluation protocol. Models: Claude Haiku 4.5 (in the submission) and Qwen3.8-27B, a newer model not in the submission (distinct from Qwen3.6-27B). 150 saved tasks per model, 3 from each of the 50 environments, sampled before any outcome was seen; the exact workspace and notes are restored, and task, tools and grading are held fixed.
Table R4 (short form) — All-check success over 150 tasks.
Residual violations among surface-successful tasks (on tasks with both surface and rule checks):
A substantial part of the gap is present when the rules are simply instructions in context, consistent with the difficulty of satisfying multiple simultaneous constraints reported by instruction-following benchmarks [1, 2]. The oracle recovers a modest part; with two models, 150 tasks each and one run per condition, we cannot say precisely how much the longitudinal setting adds beyond that, and we do not claim it adds nothing.
Denominators: residuals use the tasks carrying both surface and rule checks (137 of 150 for Haiku 4.5, 136 for Qwen3.8-27B), of which 123 and 125 are surface-successful under the oracle; all-check success on that subset under the oracle is 0.635 and 0.743. The full table and limits are in a new subsection of Section 3.
How we position the contribution. We do not claim a clean boundary between the two. Our target is narrower than personalization in general: tracking which explicitly stated user rules are in force, including replaced ones, and reflecting them in later artifacts. Instruction following is part of that capability. We have retitled the relevant paragraph, removed the implication that failures mainly reflect memory maintenance, and stated that implicit or uncertain preferences are out of scope.
References [1] J. Zhou et al. Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911, 2023. [2] Y. Jiang et al. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. ACL 2024.