acceptodds
Under review as a conference paper at ICLR 2027

PersonalWorkEnv: Evaluating and Training Personalized AI Assistants with Synthetic Work Environments

Abstract

A personal work assistant can complete a task while violating a user-specific norm revealed in an earlier interaction. We call this failure mode the and introduce , a synthetic environment-generation pipeline for evaluating and training : remembering and applying previously revealed work norms to later tool-produced artifacts. Each environment combines a user persona, six simulated workdays, roughly twenty tasks, simple CLI tools, and ten progressively introduced profiles, including temporary norms that are later replaced. Deterministic validators and rubric-based soft checks distinguish surface task completion from profile compliance. We instantiate , a double-audited suite of fifty environments, and find a substantial personalization gap across seven proprietary and open models, with larger gaps on day 6 than day 1. As a baseline training use case, mixed-curriculum RL improves small Qwen3.5 models' all-check success. Conditional-compliance diagnostics show that the 9B gains primarily reflect better surface execution, whereas the 35B-A3B model also improves profile compliance among surface-successful tasks.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, synthetic environments, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AzuLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PersonalWorkEnv, a synthetic benchmark and training framework for evaluating personalized AI assistants in multi-day work environments. The benchmark focuses on whether agents can remember and apply user-specific work norms revealed in previous interactions, including rules that may be updated or replaced over time.

The authors construct 50 synthetic environments spanning seven work sectors, comprising 1,126 tasks and 500 personalization profiles. They introduce the personalization gap to distinguish successful task completion from compliance with user-specific constraints, using both deterministic validators and LLM-based evaluations.

Experiments across seven language models reveal substantial personalization gaps, including cases where agents fail to apply rules already available in their memory. The authors also demonstrate that the environments can support reinforcement learning through a mixed-curriculum training strategy. While RL improves overall task success, additional analyses show that these improvements do not necessarily translate into better personalization compliance.

Critical strengths

1. Well-designed and carefully validated synthetic work environments.

The paper presents a systematic pipeline for generating executable work environments, including user personas, multi-day tasks, stateful CLI tools, personalization profiles, and corresponding validators. I particularly appreciate the effort devoted to environment quality, including automated consistency checks, golden solutions, and human auditing. The resulting benchmark appears carefully constructed rather than being a simple collection of LLM-generated instructions. I believe this provides a useful evaluation and training resource for studying personalized work assistants.

2. Meaningful evaluation framework and insightful empirical findings.

The distinction between surface task success and personalization compliance is useful, as conventional agent benchmarks may overlook failures to respect previously established user-specific constraints. The experiments demonstrate that such failures remain common even among strong models. I also find the conditional-compliance analysis valuable: it shows that improvements in overall task success through RL do not necessarily imply improvements in personalization compliance. These findings provide useful insights for future research on personalized agents.

Critical weaknesses

1. The operationalization of personalization is somewhat narrow.

The benchmark primarily evaluates whether agents follow explicitly introduced user-specific rules across multiple tasks. While this is a meaningful aspect of personalization, it does not fully capture the broader challenge of learning user preferences from interaction histories, particularly when preferences are implicit or uncertain. Moreover, since the harness loads the entire MEMORY.md into the agent's context before each task, some of the observed failures may reflect general instruction-following or multi-constraint satisfaction difficulties rather than personalization-specific limitations. A comparison with an oracle setting where all active rules are explicitly provided in the current task would help clarify this distinction.

2. Limited evidence of generalization to realistic work environments.

Although the synthetic environments are carefully constructed, several design choices simplify real-world workflows. For example, the fixed schedule for introducing profiles and the resetting of task workspaces reduce the complexity of long-term interactions and dependencies between tasks. Furthermore, the RL experiments use held-out environments generated by the same pipeline, with checkpoint selection performed on the validation split. The external transfer evaluation is also relatively limited and relies on adapted tools and validators. These factors make it difficult to determine how well the reported improvements would generalize to more realistic personalized work settings.

Questions for authors

Q1. Distinguishing personalization from general instruction following.

Since the benchmark primarily evaluates compliance with explicitly revealed user-specific rules, and the full MEMORY.md is provided in context, how much of the observed personalization gap can be attributed to personalization-specific difficulties rather than general multi-constraint instruction following? Have the authors considered an oracle-profile baseline in which all active user rules are explicitly included in each task prompt? Such a comparison would help establish what additional challenges arise from longitudinal personalization.

Q2. Generalization and environmental realism.

The environments use a fixed profile introduction schedule and reset the workspace between tasks, while the held-out environments share the same generation pipeline as the training environments. Could the authors provide additional evidence that the benchmark captures meaningful challenges beyond these controlled synthetic settings? In particular, how was the realism of the generated work scenarios assessed, and to what extent do the reported RL improvements generalize beyond the shared environment-generation and validation conventions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AzuL (Part 1/2)

We thank the reviewer for the supportive and precise review. We answer the two questions in turn.

Q1. Does the benchmark measure personalization or general instruction following?

Both are involved, and the oracle comparison you suggested gives a first measure of the instruction-following part. With all active rules stated in the request, 29% (Claude Haiku 4.5) and 19% (Qwen3.8-27B) of surface-successful tasks still violate at least one rule. A substantial part of the gap is therefore the multi-constraint instruction-following component you describe, and an explicit oracle recovers a modest part of it.

Setup. We ran the oracle comparison together with a corrected-memory condition requested by another reviewer, under the paper's evaluation protocol. Models: Claude Haiku 4.5 (in the submission) and Qwen3.8-27B, a newer model not in the submission (distinct from Qwen3.6-27B). 150 saved tasks per model, 3 from each of the 50 environments, sampled before any outcome was seen; the exact workspace and notes are restored, and task, tools and grading are held fixed.

Table R4 (short form) — All-check success over 150 tasks.

ModelOwn notesNotes corrected to accurate rulesAccurate rules in the request (oracle)Oracle − own notes (95% CI)
Claude Haiku 4.50.5800.6200.640+0.060 (−0.033, +0.147)
Qwen3.8-27B0.6800.7070.740+0.060 (0.000, +0.120)

Residual violations among surface-successful tasks (on tasks with both surface and rule checks):

ModelOwn notesOracle
Claude Haiku 4.536% (44/123)29% (36/123)
Qwen3.8-27B25% (32/126)19% (24/125)

A substantial part of the gap is present when the rules are simply instructions in context, consistent with the difficulty of satisfying multiple simultaneous constraints reported by instruction-following benchmarks [1, 2]. The oracle recovers a modest part; with two models, 150 tasks each and one run per condition, we cannot say precisely how much the longitudinal setting adds beyond that, and we do not claim it adds nothing.

Denominators: residuals use the tasks carrying both surface and rule checks (137 of 150 for Haiku 4.5, 136 for Qwen3.8-27B), of which 123 and 125 are surface-successful under the oracle; all-check success on that subset under the oracle is 0.635 and 0.743. The full table and limits are in a new subsection of Section 3.

How we position the contribution. We do not claim a clean boundary between the two. Our target is narrower than personalization in general: tracking which explicitly stated user rules are in force, including replaced ones, and reflecting them in later artifacts. Instruction following is part of that capability. We have retitled the relevant paragraph, removed the implication that failures mainly reflect memory maintenance, and stated that implicit or uncertain preferences are out of scope.

References [1] J. Zhou et al. Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911, 2023. [2] Y. Jiang et al. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. ACL 2024.

AuthorsOpenReview ↗

Response to Reviewer AzuL (Part 2/2)

Q2. How realistic is the setting, and how far do the results generalize?

We agree these should be stated more carefully. The revision separates four kinds of evidence and what each does and does not establish (new table below), and corrects one description of the environment.

EvidenceWhat it establishesWhat it does not
Scoring validity (tool tests, golden end states, auditor review)Checks grade what they claim to gradeHow realistic scenarios are
Scenario coverage (50 environments, explicit rules on a fixed schedule)Controlled, comparable rule exposure across modelsHow people actually disclose preferences
Within-generator learning (held-out environments)Learning signal under shared conventionsAnything beyond this generator
External transfer (one adapted suite)A limited portability checkBroad transfer

What is controlled. Rules are stated explicitly, two per day on a fixed schedule. This is an experimental control so that every model gets the same opportunity to learn each rule, not a model of how people disclose preferences. 82% of rule checks fall on tasks where the rule is not restated, so most of the benchmark tests use of previously stated rules.

What persists between tasks (corrected). Thank you for raising this; it made us correct our description.

  • Conversation: each task starts a new conversation; the memory file is the only channel carrying conversational context forward.

  • Workspace: tool state and files persist across the tasks of one simulated day and are re-seeded at each day boundary, as described in Appendix A. All reported evaluation and training runs use this per-day reset.

  • Correction: the sentence in Section 2.1 saying the workspace is reset before every task was inaccurate; we have corrected it there and in Sections 3 and 4.1.

  • Grading: validators scope their checks to records created by the current task; an artifact written earlier the same day remains visible to the assistant, and we now state this.

How scenarios and graders were checked. Every environment ships tool tests and a golden end state; each environment's checks are verified to fail on the untouched seed workspace and pass on the golden state, and task-check pairs were reviewed by two independent auditors (Section 3, Appendix J.1). This establishes scoring validity, not realism: we have no real-user or workplace data, and we now say so directly.

How far the training result carries. The held-out environments come from the same generator, so they show learning signal under shared conventions and nothing broader. The external environment is a single adapted suite with remapped tools and re-implemented validators; we keep its result in the appendix (Table 15) but describe it in the main text as a limited portability check, not as evidence of transfer.

Thank you for the detailed and thoughtful rebuttal. I appreciate the additional oracle-profile experiments and the authors' careful clarification of the benchmark's scope.

The oracle results help distinguish longitudinal personalization challenges from general multi-constraint instruction-following difficulties, although the relative contribution of each remains somewhat unclear. I also appreciate the clarification regarding workspace persistence and the more appropriately qualified claims about realism and generalization.

My remaining concern is that the evidence for generalization beyond the controlled synthetic environments remains limited. Nevertheless, I continue to find the benchmark construction and empirical analyses valuable contributions.

Overall, the rebuttal has strengthened my confidence in the paper, but I do not see sufficient new evidence to warrant a higher rating. I will maintain my score of 3.

Reviewer_omyzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PersonalWorkEnv, a framework for generating synthetic work environments that test whether assistants remember and apply user-specific norms across tasks. The benchmark contains 50 environments with progressively introduced and sometimes replaced norms. The evaluation of the proposed framework separates surface task completion from compliance with the norms. Experiments with seven models reveal substantial compliance failures, including cases where the relevant information is already in memory. Additional RL experiments demonstrate improved task success, although improvements in personalisation are less consistent.

Critical strengths
  • The benchmark targets a concrete problem where an assistant can complete the requested action while producing an artifact that violates previously established user requirements. Progressive norm introduction and replacement make this a meaningful longitudinal setting. Separating surface success from profile compliance also helps expose failures that an aggregate task score would obscure.
  • The evaluation includes substantial quality control and useful diagnostics. Automated checks, independent environment audits, and sampled human validation support the benchmark's reliability. The analysis of failures where relevant information is already in memory is particularly informative. I also find the distinction between better task execution and better personalisation interesting when interpreting the RL results.
Critical weaknesses
  • The evaluation lacks controlled baselines that establish how much difficulty comes from acquiring and maintaining norms versus applying instructions already in context. The finding that 68.9% of sampled failures involve profiles already recorded in memory is helpful, but it does not resolve this distinction quantitatively. Comparisons using an accurate, up-to-date profile memory and explicit reminders in the current request would make the benchmark substantially more diagnostic. This is important because the full memory file is loaded before every task, and later jointly change retention horizon, profile load, and replacement requirements.
  • The RL evidence remains limited by the evaluation protocol. The same ten environments are used for checkpoint selection and reporting, and selected mixed-schedule checkpoints are compared with final naive-schedule checkpoints. The paper acknowledges these limitations, but they still prevent a convincing evaluation of the scheduling benefit and generalisation.
Questions for authors
  1. How do performance and the personalisation gap change when the agent receives an oracle memory containing the currently active norms, or when those norms are repeated in the current request? These controls would clarify what the longitudinal setting contributes to the observed difficulty.
  2. Can you compare both scheduling strategies using the same checkpoint-selection rule? Results on an untouched environment split would also strengthen the training conclusions.
2 comments
AuthorsOpenReview ↗

Response to Reviewer omyz (Part 1/2)

We thank the reviewer for two specific requests. Both pointed at real gaps, and we have addressed each during the rebuttal.

Q1. Does correcting the memory, or restating the rules, close the gap?

Only partly. With every active rule stated in the request, 29% (Claude Haiku 4.5) and 19% (Qwen3.8-27B) of surface-successful tasks still violate at least one rule, and the reminder raises all-check success by a modest 6 points for both models. We agree that our earlier observation (most sampled failures had the relevant rule in memory) did not quantify this, and we have removed wording that attributed failures mainly to memory maintenance.

We ran the control you describe as a paired study on saved task snapshots, under the paper's evaluation protocol:

  • Models. Claude Haiku 4.5 (in the submission) and Qwen3.8-27B, a newer model not in the submission (distinct from Qwen3.6-27B).

  • Tasks. 150 per model, 3 per environment across all 50, sampled from task metadata before any outcome was seen (covering original successes/failures, retained/replaced rules, and different numbers of checks). All 150 completed in all three conditions.

  • Conditions. Each task is restored to the exact workspace and notes that model had, then run three times:

    • B — the assistant's own notes;
    • O — same notes, rule content replaced by the accurate list of rules in force;
    • R — own notes, with that same list placed in the request.

    The list contains every rule disclosed so far with replacements applied, in the user's own wording, and nothing from validators, later days or reference solutions.

Table R4 — Memory controls. Columns marked † use tasks carrying both surface and rule checks (137 Haiku 4.5, 136 Qwen3.8-27B), so surface success − all-check success = gap.

Model / conditionAll-check success ↑Surface success †All-check success †Gap † ↓Surface-successful but violating † ↓Checks violated / task ↓
Claude Haiku 4.5
B · own notes0.5800.8980.5770.32136% (44/123)0.216
O · corrected notes0.6200.9050.6280.27731% (38/124)0.182
R · rules in request0.6400.8980.6350.26329% (36/123)0.181
Qwen3.8-27B
B · own notes0.6800.9260.6910.23525% (32/126)0.151
O · corrected notes0.7070.9040.7060.19922% (27/123)0.134
R · rules in request0.7400.9190.7430.17619% (24/125)0.110

Paired differences in all-check success (95% intervals, resampling environments):

O − B (correct notes)R − O (move to request)R − B (total)
Haiku 4.5+0.040 (−0.053, +0.133)+0.020 (−0.040, +0.087)+0.060 (−0.033, +0.147)
Qwen3.8-27B+0.027 (−0.033, +0.087)+0.033 (−0.033, +0.100)+0.060 (0.000, +0.120)
AuthorsOpenReview ↗

Response to Reviewer omyz (Part 2/2)

The study therefore supports the narrower statement you anticipated: a substantial residual of rule violations remains when acquiring and maintaining the rules is taken out of the picture, and an explicit reminder recovers a modest part of the gap. Surface success does not change materially in any condition. In more detail:

  • R − B. +6 points for both models; Haiku's interval includes zero and Qwen's starts at zero. For Qwen, the reductions in gap (−0.059; −0.121, −0.007) and checks violated per task (−0.041; −0.079, −0.006) are distinguishable from zero.

  • O − B. Correcting the notes alone moves all-check success by 3–4 points with intervals including zero, so we claim neither an effect of corrected notes nor that the notes do not matter.

  • Sampling. Each condition was run once; tasks change outcome in both directions (B → R: 25 fail→pass and 16 pass→fail for Haiku 4.5; 17 and 8 for Qwen3.8-27B), so single-task changes are largely sampling variation.

  • Scope. This is a snapshot diagnostic: it changes one decision point and does not reproduce the notes that would have developed had the assistant been helped from day 1.

  • Contrast purity. O − B can mix completeness, freshness and wording; correcting the notes also removed some lines mixing a rule with other facts (687 of 7,230 removed lines for Haiku 4.5), which may make O slightly conservative. R − O can reflect position and emphasis.

The study and its limits are added as a new subsection of Section 3.

Q2. Is the scheduling comparison like-for-like?

You are right that a selected mixed-schedule checkpoint against a final ordinary-schedule checkpoint is not like-for-like. Table 2 now reports both schedules at step 100, with the step-60 rows kept separately and labelled as selected on validation:

Table R5 — Step 100, both schedules (score / all-check success; values from the paper's own tables).

ModelOrdinary scheduleMixed scheduleΔ (mixed − ordinary)
Qwen3.5-9B0.699 / 0.5110.848 / 0.579+0.149 / +0.068
Qwen3.5-35B-A3B0.815 / 0.5440.850 / 0.601+0.035 / +0.057

Aggregation note: ordinary-schedule values come from Table 2 and mixed-schedule values from the per-day appendix tables, which average within environment first (the base-model rows differ by 0.004–0.005 between those tables), so we present the step-100 difference as indicative.

What we now state explicitly. We did not evaluate on a split untouched by checkpoint selection, and applying the same rule to both schedules does not change that. The revision says plainly that (a) these are results on the held-out validation environments, (b) the advantage of mixing is suggestive for the smaller model and small for the larger one, and (c) the training section is a baseline use of the benchmark, not evidence of generalization beyond it. The finding we keep is the one you noted: training improved task execution more consistently than compliance among surface-successful tasks.

Reviewer_eGL2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a synthetic environment-generation framework for evaluating and training personalized AI assistants. The framework focuses on measuring whether agents can retain and apply user-specific work norms introduced in earlier interactions. To support this, the authors construct 50 environments, 1,126 tasks, and 500 profiles for deterministic and LLM-based checks to evaluate the resulting work artifacts.

The paper distinguishes surface task completion from true compliance where success satisfies all applicable personalization constraints. By using the personalization gap and conditional profile success as diagnostic metrics, evaluations across seven language models reveal substantial personalization failures. The authors also explore a reinforcement learning strategy for two Qwen3.5 models, reporting improvements in overall task success, although the gains in conditional personalization compliance are not consistent across models.

Critical strengths

This paper addresses a relevant failure mode in AI agents, which is on completing an explicit task while failing to incorporate previously established user preferences.

One strength of the proposed benchmark is its evaluation framework. Rather than just assessing preference recall, it evaluates whether agents actually apply preferences in generated tool artifacts. Furthermore, incorporating evolving and superseded user profiles creates a more realistic and informative dynamic evaluation setting compared to static benchmarks.

The evaluation framework is well designed. Combining deterministic checks for structured requirements with rubric-based LLM assessments for subjective criteria provides a balanced approach to measurement. The additional auditing and judge-agreement analyses add to the consistency of the evaluation. The experiments effectively highlight that optimizing an agent for general task success does not inherently improve its profile compliance.

Critical weaknesses

Because the multi-day experiments mix several factors by simultaneously scaling task complexity, profile counts, active constraints, and obsolete profile replacements. Hence it could be performance declines over time may simply be due to the increasing number of evaluation checks rather than degraded memory or adaptation.

The ability to generalize is also not immediately evident, and could be more clearly discussed by the authors.

Questions for authors

Could the authors provide a more fine-grained analysis of personalization performance while controlling for the number of applicable profile checks and task difficulty? In particular, how do the results differ across profile-retention and profile-replacement tasks, and does the day-wise degradation remain when comparing tasks with similar numbers and types of active constraints?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eGL2 (Part 1/3)

We thank the reviewer for the careful reading. You asked whether the day-wise decline survives when tasks with similar numbers and types of active constraints are compared, and how retention and replacement differ. We ran this analysis on two models.

In short, most of the decline tracks the growing number of rule checks. At a fixed number of checks, the late-minus-early difference in all-check success shrinks from −0.218 to −0.083 for Claude Haiku 4.5 and from −0.172 to −0.075 for Qwen3.8-27B, and both remaining 95% intervals include zero. The per-check violation rate is still 7–8 points higher late, but once checks on rules stated in the same request and on replaced rules are set aside, Haiku 4.5 shows no late increase and Qwen3.8-27B a small one (4–5 points) whose interval reaches zero. We have therefore withdrawn the reading of the raw day curve as forgetting and report the remainder as an association.

Setup. We made two full runs over all 50 environments and 1,126 tasks under the paper's evaluation protocol (same judge, temperature 0.6, top-p 0.95), with per-check logs kept for this analysis: Claude Haiku 4.5 (in the submission) and Qwen3.8-27B, a newer model not in the submission (distinct from Qwen3.6-27B). Haiku 4.5 was rerun, so its values differ slightly from Table 9 through sampling (overall all-check success 0.660 vs. 0.639). Qwen3.8-27B: overall all-check success 0.754, gap 0.173.

Table R1 — Per-day trends. Load rises ~2.8× from day 1 to day 6; all-check success falls with it.

MetricDay 1Day 2Day 3Day 4Day 5Day 6Δ (6 − 1)
Rule checks per task1.471.852.392.712.994.06+2.59 ↑
Claude Haiku 4.5
All-check success0.8300.7230.6890.6530.6280.480−0.350 ↓
Rule checks violated (share)0.1200.1740.1770.1860.1740.207+0.087 ↑
Pass all rule checks — observed0.8300.7380.6940.6730.6330.495−0.335 ↓
Pass all rule checks — constant-rate yardstick0.7550.6930.6400.6050.5710.466−0.289 ↓
Qwen3.8-27B
All-check success0.8760.8100.8110.7240.7400.596−0.280 ↓
Rule checks violated (share)0.1020.1090.0960.1320.1290.189+0.087 ↑
Pass all rule checks — observed0.8890.8310.8160.7390.7500.606−0.283 ↓
Pass all rule checks — constant-rate yardstick0.8130.7630.7170.6870.6580.566−0.247 ↓

For both models the observed row lies 3–10 points above the yardstick on every day and falls by a similar amount, so the growth in checks alone would produce a decline of roughly the observed size. The yardstick is the share of tasks that would pass all their rule checks if every check failed independently at that run's overall hard- and soft-check rates, given each day's actual number of checks. We use it descriptively only, since checks are not independent.

AuthorsOpenReview ↗

Response to Reviewer eGL2 (Part 2/3)

Comparison at a fixed number of rule checks. Table R2 compares late and early days within strata of equal load.

Table R2 — Late (days 5–6) minus early (days 1–2), 95% intervals. AC = all-check success (negative = worse late); V = share of rule checks violated (positive = worse late). Bold = interval excludes zero; ★ = key rows.

ComparisonHaiku 4.5 — ACHaiku 4.5 — VQwen3.8-27B — ACQwen3.8-27B — V
Main contrasts
Unmatched−0.218 (−0.290, −0.145)+0.054 (+0.008, +0.104)−0.172 (−0.241, −0.109)+0.066 (+0.017, +0.112)
★ Same number of rule checks−0.083 (−0.193, +0.022)+0.083 (+0.014, +0.155)−0.075 (−0.167, +0.024)+0.069 (−0.007, +0.143)
Robustness
+ same tool family−0.054 (−0.156, +0.037)+0.068 (+0.004, +0.133)−0.052 (−0.147, +0.029)+0.063 (−0.002, +0.136)
Averaged within environment−0.108 (−0.221, +0.005)+0.079 (+0.009, +0.152)−0.048 (−0.130, +0.030)+0.038 (−0.019, +0.097)
Hard checks only—+0.080 (+0.006, +0.157)—+0.070 (−0.008, +0.146)
Previously stated rules only (days 2–3 vs. 5–6)
Unmatched−0.121 (−0.215, −0.031)+0.004 (−0.063, +0.072)−0.145 (−0.223, −0.071)+0.059 (+0.012, +0.108)
★ Same number of rule checks−0.025 (−0.127, +0.114)+0.012 (−0.072, +0.082)−0.087 (−0.167, +0.016)+0.067 (+0.010, +0.118)
+ same tool family—−0.001 (−0.070, +0.076)—+0.054 (−0.006, +0.119)
Never-replaced rules only, same number of checks †—+0.023 (−0.040, +0.073)—+0.044 (−0.004, +0.086)

† Additional check, not one of the contrasts fixed in advance.

Matching procedure.

  • Strata. Tasks are stratified by their exact number of rule checks (and, for family rows, by which tool stores the graders read: email only / email + another tool / other tools only / workspace files only). Strata and families are fixed from the task definitions before any outcome is read.

  • Weighting. Only strata present on both sides are used; each stratum's difference is weighted by its pooled task count.

  • Intervals. Cluster bootstrap over whole environments, 2,000 resamples [1], recomputing common strata and weights inside each resample.

  • Coverage. The days 1–2 vs. 5–6 contrast retains all 336 early tasks and 298 of 394 late tasks; tasks with ≥5 checks have no early counterpart. Strata (early tasks/envs; late tasks/envs): 1 check: 161/46, 36/23; 2: 130/46, 75/37; 3: 36/21, 111/43; 4: 9/6, 76/40. Exact day 1 vs. day 6 retains only 89 of 198 day-6 tasks (smallest stratum: 4 tasks), so we do not base conclusions on it. Family and previously-stated rows include single-task strata and are sensitivity checks.

References [1] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.

AuthorsOpenReview ↗

Response to Reviewer eGL2 (Part 3/3)

Holding the number of checks fixed removes more than half of the late-early difference in all-check success for both models. The per-check rate does not shrink in the same way, and two things other than time can raise the late per-check rate: (i) early days contain many checks on a rule stated in the same request (106 of 225 on day 1, none on day 6), and (ii) late days carry almost all replaced rules. Restricting to previously stated rules addresses (i): Haiku 4.5 then shows no late-early difference (+0.012), Qwen3.8-27B keeps one (+0.067). The never-replaced row (†) addresses both: +0.023 for Haiku 4.5 and +0.044 for Qwen3.8-27B, both intervals reaching zero, and the Qwen value is sensitive to a few sparse strata. We therefore claim neither a retention effect nor its absence. All of these are associations; task content still varies across days.

Retention and replacement. Table R3 breaks down all 2,962 rule checks by rule status.

Table R3 — Per-check violation rate by rule status.

Rule checkedChecksHaiku 4.5 violated / rateQwen3.8-27B violated / rate
Unchanged rule, in the request that states it41354 / 0.13143 / 0.104
Replacing rule, in the request that states it12121 / 0.17415 / 0.124
Unchanged rule, in a later request2,192404 / 0.184295 / 0.135
Replacing rule, in a later request23658 / 0.24651 / 0.216

The ordering is the same for both models. A rule checked in a later request is violated more often than one checked where it is stated (unchanged rules: +0.054 (+0.013, +0.094) for Haiku 4.5, +0.030 (−0.005, +0.064) for Qwen3.8-27B), and replacing rules are violated more often than unchanged rules on the same days (+0.064 (−0.006, +0.133) and +0.060 (−0.001, +0.127); hard checks only +0.031 and +0.039), although these intervals reach zero.

This table also led us to correct two descriptions in the paper:

  1. Timing of replacement. "From day 4 onward" is true, but only 3 of 100 replacements are disclosed on day 4 and 97 on day 5, so replacement is almost entirely a day-5/6 phenomenon and cannot be separated from day.
  2. "Suppress the obsolete rule." Every check after a replacement targets the new rule; none targets the old one. We measure failures to follow the new rule, not continued use of the old one, and have reworded "suppression" accordingly.

Changes to the paper. The submission already cautioned against a causal reading of the day trend (Section 3, Appendix G). The revision replaces that caution with these numbers: Tables R1–R3 become a new diagnostic subsection, and the day-wise paragraph now states that (a) all-check success falls as checks accumulate, (b) more than half of the late-early difference is absent at a fixed number of checks, and (c) what remains per check is small and uncertain. The two descriptions above are corrected.

Generalization. You are right that the submission did not make this clear, and a scope statement alone would not answer it. What we can add is limited but specific:

  1. A second, previously unevaluated model. The analysis above holds on Qwen3.8-27B with the same main pattern and one stated difference.
  2. Difficulty is not specific to our memory mechanism. The oracle-rule study requested by the other reviewers shows that with every active rule placed in the request, 29% (Haiku 4.5) and 19% (Qwen3.8-27B) of surface-successful tasks still violate at least one rule.

What we cannot show is transfer beyond this generator: all 50 environments share one construction pipeline, rules are explicit and arrive on a fixed schedule, and we have no real-workplace data. The revised limitations paragraph says this in those terms and presents the single adapted external environment as a limited portability check, not as evidence of broad transfer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.