acceptodds
Withdrawn from ICLR 2027

Sleep-Time Training: From Deployment Experience to Persistent Agent Improvement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large-scale pre-training and post-training produce capable LLM agents, but their parameters typically remain static after deployment. Existing deployment-time improvement methods mainly rely on additional computation during user interaction or external memory, which either increases user-facing latency or does not directly improve the agent's policy. We introduce Sleep-Time Training (STT), a deployment-time learning paradigm in which agents improve during user-idle periods. During user-facing sessions, the agent collects interaction trajectories. During sleep time, an auxiliary model converts these trajectories into corrected demonstrations and identifies defective intermediate actions. We first learn from replayed demonstrations with supervised fine-tuning, and then perform group-relative reinforcement learning over original and iteratively repaired trajectories. To study persistent improvement across repeated interactions, we introduce STT-Bench, a four-day benchmark containing 20 multi-day base task families and 20 held-out generalization families, with hidden user preferences and different levels of preference disclosure. Experiments show that STT improves future task completion and preference adherence without increasing user-facing inference cost, while retaining gains across multiple days and transferring to related unseen tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agents, Supervised Fine-Tuning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GsLqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Sleep-Time Training (STT), which combines supervised fine-tuning on corrected demonstrations with group-relative reinforcement learning over original and iteratively repaired trajectories to improve agents from deployment experience during user-idle periods.

Critical strengths
  • This paper addresses an important problem: helping deployed agents learn from past interactions and improve on future tasks.

  • This paper introduces STT-Bench to evaluate improvement across days, transfer to unseen tasks, and adherence to user preferences.

Critical weaknesses
  • “Sleep-time” and “user-idle periods” describe a practical deployment setting, but the underlying approach appears to be periodic training between interaction sessions. It is unclear whether this setting introduces a distinct research contribution.

  • The method appears to distill knowledge and behavior from a stronger model through SFT demonstrations, trajectory corrections, and GRPO reward judgments. The contribution beyond this teacher-guided training pipeline is unclear.

  • Evaluation is limited to the proposed STT-Bench, which uses a single synthetic persona and simulated tools. Results on established agent benchmarks would help assess whether the improvements generalize beyond this setting.

Questions for authors
  • How does STT differ from simply distilling a stronger model’s knowledge through SFT/GRPO? A more explicit comparison with existing approaches would help clarify the contribution.
  • All experiments use STT-Bench. It would be helpful to include results on established benchmarks to see whether the gains extend beyond the proposed setting.
  • On page 4, the paper states that 'C', 'P', and the overall score are reported separately, but I could only find the overall scores. Please include the separate results, especially 'P', so that improvements in preference adherence can be assessed directly.
  • The SFT trajectory correction stage uses deterministic task and preference checks. Why are these checks not also used to provide rewards during GRPO? Have the authors compared these two reward sources?
Reviewer_HTQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sleep-Time Training (STT): a personal agent collects trajectories during use and updates its weights while the user is idle. An auxiliary model (GPT-5.6-Luna) first rewrites sampled trajectories into corrected demonstrations for SFT. For RL, it repeatedly replaces the earliest defective step and lets the agent continue; GRPO then contrasts original and repaired trajectories under a binary judge reward. STT-Bench has one fictional persona, 20 base task families with four daily instances, 20 held-out generalization families, and four levels of preference disclosure. After three days on Qwen3.5-4B, STT scores 40.8 overall vs. 21.8 for OpenClaw-RL, including 40.3 vs. 23.5 on held-out Day-4 instances and 26.4 vs. 12.8 on generalization families. Gains are smaller on 9B and 30B.

Critical strengths

S1. A timely problem with a clean, reusable protocol. Repeated-session use becomes a concrete objective (Eq. 4) and a multi-day protocol in which STT carries knowledge forward only through its weights (L262-L263), with held-out Day-4 instances and paired generalization families (L727-L733). The benchmark is carefully built: audit-log grading (L188-L190), reference policies at C = E = P = 1 on all 100 instances (L760-L762), and a null-trajectory check (L207-L209).

S2. A careful repair loop. The auxiliary model never writes tool results; every action is re-executed in a reset environment (L272-L273), and a replacement step may only use information available before the defect (L295-L296).

S3. Large gains on held-out work at 4B. STT beats the strongest baseline by 16.8 points on Day-4 Base and 13.6 on generalization families (Table 3), and the two stages are complementary (Table 4: 24.3 and 23.7 alone vs. 40.8).

Critical weaknesses

W1. The SFT stage uses checks that a deployed agent and the baselines do not have. Demonstrations are revised until "deterministic task and preference checks pass", with failed checks fed back (L259-L261, L278-L281, L797-L799). App. B places deterministic checks inside each task's evaluator (L755-L758); L786-L788 says grader feedback never reaches the repairer, which fits only if "repairer" means the RL repairer (L261, L803). If these are the grader's checks, the Seen columns of Table 3 score the very instances whose checks shaped the training data, and Day-4 Base instances share each family's preferences (L730). As described, the baselines learn from public-task success or judge signals (L772-L781), and nothing says they get these checks. Without SFT, STT is close to them (23.7 in Table 4 vs. 21.8 in Table 3). So I cannot tell how much of the gain comes from deployment experience rather than evaluator feedback.

W2. Key reference points are missing. The base model is a listed baseline (L351-L352, L771), and its scores exist (STT before Day-1 SFT, L782), but they appear nowhere. In Fig. 2 (Day-4 Base+Gen), both memory baselines end below their first points (about 18 to 16.8; about 19 to 16.0), and STT after Day-1 SFT (about 15) sits below both; if those first points are near the base model (L783), the memory baselines lose ground. C and P are promised separately (L199-L200) but never reported. This hurts Table 5 most: the baselines barely move as cues are removed (ReasoningBank even rises, 20.9 to 23.5), so STT's nearly flat curve (40.8 to 39.3) does not by itself show learned preferences. With a single persona (L733), the claim may also need narrowing to this benchmark's conventions.

W3. The new RL component is not isolated. L337-L338 promises "teacher-token and repair ablations in Section 5.3", but Sec. 5.3 has only Tables 4-6. Adding RL after SFT lifts the score from 24.3 to 40.8 (Table 4), yet we cannot tell whether iterative repair causes this, or whether plain GRPO after SFT would match it once successes are no longer rare. The share of RL tokens written by the auxiliary model (L334) is also unknown; if large, RL is partly advantage-weighted imitation of the teacher.

Questions for authors

Q1. Which checks gate SFT acceptance (L797-L799): the grader.py checks or a separate set, and all three preferences or only the one in that day's L1 prompt?

Q2. Please add the base model and report C, E, and P separately for all methods on the Day-4 splits at L0-L3.

Q3. How does plain GRPO after SFT, with the same rollouts and steps, compare with iterative-repair RL? What share of tokens in repaired trajectories is teacher-written?

Reviewer_KA9HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sleep-Time Training (STT) converts deployment interactions into parameter updates during user-idle periods. It first uses an auxiliary model to correct trajectories, validates them through environment replay, and performs supervised fine-tuning. It then replaces the earliest defective action in an original rollout, lets the current policy continue, and trains on groups containing original and repaired trajectories with a GRPO-style objective. The authors introduce STT-Bench, comprising 20 base and 20 generalization task families across four simulated days with varying preference disclosure.

Critical strengths
  1. Corrected trajectories obtain observations through actual tool execution and replay rather than auxiliary-model fabrication. After local repair, the policy generates the continuation, exposing training to states arising from its own behavior.
  2. Evaluation covers three models, multiple days, and task families excluded from learning. Stage ablations support combining SFT and RL, while using the final daily checkpoint reduces opportunities for test-based model selection.
Critical weaknesses
  1. Preference supervision during SFT needs clearer separation from daytime experience. Section 4.2 returns failed deterministic checks to the auxiliary model as rule feedback. Appendix C further requires task and preference checks to pass, allowing up to 16 repair rounds. If these checks involve preferences absent from the current L1 prompt, corrected demonstrations contain additional preference supervision. Appendix C also states that training-time scorers and repairers do not receive grader feedback, requiring clarification of the SFT exception. This affects what is learned from public interactions versus information supplied by the repair procedure.
  2. Repair-based RL is not isolated from direct learning of teacher actions. Equation (7) trains on all retained assistant tokens, including auxiliary-written corrections. The authors appropriately acknowledge mixed-data GRPO-style training, but the teacher-token and repair ablations referenced on page 7 do not appear in Section 5.3 or the appendix. Table 4 only removes entire training stages. The gains may therefore combine continued teacher supervision, successful-trajectory selection, and learning from policy-generated continuations, without establishing their separate contributions.
  3. The reported metrics do not independently establish preference internalization. Section 3 promises separate completion CC, preference PP, and overall results, but the main tables report only C×E×PC\times E\times P. Improvements could arise from execution correctness or step efficiency without demonstrating better preference adherence. Appendix B also explicitly discloses each base family’s three preferences across Days 1–3, so final L3 evaluation primarily tests retention of previously stated preferences rather than discovering them entirely from implicit feedback. One fictional persona and three update cycles further limit conclusions about long-term personalization.
  4. Supervision and interaction budgets are not adequately aligned across methods. Beyond initial sampling, STT uses full demonstration rewriting, repeated replay, iterative repair, and auxiliary scoring. OpenClaw-RL and memory baselines do not receive the same data-construction procedure. Table 6 reports SFT and RL GPU time without auxiliary-model calls, tokens, or replay counts. The results support the complete pipeline but do not establish its advantage under equal budgets or isolate the contribution of additional corrected experience. Moving computation to idle periods does not itself reduce those resource requirements.
  5. The paradigm-level novelty needs a narrower distinction from discussed work. Section 2 explicitly describes MetaClaw as updating policy parameters during user-inactive periods, while Table 1 presents idle-time parameter learning and persistent gains as a unique STT combination. The more clearly supported contribution is combining validated-demonstration SFT with group-relative learning from locally repaired trajectories. Without a direct MetaClaw comparison, the incremental benefit over this close approach remains unclear.
Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.