acceptodds
Under review as a conference paper at ICLR 2027

Vivid Memories: Fine-Grained Trajectory Memory for On-Policy Agent Self-Distillation

Abstract

Memory is central to self-improving language agents, but most agent memory systems compress experience into skills, procedures, or reflections. This is useful for inference-time prompting, but it discards the step-level state–action evidence needed for training-time credit assignment. We introduce VIVID, an on-policy self-improvement method that treats memory as a training-time supervision bank rather than an inference-time prompt aid. VIVID stores both successful and failed rollouts as raw turn sequences, annotates each turn with an offline outcome label and a continuous value, retrieves a balanced mixture of state-matched successful and failed precedents during training rollouts, and uses the annotations for reward-consistent turn-level advantage shaping of GRPO: the direction of every update is inherited from the verified environment reward, and the judge modulates only its magnitude. Evaluation is retrieval-free: the trained policy uses no memory, retrieval, or judge. On AppWorld, ALFWorld, and Sokoban with Qwen3-1.7B, Qwen3-4B, and Llama-3.1-8B backbones, VIVID is best on every benchmark, metric, and backbone, improving Qwen3-4B average accuracy from 52.5 (GRPO) and 64.1 (Skill-SD) to 72.0 (3 seeds, std 1.0). Controlled analyses locate the gain in the content of per-turn labels rather than in reweighting machinery or external-LLM spend: uninformative-label controls collapse to the GRPO floor, judge-free retrieval already matches Skill-SD with zero external LLM, budget-matched baselines do not close the gap, a future-blind (causal) judge retains most of the improvement, and letting the judge set the update sign is measurably worse than magnitude-only shaping.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, agent memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xRRLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VIVID, a training-time trajectory-memory framework for improving multi-turn language agents. Instead of compressing experience into skills or reflections, VIVID stores raw successful and failed trajectories annotated by an offline LLM judge with turn-level outcome labels, continuous QQ scores, and reflections. During training, the policy retrieves a balanced set of state-matched precedents to condition each rollout turn, while nonnegative, annotation-dependent weights modulate the trajectory-level GRPO advantage for policy updates. Evaluation uses the trained policy alone, without retrieval or a judge. Across AppWorld, ALFWorld, and Sokoban with three model backbones, VIVID reports consistent improvements over GRPO and Skill-SD, supported by several component, judge, and budget analyses.

Critical strengths

The paper addresses an important problem in multi-turn agent learning: how to preserve useful local state-action evidence from successful and failed experience rather than reducing all experience to compact skills. The training-time use of memory, followed by retrieval-free evaluation, is a meaningful design choice. The experiments cover three environments and three model backbones, and the authors provide more analysis than a standard end-to-end comparison, including retrieval-versus-weighting ablations, success/failure memory composition, raw-versus-compressed memory, judge-label controls, and external-supervision budget comparisons. These results offer evidence that the overall training pipeline is empirically effective.

Critical weaknesses

The manually specified advantage-weighting rule lacks a clear rationale and targeted validation. Equation (6) reweights each trajectory-level advantage with wi,t=ω(oi,t)∣Qi,t∣(1+αfi,t)w_{i,t}=\omega(o_{i,t})|Q_{i,t}|(1+\alpha f_{i,t}). The five outcome coefficients are fixed at 1.0,1.5,1.3,0.5,1.0, 1.5, 1.3, 0.5, and 0.30.3. The paper gives an intuitive reason to emphasize errors and recoveries while downweighting neutral or redundant turns, but it does not explain how these precise numerical values and relative gaps were determined. Appendix C.5 reports a sensitivity curve for an "outcome-weight scale," without specifying exactly how the scale is applied; this curve alone does not establish the contribution of each individual class coefficient. The authors should clarify whether this sweep includes a uniform-category baseline (ω(o)=1\omega(o)=1 for all outcome classes) and test one-at-a-time variations of the coefficients while keeping the others fixed. This matters because the heuristic can amplify potentially undesirable updates: in a positive-advantage trajectory, a turn classified as CRITICAL_ERROR with Q=−1Q=-1 receives a weight between 1.5 and 1.8, despite the judge explicitly identifying the action as harmful. The authors acknowledge this case and find that their signed-QQ variant performs worse, but this comparison does not by itself justify interpreting ∣Q∣|Q| as confidence when the judge prompt defines signed QQ as action quality, or explain the choice of these five numerical coefficients. The empirical gain of the complete method alone is not sufficient to validate these consequential design choices.

Same-task historical retrieval is a significant unresolved confound in attributing the reported gains. VIVID prevents retrieval from the current trajectory and from sibling rollouts in the same iteration, but does not exclude older trajectories of the current task; the per-source-task cap still permits up to two such precedents in a retrieval block. Because training tasks are revisited, the retrieved context can contain previously successful actions for the very task currently being solved. This may improve training rollouts through reuse of earlier local solutions, independently of whether the retrieved experience transfers across tasks. The reduced-repetition and out-of-domain analyses are helpful, but neither isolates this mechanism while keeping the rest of the training procedure fixed. The concern is consequential because the retrieval-only configuration with judge annotations, even without advantage weighting, already improves average accuracy from 52.5 (GRPO) to 68.8. Without a strict same-task-exclusion ablation, it remains unclear how much of the retrieval benefit depends on access to previous solutions for the current training task, rather than on precedents drawn from other tasks.

Questions for authors
  1. Why were the five outcome coefficients fixed at 1.0,1.5,1.3,0.5,1.0, 1.5, 1.3, 0.5, and 0.30.3? Could the authors clarify how the "outcome-weight scale" in Appendix C.5 is implemented, state whether it already includes a uniform-weight setting (ω(o)=1\omega(o)=1), and report one-at-a-time variations of the individual coefficients?
  2. How do the authors justify amplifying a CRITICAL_ERROR turn with Q=−1Q=-1 in a positive-advantage trajectory and treating ∣Q∣|Q| as confidence when the signed QQ is defined as an action-quality score?
  3. What fraction of retrieved precedents come from the current task ID, and how do full VIVID and the retrieval-only variant perform when all such same-task historical precedents are excluded under an otherwise identical training protocol?
  4. Could the authors provide the missing per-benchmark results in Table 4?
Reviewer_pvoqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VIVID, a method for RL post-training of LLMs for multi-turn tasks. VIVID uses an external LLM as a judge that reflects on and scores each turn made by the policy model during training rollouts. These scores are then used in two ways: (1) for turn-wise advantage shaping and (2) as in-context hints incorporated into the prompts for the next training rollouts.

The method is demonstrated using Qwen and Llama backbones and the AppWorld, ALFWorld, and Sokoban benchmarks. The paper provides performance comparisons with selected existing training methods and a series of ablation studies.

Critical strengths
  1. The authors demonstrate performance gains across a convincing matrix of 3 backbones, 3 benchmarks, and 5 competing methods. This strengthens the paper and provides strong justification for the proposed method.
  2. The authors show an large set of ablation studies and analyses of their method, investigating different aspects. I find the specific analysis of “Retrieval-only” and “Weighting-only” (presented in Table 4) a critical one, as the paper essentially combines two mechanisms in the training process, each of which demands its own analysis.
Critical weaknesses
  1. Choice of baseline: One of the two components of the proposed training method is turn-wise credit assignment. However, despite mentioning them in related work section, methods like ArCHer and GiGPO (I would also mention Turn-PPO [1] as a relevant one) weren’t listed under the compared methods, making it difficult to assess the contribution of the suggested turn-wise advantage shaping approach.
  2. The use of merely magnitude shaping. I acknowledge that the authors have shown empirically that using Q as-is shows worse performance, but it still seems to damage the theory behind their method. The authors emphasize their contribution in obtaining fine-grained, step-wise advantages. However, recovery attempts will be depressed in failed rollouts, and errors will be reinforced in successful rollouts. The method actually makes this problem a bit worse: steps that are classified as redundant get smaller magnitudes in successful rollouts, but critical errors are explicitly reinforced, a hierarchy that doesn’t make sense. If I have misread Eq. (6), I would welcome clarification in the rebuttal.

[1] Li, J., Zhou, P., Meng, R., Vadera, M. P., Li, L., & Li, Y. (2026, March). Turn-ppo: Turn-level advantage estimation with ppo for improved multi-turn rl in agentic llms. In Findings of the Association for Computational Linguistics: EACL 2026 (pp. 6227-6243).

Questions for authors
  1. Why are there missing cells in Table 4? Could the authors report the per-benchmark scores? I find these ablation studies important, and the results are incomplete as currently presented.
  2. The method relies on indirect distillation from an external model (GPT-4.1-mini). Can the authors clarify which of the compared methods also rely on learning from an external, stronger model and specify which external model was used for each baseline?
  3. In Equation 6, both ω(oi,t)\omega(o_{i,t}) and Qi,tQ_{i,t} are used, but they seem to overlap substantially (according to the Q instructions in the prompt). Could the authors clarify why both are needed?
  4. (Suggestion) Regarding the choice of magnitude shaping vs. sign shaping, I’m wondering what the authors think about adapting Q to the actual reward signal. For instance, setting Q = 0 for critical errors when the rollout receives a positive reward and Q = 1 when it receives a negative reward, and vice versa for positive labels.
  5. (Suggestion) All the judges tested seem to be stronger/larger than the policy model. I wonder whether the authors considered testing self-judging with their method, using the same model as both the judge and the policy model. I find it a much stronger setting if the model can improve itself without relying on other models.
Reviewer_nrJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIVID, a method for improving agent training through fine-grained trajectory memory. It stores both successful and failed trajectories, uses an external LLM to annotate each step, and retrieves similar past experiences to guide training rollouts. These annotations also adjust the magnitude of each step’s GRPO advantage while preserving its sign. At evaluation time, the trained policy operates without memory retrieval or a judge. Experiments on AppWorld, ALFWorld, and Sokoban show improvements over skill-augmented GRPO and Skill-SD.

Critical strengths
  1. The paper offers a valuable perspective on using fine-grained trajectory memory as a training supervision resource. Memory supports both rollout generation and policy updates, and the trained model achieves strong performance without memory retrieval or an external judge at evaluation time, demonstrating the potential of this approach.

  2. The step-level advantage weighting is thoughtfully designed. It adjusts update strength based on step categories, judge scores, and action divergence while preserving the advantage sign determined by environment rewards. This design balances local supervision with potential judge errors, and the ablation studies provide further support for this choice.

  3. The experimental evaluation covers multiple environments and model backbones. Component ablations, annotation quality checks, and matched external supervision budgets provide a systematic analysis of the sources of performance improvement.

Critical weaknesses
  1. Judge reliability and the role of external supervision need clarification. Human agreement tests primarily assess discrete labels, leaving the stability and calibration of continuous Q-values insufficiently validated. Interpreting ∣Q∣|Q| as confidence also requires justification, since action value and confidence in that assessment are distinct. Because the labels, scores, and explanations come from an external LLM, the paper should more clearly distinguish the contribution of learning from the agent’s own experience from that of external teacher supervision.

  2. Local credit assignment remains limited, and the signed-Q comparison is inconclusive. Non-negative weights only rescale trajectory-level advantages, so incorrect steps in positive-advantage trajectories may still be reinforced, while correct steps in negative-advantage trajectories may still be penalized. As described, the signed-Q variant multiplies the advantage by a signed Q-value, producing a positive update when both are negative. Its poorer performance therefore does not establish that using the judge’s local directional information is inherently ineffective.

  3. The context mismatch between training and evaluation needs further analysis. The model generates actions and receives updates with retrieved experiences in its context, but these experiences are removed at evaluation time. Although retrieval-free performance is strong, the paper offers limited explanation of how this transfer occurs. A systematic comparison of the same trained model with and without retrieval would help assess its remaining dependence on memory and the performance cost of removing it.

  4. Direct comparisons with fine-grained credit assignment methods are missing. The baselines primarily cover standard GRPO and skill-based approaches, without including methods discussed in the paper, such as GiGPO and ArCHer. Since step-level credit assignment is a central motivation, these comparisons would help establish whether VIVID offers benefits beyond existing turn-level advantage estimation or critic-based methods.

Questions for authors

Please refer to the weaknesses section. Meanwhile, here is a follow-up question I have:

  1. Since VIVID relies on an external LLM for step-level supervision, its characterization as self-distillation needs clarification. Have the authors tried using the policy model itself to generate these annotations, and how does its performance compare with external judging?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.