- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VIVID, a training-time trajectory-memory framework for improving multi-turn language agents. Instead of compressing experience into skills or reflections, VIVID stores raw successful and failed trajectories annotated by an offline LLM judge with turn-level outcome labels, continuous scores, and reflections. During training, the policy retrieves a balanced set of state-matched precedents to condition each rollout turn, while nonnegative, annotation-dependent weights modulate the trajectory-level GRPO advantage for policy updates. Evaluation uses the trained policy alone, without retrieval or a judge. Across AppWorld, ALFWorld, and Sokoban with three model backbones, VIVID reports consistent improvements over GRPO and Skill-SD, supported by several component, judge, and budget analyses.
Critical strengths
The paper addresses an important problem in multi-turn agent learning: how to preserve useful local state-action evidence from successful and failed experience rather than reducing all experience to compact skills. The training-time use of memory, followed by retrieval-free evaluation, is a meaningful design choice. The experiments cover three environments and three model backbones, and the authors provide more analysis than a standard end-to-end comparison, including retrieval-versus-weighting ablations, success/failure memory composition, raw-versus-compressed memory, judge-label controls, and external-supervision budget comparisons. These results offer evidence that the overall training pipeline is empirically effective.
Critical weaknesses
The manually specified advantage-weighting rule lacks a clear rationale and targeted validation. Equation (6) reweights each trajectory-level advantage with . The five outcome coefficients are fixed at and . The paper gives an intuitive reason to emphasize errors and recoveries while downweighting neutral or redundant turns, but it does not explain how these precise numerical values and relative gaps were determined. Appendix C.5 reports a sensitivity curve for an "outcome-weight scale," without specifying exactly how the scale is applied; this curve alone does not establish the contribution of each individual class coefficient. The authors should clarify whether this sweep includes a uniform-category baseline ( for all outcome classes) and test one-at-a-time variations of the coefficients while keeping the others fixed. This matters because the heuristic can amplify potentially undesirable updates: in a positive-advantage trajectory, a turn classified as CRITICAL_ERROR with receives a weight between 1.5 and 1.8, despite the judge explicitly identifying the action as harmful. The authors acknowledge this case and find that their signed- variant performs worse, but this comparison does not by itself justify interpreting as confidence when the judge prompt defines signed as action quality, or explain the choice of these five numerical coefficients. The empirical gain of the complete method alone is not sufficient to validate these consequential design choices.
Same-task historical retrieval is a significant unresolved confound in attributing the reported gains. VIVID prevents retrieval from the current trajectory and from sibling rollouts in the same iteration, but does not exclude older trajectories of the current task; the per-source-task cap still permits up to two such precedents in a retrieval block. Because training tasks are revisited, the retrieved context can contain previously successful actions for the very task currently being solved. This may improve training rollouts through reuse of earlier local solutions, independently of whether the retrieved experience transfers across tasks. The reduced-repetition and out-of-domain analyses are helpful, but neither isolates this mechanism while keeping the rest of the training procedure fixed. The concern is consequential because the retrieval-only configuration with judge annotations, even without advantage weighting, already improves average accuracy from 52.5 (GRPO) to 68.8. Without a strict same-task-exclusion ablation, it remains unclear how much of the retrieval benefit depends on access to previous solutions for the current training task, rather than on precedents drawn from other tasks.
Questions for authors
- Why were the five outcome coefficients fixed at and ? Could the authors clarify how the "outcome-weight scale" in Appendix C.5 is implemented, state whether it already includes a uniform-weight setting (), and report one-at-a-time variations of the individual coefficients?
- How do the authors justify amplifying a CRITICAL_ERROR turn with in a positive-advantage trajectory and treating as confidence when the signed is defined as an action-quality score?
- What fraction of retrieved precedents come from the current task ID, and how do full VIVID and the retrieval-only variant perform when all such same-task historical precedents are excluded under an otherwise identical training protocol?
- Could the authors provide the missing per-benchmark results in Table 4?