acceptodds
Under review as a conference paper at ICLR 2027

VerMem: Verifier-Guided Command-Level Credit Assignment for Unified Memory Management

Abstract

Long-horizon language model agents must preserve useful knowledge, manage bounded context, and recover earlier task evidence. In existing memory policies that broadcast trajectory-level advantages, the shared signal does not directly assess the semantic quality of individual commands. We introduce VerMem, a unified memory policy with verifier-guided, command-level credit assignment. The local branch evaluates executable memory transitions and normalizes their semantic scores across commands of the same operation type within an update. The global branch normalizes composite feedback across candidate trajectories for the same task. Combining these advantages with direct constraint costs provides differentiated credit while retaining the task-level objective. The policy optimizes complete commands with equal command-level weight and coordinates versioned long-term memory, active context, and same-task history. Supervised initialization and staged reinforcement learning develop these capabilities. Both semantic verifiers provide training feedback and are disabled during evaluation. Experiments on question answering and interactive tasks show improved overall task performance, stronger results from the combined feedback branches, and better performance under constrained online-token budgets.

Primary area:
reinforcement learning
Keywords:
Large Language Model Agents, Agentic Memory, Memory Management, Hierarchical Credit Assignment, Verifier-Guided Reinforcement Learning, Long-Horizon Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mi7BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes VerMem to address challenges faced by long-horizon LLM agents, such as heterogeneous memory coordination, historical context recovery, and multi-granularity credit assignment. The authors claim that the method achieves unified memory control with explicit history recovery, verifier-guided hierarchical credit assignment over different comparison sets, and full-command policy learning with uniform command-level weights. Single-run results on a limited evaluation suite suggest improvements over conventional methods.

Critical strengths
  1. The flow diagrams are well drawn (e.g., Figures 1 and 2, and Figures A.1 and A.2). Their minimalist composition makes them relatively easy to follow.
Critical weaknesses

1. The paper reads very poorly. Specifically:

  1. First, the algorithm and experiment sections repeatedly point to the appendix, and the main text alone does not let readers clearly grasp the whole algorithm. I would suggest presenting the entire algorithm compactly in pseudocode with a detailed natural-language explanation, while moving to the appendix the parts that do not affect the overall exposition (e.g., the rationale for choosing specific parameters).
  2. Second, the placement of figures and tables is haphazard: Figures 3–6 are all crowded onto page 8 even though Figure 3 is first referenced on page 6; small tables such as Tables 2, 3, and 4 could be placed on the same row or otherwise worked into the running text.
  3. Third, when presenting experiments, the paper should first explain what each symbol denotes, then report the results, and only then analyze them. The current experimental section is fragmented and in places reports results before giving definitions, which confuses the reader.

2. The paper is hard to reproduce.

In the algorithm section, the authors genuinely try to show readers the full VerMem pipeline through detailed description. But the pipeline involves many computation steps, each with several parameters, and there is insufficient evidence to support all of these design choices and settings; a reader therefore cannot determine from the paper alone whether the method is reproducible.

Questions for authors
  1. Lines 162–165 mention token-count definitions such as 8192 tokens. What is their significance, and how were these parameters chosen? Similarly confusing concrete values include {0, 1, 2, 3, 4} at line 174.
  2. The experimental results appear to come from single runs. How is the robustness of this work assessed?
  3. I could not understand how the multiple experiments support the paper's key claim — namely that the results show VerMem achieves effective memory control, history recovery, credit assignment, and policy learning relative to prior work, and delivers better performance. Could you explain whether the experiments strongly support this claim, and how each of them does so?
1 comment

The authors did not respond early in the discussion period. In my assessment, the work still has substantial room for improvement, and I encourage the authors to continue developing it.

Reviewer_R7ZSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to train memory management during long interactions: task success does not establish that every memory command helped, while failure need not invalidate an earlier justified write. VerMem separates versioned long-term storage with provenance, the active context supplied to a task solver, and ordered history from the same task. Its policy uses Add, Update, and Delete to maintain stored entries, Retrieve, Filter, and Summarize to manage context, and SelectEpisode to restore historical fragments that have left active context, alongside a trainable null action. After a memory command executes, the task solver acts using the updated context and the environment produces subsequent observations.

Training combines local and global feedback. A frozen local verifier scores relevance, grounding, local progress, and information fidelity using currently visible evidence and the realized memory transition, without future observations or the final answer. Local advantages are normalized across commands of the same operation type within an update. The global reward averages task correctness, evidence quality, terminal-memory consistency, and efficiency. Supporting-fact recall and resource costs are computed programmatically; the global verifier supplies evidence coherence and memory consistency. Global advantages compare eight candidate trajectories for the same task. Both advantages combine with command constraint costs in a PPO-clipped objective over complete commands. Token losses are averaged within each command and then across commands, giving commands equal weight regardless of length.

Supervised initialization is followed by stages for long-term maintenance, context control, and joint coordination, all trained on HotpotQA. Experiments use Qwen2.5-7B and Qwen3-4B on held-out HotpotQA questions and ALFWorld, SciWorld, PDDL, and BabyAI, with no further policy training on the latter four. Storage and history reset at each evaluation episode, so the study evaluates within-task memory control and transfer across environments. Both training verifiers are disabled during evaluation. Experiments compare feedback branches, cumulative tool configurations, and online-token budgets.

Critical strengths
  1. The local verifier judges currently available evidence and a realized memory transition, while the global branch assesses the utility of the completed strategy. This distinguishes whether a command was justified from whether the task ultimately succeeded. The appendices provide operation-specific criteria, scoring prompts, and information boundaries, and specify that both training verifiers are disabled during evaluation. The dependence on labels and the source of additional training costs are reasonably clear.
  2. In Table 1, noVerify shares tools, supervised initialization, curriculum, and constraints with the full method. The arithmetic mean of the five displayed metrics improves by 6.25 and 6.21 points for the two backbones, with gains in nine of ten task comparisons. In Table 3, combining local and global branches outperforms either alone on all three reported tasks. Table 2 also provides a separate budget-controlled comparison: at a 2,500-token budget, macro-average success across three interactive environments is 44.30%, versus 39.00% for AgeMem. These results support the practical value of the complete feedback configuration and transfer from HotpotQA training to interactive tasks.
Critical weaknesses
  1. Moving from noVerify to the full method in Table 3 changes reward allocation while also adding local semantic scores and rewards for supporting evidence, terminal-memory consistency, and efficiency. The branch ablation supports these feedback combinations, but there is no control using the same scorers and reward components with local scores aggregated and broadcast across the trajectory. The gains therefore do not distinguish richer reward information from per-command allocation and operation-specific normalization. The equal-command weighting emphasized in Eq. (9) is also not compared with token weighting. These are explicit contributions, yet the experiments do not establish how much each adds.
  2. Fine-Mem, published at ACL in July 2026, is neither cited nor discussed. It provides local feedback through chunk-specific auxiliary QA and redistributes global rewards by tracing retrieved evidence back to relevant memory operations. It therefore already addresses fine-grained memory training and credit assignment. VerMem differs by directly judging the semantics of realized command transitions and computing relative advantages within operation types. However, omitting this work leaves unclear what additional problem these differences solve. The contribution should be positioned around direct command verification and its comparison groups, with evidence for their value.
  3. Figure 5 changes both available tools and training, without removing SelectEpisode while keeping the remaining configuration fixed. It therefore does not establish how much explicit history recovery contributes or whether Retrieve and Summarize already suffice. Page 9 acknowledges this limitation, but history recovery remains a principal contribution. Appendix F.2 does provide scoring dimensions, operation-specific criteria, and a local-verifier prompt; what is missing is independent validation on these commands, such as agreement with human judgments or controlled errors. The results support the complete system more directly than they explain the separate roles of history recovery and semantic command feedback.
Questions for authors
  1. With scorers, reward components, and training budget fixed, what happens if local feedback is aggregated into a trajectory-level reward and broadcast? Beyond that control, how much do operation-specific normalization and equal-command weighting add relative to alternative grouping and token weighting?
  2. With the remaining configuration fixed, how does removing SelectEpisode affect tasks requiring recovery of earlier evidence? For commands that pass execution checks but contain partial omissions or irrelevant content, do local-verifier scores agree with independent judgments?
  3. Please provide the actual number of evaluated episodes per environment, individual results for all three seeds, and success counts and aggregation for the success-rate tasks. Seeds and variability bands for budget curves are already reported, but they do not suffice to independently check the main-table means and their uncertainty.
Reviewer_6gs3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VerMem, which trains one memory policy that manages a versioned long-term memory, a bounded active context, and the current task’s history through seven atomic tools plus a null action. The new tool, SelectEpisode, restores earlier evidence from the same task’s history, separately from retrieving stored facts from long-term memory. The main contribution is how credit is assigned. A frozen DeepSeek-V3.2 verifier scores each executable command and the scores are standardized against other commands of the same operation type, while a composite trajectory reward (task score, evidence quality, terminal-memory consistency, efficiency) is standardized across eight rollouts of the same task. The two advantages are summed with a constraint cost and shared by all tokens of a command. The policy is initialized with SFT from a DeepSeek-V3.2 teacher and then trained with a three-phase RL curriculum on HotpotQA only, with verifiers used solely during training. On Qwen2.5-7B-Instruct and Qwen3-4B-Instruct, VerMem improves the five-benchmark Average by about 6 points over a matched task-outcome-only control and over cited AgeMem numbers, transfers zero-shot to ALFWorld, SciWorld, PDDL and BabyAI, and reaches the same success rate as AgeMem with about 20% fewer online tokens. Ablations show that local and global credit each help and combine best.

Critical strengths
  1. The internal ablation is clean. noVerify, +Local, +Global and +Local+Global share everything except the credit signal, both branches help on their own, and the combination wins on all three tasks.
  2. The information boundaries are thoughtfully designed. I like that the local verifier never sees the final answer or later states, so the per-command credit is causal rather than hindsight leakage.
Critical weaknesses
  1. Long-term memory is reset after every episode, so the “long-term” store is really within-episode. The motivating story about preserving knowledge across tasks is kind of disconnected to what was really proposed.
  2. There is no analysis of learned behavior. I would like to see when and how often SelectEpisode fires on the interactive benchmarks.
Questions for authors
  1. Were the baseline curves in Figure 4 produced by rerunning released code or checkpoints?
  2. How many DeepSeek-V3.2 verifier calls does one training run make, and roughly what does that cost relative to training noVerify?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.