acceptodds
Withdrawn from ICLR 2027

MemAgent-Critic: Adaptive Self-Supervised Critic Training for Long-Context Memory Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training long-context memory agents with reinforcement learning faces sparse feedback: a single verifiable reward arrives only after the entire document is processed. A learned memory-trace critic supplies additional trajectory-level guidance, but a fixed critic can become misaligned with the evolving policy. To adapt this guidance to current-policy trajectories, we propose **MemAgent-Critic**, which updates a 1.5B critic online alongside a 7B memory policy. After initialization with GPT-5.4- and policy-generated traces, the critic is updated using within-question comparisons of combined answer and critic rewards. These updates reuse existing policy rollouts without additional teacher annotations. On RULER-HotpotQA, the resulting 7B policy achieves **93.75%** accuracy at 7K tokens and **80.47%** at 3.5M tokens. Across ten context lengths, the adaptive variant obtains an average accuracy of **88.75%**, compared with 87.27% for the frozen-critic variant, although the variants use different training durations and the frozen critic performs better at the two longest lengths. Additional evaluations report gains on several document QA, summarization, and code-completion tasks.

Primary area:
reinforcement learning
Keywords:
long-context language models, GRPO, post train

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fyqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MemAgent-Critic, a reinforcement learning framework for long-context memory agents that augments sparse terminal answer rewards with a learned trajectory-level “memory-trace critic.” The critic (1.5B) scores full sequences of natural-language memory states produced by a 7B recurrent memory agent and contributes a small auxiliary terminal reward; crucially, the critic is adapted online using within-question, group-normalized combined rewards (answer reward + critic score) as positive/negative labels, thereby reusing existing policy rollouts without extra annotations. On RULER-HotpotQA across 10 context lengths (7K–3.5M tokens), the approach achieves strong accuracy (93.75% at 7K and 80.47% at 3.5M) and improves the 10-length average over a no-critic baseline and a frozen-critic variant, with additional (though limited) gains on LongBench-QA, summarization, and code completion.

Critical strengths

Technical novelty and innovation Introduces a trajectory-level memory-trace critic specifically tailored to recurrent memory agents and uses within-question group-relative labels to adapt the critic online without extra supervision. The critic operates on complete memory traces (not token or step-level), aligning the reward signal with the agent’s recurrent state evolution and addressing an important long-horizon credit-assignment gap. Practical design choices (batch z-scoring of critic scores, small λ weighting, leveraging GRPO/DAPO infrastructure) make the method easy to integrate into existing RLVR pipelines. Experimental rigor and validation Evaluates across ten context lengths up to 3.5M tokens on RULER-HotpotQA, showing competitive performance and length generalization. Includes ablations among No-Critic, Frozen-Critic, and Adaptive-Critic variants and reports training dynamics over multiple stages. Provides some task transfer to LongBench-QA and non-QA tasks (summarization/code completion).

Critical weaknesses

Clarity or presentation issues

Details of critic input truncation/serialization are unclear (e.g., how multiple memory states fit within an 8K critic context; what is dropped or compressed). The effect size of λ after double normalization (batch z-scoring then group normalization on the sum) is difficult to intuit; a signal analysis would help.

Missing related work or comparisons

Does not empirically compare to PAPO-style outcome-anchored process rewards or to decoupled normalization (GDPO), despite being directly relevant to the multi-reward, group-normalized setting. Process-level credit assignment alternatives like TRACE are not compared; these could be complementary or superior in long-horizon settings. For discussion of credit granularity, the authors could also consider Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting offers a conceptual comparison with full-memory-trace scoring. The connection is methodological rather than a directly interchangeable memory-agent baseline, so a discussion of their differing roles would be more appropriate than requiring a new benchmark comparison.

Questions for authors

1.How is the critic input serialized and truncated when memory traces exceed its context window? Do you drop early memory states, compress, or sample? How sensitive are results to this choice? 2.Why are critic scores batch-normalized across all scored trajectories rather than per-question? Did you try per-question standardization or decoupled normalization of outcome and critic rewards (à la GDPO/PAPO), and what were the effects? 3.In groups with uniform outcome rewards (all-correct or all-wrong), the critic essentially labels itself. How often does this occur during later training, and what fraction of critic updates come from such groups? 4.Can you provide matched-compute comparisons for No-Critic, Frozen-Critic, and Adaptive-Critic, and report run-to-run variance or confidence intervals on RULER-HotpotQA?

Reviewer_Jo3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper extends MemAgent with a learned critic for memory traces. The policy reads a long document in chunks, updates a fixed-length memory, and produces a final answer. The critic give the scores for full memory trace, and this score is added to the answer reward during reinforcement learning. The critic is also updated online using labels based on the relative rewards of current policy rollouts.

The method is evaluated on long-context QA, document QA, summarization, and code completion. The adaptive version reports an average accuracy of 88.75% across ten RULER-HotpotQA context lengths, compared with 87.27% for a frozen critic and 84.22% without a critic.

Critical strengths
  1. Strong results over very long contexts. The model performs well from 7K to 3.5M tokens and improves over the no-critic baseline at all tested lengths. This shows that a learned memory-trace reward can be useful for training long-context memory agents.

  2. Evaluation on several tasks. The paper includes QA, summarization, code completion, and different memory budgets. The gains on LongBench-QA, especially Qasper, suggest that the full system may transfer beyond its main training setting.

Critical weaknesses
  1. The novelty beyond MemAgent is limited, and the value of the main new component is not clearly shown. The paper keeps the main MemAgent workflow, and Figure 1 closely resembles Figure 2 of MemAgent. The main new component is the adaptive critic. However, the adaptive stage continues from the frozen-critic checkpoint and uses more training. Therefore, its higher average accuracy may come from additional policy training rather than critic adaptation. The figure caption should also clearly state whether Figure 1 is adapted from MemAgent.

  2. It is unclear whether the critic measures memory quality. The online labels partly depend on the critic's own scores, and there is no independent evaluation of these scores. As a result, the critic may simply reinforce its existing rankings rather than learn to identify better memory traces..

Questions for authors
  1. If both methods start from the same checkpoint and use the same training time and compute, does the adaptive critic still perform better than the frozen critic?

  2. Is there any independent evidence that the critic can correctly judge memory quality? For example, do its scores agree with human judgments or judgments from an independent strong model?

  3. When all rollouts for a question receive the same answer reward, the training labels mainly come from the critic's own scores. How often does this happen? Does the method still work if these rollouts are not used to train the critic?

Reviewer_ka7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MemAgent-Critic, an extension of recurrent long-context memory agents that combines verifiable final-answer rewards with a learned reward over complete memory traces. During subsequent training, a 1.5B critic is updated using binary labels derived from whether each trajectory’s combined answer-and-critic reward exceeds its within-question group mean. This reuses existing policy rollouts without further online teacher annotation. The 7B policy model achieves 93.75% accuracy at 7K tokens and 80.47% at 3.5M tokens on RULER-HotpotQA.

Critical strengths
  1. A practical auxiliary signal for a consequential training problem. Sparse final-answer rewards provide limited discrimination among memory trajectories with identical outcomes. Scoring the memory sequence offers an implementable way to distinguish such trajectories within retained rollout groups.

  2. Table 3 reports higher accuracy for both critic variants than for the no-critic continuation, while the additional evaluations extend beyond the primary QA setting. These results make learned memory-trace rewards worth investigating.

Critical weaknesses
  1. The million-token evaluation relies on RULER-HotpotQA, leaving it unclear whether the reported length generalization transfers to realistic long qa benchmarks, such as CorpusQA, that require finding and integrating evidence across large document collections.
  2. The adaptive variant continues training from the frozen-critic checkpoint, so its gains could reflect additional training rather than critic adaptation.
Questions for authors
  1. Can you compare frozen-critic and adaptive-critic continuations from the same policy and critic checkpoint, using identical training data and matched additional policy updates?
  2. What fraction of critic training examples comes from uniform-verifier-reward groups? What happens when these groups are excluded from critic updates, or when online labels use verifier outcomes alone in mixed-outcome groups?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.