- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MemAgent-Critic, a reinforcement learning framework for long-context memory agents that augments sparse terminal answer rewards with a learned trajectory-level “memory-trace critic.” The critic (1.5B) scores full sequences of natural-language memory states produced by a 7B recurrent memory agent and contributes a small auxiliary terminal reward; crucially, the critic is adapted online using within-question, group-normalized combined rewards (answer reward + critic score) as positive/negative labels, thereby reusing existing policy rollouts without extra annotations. On RULER-HotpotQA across 10 context lengths (7K–3.5M tokens), the approach achieves strong accuracy (93.75% at 7K and 80.47% at 3.5M) and improves the 10-length average over a no-critic baseline and a frozen-critic variant, with additional (though limited) gains on LongBench-QA, summarization, and code completion.
Critical strengths
Technical novelty and innovation Introduces a trajectory-level memory-trace critic specifically tailored to recurrent memory agents and uses within-question group-relative labels to adapt the critic online without extra supervision. The critic operates on complete memory traces (not token or step-level), aligning the reward signal with the agent’s recurrent state evolution and addressing an important long-horizon credit-assignment gap. Practical design choices (batch z-scoring of critic scores, small λ weighting, leveraging GRPO/DAPO infrastructure) make the method easy to integrate into existing RLVR pipelines. Experimental rigor and validation Evaluates across ten context lengths up to 3.5M tokens on RULER-HotpotQA, showing competitive performance and length generalization. Includes ablations among No-Critic, Frozen-Critic, and Adaptive-Critic variants and reports training dynamics over multiple stages. Provides some task transfer to LongBench-QA and non-QA tasks (summarization/code completion).
Critical weaknesses
Clarity or presentation issues
Details of critic input truncation/serialization are unclear (e.g., how multiple memory states fit within an 8K critic context; what is dropped or compressed). The effect size of λ after double normalization (batch z-scoring then group normalization on the sum) is difficult to intuit; a signal analysis would help.
Missing related work or comparisons
Does not empirically compare to PAPO-style outcome-anchored process rewards or to decoupled normalization (GDPO), despite being directly relevant to the multi-reward, group-normalized setting. Process-level credit assignment alternatives like TRACE are not compared; these could be complementary or superior in long-horizon settings. For discussion of credit granularity, the authors could also consider Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting offers a conceptual comparison with full-memory-trace scoring. The connection is methodological rather than a directly interchangeable memory-agent baseline, so a discussion of their differing roles would be more appropriate than requiring a new benchmark comparison.
Questions for authors
1.How is the critic input serialized and truncated when memory traces exceed its context window? Do you drop early memory states, compress, or sample? How sensitive are results to this choice? 2.Why are critic scores batch-normalized across all scored trajectories rather than per-question? Did you try per-question standardization or decoupled normalization of outcome and critic rewards (à la GDPO/PAPO), and what were the effects? 3.In groups with uniform outcome rewards (all-correct or all-wrong), the critic essentially labels itself. How often does this occur during later training, and what fraction of critic updates come from such groups? 4.Can you provide matched-compute comparisons for No-Critic, Frozen-Critic, and Adaptive-Critic, and report run-to-run variance or confidence intervals on RULER-HotpotQA?