- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies temporal credit assignment in RL training of long-horizon memory agents. It argues that prior work treats credit assignment as a one-dimensional question of reward density, and proposes "Credit Assignment Geometry," which describes step-level credit along three dimensions: Quantity (Q), Heterogeneity/Coverage (H), and Utility (U). Building on this view, the authors propose Mem-C, a tree-based RL method whose branch selection combines a retrieval-coverage score with a utility proxy Û, defined as the mean pairwise difference in outcomes across a node’s child branches. Expansion budget is thus steered toward decisions whose continuations lead to more divergent outcomes.
Mem-C is validated on LoCoMo (59.41 F1, vs. 58.65 for Mem-T) and on three transfer benchmarks (HotpotQA, LongMemEval, NarrativeQA), where it outperforms all baselines. The paper also includes three analysis experiments, varying search budget, the coverage parameter, and the utility weight βU, which are presented as evidence for diminishing returns and heterogeneous credit utility.
Critical strengths
-
The paper reframes credit assignment in tree-based agentic RL as a budget allocation problem, pointing out that under the same rollout budget, where branches are placed matters as much as how dense the reward is.
-
The utility proxy reuses already-scored branch outcomes, so it adds no extra rollouts and can be plugged into existing tree-based RL pipelines with little effort.
Critical weaknesses
-
The paper is motivated by credit assignment over long-horizon memory operations, yet Appendix A.2 shows that RL only optimizes a retrieval policy over a pre-built memory store, with memory construction kept outside the RL loop. The construction and revision decisions emphasized in the motivation are therefore never trained.
-
The paper claims to address long-horizon credit assignment, but Fig. 8 shows a median inference trace length of only 2 steps, with two-step traces making up over 80% of records. The training configuration also caps retrieval at 6 turns (Table 6). This setting does not convincingly support the "long-horizon" claim.
-
The central question covers both how learning scales with credit and how credit should be allocated. Mem-C only addresses allocation under a fixed budget, and the scaling question receives no methodological answer.
-
The paper does not explain why the mean pairwise absolute difference of branch outcomes is used as the utility proxy rather than variance or another dispersion measure.
-
No justification is given for sampling a random anchor node at the start of each expansion round.
-
Figs. 1 and 2 repeatedly state that "high-entropy steps need more reward," implying an entropy-based method. The method described in the main text instead relies on branch-outcome dispersion and never involves entropy.
-
Fig. 2 includes a Retry Gate module that is not described anywhere in the method section. The paper does not state whether it is used in the main results.
Questions for authors
-
Mem-C improves over Mem-T by only 0.76 F1 on in-domain LoCoMo but by 12.82 F1 on out-of-domain HotpotQA. What explains this gap? Is the HotpotQA gain attributable to utility-aware allocation, or to other factors such as the SFT stage, the Retry Gate, or differences in evaluation settings?
-
Outcome dispersion across child branches also reflects sampling randomness (temperature 1.0) and reward noise, not only the decision at node v. How do the authors ensure that a high Û indicates a consequential decision rather than a noisy downstream subtree?
-
Since Û is zero for nodes with fewer than two scored branches, the selector appears biased toward already-expanded nodes. How does Mem-C avoid this exploitation bias and still discover new high-utility decisions?
11 comments
Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy
Question 2 & Weakness 4:
Outcome dispersion across child branches also reflects sampling randomness (temperature 1.0) and reward noise, not only the decision at node v. How do the authors ensure that a high Û indicates a consequential decision rather than a noisy downstream subtree?
The paper does not explain why the mean pairwise absolute difference of branch outcomes is used as the utility proxy rather than variance or another dispersion measure.
Authors’ Response:
-
1. Estimator definition and rationale.
- Our implementation first computes the mean observed return within each scored child branch:
- $ V_i(v)=\frac{1}{|\mathcal L_i^{\mathrm{scored}}|} \sum_{\ell\in\mathcal L_i^{\mathrm{scored}}}R_\ell. $
- For scored child branches, the utility estimate is defined as
- $ \hat U(v)=\frac{2}{B(B-1)} \sum_{i<j}|V_i(v)-V_j(v)|. $
- When fewer than two scored branches are available, the estimate is marked as unavailable and contributes zero to the selection score. It is updated as additional scored outcomes become available.
- We choose the mean pairwise absolute difference for three reasons:
- Interpretability: It directly measures the average observed return difference between two distinct child branches, providing a simple indicator of outcome sensitivity.
- Linear reward scale: Unlike variance, which uses squared differences, the estimator remains in the same units as the returns, making its combination with the coverage score through straightforward.
- Branch-count normalization: Averaging across branch pairs avoids the mechanical increase associated with summing pairwise differences as more branches become available.
- Variance is also a reasonable alternative. For exactly two branch values, population variance equals , so both estimators produce the same ranking. With more branches, their rankings may differ because variance places greater weight on extreme values. We do not claim that our estimator is inherently superior to variance or other dispersion measures.
- Our implementation first computes the mean observed return within each scored child branch:
-
2. Sampling randomness and interpretation of utility.
- We agree that outcome dispersion reflects not only differences between child decisions but also downstream sampling randomness (temperature 1.0) and reward noise.
- Our estimator partially mitigates individual-outcome fluctuations by averaging all available scored descendant returns within each child branch, rather than comparing individual rollouts directly.
- Since child branches share the same decision prefix, the resulting statistic measures differences in observed outcomes among alternative continuations from that prefix.
- However, averaging does not eliminate sampling noise or establish that the observed differences are caused by the branching decisions.
- In particular, finite-sample fluctuations and downstream stochasticity may still produce high utility estimates.
- Therefore, we interpret a high as evidence of observed outcome sensitivity, rather than a guaranteed indicator of causal decision importance or marginal learning benefit.
- Its role is to provide a heuristic signal for prioritizing further branching.
-
3. Empirical evidence and limitations.
-
The supplementary utility-weight sweep provides a diagnostic of how the estimator affects node selection:
-
Mean selected-node Mean logged training reward 0.0 0.0334 0.1557 0.5 0.0611 0.1258 1.0 0.0762 0.1363 1.5 0.0779 0.1317 2.0 0.0772 0.1247 -
These statistics cover the common first 29 updates and all three expansion rounds, including the initial round with zero utility estimates. Increasing the utility weight generally leads to the selection of nodes with higher observed utility, confirming that the estimator influences expansion decisions.
- We will revise the manuscript to clarify both the motivation for the mean pairwise absolute difference and its limitations, consistently describing as an observed outcome-sensitivity heuristic rather than a noise-free measure of causal decision importance.
-
-
Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy
Question 3:
Since Û is zero for nodes with fewer than two scored branches, the selector appears biased toward already-expanded nodes. How does Mem-C avoid this exploitation bias and still discover new high-utility decisions?
Authors’ Response:
Insufficient utility evidence does not exclude a node from selection. The implementation provides three mechanisms support exploration:
Random anchor.
- Each expansion round first samples uniformly from the full candidate set—the root and all nonterminal descendants—regardless of utility readiness.
- One of the slots therefore bypasses utility ranking: half the slots in Exp-C ().
Coverage-based selection.
- The remaining slots use . A node without a utility estimate can still compete through coverage, including retrieved-evidence novelty.
- Candidates are selected without replacement until the pool is exhausted.
Estimate refresh.
- Newly generated leaves are scored, and utilities are recomputed before the next round, allowing exploration to establish utility estimates for previously unassessed nodes.
Empirical check. In a frozen-policy audit with eight validation questions and three sampling seeds, utility-unready nodes continue receiving expansion slots:
-
Expansion round Selection slots assigned to utility-unready nodes 1 100.00% ± 0.00% 2 46.88% ± 8.27% 3 38.54% ± 3.61% -
Utility-unready denotes fewer than two scored child branches. Values are mean ± sample SD across three sampling seeds.
- These measurements demonstrate continued expansion of nodes whose utility cannot yet be estimated.
-
We sincerely appreciate your constructive feedback and positive assessment of the novelty, performance, and contributions of our work.
- We understand that your remaining concerns primarily focus on the need for additional experimental validation and clarification of specific details,
- rather than questioning the core novelty or effectiveness of our approach.
- To address these concerns, we have conducted supplementary experiments and provided more detailed explanations.
- We hope that these additional results and clarifications have adequately addressed your concerns.
- If so, we would be sincerely grateful if you could reconsider your overall score.
- We hope that these additional results and clarifications have adequately addressed your concerns.
- We understand that your remaining concerns primarily focus on the need for additional experimental validation and clarification of specific details,
Weakness 1:
The paper is motivated by credit assignment over long-horizon memory operations, yet Appendix A.2 shows that RL only optimizes a retrieval policy over a pre-built memory store, with memory construction kept outside the RL loop. The construction and revision decisions emphasized in the motivation are therefore never trained.
Authors’ Response:
-
Mem-C instantiates credit allocation over the retrieval policy of a memory agent.
-
Memory construction and revision prepare the store outside the RL loop; the optimized decisions are summarized below.
-
Operation Implementation Role during RL Memory construction Form memories from dialogue Memory preparation outside the RL loop Memory revision Update stored memory entries Memory preparation outside the RL loop Memory-source selection Choose facts, experiences, personas, summaries, or raw turns RL-optimized policy decision Query formulation Generate tool arguments and reformulate queries RL-optimized policy decision Stopping and answering Emit finishwith the final answerRL-optimized policy decision -
A fixed memory store still leaves consequential sequential choices: the selected source and query determine the evidence available to subsequent decisions.
- Mem-C uses observed continuation outcomes to prioritize branching within this retrieval sequence.
- Keeping the store fixed provides a common environment for these comparisons.
- Mem-C uses observed continuation outcomes to prioritize branching within this retrieval sequence.
-
We will specify this action space consistently in the introduction and figures, distinguishing memory preparation from the retrieval decisions optimized by RL.
-
Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy
Question 1 & Weakness 7:
Authors’ Response:
Thank you for raising the concerns.
We will clarify the role of the Retry Gate and isolate the contribution of utility-aware allocation to explain the substantial performance gain on HotpotQA.
1. Clarification of the Retry Gate.
2. Ablation of utility-aware allocation and the Retry Gate.
To isolate their respective contributions, we conduct a 2 × 2 controlled ablation on HotpotQA, independently varying the utility coefficient and the Retry Gate.
Specifically, we compare coverage-only allocation βU=0 with utility-aware allocation βU=1, under both enabled and disabled Retry Gate settings.
All four configurations use the same initial policy checkpoint, training protocol, memory store, test examples, and evaluation settings.
The results are presented in Table R3-1.
Table R3-1: Controlled Ablation of Utility-aware Allocation and Retry Gate on HotpotQA.
These results indicate that utility-aware credit allocation accounts for the dominant part of the observed HotpotQA improvement, whereas the Retry Gate provides a relatively small additional benefit.