- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies credit assignment for external-memory-based self-evolving agents. It identifies memory credit leakage, where co-retrieved memories share a trajectory-level return and irrelevant or harmful memories can therefore receive spurious positive credit. CMPO treats memory exposure as a stochastic Bernoulli policy and uses a propensity-corrected single-rollout estimator to update per-memory retrieval residuals. The paper proves unbiasedness for a gate-local inclusion–exclusion contrast and evaluates the method extensively on LifelongAgentBench, ALFWorld, and AppWorld with both white-box and black-box backbones.
Critical strengths
The paper identifies an important and practically relevant credit-assignment failure mode in online memory systems, clearly distinguishing the return observed when a memory is retrieved from the incremental value caused by including that memory. CMPO provides a practical black-box solution that requires no access to model parameters, gradients, or action log-probabilities, and avoids an additional counterfactual rollout for every memory decision. The empirical evaluation is unusually comprehensive, including multiple domains and backbones, a controlled credit-recovery study, online adaptation, component ablations, sensitivity analysis, paired uncertainty estimates, and held-out generalization.
Critical weaknesses
The theoretical credit is context-dependent, but the learned feedback residual is state-independent. Proposition 2 defines credit conditionally on the pre-gate history , so the same memory may be helpful in one context and harmful in another. CMPO does have a state-dependent semantic score , but the feedback learned from interaction outcomes is accumulated into a single persistent scalar residual for each memory and reused across states. Such a global learned correction cannot in general encode contextual credit whose sign changes, e.g., but , unless the fixed semantic relevance term happens to separate those contexts appropriately. The current controlled credit-recovery experiment does not test this case: for each task instance it constructs one helpful, one irrelevant, and one harmful memory, removes the token budget, and uses independent Bernoulli(0.5) gates. I would therefore like to see a more non-ideal credit-assignment experiment that explicitly studies contextual sign flips, memory interactions, and/or budget-induced competition, together with a detailed analysis of both successful and failure cases. This is the main issue preventing a clear-accept recommendation. If the authors can convincingly address it, including with a well-designed non-ideal credit-assignment analysis, I would consider raising my rating.
A minor presentation issue is that claims such as “substantial and consistent” downstream gains should be softened or qualified, since several paired confidence intervals still include zero.
Questions for authors
-
How does CMPO handle the same memory being helpful in one state but harmful in another, given that is a single state-independent scalar?
-
Could the authors add a more non-ideal controlled credit-assignment experiment, e.g. contextual sign flips, memory interactions, or budget-induced competition, and analyze both where CMPO works and where it fails?
-
Eq. (12) applies the estimator of directly to the logit residual , omitting the sigmoid factor . Is this best interpreted as a preconditioned or natural-gradient-like update?
-
Do the authors plan to release the implementation and evaluation code needed to reproduce the experiments?
5 comments
Thank you for the additional experiments. I appreciate the context-level analysis, which helps clarify the limitations of the global residual.
I have two remaining questions:
-
Context-specific credit estimation. Can the authors show whether CMPO can accurately estimate the memory's contribution separately in contexts A and B? In particular, I would like to see whether the estimated credits correctly capture the opposite signs of the memory's contribution in these two contexts, compared with the corresponding reference credits.
-
Global vs. context-conditioned residuals. Can the authors provide a lightweight comparison between the current global residual and a context-conditioned residual? I would like to see whether a context-conditioned variant can better distinguish the two contexts, increasing the exposure of helpful memories in A while suppressing them in B, and whether this leads to better task performance in both contexts.
These results would help clarify whether the remaining limitation mainly comes from the global residual parameterization or whether there are also limitations in the credit estimation mechanism itself.
We thank the reviewer for the follow-up questions and address them in turn.
1. Context-specific credit estimation.
To test whether CMPO credits capture opposite signs under rapid switching, we compare them with contributions measured by additional paired rollouts under medium and high pressure (GPT-5.6-Luna, 30 LAB DB and 30 ALFWorld tasks, three paired seeds). For each pressure, task, context, and seed, we average the matured target-memory credits, assigning each to the context in which its gate was sampled. For the reference, we uniformly sample four saved states, two from each half of the gate history in that context, and run five inclusion/exclusion pairs from each with the original budget, task sequence, and continuation updates (, ). Each half is weighted by its gate count to match the history distribution of the credits. MAE is the absolute difference between estimated and reference task-level averages, averaged over tasks. Entries are mean ± sample SD over three seeds.
| Dataset | Pressure | Context | Estimated | Reference | MAE |
|---|---|---|---|---|---|
| LAB DB | Medium | A | +0.28±0.09 | +0.20±0.07 | 0.15±0.07 |
| LAB DB | Medium | B | −0.15±0.17 | −0.18±0.05 | 0.11±0.08 |
| LAB DB | High | A | +0.24±0.15 | +0.21±0.11 | 0.09±0.05 |
| LAB DB | High | B | −0.27±0.20 | −0.18±0.09 | 0.14±0.13 |
| ALFWorld | Medium | A | +0.23±0.05 | +0.17±0.04 | 0.16±0.02 |
| ALFWorld | Medium | B | −0.26±0.08 | −0.23±0.04 | 0.08±0.06 |
| ALFWorld | High | A | +0.19±0.12 | +0.16±0.08 | 0.17±0.08 |
| ALFWorld | High | B | −0.20±0.03 | −0.22±0.06 | 0.12±0.04 |
CMPO recovers the opposite signs in both contexts. The mean credit matches the sign of the reference in all eight conditions and differs from it by only 0.02–0.09 return units under both pressures. The task-level MAE (0.08–0.17) is larger because each estimate averages noisy single-rollout credits and each reference uses only 20 pairs. This noise is why CMPO accumulates credits in a residual rather than acting on single estimates.
2. Global versus context-conditioned residuals.
To test whether this signal can be exploited, we compare global CMPO under high pressure with the fixed-zero control and a known-context variant. The variant replaces with two zero-initialized residuals, and . Before each task, the task generator records an index from the stated objective: distinct buyers (1) or orders (2) in LAB DB, and one apple (1) or two apples (2) in ALFWorld. Every memory uses in Eq. (7), and each gate record caches , so its delayed credit updates only that residual through Eq. (12). The index identifies the task requirement, not whether a memory helps or hurts. No contribution labels enter training, and all other settings are unchanged. We report exposure and success separately for interactions 1–100 and 101–200 as mean ± sample SD over three paired seeds.
| Dataset | Residual | 1–100 Exposure A/B | 1–100 Success A/B | 101–200 Exposure A/B | 101–200 Success A/B |
|---|---|---|---|---|---|
| LAB DB | Fixed-zero | 6.7±0.2/5.5±0.1 | 41.1±4.1/33.0±3.1 | 6.9±0.2/5.3±0.4 | 42.9±4.3/32.2±2.2 |
| LAB DB | Global | 13.6±2.5/11.5±2.0 | 43.2±4.8/27.3±5.0 | 19.2±2.6/18.5±2.1 | 45.6±4.9/23.5±5.9 |
| LAB DB | Conditioned | 9.2±1.1/5.7±0.4 | 42.4±2.3/26.7±3.4 | 19.7±1.8/2.6±0.1 | 43.0±2.6/34.7±3.6 |
| ALFWorld | Fixed-zero | 7.1±0.3/5.9±0.3 | 77.2±2.6/52.4±0.7 | 7.1±0.2/6.1±0.2 | 76.8±2.6/53.2±1.5 |
| ALFWorld | Global | 1.6±0.8/2.2±0.2 | 76.2±2.9/55.9±3.0 | 0.8±0.3/0.8±0.4 | 74.6±3.2/58.1±2.7 |
| ALFWorld | Conditioned | 7.8±1.0/3.4±0.4 | 77.3±1.6/53.6±2.1 | 13.3±1.4/1.8±0.6 | 79.1±0.6/54.3±1.0 |
With the same credits, separate residuals split exposure as intended, but slower learning offsets their success gains. By interactions 101–200, the conditioned variant exposes the memory far more in A than in B, with gaps of 17.1 and 11.5 points on LAB DB and ALFWorld. The global residual instead follows the net sign of the memory: it raises exposure in both contexts on LAB DB, hurting B, and suppresses it in both on ALFWorld, hurting A. Conditioning recovers success in exactly these two contexts but loses some in the other two, because each residual receives only about half of the feedback. Over the full run, the two variants therefore stay within 1.8 points of each other on both benchmarks. The limit under rapid sign flips thus lies in what a shared scalar can represent, not in the credits, which are correct in sign in both contexts. Conditioning addresses this limit at the cost of variance, because splitting unbiased but noisy credits (Proposition 3) across contexts slows learning. CMPO therefore pools all feedback into one residual per memory by default. Since the estimator does not depend on the parameterization (Proposition 2), conditioning remains an option for long deployments with a task-type signal.
Dear Reviewer VtZS,
Thank you again for your constructive review. You noted that tests under non-ideal conditions would be the most important addition, so we highlight the new results:
Do these tests and the failure analysis address your main concern? We are happy to report any additional analysis before the discussion period closes.
Best regards, The Authors