- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper proposes Memory Gain Policy Optimization (MGPO), a reinforcement-learning approach for persistent memory optimization. It addresses the problem of attributing downstream utility to individual memory rewrites rather than rewarding information already present in memory. MGPO introduces a long-horizon counterfactual reward that measures the incremental contribution of each rewrite to current and future tasks, while preserving the underlying expected policy gradient. Experiments primarily focus on document-level information extraction, with additional evaluations of reward variants, cross-domain transfer, downstream readers, and long-context question answering.
Critical strengths
1. Problem Reformulation
- The distinction between inherited memory utility and the incremental contribution of individual rewrites provides an interesting perspective on persistent-memory credit assignment.
- The formulation considers delayed usefulness rather than immediate downstream performance alone.
2. MGPO Counterfactual Intuition
- Comparing pre- and post-rewrite memories across future tasks provides an intuitive and principled learning signal.
- The connection to potential-based reward shaping and preservation of the expected policy gradient strengthens the mathematical formulation.
3. Experimental Depth
- The paper includes extensive baseline comparisons, reward ablations, cross-domain transfer, multiple downstream readers, and long-context evaluations.
- Analyses of future memory gains, chunk-distance performance, gradient variance, and memory compactness provide useful experimental insights.
Critical weaknesses
W1. Limited Retention Validation and Generalizability of MGPO Training (Primary Blocker)
- Improved downstream extraction and compact memory do not necessarily establish reliable retention of relevant information, selective forgetting, or resistance to harmful information loss across successive rewrites. Direct retention-oriented evaluations remain limited.
- More importantly, MGPO is proposed as a reward-based training methodology, but its main training experiments use only a Qwen3-8B memory writer. Evaluating the trained writer with different frozen readers does not establish the generalizability of the proposed training strategy.
- A controlled investigation across representative trainable writer families or scales is important to determine whether MGPO provides meaningful benefits beyond a single model configuration and under what conditions it remains effective. Uniformly large improvements across all models are not necessary, but experimental investigation of this generality is important for supporting the methodological contribution.
W2. Limited Investigation of Relevance Weighting and Future Utility Balancing
- MGPO aggregates signed utility differences across the remaining horizon without explicit temporal weighting or a mechanism to prioritize future targets based on their relative importance.
- A substantial immediate improvement may be offset by small, persistent future degradations, or vice versa. Although the formulation mathematically aggregates these contributions, the paper does not sufficiently investigate how such conflicting utilities should be balanced.
- This limits understanding of when the proposed reward encourages beneficial long-term retention versus optimizing cumulative downstream utility under a particular evaluation horizon.
Questions for authors
Q1 Cross-Family Generalizability: Could the authors evaluate MGPO by independently training memory writers from different model families or scales, comparing against factual and myopic reward formulations under matched conditions? This would help establish the generality and conditions of effectiveness of the proposed training strategy.
Q2. Retention and Selective Forgetting: Can the authors provide direct evaluations of how reliably MGPO preserves relevant information, discards irrelevant content, and avoids harmful forgetting across successive memory rewrites, beyond downstream extraction performance?
Q3. Conflicting Future Utilities: How does MGPO balance large immediate gains against smaller but persistent future degradations, especially when future targets differ in relevance or importance? Could the authors investigate the sensitivity of the undiscounted reward aggregation to such scenarios?
5 comments
W1 & Q2: Direct Evidence of Retention and Selective Forgetting
We first analyze the proportion of salient entities in the memory generated by each method and introduce two complementary metrics:
- Harmful Forgetting Rate (HFR ↓): The proportion of previously stored salient entities prematurely removed despite remaining relevant to future ground-truth relations, measuring the loss of future-relevant information.
- Selective Forgetting Rate (SFR ↑): The proportion of previously stored non-salient entities subsequently removed, measuring the selective removal of non-essential information.
HFR and SFR are computed over entity–rewrite transitions, conditioned on the entity being present immediately before the rewrite.
Table C2. Entity-level memory behavior on SciREX. Rates are percentages. HFR/SFR show bootstrap mean ± bootstrap SD over 2,000 document-level resamples.
| Method | Salient entities (%) ↑ | Memory tokens | HFR ↓ | SFR ↑ |
|---|---|---|---|---|
| BASE | 19.55 | 201.2 | 48.21 ± 6.46 | 51.20 ± 2.39 |
| FACTUAL | 16.78 | 198.0 | 37.29 ± 6.00 | 45.84 ± 2.72 |
| MYOPIC | 22.36 | 179.7 | 42.04 ± 5.80 | 56.97 ± 2.38 |
| FULL | 51.36 | 41.5 | 14.19 ± 6.38 | 88.73 ± 2.49 |
FULL combines more concentrated salient information, lower harmful forgetting, and greater removal of non-salient entities in substantially shorter memories.
The preceding analysis uses each method's naturally generated memory trajectories, reflecting both initial information selection and subsequent retention behavior. To disentangle these factors, we conduct a controlled retention experiment. We start all writers from shared initial memories, sampled from FULL trajectories, and track 114 salient entities. After four rewrites, retention is 87.74% for FULL, 89.29% for FACTUAL, 42.93% for MYOPIC, and 27.48% for BASE.
Table C3. Salient entity retention under shared initial memory (%).
| Method | 1 Rewrite | 2 Rewrites | 3 Rewrites | 4 Rewrites |
|---|---|---|---|---|
| FULL | 92.55 ± 2.55 | 91.15 ± 4.92 | 90.53 ± 7.64 | 87.74 ± 12.46 |
| FACTUAL | 92.39 ± 3.11 | 94.47 ± 3.51 | 93.62 ± 5.04 | 89.29 ± 8.36 |
| MYOPIC | 84.57 ± 6.46 | 67.73 ± 16.44 | 50.79 ± 27.01 | 42.93 ± 28.80 |
| BASE | 73.61 ± 8.70 | 55.81 ± 15.79 | 44.92 ± 24.44 | 27.48 ± 19.34 |
Under shared initial memory, both FULL and FACTUAL exhibit strong retention across successive updates. After four rewrites, they retain 87.74% and 89.29% of salient entities, respectively, whereas MYOPIC and BASE decline to 42.93% and 27.48%.
The two evaluations address complementary aspects of memory behavior. On natural trajectories, FULL maintains a higher concentration of salient entities while removing fewer future-relevant entities and more non-salient entities. From shared initial memories, FULL preserves already stored salient information at a rate similar to FACTUAL. Together, these results support task-selective memory updating, with the clearest distinction appearing along each policy’s naturally generated trajectories.
W2 & Q3. Conflicting Future Utilities and Temporal Weighting
We thank the reviewer for raising questions regarding "conflicting future utilities" and "temporal weighting." To directly examine the impact of different weighting mechanisms on credit assignment in MGPO, we have established a unified Credit Robustness Bound and conducted sensitivity experiments guided by this theory. We would be happy to include these findings in the final version of the paper.
Credit-sign stability under reweighting. We provide a checkable certificate for credit stability under reweighting, holding memory trajectories and reader outputs fixed. For each rewrite, we compute a perturbation bound by summing the absolute change in each target’s weight multiplied by the corresponding absolute utility difference. The triangle inequality guarantees that the absolute change in total credit cannot exceed this bound. Thus, absolute original credit > perturbation bound is a sufficient condition for preserving the credit’s nonzero sign. This calculation applies to target importance, temporal discounting, and horizon truncation. Under a uniform cap on weight changes, a bound valid for every admissible assignment is weight-change cap × sum of absolute target-level utility differences. Pairwise ordering is similarly preserved when original credit gap > sum of the two perturbation bounds. We call the bound divided by the sum of absolute utility differences the normalized perturbation bound. In our target-importance experiments, every target is upweighted or downweighted by exactly the specified percentage, with its weight shared across rewrites. The normalized bound therefore equals that percentage for every rewrite with a positive denominator, explaining the identical median and 90th-percentile values.
Experimental Setup. We analyze the counterfactual utility matrix from MGPO training (seed 1, step 45), keeping memory trajectories fixed. We consider three perturbations:
- Target importance: Fixed, rewrite-independent target weights with maximum deviations of 10%–70%. Actual sensitivity is evaluated over 1,000 random global weight assignments.
- Temporal discounting: Discount factors of 0.9, 0.7, 0.5, and 0.2 on memory gain.
- Horizon truncation: Retaining contributions within the next 0, 1, 2, 3, or 5 chunks.
Evaluation Metrics.
- β P50/P90: Median and 90th percentile of the normalized perturbation bound.
- Certified Stability ↑: Percentage of rewrites whose credit signs are theoretically guaranteed to remain unchanged.
- Strict Flip ↓: Percentage of rewrites exhibiting actual positive-to-negative or negative-to-positive credit changes.
- Rank Inversion ↓: Percentage of rewrite pairs whose relative credit ordering is reversed.
We report results for all rewrites and separately for conflicting rewrites, where future utility contributions contain both positive and negative values including current contributions.
1. Memory Gain exhibits substantial stability under moderate target-importance perturbations in the analyzed trajectories.
At a 30% maximum weight deviation, 89.2% of rewrites have certified sign stability, with only 2.4% actual sign flips. For conflicting rewrites, these figures become 67.4% and 6.7%, respectively. This confirms that MGPO's credit assignment is generally stable under moderate importance changes, while conflicting utilities naturally yield smaller robustness margins.
2. Future utilities materially determine credit assignment.
For conflicting rewrites, temporal discounting with a factor of 0.7 changes the credit sign in 31.5% of cases. Restricting credit to the current chunk increases this rate to 61.8%, alongside 42.3% rank inversions. These results demonstrate that future contributions can reverse immediate credit in either direction, highlighting the importance of evaluating both persistent gains and losses, and that discounting or truncating future contributions can substantially alter both credit direction and rewrite preference.
3. Why undiscounted full horizon?
Our choice is grounded in objective consistency with the Factual method rather than hyperparameter preference. Fixed, rewrite-independent target weights preserve MGPO's exact telescoping credit decomposition, corresponding to a reweighted factual objective. In contrast, applying rewrite-relative temporal discounts to the marginal contributions generally breaks the exact telescoping identity with our original undiscounted factual objective.
Thus, undiscounted full-horizon aggregation preserves alignment with the original factual objective while fully accounting for persistent future gains and losses. Task-specific target weighting remains a compatible extension, but learning such weights and modeling interactions between rewrites are beyond the scope of this work.
W1 & Q1: Cross-family Generalizability of MGPO
Thank you for identifying cross-writer training and direct retention evaluation as the key evidence needed to support the methodological contribution. We have added both, together with a sensitivity analysis of conflicting future utilities.
We independently train Llama-3.1-8B and Qwen3-4B writers using the same SciREX data, frozen reader, memory budget, and evaluation protocol. This directly tests the training method, complementing the original evaluation of an already trained writer across readers.
Table C1. SciREX performance across writer configurations. Scores are averaged over four evaluation repeats. These repeats measure evaluation variability, not independent training variability.
FULL improves over BASE and achieves the highest mean performance in all three configurations. On Qwen3-4B, it exceeds the strongest alternative by 3.40 Cluster F1 and 4.53 Relation F1. On Llama-3.1-8B, FULL and MYOPIC perform similarly.
These findings directly complement the cross-reader evaluation in the original paper. The new experiments establish the applicability of the MGPO optimization strategy across different writers, while the original evaluation demonstrates the reusability of learned memory representations across different readers.