acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Remember: Long-horizon Counterfactual Memory Optimization

Abstract

Persistent textual memory allows language models to carry information across long interactions, but learning what to remember is fundamentally a credit-assignment problem. A memory rewrite may only become useful many steps later, while much of the observed utility may be inherited from information already stored before the rewrite. We introduce Memory Gain Policy Optimization (MGPO), which isolates the incremental value of each memory rewrite by crediting it for its marginal contribution to current and future downstream utility. This turns delayed memory utility into a direct learning signal for optimizing what information should persist. We study MGPO on document-level information extraction, where structured supervision makes the effects of individual memory updates directly measurable. MGPO improves extraction while reducing average memory length by nearly 80% relative to the initial memory policy before optimization. The learned memory policy also supports reuse and transfer across domains, downstream models without further training. These results show that effective memory learning depends not only on preserving useful information, but on identifying which memory updates create lasting incremental value.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agent Memory, Counterfactual Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AGSyOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposes Memory Gain Policy Optimization (MGPO), a reinforcement-learning approach for persistent memory optimization. It addresses the problem of attributing downstream utility to individual memory rewrites rather than rewarding information already present in memory. MGPO introduces a long-horizon counterfactual reward that measures the incremental contribution of each rewrite to current and future tasks, while preserving the underlying expected policy gradient. Experiments primarily focus on document-level information extraction, with additional evaluations of reward variants, cross-domain transfer, downstream readers, and long-context question answering.

Critical strengths

1. Problem Reformulation

  • The distinction between inherited memory utility and the incremental contribution of individual rewrites provides an interesting perspective on persistent-memory credit assignment.
  • The formulation considers delayed usefulness rather than immediate downstream performance alone.

2. MGPO Counterfactual Intuition

  • Comparing pre- and post-rewrite memories across future tasks provides an intuitive and principled learning signal.
  • The connection to potential-based reward shaping and preservation of the expected policy gradient strengthens the mathematical formulation.

3. Experimental Depth

  • The paper includes extensive baseline comparisons, reward ablations, cross-domain transfer, multiple downstream readers, and long-context evaluations.
  • Analyses of future memory gains, chunk-distance performance, gradient variance, and memory compactness provide useful experimental insights.
Critical weaknesses

W1. Limited Retention Validation and Generalizability of MGPO Training (Primary Blocker)

  • Improved downstream extraction and compact memory do not necessarily establish reliable retention of relevant information, selective forgetting, or resistance to harmful information loss across successive rewrites. Direct retention-oriented evaluations remain limited.
  • More importantly, MGPO is proposed as a reward-based training methodology, but its main training experiments use only a Qwen3-8B memory writer. Evaluating the trained writer with different frozen readers does not establish the generalizability of the proposed training strategy.
  • A controlled investigation across representative trainable writer families or scales is important to determine whether MGPO provides meaningful benefits beyond a single model configuration and under what conditions it remains effective. Uniformly large improvements across all models are not necessary, but experimental investigation of this generality is important for supporting the methodological contribution.

W2. Limited Investigation of Relevance Weighting and Future Utility Balancing

  • MGPO aggregates signed utility differences across the remaining horizon without explicit temporal weighting or a mechanism to prioritize future targets based on their relative importance.
  • A substantial immediate improvement may be offset by small, persistent future degradations, or vice versa. Although the formulation mathematically aggregates these contributions, the paper does not sufficiently investigate how such conflicting utilities should be balanced.
  • This limits understanding of when the proposed reward encourages beneficial long-term retention versus optimizing cumulative downstream utility under a particular evaluation horizon.
Questions for authors

Q1 Cross-Family Generalizability: Could the authors evaluate MGPO by independently training memory writers from different model families or scales, comparing against factual and myopic reward formulations under matched conditions? This would help establish the generality and conditions of effectiveness of the proposed training strategy.

Q2. Retention and Selective Forgetting: Can the authors provide direct evaluations of how reliably MGPO preserves relevant information, discards irrelevant content, and avoids harmful forgetting across successive memory rewrites, beyond downstream extraction performance?

Q3. Conflicting Future Utilities: How does MGPO balance large immediate gains against smaller but persistent future degradations, especially when future targets differ in relevance or importance? Could the authors investigate the sensitivity of the undiscounted reward aggregation to such scenarios?

5 comments
AuthorsOpenReview ↗

W1 & Q1: Cross-family Generalizability of MGPO

Thank you for identifying cross-writer training and direct retention evaluation as the key evidence needed to support the methodological contribution. We have added both, together with a sensitivity analysis of conflicting future utilities.

We independently train Llama-3.1-8B and Qwen3-4B writers using the same SciREX data, frozen reader, memory budget, and evaluation protocol. This directly tests the training method, complementing the original evaluation of an already trained writer across readers.

Table C1. SciREX performance across writer configurations. Scores are averaged over four evaluation repeats. These repeats measure evaluation variability, not independent training variability.

WriterBASEFACTUALMYOPICFULL
Cluster F1Relation F1Cluster F1Relation F1Cluster F1Relation F1Cluster F1Relation F1
Llama-3.1-8B38.1816.2841.5818.1943.8820.0644.5020.59
Qwen3-4B40.6816.9946.6921.8044.6820.0450.0926.33
Qwen3-8B37.8015.5138.6715.3238.4015.1551.2926.54

FULL improves over BASE and achieves the highest mean performance in all three configurations. On Qwen3-4B, it exceeds the strongest alternative by 3.40 Cluster F1 and 4.53 Relation F1. On Llama-3.1-8B, FULL and MYOPIC perform similarly.

These findings directly complement the cross-reader evaluation in the original paper. The new experiments establish the applicability of the MGPO optimization strategy across different writers, while the original evaluation demonstrates the reusability of learned memory representations across different readers.

AuthorsOpenReview ↗

W1 & Q2: Direct Evidence of Retention and Selective Forgetting

We first analyze the proportion of salient entities in the memory generated by each method and introduce two complementary metrics:

  • Harmful Forgetting Rate (HFR ↓): The proportion of previously stored salient entities prematurely removed despite remaining relevant to future ground-truth relations, measuring the loss of future-relevant information.
  • Selective Forgetting Rate (SFR ↑): The proportion of previously stored non-salient entities subsequently removed, measuring the selective removal of non-essential information.

HFR and SFR are computed over entity–rewrite transitions, conditioned on the entity being present immediately before the rewrite.

Table C2. Entity-level memory behavior on SciREX. Rates are percentages. HFR/SFR show bootstrap mean ± bootstrap SD over 2,000 document-level resamples.

MethodSalient entities (%) ↑Memory tokensHFR ↓SFR ↑
BASE19.55201.248.21 ± 6.4651.20 ± 2.39
FACTUAL16.78198.037.29 ± 6.0045.84 ± 2.72
MYOPIC22.36179.742.04 ± 5.8056.97 ± 2.38
FULL51.3641.514.19 ± 6.3888.73 ± 2.49

FULL combines more concentrated salient information, lower harmful forgetting, and greater removal of non-salient entities in substantially shorter memories.

The preceding analysis uses each method's naturally generated memory trajectories, reflecting both initial information selection and subsequent retention behavior. To disentangle these factors, we conduct a controlled retention experiment. We start all writers from shared initial memories, sampled from FULL trajectories, and track 114 salient entities. After four rewrites, retention is 87.74% for FULL, 89.29% for FACTUAL, 42.93% for MYOPIC, and 27.48% for BASE.

Table C3. Salient entity retention under shared initial memory (%).

Method1 Rewrite2 Rewrites3 Rewrites4 Rewrites
FULL92.55 ± 2.5591.15 ± 4.9290.53 ± 7.6487.74 ± 12.46
FACTUAL92.39 ± 3.1194.47 ± 3.5193.62 ± 5.0489.29 ± 8.36
MYOPIC84.57 ± 6.4667.73 ± 16.4450.79 ± 27.0142.93 ± 28.80
BASE73.61 ± 8.7055.81 ± 15.7944.92 ± 24.4427.48 ± 19.34

Under shared initial memory, both FULL and FACTUAL exhibit strong retention across successive updates. After four rewrites, they retain 87.74% and 89.29% of salient entities, respectively, whereas MYOPIC and BASE decline to 42.93% and 27.48%.

The two evaluations address complementary aspects of memory behavior. On natural trajectories, FULL maintains a higher concentration of salient entities while removing fewer future-relevant entities and more non-salient entities. From shared initial memories, FULL preserves already stored salient information at a rate similar to FACTUAL. Together, these results support task-selective memory updating, with the clearest distinction appearing along each policy’s naturally generated trajectories.

AuthorsOpenReview ↗

W2 & Q3. Conflicting Future Utilities and Temporal Weighting

We thank the reviewer for raising questions regarding "conflicting future utilities" and "temporal weighting." To directly examine the impact of different weighting mechanisms on credit assignment in MGPO, we have established a unified Credit Robustness Bound and conducted sensitivity experiments guided by this theory. We would be happy to include these findings in the final version of the paper.

Credit-sign stability under reweighting. We provide a checkable certificate for credit stability under reweighting, holding memory trajectories and reader outputs fixed. For each rewrite, we compute a perturbation bound by summing the absolute change in each target’s weight multiplied by the corresponding absolute utility difference. The triangle inequality guarantees that the absolute change in total credit cannot exceed this bound. Thus, absolute original credit > perturbation bound is a sufficient condition for preserving the credit’s nonzero sign. This calculation applies to target importance, temporal discounting, and horizon truncation. Under a uniform cap on weight changes, a bound valid for every admissible assignment is weight-change cap × sum of absolute target-level utility differences. Pairwise ordering is similarly preserved when original credit gap > sum of the two perturbation bounds. We call the bound divided by the sum of absolute utility differences the normalized perturbation bound. In our target-importance experiments, every target is upweighted or downweighted by exactly the specified percentage, with its weight shared across rewrites. The normalized bound therefore equals that percentage for every rewrite with a positive denominator, explaining the identical median and 90th-percentile values.

Experimental Setup. We analyze the counterfactual utility matrix from MGPO training (seed 1, step 45), keeping memory trajectories fixed. We consider three perturbations:

  • Target importance: Fixed, rewrite-independent target weights with maximum deviations of 10%–70%. Actual sensitivity is evaluated over 1,000 random global weight assignments.
  • Temporal discounting: Discount factors of 0.9, 0.7, 0.5, and 0.2 on memory gain.
  • Horizon truncation: Retaining contributions within the next 0, 1, 2, 3, or 5 chunks.

Evaluation Metrics.

  • β P50/P90: Median and 90th percentile of the normalized perturbation bound.
  • Certified Stability ↑: Percentage of rewrites whose credit signs are theoretically guaranteed to remain unchanged.
  • Strict Flip ↓: Percentage of rewrites exhibiting actual positive-to-negative or negative-to-positive credit changes.
  • Rank Inversion ↓: Percentage of rewrite pairs whose relative credit ordering is reversed.

We report results for all rewrites and separately for conflicting rewrites, where future utility contributions contain both positive and negative values including current contributions.

1. Memory Gain exhibits substantial stability under moderate target-importance perturbations in the analyzed trajectories.

At a 30% maximum weight deviation, 89.2% of rewrites have certified sign stability, with only 2.4% actual sign flips. For conflicting rewrites, these figures become 67.4% and 6.7%, respectively. This confirms that MGPO's credit assignment is generally stable under moderate importance changes, while conflicting utilities naturally yield smaller robustness margins.

2. Future utilities materially determine credit assignment.

For conflicting rewrites, temporal discounting with a factor of 0.7 changes the credit sign in 31.5% of cases. Restricting credit to the current chunk increases this rate to 61.8%, alongside 42.3% rank inversions. These results demonstrate that future contributions can reverse immediate credit in either direction, highlighting the importance of evaluating both persistent gains and losses, and that discounting or truncating future contributions can substantially alter both credit direction and rewrite preference.

3. Why undiscounted full horizon?

Our choice is grounded in objective consistency with the Factual method rather than hyperparameter preference. Fixed, rewrite-independent target weights preserve MGPO's exact telescoping credit decomposition, corresponding to a reweighted factual objective. In contrast, applying rewrite-relative temporal discounts to the marginal contributions generally breaks the exact telescoping identity with our original undiscounted factual objective.

Thus, undiscounted full-horizon aggregation preserves alignment with the original factual objective while fully accounting for persistent future gains and losses. Task-specific target weighting remains a compatible extension, but learning such weights and modeling interactions between rewrites are beyond the scope of this work.

Reviewer_rpFROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Memory Gain Policy Optimization (MGPO) for learning a persistent textual memory writer against a frozen downstream reader. It compares memories before and after each rewrite on the same current and future targets, separating the rewrite’s incremental utility from information already stored. The resulting counterfactual return preserves the underlying expected policy gradient while changing the credit-assignment estimator. Experiments demonstrate improved document-level information extraction with shorter memories, alongside transfer across readers, domains, and selected long-context QA tasks.

Critical strengths
  1. The counterfactual formulation is intuitive and well motivated. Comparing adjacent memory states provides a principled way to distinguish inherited utility from the contribution of a new rewrite, addressing an important credit-assignment problem in persistent memory learning.

  2. The performance–memory trade-off is compelling. On SciREX, MGPO reduces average memory length from 202 to 43 tokens while improving entity-cluster and relation F1 over the unoptimized writer. Transfer to other readers and to AIPAN-10K further supports the usefulness of the learned memory policy.

  3. Section 3.3 provides meaningful evidence for the proposed design. In particular, FACTUAL, MYOPIC, and FULL share supervision and calibration, making their comparison informative. The gradient-variance analysis and delayed-utility diagnostics further support the benefit of assigning credit across the memory’s future reuse horizon.

Critical weaknesses
  1. Although the pre-rewrite counterfactual is well motivated, its advantage over alternative baselines remains incompletely established. The gradient-preservation argument also applies to other action-independent baselines; it does not by itself establish that this particular choice gives the most informative learning signal. Comparisons with an empty-memory, appropriately shuffled-memory, or learned value baseline under comparable computational budgets would clarify the specific benefit of using the actual preceding memory. Confirming the variance reduction at trained checkpoints would also strengthen the current analysis based on the initial policy.

  2. The boundary of transfer beyond information extraction needs clearer characterization. The BABILong evaluation is valuable, but gains over MGPO-base are mixed, including lower accuracy across all evaluated QA2 context lengths. An analysis of these cases would help distinguish task-specific specialization from memory loss or reader limitations, rather than attributing them directly to optimization failure. Additional compositional reasoning experiments, or memory evaluation where future questions are unknown during writing, would further clarify the method’s broader applicability.

Questions for authors

See weakness

8 comments
AuthorsOpenReview ↗

W1. Variance Analysis

We thank the reviewer for the constructive assessment. We clarify the preceding-memory counterfactual and the scope of the transfer evidence below.

First, thank you for these suggestions on training variance. We compared the actual preceding memory with empty-memory and shuffled-memory references and extended the variance analysis to trained checkpoints.

We evaluate 24 checkpoints from 3 independent training seeds, 2 training rewards (FULL and FACTUAL), and training steps 10, 30, 50, and 76, with 76 being the final step. At each checkpoint, we sample 4 complete trajectories for each of 16 training documents and evaluate every reference on the same trajectories. All comparisons are therefore paired. Each reference is subtracted from the factual return. The shuffled reference uses the memory at the same position in another document’s trajectory and is fixed before sampling.

  • Return variance is computed after centering each estimator at every rewrite position using a leave-one-document-out mean. We measure it both per step and for the sum over a trajectory.
  • Gradient variance is the trace of the covariance across each document’s 4 trajectories. We use the unclipped, token-summed score-function gradient of the complete trajectory, with the estimators defined in the paper and no additional baseline. The measurement covers the attention projections in the first, middle, and last transformer blocks, comprising 1.26 × 10^8 parameters.

All entries are ratios relative to FACTUAL, reported as geometric means over checkpoints with 95% document-bootstrap intervals. These intervals capture uncertainty from document sampling, not variation across independent training runs.

The actual preceding memory yields the lowest variance point estimates on all three measures (Table A). Its trajectory-level return-variance ratio is 0.53 [0.38, 0.70], compared with 0.89 for empty memory and 0.87 for shuffled memory, whose intervals both include 1. The gradient-variance ratios follow the same ordering. During training, the preceding-memory baseline achieves the lowest variance estimates among the evaluated baselines.

AuthorsOpenReview ↗

W1. Variance Analysis (Results)

Table B1. Action-independent references on the same trajectories, relative to FACTUAL. Values below 1 indicate lower variance.

ReferencePer-step return variancePer-trajectory return varianceFull-trajectory gradient variance
Actual preceding memory (MGPO)0.85 [0.67, 1.01]0.53 [0.38, 0.70]0.69 [0.56, 0.85]
Empty memory0.96 [0.76, 1.12]0.89 [0.62, 1.14]0.77 [0.51, 1.04]
Shuffled memory1.08 [0.83, 1.33]0.87 [0.60, 1.11]0.87 [0.64, 1.13]

Trajectory-level return variance is reduced at every sampled training step (Table B). The overall gradient-variance ratio is 0.69 [0.56, 0.85], with intervals excluding 1 at steps 10, 30, and 50, but not at the final step (0.78 [0.53, 1.09]). The fixed-state analysis also confirms a reduction for the FULL-trained policy (Table C). On FULL-trained states, the ratio is 0.83 [0.70, 0.96] across all states and much lower after the first rewrite. On Factual-trained states, the ratio follows the same pattern. These results show that MGPO consistently reduces the measured gradient-estimator variance across multiple training stages while preserving the same underlying factual objective.

Table B2. Actual preceding memory relative to FACTUAL, by training step.

Training stepCheckpointsPer-step return variancePer-trajectory return varianceFull-trajectory gradient variance
Overall240.85 [0.67, 1.01]0.53 [0.38, 0.70]0.69 [0.56, 0.85]
1060.82 [0.60, 1.06]0.48 [0.32, 0.72]0.68 [0.54, 0.84]
3060.91 [0.72, 1.07]0.65 [0.44, 0.85]0.71 [0.59, 0.85]
5060.87 [0.68, 1.03]0.56 [0.37, 0.76]0.60 [0.48, 0.86]
7660.80 [0.60, 1.00]0.47 [0.33, 0.61]0.78 [0.53, 1.09]
AuthorsOpenReview ↗

W2. Characterizing Cross-Task Transfer Beyond Information Extraction

We thank the reviewer for suggesting a more detailed characterization of cross-task transfer. We examine the results across QA1–QA5 to identify where MGPO provides the clearest benefits, and complement this comparison with a paired QA2 analysis that distinguishes memory evidence loss from reader limitations.

Transfer beyond IE without QA-specific training. BABILong serves as a supplementary evaluation of cross-task transfer. Trained exclusively on SciREX, MGPO is applied to these QA tasks without further optimization, with all writers evaluated using the same frozen Qwen3-14B reader. MGPO outperforms HiMPO in 19 of 20 shared task–length settings and MemAgent in 16. Comparison with MGPO-base further identifies which capabilities benefit from SciREX training.

Table B4. Task-level transfer on BABILong. Accuracy is averaged over the four shared context lengths: 8K, 16K, 32K, and 128K, with 100 examples per task and length. Δ denotes the percentage-point change relative to MGPO-base.

TaskEvidence requirementMGPO (%)MGPO-base (%)Δ (pp)
QA1Latest location of a person77.5077.500.00
QA2Object location through possession and movement events26.0030.75−4.75
QA3Historical object location from an ordered evidence chain30.7526.50+4.25
QA4Directional relations between locations56.5056.500.00
QA5Giver–object–recipient relations69.7568.50+1.25

Where MGPO helps and why. The clearest improvement occurs on QA3, which requires preserving an object–holder association together with an ordered location history. MGPO gains 7, 7, and 4 points at 8K, 16K, and 128K, respectively, with a 1-point decrease at 32K. QA5, which requires distinguishing participants in a transfer event, improves by 6 and 5 points at 16K and 32K, although it declines at 8K and matches the base at 128K. QA1 and QA4 remain essentially unchanged.

These results are consistent with transfer of the evidence-selection strategy encouraged by the SciREX objective: retaining entities together with relationships needed for downstream predictions. A plausible explanation for the QA3 and QA5 gains is that their queries provide explicit cues for selecting relational evidence. QA3 names both an object and a reference location, while QA5 specifies roles in a transfer event. These cues can help guide memory construction toward the relevant history or relation. Because the reader is frozen, the gains reflect improvements in the usefulness of the generated memory for these queries. The differing outcomes across tasks also indicate that the structure of the required evidence matters more than supporting-fact count alone.

Reviewer_8dFpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies learning a bounded textual memory for streaming documents. A trainable writer (Qwen3-8B) rewrites a ≤256-token memory after each 1,024-token chunk, and a frozen reader (Qwen3-14B) uses memory plus the current chunk to extract entity clusters and binary relations. The authors identify “inherited utility” as the central credit-assignment problem: the reader’s score under m_t largely reflects content already in m_{t−1}. They propose Memory Gain Policy Optimization (MGPO), which credits rewrite t with MG_t = Σ_{j≥t}(F_{t,j} − F_{t−1,j}), computed from a triangular counterfactual matrix of reader evaluations. They show this equals the factual return minus a pre-rewrite potential Φ_t, preserving the expected policy gradient, and they train with a position-EMA advantage and PPO-style clipping. On SciREX, MGPO reaches 51.3 cluster F1 and 26.5 relation F1 with a 43-token average memory. They also report transfer to AIPAN-10K, to other readers (Table 2), and to BABILong.

Critical strengths

Clear problem framing. The paper identifies inherited utility as the confound in memory learning, and Fig. 3 supports it empirically: pre- and post-rewrite utilities track each other closely once memory is established. Well-motivated, computable method. Memory Gain compares pre- and post-rewrite memories under the same reader and target, which makes the counterfactual clean. Chunk-aligned targets and the linearized F1 surrogate make the credit additive, and the surrogate tracks exact document-level F1 gain closely (Spearman 0.988, App. E.1). Reuse across readers. Decoupling the writer from the reader lets one trained writer improve six readers across four model families without retraining (Table 2). Out-of-domain evidence. The SciREX-trained writer transfers to a different schema (AIPAN-10K) and to BABILong without further training.

Critical weaknesses

Variance reduction is not shown to explain the gain. The paper attributes the Factual-vs-Full gap to Φ_t acting as a control variate, but the only supporting measurement is a 25% gradient-variance reduction (CI 6-43%) at fixed states under the untrained policy, computed over a small parameter subset. That is modest next to a roughly 12-point cluster F1 gap, and training-time variance is not reported. Nothing rules out that Φ_t also changes the advantage scale, which interacts with the global normalization and PPO clipping, and it is unclear whether the error bars reflect independent training seeds or only repeated evaluation of one trained model (check this against the paper before including it). Please report variance during training, results across independent training seeds, and a scale-matched Factual control.

Possible optimization twds predictions. Memory Gain scores rewrite t by holding m_t fixed over all later chunks, but at inference later memories keep updating. For j > t+1, a correct guess in m_t about the intermediate chunks (or a dataset-level prior learned from SciREX) raises F_{t,j} and earns credit. Thus, there is optimization pressure towards predicting the future input. Could you elaborate if this is a real concern?

Experimental evidence is limited. MGPO trains on chunk-aligned targets, while R1-RE, MemAgent, and HiMPO train on document-level targets, so part of the gap may be denser supervision and not better credit assignment. The test sets are small (66 and 50 documents) and Table 1 reports no standard deviations. The cluster margin over R1-RE and MemAgent is under 2 F1, and the AIPAN gains (0.4 and 1.6 F1) are within the variation seen elsewhere (±1 to 1.4).

Questions for authors

Could the framework be applied, and evaluated, with a reader that also sees the full document, so that the memory is trained as a learned guide rather than only as a substitute for context?

Should the learned memory be independent of the downstream query, so it can be built once and reused, and if so, isn't what we want just a plain summarizer?

6 comments
AuthorsOpenReview ↗

R1. Variance during training

Thank you for requesting independent training seeds, training-time variance measurements, and a direct examination of advantage scaling. We address these separately.

We evaluate 24 checkpoints spanning three seeds, FULL- and FACTUAL-trained policies, and steps 10, 30, 50, and 76. At each checkpoint, both estimators are evaluated on the same four trajectories per document for 16 training documents.

Table A1. MGPO relative to FACTUAL during training. Values below 1 indicate lower measured variance for MGPO.

Training stepCheckpointsPer-step return variancePer-trajectory return varianceFull-trajectory gradient variance
Overall240.85 [0.67, 1.01]0.53 [0.38, 0.70]0.69 [0.56, 0.85]
1060.82 [0.60, 1.06]0.48 [0.32, 0.72]0.68 [0.54, 0.84]
3060.91 [0.72, 1.07]0.65 [0.44, 0.85]0.71 [0.59, 0.85]
5060.87 [0.68, 1.03]0.56 [0.37, 0.76]0.60 [0.48, 0.86]
7660.80 [0.60, 1.00]0.47 [0.33, 0.61]0.78 [0.53, 1.09]

Gradient measurements use unclipped, token-summed score-function gradients with no additional baseline, covering 1.26×1081.26\times10^8 parameters in selected attention projections. Return measurements use position-wise centering. The intervals reflect document sampling rather than training-seed uncertainty.

Trajectory-return variance is reduced at every sampled step. Gradient-variance intervals exclude 1 at steps 10, 30, and 50; at the final step, the estimate is 0.78 [0.53, 1.09].

These measurements extend the control-variate evidence beyond the initial policy. We will revise Section 3.3 to distinguish this estimator-level result from the downstream performance gain: variance reduction supports the proposed mechanism but does not, by itself, explain the full F1 gap. The expected-gradient result likewise concerns the score-function estimator, rather than every normalized and clipped PPO update.

AuthorsOpenReview ↗

R2. Robustness Across Training Seeds

We thank the reviewer for highlighting training-seed variability. Due to computational constraints, we conducted additional experiments comparing FULL and FACTUAL across three independent training seeds. Unlike the four evaluation runs reported in the original paper, these experiments assess variability across independent training runs.

Table A2. Multi-seed performance comparison between FULL and FACTUAL.

MethodCluster F1Relation F1
FULL (3 seeds)47.76 ± 3.4124.07 ± 2.81
FACTUAL (3 seeds)41.58 ± 2.6418.43 ± 2.69
SeedFULL Cluster F1FACTUAL Cluster F1FULL Relation F1FACTUAL Relation F1
151.2938.6726.5415.32
244.4942.2621.0219.83
347.5043.8124.6520.13

FULL achieves average gains of 6.18 and 5.64 F1 points on clustering and relation extraction, respectively, outperforming FACTUAL on both metrics in all three seeds. Despite training variability, these results provide further evidence that counterfactual credit assignment improves memory optimization beyond factual returns alone.

AuthorsOpenReview ↗

R3. Does Memory Gain encourage predicting future inputs?

We thank the reviewer for raising this concern. MGPO does not directly reward the writer for anticipating future annotations. Future targets are used only to evaluate the incremental downstream utility of a memory rewrite through a frozen reader. Moreover, the utility is aligned with extraction F1, where false positives can incur negative contributions, and a correctly anticipated fact provides no additional gain unless it improves downstream extraction. Importantly, as established in Eq. (11), MGPO preserves the expected policy gradient of the factual objective rather than introducing an auxiliary prediction objective. Thus, future-oriented credit encourages information that remains useful for subsequent extraction, without explicitly rewarding the generation of unseen content.

To examine whether future-oriented credit encourages unsupported content generation, we conduct a prefix-grounding analysis using the stored SciREX test trajectories (66 documents, four runs per writer), without additional model inference.

For each newly written entity, we check whether it is supported by the observed document prefix using normalized lexical matching. As shown in Table R3, MGPO achieves an unsupported entity rate of 1.93%, lower than both the untrained writer (2.39%) and Factual (3.50%). Notably, all lexically unsupported entries of FULL occur in the first chunk, with none observed in subsequent chunks.

Manual inspection suggests that many lexically unsupported entries arise from morphological variations or extraction errors rather than clear anticipation of unseen document content. Examples include extracting "accuracy" instead of "accurate," "classification" instead of "classifier," and "Performance" instead of "performs."

These findings provide no evidence that MGPO systematically encourages the generation of unseen entities.

Table A3. Prefix-based lexical screening for unsupported entities

WriterUnsupported Entities ↓First Chunk ↓Later Chunks ↓
Untrained2.39%3.56%1.92%
Terminal7.17%4.92%7.82%
Factual3.50%6.02%1.73%
Myopic1.98%2.22%1.88%
FULL1.93%2.84%0.00%

Comments on the paper

2 comments
Reviewer_AGSyOpenReview ↗

Good Rebuttal! Appreciable

Thank you to the authors for the detailed rebuttal and additional experiments. My original Weak Reject was primarily based on insufficient validation of MGPO as a generalizable training strategy, limited direct evidence of memory retention, and the handling of conflicting future utilities. I have reassessed these concerns as follows.

1. Cross-family Generalizability (W1 and Q1)

The additional experiments with Llama-3.1-8B and Qwen3-4B directly address my primary concern about evaluating MGPO training beyond a single writer configuration. FULL achieves the highest reported mean performance across all three writers, which is encouraging.

However, the improvements are relatively small for Llama-3.1-8B (+0.62 Cluster F1 and +0.53 Relation F1 over MYOPIC), compared with the larger improvements for Qwen models. Furthermore, the reported repetitions capture evaluation variability rather than independent training variability.

Substantially resolved: The authors have made a reasonable cross-family experimental attempt, which was my primary requirement. However, the small gains in some configurations and the lack of independent training-run variability limit stronger claims of consistent generalizability.

2. Retention and Selective Forgetting (W1 and Q2)

The new HFR and SFR evaluations directly address my request for retention-oriented evidence beyond downstream performance. The natural-trajectory results indicate better preservation of future-relevant information and more selective removal of non-salient content.

However, under shared initial memories, FULL retains 87.74% of salient entities after four rewrites, compared with 89.29% for FACTUAL. Thus, FULL does not clearly outperform FACTUAL in preserving already stored relevant information.

Substantially addressed: The results support selective memory maintenance, although superior retention over FACTUAL and reliability across longer rewrite horizons remain unestablished.

3. Conflicting Future Utilities (W2 and Q3)

The sensitivity experiments investigate target-importance changes, temporal discounting, and horizon truncation. The authors also provide a reasonable mathematical justification for undiscounted aggregation through its consistency with the original factual objective.

However, these experiments examine credit changes on fixed trajectories rather than the performance of memory writers retrained under alternative weighting strategies. The prioritization of future targets with different relevance or importance remains open.

Partially resolved: The reward's sensitivity and design rationale are better established, but the effects of alternative weighting choices on actual memory learning remain unverified.

Overall Reassessment

The rebuttal substantially addresses my primary reason for Weak Reject, particularly through cross-family training experiments and direct retention evaluations. Nevertheless, the relatively small gains for some writer configurations, absence of independent training-run variability, comparable controlled retention between FULL and FACTUAL, and unresolved implications of alternative utility weighting limit the strength of the conclusions.

Given that my primary experimental requirements have now been reasonably addressed, I am updating my recommendation from Weak Reject (2) to Weak Accept (3), while keeping my confidence at 3. The remaining limitations prevent me from supporting a stronger acceptance recommendation.

AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for carefully reading our paper and providing thoughtful and constructive feedback. We greatly appreciate your suggestions, which have motivated us to conduct additional experiments and strengthen the theoretical and empirical evidence supporting MGPO.

We have provided point-by-point responses to all comments, including new experiments, theoretical analyses, and clarifications of the method's scope and limitations.

We summarize the main additions made during the rebuttal below. We have also uploaded a draft of the revised manuscript, with changes highlighted in blue, and indicate where each item is addressed below.

  1. Cross-Writer Generalization and Training Robustness: Additional experiments with Llama-3.1-8B and Qwen3-4B writers, together with three independent training seeds, demonstrate the effectiveness and reliability of MGPO across different model configurations (§3.4 P9–10; §D.3 P26).

AGSy, 8dFp

  1. Gradient Variance and Counterfactual Baselines: Training-time and fixed-state gradient analyses, comparisons with alternative counterfactual references (§3.3 P7; §C.4 P23; §E.5.1 P28), and advantage-scaling controls provide further evidence for the optimization benefits of MGPO.

8dFp, rpFR

  1. Memory Retention and Selective Forgetting: Entity-level analyses and controlled retention experiments directly examine how MGPO learns to preserve relevant information while discarding less useful content (§3.3 P8–9).

AGSy

  1. Credit Assignment Robustness: Theoretical bounds and sensitivity experiments examine conflicting future utilities, temporal discounting, and horizon truncation (§E.5.2 P28–30).

AGSy

  1. Cross-Task Transfer and Failure Analysis: Additional BABILong analyses characterize transfer across reasoning tasks, while targeted QA2 diagnostics identify limitations in evidence retention and downstream reading (§D.2 P25–26).

rpFR

  1. Additional Clarifications: We further address questions related to unsupported memory content (§2.2 P3), supervision fairness (§C.2 P22), statistical uncertainty, and the distinction between task-conditioned and query-agnostic memory (§E.6 P30–31).

8dFp, rpFR

Together, these additions provide stronger evidence for our central contribution: Toward Learning Memory Through Its Future Utility.

We once again thank all reviewers for their valuable feedback, which has helped us strengthen this work. We look forward to your further comments and discussion.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.