acceptodds
Under review as a conference paper at ICLR 2027

Which Experiences Should Be Replayed? Causal Selection for Latent-Memory Agents

Abstract

The latent-memory paradigm in self-evolving agents, exemplified by MemGen, augments reasoning with trainable Weaver latent vectors; however, it still exhibits substantial forgetting in the tested four-task staged No-replay sequential-learning setting. Replaying historical experience can mitigate this forgetting; the key is to identify examples worth replaying. To this end, we diagnose Weaver event-slot trajectories and find that static saliency signals can nominate candidates but cannot independently provide the bilateral helpful/non-harmful evidence required for replay admission; a slot's intervention effect also depends on its specific injection event. We therefore propose Causal Experience Selection (Causal replay), which uses static signals to narrow the event-slot candidate set, independently masks each candidate at the incoming checkpoint, and determines sample admission from bilateral teacher-forced evidence before constructing a fixed-budget replay set. In four-stage continual learning with Qwen2.5-1.5B-Instruct and a MemGen Weaver, Causal replay improves final four-task mean accuracy (Macro) from No replay's to , indicating that replay mitigates forgetting in the tested MemGen setting. Under strictly matched actual per-source replay counts and training volume, differing only in historical-example identity, Causal replay obtains a mean paired gain of percentage points (pp) over quantity-matched Random across three seeds (95% paired- interval ). This supports selection based on event-slot evidence over random selection in this controlled comparison; task-order permutation and Qwen3-4B also show same-direction results.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
latent-memory agents, continual learning, experience replay, causal selection, self-evolving agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WepNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies which historical examples to replay when sequentially training the Weaver in a MemGen-style latent-memory model with a frozen reasoner. It first shows that static latent rankings identify influential event-slots without reliably distinguishing helpful from harmful effects, and that these effects depend on the injection event. The proposed selector nominates four event-slots by norm, independently zeroes each selected contextualized hidden state before projection, and admits an example if teacher-forced loss changes satisfy helpfulness and harm-avoidance thresholds. It then samples original, unmasked examples under source, pattern, and budget constraints. On a four-task sequence, the method improves final macro accuracy by 2.19 percentage points over per-source quantity-matched random replay across three seeds. Component ablations and single-run checks with another task order and Qwen3-4B provide additional support.

Critical strengths
  • The paper makes a useful connection between interventions on latent memory and replay selection. I find the distinction between identifying an influential vector and determining the direction of its effect helpful. The event-conditioned, bilateral admission rule turns this observation into a concrete training procedure.
  • The main comparison separates the benefit of replay from the benefit of selecting particular examples. Matching actual per-source replay counts, training-example counts, and optimization settings makes Table 2 informative about selection quality. The gains over random replay are positive for all three seeds (1.921.92, 2.682.68, and 1.971.97 points). Table 3 further strengthens this result: bilateral admission without Pattern Cap still improves over random replay by 1.41 points, while removing harm avoidance reduces the full method's mean score by 1.091.09 points.
Critical weaknesses
  • The current evidence primarily establishes an advantage over quantity-matched random replay and internal variants. Although Table 7 includes alternative selection strategies, it uses a separate experimental protocol and does not report Causal replay under that protocol, leaving a direct comparison unresolved. Moreover, EWC and LwF provide useful classical continual-learning references, but neither is a replay selection method. They therefore do not substitute for comparisons with competitive, directly relevant replay or data selection approaches under a shared protocol. These gaps make it difficult to assess the advance over existing work and limit my assessment of the contribution’s significance.

  • Section 4.5 states that each replay stage requires five gradient-free forward passes per historical example. The method retains complete historical candidate pools and re-scores them at subsequent checkpoints. For equally sized tasks and fixed candidate-testing budget k, this yields quadratic cumulative scoring growth with task count, even when the replay training budget is fixed. The provided PDF does not report measured scoring time, end-to-end training time, or performance under a matched total compute budget. Furthermore, the main experiments match training-example counts but not token counts. The results therefore support effectiveness under a matched example budget, but do not establish whether the average 2.19-percentage-point gain over random replay justifies the scoring and storage costs, particularly for longer task sequences.

Questions for authors
  1. Comparison request: Can the authors compare Similarity with Causal replay under the same per-source replay counts and training protocol, or report Causal replay under the full-pool protocol of Table 7? Comparisons with relevant loss-, interference-, or diversity-based selectors under the same protocol would also help establish the contribution beyond random selection. Please account for selection costs when assessing these alternatives.

  2. Robustness request: The main multi-seed evidence comes from a single four-task sequence within the MemGen architecture. The alternative task order and Qwen3-4B results in Table 4 each use a single run. Could the authors provide multi-seed results for these two settings, including per-task performance and paired gains over quantity-matched random replay with uncertainty estimates? This would help determine whether the improvements are stable across task orders and backbones.

  3. Could the authors report the computational budget and measured runtime at each stage, separating historical-example scoring and selection from subsequent training, together with scoring-token counts and memory/storage costs? Please clarify whether the quantity-matched Random branch also performs checkpoint rescoring, as suggested by line 364, and distinguish costs required by the selector from costs introduced by the experimental matching protocol. A comparison under a matched total compute budget would clarify the cost–benefit tradeoff. A longer task stream or a bounded/subsampled historical-pool evaluation would also materially inform the scalability assessment.

  4. Section 4.3 explains why independent measurements implement the chosen per-candidate harm constraint, but does not establish the downstream value of this granularity. Can the authors compare the individual-slot rule with a block-level rule that jointly intervenes on all eight slots at a single injection event, at comparable scoring cost, and examine whether slot interactions change admission decisions and final replay performance?

  5. Figure 1 examines different candidate-testing budgets using diagnostic discovery rates, but does not report the sensitivity of final replay performance to kk. Could the authors clarify the validation evidence behind choosing k=4k=4, including the search range and selection criterion, and report admission rates, final performance, and scoring overhead for alternative values of kk?

  6. Lines 279–281 zero the contextualized state before projection, whereas lines 287–289 describe the intervention as candidate removal. Please specify projection bias, attention-mask and position handling, cache handling, and prediction-target alignment: depending on the implementation, zero replacement can retain a projected bias or attention/position effects and need not completely remove the slot's influence. Can the authors assess effect-sign agreement, admission overlap, and a representative downstream replay outcome under another reasonable intervention baseline, such as mean replacement or graded attenuation? The helpful/harmful interpretation should remain explicitly relative to the tested baseline.

  7. The source constraints impose upper caps rather than minimum replay allocations. What happens when no example from a historical task passes admission, or when the remaining budget is exhausted before a source is processed? Please report source-level zero-replay frequencies and subsequent task retention, and assess whether a minimum allocation or fallback policy is needed. This concerns a possibility allowed by the algorithm, rather than an assumption that it occurred in the reported runs.

  8. How is the stage training size NtN_t chosen, and how are the old/new and historical-source mixture proportions controlled under unequal task sizes and checkpoint-dependent admission rates? Please report source-level exposure and retention, particularly for GPQA. A feasible fixed/normalized mixture or reweighting control would clarify whether the allocation preserves small tasks; please specify whether sampling is with replacement. Varying proportions alone do not establish that the allocation is harmful, but their effects are not isolated by the current comparisons.

  9. Section 4.1 mentions deduplication and availability filtering without specifying their executable rules in the provided PDF. Please provide dataset versions and splits, deduplication fields and matching criteria, availability conditions, counts or item identifiers before and after each step, the AQuA8000 sampling procedure, and the held-in validation split. If these details are supplied elsewhere, please identify the exact files.

10 comments
AuthorsOpenReview ↗

Response to Reviewer WepN (1/5) — Q1: Direct Selector Comparison

Thank you for the detailed questions and for recognizing the distinction between identifying influential memory states and determining the direction of their effects. This response is posted in five parts: Part 1 addresses Q1, Part 2 Q2–Q3, Part 3 Q4–Q6, Part 4 Q7, and Part 5 Q8–Q9.

Q1. Direct selector comparison

We add both alternatives requested in Q1. First, under the same per-source counts at S4, all selectors start from Full's S3 checkpoint of the same seed, match Full's per-source replay counts m_4 on that seed (seed 43: (1,765, 1,651, 104)), and share the training-set size and training recipe (seeds 42–44). S4 is the final stage, so the effect appears directly in the final results without interference from later-stage divergence; it is also the only stage that replays all three historical sources (S3 has only two historical sources and replays only 99 examples). Selection cost is counted as forwards over the 13,944 historical examples at S4; Similarity additionally embeds the 8,000 current-task prompts:

SelectorS4 selection costGSM8KKodCodeGPQAAQuAMacroCausal − selector
Loss-high13,944 forwards54.38 ± 0.0954.60 ± 0.1523.91 ± 0.5843.04 ± 0.2343.98 ± 0.12+1.90 [0.12, 3.68]
MIR-offline27,888 forwards + one virtual update54.44 ± 0.2754.48 ± 0.0824.41 ± 0.7743.18 ± 0.6044.13 ± 0.35+1.76 [0.84, 2.67]
Similarity21,944 prompt-only forwards, frozen reasoner51.07 ± 0.3653.17 ± 0.2526.43 ± 0.5841.86 ± 0.6043.13 ± 0.19+2.75 [0.79, 4.71]
Causal replay, k=469,720 forwards, 5.36 GPU-h54.54 ± 0.5355.32 ± 0.5527.61 ± 1.0546.06 ± 1.0445.88 ± 0.71—

Causal replay is above all three selectors on every seed. Second, under the full-pool protocol, the supplementary seed-43 results are:

SelectorGSM8KKodCodeGPQAAQuAMacro
Full-pool Random53.1553.7527.2737.8042.99
Similarity51.0252.9028.7943.7044.10
Causal replay55.1956.0530.8146.0647.03

This merged-pool protocol, with approximately 4,000 examples, differs from the per-source-matched S4 comparison; within this protocol, the Macro gains are +2.93 pp over Similarity and +4.04 pp over Random (single seed). Diversity-based selection (GSS) requires per-example gradients and is not included in this round; in our method, set-level diversity is enforced by Pattern Cap, which the ablation in Table 3 shows contributes a +0.78 pp gain. We agree that EWC and LwF are not replay-selection methods: the revision explicitly positions them as classical continual-learning references in the caption of Table 6, and the direct comparison with replay selectors is given by the new tables above.

AuthorsOpenReview ↗

Response to Reviewer WepN (2/5) — Q2–Q3: Robustness and Compute

Q2. Multi-seed order and backbone checks

Both settings now use seeds 42–44. In the tables below, values are mean ± sample SD; differences are given with two-sided 95% paired-t intervals, n=3, df=2.

Permuted order: GPQA → KodCode → GSM8K → AQuA.

TaskNo replayRandomCausalPaired gain over Random
GSM8K10.66 ± 0.0450.67 ± 0.0951.58 ± 0.12+0.91 [0.41, 1.41]
KodCode10.15 ± 0.1549.43 ± 0.1351.00 ± 0.18+1.57 [1.38, 1.76]
GPQA29.29 ± 0.5127.27 ± 0.0026.94 ± 0.77−0.34 [−2.25, 1.58]
AQuA33.33 ± 0.9937.93 ± 1.4943.57 ± 0.45+5.64 [2.21, 9.08]
Macro20.86 ± 0.3741.33 ± 0.3943.27 ± 0.09+1.95 [0.80, 3.10]

Qwen3-4B, main order.

TaskNo replayRandomCausalPaired gain over Random
GSM8K63.41 ± 0.3463.74 ± 0.1265.73 ± 0.08+2.00 [1.52, 2.47]
KodCode17.18 ± 0.2553.82 ± 0.1863.30 ± 0.20+9.48 [8.55, 10.42]
GPQA26.60 ± 1.0527.44 ± 1.9128.62 ± 0.77+1.18 [−5.01, 7.37]
AQuA63.78 ± 0.7958.53 ± 0.9964.70 ± 1.14+6.17 [1.15, 11.19]
Macro42.74 ± 0.0950.88 ± 0.6755.59 ± 0.26+4.71 [2.42, 6.99]

The seed-wise Macro gains in the two settings are 1.62/2.47/1.75 and 4.38/4.00/5.75, respectively, positive on every seed. In both settings, the paired intervals for GSM8K, KodCode, and AQuA exclude zero; GPQA is the only task whose interval includes zero. The single-seed result on the larger Qwen3.5-9B backbone (Macro +2.45) is reported in our response to sULK Q1.

Q3. Cost accounting, Random rescoring, and restricted pools

On the 2×H800 seed-43 chain, S2/S3/S4 scoring takes 2.46/5.24/5.36 GPU-hours and processes 7.27M/20.21M/20.73M tokens; the subsequent training takes 2.14/0.07/2.39 GPU-hours. Scoring totals 171,955 forwards and 13.06 GPU-hours; training adds 4.60. Scoring is gradient-free, with batch size one and no key–value cache reused across passes. The stored records contain candidate identifiers, losses/effects, and signed patterns rather than hidden-state tensors. Scoring therefore needs less GPU memory than training, its records are all scalars, and selection is a single CPU pass over them; we report GPU-hours as the primary cost.

Random does not re-score. Q-matched Random receives Full's actual per-source counts; Proportional Random receives the same total historical count, allocated by source size. Both sample uniformly within sources. We have corrected "each branch re-scores" at line 364 of the submitted version: scoring applies only to Full and to the intervention-based ablations in Table 3 (gate only, w/o harm, gate + rank), which diverge after S2 and therefore score at their own checkpoints. Deployment requires scoring only along Full's chain, i.e., the 13.06 GPU-hours above; re-scoring for the ablation branches is an experimental cost and is not counted as part of the method's cost.

At matched total compute, more Random epochs do not close the gap. Eight-epoch Random takes 18.40 GPU-hours, 0.74 more than Causal k=4, and reaches 43.48 ± 0.40 Macro; Causal k=4 takes 17.66 and reaches 45.88 ± 0.71, a paired gain of +2.40 [0.70, 4.10] pp. On token matching: eight-epoch Random trains on four times the tokens of two-epoch Random without improving Macro (43.48 vs 43.69), so the gap does not stem from the total number of training tokens.

Restricted scoring retains the selection advantage. Scoring only half/one third of S4 history lowers the total cost to 14.98/14.09 GPU-hours; Causal then reaches 42.51/39.20 Macro versus 39.88/36.92 for matched Random (seed 43), a relative gain of +2.63/+2.28 pp (+2.68 with full scoring); the absolute Macro of both methods decreases similarly as the replay set shrinks. For equal-size tasks, the cumulative cost of full re-scoring grows quadratically with the number of tasks; scoring only a fixed M candidates per stage gives (k+1)M(T−1) cumulative forwards, which grows linearly with the number of tasks. These experiments support that the selection advantage is retained when only a subset of history is scored; combined with this analytical result, restricted scoring offers a promising, cost-controlled path toward longer task streams.

AuthorsOpenReview ↗

Response to Reviewer WepN (3/5) — Q4–Q6: Intervention Granularity, k, and Baseline

Q4. Individual slots versus an event-level block

A block-level intervention can enforce its own block-level harm constraint. Our independent-slot rule instead checks each nominated event-slot individually, so positive and negative effects within the same event can be detected separately. The diagnostic in Section 3 shows that repair and damage rates differ across slots of the same event (Figure 2a,b), so a block-level result can aggregate opposite effects.

Removing harm avoidance in Table 3 lowers Macro by 1.09 pp, showing that the harm constraint contributes to downstream replay; block-level masking cannot apply this constraint slot by slot. A comparable-cost replay control between block-level and slot-level rules is not yet complete, so we have restricted the wording in Section 4.3 to the difference between the constraints the two rules can enforce.

Q5. Final replay sensitivity to the candidate budget k

The additional end-to-end results show an accuracy–cost tradeoff. Each historical example requires one baseline forward plus k intervention forwards:

kForwards/exampleScoring GPU-hTotal GPU-hMacro, seeds 42–44
237.8412.4444.54 ± 0.53
4, default513.0617.6645.88 ± 0.71
8923.5128.1147.16 ± 0.52

Increasing k from 2 to 4 costs 5.22 additional GPU-hours and raises Macro by 1.34 points; increasing it from 4 to 8 costs 10.45 additional GPU-hours and raises Macro by 1.28 points. This corresponds to roughly 0.26 and 0.12 points per additional GPU-hour. k=4 is a compromise operating point; k=8 is better when compute permits. All three values exceed every tested Random configuration (at most 43.84).

Admission rates (%, admitted / scored, seed 43; G/K/P = GSM8K/KodCode/GPQA; ≥ marks a count truncated at the per-source cap):

kS2, GS3, G / KS4, G / K / P
225.10.77 / 0.73≥29.7 / 26.5 / 52.5
4, default25.30.79 / 0.77≥29.7 / 26.2 / 53.4
825.80.79 / ≥0.80≥29.7 / 27.1 / 55.2

Admission rates change little with k. S2 shares the Stage-1 checkpoint across k, so its rise from 25.1% to 25.8% reflects k alone; at S3 and S4 the incoming checkpoints also depend on earlier replay.

How k was chosen. We need to correct one statement in Section 5.1: k ∈ {2, 4, 8} and the remaining selector settings were not chosen on a separate held-in validation split, but during development by weighing scoring cost against Macro, and the Figure 1 diagnostic also uses test questions. All three k values in the table exceed every Random configuration, so the conclusion does not depend on the specific k chosen. Once chosen, the settings were fixed across stages, seeds, and branches and reused unchanged, without retuning, for the permuted order, Qwen3-4B, and Qwen3.5-9B, where the gains are +1.95 [0.80, 3.10], +4.71 [2.42, 6.99], and +2.45 (single seed), respectively.

Q6. What exactly does zero replacement intervene on?

The intervention replaces content relative to a specified baseline; it does not delete the slot. The implementation is _intervene in memgen/model/modeling_memgen.py. Specifically:

  1. Projection: the contextualized Weaver state is zeroed before the biased Weaver→reasoner projection, so the reasoner receives the projection bias at that position.
  2. Mask and position: attention masks and positions are unchanged; the slot remains present and can still be attended to.
  3. Cache: every scoring forward is an independent, full teacher-forced pass over the whole sequence (score_example in replay/scoring.py), and no key–value cache is carried across passes. Effects may propagate through later injection events.
  4. Targets: logits at latent positions are excluded, and the losses before and after intervention use the same reference-completion tokens and labels.

We have replaced "removing the candidate" at lines 287–289 of the submitted version with "zeroing the candidate" and added:

Helpfulness and harm are both relative to this zero-replacement reference: the reasoner still receives the projection bias, and positions and attention masks are unchanged.

The intervention is used only for admission: the replayed example is the original, unmasked one, and the rule's value is tested by downstream replay results (Tables 2 and 3). Mean replacement and graded attenuation have not yet been compared as alternative references; following the revised text above, helpfulness and harm are explicitly defined relative to the zero-replacement reference.

Reviewer_sULKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts a diagnose study for Weaver latent vectors, which is MemGen-style latent memory for self-evolving agents. The Weaver is fine-tuned on for datasets in sequence (GSM8K -> KodCode -> GPQA -> AQuA8000) starting from the previous checkpoint of each stage. And the study finds static saliency signals can nominate candidates but can-not independently provide the bilateral helpful/non-harmful evidence required for replay admission; a slot’s intervention effect also depends on its specific injection event. So the paper proposed "Causal Experience Selection". At each stage's incoming checkpoint, it uses static signals to narrow the event-slot candidate setn and independently masks each candidate at the incoming checkpoint and determines sample admission from bilateral techer-forcing evidenc before constructing a fixed budget replay. The experiment is done with Qwen 2.5-1.5B-Instruct and a MemGen Weaver and cross-validated with task-order permutation and Qwen3-4B. The Qwen 2.5-1.5B Instruct shows final four-task mean accuracy (Macro) from No replay’s 25.41 ±0.38 to 45.88 ±0.71, indicating that replay mitigates forgetting in the tested MemGen setting.

Critical strengths

1, the comparison protocol is clearly controlled, e.g. shared Stage-1 checkpoint, identical per-source replay counts, identical example counts and epoches, which mitigates the typical confounding impact from selection with composition. 2, the diagnose study is informative. The figures (1, 2) show that masking a single event-slot can either repair or damage a greedy answer.

Critical weaknesses

1, major results are supported by Qwen 2.5-1.5B-Instruct, while Qwen3-4B shows consistent result but less significant, the phenomenon may not exist at scale. 2, not significant concern but no detailed training hyperparameters (learning rate, optimizer, batch size, etc), no code and reproducibility statement.

Questions for authors

1, run the main protocol on a some latest generation backbone LLMs to show scaling capability. 2, please share more details about hyperparameters including Weaver's parameter count, optimizer, LR, etc

4 comments
AuthorsOpenReview ↗

Response to Reviewer sULK — Backbone Robustness and Reproducibility

Thank you for recognizing the controlled protocol and the diagnostic evidence. Below, we address the uncertainty left by the original single-run backbone check and provide the main configuration.

Q1. Multi-seed evidence on Qwen3-4B and a Qwen3.5-9B check

The Qwen3-4B results now cover seeds 42–44. Causal replay reaches 55.59 ± 0.26 Macro versus 50.88 ± 0.67 for quantity-matched Random. The paired gain is +4.71 [2.42, 6.99] pp, with seed-wise gains of 4.38 / 4.00 / 5.75.

TaskRandomCausal replayPaired gain [95% interval]
GSM8K63.74 ± 0.1265.73 ± 0.08+2.00 [1.52, 2.47]
KodCode53.82 ± 0.1863.30 ± 0.20+9.48 [8.55, 10.42]
GPQA27.44 ± 1.9128.62 ± 0.77+1.18 [−5.01, 7.37]
AQuA58.53 ± 0.9964.70 ± 1.14+6.17 [1.15, 11.19]

Values are accuracy (%), mean ± sample SD; intervals are two-sided paired-t intervals computed per contrast (n=3, df=2). The GPQA test set has only 198 questions, so one question corresponds to 0.51 pp; its gain therefore varies considerably across seeds and its interval is wide. Overall, the results on this backbone provide positive evidence that Causal replay outperforms Random.

We also ran the main protocol on the newer and larger Qwen3.5-9B. Owing to time constraints during the discussion period, only seed 43 is complete so far, so we report no interval:

Qwen3.5-9B, main order, seed 43GSM8KKodCodeGPQAAQuAMacro
No replay74.8317.2541.9266.1450.04
Q-matched Random81.6578.6539.9067.7266.98
Causal replay85.5280.9543.9467.3269.43
Causal − Random+3.87+2.30+4.04−0.39+2.45

The 9B run reuses all 1.5B hyperparameters without retuning. The Macro gains on 1.5B, 4B, and 9B are +2.19, +4.71, and +2.45, respectively, showing that the gain persists on the larger backbone in this single-seed run. The remaining two 9B seeds will be added once complete.

The task-order permutation also provides results over three paired seeds, with a Macro gain of +1.95 [0.80, 3.10] pp that is positive on every seed (1.62 / 2.47 / 1.75); the Macro summary is in General Response §3, and per-task results are in our response to WepN Q2.

Q2. Training configuration and trainable parameter count

The main Qwen2.5-1.5B configuration is as follows:

ComponentSetting
ReasonerQwen2.5-1.5B-Instruct, frozen (≈1.54B)
WeaverCopy of Qwen2.5-1.5B-Instruct; LoRA on q_proj, v_proj
LoRARank 16, alpha 32, dropout 0.1
Query latents8 prompt latents + 8 inference latents, dimension 1,536; LayerNorm and scalar scale
ProjectionsTwo Linear(1536, 1536) layers with bias
Trainable parameters6,931,458 in total; approximately LoRA 2.18M, projections 4.72M, latents 0.025M, LN/scales 0.006M
InjectionPrompt boundary plus up to five continuation delimiters (,, ., \n); at most 6 × 8 event-slots
OptimizerAdamW; learning rate 1e-5; cosine schedule; warmup ratio 0.1
Effective batch2 per GPU × 16 gradient accumulation × 2 GPUs = 64
Training2 epochs per stage; length 1,024; loss on assistant tokens only
Runtimebf16; DeepSpeed ZeRO-2; 2×H800 80GB
EvaluationGreedy decoding (temperature 0); at most 1,024 new tokens
Selectork=4; helpful/harm thresholds 0.02; epsilon 1e-6; Pattern Cap 32
Budgets and seedsReplay cap 0.5N_t; per-source cap 0.25N_t (with the specified rounding); seeds 42–44; split seed 42

This trainable parameter count is about 0.224% of the 3.09B-parameter combined system. It describes the 1.5B configuration. The task-order permutation, Qwen3-4B, and Qwen3.5-9B reuse the same training and selection hyperparameters without retuning.

Anonymous implementation: code repository. The repository contains the training, scoring, selection, and evaluation scripts and all configuration files; the README maps each part of the paper to the code and gives the full run commands.

In the revision, we have further consolidated the configuration, data construction, and reproducibility statement.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer sULK,

Thank you again for your helpful feedback. In our response titled “Backbone Robustness and Reproducibility,” we have added three-seed Qwen3-4B results, a single-seed Qwen3.5-9B experiment, detailed training configurations, and an anonymous code repository.

If your schedule permits, we would greatly appreciate your thoughts on these additions and any remaining concerns about backbone robustness or reproducibility. We would be happy to provide further clarification.

Thank you for your time and consideration. The Authors

Thank you for the follow up studies, I agree it shows the scalability. I increase my score to a positive score 3.

AuthorsOpenReview ↗

Appreciation for Raising the Score and Your Constructive Review

Dear Reviewer sULK,

Thank you very much for acknowledging our follow-up scalability studies and for raising your rating to a positive score! We truly appreciate your constructive comments throughout the review process, which have greatly improved the rigor and completeness of our work.

Reviewer_Z1SJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates experience replay selection for latent-memory agents in a continual learning setting. Building on the MemGen framework, the authors observe that static latent-slot importance measures cannot reliably distinguish helpful from potentially harmful memory contributions, and that intervention effects depend on specific injection events.

To address this issue, the paper proposes Causal Experience Selection, which first nominates candidate event-slot pairs based on latent norms and then evaluates their contributions through individual masking interventions. Historical examples are admitted for replay based on bilateral helpfulness and non-harmfulness criteria, followed by constrained sampling under source quotas, pattern deduplication, and a fixed replay budget.

Experiments on four sequential tasks using Qwen2.5-1.5B-Instruct show that the proposed method improves final Macro accuracy from 25.41% without replay to 45.88%. Compared with quantity-matched Random replay, the method achieves an average improvement of 2.19 percentage points across three seeds. Additional ablations and robustness experiments provide supporting evidence for the proposed design.

Critical strengths

1. Interesting motivation and a well-structured intervention-based selection mechanism.

The paper identifies an interesting limitation of static saliency measures in latent-memory systems: highly influential latent slots are not necessarily beneficial, and their effects can depend on the injection event. The diagnostic experiments provide useful motivation for the proposed event-conditioned intervention approach. The resulting selection pipeline is reasonably well designed and connects these observations to a practical replay admission strategy.

2. Controlled experiments provide evidence of improvements over Random replay.

The authors carefully control the actual per-source replay counts and training-example counts when comparing against Random replay, which helps isolate the contribution of sample selection. The proposed method consistently outperforms the quantity-matched Random baseline across three paired seeds, achieving an average improvement of 2.19 percentage points. The component ablations and additional robustness checks further support the effectiveness of the overall approach within the evaluated setting.

Critical weaknesses

1. Insufficient positioning within the extensive continual learning literature.

Experience replay and replay sample selection have been extensively studied in continual learning. Prior methods, such as Maximally Interfered Retrieval (MIR), Gradient-based Sample Selection (GSS), and Adversarial Shapley Value Experience Replay (ASER), have explored selecting historical samples based on interference, gradient information, and estimated sample utility.

Although the proposed event-slot intervention mechanism is interesting, the paper does not sufficiently clarify its conceptual relationship to these established approaches or explain what fundamentally new selection principle it introduces beyond existing replay-selection strategies. Moreover, the main experiments primarily compare against quantity-matched Random replay, while competitive replay-selection methods are not evaluated under the same controlled protocol. This makes it difficult to assess the incremental novelty and significance of the proposed approach within the broader continual learning literature.

2. Limited performance gains relative to the additional scoring overhead.

While the proposed method improves final Macro accuracy by 2.19 percentage points over quantity-matched Random replay, it introduces nontrivial computational overhead. Specifically, the method requires five gradient-free forward passes per historical example at each incoming checkpoint, whereas Random replay incurs negligible selection overhead.

Furthermore, the method re-scores the historical pool at each new stage, potentially making the computational cost increasingly substantial as the number of tasks and stored experiences grows. Although the paper carefully matches replay-example counts, it does not report the actual scoring cost or provide an end-to-end compute-matched comparison. Consequently, it remains unclear whether the relatively modest accuracy improvement justifies the additional computation, particularly in longer continual learning scenarios.

Questions for authors

Q1. Positioning and novelty relative to existing continual learning methods.

Experience replay selection has a long history in continual learning, including methods based on interference, gradient diversity, and sample utility (e.g., MIR, GSS, and ASER). Could the authors provide a more detailed discussion of how the proposed method relates to these existing approaches, and clarify what new insights or advantages event-slot-level interventions provide beyond established replay-selection principles?

In particular, it would be useful to position the method within the broader continual learning literature, including settings where the full historical dataset is available but the replay budget is limited. If feasible, comparisons with representative continual-learning replay selectors under the same replay budget and source composition would further help establish the method's incremental contribution.

Q2. Computational overhead and scalability.

The proposed method achieves an average improvement of 2.19 percentage points over quantity-matched Random replay, but requires five forward passes per historical example at each incoming checkpoint.

Could the authors report the actual computational overhead of the selection procedure (e.g., GPU-hours, wall-clock time, or FLOPs), and compare the total computational cost, including both selection and training, against Random replay? It would also be helpful to understand whether the performance advantage persists under a matched end-to-end computational budget.

Furthermore, since the entire historical pool is re-scored at each new stage, the cumulative selection cost may grow substantially as the replay database expands. Could the authors analyze how this cost scales with the number of tasks and historical examples, and discuss whether the additional performance gain remains worthwhile in longer continual learning sequences?

Clarifications and additional evidence addressing these two concerns would materially affect my overall assessment of the paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Z1SJ (1/3) — Positioning Relative to Replay-Selection Methods

Thank you for recognizing the intervention-based motivation and the controlled main comparison. We agree that the key question is the incremental selection evidence relative to established replay methods, not merely replay versus no replay.

Q1. Relationship to existing replay-selection principles

Our criterion tests the direction of an example's dependence on its own internal memory at the incoming checkpoint. Static norm nominates event-slot candidates; independent interventions measure the resulting changes in reference-completion loss. An example is admitted only when at least one tested candidate provides helpful evidence and no candidate exceeds the harm threshold. The original, unmodified example is then replayed.

This differs from the evidence used by MIR, GSS, and ASER:

MethodSelection evidence
MIRInterference predicted from a prospective parameter update.
GSSGradient diversity for selecting representative replay constraints.
ASERShapley-based sample utility for stability and plasticity.
Causal replayLoss effects of interventions on the example's own event-slot states at the current checkpoint.

We do not claim that considering both positive and negative utility is new; ASER already accounts for adversarial and cooperative contributions. The distinctive evidence here is internal, event-conditioned, and obtained before any prospective parameter update. It is a local activation-intervention effect, not an estimate of the causal effect of replay on future-task performance.

Admission is also separated from ranking. The helpfulness score determines eligibility; eligible examples are then sampled under source, pattern, and total-budget constraints. In submitted Table 3, removing harm avoidance lowers Macro by 1.09 pp, and replacing the non-ranking scan with helpfulness ranking lowers it by 0.46 pp. These are ablation point estimates that support the tested design, not general claims against ranking or other selectors.

AuthorsOpenReview ↗

Response to Reviewer Z1SJ (2/3) — Direct Comparisons

Direct comparison under matched per-source counts

Following your request to fix the budget and source composition, we compare several selectors at S4 on seeds 42–44. All rows start from Full's S3 checkpoint of the same seed, match Full's per-source replay counts on that seed (seed 43: (1,765, 1,651, 104)), and use the same training-set size and training recipe; only the within-source choice differs. The offline adaptations are described in a new appendix.

We place the substitution at S4 for three reasons. First, S4 is the final stage, so the selector's effect is measured directly on the reported final results. Substituting at an earlier stage would require the later stages to inherit either the baseline's or Causal replay's selections, mixing the effect of a single stage's choice with divergence at later stages. Second, S4 is the only stage whose history covers all three prior sources and whose replay set is large (3,520 of 8,000 training examples are replayed). Third, it reuses the final-stage protocol of the paper's replay-scale and full-pool checks, which also start from Full's incoming checkpoint at this stage.

SelectorSelection evidence (offline adaptation)S4 selection forwardsGSM8KKodCodeGPQAAQuAMacroCausal − selector
Loss-highTeacher-forced loss at the current checkpoint, highest per source13,94454.38 ± 0.0954.60 ± 0.1523.91 ± 0.5843.04 ± 0.2343.98 ± 0.12+1.90 [0.12, 3.68]
MIR-offlineLoss increase after one virtual update on a random current-task batch27,888 + one virtual update54.44 ± 0.2754.48 ± 0.0824.41 ± 0.7743.18 ± 0.6044.13 ± 0.35+1.76 [0.84, 2.67]
SimilarityCosine similarity between frozen-reasoner prompt embeddings and the current-task centroid, highest per source21,944 (prompt only)51.07 ± 0.3653.17 ± 0.2526.43 ± 0.5841.86 ± 0.6043.13 ± 0.19+2.75 [0.79, 4.71]
Causal replay, k=4Bilateral intervention gate69,720 (5.36 GPU-h)54.54 ± 0.5355.32 ± 0.5527.61 ± 1.0546.06 ± 1.0445.88 ± 0.71—
Q-matched Random†None053.53 ± 0.4554.50 ± 0.4624.75 ± 0.5141.99 ± 0.6043.69 ± 0.50—

Mean ± sample SD over seeds 42–44; differences are two-sided 95% paired-t intervals (df=2). The Causal row is Full's final result on each seed. † From its own S2–S4 chain rather than Full's S3 checkpoint; shown for reference only.

Causal replay is above every selector on every seed (all nine paired Macro differences are positive). Against MIR-offline and Loss-high, the gains come mainly from GPQA and the current task AQuA; against Similarity, they come mainly from GSM8K, KodCode, and AQuA.

MIR selects the historical examples that the next update would hurt most, which protects earlier tasks; its criterion does not consider whether the replayed examples themselves hinder learning of the current task. Our results also show a difference on this side. With identical current-task counts (4,480 examples on 1.5B) and starting from the same S3 checkpoint, Causal replay is +2.89 [1.39, 4.38] pp above MIR-offline on the current task AQuA. Under the main protocol (Random from its own S2–S4 chain), Causal replay is also 4.07 pp (1.5B) and 6.17 pp (Qwen3-4B) above quantity-matched Random on AQuA.

Direct comparison under the full-pool protocol

We add the missing Causal replay result to the supplementary merged-history protocol of Table 7:

SelectorFinal Macro, seed 43
Norm-proxy40.28
TF-delta41.63
Full-pool Random42.99
Similarity44.10
Causal replay47.03

In this run, Causal replay is 2.93 pp above Similarity and 4.04 pp above full-pool Random. This protocol replays approximately 4,000 examples from the merged history, and its source composition differs from that of the main experiment; it provides complementary evidence for the advantage of Causal replay under a different replay strategy.

To be candid, the direct comparison under matched per-source counts currently covers only S4, and GSS and ASER have not been ported. Their standard formulations target online buffers and differ from our stage-boundary setting, but this calls for careful adaptation rather than making the comparison unnecessary. We have therefore narrowed our empirical claims to the comparisons actually available. The EWC and LwF baselines already in the paper remain classical continual-learning references and do not substitute for replay-selector baselines.

AuthorsOpenReview ↗

Response to Reviewer Z1SJ (3/3) — Compute and Scalability

Q2. Measured scoring overhead

We now report explicitly the selector cost you raised. On the seed-43 chain using 2×H800, k=4 requires one baseline forward plus four intervention forwards per historical example. Stage 1 is shared by all methods and is excluded below.

StageHistorical examplesScoring tokensScoring GPU-hTraining GPU-h
S26,7267.27M2.462.14
S313,72120.21M5.240.07
S413,94420.73M5.362.39
Total—48.21M13.064.60

This amounts to 171,955 gradient-free scoring forwards. S3 has only 223 current-task examples, which explains why scoring dominates its very short training stage. The total scoring-plus-training cost is 17.66 GPU-hours, about 3.84× that of two-epoch Random. Throughout the pipeline, scoring consists of gradient-free forwards and needs far less GPU memory than training the same model; each scored example keeps only k+1 losses, candidate identifiers, and a signed pattern, without storing hidden states; and selection is a single CPU pass over these records. Memory and storage therefore impose no requirement beyond what training itself needs, and we use GPU-hours as the primary cost measure.

Does giving Random more training close the gap?

Not within the tested 2–8 epoch configurations. The additional comparison is as follows:

MethodTotal GPU-hMacro, mean ± SD
Q-matched Random, 2 epochs4.6043.69 ± 0.50
Q-matched Random, 4 epochs9.2043.84 ± 0.22
Q-matched Random, 8 epochs18.4043.48 ± 0.40
Causal, k=212.4444.54 ± 0.53
Causal, k=417.6645.88 ± 0.71
Causal, k=828.1147.16 ± 0.52

Macro statistics use seeds 42, 43, and 44. The paired gain of k=4 over eight-epoch Random is +2.40 [0.70, 4.10] pp (95% paired-t interval, df=2). Eight-epoch Random uses 0.74 GPU-hours more than Causal k=4, so this comparison is matched in total compute. k=2 offers a lower-cost option: it is numerically 1.06 pp above eight-epoch Random at roughly 32% lower total cost.

Longer histories and restricted scoring pools

For T tasks of n examples each, full re-scoring requires (k+1)nT(T−1)/2 forwards, so the cumulative cost grows quadratically with the number of tasks. If only a fixed M historical candidates are scored per stage, cumulative scoring becomes (k+1)M(T−1) forwards, growing linearly with the number of tasks. We note that this linear growth is an analytical result; we have not yet measured fixed-M scoring across multiple stages. The experiment below scores different fractions of history at S4 only, testing whether the selection advantage is retained when only a subset of history is scored (the one-third fraction is one instance with M ≈ 4,648).

The available reduced-pool experiment scores a random fraction of S4 history. Replay counts change as the admitted pool shrinks, and at each fraction Random matches the resulting per-source replay counts:

S4 scoring fractionS4 scoring forwardsCausal total GPU-hReplay counts, G/K/PRandom MacroCausal MacroDifference, pp
Full69,72017.66(1,765, 1,651, 104)43.7546.43+2.68
Half34,86014.98(930, 861, 58)39.8842.51+2.63
One third23,24014.09(628, 594, 46)36.9239.20+2.28

At both scoring fractions, Causal replay largely keeps its advantage over matched Random (+2.63 / +2.28, versus +2.68 with full scoring); the drop in absolute Macro comes from the smaller replay set and is similar for both methods.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2) — Direct Selector Comparisons and Compute Accounting

We thank the reviewers for recognizing the diagnostic insights of this work and for raising many constructive questions and suggestions. Our responses focus on the additional value of event-slot evidence, its computational cost, and its robustness in the tested settings, and aim to address the reviewers' concerns as fully as possible.

A note on the number of seeds. Owing to compute and time constraints during the discussion period, some supplementary experiments are currently complete for a single seed (seed 43) only: the full-pool protocol comparison, restricted scoring, the measured GPU-hour costs, and the Qwen3.5-9B backbone. We apologize for any inconvenience this may cause; as compute permits, we will add the remaining seeds for these experiments in the revision.

1. Direct selector comparisons — Z1SJ and WepN

Same per-source counts at S4. So that the comparison reflects only differences in the selection criterion itself, we set up an additional experiment: all selectors start from Full's S3 checkpoint of the same seed, match Full's per-source replay counts on that seed (seed 43: (1,765, 1,651, 104)), and share the training-set size and training recipe. The only difference is which examples are replayed, so the selector's effect appears directly in the reported final results. Over seeds 42–44, Causal replay reaches 45.88 ± 0.71 Macro, versus 44.13 ± 0.35 for an offline MIR-style selector, 43.98 ± 0.12 for highest-loss selection, and 43.13 ± 0.19 for Similarity. The paired gains are +1.76 [0.84, 2.67], +1.90 [0.12, 3.68], and +2.75 [0.79, 4.71] pp, respectively, positive on every seed.

Full-pool protocol. We add Causal replay to the merged-history protocol of submitted Table 7. On seed 43, its Macro is 47.03, compared with 44.10 for Similarity and 42.99 for full-pool Random. This protocol replays approximately 4,000 examples and differs from the main per-source-matched protocol; on this seed, it likewise shows the advantage of Causal replay.

2. Selection cost and additional-training controls — Z1SJ and WepN

Equal-compute comparison. For S2–S4, the measured seed-43 profile is 13.06 GPU-hours for scoring and 4.60 for training, so Causal k=4 costs 17.66 GPU-hours in total. We give quantity-matched Random the same compute budget by extending its training to eight epochs (18.40 GPU-hours, 0.74 more than Causal k=4). At matched compute, Causal replay reaches a three-seed Macro of 45.88 ± 0.71 versus 43.48 ± 0.40 for Random, a paired gain of +2.40 [0.70, 4.10] pp; extending training to eight epochs does not noticeably improve Random either (43.48 vs 43.69). The conclusion holds at a lower budget: Causal k=2 costs 12.44 GPU-hours, less than eight-epoch Random, and reaches 44.54 ± 0.53 Macro, above every tested Random configuration.

Restricted scoring. Scoring only a random half or one third of S4 history lowers Causal's total cost to 14.98 / 14.09 GPU-hours. With Random matched to the resulting replay counts, Causal replay remains ahead by +2.63 / +2.28 pp (seed 43; +2.68 with full scoring). Absolute Macro decreases by similar amounts for both methods (at one half, Causal 46.43 → 42.51, Random 43.75 → 39.88), indicating that the drop comes from the smaller replay set while the selection advantage itself is retained.

AuthorsOpenReview ↗

General Response (2/2) — Multi-seed Robustness, Forgetting, and Revisions

3. Multi-seed task-order and backbone checks — sULK and WepN

The additional results extend the two single-run checks in Table 4 to seeds 42–44:

SettingRandom MacroCausal MacroPaired gain [95% interval]
Qwen2.5-1.5B, main order43.69 ± 0.5045.88 ± 0.71+2.19 [1.13, 3.25]
Qwen2.5-1.5B, permuted order41.33 ± 0.3943.27 ± 0.09+1.95 [0.80, 3.10]
Qwen3-4B, main order50.88 ± 0.6755.59 ± 0.26+4.71 [2.42, 6.99]
Qwen3.5-9B, main order (seed 43 only)66.9869.43+2.45 (single seed, no interval)

Values are mean ± sample SD; intervals are two-sided paired-t intervals over three seeds with df=2. Within each three-seed setting, every seed-wise Macro difference is positive. The 9B run reuses the 1.5B hyperparameters and, owing to time constraints, covers only seed 43, so it is reported separately. The results point in the same direction across backbone sizes.

4. Forgetting — WepN

At identical replay counts, Causal replay reduces the mean BWT over historical tasks in the main order from −1.60 ± 0.42 for Random to −0.79 ± 0.42 (no replay: −26.49 ± 0.36), a paired difference of +0.81 [0.08, 1.53] pp that is positive on all three seeds, roughly halving forgetting; per-task results are given in our response to WepN Q7.

5. Revisions to the PDF

The supplementary results and the clarifications below have been incorporated into the revised PDF; changes are colored by reviewer and tagged with the question they address (e.g., "(WepN Q6)"), and the main text stays within the 10-page limit.

  • Abstract and §1: three-seed wording for the task-order and Qwen3-4B checks.
  • §2.2: relation to MIR, GSS, and ASER.
  • §4.1, §4.3, §4.5: N_t=|D_t|; the zero-replacement reference; block- versus slot-level constraint wording; cost pointer.
  • §5.1: how k and the other settings were chosen; only Full and the intervention ablations score; Table 1 column description.
  • §5.2–5.4: BWT; direct S4 selector comparison (Table 3B); three-seed Table 4; Qwen3.5-9B; selection cost.
  • §6 and Reproducibility Statement: narrowed claims, single-seed scope, anonymous code.
  • Appendix: intervention details (Appendix A); Table 6 caption (EWC/LwF as continual-learning references); Causal row in Table 7; new Appendices E–H (selector comparison; cost, k, and restricted scoring; robustness and forgetting; stage composition, data construction, and configuration).

Across selector comparisons, matched compute, restricted scoring, forgetting, task order, and backbone scale, the supplementary experiments support the same conclusion: example-specific, checkpoint-specific event-slot evidence yields gains beyond quantity-matched Random and a range of existing continual-learning selectors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.