- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces SchemaMem, a recurrent memory architecture combining chunk-local causal attention with a learned schema bank and an episode-specific phase state. Reads use the schema bank modified by the current state, whereas writes exclude direct feedback from the same layer’s state and accumulate additive phase updates committed at chunk boundaries. The model also supports full-history attention during training. Evaluation uses a synthetic address–value maintenance task covering retention, disjoint-write interference, and overwriting, with nine address/value settings and three training seeds. Compared with approximately parameter- and persistent-state-size-matched Mamba-3 and Gated DeltaNet models, SchemaMem achieves higher mean written-value retention at four times the maximum training delay. These gains coexist with substantially more optimization steps, weaker default-value prediction, and some losses on overwritten values at higher address loads. A state-zeroing ablation confirms that written-value retrieval depends on the dynamic state.
Critical strengths
1. The diagnostic identifies a meaningful distinction between task mastery and delayed retention. The baselines reach higher in-range accuracy targets, yet their written-value retrieval can deteriorate much more strongly at longer delays. For example, in A3V3, overwritten-value accuracy changes from 95.35/99.58/99.74% at delay 384 to 74.64/26.89/27.15% at delay 1,536 for SchemaMem/Mamba-3/Gated DeltaNet, respectively (Section 5). This is a substantial empirical effect and motivates evaluating state preservation separately from performance within the training range.
2. The evaluation exposes distinct memory failures and verifies dependence on recurrent state. The paper reports retained, updated, and default values separately, includes individual-seed outcomes, and retains unfavorable results. Clearing the dynamic state after every commit reduces written-value retrieval below 1% for all 27 SchemaMem checkpoints at both tested delays, while default predictions largely survive (Appendix C). This establishes that episode-specific information is conveyed through the dynamic state and reveals that the same state can impair predictions for untouched addresses.
Critical weaknesses
-
The architectural contribution is insufficiently isolated, because SchemaMem uses full-history training and substantially more optimization steps than the recurrently trained baselines. Without controlled training comparisons or ablations isolating the learned schema and phase representation, the observed gains cannot be clearly attributed to the proposed architectural choices.
-
Practical applicability remains insufficiently validated, as evaluation is confined to a small synthetic address–value task without downstream-task experiments or measured runtime, throughput, and peak memory. Parameter and persistent-state counts alone do not establish actual resource costs, leaving it unclear whether the retention gains translate into useful improvements at an acceptable cost.
Questions for authors
-
Section 4 states that evaluation reuses the validation generator seed. Does this reproduce the same in-range episodes used for the stopping criterion? Please clarify the overlap and its implications for interpreting the reported evaluation.
-
Under the same SchemaMem training schedule and budget, does a learned schema bank outperform a fixed bank? Such a controlled comparison would directly strengthen the evidence for the central schema-based contribution.
-
What are the measured training time, inference throughput or latency, and peak memory usage of the three models under comparable hardware, precision, batch size, and sequence lengths? These measurements would make the retention–resource trade-off assessable.
-
Can the retention advantage improve a downstream task requiring both preservation and revision of information, with a common backbone and controlled resource budget? Evidence of improved final task performance, including the consequences of errors on untouched information, would materially strengthen the paper’s significance.