acceptodds
Under review as a conference paper at ICLR 2027

SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

Abstract

Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected information while processing subsequent inputs. We introduce SchemaMem, an attention-based recurrent memory architecture combining chunk-local attention with a persistent, schema-indexed phase state. Learned schema embeddings provide a shared representational reference for reading and writing. Reads use the current state, whereas writes use the layer input and static schema embeddings, excluding direct feedback from that layer's own state. Chunk-boundary commits aggregate bounded phase increments through forward computation. The same parameters also support full-history attention training before and during recurrent training. We studied selective updates, preservation, and delayed retrieval in a controlled address–value task, comparing three-layer models with approximately matched parameter counts and persistent-state dimensions. Across nine address/value settings and three training seeds, SchemaMem has higher mean written-value retention at four times the maximum training delay than both baselines, which are trained toward a higher in-range accuracy target. Updated-value recovery favors SchemaMem in all nine settings against Mamba-3 and seven against Gated DeltaNet. Defaults consistently favor Gated DeltaNet over SchemaMem at that delay, and SchemaMem requires substantially more optimization steps. These results identify a promising retention–optimization trade-off in schema-indexed recurrence.

Primary area:
learning on time series and dynamical systems
Keywords:
Recurrent memory, attention mechanisms, memory architectures, state maintenance, associative memory, memory interference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KJYKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SchemaMem, a recurrent memory architecture combining chunk-local causal attention with a learned schema bank and an episode-specific phase state. Reads use the schema bank modified by the current state, whereas writes exclude direct feedback from the same layer’s state and accumulate additive phase updates committed at chunk boundaries. The model also supports full-history attention during training. Evaluation uses a synthetic address–value maintenance task covering retention, disjoint-write interference, and overwriting, with nine address/value settings and three training seeds. Compared with approximately parameter- and persistent-state-size-matched Mamba-3 and Gated DeltaNet models, SchemaMem achieves higher mean written-value retention at four times the maximum training delay. These gains coexist with substantially more optimization steps, weaker default-value prediction, and some losses on overwritten values at higher address loads. A state-zeroing ablation confirms that written-value retrieval depends on the dynamic state.

Critical strengths

1. The diagnostic identifies a meaningful distinction between task mastery and delayed retention. The baselines reach higher in-range accuracy targets, yet their written-value retrieval can deteriorate much more strongly at longer delays. For example, in A3V3, overwritten-value accuracy changes from 95.35/99.58/99.74% at delay 384 to 74.64/26.89/27.15% at delay 1,536 for SchemaMem/Mamba-3/Gated DeltaNet, respectively (Section 5). This is a substantial empirical effect and motivates evaluating state preservation separately from performance within the training range.

2. The evaluation exposes distinct memory failures and verifies dependence on recurrent state. The paper reports retained, updated, and default values separately, includes individual-seed outcomes, and retains unfavorable results. Clearing the dynamic state after every commit reduces written-value retrieval below 1% for all 27 SchemaMem checkpoints at both tested delays, while default predictions largely survive (Appendix C). This establishes that episode-specific information is conveyed through the dynamic state and reveals that the same state can impair predictions for untouched addresses.

Critical weaknesses
  1. The architectural contribution is insufficiently isolated, because SchemaMem uses full-history training and substantially more optimization steps than the recurrently trained baselines. Without controlled training comparisons or ablations isolating the learned schema and phase representation, the observed gains cannot be clearly attributed to the proposed architectural choices.

  2. Practical applicability remains insufficiently validated, as evaluation is confined to a small synthetic address–value task without downstream-task experiments or measured runtime, throughput, and peak memory. Parameter and persistent-state counts alone do not establish actual resource costs, leaving it unclear whether the retention gains translate into useful improvements at an acceptable cost.

Questions for authors
  1. Section 4 states that evaluation reuses the validation generator seed. Does this reproduce the same in-range episodes used for the stopping criterion? Please clarify the overlap and its implications for interpreting the reported evaluation.

  2. Under the same SchemaMem training schedule and budget, does a learned schema bank outperform a fixed bank? Such a controlled comparison would directly strengthen the evidence for the central schema-based contribution.

  3. What are the measured training time, inference throughput or latency, and peak memory usage of the three models under comparable hardware, precision, batch size, and sequence lengths? These measurements would make the retention–resource trade-off assessable.

  4. Can the retention advantage improve a downstream task requiring both preservation and revision of information, with a common backbone and controlled resource budget? Evidence of improved final task performance, including the consequences of errors on untouched information, would materially strengthen the paper’s significance.

Reviewer_7AGUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SchemaMem combines attention within each input chunk with a recurrent memory that carries information across chunks. It reads from the current memory state, but computes writes using learned schema vectors without direct feedback from the same layer’s state. These writes take effect at chunk boundaries. On synthetic address–value tasks, SchemaMem retains written values better than the tested Mamba-3 and Gated DeltaNet models at delays longer than those seen during training. However, it requires many more training steps and is less accurate on addresses that retain their default values.

Critical strengths
  1. High accuracy within the training range does not guarantee long-delay retention. The paper reports results separately for retained, updated, and default values, revealing errors that an overall accuracy score would hide. At the longest evaluated delay, SchemaMem has higher mean written-value retention than both baselines across the reported address/value settings. The baselines generally predict updated values more accurately within the training range, making this reversal a useful finding.

  2. Useful controls and limitations are presented clearly. Resetting the dynamic state across the evaluated SchemaMem checkpoints confirms that the models use it to retrieve values written during an episode. The paper includes source code and per-seed results, and reports the high training cost, variation across seeds, poorer default-value predictions, and limits of the evaluation. These details make the results easier to assess.

Critical weaknesses
  1. The added value of the memory design remains unclear. Local attention, recurrent storage, and learned state identifiers already appear in [1]. SchemaMem adds phase states and static-schema writing, but provides no component comparisons establishing their benefit. For fixed inputs, writes ignore the same-layer state; history can still influence higher-layer writes through context and lower-layer memory. Unlike the corrections in Gated DeltaNet [2], this update leaves the role of obsolete associations unclear. Writing uses static keys, while read keys and values change with state; the benefit of this asymmetry is also untested. The gradient-stability argument rests on preliminary observations, and state resets establish memory use rather than the value of these choices. Differences in training procedure and optimization exposure further limit how confidently the retention gains can be attributed to these architectural choices.

  2. The update tests are limited, and better retention does not consistently improve overall accuracy. The protocol includes only a single overwrite phase, excludes repeated identical writes and restoration of defaults, and uses transaction-aligned commits. Delay inputs query only never-written addresses, leaving repeated revision and mixed read/write streams untested. The retention gain also comes with more errors on untouched addresses. Weighting the overwrite groups by their address proportions favors SchemaMem at a lower load but Gated DeltaNet at a higher load. The useful operating range of this trade-off remains unclear. These gaps in design validation, comparison, and revision coverage leave the methodological and empirical contribution insufficient for acceptance.

[1] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-Recurrent Transformers. NeurIPS, 2022.

[2] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.

Questions for authors
  1. What evidence shows that excluding direct same-layer state feedback improves optimization while preserving the ability to revise stored values?
  2. Does the retention advantage persist when training exposure is controlled, particularly against ungated DeltaNet or a closely related recurrent-attention model? This would help distinguish the memory design's contribution from training and gating effects.
  3. Can SchemaMem maintain accurate current values through ongoing writes and revisions, and how strongly does this ability depend on the supplied write/read boundaries and transaction-aligned commits?
  4. Could the authors clarify SchemaMem’s contribution relative to PMNet [3], particularly the role of static-schema writing and chunk-boundary commits?

[3] Sungwoo Goo, Hwi-yeol Yun, and Sangkeun Jung. Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory. 2026. arXiv:2605.13370.

Reviewer_2whJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SchemaMem, a recurrent memory layer in which each layer keeps a learned schema bank E and an episode-specific phase state S of the same shape (176 slots by 32 features), read through softmax attention over cosine and sine features of E + S. Writes are addressed against the static bank only, so a layer’s update never depends on its own state, and bounded per-token increments are summed over a chunk and committed modulo 2π at chunk boundaries, with ordinary causal attention inside each chunk. The same weights can also process a whole episode as one full-history chunk, and training interleaves such batches with recurrent ones. Experiments use a synthetic address-value database with 8 to 32 addresses and values that scores retained writes, overwritten values and never-written defaults separately, comparing three-layer models of about 55k parameters and 17k persistent state elements against Mamba-3 and Gated DeltaNet. Trained with up to 384 intervening filler queries and tested at 1,536, SchemaMem keeps more written values on average in all nine settings, but it trails Gated DeltaNet on defaults everywhere and on updated values in two of the three 32-address settings. A state-zeroing ablation confirms that written values travel through S, and the authors are explicit that SchemaMem needs substantially more optimizer steps than the baselines.

Critical strengths
  1. Splitting the final read into retained, overwritten and never-written groups, and pushing the delay well past the training range, exposes failure modes that a single aggregate accuracy would hide. The paper reports all 81 terminal checkpoints, including the cells where SchemaMem loses. The paired S = 0 intervention is also a good control, showing that written values really pass through the recurrent state rather than leaking through local attention.

  2. A large and consistent effect against Mamba-3 at matched state size. At delay 1,536, every SchemaMem seed beats every Mamba-3 seed in 17 of the 18 setting and group combinations reported per seed in Table 5 (A5V3 updated is the only overlap), typically by 40 to 60 points in mean accuracy. The layer gets there with standard attention, linear projections and a matrix-product commit, needs no custom scan kernel, and runs the same weights in full-history or recurrent mode. If the effect holds up under proper controls, this is a simple design that others could pick up easily.

Critical weaknesses
  1. SchemaMem gets up to about 24 times more optimizer steps (88667 versus 3750 for Gated DeltaNet in A3V3), interleaved full-history attention batches, commits aligned with the oracle write and read boundaries, and a different stopping threshold. The baselines stop as soon as they reach 99%, with no reported tuning of their decay or step-size initialization. Notably, the one baseline run with a SchemaMem-sized budget, Gated DeltaNet seed 43 on A5V4 (73750 steps), beats every SchemaMem A5V4 seed at delay 1536 on both retained (88.1%) and updated (87.1%) values. Degradation beyond the training length is a known failure of modern recurrent models (e.g., the state collapse analysis of Mamba-2 in Stuffed Mamba, Chen et al., 2024), so matched-budget baselines, decay-free variants such as plain DeltaNet, and ablations of the bank, writer and phase design are needed first.

  2. All results come from one synthetic task with a single, homogeneous filler distribution at 55k parameters, with no standard recall benchmark such as MQAR (Arora et al., 2023), no language modeling and no long-context evaluation. Within the task, SchemaMem is weaker in range (95 to 97% on updated values at delay 384, where the baselines were trained to 99%), and its state corrupts never-written addresses at long delay, since zeroing S lifts defaults from 64.2% to 99.5%. When Table 6 is weighted by the number of addresses in each group, Gated DeltaNet has the higher whole-database accuracy in every scenario at 32 addresses (roughly 80% versus 63% for A5V4 overwrite). SchemaMem’s seed ranges also overlap Gated DeltaNet’s there and in A3V5 and A4V3, so the nine-setting summary in the abstract reads stronger than the data allow.

Questions for authors
  1. How do the baselines behave with a matched optimizer budget, with training delays extended toward 1536, and in decay-free form (plain DeltaNet, or Gated DeltaNet and Mamba-3 with decay initialized close to one)?

  2. Why is SchemaMem’s stopping threshold 95% rather than 99%? Updated-value accuracy at delay 384 is between 95 and 97% in every setting (Table 7), which suggests the update group is the binding constraint. Can SchemaMem reach 99% on updated values at all, and at what cost?

  3. If I read Equations 2 and 3 correctly, and the input projection carries no positional signal, the bottom layer’s committed state is an order-insensitive sum of per-token increments. That layer alone then cannot tell an old value from its replacement. Are overwrites resolved by upper layers reading lower-layer state, and how does updated-value accuracy change with depth?

  4. How large are the increments written by filler queries, and how far do the phases of slots holding written values drift between delays 384 and 1,536? The drop in defaults from delay 384 to 1536, which disappears when S is zeroed, suggests that small filler writes accumulate in the state. Would a learned write gate remove both the default corruption and the retention loss?

  5. What does the periodic phase parameterization buy over a real-valued additive state of the same shape with identical read and write paths? How does it relate to complex-valued holographic memories such as Associative LSTM (Danihelka et al., 2016)? Results with a fixed random bank in place of the learned E, and with a writer that can see its own state (you mention instability in preliminary runs but show no data), would also help isolate what each component contributes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.