acceptodds
Withdrawn from ICLR 2027

PAW-Mem: Predict, Adapt, and Write for Efficient Online Associative Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models increasingly need to retain and reuse information across long interaction histories, yet repeatedly carrying the entire history in the context window incurs growing memory and computation costs. Compact online associative memory offers an appealing alternative by compressing the past into a fixed-size recurrent state. Under aggressive compression, however, each persistent update must preserve useful history while incorporating an entire new segment, and intermediate memory states receive only indirect supervision from the final task objective. We introduce PAW-Mem (Predict, Adapt, and Write), an online associative-memory framework that addresses both issues. Predict uses the pre-write memory state to autoregressively predict the current segment, providing Forward Dynamics supervision along the memory trajectory. Adapt maintains a diagonal online scaling state that adjusts the write geometry across memory dimensions. Write then compresses token-level representations and performs one persistent associative update per segment. With a frozen Qwen3-4B backbone, PAW-Mem improves LoCoMo from 46.73 for the original -Mem to 50.02 while preserving competitive performance on HotpotQA and IFEval. Controlled ablations show complementary gains from Online Scaling and Forward Dynamics, and experiments with Qwen3-8B indicate that the design transfers across backbone scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
online associative memory, long-context language models, recurrent memory, delta-rule learning, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Rpr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PAW-Mem (Predict, Adapt, and Write), a framework designed to improve compact online associative memory for large language models.

Critical strengths
  • The motivation to decouple token-level backbone computation from segment-level persistent memory updates is highly practical. Utilizing a frozen backbone and updating only a lightweight memory interface offers tangible application value for scenarios requiring long-horizon memory without ballooning context window costs.

  • The empirical results demonstrate that the proposed mechanisms provide complementary benefits.

  • The authors appropriately test the method on a larger 8B backbone and remain objective about the results, avoiding overblown claims about monotonic improvements across scales..

Critical weaknesses
  • The primary contribution lies in integrating existing concepts (predictive supervision and adaptive writing) into the δ-Mem interface. While combinational innovation is acceptable, merely naming the pipeline "Predict-Adapt-Write" without uncovering new behavioral laws or proving definitive end-to-end advantages makes the novelty marginal for an ICLR acceptance.

  • The performance improvements are heavily concentrated on the LoCoMo benchmark, while performance on HotpotQA and IFEval slightly regresses on the 4B model (e.g., IFEval drops from 82.44 to 82.07). Furthermore, the original δ-Mem baseline is missing for the 8B experiments, and there is an unexplained discrepancy in the 4B δ-Mem baseline scores between Table 1 (46.73) and Table 2 (45.96). This lack of consistency, combined with the absence of variance reporting (e.g., error bars or multiple seeds), makes it difficult to reliably assess the stability of the proposed gains.

  • The forward dynamics objective is logically reasonable. However, lower loss cannot directly confirm that the model makes better use of historical memory. The model may only use the prefix within the current segment for token prediction. To distinguish true usage of cross-segment historical memory from ordinary language modeling improvements, stronger mechanistic evidence is required. Possible validations include: measuring prediction loss when historical memory is zeroed or shuffled, or testing performance only on tokens that must depend on cross-segment history.

Questions for authors

Refer to weakness

Reviewer_Q5fvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies efficient long-term associative memory for frozen language models. It proposes PAW-Mem, which extends δ\delta-Mem with a predict–adapt–write framework that combines autoregressive supervision of intermediate states, dimension-wise online scaling, and segment-level associative updates. Experiments evaluate long-term conversational memory, multi-document question answering, instruction following, and incremental memory tasks using Qwen3-4B and Qwen3-8B backbones, with ablations of the proposed components.

Critical strengths

The paper addresses a relevant problem in efficient long-term memory and presents a straightforward integration of predictive supervision and adaptive memory updates. The proposed method yield improvements in the reported ablations, and the 4B-scale experiments show gains on LoCoMo while generally preserving downstream performance.

Critical weaknesses
  1. Limited novelty relative to existing approaches.
  • The proposed architecture largely depends on previously established techniques. The underlying memory mechanism follows δ\delta-Mem, while dimension-wise online scaling closely follows the diagonal preconditioning approach introduced by OSDN [1]. Autoregressive supervision of compressed context representations has also been explored in prior work, including AutoCompressor [2].
  • The main contribution therefore seems to be limited to the combination of existing mechanisms within a segment-level memory framework. It's unclear what fundamentally new memory mechanism is introduced beyond this.
  1. Limited empirical gains and insufficient validation.
  • The performance improvements are relatively modest and inconsistent across backbone scales. While PAW-Mem improves LoCoMo over δ\delta-Mem at 4B, it essentially matches the strongest reported comparator at 8B, with limited improvements on other benchmarks.
  • The ablations demonstrate individual component gains but not fully isolate the effects of predictive supervision, online scaling, and segmentation. In particular, changing the segmentation strategy also changes the frequency and granularity of memory updates.
  • Furthermore, despite efficiency being a central motivation, the paper doesn't report end-to-end runtime and memory/throughput comparisons. The computational advantages are not sufficiently validated experimentally. The absence of run-to-run variability or confidence intervals also makes it difficult to assess the reliability of the reported gains.

[1] Zhou et al., 2026. OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention.

[2] Chevalier et al., 2023. Adapting Language Models to Compress Contexts.

Questions for authors
  1. Please clarify the novel contribution beyond integration of existing works δ\delta-Mem, OSDN, and existing segment-level predictive supervision methods?

  2. Could the authors provide matched ablations under identical segmentation and training budgets, together with multiple-seed results and end-to-end efficiency measurements? These would help establish whether the proposed method provide consistent improvements beyond the existing baseline.

Reviewer_SCJLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Carrying long interaction histories in the context window becomes costly, yet compressing them into a fixed-size memory forces each update to preserve past information while absorbing a new segment, with intermediate states receiving only indirect supervision from the final objective. PAW-Mem is an online associative-memory framework with three steps: Predict, Adapt, and Write. Predict uses the pre-write memory to autoregressively predict the current segment (Forward Dynamics supervision), Adapt applies a diagonal online scaling state to adjust write geometry across dimensions, and Write compresses token representations into one persistent associative update per segment. On a frozen Qwen3-4B, PAW-Mem raises LoCoMo from 46.73 to 50.02 while staying competitive on HotpotQA and IFEval, with additional analysis provided.

Critical strengths
  • The problem is clearly stated, and the solution intuitively matches it.
Critical weaknesses
  • Undefined terms and symbols make it impossible to understand the method details (e.g., L_sparse, alpha, etc).
  • The evaluation is far too weak. The backbone is Qwen3-only, baselines are very weak, and comparisons are insufficient. more baselines should be included. More importantly, the core contribution (introduced in Introduction) is not validated: the authors should show a curve which shows δ-Mem degrading with varying segment lengths, and demonstrating PAW-Mem’s retention and residual controls of δ-Mem.
  • The gains in Table 1 are too heterogeneous, and Table 2 shows strong inconsistencies in accuracy changes as each component is incorporated.
  • There is no separate related work section, it is replaced by a single introductory paragraph, and the differences from the closest works (AutoCompressor, OSDN, Titans, etc) are not specifically addressed. A vague citation is also present (LeCun et al., 2015).
  • Overall, while the problem to tackle is clear, the method details and evaluation are so thin that it feels hastily written.
Questions for authors
  1. Is δ-Mem baseline at the token level or the segment level?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.