- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PAW-Mem (Predict, Adapt, and Write), a framework designed to improve compact online associative memory for large language models.
Critical strengths
-
The motivation to decouple token-level backbone computation from segment-level persistent memory updates is highly practical. Utilizing a frozen backbone and updating only a lightweight memory interface offers tangible application value for scenarios requiring long-horizon memory without ballooning context window costs.
-
The empirical results demonstrate that the proposed mechanisms provide complementary benefits.
-
The authors appropriately test the method on a larger 8B backbone and remain objective about the results, avoiding overblown claims about monotonic improvements across scales..
Critical weaknesses
-
The primary contribution lies in integrating existing concepts (predictive supervision and adaptive writing) into the δ-Mem interface. While combinational innovation is acceptable, merely naming the pipeline "Predict-Adapt-Write" without uncovering new behavioral laws or proving definitive end-to-end advantages makes the novelty marginal for an ICLR acceptance.
-
The performance improvements are heavily concentrated on the LoCoMo benchmark, while performance on HotpotQA and IFEval slightly regresses on the 4B model (e.g., IFEval drops from 82.44 to 82.07). Furthermore, the original δ-Mem baseline is missing for the 8B experiments, and there is an unexplained discrepancy in the 4B δ-Mem baseline scores between Table 1 (46.73) and Table 2 (45.96). This lack of consistency, combined with the absence of variance reporting (e.g., error bars or multiple seeds), makes it difficult to reliably assess the stability of the proposed gains.
-
The forward dynamics objective is logically reasonable. However, lower loss cannot directly confirm that the model makes better use of historical memory. The model may only use the prefix within the current segment for token prediction. To distinguish true usage of cross-segment historical memory from ordinary language modeling improvements, stronger mechanistic evidence is required. Possible validations include: measuring prediction loss when historical memory is zeroed or shuffled, or testing performance only on tokens that must depend on cross-segment history.
Questions for authors
Refer to weakness