- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PRISM, a suite of seven synthetic tasks for studying in-context retrieval. The tasks cover randomly placed key–value bindings, composite-key retrieval, context shifts, multi-hop retrieval, and interference. The authors train six architectures under a shared training protocol and report results. Under this small-scale setting with dense next-token supervision, RWKV-7 solves nearly all task conditions, while GDN and GDP solve task-dependent subsets. Transformer, MLA, and Mamba2 do not reach the solving threshold. Further experiments show that concentrating supervision on answer positions helps attention models learn retrieval.
Critical strengths
-
PRISM tests several retrieval operations and interference patterns. Separate scores for different phases, narratives, and topic positions help reveal failures that overall accuracy could hide. The differences across architectures suggest that the suite provides useful information beyond standard key–value recall tests.
-
The paper includes MQAR positive controls, smaller task settings, an expanded hyperparameter search for Mamba2, and interventions on supervision, vocabulary size, sequence length, and binding count. In particular, training only on answer positions raises Transformer and MLA accuracy on AKR-easy to at least 0.999, providing direct evidence that the training objective affects retrieval learning.
Critical weaknesses
- The capacity claim in §3.1 —that no observed failure can be attributed to insufficient per-head state—is stronger than the evidence supports. Nominal state dimensions and rank bounds do not establish that each architecture can effectively store and retrieve information under interference, composite addressing, or multi-hop retrieval. The setup controls an important state-size confound, but state-size ablations or constructive arguments are needed to rule out effective capacity limitations.
- Section 2.1 defines scope as the number of bindings held simultaneously and states that the difficulty knob isolates a single axis. However, NIS-moderate and NIS-hard both contain 32 bindings and differ in the number of conflicting keys (§2.2, p. 4), so this comparison changes interference rather than binding count. Across tasks, query formats and scoring targets also differ: AKR primarily scores one early signal, while NCR scores all bindings. These differences limit interpretations based on a clean separation between dispersion and scope.
Questions for authors
- To support the capacity claim in §3.1 , could you provide state-size ablations or a constructive memory solution for representative failures, particularly CAR and THR? These would help distinguish effective capacity limitations from learning difficulties. If that distinction remains unresolved, could the claim that no failure can be attributed to insufficient state be narrowed?
- Given the scope definition in §2.1 , could binding count and conflict level be treated as separate factors, particularly for NIS? Could the effect of dispersion also be tested within one task by changing only binding layout while keeping binding count, query format, vocabulary partitions, and scoring rules fixed?
- Could you clarify whether the “fixed compute budget” in §3.2 means matched training exposure, optimization steps, or computational cost? Reporting training tokens together with FLOPs or runtime would make the comparison easier to interpret. Additional seeds for a few key conditions would also help establish whether the main success and failure patterns are stable.