acceptodds
Withdrawn from ICLR 2027

PRISM: A Probe Suite for Retrieval, Interference, and Selective Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce PRISM, a suite of seven synthetic probes for in-context retrieval. The probes vary how key-value bindings are arranged in the context (dispersion) and how many must be held at once (scope), with sequence length and vocabulary fixed across three difficulty tiers. We train six architectures (GDN, GDP, RWKV-7, Mamba2, MLA, Transformer) from scratch with dense next-token cross-entropy at matched parameters and training budget, with per-seed reporting and length and complexity generalisation tests. We control for the most direct state-size confound by matching nominal recurrent state dimension to the hardest binding count. We find that: (i) under this small-scale, fixed-budget regime, RWKV-7 solves nearly the entire suite and GDN/GDP solve task-dependent subsets, while Transformer, MLA, and Mamba2 solve no task; (ii) controlled interventions show that retrieval-supervision density is a major factor in whether softmax attention learns to retrieve, which helps reconcile PRISM with MQAR and MAD, while Mamba2 solves retrieval only at smaller vocabulary and shorter sequences; (iii) the structured-versus-unstructured dispersion axis dissociates models that the difficulty knob alone does not; (iv) when sequence length and the difficulty knob are each scaled by the same factor (2× or 4×), complexity generalisation degrades more than length generalisation on five of seven tasks.

Primary area:
datasets and benchmarks
Keywords:
in-context retrieval, retrieval circuits, associative recall, sub-quadratic sequence models, complexity generalization, length generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BMhcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PRISM, a suite of seven synthetic tasks for studying in-context retrieval. The tasks cover randomly placed key–value bindings, composite-key retrieval, context shifts, multi-hop retrieval, and interference. The authors train six architectures under a shared training protocol and report results. Under this small-scale setting with dense next-token supervision, RWKV-7 solves nearly all task conditions, while GDN and GDP solve task-dependent subsets. Transformer, MLA, and Mamba2 do not reach the solving threshold. Further experiments show that concentrating supervision on answer positions helps attention models learn retrieval.

Critical strengths
  1. PRISM tests several retrieval operations and interference patterns. Separate scores for different phases, narratives, and topic positions help reveal failures that overall accuracy could hide. The differences across architectures suggest that the suite provides useful information beyond standard key–value recall tests.

  2. The paper includes MQAR positive controls, smaller task settings, an expanded hyperparameter search for Mamba2, and interventions on supervision, vocabulary size, sequence length, and binding count. In particular, training only on answer positions raises Transformer and MLA accuracy on AKR-easy to at least 0.999, providing direct evidence that the training objective affects retrieval learning.

Critical weaknesses
  1. The capacity claim in §3.1 —that no observed failure can be attributed to insufficient per-head state—is stronger than the evidence supports. Nominal state dimensions and rank bounds do not establish that each architecture can effectively store and retrieve information under interference, composite addressing, or multi-hop retrieval. The setup controls an important state-size confound, but state-size ablations or constructive arguments are needed to rule out effective capacity limitations.
  2. Section 2.1 defines scope as the number of bindings held simultaneously and states that the difficulty knob isolates a single axis. However, NIS-moderate and NIS-hard both contain 32 bindings and differ in the number of conflicting keys (§2.2, p. 4), so this comparison changes interference rather than binding count. Across tasks, query formats and scoring targets also differ: AKR primarily scores one early signal, while NCR scores all bindings. These differences limit interpretations based on a clean separation between dispersion and scope.
Questions for authors
  1. To support the capacity claim in §3.1 , could you provide state-size ablations or a constructive memory solution for representative failures, particularly CAR and THR? These would help distinguish effective capacity limitations from learning difficulties. If that distinction remains unresolved, could the claim that no failure can be attributed to insufficient state be narrowed?
  2. Given the scope definition in §2.1 , could binding count and conflict level be treated as separate factors, particularly for NIS? Could the effect of dispersion also be tested within one task by changing only binding layout while keeping binding count, query format, vocabulary partitions, and scoring rules fixed?
  3. Could you clarify whether the “fixed compute budget” in §3.2 means matched training exposure, optimization steps, or computational cost? Reporting training tokens together with FLOPs or runtime would make the comparison easier to interpret. Additional seeds for a few key conditions would also help establish whether the main success and failure patterns are stable.
Reviewer_kxc7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PRISM, a suite of seven synthetic probes for in-context retrieval, interference, and selective memory. It compares six architectures trained from scratch under a shared training recipe, with approximately matched parameter counts, three difficulty tiers, and per-seed reporting. The experiments reveal substantial differences in task acquisition across architectures and show that changing retrieval supervision can reverse some failures of softmax-attention models. The paper also evaluates generalization to longer sequences and increased task complexity, framing the suite around dispersion and scope in a purportedly capacity-sufficient regime.

Critical strengths
  1. The suite covers a useful range of retrieval settings, including conflicting bindings, context shifts, composite addressing, and retention under distraction. The auxiliary metrics and per-seed reporting expose differences that aggregate recall accuracy would obscure.
  2. The supervision interventions provide useful evidence that the training objective materially affects whether an architecture acquires retrieval. In particular, the substantial improvement of Transformer and MLA on AKR under answer-only supervision helps qualify the headline architectural comparisons. The positive controls on MQAR and reduced task settings also strengthen the empirical investigation.
Critical weaknesses
  1. The interpretation of the suite as isolating orthogonal dispersion and scope axes is insufficiently validated. Across tasks, the comparisons change addressing operations, vocabulary partitions, query formats, and placement structure together. Within tasks, the difficulty knobs also change quantities beyond the stated memory demand. For example, increasing the number of NCR bindings at fixed context length increases retrieval-supervision density, which the paper itself identifies as a major determinant of acquisition. In NIS, moderate and hard have the same number of narrative-specific bindings, while key overlap increases. These settings provide useful stress tests, but their outcomes do not cleanly isolate the proposed explanatory axes. For a benchmark whose contribution rests on diagnostic interpretation, stronger justification of what each probe measures is central.
  2. The claimed separation between capacity and acquisition is not adequately established. Section 3.1 interprets a head’s rank bound as a capacity for a corresponding number of bindings, then concludes that observed failures cannot be attributed to insufficient state. A rank ceiling does not establish that the architecture’s actual encoding, update, and readout can realize the required task. Composite-key addressing and two-hop retrieval impose computational requirements beyond the number of stored bindings, and nominal hidden-state dimensions are not directly interchangeable across architectures. The paper provides neither task-specific realizations or resource analysis nor systematic state-size controls establishing the claimed capacity-sufficient regime. Consequently, the evidence does not support attributing the remaining failures specifically to acquisition as strongly as the framing suggests.
Questions for authors
  1. What evidence isolates dispersion and scope from changes in addressing operation, query format, and retrieval-supervision density? In particular, how should the NCR difficulty comparisons and the fixed-binding-count NIS comparisons be interpreted under the proposed decomposition?
  2. What establishes effective capacity sufficiency for each architecture and task beyond nominal state dimensions or rank bounds?
Reviewer_4oiQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author introduce a suite of synthetic in-context retrieval tasks that focus on retrieval, interference, and selective memory. The author compared six architectures trained from scratch on these tasks.

Critical strengths
  1. The PRISM tasks are clearly explained.
  2. The author conducted controlled experiments.
Critical weaknesses
  1. Most experiments are on synthetic tasks (trained on synthetic data). Some models have inductive biases that are better suited to certain tasks. The observation might not be true when trained on general pretraining on language modeling tasks and on downstream tasks.
  2. There are already many previous works on synthetic retrieval and associative recall. It is unclear how PRISM provides additional signal apart from these tasks.
Questions for authors

Check the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.