acceptodds
Withdrawn from ICLR 2027

Graph Machine: Pretraining with Sparse Dynamic Pointers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce the Graph Machine (GM), an architecture that maintains an -sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves complexity in its sparse layers without restricting the potentially accessible state size to . Instead, GM uses pointer-like edges with discrete indices and differentiable weights, updated through a referral mechanism resembling pointer chasing. We replace 75% of Qwen3-0.6B's dense Transformer layers with GM sparse layers and train the resulting models end to end from scratch on 15.7B tokens using a standard pretraining recipe. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, dynamic routing, neural memory, efficient sequence models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jaGbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an alternative mechanism to sparse attention layers through a technique inspired by dynamic pointers with the goal to retain O(n) state but reducing the O(n) access of vanilla transformers. Each token keeps a list of pointers down the KV, cache. Initially it starts as a sliding windows and can grow down the KV history through "referral" from previous pointers.

Critical strengths

-Alternatives to sparse attention mechanisms are an important research direction, especially given the scale of retained context in current LLM and the resulting computational cost.

Critical weaknesses
  • My main concern with the method is a) it is very hard to properly understand the method due to its presentation and b) it is unclear what the benefits of the proposed method would be. It is argued that it can replace existing SA mechanisms, however, to what end? If I understand it correctly its not about memory reduction as KV is maintained, nor is it faster in training / inference (or is lacking empirical support for it). Further, there is no evidence on a clear improvement of quality (see eval scores), especially if considering it under equal resource budget.
  • The description of the mechanism is lackeing: what is its associated computational cost for the different steps and consequences for speed / memory access
  • Further, the method is not evaluated in the context where SA method would be actually required, ie. long context retrieval / efficiency or against relevant baseline methods (even a sliding window) -Many of the references are missing further information such as venues, proceedings or a proper reference link to identify them.
Questions for authors
  • Can you provide further details on the computational resources used for the experiments, ie. walltime and what tradeoffs your proposed mechanism aims at
  • Please include more comparisons for the results, ie. Table 4. As I understand it the performance is near random chance on the evaluated downstream benchmarks?
Reviewer_7DkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Graph Machine (GM) layers, which update their edges through referral and attend only to the positions that the edges name. The edge weights are differentiable, and the pointers are learned. The experiments train models from scratch on about 15B tokens: a dense baseline of Qwen3-0.6B size, and the proposed model, in which 21 of the 28 layers are replaced with GM layers. The paper claims performance close to the baseline with better theoretical efficiency.

Critical strengths
  1. This is a genuinely different design. Compared with dynamic sparse attention, the address computation is constant per token, and the pointers are learned end to end through differentiable edge weights.

  2. The experiments are clearly described, including the dataset and recipe, with a dense baseline.

Critical weaknesses
  1. The core of the paper, the pointer mechanism, is not isolated. The paper mentions clearly competitive baselines such as sliding-window attention and dynamic sparse attention, and a plain local/global hybrid attention would be another. Sliding-window attention, for example, has better efficiency, yet it is never compared.

  2. Each setting appears to have been run only once, and tiny gaps are read as findings. For example, a 0.003 loss gap is called a "moderate improvement", which does not make sense given the randomness of training on 15B tokens. No other model family or larger model is tested to make the results more convincing and general.

  3. Sparse layers may be relatively worse than full attention on long context (all variants trail on RACE-high and LAMBADA), yet there is no mention of any testing, such as a needle-in-a-haystack test showing that the pointers can retrieve the relevant context.

  4. The efficiency gains are only theoretical estimates. No measured inference or training speedup is reported, and the authors admit that the model is several times slower.

Questions for authors

Can you add a basic sliding-window baseline, which has even better efficiency?

Given the small differences in performance, can you measure statistical significance?

Reviewer_hhvYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Graph Machine stores sparse weighted pointers with token states, composes them through referral, and uses sparse attention to read their targets. It replaces 75% of Transformer layers and reports near-baseline loss after pretraining from scratch.

Critical strengths

The paper gives a concrete sparse implementation of address propagation alongside content aggregation. The pretraining study demonstrates feasibility.

Critical weaknesses
  1. The central efficiency claim is not supported by the evaluated system. Every model retains 25% dense attention, so the full architecture remains quadratic in sequence length. The FLOP table estimates only 5–15% total-compute savings; meanwhile, the reference implementation is several times slower than Qwen3. The faster-kernel result is preliminary and does not establish a measured speedup in the reported setting. The paper therefore demonstrates reduced estimated arithmetic in selected layers, not a practically more efficient language model.
  2. The referral ablation is not budget-matched: Theia-K24 has 601M parameters/62.3 EFLOPs, while K24-R3 has 700M/71.6 EFLOPs. The reported gain cannot be attributed to referral independently of added capacity and compute.
  3. The paper cites relevant modern work (Routing Transformer, NSA, DeepSeek-V3.2/DSA) and pointer-memory work, but does not clearly establish what layer-wise persistent edges and referral add over learned sparse selection.
Questions for authors
  1. What measured speed or memory advantage does the full hybrid achieve at the evaluated sequence length?
  2. What does referral add over a sparse/dense baseline with comparable parameters and compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.