acceptodds
Desk rejected at ICLR 2027

What to Write: Structured Sparse Writing for Test-Time Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time training turns an input stream into fast weights through successive memory writes. Each write carries a decision that is usually left implicit: when only a fraction of the available contributions can enter memory, which ones should, and which ones should enter together? The choice is separate from the update rule and the chunk schedule a backend already fixes, and it is not about volume alone, because an update combines its contributions before any read sees them. We introduce Structured Sparse Writing (SSW), a model-independent operator that makes the decision explicit: it scores contiguous blocks of contributions with signals the update path already produces, selects the highest-scoring blocks under a budget, and gathers their contributors in temporal order before invoking the original write. SSW trains no selector and preserves the full read stream, target construction and commit schedule, so one interface precedes both optimizer-based and additive memories while exposing the selection unit, the ranking signal and the budget as separate design choices. Analysing the objective this induces delimits what it explains: it admits a tight worst-case omission guarantee for norm-scored additive contributions, yet the same argument proves it must prefer token-wise selection to any grouping — while grouping is what measurably helps. That advantage therefore lies in the interactions between retained contributions, which no scalar score encodes. Experiments on two backends agree: at a matched budget the retained identities govern quality, sparse writes reach the dense update's quality using a fraction of its contributions, and accuracy and answer likelihood prefer different budgets on identical inputs. Write allocation is thus a design axis of streaming memory, defined by the selection unit, the ranking signal and the readout it serves.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time training, fast-weight memory, sparse updates, streaming language models, long-context modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bZNhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies selective memory updates ("write allocation") in chunk-wise Test-Time Training (TTT). The authors propose Structured Sparse Writing (SSW), which scores contiguous blocks of tokens within each update chunk using backend-supplied signals, selects the top-KK blocks under a budget, and gathers them in temporal order before invoking the original update operator. Evaluations on LaCT (760M) and In-Place TTT (Qwen3-4B) show that sparse block writing can improve RULER retrieval accuracy over dense writing and token-wise selection, while dense writing remains preferable for NLL.

Critical strengths
  1. Clean problem formulation: Decoupling internal write selection from the read path and commit schedule is a clean way to study fast-weight capacity without disrupting local context representations.

  2. Insightful findings on block vs. token selection: The controlled experiments on LaCT (Figure 2, Table 3) convincingly show that retaining contiguous blocks outperforms token-wise top-KK and dispersed groups by a wide margin (32.75% vs. 18.69% at 16K). This highlights the importance of local token interactions when updating memory via nonlinear optimizers like Muon.

  3. Figure 3 demonstrates that free-generation retrieval accuracy peaks at sparse budgets (1/8 or 1/4), whereas answer NLL and continuation NLL favor denser writes.

Critical weaknesses
  1. Missing reduced-learning-rate dense control on LaCT: Dropping 87.5% of tokens without inverse-retention rescaling substantially reduces the total update magnitude per chunk. In the archived In-Place TTT study (Appendix A.3, Line 744), simply halving the learning rate of the dense baseline recovers most of the performance deficit and leaves an statistically insignificant gap to SSW. However, this crucial control is missing for the primary LaCT model. It is unclear how much of SSW's gain over Dense on LaCT comes from selective block pruning versus implicitly shrinking an overly large default update step size at 16K/32K.

  2. Incomplete experimental records and unexplained discrepancies: Several notes in the appendix raise reproducibility concerns. For In-Place TTT (Appendix A.3, Line 734), the random baseline scored 76.17 (32K) and 69.30 (64K) in earlier archived runs (matching SSW's 76.93 and 69.35), but drops to 56.17 and 59.30 in Table 1's "updated aggregates," with the authors noting that per-input logs and execution configs for the updated numbers are unavailable. Similarly, Appendix A.1 (Line 673) notes that the exact head/step reduction for LaCT's learning-rate score is unavailable in the supplied records.

  3. Limited theoretical depth and baseline mismatches: Section 3 relies on basic identities (triangle inequality in Prop. 1 and Frobenius norm expansion in Eq. 5) under linear additive assumptions, which do not directly explain why contiguous blocks align better or how they interact with LaCT's nonlinear Muon optimizer. Additionally, LaCT's random baseline in Table 1 uses full-size masking rather than tensor gathering, though reverse ranking in Table 3 partially addresses this.

Questions for authors
  1. How does a reduced-learning-rate (or scaled-down) Dense baseline perform on LaCT at 16K and 32K (e.g., scaling the update or inner LR by 1/81/8, 1/41/4, or 1/21/2)?

  2. Could you clarify why the In-Place TTT random baseline dropped by ~20 points between the archived B records and Table 1, and why the exact score reduction details for LaCT and per-input logs for the updated In-Place runs are unavailable?

  3. In Table 3, Block16 consistently outperforms Block64 across both Dev and New splits (e.g., 24.75 vs. 21.44 at 32K). Why was Block64 kept as the main configuration?

Reviewer_3GfWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies write allocation in test-time training (TTT) with fast-weight memories. At each write, only a fraction of the available per-token contributions can enter the memory update. The authors propose Structured Sparse Writing (SSW). It splits a chunk into contiguous blocks, scores each block by summing per-token priorities the backend already computes, keeps the top-K blocks under a budget, and passes the retained contributions, in temporal order, to the unchanged update operator. The read path and chunk schedule are untouched.

The theory has two parts. Proposition 1 gives a tight worst-case omission guarantee for norm-scored additive contributions under a block budget, solved exactly by top-K. The authors also give a pairwise-interaction decomposition (Eq. 5) to argue that joint retention matters beyond scalar scores.

The experiments use two backends:

LaCT (760M): at 12.5% of writes, SSW improves RULER needle-in-a-haystack retrieval over dense writing by 6.56 points at 16K and 2.12 at 32K. In-Place TTT (Qwen3-4B): at 50% of writes, SSW improves over dense by 13.46 points at 32K and 0.89 at 64K. Controls: block selection beats token-wise selection and dispersed grouping, reverse ranking performs near random, and accuracy and answer NLL favor different budgets on the same inputs.

The authors conclude that write allocation is a design axis of TTT.

Critical strengths
  1. The question is framed clearly and the controls are well chosen. Treating write allocation as a decision separate from the update rule and the schedule is a useful framing. The paper backs it with matched-budget comparisons, a reverse-ranking control that shares SSW’s execution path, and a dispersed-groups control. These isolate the effect of ranking and grouping better than a plain “sparse vs. dense” comparison would. The reverse-ranking control matters most. It scores 10.31 and 4.31 against SSW’s 33.50 and 21.44, which shows the ranking signal is doing real work on the LaCT backend.

  2. The paper is clear about its limits. It reports bootstrap intervals, says plainly that the 16K dense-control interval reaches zero, and flags missing raw records. Its finding that accuracy and answer NLL prefer different budgets on identical inputs (Fig. 3) is a practically relevant caution for anyone tuning write budgets.

Critical weaknesses
  1. The empirical case is narrow, and part of it cannot be verified. Most of the evidence comes from one 760M LaCT checkpoint on synthetic needle-in-a-haystack retrieval, plus one author’s prose for the NLL study. The In-Place results in the main table (Table 1) come from aggregate scores without per-input records or intervals. Appendix A.3 says the earlier records of the same experiment gave random-selection means of 76.17 and 69.30. Table 1 reports 56.17 and 59.30, and the paper says it cannot attribute the change to a specific cause. A 20-point shift in the control makes it hard to know how large SSW’s real advantage over random is on this backend. Other evidence is also thin. The dense-control interval at 16K reaches zero. The token-wise comparison in Table 3 exists only on development inputs. On natural-text continuation, dense writing is best and SSW’s gain over random is about 0.003 to 0.005 nats. No wall-clock, memory or FLOP measurements support the efficiency framing (only contributor counts are reported).

  2. The explanation and theory do not yet support the paper’s main conceptual claim. Proposition 1 follows from the triangle inequality plus top-K, and it assumes norm scores. Appendix A.1 acknowledges that LaCT’s actual score (a learning-rate statistic) is not an optimizer-aware norm, so the guarantee does not cover the main experimental setting. The claim that grouping helps because of interactions between retained contributions is not isolated empirically. Contiguous blocks may simply keep the needle’s tokens together, and the dispersed-group result (8.00 vs. 32.75) is consistent with that locality explanation. Appendix B shows that uniform random selection matches a reduced-learning-rate dense write in expectation. The Limitations section admits a half-learning-rate dense arm recovers much of the dense deficit, with an unresolved gap to SSW. So it remains unclear how much of the gain is selective writing and how much is simply smaller or regularized updates.

Questions for authors
  1. In Appendix A.3, what explains the change in the In-Place random-selection scores (76.17 and 69.30 earlier vs. 56.17 and 59.30 now)? What were the dense and SSW numbers under the earlier records? Can you provide per-input outputs and intervals for Table 1?

  2. Can you report token-wise selection on the new-input populations as well as development inputs, so the block-vs-token claim does not rest on development data only?

  3. Can you report a tuned reduced-learning-rate dense baseline on LaCT at matched effective update size, so the gain can be separated from a smaller-update effect?

  4. Can you report latency or memory for SSW including scoring, sorting and gathering, so the efficiency framing has measurements behind it?


Also, some suggestions for future work:

  1. Evaluating on non-needle tasks, where the relevant evidence is not one contiguous span, would show whether contiguity helps because of locality or because of joint-retention effects.

  2. Testing more than one scoring signal per backend would show whether SSW depends on the specific signals used here.

Reviewer_fdDuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structured Sparse Writing (SSW), a plug-and-play processor for selectively writing information to memory in test-time training (TTT) models under a specified write-budget. SSW computes a net score for each token block and writes only the top-k blocks to memory. The paper compares SSW with random selection across recent backbones, including in-place TTT, and write ratios ranging from 1/32 to 1.

Critical strengths
  1. The paper proposes a novel model-agnostic structured variant of TTT, a recent sequence modeling approach.
  2. The experiments cover multiple backbones and write ratios.
Critical weaknesses
  1. It is unclear how practical the write-budgeted setting addressed by the paper is. Since the current selection mechanism must compute scores for all tokens and read them all, it is unclear how much computational benefit selective writing provides.
  2. The evaluation uses random selection as its only baseline. Other intuitive policies, such as selecting the most recent blocks under the same write budget, would provide a more informative comparison.
Questions for authors
  1. What are the end-to-end computational and memory-efficiency gains from using SSW?
  2. Could the authors compare SSW against additional baselines, including a policy that selects recent blocks under the same write budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.