acceptodds
Withdrawn from ICLR 2027

DisLA: Linear Attention with Distinct Use and Update Decisions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recurrent linear attention stores sequence history in a fixed-size state. Existing gated Delta models use the current input to correct key–value associations in the state, then read the updated state to form an output. Thus, the same correction is committed to the persistent state and contributes to the current output. However, persistent state updates need to account for future outputs, whereas the current output serves only the present step: the two roles may require different state corrections. We introduce DisLA, which learns separate corrections for state updates and current outputs. The two control projections share a base, with a low-rank branch representing their difference. We instantiate this mechanism in Gated DeltaNet-2 (GDN2), preserving its feed-forward width and recurrent state size while supporting chunkwise parallel training. Joint erase–write decoupling in DisLA-GDN2 adds only 0.17% parameters. In matched experiments with approximately 133M parameters and 3B valid next-token prediction targets from FineWeb-Edu per model, DisLA-GDN2 reduces WikiText-2 perplexity by 2.6% and improves average accuracy across LAMBADA and eight reasoning tasks by 1.22 percentage points over GDN2. It also raises the mean score across all 13 RULER tasks at 4K from 1.81 to 2.65.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, recurrent language models, memory updates, low-rank parameterization, delta rule

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KrAiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DisLA, a residual linear-attention design that separates the correction used for the current token output from the correction committed to persistant state. The idea can be summarized as "Joint use-commit decoupling". The method is instantiated within GDN2 through a shared 133M-parameter backbone and a rank-8 difference branch. The paper evaluates the design on nine standard tasks and RULER, reporting modest average gains over the parent model with limited additional parameters and negligible runtime overhead.

Critical strengths

S1. The distinction between the correction used for the current output and the correction retained for later tokens is intuitive and technically well defined. The formulation also recovers the parent GDN2 model when the two residuals coincide, which makes the proposed change easy to interpret.

S2. The low-rank difference branch adds a few parameters without enlarging the recurrent state. The model variants share the same backbone and training setup. In addition, the paper reports component ablations, efficiency measurements, and paired evaluation intervals.

Critical weaknesses

W1. DisLA simultaneously introduces use/commit decoupling, a rank-8 difference branch, and approximately 0.17% more parameters. Although the write-only and erase-only variants isolate the scope of decoupling, the experiments do not establish whether the gain arises from the decoupling itself, the specific low-rank design, or the added capacity in model parameters. It would make the attribution clearer if this paper discusses a parameter-matched GDN2 control, a rank sensitivity study, or an alternative implementation of the use/commit split. The conceptual distinction from related read/update separation methods could also be explained more precisely.

W2. The main results use a single model with approximately 133M parameters and one training seed. The average gain is driven largely by BoolQ. We can see that the performance on several tasks decline with the proposed method. By the way, the write-only variant is at least as competitive as the joint variant on some aggregate results. Some long-context tasks also remain at floor. Bootstrap intervals across evaluation samples do not measure variability across training runs, so the robustness of the improvement remains uncertain.

Questions for authors

Q1. Is there evidence from a matched GDN2 setting that compares DisLA with SILA, read-before-write, or another read/update-decoupling baseline? Such a comparison would help identify which gain depends specifically on the use/commit distinction.

Q2. Can the authors provide a parameter-matched GDN2 widening control, or otherwise show that the gain is not mainly due to the additional parameters or model capacity?

Q3. Can the authors report variability across independent training runs? The current paired bootstrap intervals capture uncertainty across evaluation samples, but not variation across training runs, which is important for judging the robustness of the modest average gain.

Q4. Given that the gain is concentrated in a subset of tasks and the write-only variant is competitive with the joint model on some aggregate results, is Joint intended as a generally preferred design or as a task-dependent trade-off?

Reviewer_j6boOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DisLA separates the correction used for the current output from the correction committed to recurrent memory. It implements this distinction through shared projections with low-rank differences, preserving recurrent-state capacity and chunkwise computation. Experiments on GDN2 show improved language modeling and task-dependent downstream gains.

Critical strengths
  • The central insight is clear and meaningful: immediate computation and persistent memory need not use the same correction. The architectural change follows directly from this observation.
  • The method preserves fixed-size recurrent memory and supports chunkwise computation, with a clear derivation of the resulting computation.
Critical weaknesses
  1. The experiments do not fully isolate the proposed mechanism. The scope ablations establish where decoupling is useful, but do not distinguish its benefit from the additional low-rank parameterization and optimization flexibility. A comparable low-rank baseline that retains tied use/update controls would test the central hypothesis more directly.

  2. The task-dependent effects remain insufficiently explained. Aggregation and QA improve, while several retrieval tasks regress and all configurations score zero on MK-2, MK-3, and VT. These results leave unclear whether the method primarily improves persistent storage, historical readout, or information combination. The zero-score tasks also provide little evidence for distinguishing the configurations.

Questions for authors
  1. Would a baseline with a comparable low-rank parameter budget, while retaining tied use/update controls, recover similar gains?
  2. Can output-level analysis explain why aggregation and QA benefit more consistently than retrieval and tracking? For the zero-score tasks, can the authors distinguish memory failures from generation or task-following failures?
Reviewer_VLokOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces DisLA, which separates the correction term for the state from the correction term used to produce the output. This new recurrence is instantiated using GDN2 using shared erase/write control projections with low-rank differences between the use and commit paths, without increasing recurrent state size. The authors derive a chunkwise parallel implementation (by trivially modifying the existing implementation of GDN2) and evaluate joint, write-only, and erase-only decoupling. They report improvements in language modeling evals, with marginal throughput overhead.

Critical strengths
  1. Very lightweight and easy to follow modification: DisLA retains the parent model’s recurrent state size and feedforward width, adds few parameters, and preserves an effieicnt chunkwise parallel form. The formulation also recovers the parent model when use and commit controls coincide, making the relationship to GDN2 explicit.
Critical weaknesses
  1. Limited novelty: DisLA is a relatively trivial extension of GDN2 that introduces separate erase/write gates for the current output and persistent state update, while otherwise preserving the original recurrence and computational structure. The paper does not introduce a fundamentally new memory mechanism, learning algorithm, or substantial theoretical insight. Given the modest empirical improvements and limited experimental scale, it is difficult to see this modification as a sufficiently significant standalone contribution.
Questions for authors
  1. Can the authors demonstrate a concrete limitation of coupling current use and persistent update decisions in GDN2? In particular, are there settings where the optimal decisions necessarily differ, and does DisLA meaningfully address this limitation? Maybe a synthetic setting would be illuminating here

  2. Beyond introducing separate gates, what fundamental insight or generalizable principle does DisLA establish about recurrent memory architectures? Can the authors develop a broader characterization of when and why separating use and update decisions is beneficial?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.