- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DisLA, a residual linear-attention design that separates the correction used for the current token output from the correction committed to persistant state. The idea can be summarized as "Joint use-commit decoupling". The method is instantiated within GDN2 through a shared 133M-parameter backbone and a rank-8 difference branch. The paper evaluates the design on nine standard tasks and RULER, reporting modest average gains over the parent model with limited additional parameters and negligible runtime overhead.
Critical strengths
S1. The distinction between the correction used for the current output and the correction retained for later tokens is intuitive and technically well defined. The formulation also recovers the parent GDN2 model when the two residuals coincide, which makes the proposed change easy to interpret.
S2. The low-rank difference branch adds a few parameters without enlarging the recurrent state. The model variants share the same backbone and training setup. In addition, the paper reports component ablations, efficiency measurements, and paired evaluation intervals.
Critical weaknesses
W1. DisLA simultaneously introduces use/commit decoupling, a rank-8 difference branch, and approximately 0.17% more parameters. Although the write-only and erase-only variants isolate the scope of decoupling, the experiments do not establish whether the gain arises from the decoupling itself, the specific low-rank design, or the added capacity in model parameters. It would make the attribution clearer if this paper discusses a parameter-matched GDN2 control, a rank sensitivity study, or an alternative implementation of the use/commit split. The conceptual distinction from related read/update separation methods could also be explained more precisely.
W2. The main results use a single model with approximately 133M parameters and one training seed. The average gain is driven largely by BoolQ. We can see that the performance on several tasks decline with the proposed method. By the way, the write-only variant is at least as competitive as the joint variant on some aggregate results. Some long-context tasks also remain at floor. Bootstrap intervals across evaluation samples do not measure variability across training runs, so the robustness of the improvement remains uncertain.
Questions for authors
Q1. Is there evidence from a matched GDN2 setting that compares DisLA with SILA, read-before-write, or another read/update-decoupling baseline? Such a comparison would help identify which gain depends specifically on the use/commit distinction.
Q2. Can the authors provide a parameter-matched GDN2 widening control, or otherwise show that the gain is not mainly due to the additional parameters or model capacity?
Q3. Can the authors report variability across independent training runs? The current paired bootstrap intervals capture uncertainty across evaluation samples, but not variation across training runs, which is important for judging the robustness of the modest average gain.
Q4. Given that the gain is concentrated in a subset of tasks and the write-only variant is competitive with the joint model on some aggregate results, is Joint intended as a generally preferred design or as a task-dependent trade-off?