- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CQFA, an extension of Gated DeltaNet-2 that adds query feedback to memory updates through input-dependent channel-wise gates. Unlike scalar feedback, these gates allow different channels to contribute with different strengths, changing both the direction and magnitude of the correction. The method keeps erasure and writing separate and preserves the recurrent memory size. The authors also derive a condition for one-step residual contraction and provide a chunkwise implementation for parallel training.
Experiments compare 340M and 1.7B models with several recurrent baselines on language modeling and zero-shot reasoning. Natural and controlled retrieval are evaluated at 340M. The paper reports higher average task scores while retaining roughly 96% of GDN-2’s training throughput. Additional analyses examine query and key readouts, feedback directions, and local residual behavior.
Critical strengths
-
The method adds flexibility while preserving the efficient recurrent structure. Channel-wise gates can change both the strength and direction of query feedback without increasing the memory-state size or changing the rank-one update. This is the strongest positive aspect of the proposal because it offers a practical way to make memory correction more expressive.
-
The core computation is explicit and checkable. The paper provides sequential and chunkwise equations, a reference implementation, and a precise condition for local residual contraction. These details support the basic formulation and provide a useful foundation for implementing and testing the method.
Critical weaknesses
-
The retrieval results are inconsistent with the stated scoring protocol. Appendix A specifies 384 examples per evaluation set and one binary target per S-NIAH example. The highest nonperfect score would therefore be 383/384, or 99.74%. Yet Table 5 reports scores such as 99.98% for GDN-2 and 99.86% for Gated DeltaNet. Taking the best of three evaluations cannot explain these values. This directly affects the reliability of the retrieval evidence. The actual sample counts, aggregation procedure, and per-example outputs need to be provided before these results can support acceptance.
-
The main ablation does not isolate the contribution of channel-wise feedback. Appendix B uses a feedback bound of 0.4 for CQFA and 1.0 for Scalar-Q. The comparison therefore changes feedback strength as well as gate granularity. Matching the parameter counts does not remove this confound, and the results cannot establish whether channel-wise control or the different feedback bound explains the gains. Since channel-wise control is the paper’s main contribution, a comparison with identical bounds and comparable tuning is necessary to substantiate its central claim.
Questions for authors
-
How were Table 5 scores such as 99.98% obtained from 384 single-target examples with binary scoring? Please clarify the denominator and aggregation.
-
Why were feedback bounds set to 0.4 for CQFA and 1.0 for Scalar-Q? Were both tuned under the same protocol?
-
Do you have CQFA and Scalar-Q results with identical feedback bounds to isolate the effect of channel-wise control?
-
How were checkpoints and best results selected? Were all models evaluated on identical examples?
-
Can you provide results for all three training seeds, including means and uncertainty, to show whether the gains are consistent?
2 comments
Thank you for the clear reply. Using 28,000 examples makes the reported percentages possible and explains the mismatch with the earlier description. Please include the exact S-NIAH hit counts so readers can check the scores directly.
My main concern remains the comparison between CQFA and Scalar-Q. The feedback bounds differ, and there was no shared tuning process. This means the current results do not separate the effect of channel-wise gates from the effect of feedback strength.
Using the same stable seed for all three models makes the selection clearer. However, results from one selected seed do not show whether the gains hold across runs. Results with the same feedback bounds and scores from all three seeds are still needed to support the paper's central claim.
Thank you for identifying the reporting inconsistency and the controls needed for the central comparison. We address each question below.
Q1: Retrieval denominator and aggregation. The descriptions of 384 examples and best-of-three selection were retained from the preliminary evaluation. The final results use 28,000 scored examples for each variant/length condition in Table 5 and each task in Table 6. All models use the same examples, with no exclusions. Each condition yields one pooled score under Equation (18). For S-NIAH, this is 100H/28,000, where H counts successful examples. MV/MQ use mean per-example target recall; Table 6 averages its three task scores. The S-NIAH resolution is approximately 0.003571 percentage points, and scores are rounded to two decimals. Values such as 99.98% and 99.86% are therefore compatible with the final sample count. No best-of-three selection is applied. We have corrected the protocol statements and captions; the table values already represent this final pooled evaluation. Exact hit counts and per-example outputs are not supplied in this response.
Q2: Choice and tuning of feedback bounds. The bound Ls=1.0 follows the (0,1) range of Q-Delta's adaptive per-head scalar feedback. Scalar-Q adapts its parameterization to GDN-2 using the same projection parameter count as CQFA. CQFA's L=0.4 constrains each coefficient while allowing separate channel contributions. For projected logits near zero, the fixed negative offset favors smaller coefficients. These are design choices made without a shared tuning sweep. See Q-Delta, Appendix B.
Q3: Comparisons with identical bounds. A matched-bound experiment is not included in the current results. We have clarified that the reported Scalar-Q/CQFA comparison is parameter-matched and evaluates the two specified feedback configurations. The common-bound analysis characterizes additional directional flexibility; measuring its independent performance contribution requires common bounds and comparable feedback strengths.
Q4: Checkpoint selection and shared evaluation examples. For the main GDN-2, Scalar-Q, and CQFA comparison, we used three common training seeds under otherwise identical experimental settings. We selected the same seed for which all three training-loss curves were stable, based on inspection of those curves. The 340M task tables use the completed 15B-token checkpoint from each selected run, with one checkpoint per model shared across tasks. We have replaced “best results” with this selection procedure. The final controlled-retrieval evaluation uses identical examples for all models and scores the complete pooled set, separately from training-seed selection.
Q5: Variability across training seeds. Complete per-seed task summaries and uncertainty estimates are not included in the present results. The reported differences therefore describe the selected runs; their consistency across training seeds remains to be evaluated. The similar timing of training instability is an observation about the loss curves, rather than evidence for seed-averaged task performance.