- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce HALA, a block that uses adjacent sliding-window attention (SWA) and linear recurrence blocks for parallel processing of sequences before mixing their outputs using a learned per-head gating function.
Critical strengths
The experiments seem to have been conducted with some technical care and the supplementary material does contain code that appears to match what is written in the work.
Critical weaknesses
Novelty or Uniqueness of Question of Interest
The novelty of the method appears to be fairly limited, being extremely similar to the Hymba architecture in its use of two adjacent components, namely a SWA component and a linear component, that processes the sequence separately before being mixed together by a gated projection. The authors admit this themselves in the introduction but there isn't any additional details that highlight how HALA is motivated differently and what exactly is the benefit of this specific instantiation compared to existing techniques.
The authors then frame the motivation of the work is to answer "whether the two paths actually divide labour the way the design assumes". However, there are no real experiments to validate this. Comparing against full attention or fixed recurrence is limited to AR accuracy or perplexity evaluations, thus there is no way of understanding directly what information is being handled by each individual branch. For example, in AR, this could be done with measing some type of shapely value or influence score just by seeing how marginal changes in target tokens influence the outputs of different branches. While this could be evaluated in many ways, the paper in its current form doesn't support the claim that is being made in the introduction.
Finally, the authors claim that their method introduces the idea of a learned per-head mixing gate rather than a fixed rule, however the actual implementation is simply modifying a mixing constant based on an additional MLP + sigmoid mixture, which is not novel. The ablations using the static and hard method show that this doesn't lead to very meaningful changes in performance nor does the learnt version show a distinct advantage over the other variants, which further raises issues with the actual contributions of the work.
Experimental Settings
- The size of the models being experimented with are very small (<100M non-embedding parameters), thus there are signficant questions about the actual abilites of these models beyond synthetic AR accuracy. Perplexity results shows HALA to generally lag behind the baselines (Table 3 in Appendix G.1), which is a potentially problematic signal of performance beyond synethic examples where the representation of information is not as compact and simple to retrieve compared to AR tasks.
- For throughput and memory results (Table 2), HALA doesn't seem to show a major improvement untill 128K sequence lengths, so the practical benefit of such an architecture just from this perspective is also limited. While it is possible that shorter lengths could see some benefit as well, the trend being represented would suggest that only for inordinately long sequences could such a paradigm be useful, which is a rather limiting factor.
Presentation
The work reads as having been written directly by an AI tool or at the very least having undergone substantial AI editing, which makes the narrative and flow of the work very difficult to follow at times. Details are often scattered in individual paragraphs that are difficult to follow. This is the same with the code, which appears to be primarily AI written and thus somewhat truncated in organization, making specific components or details somewhat hard to find or navigate.
Questions for authors
Please refer to weaknesses. While I do not want to constrain the authors to specific experiments and prefer to leave them to decide what is of importance without my biasing, there are suggestions regarding what comparisons could be useful or the type of experiments that would be necessary to make this work convincing or clarify any issues that were raised.
3 comments
I appreciate the response from the authors.
However there are many issues that remain; the scale of experiments still is quite small to enable any assessments about the applicability of the approach to any particular task of use, while evidence of superiority doesn't appear to be stable. The main benefit is that there is an aggregate speed-up at long contexts, but I do question again if this would be practically at large scales where it is not only the architecture format that affects this measure.
Thus I prefer to maintain my current assessment, unless there is some scaling to significantly larger models or more realistic (or meaningful, per the authors own interpretation of this word) tasks that show advantages that are relevant in a more practical sense.
(Do further note that parts of the rebuttal and general response reads as having been significantly edited through the use of AI as well, which makes some of the additional details added difficult to understand.)
Response to Reviewer 9MDe
Thank you for reading our response.
We agree on scale. All models are below 100M parameters, and we cannot train significantly larger models within the discussion period. The paper lists this as its main limitation.
One clarification, since our framing may have been unclear. The paper does not claim that HALA is superior, or that the speed-up is its main benefit. On aggregate perplexity HALA trails the baselines, and the revision says so. The claim is narrower: a measurement of where a hybrid's recurrent path helps. It helps only beyond the attention window, and this held against a capacity-matched, tuned baseline. Whether it holds at larger scale is an open question, and the paper says that too.
AI assistance is disclosed in the paper. We are sorry that parts were hard to follow. If a specific point is unclear, we will answer it directly.
Response to Reviewer 9MDe
Thank you for the direct critique. We agree with much of it and restructured the paper accordingly.
Novelty vs Hymba. We agree the block is not novel and no longer present it as a contribution. The contribution is a measurement: where, by retrieval distance, a hybrid's recurrent path helps. Hymba and other within-block hybrids report aggregate quality, which mixes regimes where the recurrence helps with regimes where it does not.
Direct evidence of what each branch handles. We added two branch-level analyses.
The gate. Agreed: it is a standard MLP followed by a sigmoid. A learned gate beats a fixed 0.5 blend by 1.36, 0.40 and 1.63 perplexity on the three domains, and hard and soft gating perform the same. We no longer claim it as novel. Its main use in the paper is as an instrument for measuring routing.
What this instantiation buys. We tested the block's design choices at equal parameters (§5.4). Restoring an FFN of width 480, in exchange for two layers, improves perplexity by about 2.7 and also improves far-token loss. A smaller FFN, paid for with latent rank, leaves far-token performance unchanged, so latent rank is not the bottleneck. Omitting the FFN is therefore a cost, and the paper now states it.
Scale and perplexity. Agreed. The models have 11.9–74M non-embedding parameters. HALA trails the baselines on aggregate perplexity when each architecture uses one learning rate. Tuning HALA alone changes its books perplexity from 44.75 to 40.77 (3 seeds), so we no longer rank architectures against each other.
Throughput. One correction: HALA's time to first token breaks even with the Transformer between 8k and 16k tokens, not at 128k. At 131k it is 6.5× faster (127.4 vs 831.4 ms). We agree the benefit is limited to long contexts, and HALA has the highest peak memory.
Presentation and code. The abstract, introduction, results and conclusion now state the same three findings. We rewrote the README, which now has a table mapping every figure and table to its script and command. AI use is disclosed in the paper.