- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
RAVEL adds an explicit memory structure that contains multiple “banks”, each one of which maps the current token’s ID to a dynamically updated representation of previous tokens’ successors. Selected vectors are then aggregated, projected, and gated before added to the base model backbone’s logits prior to decode. Experiments up to 24M parameter language models demonstrate that adding RAVEL reduces the NTP loss over baseline architectures. Additional ablations show that increasing the number of banks provides diminishing returns in modeling performance, and while additional scaling experiments investigate performance across three model sizes, results at scale remain limited.
Critical strengths
The authors provide comparisons across three model sizes and three random seeds, which shows that RAVEL improves upon the baseline backbone in their experimental setting.
Critical weaknesses
The scale at which the RAVEL method is tested is in general too small to give signal (model size caps at ~20M and trained with only 2M tokens). While there are gains at the 12M model scale, Figure 3 seems to suggest that the overall gains of RAVEL seem to decrease as the model parameters increase. In addition, the training horizon of the models is extremely small, which may bias performance towards the RAVEL module as the baselines are vastly undertrained. Furthermore, there is a lack of experiments that test RAVEL’s impact on MQAR or other synthetic tasks in which a model of such scale might perform better at and give better signal.
There also is a lack of baselines and ablations that isolate the effect of RAVEL. For instance, RAVEL at its core is a learned gated bigram; one suitable comparison would be to DeekSeek’s Engram architecture or a simple online bigram cache. Additional ablations on the hash collision and memory correct components would also help disentangle the multiple components of RAVEL. It also seems like a single bank is able to recover most of the overall performance, raising questions about the multi-bank design.
Questions for authors
What are the specifics of the experimental protocol, e.g., peak learning rate, hyperparameters used, etc.
How does RAVEL scale with additional compute, either with model scale or more training data?
How does RAVEL perform on synthetic tasks, such as MQAR, select copying, fuzzy recall, etc. (please see Mechanistic Design and Scaling of Hybrid Architectures), if scaling up language model training is infeasible.
Thank you for your response.
My concern about lack of baselines and ablations remains unanswered. How well does RAVEL compare against a simple learned gated bigram or an online bigram?
While I appreciate the new experiments, the protocol could be improved. For instance, the largest 55M model is very undertrained compared to the smallest model, making extrapolation hard.
What is the performance of a baseline Transformer on the MQAR tasks? What was the protocol used for training and evaluating on MQAR?
Also could you clarify the protocol? Your response says the extended experiments use 8k cosine steps (~32M tokens) while the results report after 8M and 2M tokens.