- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a language-model architecture featuring 31 interleaved cells categorized as Primary, Intermediate, Hub, and Association cells. Local convolutions, windowed attention, and global attention operate on persistent, specialized states; incremental updates are propagated via "Telscoping Residual Inheritance," while the output of each cell is retained as a form of "deep memory." Subsequently, a terminal Association operator performs two sequential "deep reads" to update the model's reasoning and discourse states. A "LatentBank" component introduces compact, specialized states used to modulate the attention inputs within the backbone. A key experimental finding is that employing three-step recursion during training enhances downstream performance, even when inference is restricted to zero or one recursion step. At a scale of 220 million parameters, a configuration using three-step training and one-step inference achieves an average score of 34.66% across nine tasks. The paper also extends these "three-seed comparisons" to model scales of 400 million and 800 million parameters.
Critical strengths
This idea is clear and easy to follow; the three components of the structure: Y, Z, and LatentBank, are also clear.
This finding that increasing the number of recursive steps during training allows for the retention of most performance gains, even when the number of steps is reduced during inference, is valuable. And the experimental results demonstrate a degree of stability; improvements were observed across three different model scales and multiple random seeds, indicating that this advantage is not merely an artifact of a single experiment.
Critical weaknesses
It remains unclear whether full-depth memory is necessary. Reading only the final state yields an average accuracy of 34.74%, while using the complete trajectory results in 34.66% (Table 1). By training a model that has access only to the final state, one can verify whether full-depth memory is essential during the training process.
Some performance gains may stem from the additional computational effort during training. With the number of recursion steps fixed at zero during inference, increasing the number of recursion steps during training from zero to three improves the average accuracy from 32.99% to 34.52% (Table 3(c)). However, having the same number of parameters and the same volume of training data does not necessarily imply an equivalent computational cost. Conducting comparisons under an "equal-compute" setting helps to disentangle the advantages inherent to the architecture itself from the gains derived from additional computation.
Questions for authors
-
Why is the accuracy in MATH of Table 1 all 0? It should be clearly explained the reason for this in the main text.
-
The author states in the paper that "Our design is also motivated by evidence that biological language processing is distributed across differentiated systems," yet this point is not addressed in the subsequent method section—which is quite confusing.
-
"Shallow inference after recursive training"—can this paragraph can further analysis to understand why performance on certain tasks falls short of the baseline?
-
There are a few typos: at the end of the "Inference profiling" paragraph, type an extra ".".This description in Table 3: "Panel(a)".
4 comments
PART 2
3. Why is MATH accuracy zero?
We appreciate spotting this one for clarification.
Our evaluation uses MATH 500, zero shot greedy generation, a maximum of 128 generated tokens, and normalised exact matching of generated boxed answer against the reference boxed answer (Appendix B.4).
Under this protocol, all compared models at 220M, 400M, and 800M obtain 0.00%. This is measured outcome of specified exact answer evaluation.
Our models are relatively small pretrained language models evaluated without task specific mathematical fine tuning.
We agree that main text should explicitly explain evaluation setup and avoid interpreting zero exact match accuracy more broadly than warranted and we will make these clarifications accordingly.
4. Neuroscience motivation
We agree that distinction between architectural inspiration and biological correspondence should appear more clearly in Section 2.
The neuroscience evidence motivates an organisational prior of interacting, differentiated language processing systems, not a claim that RMP reproduces biological language networks.
This prior is instantiated computationally through:
- Primary cells: local causal convolution for short range information.
- Intermediate/Hub cells: windowed attention for contextual processing and interaction with global representations.
- Association cells: global causal attention for long range integration.
- TRI and LatentBank: persistent information exchange while retaining differentiated specialist states.
- Recursive Memory Plane: explicit access to representations at different stages of this hierarchy.
These roles are our engineering choices. We will connect this motivation to method more explicitly.
5. Why do some tasks underperform baselines?
We agree that overall aggregate should not obscure heterogeneous task level behavior.
At 220M, RMP performs well on BLiMP (76.59%) and BoolQ (59.57%), but trails TRM on ARC Challenge (21.74% versus 23.41%) and GSM8K (0.99% versus 1.44%), and slightly trails mHC on OpenBookQA (15.80% versus 16.20%).
These results suggest that improvements in linguistic and contextual representations do not automatically translate into improvements in every form of multi step reasoning or knowledge retrieval.
The present experiments do not isolate a unique causal explanation for these deficits, and we will avoid making one. We will expand discussion of task dependent behavior and make clear that Z=1 is best observed depth for reported nine task aggregate, not a universally optimal inference setting for sure.
Indeed, WikiText-103 perplexity continues improving through Z=3 (58.08 at Z=0, 50.83 at Z=1, and 44.48 at Z=3), establishing an objective dependent trade off.
As an additional robustness check, strict BBH final answer rescoring preserves RMP's lead at all three capacities, and nine task aggregate remains highest for RMP after replacing original BBH results.
6. Presentation corrections
We thank reviewer for identifying typographical issues. We will remove the duplicated period in the inference profiling paragraph and correct Table 3 panel description. We will also strengthen main text explanation of MATH evaluation, clarify neuroscience to method connection, and foreground limitations of downstream aggregate.
To conclude our responses, We hope new results directly address reviewer's central architectural question i.e. full depth memory is not required to preserve shallow downstream accuracy after full memory training, but trained terminal only models consistently underperform full RMP at 220M, 400M, and 800M.
We sincerely appreciate constructive suggestions and hope we have addressed reviewer’s concerns accordingly. Thank you very much for your time reading and reviewing our paper and sharing your feedback with us.
Thank you for your response which resolves my questions. I keep my original score.
Glad to read that we’ve addressed your concerns. Thank you 🙏🏼
Rebuttal 1 - Thanks for your valuable feedback
We appreciate reviewer for spelling out clarity of our architectural contributions, value of recursive training for shallow inference, and consistent results across three model scales and independent seeds. We especially appreciate positive soundness assessment.
Due to character limit, we provide our responses in two parts.
We’ve conducted additional experiments addressing two principal concerns. trained terminal only controls at 220M, 400M, and 800M, and training resource accounting. We also clarify MATH results, neuroscience motivation, and task specific limitations.
1. Is full depth memory necessary during training?
Our new experiments provide direct evidence that full depth access improves training outcomes.
You correctly distinguish existing terminal only inference intervention (Appendix E, Table E.1) from a model trained without full depth retrieval. former achieves 34.74% versus 34.66% because it uses a checkpoint originally optimised with access to all 31 depth states. It cannot determine whether this access contributed during training.
To address this question, we trained terminal only recursive models at all three capacities. These retain the 31 cell specialist backbone, TRI, LatentBank, and recursive R/D updates, but restrict both recursive reads to Y31 throughout training across 3 seeds per param config.
New results (nine-task mean accuracy)
Full trajectory training consistently achieves higher downstream accuracy at every tested capacity. This is precisely comparison needed to distinguish training time utility from inference time redundancy.
The surprising terminal only inference result is now making kinda sense. full depth access during optimisation improves learned representations, yet those representations can subsequently support effective terminal only inference.
Our held out language modeling experiment provides complementary evidence. At Z=3, full trajectory retrieval achieves NLL 3.7951 (PPL 44.48), whereas terminal only retrieval achieves NLL 3.8266 (PPL 45.91). Thus, complete depth access also retains measurable utility for next token likelihood.
2. Training compute and architectural attribution
We agree that parameter matching and equal processed tokens don’t imply identical training computation. We additionally inspected our W&B training logs.
220M resource measurements
These measurements explicitly expose the increased resource requirements of recursive training.
Importantly, trained terminal only controls provide a kind of targeted architectural comparison than external parameter matching i.e. they retain same backbone and recursive structure while changing information supplied to recursion.
Our inference analysis separately demonstrates that, at 220M, RMP with Z=0 achieves 34.52% at 16.22k tokens/s, compared with mHC's 33.87% at 15.93k tokens/s under same full sequence profiling protocol. Hence deeper training can yield a useful shallow inference operating point, while additional training expense remains acknowledged.
<< CONTINUES IN FOLLOWING PART >>