acceptodds
Under review as a conference paper at ICLR 2027

What's in a Tree ? Hierarchical -gram models as controlled approximators of language

Abstract

Hierarchical models have been proposed as synthetic settings for studying language. However, the relevance to realistic language corpora is under-explored. In this work, we introduce tree-grams, autoregressive generative -gram models that model the conditional probability tensor through a hierarchy of order-2 and order-3 merges augmented with normalization layers, residual connections, and MLPs. At fixed , a rank hyper-parameter, , controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. Tree-grams are shown to out-perform existing -gram baselines for higher , while remaining competitive with deep neural sequence models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyper-parameter transfer under P can be made more robust when narrower models are tuned on simpler data. In addition, tree-grams allow us to investigate the effect of pretraining and finetuning distributions. We show that performance increasingly worsens as the complexity of the finetuning distribution increases. However, the pre-training distribution complexity has little effect in a capacity-limited regime, while there is a non-monotonic effect in a horizon-limited regime. Tree-grams are interpretable by construction. In particular, terminal nodes in different layers have receptive fields whose information can be independently recovered through linear probes and activation patching. When allowed to route across candidate topologies, we observe emergent sparsity where tree-grams produce concentrated path probabilities fit by a power law with a generalized exponential cutoff.

Primary area:
generative models
Keywords:
n-gram models, language modeling, tensor factorization, tree topologies, interpretability, hierarchical grammar

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yHbHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents tree-grams which are a hierarchical n-gram model to approximately model language. They demonstrate that tree-grams can outperform existing n-gram baselines and can match some capabilites of deep neural models like transformers. The paper demonstrates applications including understanding hyperparameter transfer and distribution shifts. Furthermore, they demonstrate that with the aligned topology, the model has effective receptive fields, something lacking in other models.

Critical strengths
  • The proposed method is novel while outperforming other n-gram estimators and provides a way for exploring controlled approximations of language which can be used for hyperparameter tuning or studying topologies.

  • The experiments are thorough, considering a wide range of baselines and conditions, providing a detailed view of how tree-grams work across different settings and how distributions affect training/tuning performance.

  • The demonstration of the effective receptive fields in the aligned topology with verification using probes and patching shows that tree-grams may be useful for interpretability and provides an alternative view of model computation.

Critical weaknesses
  • Figure clarity can be improved as some have difficult to read legends (Figure 1b, 3a/b) or seem to have undefined components (Figure 3 dashed lines).

  • The method could be more clearly explained and use improved notation. In particular, one confusion is around hi(l)h_i^{(l)} which is introduced but not used in the following parts, making it difficult to follow exactly how the parts come together.

Questions for authors
  • Would you be able to clarify how the different components in the method come together?

  • Could you provide additional explanation about Figures 3a/b and in particular, how should I interpret the Figure 3a where it seems like excess loss vs. data complexity seems to have a more complex relationship for L=4?

3 comments
AuthorsOpenReview ↗

Response to reviewer yHbH

We thank the reviewer for their substantial time and effort in reviewing our work. We address their main concerns and questions below.

W1 : We have now updated all the Figures in the main text, to make it more legible. Some of the changes made are as follows:

  1. Fig 1 a,b,c are all bigger now. The legend describing the baselines is made bigger and pushed to the side for added clarity.

  2. Fig 3 is now split into two (larger) figures with captions and intuitive labels to explain the trends in the figure. Additional plots are relegated to the appendix, for the sake of brevity.

  3. In general, we have made changes to the figures to only keep the most informative plots. These plots are supplemented with additional descriptive text wherever possible. All components are clearly marked and labelled as well.

W2 : We avoid reusing hi(ℓ)h^{(\ell)}_i since we use that to denote the output of the Merge operation. As the reader would notice, the qi(ℓ)q^{(\ell)}_i used in the Forward Pass equation, denotes the output of the Merge operation and the residual connection. We have also made some changes in the diagram to more clearly illustrate the Forward Pass (Fig 1a). We would be grateful if the reviewer could point out if there are further changes that need to be made in the description.

Q1 : The different components come together, as follows.

  1. Embed. Each of the n−1n-1 context tokens becomes zi(0)=Emb(xi)+piz^{(0)}_i = \mathrm{Emb}(x_i) + p_i.
  2. Upward pass. Layer by layer, each parent merges one or two children (decided by the topology), followed by the residual and the MLP block above. Each upward node summarises a contiguous block of the context.
  3. Downward pass. Along the terminal node of each layer, from the top down, a downward merge combines the state carried from the older context with the terminal summary of the current layer, with the same residual/MLP pattern.
  4. Readout. The bottom terminal state passes through the tied embedding matrix and a softmax to give p(xn∣x<n)p(x_n \mid x_{<n}).
  5. Training. Training unfolds each window into its n−1n-1 left-padded prefixes. It is then trained using standard cross-entropy loss for auto-regressive training.

Q2 : We have made modifications to the current main text figure to make it easier to read the trend. Briefly, we show the trend of the loss as a function of the learning rate (LR) for the target (wider) transformer model. The drift of the optimal LR for the proxy (narrower) transformers (relative to the actual optimal LR for the target model) determines the excess loss. Due to the asymmetric nature of the loss curve around the learning rate, it is safer to underestimate the LR.

We would like to thank the referee for their time, effort and feedback which considerably strengthened our paper. We would appreciate it if you could indicate whether we were able to address your concerns adequately.

Thank you for the thorough response. The revisions have helped significantly with the clarity, and my concerns have been addressed. I will maintain my positive rating of the paper.

Reviewer_8EtkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the present paper, the authors introduce tree-grams: fixed-context autoregressive models of language that combine hierarchical merges, residual connections, normalization, and MLPs. Context length and a hidden width dimension control the model horizon and its capacity. The authors compare tree-grams with count-based (worse) and neural baselines (better at long context), then use generated corpora to study learning-rate transfer between narrow and wide transformers and adaptation between pretraining and fine-tuning distributions (experiments on TinyStories). They also show initial investigations into how topology affects token prediction, reporting concentrated routing probabilities when models can select among candidate topologies.

Critical strengths
  • The paper proposes a new construction to define a generator fitted to a text corpus whose context horizon and model capacity can be varied separately.

  • The empirical evaluation includes a broad range of count-based and neural baselines, as well as parameter-matched transformer comparisons.

  • The finding that learning-rate tuning on synthetic approximations can improve transfer from narrow to wide transformers is interesting and merits further investigation.

Critical weaknesses
  • The paper introduces an interesting and potentially useful family of controlled language generators, the tree-grams. However, it remains unclear what is gained specifically from the hierarchical construction. The comparisons show that tree-grams learn useful short-context statistics, but do not isolate the contribution of the tree structure from that of the residual connections, normalization, and MLPs.

  • The topology is restricted a priori to a narrow family of nearly balanced trees, and in almost all experiments a single topology is selected by hand. The rationale for the restriction, the selected alternatives, and the particular merge operations is not sufficiently developed. The ablations do not establish how sensitive the conclusions are to these choices, or whether more expressive interactions would substantially change performance (or even simpler ones, like a standard balanced binary hierarchy).

  • The experiments are limited to contexts of at most 31 preceding tokens, substantially shorter than the paragraph-length contexts relevant even to TinyStories. There seems to be a considerable computational bottleneck preventing longer-context experiments, limiting the demonstrated scalability and fidelity of the proposed approximation.

  • The hyperparameter-transfer and pretraining–finetuning experiments suggest interesting effects, but their robustness and practical significance remain unclear, particularly whether the procedure offers computational savings over alternative tuning strategies. Moreover, substantial seed instability affects some transfer configurations, making it unclear whether the nonmonotonic behavior could be an optimization artifact.

  • The quadratic toy model has a weak connection to the empirical results. Its conclusions depend on assumptions about signal normalization, feature alignment, and optimization that are not validated in the trained transformers. It illustrates a possible learning-rate drift, but does not establish an explanation for the observed behavior, and its role in supporting the paper’s main conclusions remains unclear.

  • The interpretability experiments do not clearly separate learned organization from architectural constraints and information preserved through residual connections. The aligned topology shows more separated token-recovery patterns, while the anti-aligned topology performs better, and the implications of this tradeoff are not explored.

  • The routing experiment establishes concentrated path probabilities, but does not disentangle initialization or router-optimization effects. The preferred topologies depend on the seed, and no clear connection is established between routing concentration, predictive performance, and the structure of the data, making this preliminary exploration inconclusive.

  • Overall, many observations remain insufficiently connected to a clear, tested interpretation.

Questions for authors
  • What advantage does the hierarchical construction provide over other finite-context neural generators with adjustable width? Can the authors isolate the contribution of hierarchy while retaining comparable residual connections, normalization, and MLPs?

  • How did the authors choose the layer-size recurrence in Eq. (1) and the particular topology family explored? How would a standard balanced binary hierarchy compare? What motivates the factorized upward merge and the asymmetric downward operation, and how sensitive are the results to these choices?

  • Can the authors provide longer-context results, or quantify the computational and memory costs of reaching paragraph-length contexts?

  • How is the empirical entropy reference HnH_n in Figure 2 estimated, and how reliable is it at larger nn? There should be a distinction from the entropy estimated directly from a known synthetic generator.

  • Does tuning on the tree-gram synthetic proxy offer a net computational saving once generator training, sampling, and proxy selection are included? How does it compare with simpler alternatives, such as reduced-data tuning on the original corpus?

  • Can any quantities driving the quadratic toy model—such as the allocation of predictive power or the relevant curvature—be measured in the transformer experiments? What testable prediction does the model provide beyond illustrating a possible learning-rate drift?

  • What precisely are the “directional components” proposed to explain distributional alignment? Can the authors measure them or test a concrete alignment criterion beyond entropy?

  • Do the token-recovery patterns persist in untrained networks or after removing residual connections?

  • Does concentrated routing improve prediction or permit pruning without loss? How do routing distributions compare across seeds?

  • Please explain μP\mu P on first use, clarify what topology “alignment” refers to, and make Figure 3 easier to interpret. Its caption should also distinguish source-complexity effects from target-complexity effects consistently with the main text.

5 comments
AuthorsOpenReview ↗

Response to reviewer 8Etk

We thank the reviewer for their substantial time and effort in reviewing our work. We address their critical concerns and questions below.

W1 : We have added the role of LayerNorm and residual connections in Section 5 (Ablations). We have previously shown that MLPs mainly help with stability - even without MLPs, tree-grams can learn the structure of language. In addition, we emphasize that the importance of our tree-grams lies in their experimental value - they serve to test the relevance of hierarchical structures in language modelling. In this respect, MLPs, normalization and residual connections are strategies that we borrow from ML to stabilize and train tree-grams.

W2 : The choice of topologies is driven by (1) maximum merge of order-3 based on the CP-type Hadamard decomposition of [1] and (2) restriction on the maximum number of order-2 merges to approximately halve the nodes at each layer and have logarithmic depth (inspired from the Random Hierarchy Model [2]). Neither of these choices are unique or privileged.

We select the aligned topology τ0\tau_0 for most experiments since the smallest tree-gram n=3n=3 only has two topologies. The effective receptive fields in this aligned topology make it suitable for investigating the role of hidden nodes as a function of the context horizon. We leave the latter for future work.

Standard binary hierarchy restricts testable n-grams to only powers of 2.

It is plausible that richer interactions would improve the representative capacity of such models, but training such models is already challenging. Without explicit experimentation, we prefer not to speculate on the sensitivity of the conclusions to these architectural choices. However, we emphasize that even with such a restricted class of topologies, tree-grams beat nn-gram baselines (Fig 1) and are scalable (Fig 2).

W3 : This is a challenge that’s already highlighted under Limitations, with a possible solution. Our experimental range already encompasses regimes where tree-grams outperform or under-perform deep transformers. We clarify however that our main basis of comparison are the n-gram baselines. Tree-grams are more scalable than tested baselines (Fig 1). In addition, we show they do not plateau as quickly as shallow transformers (Fig 2). We also emphasize that the value of tree-grams is experimental (as a controlled scalable model to study the structure of data) and not empirical (a novel method which out-performs transformers).

W4 : New experiments were added to assess the generalizability of these results to richer datasets like WikiText and FineWeb-Edu. Our results hold for WikiText. For FineWeb-Edu, the main observation that “optimal learning rate drifts” holds as well. In addition we have fixed the optimization issues by re-running cells with higher standard deviations. These new results are included in Fig 3,4,5. Tree-grams provide explicit controls on the data distribution, giving an experimental setting to test hypotheses about language data. We also connect the PT-FT performance gap results to the joint effect of entropy and effective range of correlations. We will open-source these synthetic datasets. Thus computational savings happen because of the amortization of generation costs.

W5 : We emphasize that the observed learning rate drift is the novels empirical phenomena, that happens across three different datasets that we test. It is also posited as the explanation for why HP transfer can be robust with simpler data. Because the loss is asymmetric (Fig 3 and related works in [3,4,5]), it is better to under-estimate the optimal learning rate. We now motivate how the toy model serves as an approximation of our empirical models. In addition, we compare the relation of the optimal learning rate and the empirically measured top-scale share of power, which matches pretty well with the toy model. This experiment also provides a more solid foundation for the connection between the toy model and the tree-gram data.

[1] Loconte, L., Mari, A., Gala, G., Peharz, R., de Campos, C., Quaeghebeur, E., Vessio, G., & Vergari, A. (2025). What is the relationship between tensor factorizations and circuits (and how can we exploit it)? Transactions on Machine Learning Research.

[2] Cagnetta, F., Petrini, L., Tomasini, U. M., Favero, A., & Wyart, M. (2024). How deep neural networks learn compositional data: The random hierarchy model. Physical Review X, 14(3), Article 031001.

[3] Kosson, A., Welborn, J., Liu, Y., Jaggi, M., & Chen, X. (2026). Weight decay may matter more than µP for learning rate transfer in practice. In International Conference on Learning Representations.

[4] Ishikawa, S., Yokota, R., & Karakida, R. (2026). Measuring the local shape of learning-rate transfer curves. arXiv.

[5] Filatov, O., Ebert, J., Wang, J., & Kesselheim, S. (2025). Time transfer: On optimal learning rate and batch size in the infinite data limit. OpenReview.

AuthorsOpenReview ↗

Response to reviewer 8Etk (ctd.)

W6 : The anti-aligned tree's patterns are the same before and after training, so they are architectural. The aligned tree's separated bands are absent at initialisation and are produced by training. Without residual connections, an untrained anti-aligned tree recovers nothing above chance. A tree trained without residuals, however, reproduces the aligned tree's pattern node for node, at slightly lower recovery (this is now placed in Appendix K for constraints of space). The paper reports the “slightly better performance of anti-aligned trees” as an observation. Such a small difference is not expected to materially alter the utility of the tree-grams as a generator of synthetic data. Thus we did not pursue this trade-off.

W7 : We now perform a pruning experiment, where we show that we can remove upto 95% of the each level’s merge candidates without a meaningful change in the loss. Thus, the observation of emergent sparsity can be tied into a concrete statement about tree-gram performance. This experiment suggests that we can increase tree-gram capacity by training only sparse subsets of all topologies.

W8 : Our revision now either provides testable explanations for the model or withdraws claims that are insufficiently supported.

Q1 : As mentioned in our response to W1, the tree-gram keeps improving as the horizon grows, with parameters linear and depth logarithmic in the context (Fig 1, 2). With the MLPs removed, every interaction between tokens runs along the tree, and this model still beats every flat and tensor-network baseline at matched size by more than half a nat at 31 tokens of context. (Fig 1 and Fig 6).

Q2: The recurrence, the topology family and the comparison with a balanced binary hierarchy are covered in our response to W2. On the operations:

  • Upward merge. The merge is a CP-type (Hadamard) factorisation of an order-3 tensor, as in the tensor circuits of Loconte et al. (2025). It needs d2d^2 parameters per node instead of the d3d^3 f a full order-3 core. A two-child merge is the smallest that combines information, and it keeps the depth logarithmic.
  • The two passes. They follow the factorisation of p(xn∣x1:n−1)p(x_n \mid x_{1:n-1} ) first summarise the n−1n-1 context tokens, then use the summary to predict xnx_n.The upward pass compresses the context into dd-dimensional summaries of contiguous blocks at dyadic scales. The downward pass then walks along the terminal nodes, from the oldest block to the block that contains xn−1x_{n-1}. At each step it combines the state carried from the older context with that block's summary. The tied softmax reads out the final state.
  • *Asymmetric downward merge. It applies WTW^T to the parent state because its two inputs play different roles. One is the state carried down from the older context, and the other is the summary of the current block. As mentioned in W2, we have not tested other merge operators, so we cannot say how sensitive the results are to this choice. We also once again emphasize that our claim in this work is not that our current implementation of tree-grams is the best or most efficient or most optimized. Since this is a fundamentally new class of architecture, our goal in this paper was to set up a basic working tree-like models, which can train and scale to higher nn-grams, as opposed to previous nn-gram baselines. We have now explicitly highlighted this in our Limitations section.

Q3 : Our reason for stopping at 31 tokens is in our response to W3. The measured costs at width 1024: Parameters grow linearly with the context, with one weight matrix per node. Training slows as the context grows, because it unfolds each window into all of its prefixes. Throughput falls about fivefold between 3 and 31 tokens of context. The compute for one unfolded window grows quadratically, and is about 15 times larger at 127 tokens than at 31. Generation needs only the last row, so its cost per token grows linearly: about 3.6 times larger at 127 tokens than at 31. The unfolding drives the training cost, so an objective that supervises only some of the prefixes in each window is the natural route to longer horizons. We leave this to future work.

AuthorsOpenReview ↗

Response (ctd.)

Q4 : For real text, the true conditional entropy is unknown. The reference in Figs. 1, 2 and 7 is taken at each context length: it is the lowest per-position validation loss among four converged reference models trained with long contexts. [6] show that different architectures converge to the same per-position curve. We take the agreement of these four models as evidence that the envelope reflects the data more than any single model. They differ by 0.09–0.18 nats, and the spread does not grow with context length. The envelope is still an empirical reference, not an entropy, and a model could in principle fall below it. For corpora sampled from a tree-gram, the conditional entropy is instead computed exactly from the generator's softmax, averaged over sampled contexts. Section 3.1 now gives both definitions, with different symbols: HnH_n for the empirical reference and HH for a generator's entropy rate.

Q5 : In our setting, one proxy run costs a fifth (2-layer target) to an eleventh (4-layer target) of one target run. A sweep on one proxy corpus therefore costs about 11 A100-hours, against about 110 for the exhaustive three-seed sweep at the target. The proxy's cost is fixed while the target's grows, so this ratio widens as the target gets larger. The one-time cost of a tree-gram corpus is paid once per corpus, as mentioned in our response to Reviewer WxSj (W3), and we will release the corpora and generators. Maximum generation time for bigram corpus is 30 minutes on a single CPU for TinyStories. We have not run reduced-data tuning on the original corpus. It is a reasonable baseline, and the Limitations list it.

Q6 : We now empirically test a concrete prediction from the toy model. We show that the optimal learning rate v/s allocation of predictive power matches the theoretical toy model prediction.

Q7 : We improve the previous statement. We now find that the target’s entropy rate and the effective range of its correlations jointly account for the fine-tuning gap. The physical interpretation of our finding is that generators with comparable entropies, perform worse if the effective correlation length is higher. We quantify this through a four-parameter fit. This explains 82–99% of the variance across all 81 source–target pairs at each depth and dataset, and predicts held-out targets with comparable error.

Q8 : As mentioned in our response to W6, the anti-aligned tree's patterns are the same before and after training, so they are architectural. The aligned tree's separated bands are absent at initialisation and are produced by training. Without residual connections, an untrained anti-aligned tree recovers nothing above chance. A tree trained without residuals, however, reproduces the aligned tree's pattern node for node, at slightly lower recovery (this is now placed in Appendix K).

Q9 : We show that routing preserves prediction performance. It also permits pruning which we have now detailed in the manuscript. Different seeds share the shape of the routing distribution, but which paths get used matches only at chance level.

Q10 : μ\muP refers to the Maximal-Update Parametrization due to [Yang et. al]. We refer to it now explicitly on the first instance. Alignment describes the location of the order-2 merges in a topology. This naming convention comes from the hypothesis that English language is predominantly right-branching [Berg, 2012] We emphasize that we do not test the validity of this assumption Fig 3 is divided into three figures now, for better clarity. The caption has also been made consistent with the main text now.

[6] Cagnetta, F., Raventós, A., Ganguli, S., & Wyart, M. (2026). Deriving neural scaling laws from the statistics of natural language. arXiv.

Reviewer_WxSjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces a new n-gram architecture called tree-grams, in which the horizon and model capacity control how closely the model approximates a corpus. The authors show that on TinyStories, tree-grams match or outperform other n-gram baselines, and are comparable to deep sequence models up to roughly n = 8. The authors use text sampled from tree-grams as a lower-complexity approximation of real data. They find that hyperparameter transfer is more robust when the proxy is tuned on such data than on real data. They also study pretraining/finetuning mismatch across these corpora, finding that the finetuning gap grows with target complexity and depends differently on horizon and capacity mismatch.

Critical strengths

The paper is generally clear and well-organized. The authors provide the right amount of detail and are honest about the limitations of their work. The architecture is genuinely novel and achieves strong performance against a wide range of baselines and even some transformer-based and SMM-based deep neural models. Additionally, the architecture allows two knobs, the horizon length and model capacity, with which one can control how closely data sampled from tree-grams approximates real data. A model with this kind of control, whose samples also have a computable entropy rate, could be a useful tool for controlled studies of training dynamics. The finding that hyperparameter tuning is more robust when the proxy is tuned on low-complexity synthetic data rather than real data is surprising and practically relevant.

Critical weaknesses
  1. All experiments are done on a single corpus, namely TinyStories, which makes it hard to assess the generalizability of the results. This is a very simple synthetic dataset, with restricted vocabulary and syntax, so I find it rather unsurprising that an n-gram model can approximate it well. Also, I'm unsure whether the hyperparameter transfer result is just specific to this dataset. The paper would be much stronger if the authors provided additional experiments on some other datasets.
  2. The interpretability contribution seems rather unconvincing compared to the rest of the paper. If I understand correctly, these experiments generally show that the model behaves as expected. I might be wrong about this, so can the authors explain what the purpose of these experiments is?
  3. Training and sampling from tree-grams seem costly compared to training and sampling from transformers, for instance, but I don't see any direct comparison. Can the authors provide some numbers for comparison between the models? Since tree-grams seem to be most useful for generating synthetic data, the cost of generation seems particularly relevant.
Questions for authors

See the questions from the weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to reviewer WxSj

We thank the reviewer for their substantial effort in reviewing our work. Below we address their main concerns about our paper.

W1 : We have now added results on richer datasets like Wiki-Text and FineWeb-Edu. We observe that tree-grams are competitive with deep neural language models on these datasets too. The observations regarding hyper-parameter transfer and pre-training, fine-tuning distribution gap holds for both WikiText and TinyStories. For FineWeb-Edu, drift in optimal learning rate is also observed to happen. Hyper-parameter transfer is not as reliable however, since the optimal learning rate increases with width. The latter observation is beyond the scope of this work. In addition, we want to emphasize that even on simple datasets like TinyStories, our tree-grams out-perform tested nn-gram baselines.

W2 : We observe that we can prune upto 95% of topologies without significant loss in performance for top-2 routing, which is related to the emergent sparsity we observe through topology-routing. (Fig 7) In addition, we want to clarify that the purpose of the “effective receptive fields” subsection was to provide evidence that topology τ0\tau_0 provides an alternate view of model computation from transformers which behave closer to τ−1\tau_{-1}. We have placed this in the Appendix for constraints on space.

W3 : The exact generation times are reported in the Appendix now. On dual AMD 7763 CPUs, the estimated cost for generating a synthetic corpora on average is 3 hours for Wiki-Text (130MT, vocab=8192), 10 hours for TinyStories (530MT, vocab=8192) and 36 hours for FineWeb-Edu (1.05BT, vocab = 50257). In contrast, transformers typically take 1.5-2x less time. Moreover, these costs are amortized per dataset. We will open-source these datasets so that the community need not retrain.

Thank you very much for carefully addressing my concerns! I believe all the points I raised are now resolved (I also read the other reviewers' comments and the responses) and the revision is indeed much stronger. For this reason, I am now in favour of the acceptance of this paper and will increase my score.

We thank the referee for their time, effort and feedback which helped strengthen the paper considerably.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

We thank the reviewers for their detailed comments and feedback on our submission. We summarize the main changes made in the submission here.

  1. We now add tree-gram results on Wiki-Text and FineWeb-Edu. We see that the main observations on TinyStories (the drift in optimal learning rate and weak dependence on pre-training distribution complexity) replicate on these richer datasets. This directly addresses the concerns of all the reviewers on the generalizability of our results to different and more complicated datasets.

  2. We empirically test the toy model predictions for the HP transfer experiment. We also show that the entropy and the effective range of correlations jointly explain the pretraining-finetuning performance gap.

  3. We observe pruning (upto 95%) topologies in routing experiments can preserve predictive performance.

We also want to emphasize that the value of tree-grams is experimental. It allows us to rigorously test ideas about the structure of (language) data and learning. This is in contrast with empirical value as a model class that out-performs SOTA neural networks. Thus the focus in this work was to set up a basic version of tree-like models that scale better than existing n-gram baselines (including shallow transformers).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.