- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents tree-grams which are a hierarchical n-gram model to approximately model language. They demonstrate that tree-grams can outperform existing n-gram baselines and can match some capabilites of deep neural models like transformers. The paper demonstrates applications including understanding hyperparameter transfer and distribution shifts. Furthermore, they demonstrate that with the aligned topology, the model has effective receptive fields, something lacking in other models.
Critical strengths
-
The proposed method is novel while outperforming other n-gram estimators and provides a way for exploring controlled approximations of language which can be used for hyperparameter tuning or studying topologies.
-
The experiments are thorough, considering a wide range of baselines and conditions, providing a detailed view of how tree-grams work across different settings and how distributions affect training/tuning performance.
-
The demonstration of the effective receptive fields in the aligned topology with verification using probes and patching shows that tree-grams may be useful for interpretability and provides an alternative view of model computation.
Critical weaknesses
-
Figure clarity can be improved as some have difficult to read legends (Figure 1b, 3a/b) or seem to have undefined components (Figure 3 dashed lines).
-
The method could be more clearly explained and use improved notation. In particular, one confusion is around which is introduced but not used in the following parts, making it difficult to follow exactly how the parts come together.
Questions for authors
-
Would you be able to clarify how the different components in the method come together?
-
Could you provide additional explanation about Figures 3a/b and in particular, how should I interpret the Figure 3a where it seems like excess loss vs. data complexity seems to have a more complex relationship for L=4?
3 comments
We would like to thank the referee for their time, effort and feedback which considerably strengthened our paper. We would appreciate it if you could indicate whether we were able to address your concerns adequately.
Thank you for the thorough response. The revisions have helped significantly with the clarity, and my concerns have been addressed. I will maintain my positive rating of the paper.
Response to reviewer yHbH
We thank the reviewer for their substantial time and effort in reviewing our work. We address their main concerns and questions below.
W1 : We have now updated all the Figures in the main text, to make it more legible. Some of the changes made are as follows:
Fig 1 a,b,c are all bigger now. The legend describing the baselines is made bigger and pushed to the side for added clarity.
Fig 3 is now split into two (larger) figures with captions and intuitive labels to explain the trends in the figure. Additional plots are relegated to the appendix, for the sake of brevity.
In general, we have made changes to the figures to only keep the most informative plots. These plots are supplemented with additional descriptive text wherever possible. All components are clearly marked and labelled as well.
W2 : We avoid reusing hi(ℓ) since we use that to denote the output of the Merge operation. As the reader would notice, the qi(ℓ) used in the Forward Pass equation, denotes the output of the Merge operation and the residual connection. We have also made some changes in the diagram to more clearly illustrate the Forward Pass (Fig 1a). We would be grateful if the reviewer could point out if there are further changes that need to be made in the description.
Q1 : The different components come together, as follows.
Q2 : We have made modifications to the current main text figure to make it easier to read the trend. Briefly, we show the trend of the loss as a function of the learning rate (LR) for the target (wider) transformer model. The drift of the optimal LR for the proxy (narrower) transformers (relative to the actual optimal LR for the target model) determines the excess loss. Due to the asymmetric nature of the loss curve around the learning rate, it is safer to underestimate the LR.