- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces two architectural/methodological tweaks to language models: Boundary Bottleneck Concept (BBC) and Continuous Concept (CC). The Boundary Bottleneck Concept is a restriction of causal attention maps in an LLM to a tiled pattern as follows:
X O O O O O O O O O O O
X X O O O O O O O O O O
X X X O O O O O O O O O
O O X X O O O O O O O O
O O X X X O O O O O O O
O O X X X X O O O O O O
O O X O O X X O O O O O
O O X O O X X X O O O O
O O X O O X X X X O O O
O O X O O X O O X X O O
O O X O O X O O X X X O
O O X O O X O O X X X X
The Continuous Concept adds a regularizer to the LLM loss to prevent representations of adjacent tokens from varying too drastically, e.g., The authors show via some experiments with GPT-2 size models and 0.1-20B tokens that one can get some favorable results in LLM pretraining at two model scales. They also characterize some geometric properties of the learned representations.
Critical strengths
- The analysis and ablations of how best to situate BBC and CC are relatively thorough.
Critical weaknesses
- The paper is quite hard to read --- there is a lot of vocabulary used without introducing it or formalizing it, so it's quite hard to understand what are the precise claims being made, so it's difficult for me to evaluate the paper.
- The mathematics seems unrelated to the point of the work --- there's some analysis with Kolmogorov complexity and covering numbers that each seem unrelated to the rest of the paper, even if technically impressive (I didn't check those details carefully).
- The paper does not cite or compare against previous related work. For example, the paper's BBC seems like a strictly less effective version of the landmark attention in [1].
- Adding inductive biases ultimately requires justification that they help at scale, but the paper provides no reasoning that such accommodation will scale with model size and dataset size (only with dataset size, wherein it requires extra tuning to compete with the baseline). It also does not motivate the architectural modifications via pointing out concrete bottlenecks observed in nontrivial circumstances. As a result, I'm not given any insight into whether BBC or CC should be used in any particular situation and scale.
[1] Mohtashami, Amirkeivan, and Martin Jaggi. "Landmark attention: Random-access infinite context length for transformers." arXiv preprint arXiv:2305.16300 (2023).
Questions for authors
- How are the theoretical claims directly connected to, or inform, the empirical parts of the paper?
- Can you establish some scaling relationship where if we scale model and dataset size jointly, the improvements by this change persist or stay the same?
- How strongly tuned are the baselines? Did you do sweeps to find optimal hyperparameters for the baselines?
5 comments
(2/2)
Q2: Can you establish some scaling relationship where if we scale model and dataset size jointly, the improvements by this change persist or stay the same?
A2: We agree that establishing a joint model–data scaling relationship for SRIB would be valuable. However, Chinchilla-style co-scaling addresses the compute-optimal allocation between model size and training tokens under a fixed training-compute budget; it does not imply that model size and data must always be scaled together. In practice, many modern LLMs deliberately deviate from this prescription by substantially over-training fixed model sizes relative to the Chinchilla-optimal token budget, as in Llama 3 and Gemma 2.
Our current experiments provide two complementary forms of scaling evidence. First, the gains persist across two model sizes, gpt2-small and gpt2-medium. Second, for fixed model sizes, they persist as the training horizon is extended from 300M to 1B and up to 20B tokens. We therefore have evidence that the effect is not confined to a single model size or short training regime, but we do not yet claim a joint model–data scaling law.
Our design intentionally changes one axis at a time so that persistence of the SRIB effect can be separated from assumptions about compute-optimal co-scaling.
Since the goal of this paper is to establish that structure–residual inductive biases are beneficial and exhibit a layer preference, we leave the derivation of an SRIB-specific scaling law—including how model size, training tokens, , and should co-scale—to future work.
Q3: How strongly tuned are the baselines? Did you do sweeps to find optimal hyperparameters for the baselines?
A3: Thank you for this question, as it highlights the same minimal-intervention philosophy behind our experimental design. Because we use the standard GPT-2 architecture and training recipe, we tune the primary optimization hyperparameter, the learning rate. As described in Section 6 (Setup), we sweep
and select for gpt2-small and
for gpt2-medium.
We then fix the baseline learning rate and tune only the method-specific hyperparameters: for BBC and for CC. We do not retune the baseline separately for each comparison. We believe this controlled setup strengthens the conclusion that the observed gains come from the SRIB interventions themselves, rather than from additional baseline-specific hyperparameter optimization, and that these gains persist over long training horizons.
Addressing W1
W1: The paper is quite hard to read --- there is a lot of vocabulary used without introducing it or formalizing it, so it's quite hard to understand what are the precise claims being made, so it's difficult for me to evaluate the paper.
R1: Thank you for this feedback. We agree that the earlier version tried to compress substantial theoretical and empirical material into limited space, and some of the connections between them were therefore not stated clearly enough. In the revision, we reinstated these connections explicitly, added a roadmap in Section 2, and tied the theoretical results directly to the empirical studies and hyperparameter sweeps.
The theory now serves two concrete purposes. First, the general two-part coding view motivates the structure--residual principle, while the method-specific analysis explains how BBC and CC instantiate it. For BBC, Proposition 1 shows that controls how much completed-segment detail is collapsed into boundary carriers while preserving local realization detail; for CC, Propositions 2--3 show that CC energy controls the magnitude and finite-precision description length of the high-frequency residual. These results make precise what each intervention constrains, rather than using Kolmogorov complexity only as an abstract framework.
Second, the theory informs the hyperparameter behavior. For BBC, and both recover the vanilla model, so any benefit must arise at an intermediate , consistent with the U-shaped sweep. For CC, controls the effective regularization strength, and our gradient-ratio analysis quantitatively predicts the optimal at the layers used in both model sizes: and predicted versus and observed. We have revised the paper to distinguish clearly between these theoretical implications and the empirical questions, such as the optimal layer, that the theory does not claim to solve.
Response (1/2)
W3: The paper does not cite or compare against previous related work. For example, the paper's BBC seems like a strictly less effective version of the landmark attention in [1].
[1] Mohtashami, Amirkeivan, and Martin Jaggi. "Landmark attention: Random-access infinite context length for transformers." arXiv preprint arXiv:2305.16300 (2023).
R3: We cite several related sparse-attention and structured-communication methods in the Related Work section, but due to space constraints we were not able to enumerate every prior approach. We appreciate the reviewer pointing out Landmark Attention and added the citation.
More importantly, BBC is not intended as a more sophisticated sparse-attention architecture. The goal of this paper is to establish that structure–residual inductive biases (SRIB) can improve language-model pretraining and exhibit a clear layer preference. We therefore deliberately use a simple, fixed intervention rather than introducing additional mechanisms such as dynamically learned routing or intervention locations. We believe this simplicity strengthens the central result: even a deliberately minimal bottleneck, when applied at the appropriate depth, improves predictive performance and the gain persists over long training horizons.
This simplicity is deliberate: it isolates the effect of the structure–residual bias from additional architectural machinery.
Q1: How are the theoretical claims directly connected to, or inform, the empirical parts of the paper?
A1: We agree that the Kolmogorov-complexity and covering-number results do not by themselves predict the best K, β or layer, and that neither BBC nor CC directly optimizes Kolmogorov complexity (Section 2 now says this). Instead, for each method they make precise what counts as structure and what counts as residual.
For BBC, a segment can be described by its structure plus an index that picks it out among the segments with the same structure (Proposition 1(iii)). This is true for any model. BBC adds the same kind of split to attention: at the BBC layer, a completed segment reaches later segments only through its boundary state, while the current segment keeps its token-level detail (Proposition 1(i)). K sets where the split falls, and K=1 and K≥N both give the vanilla model (Proposition 1(ii)), so any gain from BBC has to come from an intermediate K.
In CC, the structure is the low-frequency part of the hidden-state sequence and the residual is the high-frequency rest. Proposition 2 bounds the size of the residual by the CC energy, and Proposition 3 bounds the bits needed to describe the residual to a fixed precision. Both bounds shrink as the energy falls, while the structure keeps its own description cost. CC adds this energy to the loss with weight β. A larger β is not always better, though. In a simple estimation model, a small β lowers the expected excess loss at first order while its cost grows only at second order, so the best β is positive but finite (Proposition 12, Appendix C.8). The K and β sweeps are approximately U-shaped, consistent with this (Figure 3).
The gradient analysis in Appendix C.8 also makes a quantitative prediction for β, which we tested. Let ρl be the ratio of the gradient norms of the CC loss and the next-token loss with respect to the hidden states at layer l, measured once on a vanilla checkpoint. Under Adam, CC's effective strength at layer l is then approximately βρl, and if the best effective strength is the same at every layer, the best β scales as 1/ρl. Taking that strength from
gpt2-mediumlayers 10–20, where the fitted optimum lies within our grid, predicts β=0.0013 forgpt2-mediumlayer 22 and 0.0021 forgpt2-smalllayer 10 (Table 21); the sweeps find 0.00125 and 0.0025 (Tables 3 and 5). At matched effective strength, later layers are still better (see our reply to Reviewer rZq2, Q1).Whether the split improves NLL, and which K and layer are best, are empirical questions, which our layer and hyperparameter sweeps answer. We also tested the link between the two methods (Proposition 4) on trained BBC layers. It is informative only when the representation changes little from one token to the next within a segment, which is not the case there, so we now state it only qualitatively (Appendix B.3; see our reply to Reviewer rZq2, Q3).