acceptodds
Under review as a conference paper at ICLR 2027

Structure–Residual Inductive Biases for Language Model Pretraining

Abstract

Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure–residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that low CC energy controls the error of multiplicity-corrected BBC-style boundary attention. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure–residual biases can improve language-model pretraining beyond next-token prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, inductive bias, representation learning, sparse attention, spectral regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iyCROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces two architectural/methodological tweaks to language models: Boundary Bottleneck Concept (BBC) and Continuous Concept (CC). The Boundary Bottleneck Concept is a restriction of causal attention maps in an LLM to a tiled pattern as follows:

X O O O O O O O O O O O
X X O O O O O O O O O O
X X X O O O O O O O O O
O O X X O O O O O O O O
O O X X X O O O O O O O
O O X X X X O O O O O O 
O O X O O X X O O O O O
O O X O O X X X O O O O
O O X O O X X X X O O O
O O X O O X O O X X O O
O O X O O X O O X X X O
O O X O O X O O X X X X

The Continuous Concept adds a regularizer to the LLM loss to prevent representations hth_t of adjacent tokens from varying too drastically, e.g., LCC=1T−1∑t=1T−1∣ht+1−ht∣22.\mathcal{L}_{CC} = \frac{1}{T-1}\sum_{t = 1}^{T - 1}\\|h_{t+1} - h_{t}\\|_2^2. The authors show via some experiments with GPT-2 size models and 0.1-20B tokens that one can get some favorable results in LLM pretraining at two model scales. They also characterize some geometric properties of the learned representations.

Critical strengths
  • The analysis and ablations of how best to situate BBC and CC are relatively thorough.
Critical weaknesses
  • The paper is quite hard to read --- there is a lot of vocabulary used without introducing it or formalizing it, so it's quite hard to understand what are the precise claims being made, so it's difficult for me to evaluate the paper.
  • The mathematics seems unrelated to the point of the work --- there's some analysis with Kolmogorov complexity and covering numbers that each seem unrelated to the rest of the paper, even if technically impressive (I didn't check those details carefully).
  • The paper does not cite or compare against previous related work. For example, the paper's BBC seems like a strictly less effective version of the landmark attention in [1].
  • Adding inductive biases ultimately requires justification that they help at scale, but the paper provides no reasoning that such accommodation will scale with model size and dataset size (only with dataset size, wherein it requires extra tuning to compete with the baseline). It also does not motivate the architectural modifications via pointing out concrete bottlenecks observed in nontrivial circumstances. As a result, I'm not given any insight into whether BBC or CC should be used in any particular situation and scale.

[1] Mohtashami, Amirkeivan, and Martin Jaggi. "Landmark attention: Random-access infinite context length for transformers." arXiv preprint arXiv:2305.16300 (2023).

Questions for authors
  • How are the theoretical claims directly connected to, or inform, the empirical parts of the paper?
  • Can you establish some scaling relationship where if we scale model and dataset size jointly, the improvements by this change persist or stay the same?
  • How strongly tuned are the baselines? Did you do sweeps to find optimal hyperparameters for the baselines?
5 comments
AuthorsOpenReview ↗

Response (1/2)

W3: The paper does not cite or compare against previous related work. For example, the paper's BBC seems like a strictly less effective version of the landmark attention in [1].

[1] Mohtashami, Amirkeivan, and Martin Jaggi. "Landmark attention: Random-access infinite context length for transformers." arXiv preprint arXiv:2305.16300 (2023).

R3: We cite several related sparse-attention and structured-communication methods in the Related Work section, but due to space constraints we were not able to enumerate every prior approach. We appreciate the reviewer pointing out Landmark Attention and added the citation.

More importantly, BBC is not intended as a more sophisticated sparse-attention architecture. The goal of this paper is to establish that structure–residual inductive biases (SRIB) can improve language-model pretraining and exhibit a clear layer preference. We therefore deliberately use a simple, fixed intervention rather than introducing additional mechanisms such as dynamically learned routing or intervention locations. We believe this simplicity strengthens the central result: even a deliberately minimal bottleneck, when applied at the appropriate depth, improves predictive performance and the gain persists over long training horizons.

This simplicity is deliberate: it isolates the effect of the structure–residual bias from additional architectural machinery.

Q1: How are the theoretical claims directly connected to, or inform, the empirical parts of the paper?

The mathematics seems unrelated to the point of the work — there's some analysis with Kolmogorov complexity and covering numbers that each seem unrelated to the rest of the paper, even if technically impressive (I didn't check those details carefully).

A1: We agree that the Kolmogorov-complexity and covering-number results do not by themselves predict the best KK, β\beta or layer, and that neither BBC nor CC directly optimizes Kolmogorov complexity (Section 2 now says this). Instead, for each method they make precise what counts as structure and what counts as residual.

For BBC, a segment can be described by its structure plus an index that picks it out among the segments with the same structure (Proposition 1(iii)). This is true for any model. BBC adds the same kind of split to attention: at the BBC layer, a completed segment reaches later segments only through its boundary state, while the current segment keeps its token-level detail (Proposition 1(i)). KK sets where the split falls, and K=1K=1 and K≥NK\geq N both give the vanilla model (Proposition 1(ii)), so any gain from BBC has to come from an intermediate KK.

In CC, the structure is the low-frequency part of the hidden-state sequence and the residual is the high-frequency rest. Proposition 2 bounds the size of the residual by the CC energy, and Proposition 3 bounds the bits needed to describe the residual to a fixed precision. Both bounds shrink as the energy falls, while the structure keeps its own description cost. CC adds this energy to the loss with weight β\beta. A larger β\beta is not always better, though. In a simple estimation model, a small β\beta lowers the expected excess loss at first order while its cost grows only at second order, so the best β\beta is positive but finite (Proposition 12, Appendix C.8). The KK and β\beta sweeps are approximately U-shaped, consistent with this (Figure 3).

The gradient analysis in Appendix C.8 also makes a quantitative prediction for β\beta, which we tested. Let ρl\rho_l be the ratio of the gradient norms of the CC loss and the next-token loss with respect to the hidden states at layer ll, measured once on a vanilla checkpoint. Under Adam, CC's effective strength at layer ll is then approximately βρl\beta\rho_l, and if the best effective strength is the same at every layer, the best β\beta scales as 1/ρl1/\rho_l. Taking that strength from gpt2-medium layers 10–20, where the fitted optimum lies within our grid, predicts β=0.0013\beta=0.0013 for gpt2-medium layer 22 and 0.00210.0021 for gpt2-small layer 10 (Table 21); the sweeps find 0.001250.00125 and 0.00250.0025 (Tables 3 and 5). At matched effective strength, later layers are still better (see our reply to Reviewer rZq2, Q1).

Whether the split improves NLL, and which KK and layer are best, are empirical questions, which our layer and hyperparameter sweeps answer. We also tested the link between the two methods (Proposition 4) on trained BBC layers. It is informative only when the representation changes little from one token to the next within a segment, which is not the case there, so we now state it only qualitatively (Appendix B.3; see our reply to Reviewer rZq2, Q3).

Q2: Can you establish some scaling relationship where if we scale model and dataset size jointly, the improvements by this change persist or stay the same?

A2: We agree that establishing a joint model–data scaling relationship for SRIB would be valuable. However, Chinchilla-style co-scaling addresses the compute-optimal allocation between model size and training tokens under a fixed training-compute budget; it does not imply that model size and data must always be scaled together. In practice, many modern LLMs deliberately deviate from this prescription by substantially over-training fixed model sizes relative to the Chinchilla-optimal token budget, as in Llama 3 and Gemma 2.

Our current experiments provide two complementary forms of scaling evidence. First, the gains persist across two model sizes, gpt2-small and gpt2-medium. Second, for fixed model sizes, they persist as the training horizon is extended from 300M to 1B and up to 20B tokens. We therefore have evidence that the effect is not confined to a single model size or short training regime, but we do not yet claim a joint model–data scaling law.

Our design intentionally changes one axis at a time so that persistence of the SRIB effect can be separated from assumptions about compute-optimal co-scaling.

Since the goal of this paper is to establish that structure–residual inductive biases are beneficial and exhibit a layer preference, we leave the derivation of an SRIB-specific scaling law—including how model size, training tokens, KK, and β\beta should co-scale—to future work.

Q3: How strongly tuned are the baselines? Did you do sweeps to find optimal hyperparameters for the baselines?

A3: Thank you for this question, as it highlights the same minimal-intervention philosophy behind our experimental design. Because we use the standard GPT-2 architecture and training recipe, we tune the primary optimization hyperparameter, the learning rate. As described in Section 6 (Setup), we sweep η∈3×10−4,6×10−4,10−3,1.5×10−3\eta\in\\{3\times10^{-4},6\times10^{-4},10^{-3},1.5\times10^{-3}\\} and select η=10−3\eta=10^{-3} for gpt2-small and η=6×10−4\eta=6\times10^{-4} for gpt2-medium.

We then fix the baseline learning rate and tune only the method-specific hyperparameters: KK for BBC and β\beta for CC. We do not retune the baseline separately for each comparison. We believe this controlled setup strengthens the conclusion that the observed gains come from the SRIB interventions themselves, rather than from additional baseline-specific hyperparameter optimization, and that these gains persist over long training horizons.

AuthorsOpenReview ↗

Addressing W1

W1: The paper is quite hard to read --- there is a lot of vocabulary used without introducing it or formalizing it, so it's quite hard to understand what are the precise claims being made, so it's difficult for me to evaluate the paper.

R1: Thank you for this feedback. We agree that the earlier version tried to compress substantial theoretical and empirical material into limited space, and some of the connections between them were therefore not stated clearly enough. In the revision, we reinstated these connections explicitly, added a roadmap in Section 2, and tied the theoretical results directly to the empirical studies and hyperparameter sweeps.

The theory now serves two concrete purposes. First, the general two-part coding view motivates the structure--residual principle, while the method-specific analysis explains how BBC and CC instantiate it. For BBC, Proposition 1 shows that KK controls how much completed-segment detail is collapsed into boundary carriers while preserving local realization detail; for CC, Propositions 2--3 show that CC energy controls the magnitude and finite-precision description length of the high-frequency residual. These results make precise what each intervention constrains, rather than using Kolmogorov complexity only as an abstract framework.

Second, the theory informs the hyperparameter behavior. For BBC, K=1K=1 and K≥NK\ge N both recover the vanilla model, so any benefit must arise at an intermediate KK, consistent with the U-shaped sweep. For CC, β\beta controls the effective regularization strength, and our gradient-ratio analysis quantitatively predicts the optimal β\beta at the layers used in both model sizes: 0.00130.0013 and 0.00210.0021 predicted versus 0.001250.00125 and 0.00250.0025 observed. We have revised the paper to distinguish clearly between these theoretical implications and the empirical questions, such as the optimal layer, that the theory does not claim to solve.

Reviewer_rZq2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two inductive biases for language-model pretraining. BBC limits cross-segment attention to boundary tokens at a selected layer, while CC penalizes differences between adjacent hidden states. Experiments on GPT-2 Small and Medium show that BBC works best in early layers and CC in later intermediate layers. Their combination improves held-out NLL, with gains over vanilla continuing through 20B training tokens. The paper also provides a structure–residual coding interpretation and diagnostics of boundary importance and spectral structure.

Critical strengths
  1. The contrasting depth preferences are interesting and consistent across both model sizes.

  2. The paper includes useful sparse-attention and regularization controls, with separate tuning for alternative penalties.

  3. Both methods are simple. Neither requires extra tokens or a separate prediction network. BBC+CC reduces GPT-2 Medium’s NLL from 3.615 to 3.563 at 300M tokens, and improvements over vanilla remain through 20B tokens..

Critical weaknesses
  1. BBC closely resembles fixed sparse attention [1], and CC closely matches temporal activation regularization [2], making their depth-dependent use a key contribution. However, CC penalizes raw hidden-state differences, so the same coefficient may impose different effective strengths across layers. Table 3 sweeps the coefficient but does not establish whether the depth preference remains after controlling for these differences.

  2. CC’s added value remains unclear beyond the tested setting: the main evaluation covers one data source and one architecture family, and at 20B tokens CC improves BBC’s NLL only from 2.7876 to 2.7858. The main evaluation covers only GPT-2 models and one data source. Testing a modern LLM architecture with RoPE, RMSNorm, and gated MLPs would help establish whether the depth findings and gains extend beyond GPT-2. A matched comparison with STP [3] is also missing; the adjacent-state cosine baseline does not implement STP’s three-point objective.

[1] Child et al. Generating Long Sequences with Sparse Transformers. 2019.

[2] Merity et al. Regularizing and Optimizing LSTM Language Models. ICLR, 2018.

[3] Huang et al. Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA. 2026.

Questions for authors
  1. Since CC’s effective strength may vary across layers, does its preference for later layers remain after matching representation scale or relative gradient strength?

  2. Do the depth preferences and CC’s additional gains hold in a modern architecture? Does the improvement extend to an external text distribution?

  3. For the theoretical connection, does the trained BBC layer satisfy Proposition 4’s low-energy condition, and does the bound track the measured error for multiplicity-corrected attention?

3 comments
AuthorsOpenReview ↗

Response (1/3)

Q1: Since CC’s effective strength may vary across layers, does its preference for later layers remain after matching representation scale or relative gradient strength?

BBC closely resembles fixed sparse attention [1], and CC closely matches temporal activation regularization [2], making their depth-dependent use a key contribution. However, CC penalizes raw hidden-state differences, so the same coefficient may impose different effective strengths across layers. Table 3 sweeps the coefficient but does not establish whether the depth preference remains after controlling for these differences.

A1: Yes, the preference for later layers remains after matching. We measured the relative gradient strength ρl=∣∇HLCC∣F/∣∇HLNTP∣F\rho_l=\\|\nabla_H\mathcal{L}_{\rm CC}\\|_F/\\|\nabla_H\mathcal{L}_{\rm NTP}\\|_F at β=1\beta=1 for each layer of our 100M-token vanilla checkpoints, so that the effective strength of CC at layer ll is βρl\beta\rho_l (Appendix C.8, Table 21). As the reviewer suspected, it changes with depth: in gpt2-medium, ρl\rho_l is about 0.83–0.86 for layers 2–10 and rises to 3.54 at layer 22, so the same β\beta is about four times stronger at layer 22 than at layer 2. ρl\rho_l also reflects representation scale: when the NTP loss is invariant to rescaling the hidden states, as it approximately is in our pre-LN models, rescaling them by α\alpha multiplies ρl\rho_l by α2\alpha^2 (Proposition 11). We therefore match on ρl\rho_l rather than on scale alone.

The tuned β\beta falls roughly as 1/ρl1/\rho_l, so the effective strength at the optimum stays roughly constant. Over gpt2-medium layers 10–20, where the optimum lies within our grid, the interpolated best β\beta (Table 21) times ρl\rho_l is between 0.0034 and 0.0059. Dividing its geometric mean by ρl\rho_l predicts the best β\beta at two layers we did not fit on: 0.0013 for gpt2-medium layer 22 and 0.0021 for gpt2-small layer 10, where the sweeps in Tables 3 and 5 find 0.00125 and 0.0025.

At every layer in Table 21, the grid value nearest the predicted β\beta is also that layer's best entry in Table 3. So the per-layer best in Table 3 is already the strength-matched comparison, and it still favours later layers: NLL falls from 4.094 at layer 2 to 4.040 at layer 22 in gpt2-medium, and from 4.137 to 4.106 (layers 2 to 10) in gpt2-small. At the early layers the predicted β\beta is slightly above the largest value in the layer sweep (0.005), and NLL is still falling there, by up to 0.016 per doubling of β\beta. Even if it kept falling at that rate for one more doubling, these layers would remain at least 0.01 worse than the late layers. Table 21 leaves out the embedding layer and the final post-LayerNorm state, where ρl\rho_l is below 0.001 and about 0.17, so the matched β\beta lies far outside our grid.

Q2: Do the depth preferences and CC’s additional gains hold in a modern architecture? Does the improvement extend to an external text distribution?

CC’s added value remains unclear beyond the tested setting: the main evaluation covers one data source and one architecture family, and at 20B tokens CC improves BBC’s NLL only from 2.7876 to 2.7858. The main evaluation covers only GPT-2 models and one data source. Testing a modern LLM architecture with RoPE, RMSNorm, and gated MLPs would help establish whether the depth findings and gains extend beyond GPT-2. A matched comparison with STP [3] is also missing; the adjacent-state cosine baseline does not implement STP’s three-point objective.

A2: Thank you for the suggestion. Among the architectural differences highlighted by the reviewer—RoPE, RMSNorm, and gated MLPs—we prioritized RoPE during the rebuttal because positional encoding directly changes how token positions enter the attention mechanism and is therefore most closely related to both BBC and CC. We added experiments on Pythia-410M, which uses RoPE, reusing the gpt2-medium hyperparameters without retuning. At 1B training tokens, BBC improves NLL by 0.58% and BBC+CC by 0.83%. We also sweep the fraction of head dimensions using RoPE and find that the default rotary_pct=0.25 performs best. These results provide initial evidence that the gains are not specific to GPT-2's absolute positional embeddings. The results are reported in Section 6 (line 482–490) and Appendix A.9 (line 1328–1382).

We have not yet isolated RMSNorm or gated MLPs, nor evaluated on an external text distribution, and agree that these are valuable extensions. Following the same minimal-intervention principle, we prioritized RoPE because it has the clearest mechanistic connection to BBC and CC, rather than changing several architectural components simultaneously.

We also want to clarify the 20B-token result. We believe the reduced incremental gain of BBC+CC over BBC at intermediate long-training checkpoints is at least partly a numerical-stability issue rather than evidence that the CC effect disappears with scale. We introduce adaptive dropout to avoid repeatedly applying numerically small CC updates; with this modification, BBC+CC outperforms BBC at every intermediate checkpoint, and its final improvement is comparable to the additional gain observed at the 1B-token horizon.

Finally, STP is not a directly matched pretraining baseline for our setting. LLM-JEPA and STP were designed around joint-embedding prediction between semantically structured portions of supervised examples (for fine-tuning), where the data provide an explicit question–answer relationship; STP relaxes this structure through intermediate-point prediction but its formulation and experiments rely on semantically structured question–answer partitions, whereas our pretraining setting is continuous text without an externally defined partition. A faithful STP comparison would therefore require introducing an additional segmentation and target-construction design choice. The cosine-distance baseline in our paper is intended only as a control for CC's local geometric regularization, not as an implementation of STP.

Q3: For the theoretical connection, does the trained BBC layer satisfy Proposition 4’s low-energy condition, and does the bound track the measured error for multiplicity-corrected attention?

A3: No on both counts, and the revision now says so (Appendix B.3, Table 19). We took the queries, keys and values at the BBC layer (layer 2, segment length K=16K=16) of gpt2-medium and Pythia-410M (rotary embeddings), both trained for 1B tokens, on 100k held-out tokens. We evaluated the bound on these keys and values directly instead of going through the hidden states and the constants cKc_K and cVc_V (for Pythia-410M we add the rotary term of Appendix C.9), so it is never looser than the stated version.

The low-energy condition does not hold. Inside a segment, the values change between consecutive tokens about as much as in a random reordering of the same tokens, and the keys are only somewhat smoother in gpt2-medium and barely at all in Pythia-410M. With that much path energy, the bound allows logit gaps of about 150 nats within a segment, so its tanh⁡\tanh term saturates and the bound is no better than the trivial one for every query.

The bound also does not track the measured error of multiplicity-corrected attention. Even if we skip the path energy and plug in the measured Δs\Delta_s and Δv\Delta_v (the largest logit and value gaps between a segment and its boundary token), the bound is a median 8.8 times the error for gpt2-medium and 57 times for Pythia-410M, and its within-head rank correlation with the error is 0.11 and −0.02-0.02 (median over heads). The logit gaps themselves are large: Δs\Delta_s has median 10.8 and 12.6 nats, so tanh⁡(Δs)≈1\tanh(\Delta_s)\approx1 and the bound is close to Δv+2Vmax⁡\Delta_v+2V_{\max} for almost every query, whatever the actual error. For the BBC mask itself, without the multiplicity correction, the error is explained by a simpler quantity, the share mm of dense attention that falls on the masked keys: exactly, o−oBBC=m(uˉM−oBBC)o-o_{\rm BBC}=m(\bar u_M-o_{\rm BBC}), where uˉM\bar u_M is the attention-weighted mean of the masked values, and mm has within-head rank correlation 0.46 (gpt2-medium) and 0.94 (Pythia-410M) with this error.

This check changed how we present Proposition 4. We now state it only as a qualitative result, that the error vanishes as the within-segment path energy vanishes, and we changed the abstract, Section 5 and the conclusion to match.

Reviewer_xobUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work explores small LM pretraining and contributes 2 independent architectural changes to improve next token prediction. They contribute Boundary Bottleneck Concept (BBC) -- a position based mask with a hyperparmeter K . They contribute Continuous concept (CC) loss which encourages the distance between neighbouring embeddings in a specific layer to be small. These are only applied at a single layer each. The paper provides proofs, hyperparmater tuning, extra experiments . These changes improve NLL in language modelling pretraining.

Critical strengths
  1. Simple architectural and regularization contributions. BBC and CC introduce relatively simple modifications to language-model pretraining that improve next-token prediction performance. The reported NLL improvements suggest that these inductive biases may be useful.

  2. Multiple random seeds, baseline tuning and proofs. The methodology includes tuning the vanilla GPT-2 baseline before evaluating BBC and CC, and reporting results across three random seeds with means and standard deviations and providing theoretical proofs.

  3. Investigation of scaling and longer training horizons. The paper evaluates both GPT-2 Small (124M parameters) and GPT-2 Medium (355M parameters), investigating whether improvements persist through 20B training tokens. This exceeds the approximate Chinchilla compute-optimal training budgets of 2.5B tokens for GPT-2 Small and 7.1B tokens for GPT-2 Medium, providing evidence that the improvements persist over longer training horizons.

Critical weaknesses
  1. Lack of coherent motivation and connection between contributions. The paper presents numerous theoretical proofs, ablations and additional experiments, but often fails to explain why they are necessary or how they connect to the central research question. This makes the work feel disjointed.

  2. Insufficient explanation of the underlying mechanisms. Although the paper provides many experiemnts, these do not form a sufficiently clear explanation of why BBC and CC improve NLL, particularly why their effectiveness depends on the intervention layer.

  3. Unclear generalisability and broader research implications. The experiments demonstrate improvements across two GPT-2 model sizes, but require different hyperparameter configurations and single-layer interventions. It remains unclear whether these findings generalise to other architectures or downstream applications, and what broader principles the research community should take away from the results.

Questions for authors

This work left the reader with many questions:

Questions to Authors

  1. Theoretical motivation (Sections 2–5): What is the purpose of the theoretical propositions, and how do they connect to the empirical improvements observed for BBC and CC? In particular, do the propositions give us some guarantees or info about the hyperparamters we need to use? Or which layer this should work?

  2. Coding convention (Section 2): What exactly does the coding convention Γ\Gamma represent? Does it include the encoder-decoder mechanism, model parameters or another shared specification? How should the structure and residual description lengths be interpreted?

  3. BBC implementation (Section 3): Is BBC implemented entirely through a modified causal attention mask, or does it require additional architectural changes? Does the implementation reduce computational cost, or only the number of permitted attention connections?

  4. Boundary carrier selection (Section 3): Is the boundary carrier determined exclusively by the fixed segment length KK? Why use fixed positional boundaries rather than learned or content-dependent boundaries?

  5. Layer selection (Section 6, Figure 2): Are BBC and CC evaluated independently at each candidate layer?

  6. Single-layer intervention (Section 6): Why are BBC and CC applied at only one Transformer layer each? Have the authors considered applying either method across multiple layers or learning the intervention locations dynamically?

  7. Hyperparameter selection (Section 6): Why do the optimal segment length KK and regularisation strength β\beta differ between GPT-2 Small and GPT-2 Medium? Is there a principled way to select these hyperparameters for different model sizes?

  8. Underlying mechanisms (Section 6, Figure 2): Why does BBC perform better in earlier Transformer layers while CC performs better in later layers? What evidence supports the proposed interpretation that different layers favour different representational functions?

  9. Statistical significance (Section 6, Figure 3): standard deviations are reported across the three seeds but how many? is it 1 2 or 3?

  10. Figure 3 interpretation (Section 6): Does the percentage improvement in Figure 3b refer to evaluation NLL? Is the vanilla baseline trained for the same number of tokens at every comparison point?

  11. Chinchilla scaling (Section 6): Going to 20B tokesn for a small model is overkill. Have you tried just keeping the Chinchilla-style compute-optimal scaling, particularly the approximate budgets of 2.5B tokens for GPT-2 Small and 7.1B tokens for GPT-2 Medium then for different architectures

  12. CC ablations (Section 6, Figure 4): What motivates the inclusion of the CC-2 regularisation variant, and what does the scale quantity in Figure 4b represent?

  13. Related methods (Introduction and Related Work): How does BBC compare with existing approaches such as MEGABYTE and gist tokens? What advantages does BBC offer over these alternatives?

  14. Computational reproducibility (Section 6 and Reproducibility Statement): Could the authors report the hardware used for training, the approximate GPU-hours required for the main experiments, and the computational cost of the hyperparameter sweeps?

6 comments
AuthorsOpenReview ↗

Response (1/4)

W1: Lack of coherent motivation and connection between contributions: The paper presents numerous theoretical proofs, ablations and additional experiments, but often fails to explain why they are necessary or how they connect to the central research question. This makes the work feel disjointed.

R1: Thank you for this feedback. Our central question is whether the structure of internal representations should be an explicit inductive bias rather than being left entirely to next-token prediction. BBC (an attention mask) and CC (a smoothness loss) are two independent tests of this question, each with a single parameter, KK or β\beta, that sets the balance between structure and residual detail. The theory states what each method constrains and gives a testable rule for β\beta (A1, A7). The ablations rule out simpler explanations of the gains, such as sparsity alone or a single component of the CC loss. The representation diagnostics check that each method changes the representation as intended (Table 1, Figure 6). We added a short roadmap at the end of Section 2 to make these connections explicit.

W2: Insufficient explanation of the underlying mechanisms: Although the paper provides many experiemnts, these do not form a sufficiently clear explanation of why BBC and CC improve NLL, particularly why their effectiveness depends on the intervention layer.

R2: We agree that a complete mechanism is still missing. In the revision we rule out the simplest explanation of each depth preference. For CC, later layers remain better even when each layer is compared at its own best β\beta, so the preference is not an artifact of the coefficient. For BBC, the preference is not explained by how much attention the mask removes. Details are in A8.

W3: Unclear generalisability and broader research implications: The experiments demonstrate improvements across two GPT-2 model sizes, but require different hyperparameter configurations and single-layer interventions. It remains unclear whether these findings generalise to other architectures or downstream applications, and what broader principles the research community should take away from the results.

R3: We added Pythia-410M, which uses rotary position embeddings (Table 17). With the gpt2-medium hyperparameters and no retuning, BBC improves NLL by 0.58% and BBC+CC by 0.83%. The hyperparameter differences between the two model sizes are small and partly explained by our theory (A7), and the single-layer design is discussed in A6. We have not evaluated downstream tasks; at this scale we report held-out NLL. The broader takeaway is that the layer at which a representation bias is applied matters. In our sweeps, the choice of layer changes NLL at least as much as KK or β\beta, and BBC and CC prefer opposite ends of the network.

Q1: Theoretical motivation (Sections 2–5): What is the purpose of the theoretical propositions, and how do they connect to the empirical improvements observed for BBC and CC? In particular, do the propositions give us some guarantees or info about the hyperparamters we need to use? Or which layer this should work?

A1: The propositions do not guarantee lower NLL. For each method, they pin down the split between structure and residual and show which parameter acts on it. At the BBC layer, a completed segment reaches later segments only through its boundary state, while the current segment keeps its token-level detail. KK sets this split, and K=1K=1 and K≥NK\geq N both give the vanilla model (Proposition 1), so any gain has to come from an intermediate KK. In CC, the penalty itself bounds the high-frequency residual of the hidden-state sequence and the bits needed to describe it to a fixed precision (Propositions 2 and 3), and β\beta is its weight. In a simple estimation model, the best β\beta is positive but finite (Proposition 12). The KK and β\beta sweeps are approximately U-shaped, consistent with this (Figure 3).

The theory also gives one hyperparameter rule that we could test. To first order, the effective strength of CC at layer ll is βρl\beta\rho_l, where ρl\rho_l is the ratio of the CC to the NTP gradient norm (Appendix C.8). If the best strength is the same at every layer, the best β\beta should scale as 1/ρl1/\rho_l; this rule predicts the best β\beta at the layers we use in both models (A7). For KK, Proposition 1 and Lemma 2 describe exactly which keys the mask removes, but they do not locate the optimum, which we find empirically (Figure 3). The theory does not select the layer either. Proposition 4, which connects CC and BBC, is informative only when the representation changes little from one token to the next within a segment. On trained BBC layers this does not hold (Appendix B.3), so we now present this result as a qualitative connection.

Q2: Coding convention (Section 2): What exactly does the coding convention (Γ\Gamma) represent? Does it include the encoder-decoder mechanism, model parameters or another shared specification? How should the structure and residual description lengths be interpreted?

A2: Γ\Gamma is everything the encoder and decoder agree on before coding, so it never has to be transmitted; a fixed decoding procedure only adds a constant. For BBC, Γ\Gamma consists of the vocabulary, KK and the structure map ϕ\phi; for CC, it consists of the dimensions, the path-graph basis, the cutoff, the energy budget, the precision and the codebook. It does not include the model parameters: our bounds concern explicit codes, and anything outside Γ\Gamma would have to be paid for in bits. The structure length is the number of bits needed to specify the structure (the segment label ϕ(S)\phi(S) for BBC, the low-frequency component HmH_m for CC), and the residual length is the length of an index that identifies the actual realization given that structure. Both are upper bounds on explicit codes that motivate the methods; training does not minimize them directly.

Q3: BBC implementation (Section 3): Is BBC implemented entirely through a modified causal attention mask, or does it require additional architectural changes? Does the implementation reduce computational cost, or only the number of permitted attention connections?

A3: BBC is implemented entirely by replacing the causal attention mask at one layer with the mask in Equation 1; no parameters, tokens or modules are added. Our implementation computes attention densely and then applies the mask, so its cost is the same as the vanilla model. A sparse kernel could compute only the kept query–key pairs (13% for K=8K=8 and 8% for K=16K=16 at N=1024N=1024), but because only one layer is affected, this would save at most about 1% of training FLOPs.

Q4: Boundary carrier selection (Section 3): Is the boundary carrier determined exclusively by the fixed segment length (K)? Why use fixed positional boundaries rather than learned or content-dependent boundaries?

A4: Yes, the carrier is the last position of each segment, so it is determined by KK alone and nothing is learned. Under the causal mask, the last position is the only one that has seen the entire segment. We use fixed positions so that any gain can be attributed to the routing restriction itself rather than to a learned boundary predictor. Carrier placement does matter: placing each carrier at a random position within its segment, with the same attention budget, is significantly worse (3.702 vs. 3.684 NLL on gpt2-small at 300M tokens, p=0.011p=0.011, Table 8).

Q5: Layer selection (Section 6, Figure 2): Are BBC and CC evaluated independently at each candidate layer?

A5: Yes, they are independently evaluated at each candidate layer.

Q6: Single-layer intervention (Section 6): Why are BBC and CC applied at only one Transformer layer each? Have the authors considered applying either method across multiple layers or learning the intervention locations dynamically?

A6: As mentioned in the introduction, our design is motivated by the view that different Transformer layers serve different functions and may therefore benefit from different balances between structure and residual detail. We added tests applying BBC to consecutive layers, such as layers 1, 2, and 3, but found performance similar to applying it only at layer 2. Since the goal of this paper is to establish that structure–residual inductive biases (SRIB) are beneficial and exhibit a layer preference, we believe that single-layer interventions combined with layer sweeps provide the clearest presentation. This follows our broader experimental philosophy: use the minimal intervention needed to isolate the SRIB effect before introducing additional architectural complexity.

We did not explore dynamically learned intervention locations because fixed locations already provide sufficient evidence that SRIB is beneficial and depth-dependent.

K=8K = 8K=16K = 16K=32K = 32
l=24.101 ±\pm 0.0024.098 ±\pm 0.0074.102 ±\pm 0.002
l=1,2,34.102 ±\pm 0.0014.096 ±\pm 0.0034.092 ±\pm 0.002

Q7: Hyperparameter selection (Section 6): Why do the optimal segment length (K) and regularisation strength (β\beta) differ between GPT-2 Small and GPT-2 Medium? Is there a principled way to select these hyperparameters for different model sizes?

A7: Thank you for the valuable feedback. We expanded the sweeps over both KK and β\beta. The new results (Figure 3, lines 352–371) show a broad optimal region for KK. More importantly, the favorable ranges are consistent across gpt2-small and gpt2-medium: K∈[8,32]K\in[8,32] and β∈[0.00125,0.0025]\beta\in[0.00125,0.0025].

This suggests that these hyperparameters may not require precise model-size-specific tuning, at least across the two model scales we study. It also means that the KK used in our original gpt2-small experiments was not the exact optimum, but it still lies in a regime sufficient to demonstrate the benefit of SRIB.

The remaining difference in the best β\beta is expected from the rule in A1. On our vanilla checkpoints, ρl\rho_l is 2.20 at the gpt2-small layer we use and 3.54 at the gpt2-medium layer, so gpt2-medium needs a smaller β\beta for the same effective strength. Fitting one value of βρl\beta\rho_l to the interpolated optima on gpt2-medium layers 10–20 (Table 21) predicts β=0.0013\beta=0.0013 for gpt2-medium and 0.00210.0021 for gpt2-small, in line with the sweep optima of 0.001250.00125 and 0.00250.0025 (Tables 3 and 5).

Q8: Underlying mechanisms (Section 6, Figure 2): Why does BBC perform better in earlier Transformer layers while CC performs better in later layers? What evidence supports the proposed interpretation that different layers favour different representational functions?

A8: We do not yet have a complete mechanism. For CC, a fixed β\beta acts more strongly at later layers, because the gradient ratio ρl\rho_l grows from 0.83 at gpt2-medium layer 2 to 3.54 at layer 22 (Table 21). However, when each layer is compared at its own best β\beta, NLL still improves with depth (from 4.094 at layer 2 to 4.040 at layer 22, Table 3), so the preference is not an artifact of the coefficient (details in our reply to Reviewer rZq2, Q1). For BBC, we checked whether early layers attend locally, so that the mask removes little there. They do not: at layer 2 of our vanilla checkpoints, the mask with K=8K=8 would remove 83% of the attention to tokens other than the first, and across layers the removed attention does not explain where BBC works best (Appendix C.6, Table 20). The paper only suggests that different depths serve different functions; it is a hypothesis we have not tested directly.

Q9: Statistical significance (Section 6, Figure 3): standard deviations are reported across the three seeds but how many? is it 1 2 or 3?

A9: It is 3 (as in "three seeds"). As noted in line 377, statistical significance is evaluated using paired tt-tests, with the corresponding pp-values reported in Tables 6 and 7 of Appendix A.3.

Q10: Figure 3 interpretation (Section 6): Does the percentage improvement in Figure 3b refer to evaluation NLL? Is the vanilla baseline trained for the same number of tokens at every comparison point?

A10: Yes. The improvement is the relative reduction in held-out evaluation NLL, 100×(NLLvanilla−NLLmethod)/NLLvanilla100\times(\mathrm{NLL}_{\rm vanilla}-\mathrm{NLL}_{\rm method})/\mathrm{NLL}_{\rm vanilla} (Figure 4b in the revision). At each point, the vanilla baseline and the method are trained on the same number of tokens, with the same learning-rate schedule and the same three seeds, so each comparison is paired. The corresponding numbers are in Tables 6 and 7 (Appendix A.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.