acceptodds
Under review as a conference paper at ICLR 2027

Causality is What You may Downgrade If You Look Beyond Data Distribution

Abstract

We show that the causality of language models (LMs) may not be necessary nor optimal. This is the case when system behavior (denoted as ) is incorporated as a first-principle Bayesian feature. Here, refers to extra dominant factors beyond the data space, and it especially captures the coupled effects of algorithms, hardware, runtime, and language. Despite being the de facto foundation of modern architecture, recent studies indicate persistent mismatches and contradictions with causality. These issues largely stem from system behavior rather than the data distribution. We therefore propose the System Behavior Dynamics (SBD) framework, which incorporates as an irreducible component of the evidence lower bound (ELBO). SBD theoretically reveals a counter-intuitive Causality Tax phenomenon, where causality emerges as a suboptimal approximation with an additional structural error, due to the obliviousness to . To address the challenge of latent variable analysis, we validate the SBD-predicted impact of via implicit measurements, theoretical-bound-guided controls, and Neural Tangent Kernel (NTK) evaluations. In particular, we construct Green Shell (GSH) to show the possibility of reducing Causality Tax. GSH is a non-causal mean-field variational family, and it replaces the sequential dependency chain of components with a divide-and-conquer partition. NTK spectra in the lazy-training regime confirm that GSH always achieves significantly tighter error bounds than causality, with (or more) improvement in signal-to-noise ratio. In the relatively later stage of lazy-training, GSH further leads to superior generalization (up to richer multi-scale fitting capabilities). Taken together, SBD establishes system behavior as a complementary theoretical abstraction besides causality and distribution fitting, opening new research avenues such as designing and optimizing LM base models.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Approximation Theory, Bayesian Modeling, Language Models, Latent Variable Models, Variational Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8aJmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces System Behavior Dynamics, a Bayesian framework that treats system-level factors as latent variables in language modeling. The authors argue that autoregressive causality introduces an additional approximation error, termed the Causality Tax, and propose Green Shell as an alternative variational formulation. The framework is evaluated through perturbation analysis and NTK spectral comparisons with autoregressive and block-diffusion models.

Critical strengths

The paper attempts to incorporate system-level factors into probabilistic language modeling and proposes a concrete GSH formulation. The comparisons across multiple datasets provide some preliminary observations, although they do not establish the central theoretical claims.

Critical weaknesses
  • The proposed Causality Tax is not convincingly established, since autoregressive factorization itself does not restrict the representation of arbitrary joint distributions. Moreover, Lemma 4.2 infers a strict approximation-error advantage from upper bounds that do not establish such an ordering.

  • The evaluation relies primarily on NTK spectral metrics without adequately establishing their connection to the claimed approximation-error bounds or actual generalization performance. GSH also changes the training objectives and procedures, which makes it difficult to attribute the observed improvements specifically to the relaxation of causal dependencies.

Questions for authors
  1. Why should autoregressive factorization necessarily introduce an additional approximation error? How do the upper bounds in Lemma 4.2 establish that non-causal models achieve strictly smaller errors?

  2. Table 1 mainly reports NTK SNR and MLC. Is there any direct evidence that GSH actually improves language modeling performance or generalization compared with AR and block diffusion?

  3. Since GSH also changes the training objectives, how can we tell whether the improvements come from removing causal dependencies rather than the different optimization procedure?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8aJm [Part 1/2]: Core Takeaways, End-to-End Validation, and Causality Tax

We provide point-by-point responses below. Part 1 summarizes our primary empirical validations, presents the end-to-end language modeling benchmark, and resolves the theoretical formulation of the Causality Tax (Q1, Q2).

1. Primary Empirical Evidence: End-to-End Language Modeling and Depth Scaling

The review questions whether there is direct evidence that GSH improves language modeling performance or generalization compared with AR and block diffusion. We resolve this directly through empirical evidence outside the lazy regime.

GSH was directly ported into the identical training pipeline of Block Diffusion, preserving the exact data loading, batching, evaluation protocol, and optimization schedules, completely outside the lazy-training regime. All architectures were trained under identical parameter capacity (~119M parameters, 6 layers, hidden dimension 768, 12 heads) on the complete TinyStories dataset and evaluated on the full validation set.

Model ArchitectureParametersSequence / Block SetupValidation Perplexity (PPL) ↓\downarrow
Autoregressive (AR)~119MToken-by-token11.76
Block Diffusion (BD)~119MIterative denoising9.33
Green Shell (GSH)~119MNon-causal field7.34

GSH trains stably and achieves a validation perplexity of 7.34, outperforming both AR (11.76) and BD (9.33) by a large margin. This demonstrates that tighter approximation bounds translate directly into superior end-to-end density estimation and generalization.

In addition, we highlight the architectural scalability enabled by GSH. Because GSH optimizes layers independently without inter-layer backpropagation, activation memory is strictly O(1)O(1) with respect to network depth LL. Using default PyTorch Lightning directly from the Block Diffusion codebase without infrastructure tricks (no CPU offloading, host-device paging, or virtual memory swapping), we scaled GSH to 128 layers (approximately 10B parameters) on a single commodity RTX 4090 GPU (24GB VRAM), with the training memory footprint remaining virtually identical to the 110M model. This confirms that relaxing causal constraints yields both empirical modeling superiority and extreme practical scalability.

2. Q1: Autoregressive Factorization Error and Strict Ordering in Lemma 4.2

1. Why Autoregressive Factorization Imposes Approximation Restrictions: The reviewer correctly notes that for observed data marginals p(X)p(X), autoregressive factorization p(X)=∏t=1Tp(xt∣x<t)p(X) = \prod_{t=1}^T p(x_t \mid x_{<t}) is an identity by the chain rule of probability.

However, SBD does not model data marginals in isolation. SBD formulates the joint distribution p(Y,S∣X)p(Y, S \mid X) of data YY and internal execution states S={s1,…,sT}S = \{s_1, \dots, s_T\}. When internal representation states are forced to satisfy a causal forward Markov chain:

p(S∣X)=∏t=1Tp(st∣s<t,X)p(S \mid X) = \prod_{t=1}^T p(s_t \mid s_{<t}, X)

this is no longer an identity. It enforces conditional independence constraints st⊥s>t∣(s<t,X)s_t \perp s_{>t} \mid (s_{<t}, X) across latent states. In an unconstrained system, bidirectional dependencies and global boundary conditions exist naturally across representation layers. Constraining the variational distribution to forward causal factorizations confines the hypothesis class Ωcausal\Omega_{\mathrm{causal}} to a strict proper subset of the general variational family Ωgen\Omega_{\mathrm{gen}} (Ωcausal⊊Ωgen\Omega_{\mathrm{causal}} \subsetneq \Omega_{\mathrm{gen}}).

2. How Strict Suboptimality is Established: The suboptimality of the causal family is established by Lemma 4.1 through monotonicity of the supremum:

max⁡q∈ΩgenELBO(q)≥max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) \ge \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q)

Because Ωcausal\Omega_{\mathrm{causal}} is a strictly constrained subset of Ωgen\Omega_{\mathrm{gen}}, the supremum over Ωcausal\Omega_{\mathrm{causal}} cannot exceed that over Ωgen\Omega_{\mathrm{gen}}. Whenever the target joint distribution contains bidirectional correlations, the causal constraint is active and the inequality is strict:

max⁡q∈ΩgenELBO(q)>max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) > \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q)

Lemma 4.2 complements this variational supremum by analyzing function approximation under finite parameter capacity. Forcing the network onto the causal constraint manifold Mcausal={F:∑t=1Ttt(X)=0}\mathcal{M}_{\mathrm{causal}} = \{F : \sum_{t=1}^T t_t(X) = 0\} incurs an additive projection distance. By the triangle inequality:

∥Fcausal−f∥≤∥Fcausal−(f+∑t=1Ttt)∥+∥∑t=1Ttt∥≤ϵ+∑t=1T∣tt(X)∣\|F_{\mathrm{causal}} - f\| \le \|F_{\mathrm{causal}} - (f + \sum_{t=1}^T t_t)\| + \|\sum_{t=1}^T t_t\| \le \epsilon + \sum_{t=1}^T |t_t(X)|

The term ∑t=1T∣tt(X)∣\sum_{t=1}^T |t_t(X)| quantifies the exact functional penalty required to maintain the causal chain.

Please refer to Part 2 for the ablation study disentangling causal relaxation from optimization procedures (Q3).

AuthorsOpenReview ↗

Response to Reviewer 8aJm [Part 2/2]: Disentangling Causal Relaxation from Optimization Procedures

This section details the extensive ablation study resolving Question 3: whether performance gains stem from removing causal dependencies or from the specific optimization procedure.

1. Q3: Systematic Ablation of Objectives vs. Causal Structure

To determine whether performance gains stem from removing causal dependencies or from the optimization procedure, we refer directly to the ablation study in Appendix C.2 (Section C.2 and Figure 8).

We evaluated six architectural and objective configurations under identical conditions:

  1. Standard GSH: Non-causal structure with Green-kernel continuous loss and mixed-target discrete renormalization.
  2. MSE+S2: Non-causal structure with standard MSE loss replacing the Green-kernel objective.
  3. S1+CE: Non-causal structure with standard Cross-Entropy replacing mixed-target discrete renormalization.
  4. S1-only: Continuous Green-kernel training only.
  5. S2-only: Discrete mixed-target training only.
  6. CE-only: Pure standard Cross-Entropy loss applied to the non-causal architecture.

2. Key Findings and Structural Isolation

The empirical results from this ablation isolate the impact of the non-causal architecture from the loss formulation:

  • Non-Causal Architecture Drives the Primary Gain: When the custom Green-kernel and mixed-target objectives are completely stripped away and the non-causal model is trained using only standard Cross-Entropy (CE-only) or standard Mean Squared Error (MSE-only), the model still achieves high NTK SNR (>20 dB). It maintains a substantial spectral advantage over causal AR (12.8 dB) and BD (15.8 dB) baselines. This confirms that relaxing the sequential dependency chain is the foundational factor driving tighter error bounds.
  • The Specific Role of the Green-Kernel Functional: The layer-wise Green-kernel objective functions as a regularizer across spectral modes. In the CE-only setting, signal energy concentrates into dominant modes, yielding higher Main Lobe Concentration (MLC). The Green-kernel objective distributes energy across intermediate and peripheral frequencies, reducing MLC to 0.72-0.73 and suppressing high-frequency noise bulk.
  • Continuous and Discrete Coupling: Comparing S1-only (continuous only) and S2-only (discrete only) shows that pure continuous training achieves extreme noise suppression (SNR > 30 dB) but suffers from high concentration (MLC > 0.95), whereas pure discrete training achieves rich peripheral features (MLC < 0.7) but weaker noise suppression (SNR < 16 dB). Standard GSH achieves the optimal balance by coupling continuous field regularization with discrete renormalization.

3. Summary of Evaluation Metrics: Lazy Regime vs. Feature Learning

  • Lazy Regime (Section 5, Table 1): The NTK spectral evaluation was conducted under the lazy-training regime (∥Δθ∥≪∥θ0∥\|\Delta\theta\| \ll \|\theta_0\|) specifically to maintain theoretical correspondence with UAT approximation bounds via localized Rademacher complexity. In this controlled regime, variations in SNR isolate architectural inductive biases from optimizer heuristics.
  • Feature Learning Regime (Part 1, Table 1): To confirm that these spectral advantages translate into real-world generative performance, end-to-end training within the established Block Diffusion pipeline confirms that GSH achieves a validation perplexity of 7.34, vastly superior to AR (11.76) and BD (9.33).
Reviewer_4KHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a Bayesian framework incorporating system behavior into language modeling and claims that autoregressive causality incurs an additional approximation error (Causality Tax). It proposes Green Shell (GSH) to reduce this penalty and reports improved NTK spectral metrics over autoregressive and block-diffusion baselines across four datasets.

Critical strengths
  1. Interesting methodological direction. The GSH formulation does provide a concrete, physics-inspired approach to non-causal modeling within token blocks. The proposed layer-wise Green-kernel objectives offer a potentially useful alternative to conventional autoregressive and diffusion-based training.

  2. Consistent empirical improvements. GSH achieves higher NTK SNR across four datasets, supported by additional ablation and sensitivity studies.

Critical weaknesses

Weaknesses

  1. Questionable theoretical validity of the Causality Tax. The derivation of Lemma 4.2 contains fundamental issues, which undermine the paper's central theoretical claim. (Please see Q1–Q3 below.)

  2. Insufficient theoretical justification for GSH & Unjustified NTK SNR proxy. The Green-shell decomposition does not inherently imply the proposed mean-field independence assumption, nor does it establish that GSH avoids the claimed Causality Tax. On top of that, the claimed relationship between NTK SNR and approximation error is insufficiently established. Hence, the reported spectral improvements do not necessarily support the claimed tighter approximation bounds. (Please also see Q4 below.)

Questions for authors

Q1. Conditional independence assumption in Eq. (10) (Appendix B).

Could the authors justify why Eq. (10) is a valid characterization of AR factorization? From my understanding, standard autoregressive (AR) factorization does not require the current state to be conditionally independent of future states given the past. If this is an additional assumption, it should be explicitly stated, and its implications for the generality of Lemma 4.2 should be clarified.

Q2. Inconsistency between Eq. (10) and the residual in Eq. (11).

Now even if we accept Eq. (10), the residual defined in Eq. (11) does not appear to properly characterize the corresponding conditional independence constraint. Specifically, Eq. (10) conditions on both past states and the input XX, whereas Eq. (11) conditions only on XX. The residual integrand in Eq. (11) can therefore remain nonzero even when Eq. (10) holds. Could the authors clarify how Eq. (11) quantifies the violation of Eq. (10)?

Q3. Justification of the Causality Tax bound in Eq. (12).

Even assuming Eqs. (10) and (11) are valid, it is unclear how Eq. (12) follows from the Universal Approximation Theorem (UAT). Why should the approximation error admit the proposed additive decomposition involving the residual terms? A looser upper bound in Eq. (13) does not necessarily imply a larger actual approximation error. Could the authors provide a rigorous derivation supporting the claimed strict suboptimality of the causal family?

Q4. NTK SNR as a proxy for approximation error.

Could the authors provide more justification for the claimed relationship between NTK SNR and the UAT approximation error? Why is splitting the NTK spectrum into signal and noise based on eigenvalue ordering sufficient? Under what assumptions does Eq. (35) hold across different model architectures?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4KHB [Part 1/2]: Core Empirical Validations and Conditional Independence

We provide point-by-point responses to the theoretical questions and technical inquiries below. Part 1 summarizes our primary empirical validations and resolves the conditional independence formulation (Q1, Q2).

1. Primary Empirical Evidence: End-to-End Validation and Depth Scaling

To verify that theoretical error bound improvements translate directly into real language modeling performance outside the lazy regime, GSH was directly ported into the identical training pipeline of Block Diffusion without altering data processing, batching, or optimization routines.

We evaluated 119M-parameter models on the complete TinyStories validation set:

Model ArchitectureParametersSequence / Block SetupValidation Perplexity (PPL) ↓\downarrow
Autoregressive (AR)~119MToken-by-token11.76
Block Diffusion (BD)~119MIterative denoising9.33
Green Shell (GSH)~119MNon-causal field7.34

The superior NTK SNR of GSH directly corresponds to a substantially lower validation perplexity in full end-to-end training (7.34 vs. 9.33 and 11.76).

Furthermore, we highlight the architectural scalability enabled by the backpropagation-free nature of GSH. Because layers are optimized independently without inter-layer backpropagation, activation memory is strictly O(1)O(1) with respect to total depth LL. Using default PyTorch Lightning directly from the Block Diffusion codebase without infrastructure tricks (no CPU offloading, host-device paging, or virtual memory swapping), we scaled GSH to 128 layers (approximately 10B parameters) on a single commodity RTX 4090 GPU (24GB VRAM), observing a training memory footprint virtually identical to the 110M model. This demonstrates both generative density superiority and practical computational scalability.

2. Q1: Conditional Independence in Eq. (10) (Appendix B)

In standard autoregressive modeling, generation is formalized as a directed graphical model where internal generation states S={s1,…,sT}S = \{s_1, \dots, s_T\} are generated sequentially:

p(S∣X)=∏t=1Tp(st∣s<t,X)p(S \mid X) = \prod_{t=1}^T p(s_t \mid s_{<t}, X)

In this forward generative prior, local transition p(st∣s<t,X)p(s_t \mid s_{<t}, X) cannot depend on future states s>ts_{>t} that have not yet been generated. In terms of conditional probability, this requires:

p(st∣s<t,s>t,X)=p(st∣s<t,X)p(s_t \mid s_{<t}, s_{>t}, X) = p(s_t \mid s_{<t}, X)

which corresponds to the conditional independence statement st⊥s>t∣(s<t,X)s_t \perp s_{>t} \mid (s_{<t}, X) in Eq. (10).

For an unconstrained joint distribution p(S∣X)p(S \mid X), the true posterior allows bidirectional dependencies where future context provides informative feedback to earlier states. Imposing st⊥s>t∣(s<t,X)s_t \perp s_{>t} \mid (s_{<t}, X) is the defining structural restriction that confines general joint representations into forward causal factorizations. We have clarified this distinction between the unconstrained joint posterior and the forward generative prior in the revision.

3. Q2: Conditioning Discrepancy Between Eq. (10) and Residual in Eq. (11)

The reviewer correctly identifies a notation omission in Eq. (11) of Appendix B. The integrand in Eq. (11) was written as:

∫∣p(st,s>t∣X)−p(st∣X)p(s>t∣X)∣ dX\int |p(s_t, s_{>t} \mid X) - p(s_t \mid X)p(s_{>t} \mid X)| \, dX

which conditioned only on XX and omitted explicit conditioning on past states s<ts_{<t}.

The mathematically exact formulation that quantifies the violation of Eq. (10) integrates the conditional discrepancy over the full history (s<t,X)(s_{<t}, X):

tt(AR)(X)=∫p(s<t∣X)(∬∣p(st,s>t∣s<t,X)−p(st∣s<t,X)p(s>t∣s<t,X)∣ dst ds>t)ds<tt_t^{\text{(AR)}}(X) = \int p(s_{<t} \mid X) \left( \iint \left| p(s_t, s_{>t} \mid s_{<t}, X) - p(s_t \mid s_{<t}, X) p(s_{>t} \mid s_{<t}, X) \right| \, ds_t \, ds_{>t} \right) ds_{<t}

Under this corrected formulation:

  1. When Eq. (10) holds, p(st,s>t∣s<t,X)=p(st∣s<t,X)p(s>t∣s<t,X)p(s_t, s_{>t} \mid s_{<t}, X) = p(s_t \mid s_{<t}, X) p(s_{>t} \mid s_{<t}, X) holds almost everywhere, and the inner integral vanishes, yielding tt(AR)(X)=0t_t^{\text{(AR)}}(X) = 0.
  2. When non-causal bidirectional dependencies exist such that future states cannot be conditionally factored out given the past, the integrand is strictly positive, yielding tt(AR)(X)>0t_t^{\text{(AR)}}(X) > 0.

We have updated Eq. (11) with this rigorous conditional integral.

Please refer to Part 2 for derivations of Q3 (strict suboptimality and UAT bound) and Q4 (NTK proxy justifications).

AuthorsOpenReview ↗

Response to Reviewer 4KHB [Part 2/2]: Causality Tax Derivation, NTK Spectral Proxy, and Ablations

This section provides derivations for the Causality Tax bounds (Q3) and theoretical justifications of the NTK Signal-to-Noise Ratio proxy (Q4).

1. Q3: Justification of Causality Tax and Strict Suboptimality

We address both supremum suboptimality and the additive UAT bound:

1. Strict Suboptimality via Lemma 4.1 (Monotonicity of Supremum): Suboptimality is established in Lemma 4.1 through monotonicity of the supremum:

max⁡q∈ΩgenELBO(q)≥max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) \ge \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q)

Because the causal variational family Ωcausal\Omega_{\mathrm{causal}} is constrained by sequential factorization p(S∣X)=∏t=1Tp(st∣s<t,X)p(S \mid X) = \prod_{t=1}^T p(s_t \mid s_{<t}, X), it forms a strict proper subset of the unconstrained family Ωgen\Omega_{\mathrm{gen}} (Ωcausal⊊Ωgen\Omega_{\mathrm{causal}} \subsetneq \Omega_{\mathrm{gen}}). By Rudin (Principles of Mathematical Analysis), the supremum over a strict subset cannot exceed the supremum over the full set. Whenever the true posterior exhibits non-causal dependencies, the constraint is active and inequality is strict: max⁡q∈ΩgenELBO(q)>max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) > \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q).

2. Additive Functional Bound in Lemma 4.2: In function space under finite capacity, approximating target f(X)f(X) subject to causal constraint manifold Mcausal={F:∑t=1Ttt(X)=0}\mathcal{M}_{\mathrm{causal}} = \{F : \sum_{t=1}^T t_t(X) = 0\} projects the solution onto a restricted subspace. By triangle inequality on function norms:

∥Fcausal−f∥≤∥Fcausal−(f+∑t=1Ttt)∥+∥∑t=1Ttt∥≤ϵ+∑t=1T∣tt(X)∣\|F_{\mathrm{causal}} - f\| \le \|F_{\mathrm{causal}} - (f + \sum_{t=1}^T t_t)\| + \|\sum_{t=1}^T t_t\| \le \epsilon + \sum_{t=1}^T |t_t(X)|

Here, ϵ\epsilon represents intrinsic UAT error of the unconstrained family, and ∑t=1T∣tt(X)∣\sum_{t=1}^T |t_t(X)| is the functional penalty enforcing conditional independence at every stage. Lemma 4.1 establishes strict suboptimality in ELBO, while Lemma 4.2 quantifies the constructive functional projection error.

2. Q4: Justification of NTK SNR as a Proxy for Approximation Error

1. Spectral Partitioning via Random Matrix Theory: Following RMT and the Marchenko-Pastur law (Martin & Mahoney, 2021; Gavish & Donoho, 2014), the Gram matrix spectrum separates into:

  • A continuous noise bulk bounded by the theoretical Marchenko-Pastur upper edge λ+\lambda_+, representing unregularized high-frequency variance.
  • Isolated outlier eigenvalues (spikes) above λ+\lambda_+, representing dominant low-rank task features. Dividing the spectrum into signal power PsigP_{sig} and noise bulk PnoiseP_{noise} isolates structured representations from unstructured noise.

2. Connection to UAT Error via Localized Rademacher Complexity: Under localized Rademacher complexity theory (Bartlett & Mendelson, 2002; Caponnetto & De Vito, 2007), excess risk and approximation error ϵ\epsilon of kernel regression scale with noise leakage:

ϵ∝C⋅PnoiseN⋅Psig=C⋅1N⋅SNR\epsilon \propto C \cdot \sqrt{\frac{P_{noise}}{N \cdot P_{sig}}} = C \cdot \sqrt{\frac{1}{N \cdot \text{SNR}}}

where CC is an architectural constant and NN is sample size. Consequently, higher NTK SNR directly corresponds to a tighter approximation bound.

3. Controlled Assumptions Across Architectures: As detailed in Appendix C.1, all compared models (AR, BD, GSH) were evaluated under controlled conditions preserving NTK validity: (1) identical 150M parameter capacity and embeddings, (2) identical dataset subsets and sample size NN, and (3) fixed learning rate without adaptive scheduling, ensuring updates remain in the lazy regime (∥Δθ∥≪∥θ0∥\|\Delta\theta\| \ll \|\theta_0\|).

3. Theoretical Justification for Green Shell Mean-Field Formulation

  • Spatial Partitioning vs. Temporal Factorization: In physics, concentric Green shells divide continuous field potentials into discrete equipotential surfaces. The interior field is fully specified by boundary potentials without temporal transitions. In GSH, each layer ll represents a spatial shell conditioned on clean boundaries (Eq. 25).
  • Mean-Field Independence: The joint distribution factorizes as q(Y,S)=∏i=0L−1q(si)q(yi∣si)q(Y, S) = \prod_{i=0}^{L-1} q(s^i) q(y^i \mid s^i). Because each layer optimizes local field functionals independently without sequential recursive dependencies q(si∣si−1)q(s^i \mid s^{i-1}), the architecture instantiates a mean-field spatial partition.
  • Spectral Confirmation: In Table 1, GSH achieves significantly higher NTK SNR (22.5 to 32.2 dB vs. 12.8 to 19.5 dB for AR) and lower Main Lobe Concentration (MLC: 0.72 vs. 0.94), confirming that relaxing the causal chain suppresses noise bulk and distributes signal energy evenly across modes.
Reviewer_EmNHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to model extrinsic confounders as part of a variational framework, and by applying it to language modeling, demonstrates that causal modeling incurs an intrinsic penalty on the evidence lower bound. Drawing from plasma physics and electromagnetic field modeling, authors introduce a (partially) non-causal architecture that demonstrates the potential capability to fit language distributions.

Critical strengths

An ambitious project introducing a high-level framework drawing from an impressive breadth of sources, spanning several decades of work in physics, probability, language modeling, model serving, algorithmic analysis, LLM interpretability, diffusion modeling, zeroth-order optimization, online error correction, transistor engineering, and even the treatment of narcissistic personality disorder.

Critical weaknesses

There’s discussion to be had on the rigor of the framework, the validity of the results, and the mode of presentation. But none of that matters in the face of a more fundamental epistemological flaw. In my view, a unifying theoretical framework is only a valuable contribution to the field if it can do one of two things: either it advances understanding in the field by answering a previously unanswered question, or by pointing out an unexpected new finding, or it guides us in a principled manner toward an otherwise unintuitive novel approach that works well. I don’t think SBD is accomplishing either.

On the “advances understanding” side: the modeling of “system information” S under a variational framework adds no new understanding of the confounding behaviors embodied by S, as S is a catch-all for every possible source of error, known and unknown, not arising from the data distribution or model capacity. It is claimed that incorporating S in whole or in part can reduce modeling error, but this is tautological and clearly true by construction. Conversely, the incorporation of S into a variational framework says little about variational frameworks, as we already know that additional variables can be incorporated in this way, and again S is a general catch-all rather than a specific behavioral mode. Similarly, the “discovery” of a causality tax is not new or surprising: to my knowledge, it is not a commonly held belief that causality is necessary or optimal for language modeling. From a strictly theoretical modeling standpoint, of course we should leverage future information when available. But we default to causal language models because they admit a straightforward and practical inference implementation and demonstrate desirable local-token-conditioning. Decoders are practical, not theoretically optimal.

On the “novel approach” side: concrete details on the architecture setup are scattered through appendix, but as best I can tell, the proposed GSH architecture is a tiny, 4-layer block-diffusion model that, instead of denoising iteratively, instead predicts five tokens simultaneously in a single forward pass directly from noise. It is trained on 4 million tokens using a custom layer-wise time-blurred latent reconstruction objective, transitioning into cross entropy loss with label smoothing. To what degree this architecture is novel, or exemplifies the stated non-causal, variational, and/or Green-Shell-based framings, is debatable. But more importantly, the paper gives no indication that the model even works. Based on what’s shown, I have no reason to believe that the GSH generative language model can actually generate language, which I consider the bare minimum for a proposed generative architecture.

In the absence of 1) answering an existing question or offering a fundamentally new finding or 2) demonstrating basic functionality in a downstream derived architecture, I cannot consider SBD a meaningful academic contribution.

Questions for authors

See weaknesses - what new insight does SBD offer as a unifying framework? And is the GSH architecture capable of producing realistic language?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EmNH [Part 1/3]: Executive Summary, Core Takeaways, and End-to-End Validation

We provide a direct, point-by-point response below. Part 1 summarizes core arguments, presents primary empirical validations, and resolves the reviewer's fundamental questions.

1. The Core Empirical Validation: End-to-End Language Modeling

The review questions whether Green Shell (GSH) "even works" or "can actually generate language." We resolve this through direct empirical evidence.

GSH was directly ported into the identical training pipeline of Block Diffusion (BD3), preserving the exact data loading, batching, evaluation protocol, and optimization schedules, completely outside the lazy-training regime. All models were trained under identical parameter budgets (~119M parameters, 6 layers, hidden dimension 768, 12 heads) on the complete TinyStories dataset and evaluated on the full validation set.

Model ArchitectureParametersSequence / Block SetupValidation Perplexity (PPL) ↓\downarrow
Autoregressive (AR)~119MToken-by-token11.76
Block Diffusion (BD)~119MIterative denoising9.33
Green Shell (GSH)~119MNon-causal field7.34

GSH converges stably and achieves a validation perplexity of 7.34, outperforming both the AR baseline (11.76) and the BD baseline (9.33) by a substantial margin. This confirms that GSH functions effectively as a generative language model and achieves superior density estimation within the established Block Diffusion pipeline.

2. Architectural Scalability: 10B Parameters on a Single Consumer GPU

The review speculates that GSH is restricted to tiny architectures. In reality, GSH possesses a backpropagation-free property that enables extreme depth scaling.

  • O(1) Activation Memory: Because GSH decouples layers into independent spatial field optimizations, activations are not cached across depth for backpropagation. Peak activation memory is strictly O(1)O(1) with respect to network depth LL.
  • 10B Scale-Up on a Single RTX 4090: Using default PyTorch Lightning directly from the Block Diffusion codebase without infrastructure tricks (no CPU offloading, host-device paging, or virtual memory swapping), we scaled GSH to 128 layers (approximately 10B parameters) on a single commodity GPU (RTX 4090, 24GB VRAM). The training memory footprint of this 10B setup remained virtually indistinguishable from the 110M model.
  • Why the NTK Model Was Kept Compact: When plotting empirical NTK spectra, the evaluated model was intentionally kept at a controlled scale (150M parameters, 2560 hidden dimension). Computing the empirical NTK Gram matrix and executing full eigenvalue decomposition via PyTorch (torch.linalg.eigh) causes GPU out-of-memory (OOM) failures if the model or sample size is large. This constraint reflects PyTorch eigensolver memory limits, not a capacity ceiling of GSH.

3. Executive Summary: Core Stances on Epistemology and Causality

  • SBD is Not a Tautological Catch-All: System behavior SS captures concrete physical and runtime realities (hardware execution jitter, runtime non-determinism, operator irreversibility, speculative lookahead) that directly affect generation paths. SBD derives a formal, decoupled ELBO comprising Token Alignment (TA), System Response to Input/Output (SRIO), and Approximation Internal Semantics (AIS). This isolates where modeling errors originate, which data-only variational frameworks cannot accomplish.
  • The Causality Tax is Not Trivial: In generative modeling, autoregressive factorization p(X)=∏t=1Tp(xt∣x<t)p(X) = \prod_{t=1}^T p(x_t \mid x_{<t}) is universally regarded as exact for data marginals by the probability chain rule. SBD reveals that once internal computational states SS are jointly modeled with tokens, imposing a causal DAG on SS restricts the variational family to a strict proper subset (Ωcausal⊊Ωgen\Omega_{\mathrm{causal}} \subsetneq \Omega_{\mathrm{gen}}). This restriction induces an inescapable lower bound penalty on the ELBO (Lemma 4.1) and an additive approximation error penalty ∑t=1T∣tt(X)∣\sum_{t=1}^T |t_t(X)| under UAT (Lemma 4.2).

4. Direct Answers to Reviewer Questions

  • What new insight does SBD offer as a unifying framework? SBD proves that causal sequential generation is a suboptimal inductive bias when accounting for internal computational dynamics. It provides an analytical decomposition of variational lower bounds under system factors and introduces an actionable blueprint for non-causal, spatially partitioned generative architectures.
  • Is the GSH architecture capable of producing realistic language? Yes. As established by the benchmark above, GSH trains stably and achieves a validation perplexity of 7.34, outperforming both AR (11.76) and BD (9.33).

Please refer to Part 2 for algorithmic pseudo-code and Part 3 for in-depth theoretical derivations.

AuthorsOpenReview ↗

Response to Reviewer EmNH [Part 2/3]: Algorithmic Pipeline, Pseudo-code, and Scalability

This section details the algorithmic implementation of Green Shell (GSH), presents formal pseudo-code, and analyzes its scaling properties.

1. Algorithmic Formulation of Layer-wise Training

Layers in GSH are decoupled and trained sequentially. The outer loop iterates through layers, while the inner loop iterates through data batches. Intermediate layers optimize a masked Green-kernel diversity loss, while the final layer optimizes masked cross-entropy jointly with the language modeling head.

Algorithm: Layer-wise Backpropagation-Free Training of Green Shell (GSH)
Input: Dataset D, vocabulary V, total layers L, block size B, max epochs E
Output: Optimized parameters {theta_0, theta_1, ..., theta_{L-1}, theta_proj, theta_head}

1: Initialize layer parameters theta_l for each layer l in [0, L-1], and output projection theta_proj, theta_head
2: for layer index l = 0 to L-1 do
3:     Initialize optimizer Opt_l for parameters theta_l (if l == L-1, include theta_proj and theta_head)
4:     for epoch = 1 to E do
5:         for each batch (Prompt_IDs, Target_IDs) in D do
6:             Extract clean embeddings: E_prompt, E_target
7:             Generate shell noise on demand strictly for required indices {0, l}
8:             Sample interleaved binary block mask M across sequence blocks
9:             Expand mask M to token dimension as Mask_tokens
10:            Construct base residual state:
                   X_raw_target = Mask_tokens * E_target + (1 - Mask_tokens) * Shell_Noise[0]
                   X_raw = Concat([E_prompt, X_raw_target], dim=seq)
11:            Construct layer-specific input state:
                   H_mix = Mask_tokens * E_target + (1 - Mask_tokens) * Shell_Noise[l]
                   H_in = Concat([E_prompt, H_mix], dim=seq)
12:            Forward pass through layer l only:
                   H_pred = Layer_l(tgt=H_in, x_raw=X_raw, intra_block_attn=Full, inter_block_attn=Causal)
13:            if l < L - 1 then
14:                Compute masked Green-kernel diversity loss:
                       Loss = Loss_GreenKernel(H_pred_target, E_target, mask=M)
15:            else
16:                Project to vocabulary logits: Logits = Head(Proj(H_pred))
17:                Compute masked cross-entropy loss:
                       Loss = Loss_CrossEntropy(Logits, Target_IDs, mask=Mask_tokens)
18:            end if
19:            Opt_l.zero_grad()
20:            Loss.backward()  // Gradients computed strictly within layer l
21:            Opt_l.step()
22:            Explicitly deallocate intermediate states and noise tensors
23:        end for
24:    end for
25:    Freeze layer parameters theta_l
26: end for

2. Distinction from Block Diffusion

The review categorizes GSH as a variant of block diffusion. The two architectures differ in their fundamental modeling principles:

  • Generation Dynamics: Block diffusion relies on an iterative, multi-step Gaussian denoising Markov chain over temporal diffusion steps within each block. GSH replaces temporal sequential corruption with a single-step spatial field reconstruction.
  • Spatial Equipotential Partitioning: Inspired by Green's functions in field theory, GSH treats transformer layers as concentric equipotential shells. Information propagates spatially across depth rather than temporally across artificial noise steps.
  • Boundary Condition Reconstruction: GSH employs an interleaved reconstruction mask during training. Unmasked context blocks serve as physical boundary conditions that directly constrain the latent field of masked blocks, eliminating the need for iterative time-stepping.

3. Scaling Laws and Future Work

The backpropagation-free characteristic of GSH opens unique avenues for foundation model scaling:

  • Memory Invariance: Because memory does not scale with depth, increasing layer count from 6 to 128 requires no additional GPU VRAM for activations.
  • Scaling Law Open Questions: Standard autoregressive models conform to empirical power laws (e.g., Chinchilla scaling). For decoupled non-causal architectures like GSH, it remains an open theoretical question whether scaling behavior follows Chinchilla laws on a per-layer basis, conforms to them at the aggregate system level, or adheres to an entirely different scaling trajectory. Validating these trajectories across billions of tokens requires massive compute clusters, which exceeds our current academic resources. We leave full-scale empirical scaling curves as future work.
AuthorsOpenReview ↗

Response to Reviewer EmNH [Part 3/3]: Detailed Theoretical Framework and Formal Refutations

This section provides in-depth mathematical justifications for the System Behavior Dynamics (SBD) framework and formal derivations of the Causality Tax.

1. Mathematical Grounding of System Behavior Dynamics (SBD)

The review argues that incorporating system behavior SS into a variational framework adds no meaningful insight because additional variables can always be incorporated into ELBO. This overlooks how SS is structurally coupled with generation.

In deployed language models, execution trajectories are not determined solely by the static data distribution pdata(X)p_{\text{data}}(X). Computational factors, such as hardware execution non-determinism, memory access latency, operator irreversibility, and speculative decoding verification, actively alter the effective generation distribution. Standard Bayesian formulations ignore these factors, treating observed data XX as the sole probabilistic entity and casting all execution phenomena as extrinsic noise outside the lower bound.

SBD formalizes the joint distribution p(Y,S,X)p(Y, S, X) and derives the decoupled ELBO in Proposition 4.1:

ELBO=log⁡p(X)+Eq(Y,S)[log⁡p(Y∣X)]+Eq(Y,S)[log⁡p(S∣X,Y)−log⁡q(Y,S)]\text{ELBO} = \log p(X) + \mathbb{E}_{q(Y,S)}[\log p(Y \mid X)] + \mathbb{E}_{q(Y,S)}[\log p(S \mid X, Y) - \log q(Y, S)]

This structural decomposition separates optimization into three distinct terms:

  1. Token Alignment (TA): Measures surface data likelihood under system expectations.
  2. System Response to Input/Output (SRIO): Captures coherence between internal computational states and output tokens.
  3. Approximation Internal Semantics (AIS): Quantifies the variational entropy of internal representations, governing how the hypothesis class organizes latent approximations.

This decoupling provides an analytical mechanism to evaluate how structural constraints on SS impact overall approximation bounds, transforming unmodeled system variance into tractable variational objectives.

2. Formal Derivation of the Causality Tax

The review asserts that the suboptimality of causality is trivial because future information is helpful. This confuses discriminative sequence conditioning with generative distribution modeling. In generative modeling, autoregressive factorization is widely assumed to be exact and uncompromised. SBD proves this assumption fails in the joint representation space through two complementary results:

1. Strict ELBO Suboptimality (Lemma 4.1 via Monotonicity of the Supremum): Let Ωgen\Omega_{\mathrm{gen}} denote the unrestricted variational family, and let Ωcausal\Omega_{\mathrm{causal}} denote the variational family constrained by the causal factorization q(S∣X)=∏t=1Tq(st∣s<t,X)q(S \mid X) = \prod_{t=1}^T q(s_t \mid s_{<t}, X). Because causal factorization enforces directed conditional independence constraints across latent states, Ωcausal\Omega_{\mathrm{causal}} is a strict proper subset of Ωgen\Omega_{\mathrm{gen}} (Ωcausal⊊Ωgen\Omega_{\mathrm{causal}} \subsetneq \Omega_{\mathrm{gen}}).

By the monotonicity of the supremum (Rudin, Principles of Mathematical Analysis), the supremum of any functional over a strict subset cannot exceed the supremum over the full set:

max⁡q∈ΩgenELBO(q)≥max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) \ge \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q)

Whenever the true posterior distribution p(Y,S∣X)p(Y, S \mid X) contains bidirectional dependencies or spatial couplings across computational states, the causal constraint is strictly active, yielding max⁡q∈ΩgenELBO(q)>max⁡q∈ΩcausalELBO(q)\max_{q \in \Omega_{\mathrm{gen}}} \mathrm{ELBO}(q) > \max_{q \in \Omega_{\mathrm{causal}}} \mathrm{ELBO}(q). This proves strict suboptimality without relying on upper bound heuristics.

2. Universal Approximation Functional Bound (Lemma 4.2): Under finite parameter capacity, approximating target function f(X)f(X) subject to the causal constraint manifold Mcausal={F:∑t=1Ttt(X)=0}\mathcal{M}_{\mathrm{causal}} = \{F : \sum_{t=1}^T t_t(X) = 0\} projects the solution onto a restricted functional space. By the triangle inequality on function norms:

∥Fcausal−f∥≤∥Fcausal−(f+∑t=1Ttt)∥+∥∑t=1Ttt∥≤ϵ+∑t=1T∣tt(X)∣\|F_{\mathrm{causal}} - f\| \le \|F_{\mathrm{causal}} - (f + \sum_{t=1}^T t_t)\| + \|\sum_{t=1}^T t_t\| \le \epsilon + \sum_{t=1}^T |t_t(X)|

where ϵ\epsilon is the intrinsic approximation error of the unconstrained family, and ∑t=1T∣tt(X)∣\sum_{t=1}^T |t_t(X)| is the functional penalty required to enforce conditional independence at every order of the causal chain.

3. Purpose and Rigor of the Lazy-Training Regime

To evaluate theoretical approximation bounds via empirical Neural Tangent Kernel (NTK) spectra, parameter updates must stay within the linearization region (∥Δθ∥≪∥θ0∥\|\Delta\theta\| \ll \|\theta_0\|). Entering the feature-learning regime with massive token counts breaks the kernel correspondence, rendering NTK analysis invalid. The compact data volume guarantees mathematical rigor for the spectral bounds, while the end-to-end benchmark in Part 1 demonstrates real-world generative performance outside the lazy regime.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.