- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces System Behavior Dynamics, a Bayesian framework that treats system-level factors as latent variables in language modeling. The authors argue that autoregressive causality introduces an additional approximation error, termed the Causality Tax, and propose Green Shell as an alternative variational formulation. The framework is evaluated through perturbation analysis and NTK spectral comparisons with autoregressive and block-diffusion models.
Critical strengths
The paper attempts to incorporate system-level factors into probabilistic language modeling and proposes a concrete GSH formulation. The comparisons across multiple datasets provide some preliminary observations, although they do not establish the central theoretical claims.
Critical weaknesses
-
The proposed Causality Tax is not convincingly established, since autoregressive factorization itself does not restrict the representation of arbitrary joint distributions. Moreover, Lemma 4.2 infers a strict approximation-error advantage from upper bounds that do not establish such an ordering.
-
The evaluation relies primarily on NTK spectral metrics without adequately establishing their connection to the claimed approximation-error bounds or actual generalization performance. GSH also changes the training objectives and procedures, which makes it difficult to attribute the observed improvements specifically to the relaxation of causal dependencies.
Questions for authors
-
Why should autoregressive factorization necessarily introduce an additional approximation error? How do the upper bounds in Lemma 4.2 establish that non-causal models achieve strictly smaller errors?
-
Table 1 mainly reports NTK SNR and MLC. Is there any direct evidence that GSH actually improves language modeling performance or generalization compared with AR and block diffusion?
-
Since GSH also changes the training objectives, how can we tell whether the improvements come from removing causal dependencies rather than the different optimization procedure?
2 comments
Response to Reviewer 8aJm [Part 2/2]: Disentangling Causal Relaxation from Optimization Procedures
This section details the extensive ablation study resolving Question 3: whether performance gains stem from removing causal dependencies or from the specific optimization procedure.
1. Q3: Systematic Ablation of Objectives vs. Causal Structure
To determine whether performance gains stem from removing causal dependencies or from the optimization procedure, we refer directly to the ablation study in Appendix C.2 (Section C.2 and Figure 8).
We evaluated six architectural and objective configurations under identical conditions:
- Standard GSH: Non-causal structure with Green-kernel continuous loss and mixed-target discrete renormalization.
- MSE+S2: Non-causal structure with standard MSE loss replacing the Green-kernel objective.
- S1+CE: Non-causal structure with standard Cross-Entropy replacing mixed-target discrete renormalization.
- S1-only: Continuous Green-kernel training only.
- S2-only: Discrete mixed-target training only.
- CE-only: Pure standard Cross-Entropy loss applied to the non-causal architecture.
2. Key Findings and Structural Isolation
The empirical results from this ablation isolate the impact of the non-causal architecture from the loss formulation:
- Non-Causal Architecture Drives the Primary Gain: When the custom Green-kernel and mixed-target objectives are completely stripped away and the non-causal model is trained using only standard Cross-Entropy (CE-only) or standard Mean Squared Error (MSE-only), the model still achieves high NTK SNR (>20 dB). It maintains a substantial spectral advantage over causal AR (12.8 dB) and BD (15.8 dB) baselines. This confirms that relaxing the sequential dependency chain is the foundational factor driving tighter error bounds.
- The Specific Role of the Green-Kernel Functional: The layer-wise Green-kernel objective functions as a regularizer across spectral modes. In the CE-only setting, signal energy concentrates into dominant modes, yielding higher Main Lobe Concentration (MLC). The Green-kernel objective distributes energy across intermediate and peripheral frequencies, reducing MLC to 0.72-0.73 and suppressing high-frequency noise bulk.
- Continuous and Discrete Coupling: Comparing S1-only (continuous only) and S2-only (discrete only) shows that pure continuous training achieves extreme noise suppression (SNR > 30 dB) but suffers from high concentration (MLC > 0.95), whereas pure discrete training achieves rich peripheral features (MLC < 0.7) but weaker noise suppression (SNR < 16 dB). Standard GSH achieves the optimal balance by coupling continuous field regularization with discrete renormalization.
3. Summary of Evaluation Metrics: Lazy Regime vs. Feature Learning
- Lazy Regime (Section 5, Table 1): The NTK spectral evaluation was conducted under the lazy-training regime () specifically to maintain theoretical correspondence with UAT approximation bounds via localized Rademacher complexity. In this controlled regime, variations in SNR isolate architectural inductive biases from optimizer heuristics.
- Feature Learning Regime (Part 1, Table 1): To confirm that these spectral advantages translate into real-world generative performance, end-to-end training within the established Block Diffusion pipeline confirms that GSH achieves a validation perplexity of 7.34, vastly superior to AR (11.76) and BD (9.33).
Response to Reviewer 8aJm [Part 1/2]: Core Takeaways, End-to-End Validation, and Causality Tax
We provide point-by-point responses below. Part 1 summarizes our primary empirical validations, presents the end-to-end language modeling benchmark, and resolves the theoretical formulation of the Causality Tax (Q1, Q2).
1. Primary Empirical Evidence: End-to-End Language Modeling and Depth Scaling
The review questions whether there is direct evidence that GSH improves language modeling performance or generalization compared with AR and block diffusion. We resolve this directly through empirical evidence outside the lazy regime.
GSH was directly ported into the identical training pipeline of Block Diffusion, preserving the exact data loading, batching, evaluation protocol, and optimization schedules, completely outside the lazy-training regime. All architectures were trained under identical parameter capacity (~119M parameters, 6 layers, hidden dimension 768, 12 heads) on the complete TinyStories dataset and evaluated on the full validation set.
GSH trains stably and achieves a validation perplexity of 7.34, outperforming both AR (11.76) and BD (9.33) by a large margin. This demonstrates that tighter approximation bounds translate directly into superior end-to-end density estimation and generalization.
In addition, we highlight the architectural scalability enabled by GSH. Because GSH optimizes layers independently without inter-layer backpropagation, activation memory is strictly O(1) with respect to network depth L. Using default PyTorch Lightning directly from the Block Diffusion codebase without infrastructure tricks (no CPU offloading, host-device paging, or virtual memory swapping), we scaled GSH to 128 layers (approximately 10B parameters) on a single commodity RTX 4090 GPU (24GB VRAM), with the training memory footprint remaining virtually identical to the 110M model. This confirms that relaxing causal constraints yields both empirical modeling superiority and extreme practical scalability.
2. Q1: Autoregressive Factorization Error and Strict Ordering in Lemma 4.2
1. Why Autoregressive Factorization Imposes Approximation Restrictions: The reviewer correctly notes that for observed data marginals p(X), autoregressive factorization p(X)=∏t=1Tp(xt∣x<t) is an identity by the chain rule of probability.
However, SBD does not model data marginals in isolation. SBD formulates the joint distribution p(Y,S∣X) of data Y and internal execution states S={s1,…,sT}. When internal representation states are forced to satisfy a causal forward Markov chain:
p(S∣X)=∏t=1Tp(st∣s<t,X)
this is no longer an identity. It enforces conditional independence constraints st⊥s>t∣(s<t,X) across latent states. In an unconstrained system, bidirectional dependencies and global boundary conditions exist naturally across representation layers. Constraining the variational distribution to forward causal factorizations confines the hypothesis class Ωcausal to a strict proper subset of the general variational family Ωgen (Ωcausal⊊Ωgen).
2. How Strict Suboptimality is Established: The suboptimality of the causal family is established by Lemma 4.1 through monotonicity of the supremum:
maxq∈ΩgenELBO(q)≥maxq∈ΩcausalELBO(q)
Because Ωcausal is a strictly constrained subset of Ωgen, the supremum over Ωcausal cannot exceed that over Ωgen. Whenever the target joint distribution contains bidirectional correlations, the causal constraint is active and the inequality is strict:
maxq∈ΩgenELBO(q)>maxq∈ΩcausalELBO(q)
Lemma 4.2 complements this variational supremum by analyzing function approximation under finite parameter capacity. Forcing the network onto the causal constraint manifold Mcausal={F:∑t=1Ttt(X)=0} incurs an additive projection distance. By the triangle inequality:
∥Fcausal−f∥≤∥Fcausal−(f+∑t=1Ttt)∥+∥∑t=1Ttt∥≤ϵ+∑t=1T∣tt(X)∣
The term ∑t=1T∣tt(X)∣ quantifies the exact functional penalty required to maintain the causal chain.
Please refer to Part 2 for the ablation study disentangling causal relaxation from optimization procedures (Q3).