- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper derives a fully explicit, reference-centered covering-number generalization bound for modern pre-normalized decoders such as those in the Llama family. It proves a sharp threshold beyond which the optimized bound must equal its range cap, so tightening covering constants alone cannot help. On pretrained checkpoints the bound is vacuous, and the product of per-layer Jacobian estimates far exceeds the directly estimated end-to-end Jacobian. The authors conclude that the obstruction lies in stagewise multiplicative transport and call for direction-aware composition bounds.
Critical strengths
- A precise vacuity threshold. Instead of a vague claim that the bound is loose, it identifies exactly when the optimized bound must become vacuous.
- Mechanism-level diagnosis. It separates reference-centering, transport, and entropy, and audits each one to locate where the bound fails.
- Exceptionally thorough reproducibility. The materials include hashes, frozen protocols, and FP32 reruns.
Critical weaknesses
- Narrow scope. The conclusions apply only to covering/Dudley-type proofs, while PAC-Bayes and compression-based approaches (Lotfi et al., 2024) already obtain nonvacuous bounds for LLMs. The practical impact of showing that this route fails is therefore limited.
- No positive result. The paper argues that a direction-aware transport theory is needed but does not provide such a bound.
- Reliability of the composition gap. Both the stagewise product and the composite Jacobian norm are finite-search lower estimates, so the reported gap is not guaranteed in either direction.
Questions for authors
-
Dependence of the gap on depth. The gap appears larger for models with more sublayers. Does it grow roughly linearly with depth?
-
The large layer-2 feedforward factor. Why does the layer-2 feedforward map have the largest single-stage factor?
5 comments
Reviewer wLUj — Part 2/3: Certified alignment comparisons and a matched depth study
W3. Rigorous guarantees for the product-to-composition gap.
The revision adds analytic and interval-certified comparisons of stage products and composed sensitivity. Proposition V.1 (p. 60) pairs verified feasible-direction stage lower bounds with a certified composition upper bound :
\ge \frac{\prod_s\underline\sigma_s}{\overline C}.$$ Appendix V develops compatible block-norm bounds and arithmetic-error accounting. The exact construction proves a uniform alignment gap algebraically. The trained seed/depth study verifies a ratio greater than one in all 1,280 reference model/input cases. | Decoder blocks | Median across seeds of inputwise median certified ratio | |---|---:| | 1 | 1.771 | | 2 | 4.652 | | 3 | 16.842 | | 4 | 31.695 | The displayed values are rounded downward. This study covers five seeds at each depth and supplies certified reference-point sensitivity comparisons. See Table 22 and Appendix X.4 (p. 66). For the pretrained numerical audit, independent dense FP64 calculations validate 36 selected raw-feedforward, branch, and residual comparisons across 12 record/layer/token cases, with maximum relative discrepancy $5.69\times10^{-7}$. Independent derivative-action checks have maximum relative error $2.22\times10^{-14}$. See Appendix Y.3 (p. 69). **Q1. Does the observed gap increase approximately linearly with depth?** Yes. The six-prefix study of the frozen TinyLlama checkpoint shows an approximately linear increase in the paired log-gap. All prefixes use the same 64 frozen records, providing a matched-input comparison. The numerical norm-search results are: | Residual stages | Stage-product log\$_{10}\$ | Composition log\$_{10}\$ | Paired log\$_{10}\$ gap | |---|---:|---:|---:| | 2 | 2.091 | 1.415 | 0.632 | | 4 | 3.726 | 1.481 | 2.204 | | 8 | 9.274 | 3.069 | 6.208 | | 16 | 16.460 | 3.069 | 13.402 | | 32 | 30.159 | 3.099 | 27.107 | | 44 | 40.575 | 3.658 | 36.844 | Every column reports a median across records; the paired-gap column takes the median after computing each record's difference. The gap increases at all five adjacent-depth comparisons for all 64 records: 320 increases in 320 comparisons. Recordwise linear fits have median slope 0.866 log-base-ten units per residual stage and median $R^2=0.99944$. The largest absolute fit residual is 1.15 log-base-ten units. Together, these results quantify the approximately linear trend over the tested grid. See Section 5.5 and Figure 2(a) (p. 10), and Appendix Y.1 (p. 67). **Search-budget checks supporting the depth comparison.** On four prespecified records, we increased the composite search from 12 iterations and 3 restarts to 24 iterations and 6 restarts. The largest composite increase is $2.44\times$, driven by additional restarts; extending the original three restarts to 24 iterations changes each tested prefix estimate by less than 1%. After strengthening both stage and composition searches, the full-depth paired gaps remain 37.45–39.13 decades. See Table 24 (p. 68).Reviewer wLUj — Part 3/3: Localizing the feedforward outlier and improving the presentation
Q2. Why is the “layer-2” feedforward factor so large?
We have completed an all-token decomposition of the relevant residual map. With the zero-based layer indexing used in the audit, layer 2 is decoder block 3, and its feedforward residual map is residual stage 6.
The peak is attained at position zero in all 64 evaluation records. At that same position, the median component estimates are:
| Component at block 3, position 0 | Median estimate |
|---|---|
| Learned RMS-normalization Jacobian | 6.634 |
| Raw SwiGLU Jacobian at the normalized state | 108.134 |
| Composed branch Jacobian | 421.608 |
| Residual Jacobian | 421.543 |
This decomposition identifies a large initial-position SwiGLU derivative, amplified through composition with normalization. The close agreement between the branch and residual values places the large scale in the branch. For comparison, block 1 has a larger maximum normalization estimate, with median 64.95, while block 3's distinctive feature is its raw feedforward derivative at position zero. Across all 8,128 noninitial positions in block 3, the raw SwiGLU estimate is at most 1.049.
Token-level localization and independent checks.
Recomputing the token-maximum summary of the saved Jacobian measurements over positions 1–127 reduces block 3's median maximum residual estimate from 421.54 to 5.86, with recordwise values between 4.53 and 7.85. Applying this same summary to every block places another block at the maximum in every record, further localizing the exceptional factor to position zero.
The 64 records contain 35 distinct first-token IDs. Selected dense FP64 calculations reproduce the exceptional block-3 magnitude. Together, the decomposition, position summaries, and dense checks give a concrete account of where the observed factor arises in this checkpoint. See Section 5.5 and Figure 2(b) (p. 10), Appendix Y.2 and Table 25 (p. 68), and Appendix Y.3 (p. 69).
Presentation changes addressing your assessment.
The introduction now explains the fixed-reference learning problem and uses a one-varying-matrix example to distinguish transport from entropy. The main text develops the explicit bound and threshold, the complete coefficient budget, the composed theorem, and the positive certificate in sequence. The experimental section then presents the trained certificates and the pretrained depth and token-position measurements with direct pointers to their protocols.
The most relevant places to reassess are Sections 1–2 (pp. 1–4), Section 3.2 (p. 5), Section 4 (pp. 6–7), and Sections 5.3–5.5 (pp. 9–10). Full derivations and reproducibility details are retained in the appendix.
Your questions led to the positive composition theorem, certified alignment comparisons, and the matched depth and feedforward investigations. We would be grateful if you would reconsider your assessment in light of these theoretical, experimental, and expository additions, and update your score accordingly.
Reviewer wLUj — Part 1/3: Positive direction-aware bounds and their relation to existing certificates
Thank you for your questions about scope, constructive results, and numerical reliability, and for recognizing the reproducibility effort. We have made the positive direction central to the revision and completed the requested depth and feedforward studies.
W1. The contribution alongside nonvacuous PAC-Bayes and compression results.
Section 1.1 (p. 2) and Appendix L (pp. 34–36) discuss the nonvacuous language-model results of Lotfi and collaborators and compare the corresponding statistical targets and complexity descriptions. Our constructive contribution develops verifiable improvements to uniform generalization guarantees over continuous parameter neighborhoods around a fixed reference decoder.
The revised paper provides two constructive answers. Theorem R.1 (p. 44) gives a finite-dimensional entropy envelope, logN∞(G,ϵ)≤qlog(1+2S/ϵ), with logarithmic sensitivity dependence. It can improve the certificate while retaining the original transport products. Separately, Theorems 4.1–4.2 improve the transport calculation itself. The sharp quadratic-envelope threshold and these two refinements together identify how improvements to entropy and transport translate into improved statistical certificates.
W2. A positive, parameter-uniform composed bound.
Theorem 4.1 (p. 6) combines certified reference Jacobian products with uniform derivative remainders. A finite state-tube test verifies the conditions for all parameters in the prescribed class and all proper-cover hybrids. Theorem 4.2 (p. 7) includes the scalar loss in the composition and carries the resulting sensitivity into the generalization bound. Appendix S (pp. 46–53) gives the proof and finite verification procedures.
The exact causal RMS/SwiGLU construction uses alternating shear Jacobians whose compositions cancel, together with a causal first-attention branch that makes the loss nonconstant over the allowed parameter. It establishes
K_{\mathrm{comp}}<0.002830.$$ At $N=10^5$, the original correction is capped, the composed correction is below $0.240$, and the composed total-risk certificate is below $0.597<\log 2$ with joint confidence at least 95%. This is a uniform comparison on the same one-parameter class using the same quadratic entropy envelope. The true product-to-composition Jacobian ratio exceeds $10^{20.08}$ throughout that class. See Section 4.1 (p. 7) and Proposition T.1 (p. 53). Theorem S.6 (p. 50) and Corollary S.7 (p. 52) provide a complementary construction in which every matrix can vary over a ball with nonempty interior. With per-matrix radii proportional to $D^{-2}$ and the stated uniform curvature conditions, the composed complexity term is $O(D^{-1/2}\log D)$, while the old coefficient can grow exponentially on the same class. **Uniform validation on trained models.** We also enclose two independently varying entries in five trained two-block decoders. All 25 parameter squares have completed interval enclosures over all 64 input sequences. | Square radius | Composed risk upper bounds across seeds | Below $\log 4$ | |---|---:|---:| | $10^{-6}$ | 0.8998–0.9078 | 5/5 | | $10^{-5}$ | 0.9206–0.9341 | 5/5 | | $10^{-4}$ | 0.9912–1.0614 | 5/5 | | $10^{-3}$ | 1.6550–5.1047 | 0/5 | | $10^{-2}$ | 5.1047–5.1047 | 0/5 | A fresh sample of 100,000 independent sequences supplies a joint 95% event across the squares, matched one-entry controls, and reference Hoeffding bounds. The loss Jacobian has rank two in the selected directions at every reference. Composition improves the stagewise volumetric upper bound in 19 of 25 squares; complete matched comparisons appear in Table 20 (p. 66). See Section 5.4 and Table 3 (p. 9), and Appendix X (pp. 64–67).