acceptodds
Under review as a conference paper at ICLR 2027

Reference-Centered Generalization Bounds for Modern Pre-Normalized Decoders: Explicit Constants and a Vacuity Threshold

Abstract

We derive an explicit reference-centered uniform generalization bound for pre-normalized decoders, including grouped-query attention, rotary maps, RMS normalization, and SwiGLU, and ask when such a bound can be nontrivial at all. The bound has two moving parts: a transport coefficient, which carries an early parameter perturbation through the residual stack, and a covering-entropy term. We prove a sharp threshold for the normalized stage coefficient of the truncated-Dudley objective: at or above it, the optimized certificate must equal its range cap, and in a stronger regime improvements in stage-cover constants drop out of the objective altogether. On a frozen public 1.1B checkpoint, a norm-matched Jacobian audit gives a median transport of for the product of separately estimated sublayer norms but only for the directly estimated composite. Three SmolLM2 checkpoints (135M–1.7B parameters) reproduce the separation, with median paired gaps of – decades against on the 1.1B reference; all operator norms in this audit are finite-search lower estimates. We find that, within this covering-and-transport proof structure, the obstruction sits in stagewise transport rather than in the cover constants, and our results point to direct, direction-aware composition bounds rather than further tightening of stagewise constants.

Primary area:
learning theory
Keywords:
transformer generalization, local generalization bounds, norm-based complexity, covering numbers, uniform convergence, trained neural networks, residual networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wLUjOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper derives a fully explicit, reference-centered covering-number generalization bound for modern pre-normalized decoders such as those in the Llama family. It proves a sharp threshold beyond which the optimized bound must equal its range cap, so tightening covering constants alone cannot help. On pretrained checkpoints the bound is vacuous, and the product of per-layer Jacobian estimates far exceeds the directly estimated end-to-end Jacobian. The authors conclude that the obstruction lies in stagewise multiplicative transport and call for direction-aware composition bounds.

Critical strengths
  • A precise vacuity threshold. Instead of a vague claim that the bound is loose, it identifies exactly when the optimized bound must become vacuous.
  • Mechanism-level diagnosis. It separates reference-centering, transport, and entropy, and audits each one to locate where the bound fails.
  • Exceptionally thorough reproducibility. The materials include hashes, frozen protocols, and FP32 reruns.
Critical weaknesses
  • Narrow scope. The conclusions apply only to covering/Dudley-type proofs, while PAC-Bayes and compression-based approaches (Lotfi et al., 2024) already obtain nonvacuous bounds for LLMs. The practical impact of showing that this route fails is therefore limited.
  • No positive result. The paper argues that a direction-aware transport theory is needed but does not provide such a bound.
  • Reliability of the composition gap. Both the stagewise product and the composite Jacobian norm are finite-search lower estimates, so the reported gap is not guaranteed in either direction.
Questions for authors
  • Dependence of the gap on depth. The gap appears larger for models with more sublayers. Does it grow roughly linearly with depth?

  • The large layer-2 feedforward factor. Why does the layer-2 feedforward map have the largest single-stage factor?

5 comments
AuthorsOpenReview ↗

Reviewer wLUj — Part 1/3: Positive direction-aware bounds and their relation to existing certificates

Thank you for your questions about scope, constructive results, and numerical reliability, and for recognizing the reproducibility effort. We have made the positive direction central to the revision and completed the requested depth and feedforward studies.

W1. The contribution alongside nonvacuous PAC-Bayes and compression results.

Section 1.1 (p. 2) and Appendix L (pp. 34–36) discuss the nonvacuous language-model results of Lotfi and collaborators and compare the corresponding statistical targets and complexity descriptions. Our constructive contribution develops verifiable improvements to uniform generalization guarantees over continuous parameter neighborhoods around a fixed reference decoder.

The revised paper provides two constructive answers. Theorem R.1 (p. 44) gives a finite-dimensional entropy envelope, log⁡N∞(G,ϵ)≤qlog⁡(1+2S/ϵ),\log\mathcal N_\infty(\mathcal G,\epsilon)\le q\log(1+2S/\epsilon), with logarithmic sensitivity dependence. It can improve the certificate while retaining the original transport products. Separately, Theorems 4.1–4.2 improve the transport calculation itself. The sharp quadratic-envelope threshold and these two refinements together identify how improvements to entropy and transport translate into improved statistical certificates.

W2. A positive, parameter-uniform composed bound.

Theorem 4.1 (p. 6) combines certified reference Jacobian products with uniform derivative remainders. A finite state-tube test verifies the conditions for all parameters in the prescribed class and all proper-cover hybrids. Theorem 4.2 (p. 7) includes the scalar loss in the composition and carries the resulting sensitivity into the generalization bound. Appendix S (pp. 46–53) gives the proof and finite verification procedures.

The exact causal RMS/SwiGLU construction uses alternating shear Jacobians whose compositions cancel, together with a causal first-attention branch that makes the loss nonconstant over the allowed parameter. It establishes

K_{\mathrm{comp}}<0.002830.$$ At $N=10^5$, the original correction is capped, the composed correction is below $0.240$, and the composed total-risk certificate is below $0.597<\log 2$ with joint confidence at least 95%. This is a uniform comparison on the same one-parameter class using the same quadratic entropy envelope. The true product-to-composition Jacobian ratio exceeds $10^{20.08}$ throughout that class. See Section 4.1 (p. 7) and Proposition T.1 (p. 53). Theorem S.6 (p. 50) and Corollary S.7 (p. 52) provide a complementary construction in which every matrix can vary over a ball with nonempty interior. With per-matrix radii proportional to $D^{-2}$ and the stated uniform curvature conditions, the composed complexity term is $O(D^{-1/2}\log D)$, while the old coefficient can grow exponentially on the same class. **Uniform validation on trained models.** We also enclose two independently varying entries in five trained two-block decoders. All 25 parameter squares have completed interval enclosures over all 64 input sequences. | Square radius | Composed risk upper bounds across seeds | Below $\log 4$ | |---|---:|---:| | $10^{-6}$ | 0.8998–0.9078 | 5/5 | | $10^{-5}$ | 0.9206–0.9341 | 5/5 | | $10^{-4}$ | 0.9912–1.0614 | 5/5 | | $10^{-3}$ | 1.6550–5.1047 | 0/5 | | $10^{-2}$ | 5.1047–5.1047 | 0/5 | A fresh sample of 100,000 independent sequences supplies a joint 95% event across the squares, matched one-entry controls, and reference Hoeffding bounds. The loss Jacobian has rank two in the selected directions at every reference. Composition improves the stagewise volumetric upper bound in 19 of 25 squares; complete matched comparisons appear in Table 20 (p. 66). See Section 5.4 and Table 3 (p. 9), and Appendix X (pp. 64–67).
AuthorsOpenReview ↗

Reviewer wLUj — Part 2/3: Certified alignment comparisons and a matched depth study

W3. Rigorous guarantees for the product-to-composition gap.

The revision adds analytic and interval-certified comparisons of stage products and composed sensitivity. Proposition V.1 (p. 60) pairs verified feasible-direction stage lower bounds σ‾s\underline\sigma_s with a certified composition upper bound C‾\overline C:

\ge \frac{\prod_s\underline\sigma_s}{\overline C}.$$ Appendix V develops compatible block-norm bounds and arithmetic-error accounting. The exact construction proves a uniform alignment gap algebraically. The trained seed/depth study verifies a ratio greater than one in all 1,280 reference model/input cases. | Decoder blocks | Median across seeds of inputwise median certified ratio | |---|---:| | 1 | 1.771 | | 2 | 4.652 | | 3 | 16.842 | | 4 | 31.695 | The displayed values are rounded downward. This study covers five seeds at each depth and supplies certified reference-point sensitivity comparisons. See Table 22 and Appendix X.4 (p. 66). For the pretrained numerical audit, independent dense FP64 calculations validate 36 selected raw-feedforward, branch, and residual comparisons across 12 record/layer/token cases, with maximum relative discrepancy $5.69\times10^{-7}$. Independent derivative-action checks have maximum relative error $2.22\times10^{-14}$. See Appendix Y.3 (p. 69). **Q1. Does the observed gap increase approximately linearly with depth?** Yes. The six-prefix study of the frozen TinyLlama checkpoint shows an approximately linear increase in the paired log-gap. All prefixes use the same 64 frozen records, providing a matched-input comparison. The numerical norm-search results are: | Residual stages | Stage-product log\$_{10}\$ | Composition log\$_{10}\$ | Paired log\$_{10}\$ gap | |---|---:|---:|---:| | 2 | 2.091 | 1.415 | 0.632 | | 4 | 3.726 | 1.481 | 2.204 | | 8 | 9.274 | 3.069 | 6.208 | | 16 | 16.460 | 3.069 | 13.402 | | 32 | 30.159 | 3.099 | 27.107 | | 44 | 40.575 | 3.658 | 36.844 | Every column reports a median across records; the paired-gap column takes the median after computing each record's difference. The gap increases at all five adjacent-depth comparisons for all 64 records: 320 increases in 320 comparisons. Recordwise linear fits have median slope 0.866 log-base-ten units per residual stage and median $R^2=0.99944$. The largest absolute fit residual is 1.15 log-base-ten units. Together, these results quantify the approximately linear trend over the tested grid. See Section 5.5 and Figure 2(a) (p. 10), and Appendix Y.1 (p. 67). **Search-budget checks supporting the depth comparison.** On four prespecified records, we increased the composite search from 12 iterations and 3 restarts to 24 iterations and 6 restarts. The largest composite increase is $2.44\times$, driven by additional restarts; extending the original three restarts to 24 iterations changes each tested prefix estimate by less than 1%. After strengthening both stage and composition searches, the full-depth paired gaps remain 37.45–39.13 decades. See Table 24 (p. 68).
AuthorsOpenReview ↗

Reviewer wLUj — Part 3/3: Localizing the feedforward outlier and improving the presentation

Q2. Why is the “layer-2” feedforward factor so large?

We have completed an all-token decomposition of the relevant residual map. With the zero-based layer indexing used in the audit, layer 2 is decoder block 3, and its feedforward residual map is residual stage 6.

The peak is attained at position zero in all 64 evaluation records. At that same position, the median component estimates are:

Component at block 3, position 0Median estimate
Learned RMS-normalization Jacobian NN6.634
Raw SwiGLU Jacobian at the normalized state FF108.134
Composed branch Jacobian B=JFJNB=J_FJ_N421.608
Residual Jacobian R=I+BR=I+B421.543

This decomposition identifies a large initial-position SwiGLU derivative, amplified through composition with normalization. The close agreement between the branch and residual values places the large scale in the branch. For comparison, block 1 has a larger maximum normalization estimate, with median 64.95, while block 3's distinctive feature is its raw feedforward derivative at position zero. Across all 8,128 noninitial positions in block 3, the raw SwiGLU estimate is at most 1.049.

Token-level localization and independent checks.

Recomputing the token-maximum summary of the saved Jacobian measurements over positions 1–127 reduces block 3's median maximum residual estimate from 421.54 to 5.86, with recordwise values between 4.53 and 7.85. Applying this same summary to every block places another block at the maximum in every record, further localizing the exceptional factor to position zero.

The 64 records contain 35 distinct first-token IDs. Selected dense FP64 calculations reproduce the exceptional block-3 magnitude. Together, the decomposition, position summaries, and dense checks give a concrete account of where the observed factor arises in this checkpoint. See Section 5.5 and Figure 2(b) (p. 10), Appendix Y.2 and Table 25 (p. 68), and Appendix Y.3 (p. 69).

Presentation changes addressing your assessment.

The introduction now explains the fixed-reference learning problem and uses a one-varying-matrix example to distinguish transport from entropy. The main text develops the explicit bound and threshold, the complete coefficient budget, the composed theorem, and the positive certificate in sequence. The experimental section then presents the trained certificates and the pretrained depth and token-position measurements with direct pointers to their protocols.

The most relevant places to reassess are Sections 1–2 (pp. 1–4), Section 3.2 (p. 5), Section 4 (pp. 6–7), and Sections 5.3–5.5 (pp. 9–10). Full derivations and reproducibility details are retained in the appendix.

Your questions led to the positive composition theorem, certified alignment comparisons, and the matched depth and feedforward investigations. We would be grateful if you would reconsider your assessment in light of these theoretical, experimental, and expository additions, and update your score accordingly.

Reviewer_BkdUOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work presents reference-centered bounds for a popular class of transformer architectures. Whereas other generalizations bounds are tied to sequence lengths, the bounds in this work are expressed in terms of the divergence between a trained model used as prior reference and a data-dependent posterior. The bound has two components, a transport coefficient and a covering-entropy term. The theory is verified on controlled decoders and frozen TinyLlama and SmolLM2 checkpoints.

Critical strengths
  • The mathematical claims seem to be well-supported.
  • The paper makes the source of vacuity mathematically explicit. There is an interesting find, in which a convergence obstruction sits in the transport rather than in the covering constants.
Critical weaknesses

The paper severely lacks clarity. I am familiar with some of the research in this area and I found the paper very terse and difficult to parse. Many concepts are not explained or cited and there is a very strong assumption that the reader is very familiar with most, if not all, of the relevant material. Moreover, explanations regarding the importance of the work and how the author’s contributions compare to related work are missing.

Questions for authors

I do not think any sort of rebuttal will significantly change my scores since my criticism is fundamental.

3 comments
AuthorsOpenReview ↗

Reviewer BkdU: Clearer exposition, explicit learning setup, and constructive significance

Thank you for your careful assessment of the presentation. We have addressed your comments directly in the manuscript by expanding the explanations of the learning problem, notation, proof mechanisms, and significance. The revised ten-page main text explains the statistical problem, the explicit certificate, and the constructive improvements in sequence.

1. Explain the learning problem before the constants.

Section 1 (p. 1) now states the statistical task explicitly: fix a reference decoder and a parameter neighborhood independently of the analysis sample, then control population risk simultaneously for every model in that neighborhood. The reference may be trained on independent pilot data. The sample consists of independent sequences, with dependence among token positions within each sequence allowed.

This is a fixed-class uniform-convergence analysis. The probability statement is over the fresh data sample, and the neighborhood is a deterministic constraint on the model parameters. Reference-centering specifies which parameter perturbations are covered. Context length enters logarithmically through the entropy factor. These explanations make the object, sample unit, and guarantee explicit before the architecture-specific constants in Section 2 (pp. 3–4).

2. Explain transport, entropy, and the complete risk bound concretely.

The introduction uses a one-varying-matrix example. Approximating that matrix creates an immediate error, which propagates through the remaining decoder to the loss: this is the transport cost. Counting the required approximations creates the entropy cost. A smaller neighborhood reduces the approximation budget, while composed transport exploits how perturbation directions evolve through the decoder.

Section 3.2 (p. 5) then connects these quantities to the complete risk certificate. The empirical risk, complexity correction, and confidence term must fit within a specified risk target. The two-row TinyLlama calculation in Table 1 (p. 6) makes the required coefficient improvement numerical.

TopicAdded explanation and location
Statistical setup and reference classSections 1–2, pp. 1–4
Transport versus entropyOne-matrix explanation, Section 1, pp. 1–2
Meaning of the vacuity thresholdSection 3, pp. 4–5
Complete coefficient budgetSection 3.2 and Table 1, pp. 5–6
Composed transport and finite tube verificationSection 4, pp. 6–7
Symbols and the roles of the constantsAppendix L, pp. 34–36

The main text presents the principal statements and their interpretation, with explicit pointers to complete proofs, protocols, and convergence tables.

3. Explain the relation to prior work.

Section 1.1 (p. 2) and Appendix L (pp. 34–36) distinguish reference-centered norm analysis, compositional Lipschitz certification, and nonvacuous compression/PAC-Bayes results. The comparison is organized around what each guarantee controls and how complexity enters it.

The revision also adds a finite-dimensional entropy envelope with logarithmic sensitivity dependence (Theorem R.1, p. 44). This gives a second constructive route within the same framework and clarifies precisely what the quadratic-envelope threshold diagnoses.

4. Make the significance visible through a positive result.

The revised paper answers a concrete question: which step of a local norm-based certificate should be improved, and can that improvement yield a useful guarantee on the same class?

The complete budget answers the first part. The uniform-composition theorem and exact causal decoder answer the second. With the same class and quadratic entropy envelope, composed transport changes a capped correction into a total-risk certificate below 0.597<log⁡20.597<\log 2. See Theorems 4.1–4.2 (pp. 6–7), Section 4.1 (p. 7), and Proposition T.1 (p. 53).

The experiments connect this result to trained models and the original audit: five trained decoders receive uniform two-entry certificates, while the frozen-checkpoint study measures the depth trend and localizes the large feedforward factor. Sections 5.3–5.5 (pp. 9–10) explain these results in the corresponding statistical and numerical settings.

We appreciate the opportunity to improve the paper's accessibility and significance. We would be grateful if you would reconsider your assessment based on the revised exposition and constructive results, and update your score accordingly.

Thanks for the reply.

I have gone through the Introduction, Appendix L, and the Concluding Remarks. I see improvements regarding the issues I have raised but not nearly enough to move the needle. I actually went through some of the related work cited in this new version and those papers made sense to me. I was able to understand them and connect their contributions to my current knowledge of this problem. I do not doubt this is a reviewer problem anymore. I am keeping the rating the same and I will increase my confidence.

Substantial portions of the paper seem to have changed. I cannot find the original version to verify this so I am basing it on what the authors highlighted. I skimmed the new changes. The large number of significant changes in such a short amount of time makes me question the ability of the authors to verify and assume responsibility for what is being presented in this work. But I will not factor this concern in my score.

AuthorsOpenReview ↗

A short reading route and clarifying the original contribution

Thank you for taking the time to revisit the manuscript and the related literature. We appreciate your acknowledgment of the improvements and take your continuing concern about clarity seriously. Making the contribution accessible is our responsibility.

The central question and framework are continuous with the submission.

The paper asks when we can obtain a numerically informative guarantee for loss on new data, using observed loss and a prescribed parameter neighborhood around a fixed reference decoder.

One ingredient in this guarantee is how a parameter perturbation changes the final loss. Multiplying each layer’s worst-case amplification can greatly overstate this effect: the direction most amplified by one layer may be poorly amplified by the next.

The submission made this mechanism quantitative. It derived an explicit generalization bound, identified a sharp threshold at which its optimized correction reaches the loss-range cap, and used norm-matched checkpoint measurements to motivate direction-aware composition. This reference-centered framework, threshold analysis, and original checkpoint audit remain the foundation of the revised paper. Indeed, the submitted abstract already identified direct, direction-aware composition as the direction suggested by its findings.

The revision develops that direction constructively. It adds a uniform composed bound and an explicit decoder example in which changing the transport calculation turns a capped correction into a useful risk guarantee on the same class. These additional results extend the research question and mechanism identified in the submission.

Why the revision contains substantial additions.

We pursued two goals in response to the review discussion: explaining the framework and its significance more clearly, and answering requests for a positive composition theorem, certified validation, and deeper empirical analysis. Your comments guided the expository changes; the additional theoretical and experimental sections address these further questions.

Blue highlighting covers both explanatory revisions and new results. We recognize that their combined extent makes it harder to distinguish the retained foundation from the additions. The revised main text is ten pages, with full derivations and experimental details in the appendix.

We take responsibility for the statements and results in the revision. The additions are accompanied by explicit proofs, specified experimental protocols, and documented numerical checks, including interval certification for the small-decoder studies and independent dense calculations for selected pretrained measurements.

A short route to the contribution.

To make a further assessment manageable, we suggest three specific passages:

PassageWhat to look for
Section 2.3, especially Equation (11), p. 4The statistical target: population loss is controlled by empirical loss plus an explicit correction, uniformly over the prescribed parameter neighborhood.
Theorem 3.1, p. 5The original quantitative diagnosis: an explicit threshold identifies when the optimized correction reaches its range cap.
Section 4.1 and Figure 1, pp. 7–8The constructive consequence: in the explicit decoder example, with the same class and entropy bound, composed transport yields a total-risk certificate below 0.597<log⁡20.597<\log 2 at N=105N=10^5, with joint confidence at least 95% under the specified data distribution.

These passages connect the setup, the original threshold, and the added positive result. In relation to the Lipschitz literature, the contribution is to quantify how the choice of transport estimate changes an explicit statistical guarantee: the threshold identifies when the correction becomes vacuous, and the construction demonstrates a successful improvement through composition.

If your time permits, we would be grateful for a focused rereading of these passages and any reconsideration of your assessment that they support. We would also welcome the opportunity to clarify a particular definition, inference, or connection to prior work that would help your evaluation. Thank you again for your time on this.

Reviewer_eeJXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors have derived the explicit generalization bounds for standard pre-norm transformer models and provided a sharp condition where these bounds become vacuous. They argued the conventional stagewise sensitivity products as a major bottleneck, as multiplying individal layer sensitivities often gives values sustainably larger than directly estimating their composition. The empirical results motivate bounds that track perturbation directions across layers.

Critical strengths
  1. The effort to make the analysis explicit for a realistic decoder architecture. Separating reference-centering, perturbation propagation, and complexity optimization makes it easier to understand why improving one part of the bound may leave the final certificate unchanged.

  2. The empirical audit is very useful. Comparing stagewise and composed sensitivities in the same operator norm, and repeating the comparison across four checkpoints, gives solid evidence motivating a closer study of cross-layer directional compatibility.

Critical weaknesses
  1. My main concern is the extent of the conceptual advance. The looseness of layerwise norm products has been discussed in earlier work, as the paper acknowledges. The sharp vacuity threshold is useful, but it is specific to the chosen optimization objective. A clearer explanation of what the analysis teaches us beyond making this known limitation explicit for modern decoders. would have been useful.

  2. The interpretation of the composition gap remains somewhat tentative. Both quantities are finite-search lower estimates, so their ratio does not certify the true alignment defect. The authors acknowledge this limitation, but it weakens the evidence for the paper’s most interesting explanation. It also remains unclear whether tighter composed transport would be sufficient to obtain a useful certificate after accounting for the other terms.

Questions for authors

See the weaknesses.

Additionally,

  1. How small would a certified upper bound on the relevant downstream composition need to be to cross the nonvacuity threshold? If it were comparable to the measured composite sensitivity, would the remaining terms still keep the certificate vacuous?

  2. Could the authors strengthen the validation of the composition gap—for example, through a certified composite upper bound or a smaller decoder where the relevant norms can be computed accurately? This would help establish how much of the observed separation reflects directional incompatibility rather than differences in estimation accuracy.

4 comments
AuthorsOpenReview ↗

Reviewer eeJX — Part 1/3: Constructive transport theory and certified alignment gaps

Thank you for recognizing the explicit decoder analysis and norm-matched audit, and for asking what constructive advance follows from the observed separation. Your questions led us to add a parameter-uniform composition theorem, positive decoder constructions, and directly certified alignment measurements.

W1. The conceptual advance: a uniform certificate that exploits composition.

For certified reference-segment bounds pb:a≥∥Jb0⋯Ja0∥p_{b:a}\ge\|J_b^0\cdots J_a^0\| and uniform single-stage derivative variations ϵs\epsilon_s, Theorem 4.1 (p. 6) uses Cb:a=pb:a+∑j=abpb:j+1ϵjCj−1:a.C_{b:a}=p_{b:a}+\sum_{j=a}^{b}p_{b:j+1}\epsilon_j C_{j-1:a}. This retains directional cancellation in the reference composition. A finite inequality test verifies the proposed state tubes for every model in the parameter class, including the hybrids needed for proper covers. A first-exit argument establishes the validity of the tubes. Theorem 4.2 (p. 7) appends the scalar loss as the final stage and inserts the resulting coefficient into the generalization bound. Complete proofs and finite verification procedures appear in Appendix S (pp. 46–53).

The statistical consequence is explicit. The exact causal RMS/SwiGLU decoder has d=V=2d=V=2 and context length 32. Alternating shear Jacobians cancel in composition, while the first causal attention branch makes the loss vary nontrivially with the allowed parameter. Its coefficients satisfy log⁡10Kold>34.10,Kcomp<0.002830.\log_{10}K_{\mathrm{old}}>34.10,\qquad K_{\mathrm{comp}}<0.002830. At N=105N=10^5, the original correction is capped, while the composed correction is below 0.2400.240 and the uniform total-risk certificate is below 0.597<log⁡20.597<\log 2, with joint probability at least 0.95. This comparison holds on the same class with the same quadratic mixed-norm entropy envelope. See Section 4.1 (p. 7) and Proposition T.1 (p. 53).

Theorem S.6 (p. 50) supplies a complementary full-dimensional construction: every matrix can vary over a nonempty ball with radius proportional to D−2D^{-2}. Under the stated uniform curvature and injection conditions, the composed complexity term is O(D−1/2log⁡D)O(D^{-1/2}\log D), while the original coefficient can grow exponentially on the same class.

We also identify a separate improvement through entropy. Theorem R.1 (p. 44) gives log⁡N∞(G,ϵ)≤qlog⁡(1+2S/ϵ),\log\mathcal N_\infty(\mathcal G,\epsilon)\le q\log(1+2S/\epsilon), where qq is the effective parameter dimension and SS the propagated radius. Sensitivity enters logarithmically, and this envelope can be combined pointwise with the quadratic envelope before integration. This explicit alternative complements the sharp quadratic-envelope threshold and provides a second route to improved certificates. Section 1.1 (p. 2) and Appendix L (pp. 34–36) explain these contributions and their relation to prior work.

W2. Direct certification of the alignment gap.

The revision establishes a rigorous comparison: verified stage lower bounds σ‾s\underline\sigma_s and a certified composition upper bound C‾\overline C imply

\ge \frac{\prod_s\underline\sigma_s}{\overline C}.$$ Proposition V.1 (p. 60) gives the statement, with compatible block-norm bounds and arithmetic-error accounting in Appendix V. This criterion is verified in all 1,280 reference model/input cases across five seeds and four depths. The median across seeds of each model's inputwise median certified lower bound is: | Decoder blocks | Certified ratio summary | |---|---:| | 1 | 1.771 | | 2 | 4.652 | | 3 | 16.842 | | 4 | 31.695 | The summaries are rounded downward. Table 22 and Appendix X.4 (p. 66) report these reference-point certificates across all 20 trained models. Part 3 presents the corresponding study of parameter-uniform risk certification. The exact construction additionally proves a uniform true alignment gap exceeding $10^{20.08}$ across its entire parameter class.
AuthorsOpenReview ↗

Reviewer eeJX — Part 2/3: Exact coefficient targets for a useful generalization bound

Q1. How small must the downstream bound be after accounting for the remaining terms?

We have added an exact budget for the complete objective. It answers this question quantitatively and identifies the factors that must improve together.

For fixed embeddings, let cN=3log⁡(2/δ)2N,c_N=3\sqrt{\frac{\log(2/\delta)}{2N}}, and let ρB\rho B be the desired correction target. For cN<ρ≤1c_N<\rho\le1, define kρk_\rho as the unique solution in (0,4)(0,4) of kρ{1+log⁡(4/kρ)}=(ρ−cN)/2.k_\rho\{1+\log(4/k_\rho)\}=(\rho-c_N)/2. The complete displayed correction is below ρB\rho B exactly when

=\frac{B\sqrt N}{384\sqrt{\mathcal L_N}}\,k_\rho.$$ Theorem Q.1 (p. 41) also handles a nonzero embedding term, and Corollary Q.2 (p. 42) gives the admissible region when several transport coefficients vary. Section 3.2 (p. 5) explains the calculation in the main text. At the archived setting $N=10^6$, $\delta=0.05$, and neighborhood scale $\eta=10^{-3}$, escape from the correction cap requires $K<1.54816218$. Holding the other archived upper-channel coefficients fixed yields: | Active matrices | $K$ with downstream factor 1 | Downstream requirement | |---|---:|---:| | First attention $Q/K$ | $3.83746\times10^9$ | $<4.03434\times10^{-10}$ | | Last feedforward gate/up | $1.68243\times10^9$ | Tail already equals 1 | This calculation identifies the quantitative improvement required from each part of the certificate. The last feedforward class has an empty downstream residual tail, so its coefficient identifies parameter injection and output sensitivity as additional targets. For the first $Q/K$ class, the relevant quantity is the tail after parameter injection. The revised theorem composes through the loss itself, thereby treating the full path needed for certification. See Table 1 (p. 6) and Table 15 (p. 43). **From correction-cap escape to a useful total-risk certificate.** For empirical risk $\widehat R$, a total-risk bound below $B$ uses $$\rho=1-\widehat R/B.$$ For example, inserting the archived reference capped loss $3.8191$ tightens the allowable coefficient to $1.30421701$; evaluation of another model uses that model's empirical risk. Thus the revision provides both an exact target and constructive ways to reach it: improve parameter-to-loss composition, sharpen injection/output sensitivity, or use the finite-dimensional entropy envelope. The exact decoder in Section 4.1 (p. 7) demonstrates a successful same-class, same-entropy transition to a useful total-risk certificate. Appendix Q retains the complete derivation and archived coefficient calculations.
AuthorsOpenReview ↗

Reviewer eeJX — Part 3/3: Uniform certificates for trained decoders and independent numerical checks

Q2. Certified validation on smaller decoders.

We have completed a uniform interval study of five separately trained two-block causal RMS/SwiGLU decoders, each with 352 parameters. Around each frozen reference, two entries of the first value matrix vary independently over [−r,r]2[-r,r]^2. Directed interval arithmetic encloses the relevant derivatives throughout each square and over all 64 possible input sequences.

Every one of the 25 declared squares has a completed enclosure. The table gives the range of composed volumetric risk upper bounds across the five training seeds, with displayed endpoints rounded upward.

Square radius rrComposed risk upper boundsBelow uniform guessing, log⁡4\log 4
10−610^{-6}0.8998–0.90785/5
10−510^{-5}0.9206–0.93415/5
10−410^{-4}0.9912–1.06145/5
10−310^{-3}1.6550–5.10470/5
10−210^{-2}5.1047–5.10470/5

A fresh sample of N=100,000N=100{,}000 independent sequences gives joint coverage of at least 95% across the 25 squares, their 25 matched one-entry controls, and the five reference Hoeffding bounds. A nonsingular two-by-two loss Jacobian at every trained reference verifies that the two varying directions are functionally distinct.

Composition improves the stagewise volumetric risk upper bound in 19 of the 25 squares. All outcomes and the matched stagewise and reference-plus-oscillation Hoeffding comparisons are reported in Table 20 (p. 66). Together with the exact causal construction, these results establish useful uniform risk bounds on explicit decoder parameter classes. See Section 5.4 and Table 3 (p. 9), and Appendix X (pp. 64–67).

Independent checks of the pretrained numerical audit.

We strengthened the pretrained measurements with both dense calculations and larger searches. Across 12 selected record/layer/token cases, 36 raw-feedforward, branch, and residual comparisons agree with dense FP64 singular-value calculations to a maximum relative discrepancy of 5.69×10−75.69\times10^{-7}. The maximum relative derivative-action error is 2.22×10−142.22\times10^{-14}. See Appendix Y.3 (p. 69).

On four prespecified records, the expanded search uses 24 iterations and 6 restarts, compared with the original budget of 12 iterations and 3 restarts. It improves composite estimates by up to 2.44×2.44\times. The strengthened stage and composition calculations retain full-depth paired log-gaps of 37.45–39.13 decades. These checks provide numerical support for the observed pretrained separation alongside the algebraic and interval-certified results above. See Tables 23–24 (pp. 67–68).

Your comments prompted a constructive theorem, exact coefficient targets, a same-class positive example, and certified trained-model validation. We would be grateful if you would reconsider your assessment in light of these additions and update your score to reflect the revised paper.

AuthorsOpenReview ↗

Follow up: A short route to the constructive result and certified evidence

Thank you again for your detailed questions. As the discussion period draws to a close, we wanted to highlight the passages most directly relevant to your assessment.

The revision retains the submitted reference-centered framework, explicit generalization bound, vacuity-threshold analysis, and original checkpoint audit. The new results develop the direction-aware composition approach already motivated in the submission and address your requests for a constructive result and certified validation.

A focused reading route is:

  • Section 4.1 and Figure 1, pp. 7–8: an explicit causal decoder where, for the same parameter class and entropy bound, composed transport turns a capped correction into a useful total-risk certificate.
  • Section 3.2 and Table 1, pp. 5–6: the complete coefficient budget, quantifying the required improvement while accounting for the remaining terms.
  • Sections 5.3–5.4, p. 9: certified sensitivity gaps across trained seeds and depths, followed by uniform risk bounds for small trained-decoder classes with two independently varying parameter entries.

The revised main text is ten pages; the appendix supplies the full proofs and computational details.

We would particularly appreciate your assessment of whether the constructive example and certified studies address your concerns about the conceptual contribution and reliability of the evidence. If a specific step would benefit from further explanation, we would be glad to clarify it during the remaining discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview: Uniform composition theory, useful certificates, and expanded validation

We thank the reviewers for their careful reading and constructive questions. The revision adds a positive direction-aware theory, rigorous validation, and clearer exposition in direct response to the reviews. The revised main text has ten pages, with changes marked in blue relative to the submission and complete proofs and protocols in the appendix.

1. A constructive, parameter-uniform transport theorem. Theorems 4.1–4.2 (pp. 6–7) retain composed reference Jacobians and control their variation throughout a prescribed parameter neighborhood. A finite state-tube test verifies the conditions for every admissible model and covering hybrid. Including the scalar loss in the composition connects this construction directly to the generalization bound.

2. A useful certificate on the same class and entropy bound. For an exact causal RMS/SwiGLU decoder, the original stage-product correction is range-capped, while the composed total-risk certificate is below 0.597<log⁡20.597<\log 2 at N=105N=10^5, with joint confidence at least 95%. The comparison uses the same parameter class and quadratic entropy envelope. A separate theorem extends the construction to neighborhoods in which every matrix can vary, with explicit depth-dependent radii. See Section 4.1 (p. 7), Proposition T.1 (p. 53), and Theorem S.6 (p. 50).

3. Certified results across training seeds, depths, and independent parameter directions. The seed/depth study certifies a product-to-composition ratio greater than one in all 1,280 reference model/input cases. A further experiment encloses two independently varying entries in each of five trained decoders: all 25 parameter squares are enclosed, and all 15 squares at the three smallest radii receive useful uniform risk bounds. See Sections 5.3–5.4 (p. 9) and Appendix X (pp. 64–67).

4. An exact answer to what must improve. The complete coefficient budget incorporates empirical risk, confidence, embeddings, readout, and transport. It gives explicit numerical targets for improving the archived TinyLlama certificate. An alternative finite-dimensional entropy bound supplies a second route to improvement. See Section 3.2 (p. 5), Appendix Q (pp. 41–44), and Theorem R.1 (p. 44).

5. Direct answers to the pretrained depth and feedforward questions. The six-prefix audit uses the same 64 records and finds an approximately linear increase in the paired log-gap over the tested depths. The token-level decomposition localizes the exceptional block-3 feedforward factor to position zero. Dense FP64 checks and larger searches support these numerical findings. See Section 5.5 (p. 10) and Appendix Y (pp. 67–69).

The revised exposition introduces the learning problem and sample unit first, explains transport and entropy through a one-matrix example, and links each main result to its proof and experimental protocol. Section 1.1 (p. 2) and Appendix L (pp. 34–36) clarify the connections to existing generalization and Lipschitz-certification work. All locations in these responses refer to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.