- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the residual stream of Pre-Norm Transformers and argues that its readable bandwidth is governed by the residual scale rather than by content importance . The authors introduce the Readable Information Bottleneck (RIB) as a diagnostic framework and propose Harmonic Variance Embedding (HVE), a lightweight, zero-initialized module that reads the per-token RMS of the un-normalized merged stream, encodes it through a bounded harmonic encoding, and uses a near-identity soft gate to modulate the FFN input. The claimed goal is to close the residual scale into the FFN’s variance choice so that important information occupies a larger share of the recognizable bandwidth. Experiments are conducted on 7B Qwen2.5-base (MATH-500, 1-shot) and 3B Qwen2.5-instruct (aimo-validation-amc, 2-shot), with additional mechanism probes (E1–E4, S1–S3) and ablations such as HVE_no_cos. The main empirical ordering is HVE > HVE_no_cos > Base > LoRA_without_HVE on 7B, and LoRA+HVE > LoRA+HVE_no_cos > LoRA on 3B.
Critical strengths
The experimental reporting is relatively cautious and honest in places. The authors do not overclaim that all mechanism probes are significant; for example, they explicitly note that E1/E2/E3 are weak or non-significant on the measured checkpoints, and they frame Conjecture 3.1 as a falsifiable conjecture rather than a proven theorem. This kind of restraint is appreciated and makes the empirical section more trustworthy than it might otherwise be.
The proposed HVE module is lightweight, zero-initialized, and identity-preserving at initialization. The design choice of reading with stop-gradient and injecting it through a bounded harmonic encoding is technically reasonable and could be of practical interest if the core mechanism were better validated.
Critical weaknesses
1. The central claim that “noise occupies the lane” is not supported by the E2 data.
The paper’s core pathology is that recognizable bandwidth is ruled by scale and blind to importance , with the stronger claim that unimportant or noise-like outputs “falsely occupy a large share” and “clog the bandwidth” of important information. However, the E2 measurements show that high- tokens hold about 10.9% of the readable bandwidth while low- tokens hold about 9.8%, a difference of only about 1.1 percentage points. The authors themselves write that HVE’s bandwidth re-allocation is “not significant at this coarse level of attribution.” This supports at most the weaker claim that bandwidth is not allocated by importance; it does not support the stronger and rhetorically central claim that noise is actively occupying the lane. The paper repeatedly frames the problem as “noise occupying the lane,” but the data only show a mild misallocation. This is a consequential gap between the narrative and the evidence.
2. The core concept of “importance ” is close to circular.
The entire theoretical framework is built on the idea that recognizable bandwidth should be allocated proportionally to importance . However, is defined as the “intrinsic content that deserves to be read by the next layer” (Definition 1), then said to be “implicitly estimated by the FFN from raw content” (§3.1), and finally measured in experiments by gradient norms or task-subspace projection shares (§4.3). If is defined by network behavior—FFN estimation or gradient norms—then the matched-allocation law risks becoming a tautology: the FFN output variance is large because the FFN output variance is large. The authors’ hygiene protocol (computing on the baseline, freezing it, and mis-batching it relative to the measured ) only cuts the measurement loop, not the definitional loop. is still not an independent content property. This weakens the theoretical contribution considerably.
3. Depth dilution has already been independently verified, and simpler fixes exist.
The phenomenon of depth dilution in Pre-Norm Transformers has been systematically studied in recent work, e.g., “The Curse of Depth in Large Language Models” (NeurIPS 2025), which shows across LLaMA, Mistral, DeepSeek, and Qwen that removing up to 30% of deep layers without fine-tuning barely hurts performance, and attributes this to exponential variance growth in Pre-LN. That work also proposes LayerNorm Scaling, which scales by with zero additional parameters and one line of code. In contrast, this paper re-packages the known phenomenon as RIB, proposes a much more complex HVE module with harmonic encoding, an MLP, and a bounded gate, and reports gains of roughly +0.4% to +5%, while the mechanism probes are largely non-significant. The authors do not directly address why a closed-loop gated module is needed when a simple scaling fix may suffice. This weakens the claimed significance and novelty of the repair.
Questions for authors
-
Can the authors clarify whether the E2 result actually supports the strong “noise occupies the lane” claim, or only the weaker “bandwidth is not allocated by importance” claim? If the latter, how does this affect the central motivation of RIB and HVE?
-
How is defined independently of the network’s own behavior? If is measured by gradient norms or FFN-estimated quantities, how does the matched-allocation law avoid becoming a tautology?
-
How does HVE compare against simple depth-scaling fixes such as LayerNorm Scaling (e.g., ) under equal parameter and compute budgets? If a zero-parameter scaling method already addresses depth dilution, what is the unique advantage of HVE?
We thank you for a detailed report. The problems listed under W1 are real, and the cause is on our side: the PDF that reached you was compiled from an older working tree, not from the file we had marked as the submission. We have since kept a single canonical
.texand added a pre-submission check that searches the compiled source for editorial notes, references to files outside the paper, and broken figure or table references. W1 — Drafting artifacts and internal inconsistenciesThree of the items are editorial leftovers. The note "(MOVED FROM THE FORMER THEORY CHAPTER)" in Appendix A.7 is gone, and A.7 now reads as ordinary text under the heading "The unified inequality", stating
a₀‖v‖ < 1once and showing the four places it appears (critical depth T1, anti-swamping B, readability bound C, feasibility of matched allocation in §3.2). The relocation note in Appendix F is gone; the material is now its own appendix. The line "Draft; anonymous review. Please send comments to the anonymous ICLR-2027 chairs." was a footer in the draft template and has been removed. The other four concern content. The two derivations that pointed toHVE_Appendix_B_swamping_FULL.mdandHVE_Appendix_C_readable_information_FULL.mdare now typeset in full as Appendix B (B.1 restates anti-swamping as a contribution ratio, B.3 gives the per-sample left-tail condition) and Appendix C (C.1 derives the Gaussian closed form fromX ⟂ Y, C.4–C.6 give the three viewpoints, C.7 the correspondence table); no claim depends on a file outside the submission, and no.mdfilename remains in the source. The Figure 1 caption no longer mentions "Figures 1–3": the manuscript has two figures, the caption describes only the convention used in this paper, and we re-checked every\refand\labelfor dangling references. The sentence about a 1.5B validation was left over from an earlier experimental plan and has been deleted; §4 now describes the runs that were performed, namely 7B Qwen2.5-base on error-selected OmniMath-500 (4096 context, 3 epochs, 1-shot MATH-500) and 3B Qwen2.5-instruct on error-selected S1K (1 epoch, 2-shot aimo-validation-amc), and no 1.5B result is claimed. The reference list no longer opens with "Listed alphabetically by first author (ICLR author–year style); …" and is a plain ICLR-style bibliography. Inlining the two appendices was not only a formatting change. It forced us to say, where each result is used, which quantities are assumed and which are derived, and that changed §3 and Appendix A.2 (Q3 below).W2 / Q1 — The LoRA baseline
You are right that the LoRA arm is not a tuned comparator. It was run under one fixed recipe, identical to the HVE arm (3 epochs, 4096-token context,
lr = 1e-4in the Table 2 version). We did not sweep the learning rate or the number of steps, and the submitted implementation table gave only the HVE defaults, not the LoRA rank and alpha. Those values are being added. From here on the paper treats the LoRA arm as a capacity control and does not argue from "HVE beats LoRA" alone. The 39.0 → 34.0 drop has a specific cause. The 7B run trains on the samples the frozen checkpoint cannot solve, so the supervised targets sit in the region where the model is already confident: on the probe corpus 57% of answer tokens are at CE ≤ 1e-2 (PPL ≈ 1.01), and the hardest decile has median PPL ≈ 686. Plain NLL over that mixture sharpens an already saturated distribution, which is consistent with the drop and with the reports of Pande et al. (2025) and Barnett et al. (2024) on saturated in-domain data. The clean version of the comparison is Table 2, where both arms use the same recipe (full OmniMath,lr = 1e-4, 3 epochs, TrimSFT with m = 3, τ = 2) and only the HVE arm has a warm-started gate ata ≈ 0.05. On AIME-2025 the un-finetuned checkpoint scores 0/30 at 3B and 1/30 at 7B, LoRA scores 1/30 and 1/30, and LoRA+HVE scores 3/30 and 2/30. With a recipe that does not over-sharpen, the LoRA arm is no longer below the frozen checkpoint, so the inversion in Table 1 comes from the recipe rather than from LoRA being a weak method. We state this in §4.2 and in the limitations, and we also note there that each cell has 30 items and 0–3 successes (Fisher one-sided p ≈ 0.31–0.50), so Table 2 is directional. We do not lean on the main-table margins either. The HVE–Base gap is 1.4 points on 500 items, which is inside single-run noise (±≈2.2 points at 95%), and the HVE–no_cos gap is two items. The controls that would make the comparison meaningful are running under the same budget: a learning-rate and step sweep for LoRA (lr ∈ {1e-4, 2e-4, 5e-4}× {1, 3} epochs), a gate-zero control (a ≡ 0), and a parameter-matched ρ-blind control. The frozen 3B checkpoint, which is a single evaluation pass and was missing from the table, is being added.