acceptodds
Under review as a conference paper at ICLR 2027

Harmonic Variance Embedding: A Positional-Style Representation for Counteracting Depth Dilution in Pre-Norm Transformers

Abstract

Pre-norm Transformers have become the backbone of modern large language models, yet their residual stream is not normalized, causing activation variance to grow with depth and gradually diminishing the contribution of each sublayer—a phenomenon known as the curse of depth. In this work, we propose Harmonic Variance Embedding (HVE), a simple and lightweight method that modulates residual dynamics based on input statistics. HVE computes the per-token variance of the residual stream, encodes it using a positional-encoding-style embedding mechanism.HVE maps the resulting representation into the hidden states of intermediate layers via a small MLP with zero-initialized output so that each layer can recognize the variance structure of its own input and adaptively choose the variance of the output it generates , allocating recognizable bandwidth by importance × scale in a matched way so as to partly solve the curse of depth. The positional-style representation is bounded and smooth, effectively suppressing outlier sensitivity while preserving a structured feature of variance magnitude. HVE does not alter the core architecture and adds only a negligible amount of parameters. It importantly mitigates the severe generalization degradation caused by naive variance injection. These findings suggest that our positional-encoding-style encoding of activation statistics offers a practical and robust alternative to explicit normalization or learned scaling mechanisms for deep Pre-norm Transformers.

Primary area:
generative models
Keywords:
Pre-Norm Transformers, Residual Stream, Depth Dilution, Curse of Depth, Activation Statistics, Residual Modulation, Representation Collapse, Fine-tuning Stability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZrJhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the residual stream of Pre-Norm Transformers and argues that its readable bandwidth is governed by the residual scale ρ\rho rather than by content importance π\pi. The authors introduce the Readable Information Bottleneck (RIB) as a diagnostic framework and propose Harmonic Variance Embedding (HVE), a lightweight, zero-initialized module that reads the per-token RMS of the un-normalized merged stream, encodes it through a bounded harmonic encoding, and uses a near-identity soft gate to modulate the FFN input. The claimed goal is to close the residual scale into the FFN’s variance choice so that important information occupies a larger share of the recognizable bandwidth. Experiments are conducted on 7B Qwen2.5-base (MATH-500, 1-shot) and 3B Qwen2.5-instruct (aimo-validation-amc, 2-shot), with additional mechanism probes (E1–E4, S1–S3) and ablations such as HVE_no_cos. The main empirical ordering is HVE > HVE_no_cos > Base > LoRA_without_HVE on 7B, and LoRA+HVE > LoRA+HVE_no_cos > LoRA on 3B.

Critical strengths

The experimental reporting is relatively cautious and honest in places. The authors do not overclaim that all mechanism probes are significant; for example, they explicitly note that E1/E2/E3 are weak or non-significant on the measured checkpoints, and they frame Conjecture 3.1 as a falsifiable conjecture rather than a proven theorem. This kind of restraint is appreciated and makes the empirical section more trustworthy than it might otherwise be.

The proposed HVE module is lightweight, zero-initialized, and identity-preserving at initialization. The design choice of reading ρ\rho with stop-gradient and injecting it through a bounded harmonic encoding is technically reasonable and could be of practical interest if the core mechanism were better validated.

Critical weaknesses

1. The central claim that “noise occupies the lane” is not supported by the E2 data.

The paper’s core pathology is that recognizable bandwidth is ruled by scale ρ\rho and blind to importance π\pi, with the stronger claim that unimportant or noise-like outputs “falsely occupy a large share” and “clog the bandwidth” of important information. However, the E2 measurements show that high-π\pi tokens hold about 10.9% of the readable bandwidth while low-π\pi tokens hold about 9.8%, a difference of only about 1.1 percentage points. The authors themselves write that HVE’s bandwidth re-allocation is “not significant at this coarse level of attribution.” This supports at most the weaker claim that bandwidth is not allocated by importance; it does not support the stronger and rhetorically central claim that noise is actively occupying the lane. The paper repeatedly frames the problem as “noise occupying the lane,” but the data only show a mild misallocation. This is a consequential gap between the narrative and the evidence.

2. The core concept of “importance π\pi” is close to circular.

The entire theoretical framework is built on the idea that recognizable bandwidth should be allocated proportionally to importance π\pi. However, π\pi is defined as the “intrinsic content that deserves to be read by the next layer” (Definition 1), then said to be “implicitly estimated by the FFN from raw content” (§3.1), and finally measured in experiments by gradient norms or task-subspace projection shares (§4.3). If π\pi is defined by network behavior—FFN estimation or gradient norms—then the matched-allocation law σY2∝πρ2\sigma_Y^2 \propto \pi \rho^2 risks becoming a tautology: the FFN output variance is large because the FFN output variance is large. The authors’ hygiene protocol (computing π\pi on the baseline, freezing it, and mis-batching it relative to the measured rr) only cuts the measurement loop, not the definitional loop. π\pi is still not an independent content property. This weakens the theoretical contribution considerably.

3. Depth dilution has already been independently verified, and simpler fixes exist.

The phenomenon of depth dilution in Pre-Norm Transformers has been systematically studied in recent work, e.g., “The Curse of Depth in Large Language Models” (NeurIPS 2025), which shows across LLaMA, Mistral, DeepSeek, and Qwen that removing up to 30% of deep layers without fine-tuning barely hurts performance, and attributes this to exponential variance growth in Pre-LN. That work also proposes LayerNorm Scaling, which scales by 1/depth1/\sqrt{\text{depth}} with zero additional parameters and one line of code. In contrast, this paper re-packages the known phenomenon as RIB, proposes a much more complex HVE module with harmonic encoding, an MLP, and a bounded gate, and reports gains of roughly +0.4% to +5%, while the mechanism probes are largely non-significant. The authors do not directly address why a closed-loop gated module is needed when a simple scaling fix may suffice. This weakens the claimed significance and novelty of the repair.

Questions for authors
  1. Can the authors clarify whether the E2 result actually supports the strong “noise occupies the lane” claim, or only the weaker “bandwidth is not allocated by importance” claim? If the latter, how does this affect the central motivation of RIB and HVE?

  2. How is π\pi defined independently of the network’s own behavior? If π\pi is measured by gradient norms or FFN-estimated quantities, how does the matched-allocation law avoid becoming a tautology?

  3. How does HVE compare against simple depth-scaling fixes such as LayerNorm Scaling (e.g., 1/depth1/\sqrt{\text{depth}}) under equal parameter and compute budgets? If a zero-parameter scaling method already addresses depth dilution, what is the unique advantage of HVE?

Reviewer_LvDHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the growth of residual-stream scale with depth in Pre-Norm Transformers and the resulting reduction in the relative contribution of new information from deeper layers. The authors introduce the Readable Information Bottleneck (RIB) framework, interpreting this phenomenon as a bandwidth-allocation problem in which readability is governed more by residual-stream scale than by information importance. They propose an ideal allocation principle under which the variance of the current layer's output should be proportional to both information importance and the square of residual-stream scale.

To approximate this principle, the paper proposes Harmonic Variance Embedding (HVE). HVE computes the pre-normalization RMS of each token's residual stream, maps the scale to bounded features using multi-frequency sine and cosine functions, and uses a lightweight MLP to produce a modulation vector. The vector is then injected through a gated mechanism into the normalized FFN input, allowing the FFN to condition on residual-stream scale. The paper also presents theoretical analyses drawing on information theory, optimal transport, and function approximation.

The authors evaluate HVE with LoRA fine-tuning on Qwen2.5-7B-base and Qwen2.5-3B-instruct. They report improvements over ordinary LoRA fine-tuning and a variant without harmonic encoding on the reported mathematical reasoning tasks. A set of mechanistic experiments further examines the relationships among information importance, residual scale, and readability.

Critical strengths
  1. The paper addresses a relevant problem. The interaction between residual-stream scale and the contribution of deeper layers is pertinent to understanding representations and information flow in deep Transformers.

  2. The proposed module is relatively simple to integrate. HVE encodes residual-stream scale and uses it to modulate the FFN input without changing the residual-connection topology. Its compatibility with LoRA fine-tuning gives the approach some practical potential.

Critical weaknesses

W1. The manuscript contains substantial unremoved drafting artifacts and internal inconsistencies, raising concerns about quality control and the reliability of the reported account.

The submission contains material that should not appear in a finalized paper, including editorial notes, references to unavailable files, incorrect figure references, and discrepancies between the experimental description and the reported results. These issues impair readability and make it difficult to verify that the theoretical and experimental claims have been checked carefully.

  1. Unremoved drafting and editorial notes. Appendix A.7 (lines 888–889) retains “(MOVED FROM THE FORMER THEORY CHAPTER)”; Appendix F (line 1159) similarly contains “(Moved from §3.2–3.4 of earlier drafts; ...)”. Line 1155 even states, “Draft; anonymous review. Please send comments to the anonymous ICLR-2027 chairs.” These are revision-stage notes rather than scholarly content.

  2. Key derivations point to external Markdown files that are not included. Lines 965–966 and 1071–1073 direct the reader to HVE_Appendix_B_swamping_FULL.md and HVE_Appendix_C_readable_information_FULL.md for full derivations. These files are not part of the submitted paper. Their absence prevents reviewers from checking important derivations in a paper that relies heavily on theoretical analysis.

  3. Incorrect figure reference. The caption of Figure 1 refers to a color convention shared by “Figures 1–3,” although the provided manuscript does not contain a Figure 3. This suggests that cross-references were not checked carefully.

  4. Mismatch between the experimental description and the reported results. Lines 452–455 state that the authors first validate the phenomenon on a 1.5B model and then conduct causal attribution analysis. However, the corresponding reported experiments are on 3B and 7B models; results for the stated 1.5B experiment are not provided. This discrepancy makes it unclear whether the experimental description accurately reflects the work performed.

  5. Non-scholarly instructions in the references section. The beginning of the references retains the sentence “Listed alphabetically by first author (ICLR author–year style); every in-text citation corresponds to an entry below.” This reads as an internal bibliography-preparation note, not as part of a finalized reference list.

Taken together, these problems materially affect the paper's professionalism and verifiability. The authors should systematically check the manuscript, remove drafting artifacts, provide the missing derivations, and verify all cross-references and consistency between the experimental setup, results, and claims.

W2. The experimental baselines are questionable, making the reported performance gains difficult to interpret.

A substantial part of the reported advantage is measured against an ordinary LoRA baseline that performs poorly. In the Qwen2.5-7B experiment, the untuned Base model achieves 39.0% accuracy on MATH-500, while ordinary LoRA fine-tuning reduces accuracy to 34.0%. The paper does not adequately analyze this marked degradation or establish that the LoRA baseline was reasonably tuned.

HVE reaches 40.4%, only 1.4 percentage points above the untuned Base model. The full HVE model also exceeds HVE_no_cos, the variant without harmonic encoding, by only 0.4 percentage points. These results do not rule out the possibility that the apparent gap is primarily due to a poorly configured LoRA baseline rather than HVE's proposed scale-conditioning mechanism.

For the 3B experiments, the paper does not report results for the untuned Base model. It is therefore difficult to determine whether HVE improves the original model's capability or merely outperforms one particular LoRA fine-tuning configuration.

W3. The mechanistic experiments do not substantiate the paper's central theoretical claim.

The central hypothesis is that HVE makes the readable share of residual-stream information depend more on information importance and less on residual-stream scale. The mechanistic experiments in Section 4.3 do not provide sufficient support for this claim.

In E1, when gradient norm is used as a proxy for importance, HVE's overall importance–readability Spearman correlation is 0.022, below the Base model's 0.036; the correlation does not improve in deeper layers either. E2 does not show that HVE increases the bandwidth share allocated to high-importance tokens. In E3, HVE and the baseline have similar matching slopes, while the corresponding regressions explain little variance.

Thus, although HVE improves some reported downstream results, the paper does not provide direct evidence that these gains arise from the proposed importance-aware bandwidth-allocation mechanism. The authors attribute the weak mechanistic evidence to insufficient training, but do not provide results after more extensive training. The proposed mechanism therefore remains empirically unsupported.

Questions for authors

Q1. Adequacy of the LoRA baseline

In the 7B experiment, ordinary LoRA fine-tuning reduces MATH-500 accuracy from 39.0% for the Base model to 34.0%. Can the authors explain this degradation? In particular, how extensively was the LoRA baseline tuned (e.g., learning rate and number of training steps or epochs)? Could the authors report results under alternative training configurations to establish that HVE's advantage is not primarily due to comparison against an under-optimized LoRA baseline?

Q2. Discrepancy between the mechanistic experiments and the central claim

The paper's central claim is that HVE makes readability depend more on information importance than on residual-stream scale. However, the E1, E2, and E3 experiments in Tables 2–4 do not show a clear mechanistic improvement over the baseline. How do the authors reconcile these results with the central claim? Can they provide more direct evidence that HVE implements importance-aware bandwidth allocation, rather than merely improving downstream task performance?

Q3. Connection between the theoretical analysis and the HVE implementation

The theoretical analysis assumes that output variance satisfies σY2∝πρ2\sigma_Y^2 \propto \pi\rho^2, whereas the implemented HVE module modulates the FFN input through scale encoding and gating without explicitly using importance π\pi or directly controlling output variance. Can the authors provide a more rigorous explanation of why this implementation should satisfy the theoretical matched-allocation condition? In particular, can they validate the carrier assumption in Appendix A.2 and specify the conditions under which the theoretical result applies to the implementation, rather than only to an idealized scale-matching mechanism?

3 comments

We thank you for a detailed report. The problems listed under W1 are real, and the cause is on our side: the PDF that reached you was compiled from an older working tree, not from the file we had marked as the submission. We have since kept a single canonical .tex and added a pre-submission check that searches the compiled source for editorial notes, references to files outside the paper, and broken figure or table references. W1 — Drafting artifacts and internal inconsistencies

Three of the items are editorial leftovers. The note "(MOVED FROM THE FORMER THEORY CHAPTER)" in Appendix A.7 is gone, and A.7 now reads as ordinary text under the heading "The unified inequality", stating a₀‖v‖ < 1 once and showing the four places it appears (critical depth T1, anti-swamping B, readability bound C, feasibility of matched allocation in §3.2). The relocation note in Appendix F is gone; the material is now its own appendix. The line "Draft; anonymous review. Please send comments to the anonymous ICLR-2027 chairs." was a footer in the draft template and has been removed. The other four concern content. The two derivations that pointed to HVE_Appendix_B_swamping_FULL.md and HVE_Appendix_C_readable_information_FULL.md are now typeset in full as Appendix B (B.1 restates anti-swamping as a contribution ratio, B.3 gives the per-sample left-tail condition) and Appendix C (C.1 derives the Gaussian closed form from X ⟂ Y, C.4–C.6 give the three viewpoints, C.7 the correspondence table); no claim depends on a file outside the submission, and no .md filename remains in the source. The Figure 1 caption no longer mentions "Figures 1–3": the manuscript has two figures, the caption describes only the convention used in this paper, and we re-checked every \ref and \label for dangling references. The sentence about a 1.5B validation was left over from an earlier experimental plan and has been deleted; §4 now describes the runs that were performed, namely 7B Qwen2.5-base on error-selected OmniMath-500 (4096 context, 3 epochs, 1-shot MATH-500) and 3B Qwen2.5-instruct on error-selected S1K (1 epoch, 2-shot aimo-validation-amc), and no 1.5B result is claimed. The reference list no longer opens with "Listed alphabetically by first author (ICLR author–year style); …" and is a plain ICLR-style bibliography. Inlining the two appendices was not only a formatting change. It forced us to say, where each result is used, which quantities are assumed and which are derived, and that changed §3 and Appendix A.2 (Q3 below).

W2 / Q1 — The LoRA baseline

You are right that the LoRA arm is not a tuned comparator. It was run under one fixed recipe, identical to the HVE arm (3 epochs, 4096-token context, lr = 1e-4 in the Table 2 version). We did not sweep the learning rate or the number of steps, and the submitted implementation table gave only the HVE defaults, not the LoRA rank and alpha. Those values are being added. From here on the paper treats the LoRA arm as a capacity control and does not argue from "HVE beats LoRA" alone. The 39.0 → 34.0 drop has a specific cause. The 7B run trains on the samples the frozen checkpoint cannot solve, so the supervised targets sit in the region where the model is already confident: on the probe corpus 57% of answer tokens are at CE ≤ 1e-2 (PPL ≈ 1.01), and the hardest decile has median PPL ≈ 686. Plain NLL over that mixture sharpens an already saturated distribution, which is consistent with the drop and with the reports of Pande et al. (2025) and Barnett et al. (2024) on saturated in-domain data. The clean version of the comparison is Table 2, where both arms use the same recipe (full OmniMath, lr = 1e-4, 3 epochs, TrimSFT with m = 3, τ = 2) and only the HVE arm has a warm-started gate at a ≈ 0.05. On AIME-2025 the un-finetuned checkpoint scores 0/30 at 3B and 1/30 at 7B, LoRA scores 1/30 and 1/30, and LoRA+HVE scores 3/30 and 2/30. With a recipe that does not over-sharpen, the LoRA arm is no longer below the frozen checkpoint, so the inversion in Table 1 comes from the recipe rather than from LoRA being a weak method. We state this in §4.2 and in the limitations, and we also note there that each cell has 30 items and 0–3 successes (Fisher one-sided p ≈ 0.31–0.50), so Table 2 is directional. We do not lean on the main-table margins either. The HVE–Base gap is 1.4 points on 500 items, which is inside single-run noise (±≈2.2 points at 95%), and the HVE–no_cos gap is two items. The controls that would make the comparison meaningful are running under the same budget: a learning-rate and step sweep for LoRA (lr ∈ {1e-4, 2e-4, 5e-4} × {1, 3} epochs), a gate-zero control (a ≡ 0), and a parameter-matched ρ-blind control. The frozen 3B checkpoint, which is a single evaluation pass and was missing from the table, is being added.

W3 / Q2 — What the mechanism experiment is meant to test

We should separate two claims that we previously ran together, because only one of them is load-bearing. The claim HVE actually makes is layer-level: the scale read at each wrapped layer reaches the deep output and changes it. The measurement that supports this is the intervention in E15, run on the open-gate checkpoint (a ≈ 0.05). Amplifying the pre-normalisation RMS by q = 2 at a layer (a no-op at q = 1) moves the deep answer-token output with 1 − cos² = 1.3 × 10⁻³ (95% CI [8e-4, 1.9e-3]) and mean |ΔCE| = 2.8 × 10⁻². The perturbation is norm-preserving and almost purely orthogonal (perpendicular component 3.3e-2, dlog‖f‖ ≈ 0), so it does not act by adding energy; the response is produced by the stack above the perturbed layer. It also grows by about 430× from the easiest to the ninth perplexity decile. This is the evidence the paper rests on, and it is an external intervention rather than a correlation measured inside the model. The token-decile probe (E1–E3) was an attempt at a stronger, token-resolved statement: that readable bandwidth is reallocated across tokens within a layer. That was always an additional check, not the mechanism we claim, and the current version no longer presents it as support for the layer-level claim. On the numbers you quote: they come from the earlier probe and have been replaced by a re-run. The protocol changed in three ways: eight deep layers (10, 12, …, 24) instead of all layers; π recomputed on each checkpoint (self-π), so that head drift after fine-tuning cannot be mistaken for a mechanism result; and per-layer Spearman averaged with layer-clustered bootstrap CIs, with a ρ-purified variant E1′ to remove π–ρ collinearity. All numbers in §4.3 now come from this run. E1 is −0.280 [−0.383, −0.168] for Base, −0.267 [−0.375, −0.151] for LoRA, and −0.270 [−0.375, −0.156] for LoRA+HVE; E1′ is −0.218/−0.216/−0.218 and the π–ρ Spearman is +0.170, so the negative sign is not a collinearity artifact; HVE − LoRA is −0.003, far below the CI half-width of about 0.11, so the coupling is strongly negative and HVE does not change it. In E2 the high-π decile holds 9.09% [8.66, 9.41] of the readable share against 11.56% [10.86, 12.54] for the low-π decile, a gap of 0.024 with a CI half-width under 0.004, and HVE − LoRA is −0.0002; this is a resolved null, not a power failure. In E3 the slopes are +12.96/+12.93/+13.00 with R² ≈ 0.25, so the arms are indistinguishable and the regression explains little variance; since ρ² appears in both the regressor and the denominator of the share, the coefficient is not identifiable as evidence for matched allocation, so we keep E3 as a descriptive check and replace it with the E14 protocol. The null was also predictable from the implementation, and we now say why instead of attributing it to training length. Two facts, both visible in the code and in Algorithm 1: the gate is a per-dimension vector b ∈ ℝ^D, shared by all tokens, so token dependence can enter only through ρ and cannot produce a per-token weight at all; and the main mechanism checkpoint operates at a ≈ 10⁻³, where the modulated branch contributes at the 10⁻⁶ level relative to the residual, which is below what a π-share probe can resolve. So E1–E3 test a claim the deployed parameterisation cannot express, at an operating point where the branch is nearly closed. We report that as a limitation of this implementation.

Q3 — Theory and implementation

Three points from the current text are relevant. First, the theoretical statement is an allocation rule with a stated reach, not an identity. Appendix A.2 derives an analytical surrogate and Propositions 3.1/3.2 give the bounded realisation. Outside the band where a₀‖v‖ < 1 the frozen branch caps the numerator at C₀², the share obeys s ≤ C₀²/(ρ² + C₀²), and the 1/ρ² law returns. The surrogate is a statement about the rule inside its reach, and the paper says explicitly that it must not be read as σ_Y² = cπρ² holding at arbitrary depth. Second, the bridge from rule to module is stated as an assumption in the place where it is used, not folded into the A1–A7 table. The carrier assumption (X ⟂ Y) and the reliability of the π proxies are listed as modelling assumptions alongside the result that needs them. C.1 shows that the Gaussian closed form is derived from exactly that one hypothesis, and C.6 states that the OT constant depends on which reading of the injection is taken (aligned or orthogonal), so every quantitative use of the constant names its reading. The paper also notes that at a ≈ 10⁻³ the realised cπ is near 10⁻⁶, which is the small-share regime in which the rule reduces to r ≈ cπ. Third, the implementation does not set a = cπ/‖v‖². It learns a per-dimension gate as a function of ρ through a harmonic encoding. What the theory supplies is the required allocation and the reachable set; whether optimisation reaches it is Conjecture 3.1, which is labelled a conjecture and adjudicated by E1 and the E14 protocol. The current E1 answer is that at a ≈ 10⁻³ it does not. We say this in the text rather than asserting that the matched-allocation condition is satisfied. To test the carrier assumption directly we are measuring, per wrapped layer, the cosine between the injected direction v and the FFN's own output direction f, together with the E14 input-side proxy computed only from the pre-FFN normalised input. These are the conditions under which the theoretical statement applies to this module, and we will report them with the baseline sweep in W2. Closing We are sorry that the reviewed PDF mixed an old draft with the paper. The writing defects in W1 were real and are gone from the current version, and both derivations are now inside it. On the two substantive points we do not have a defence the data would support: the LoRA control was not tuned, and the token-level reallocation is not demonstrated. The first is being fixed with a sweep and two controls; the second we have stopped claiming, because the evidence for this implementation is at the layer level and the token-level test cannot be passed by a gate of this form.

Reviewer_eT2aOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HVE, which encodes residual-stream RMS using harmonic features and feeds this information into normalized FFN inputs. It aims to mitigate depth dilution in Pre-Norm Transformers and evaluates the method through LoRA fine-tuning of Qwen2.5 checkpoints.

Critical strengths
  1. Giving FFNs access to scale information removed by normalization is a reasonable idea, and the module is straightforward to integrate into pretrained models.

  2. The paper reports downstream improvements over LoRA and attempts to investigate the underlying mechanism.

Critical weaknesses
  1. The evaluation does not match the scope of the architectural claim. Experiments are limited to brief fine-tuning of pretrained checkpoints. A controlled pretraining study, preferably across different depths, is needed to establish whether HVE improves optimization and depth utilization.

  2. The mechanism diagnostics do not support the claimed repair. In Table 2, importance–readability correlation decreases from 0.036 to 0.022 with HVE, while Table 3 shows no increased allocation to high-importance tokens. Downstream accuracy gains alone cannot establish that HVE reduces the proposed importance-blindness.

  3. The evidence for harmonic encoding is weak. Its advantage over the nonharmonic variant amounts to two additional correct answers in each setting, based on single-run results. The encoder architectures also differ, so the comparison does not isolate harmonic features at matched capacity. Comparisons with closely related scale-conditioned normalization methods are missing.

Questions for authors
  1. Does HVE improve pretraining validation loss and deep-layer contributions under a comparable training budget?

2、 What evidence connects the downstream gains to reduced depth dilution, given the results in Tables 2–3?

  1. Do the harmonic gains persist across seeds and against a capacity-matched simple encoder or scale-conditioned normalization baseline?
2 comments
AuthorsOpenReview ↗

Response to Reviewer: A Retraining-Free, Post-Hoc Adapter for Depth Utilization — Scope, Mechanism, and Controls

We thank you for the careful reading and for the specific, constructive criticisms. A revised manuscript has been submitted. It adds measured depth/utilization behaviour at two model scales, an explicit comparison with retraining-based remedies for the depth curse, and a reformulation of the mechanism and harmonic-encoding claims. The encoder controls requested in W3 are currently running and will be reported in the final version.

Our Settings used throughout. Qwen2.5-3B-instruct and Qwen2.5-7B-base. HVE wraps post_attention_layernorm at all layers except the first 4 and the last 2 (30 layers, 5–34, at 3B; 22 layers, 5–26, at 7B), adding 2D^2 + D parameters per wrapped layer, with the RMSNorm gain frozen. Fine-tuning uses: 7B, error-selected OmniMath-500, 3 epochs, 4096 context, 1-shot MATH-500; 3B, error-selected S1K, 1 epoch, 2-shot aimo-validation-amc. W1 — The evaluation does not match the scope of the architectural claim

We clarify the scope first. HVE is a post-hoc, retraining-free adapter: it is inserted into an already-pretrained checkpoint and trained under exactly the same budget as the baseline. The claim is that depth utilisation lost to pre-norm can be partially recovered without re-pretraining, not that HVE improves pretraining optimisation. A from-scratch study answers the second claim, not ours.

Within its scope we do run a controlled comparison: identical checkpoint, corpus, optimiser, steps, schedule and decoding, with HVE as the only difference, at two scales:

Method7B · MATH-5003B · AMC (n=83)
LoRA+HVE40.433/83 (39.76%)
LoRA+HVE_no_cos40.031/83 (37.35%)
Base (frozen)39.0—
LoRA34.026/83 (31.33%)

Two facts matter for the reviewer's concern. First, HVE changes downstream behaviour materially: +6.4 points over the LoRA control at 7B and +8.4 points (+7 items) at 3B, and on the AIME-2025 calibration check 3/30 vs 1/30 vs 0/30 (3B) and 2/30 vs 1/30 vs 1/30 (7B). Second, on the 7B ablation the fine-tuned LoRA arm falls below the frozen backbone (34.0 < 39.0), so HVE is the only configuration that remains above the untouched pretrained model; we read this as evidence that the gain is not a generic effect of extra trainable capacity. We also report, for the first time, that the variance reading is causally used rather than absorbed by normalisation. Amplifying the pre-normalisation RMS by a uniform factor q=2 (q=1 is a no-op by construction) on the wrapped 7B checkpoint changes the answer-token CE with 1 - cos^2 = 1.3e-3 (95% CI [8e-4, 1.9e-3]), mean absolute per-token change 2.8e-2 against a net -2.2e-3. The perturbation is norm-preserving and almost purely orthogonal (perpendicular component 3.3e-2, dlog||f|| ~ 0), i.e. it does not act by adding energy; the response is produced by the stack above the perturbed layer. It is strongly difficulty-dependent: the absolute dCE grows by ~430x from the easiest to the ninth perplexity decile (decile rank correlation ~0.95), and the fraction of tokens whose CE increases rises from 0.12 to 0.42–0.43. The control loop of §3.2 is therefore live in the trained model. On the request for a from-scratch controlled study: the closest competitors (Hyper-Connections and its manifold-constrained variants) obtain their gains by joint pretraining with additional cross-layer residual connections, i.e. by re-pretraining the whole backbone at its original pretraining budget. Such a comparison would not isolate HVE; it would confound the adapter with a different pretraining recipe, and its cost is precisely what a retrofit method exists to avoid. Conversely, the variance-constraint remedy from the depth-curse line can be applied post hoc, and in our tests doing so degrades an already-pretrained checkpoint relative to the untouched backbone. That asymmetry is the substantive point: post-hoc compatibility is not automatic, it has to be measured, and the relevant comparison class for HVE is post-hoc remedies.

W2 — The mechanism diagnostics do not support the claimed repair

We agree and have withdrawn the claim. We state explicitly that the gain showed in the former experiment cannot be attributed to importance-blindness repair, and that the earlier framing was wrong.

What we claim instead is weaker and measured: the injected scale is read and used. Amplifying the pre-norm RMS by q=2 (a no-op at q=1) changes answer-token CE with 1 - cos^2 = 1.3e-3, mean |dCE| = 2.8e-2, while being norm-preserving and almost purely orthogonal (dlog||f|| ~ 0); the response grows ~430x from the easiest to the ninth perplexity decile. Structurally, the gate is (D,) and shared across tokens, so a token-level reallocation is not expected by construction.

We also scope the null honestly. The probe is small — 49,856 tokens of OmniMath text, 8 of the 22 adapted layers — so we treat the present result as a null within this scope, not as proof that no reallocation is possible. A larger multi-domain probe (problems and generations, all adapted layers, both scales) is running and will be reported; until then we do not present downstream accuracy as mechanism evidence. W3 — The evidence for harmonic encoding is weak

We agree: the two encoders differ in capacity and the evidence rests on single runs, so the comparison does not isolate the harmonic features, and comparisons against scale-conditioned normalisation are missing in the original paper. These controls are running now and will be reported in the final version, whatever their outcome: 5 seeds at 3B and 7B (mean ± std); a capacity-matched non-harmonic encoder (equal parameters and FLOPs); a scale-conditioned normalisation baseline at the same 30/22 layers; and a parameter-matched rho-blind control, which the capacity objection in W1 also requires. If the advantage does not survive capacity matching or 5 seeds, we will revise the claim rather than the baseline, and will say so explicitly. The current 0.4-point and 2-item margins are already labelled preliminary. Q1 — Does HVE improve pretraining validation loss and deep-layer contributions under a comparable training budget?

HVE is not a pretraining modification, so we claim nothing about pretraining validation loss. It improves the downstream behaviour of an already-pretrained model at a comparable fine-tuning budget: +6.4 (7B) and +8.4 (3B) points over the matched LoRA control, and the only arm above the frozen backbone. Per-layer rho, per-layer contribution and Jacobian-norm profiles for the measured checkpoints are given in Appendix F.1; they establish the baseline pathology, not a pretraining effect. Q2 — What evidence connects the downstream gains to reduced depth dilution, given the results in Tables 2–3?

We no longer make that connection. The depth-dilution diagnostic (Appendix F.1) characterises the baseline; within the current probe scope, Tables 3–5 show HVE does not measurably reallocate bandwidth, and that probe (49,856 tokens, OmniMath, 8 of 22 adapted layers) is the limitation we are addressing with a larger and more diverse one. What we can support today is that the scale reading is causally used (readout intervention: 1 - cos^2 = 1.3e-3, mean |dCE| = 2.8e-2), and that the effect concentrates on harder tokens (~430x from easiest to ninth decile). We therefore present the gain as an empirical effect of a live control loop, and treat "reduced depth dilution" as an unverified hypothesis. Q3 — Do the harmonic gains persist across seeds and against a capacity-matched simple encoder or scale-conditioned normalization baseline?

That comparison is running now: 5 seeds, a capacity-matched non-harmonic encoder, and a scale-conditioned normalisation baseline at the same insertion points, at both 3B and 7B. Results are not complete; they will be included in the final version with a revised claim, and we will report them whether or not they preserve the harmonic advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.