acceptodds
Under review as a conference paper at ICLR 2027

Why Does Non-Text Token Norm Inflation Emerge?

Abstract

Multimodal LLMs improve at understanding images and audio, yet a modality gap between non-text and text remains. We define as an increase in non-text token L2 norms during modality alignment, with these norms exceeding text token norms in early decoder layers. To examine its emergence, we track non-text norms during alignment. We then test its information-preserving role by replacing non-text tokens with inputs of varying norms but fixed directions. We assess the effects of mitigating norm inflation using during Qwen3.5-4B audio alignment. Our findings are threefold. First, early norm inflation is driven by increases in common-component norm. Second, we find an early decoder-layer range where replacing subsequent non-text tokens with inputs preserves at least 90% of originally correct predictions. Finally, TAN applied through early decoder layers yields lower accuracy after replacement than without TAN. TAN achieves higher inter-modal cosine similarity and improves performance across tasks, with a 9.9%p gain on MMAU. These findings support the information-preserving role of norm inflation. The results suggest that mitigating norm inflation may promote earlier formation of cross-modal representations and thereby improve task performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Model, Modality gap, Norm discrepancy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GiYqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the role of text and non-text token norms in multimodal large language models (MLLMs). In particular, the authors observe a phenomenon they call non-text token norm inflation, where the norms of non-text tokens increase during modality alignment and become larger than those of text tokens. The authors analyze this phenomenon across different models and investigate its potential role in preserving non-text information through the decoder layers. They then introduce Text-Anchored Norm Regularization (TAN), which aims to mitigate this effect by aligning the norm statistics of non-text tokens with those of text tokens. They report improvements on several downstream audio tasks.

Critical strengths

The paper addresses an interesting research question. Understanding how the norms of text and non-text tokens evolve during multimodal alignment, how their distributions differ, and how they interact across decoder layers is certainly worth investigating. Moreover, the analysis goes beyond simply observing norm differences, including experiments on token replacement and the decomposition of representations into common and individual components. The proposed regularization is relatively simple, and the reported improvements on some downstream tasks are encouraging.

Critical weaknesses

My main concern is the overall execution and presentation of the study. I found the paper quite difficult to follow. Several concepts are repeated throughout the manuscript, sometimes without adding much to the discussion. In general, I struggled to identify a clear flow connecting the introduction, methodology, experiments, and conclusions.

The methodology section is particularly difficult to follow. While some mathematical definitions are provided, I think the overall formalization is insufficient. For instance, the expression (H^+ = H + F(\operatorname{Norm}(H))) is very general, since (F) can represent either attention or an FFN and the same is not linked to anything else later on in the text so even useless. In general, I would have appreciated a more precise description of the architecture, clearly defining the encoder, projector, decoder, and the representations being analyzed. Although these components are familiar in the MLLM literature, their specific roles and interactions should be made explicit in this work.

Another issue is the organization of the methodology. Several implementation and evaluation details are introduced in the Methods section, while other important details are only provided in the appendices. I think a clearer separation between the proposed methodology and experimental setup would considerably improve readability.

My main experimental concern relates to the training procedure. As stated in Section 3.1 and illustrated in Figure 1(a), the analysis of norm inflation during alignment is conducted by training only the projector, while keeping the encoder and decoder frozen. This is a reasonable setup for an initial investigation, but it only represents one stage of a typical MLLM training pipeline.

In many MLLMs, a subsequent alignment or instruction-tuning stage involves jointly training the projector and the LLM, and sometimes also the modality-specific encoder. What happens to the norms of text and non-text tokens during this stage? Does the same inflation behavior emerge, and does the proposed regularization remain beneficial?

To be precise, the authors do include a subsequent decoder LoRA fine-tuning stage in Section 3.3. However, I think the evolution of token norms during this stage deserves more attention, particularly considering the main research question of the paper. Minor problem: If also Lora finetuning is perfomed then Figure 1 is misleading. The only trainable module in that picture is the projector one. Readability and correctness of Figure 1 should be improved.

Finally, I find the interpretation of the results not entirely convincing. The authors associate reduced norm inflation with earlier cross-modal representation formation, mainly based on cosine similarity and token-replacement experiments. While these results are interesting, I am not sure they are sufficient to support this interpretation. Moreover, TAN is evaluated mainly on one audio alignment configuration, making it difficult to understand how general the proposed method actually is.

All in all, I think the paper starts from an interesting observation and investigates a relevant problem in MLLMs. However, I find the current execution rather weak for this prestigeous venue, both in terms of presentation, limited novelty and weak experimental validation. The difficulty in following the methodology, together with the limited investigation of different alignment stages, makes it hard for me to fully assess the significance and generality of the results.

Questions for authors
  1. Why is the emergence of norm inflation investigated primarily under projector-only optimization? How do text and non-text token norms evolve during decoder LoRA fine-tuning or joint projector–decoder training?

  2. What evidence establishes that increased inter-modal cosine similarity under TAN reflects improved semantic integration rather than a geometric effect of the regularizer?

Plus See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer GiYq (Part 1)

Thank you for your detailed feedback on the paper's presentation, training procedure, and interpretation of its results. We provide a revised manuscript and plan to supplement its appendix with additional experimental results. Our responses to your questions and concerns follow.

 

Question 1. Projector-only optimization and subsequent training

We froze the encoder and decoder and trained only the projector to show that norm inflation already emerges during initial alignment. This is a standard setting for initial feature alignment in MLLMs. For vision in Experiment 1, we reinitialized and trained the projector between a jointly pretrained encoder and LLM. Norms grew rapidly, then decreased and stabilized above text norms. Audio also showed an initial increase followed by a stable norm trajectory (Sections 3.2–3.3, Figure 2).

To examine the effect of subsequent decoder training, we compared layer-wise mean norms before and after fine-tuning on the same 256 samples. With the projector frozen during LoRA fine-tuning, relative changes in both text and non-text norms were below 1% for Baseline, Initialization only, TAN0\mathrm{TAN}_0 (λ=0.1\lambda=0.1), and TANe\mathrm{TAN}_e. We also jointly trained the projector and the decoder's LoRA parameters for 625 steps from the TANe\mathrm{TAN}_e alignment checkpoint at a learning rate of 10−510^{-5}. No norm regularizer was applied during this stage. The largest relative changes in layer-wise mean norm were +2.8% for non-text tokens and +0.9% for text tokens. In our setting, subsequent training changed the norms established during initial alignment only slightly.

 

Question 2. Semantic integration rather than a geometric effect of the regularizer

The manuscript discusses earlier formation of joint representations and does not use the term “semantic integration.” Smaller non-text norms can increase the relative influence of decoder updates on the residual and thereby change inter-modal cosine similarity (Section 5.3, Figure 6). We also reproduced an increase in cosine similarity through pure rescaling (Appendix E.5, Figure 20).

TANe\mathrm{TAN}_e learns projector representations under reduced norms, so the differences after training cannot be equated with post-hoc rescaling alone. To examine the representations and how the decoder uses them, we compared audio inputs with their transcript text embeddings and analyzed internal computation in a SpokenSQuAD case, as described in our response to Reviewer uyfY, Question 4. For the same recording, Initialization only and TANe\mathrm{TAN}_e differed in how representations of particular audio segments affected the answer. Selective interventions within each model confirmed these effects. The procedure and results are included in the revised manuscript (Section 5.3; Appendices F.1–F.2).

AuthorsOpenReview ↗

Response to Reviewer GiYq (Part 2)

Concern 1. The overall execution and presentation of the study

We have organized the manuscript around three experiments: the emergence of norm inflation, its role in input preservation, and the effects of mitigating it (Sections 3–5). Shared definitions appear early in the paper (Section 1), and each experiment connects its question with the results and interpretation. Methodology and experimental setup are now separate (Sections 3.1–3.2, 4.1–4.2, and 5.1–5.2). Each Experimental Setup includes the settings needed to interpret that experiment, with detailed reproducibility information in the appendix (Appendices B–C).

 

Concern 2. The overall formalization is insufficient

We deliberately use a general formulation to emphasize that our analysis rests on a property shared by pre-norm decoders.

We have also specified that the encoder extracts features from non-text inputs and the projector maps those features into the decoder's input space. A token denotes the representation vector at a decoder sequence position. For non-text tokens, depth 0 denotes the projector output, and each subsequent depth denotes the corresponding decoder block's output (Section 1).

 

Concern 3. Figure 1 is misleading if LoRA fine-tuning is also performed

Subsequent decoder LoRA fine-tuning serves downstream task evaluation. Because it is not central to our methodology, Figure 1(a) focuses on the projector alignment used to investigate norm inflation. Fine-tuning used a low maximum learning rate of 5×10−65\times10^{-6} and a small training set of 15,000 examples, with the same procedure applied to all conditions, including Baseline. Experiment 3 specifies the settings (Section 5.2; Appendix B.1).

 

Concern 4. TAN is evaluated mainly on one audio alignment configuration

TAN is a simple regularizer that matches non-text token norm statistics to text references. Although evaluated here only in Qwen3.5-4B audio alignment, it can also be applied to other modalities. We chose audio because vision models are more widely available than audio-visual models, making audio adaptation of an existing VLM a useful small-scale setting under limited resources.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts. However, two main concerns remain.

1. Paper structure and organization.

I acknowledge that the readability of the manuscript has slightly improved. However, in my honest point of view, I still find the overall structure not so much appropriate for a scientific paper. I apologize if my original comment was not sufficiently clear. What I meant was a clear separation between methodology and experimental evaluation, rather than organizing the paper into three macro-sections titled "Experiment 1", "Experiment 2", and "Experiment 3", each containing its own methodology and experimental setup.

In my view, a more appropriate organization would include a unified methodology section presenting the framework, definitions, and proposed approach, followed by a separate experimental section presenting the evaluations and analyses. The current organization remains fragmented and does not adequately address my original concern.

2. The central research question.

The paper is titled "Why Does Non-Text Token Norm Inflation Emerge?" However, I am still not convinced that the work actually answers this question.

The experiments show that norm inflation occurs during alignment, that the common component contributes substantially to its early development, and that larger token norms can help preserve input representations. These are interesting observations, but demonstrating what norm inflation does is not the same as explaining why it emerges.

I want to say again that I find the overall phenomenon interesting and worthy of further investigation. However, at its current stage, the work mainly provides empirical observations without a sufficiently developed theoretical foundation. The proposed explanation then is plausible, but the experimental evidence does not establish the claimed causal mechanism.

I sincerely appreciate the authors' efforts in investigating this topic and in rebuttal phase and I hope they find my feedback constructive.

AuthorsOpenReview ↗

Response 2 to Reviewer GiYq

Thank you for clarifying your concern that the methodology is still presented in a fragmented way. We designed the study to examine how norm inflation emerges, what role it plays, and what changes when it is mitigated. We used a different method for each question. Had we been able to establish the causal mechanism directly, we could have presented a unified methodology and organized the experimental evaluation around that mechanism. We understand your concern about the paper’s organization to be related to how directly the study explains why norm inflation emerges.

We agree that demonstrating that large norms help preserve input representations does not by itself establish why norm inflation emerges. Although our study does not directly establish a causal mechanism for the emergence of norm inflation, the following steps show how we developed our explanation and tested its predictions.

  1. In Experiment 1, we observed norm inflation when we used next-token prediction (NTP) to train projectors for feature alignment in multimodal LLMs.

  2. The NTP objective does not explicitly favor large non-text token norms.

  3. We hypothesized that large norms emerge because they help achieve a function “A” that benefits NTP optimization. During projector training, the norms of the projected inputs can change and thereby affect “A.” The benefit of “A” can therefore favor large norms during alignment. The title focuses on the emergence of norm inflation, the phenomenon we observe directly during training.

  4. To identify “A,” we considered the following hypothesis: “Making the encoder’s already structured features compatible with the decoder can help the model rapidly reduce NTP loss during early alignment by learning associations between inputs and responses in the training data. Preserving the projected encoder features across decoder layers may help the model use the learned associations for prediction.”

  5. In Experiment 2, we investigated input preservation as the proposed function “A.” We examined how closely non-text tokens retained their projected input directions and used input replacement with norm and direction controls to test the role of large norms in preserving the projected inputs.

  6. With reduced input preservation, we expected decoder updates to have greater relative influence on the residual in early layers than under Baseline, so that non-text tokens would depart from their input directions at shallower depths and accuracy after input replacement would be lower.

  7. In Experiment 3, we tested the predictions in step 6. Because input preservation cannot be manipulated in isolation in our training setup, we used TAN to mitigate norm inflation during alignment. We observed both expected changes under TANe\mathrm{TAN}_e relative to Baseline, providing indirect empirical support for our explanation of why norm inflation emerges.

The current manuscript does not explicitly state the optimization hypothesis in step 4. We will state the hypothesis at the start of Experiment 2 to explain why preserving the decoder-compatible inputs discussed in Experiment 1 may benefit early optimization and how the potential benefit motivates testing the role of large norms in input preservation.

We appreciate your constructive feedback, which prompted us to clarify how the experiments address the central research question.

Reviewer_uyfYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why non-text token norms grow during multimodal alignment and how this affects representation processing. It tracks norm growth in vision and audio models, finding that shared components contribute substantially to early inflation, and uses norm- and direction-controlled token replacements to examine input preservation within the decoder. It then proposes Text-Anchored Norm Regularization (TAN), which matches non-text norm statistics to text references across early layers. Experiments on several pretrained multimodal models support a role for token magnitude in input preservation, while Qwen3.5-4B audio alignment experiments show that early-layer TAN improves several tasks, including a 9.9-percentage-point gain on MMAU.

Critical strengths
  1. Controlled analysis of token magnitude and direction. Training-time tracking and norm-matched replacement experiments provide useful evidence about how norm inflation develops and affects input preservation, going beyond descriptive comparisons between modalities.

  2. Practical gains with a relevant ablation. Early-layer TAN improves MMAU accuracy by 9.2 percentage points over the matched-initialization and learning-rate control, supporting a benefit from internal norm regularization beyond simply changing initialization.

Critical weaknesses
  1. The explanation for why norm inflation emerges remains incomplete. Replacement experiments establish that token magnitude affects subsequent computation, but do not isolate input preservation as the driver of norm growth during training. Higher cosine similarity between modality means also does not establish earlier semantic integration. The evidence therefore supports the observed effects more strongly than the proposed mechanism.

  2. The incremental benefit and robustness of TAN are insufficiently established. The initialization-only control changes both initialization and learning rate relative to Baseline and already captures much of the transcription and QA improvement. Although TAN provides a substantial additional MMAU gain, its evaluation uses one training seed and one audio-alignment configuration. Repeated runs are needed to establish that this benefit is reliable.

Questions for authors
  1. How consistent are TAN’s gains across training runs, particularly relative to the matched-initialization and learning-rate control on MMAU? Additional seeds would help assess the reproducibility of the regularizer’s benefit, since the reported confidence intervals capture evaluation-sample uncertainty. This is a request for additional validation.

  2. How much of the difference between TAN₀ and TANₑ comes from depth coverage versus regularization strength? Since TANₑ distributes the coefficient across 12 depths, an ablation separating these factors would help clarify the benefit of internal norm control. This is a request for a targeted ablation.

  3. How do the authors interpret what TAN changes beyond representation scale? I found the preservation–transformation perspective particularly surprising and would welcome the authors’ thoughts on whether TAN primarily changes information accessibility of non-text input, its distribution across positions, or the decoder’s use of it. As a possible future direction, comparing pure rescaling with TAN and examining output-token entropy alongside accuracy could help distinguish these possibilities. This is an exploratory discussion rather than a requirement for acceptance.

My main consideration for raising the rating to Clear Acceptance is the analysis in the third point. I would particularly value a mechanistic interpretability of what token magnitude represents and what TAN changes, connected to whether models with and without TAN differ in the consistency of their semantic understanding—for example, when interpreting the same image across paraphrased questions. Since the current TAN experiments concern audio, an analogous audio example would also be appropriate. A carefully analyzed case study linking these behavioral differences to representation changes could substantially strengthen my assessment without requiring a broad new evaluation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uyfY (Part 1)

We appreciate your detailed feedback and suggestions on reproducibility, regularization strength, and the analysis of audio representations. A revised manuscript accompanies this response; we plan to add further experimental results to the appendix. We respond to your questions and concerns below.

 

Question 1. Consistency of TAN's gains across training runs

The additional seeds support the MMAU improvement under TANe\mathrm{TAN}_e in our Qwen3.5-4B audio adaptation setting. For seed 122, the manuscript reports 59.8% for Initialization only and 69.0% for TANe\mathrm{TAN}_e. With seed 123, the corresponding results are 55.7% and 66.6%. TANe\mathrm{TAN}_e with seed 124 achieves 66.1%. Initialization only and TANe\mathrm{TAN}_e use matched initialization and learning rates (Sections 5.1–5.2).

 

Question 2. Depth coverage versus regularization strength

The additional experiment shows that regularization strength is the main factor behind the difference between TAN0\mathrm{TAN}_0 and TANe\mathrm{TAN}_e. We divided TAN0\mathrm{TAN}_0's coefficient by 12, training it at 0.1/12 to match the coefficient TANe\mathrm{TAN}_e applies to its input norm penalty:

ConditionLS-clean WER ↓LibriMix WER ↓SpokenSQuAD F1 ↑MMAU ↑MMAU-Pro ↑
TAN0\mathrm{TAN}_0 (λ=0.1/12\lambda=0.1/12)2.86%8.32%86.94%67.50%36.42%
TANe\mathrm{TAN}_e2.81%7.70%87.32%69.00%35.86%

The L2 norm and inter-modal cosine similarity profiles also become closer to TANe\mathrm{TAN}_e's after adjusting the coefficient. TANe\mathrm{TAN}_e still has smaller early-layer norms and higher mean inter-modal cosine similarity, indicating a contribution from depth coverage as well. The revised manuscript includes this comparison and reports the norm and cosine profiles (Section 5.3, Table 1 and Figure 6; Appendix E.2, Figure 18).

 

Question 3. What TAN changes beyond representation scale

Changes in representation scale also affect the direction of internal representations through the residual connections of a pre-norm decoder. We interpret this as affecting how the decoder uses information. Following your suggestion, we applied pure rescaling to match TANe\mathrm{TAN}_e's scale and analyzed task performance and inter-modal cosine similarity (Appendix E.5, Figure 20 and Table 9). The task results are:

ConditionLS-clean WER ↓SpokenSQuAD F1 ↑MMAU ↑
Baseline3.26%78.47%59.1%
Baseline rescaling346.15%21.75%60.7%
Initialization only2.76%86.53%59.8%
Initialization rescaling3.64%83.44%72.4%
TANe\mathrm{TAN}_e2.81%87.32%69.0%

Pure rescaling substantially degrades performance on tasks requiring speech recognition for Baseline. The decline is much smaller for Initialization rescaling. Thus, norm differences during encoder–LLM alignment can shape the internal representations processed by the decoder.

 

Question 4. Token magnitude and consistency of semantic understanding

Token magnitude controls the relative influence of decoder updates on the existing audio representation in the residual stream. Following your suggestion, we examined how the same audio content is represented and used differently after training with TANe\mathrm{TAN}_e.

First, we examined how well audio representations correspond to text embeddings of the same content. We used all 5,559 recordings in LibriSpeech test-clean and test-other. We averaged the audio input tokens and transcript embeddings separately for each recording, subtracted each modality's mean vector across recordings, and computed cosine similarity. This retrieval test uses no additional learned mapping. The following Top-1 results count recordings whose own transcript was retrieved from all 5,559 candidate transcripts:

ConditionCorrect transcript retrievedTop-1
Baseline10 / 5,5590.18%
Initialization only14 / 5,5590.25%
TANe\mathrm{TAN}_e128 / 5,5592.30%

In this comparison, TANe\mathrm{TAN}_e's audio input directions correspond more closely to text embeddings of the same content (Section 5.3; Appendix F.1, Table 13).

AuthorsOpenReview ↗

Response to Reviewer uyfY (Part 2)

Question 4. Token magnitude and consistency of semantic understanding (continued)

We then gave Initialization only and TANe\mathrm{TAN}_e the same SpokenSQuAD recording as audio tokens and analyzed their internal representations and answers across question phrasings. The recording describes Denver's Super Bowl win and states that Gary Kubiak won as a head coach with the same franchise for which he had appeared in the Super Bowl as a player. A representative question is “Which team did Gary Kubiak play for when he was in the Super Bowl?” The reference to “the same franchise” in the audio is a clue to the answer. The question asks for the franchise's name, Denver/Broncos.

Across 12 phrasings of the same fact, Initialization only sometimes answered with the phrase “the same franchise.” TANe\mathrm{TAN}_e answered with the team name, Denver/Broncos. Both models could answer when asked explicitly whether the winning team and Kubiak's teams as a player and coach were the same.

To explain the difference, we compared the next-token computation after both models had generated “the.” We decomposed the attention output by audio segment and measured each segment's support for generating the team name or “same.” Both models read information favoring Denver/Broncos from the segment where the team is named. In Initialization only, the representation of the segment saying “the same franchise” strongly supported producing “same” as the answer. This influence was smaller in TANe\mathrm{TAN}_e. The same difference appeared in the 7 questions that allowed comparison at this shared prefix.

To test whether reading these audio segments affects the answer, we perturbed key directions within each model while retaining its audio values and key norms. This changes which audio segment is read and how strongly, while preserving the information carried by the values. We selected the 28th decoder layer through a layer-wise scan and targeted the keys at audio token positions corresponding to each relevant segment. We located the approximate spans by transcribing overlapping segments of the recording. Both models received the same full recording.

For the representative question, perturbing Initialization only's keys for the segment saying “the same franchise” changed its answer from “the same franchise he won” to “the broncos.” TANe\mathrm{TAN}_e retained its answer under the same intervention. Perturbing TANe\mathrm{TAN}_e's keys for the segment containing the team name changed “the broncos” to “the super bowl.” Applying the same intervention to an unrelated audio segment preserved the answers.

In Initialization only, the representation of the segment saying “the same franchise” strongly promoted repetition of that phrase even when the question asked for a team name. This influence was smaller in TANe\mathrm{TAN}_e. The answers across question phrasings reflect different contributions from the audio token representations of the “same franchise” and team-name segments in the two models. The case study and intervention results appear in the revised manuscript (Appendix F.2, Tables 14–17).

 

Concern 1. The explanation for why norm inflation emerges remains incomplete

We agree that the replacement experiments do not isolate input preservation as the driver of norm growth during training. They establish the input-preservation role of large norms.

We think NTP optimization may favor preserving encoder features in early layers and modifying them gradually, since non-text input vectors already carry features organized by the encoder. We consider this a plausible interpretation, though explaining why the loss landscape would favor this behavior is a substantially more complex problem.

 

Concern 2. Higher cosine similarity between modality means does not establish earlier semantic integration

We agree that higher cosine similarity alone does not establish semantic integration. The manuscript does not use the term “semantic integration.” We use joint representations for decoder states that represent information from the different modalities together.

Our interpretation of earlier joint representation formation combines earlier departure from input directions with higher audio–text cosine similarity (Section 5.3, Table 2 and Figure 6). In Question 4, we complement this interpretation by examining transcript matching and tracing how internal representations of audio segments contribute to answers (Appendices F.1–F.2).

I appreciate the authors' response and their careful answers to my questions. I think my concerns are resolved.

Reviewer_j8u9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to address the issue of modality gap in multimodal LLMs via focusing on a phenomenon of pre-norm LLMs: norm inflation in early depths. The work investigated the effect of such inflation via token replacement and proposed Text-Anchored Norm Regularization (TAN). In the end, proposed TAN yields performance improvement in the evaluation and suggests that such regularization which mitigates norm inflation is beneficial to multimodal LLM training.

Overall, the work is great in its analysis and proposed insights, however there are some concerns about the mismatch on its narrative and actual discovery (please refer to weakness 1). Moreover, part of the models it uses were not really match the assumption in this work (Layernorm vs RMSNorm, question 1). If these could be resolved, I would be glad to adjust the review score.

Critical strengths
  • Comprehensive analysis on the norm inflation phenomenon with multiple real-world LLMs.
  • Insights on why norm inflation is disadvantageous through the observation of its input preservation effect (section 4.2).
Critical weaknesses
  • Part of the models investigated in the work seemed using a different type of normalization layer, RMSNorm. Since the core discussion is established on L2 norm and the inflation of common component (mean), it does not match the process of RMSNorm (inverse scaling on non-centered rms value). Please also refer to my question below.

  • I have a concern regarding the claim of the work. Applying TAN with e=11 does improved performance. However, TAN_0 and "initialization only" performed worse than baseline on certain metrics. Moreover, in section 4.2, replacing original predicted result with input tokens (should have smaller norm) impacts accuracy. It could be argued that TAN forced earlier cross-modal alignment within the model, thus benefited the performance, not because it avoided norm inflation (e.g. the early departure from input directions mentioned in L447-451). I understand this is a positive discovery, and it is explicitly mentioned in the manuscript, but then it's not really matched with the narrative in the introduction that norm inflation is the cause of modality gap.

Questions for authors
  • According to the implementation available online, Qwen3.5-4B used RMSNorm with a 1.0 shift, while Phi-4-multimodal vision used RMSnorm. Despite if the shift of Qwen-3.5, RMSNorm lacks the zero-centering step, unlike LayerNorm. First, I would like to confirm if my finding is true. If yes, since the entire manuscript assumes the normalization is LayerNorm and thus decomposes tokens in common components and individual components, how does such difference affect the claims in the manuscript?
3 comments
AuthorsOpenReview ↗

Response to Reviewer j8u9

We thank you for your detailed review and for raising questions about RMSNorm, task performance, and the interpretation of our findings. We provide a revised manuscript and plan to include additional experimental results in the appendix. We address your question and concerns below.

 

Question. RMSNorm and the decomposition into common and individual components

Yes. The LLM decoders in Qwen3.5-4B and Phi-4-multimodal use RMSNorm, which does not subtract the mean across coordinates within a token. Our decomposition and our explanation of how magnitude affects input preservation both apply to RMSNorm.

At each depth, the common component is the vector obtained by averaging non-text tokens within each input and then across inputs. The individual component is each token's deviation from this vector (Section 3.1):

\mathbb{E}\\|h\\|_2^2=\\|c\\|_2^2+\mathbb{E}\\|h-c\\|_2^2.$$ This average is defined across inputs and token positions, independently of centering the coordinates within each token. We scaled a fixed audio vector without changing its direction and passed it through the RMSNorm module in Qwen3.5-4B's first decoder layer. The BF16 results are: | Input scale | L2 norm before RMSNorm | L2 norm after RMSNorm | |---|---:|---:| | ×1 | 11.4338 | 57.9441 | | ×4 | 45.7352 | 57.9443 | | ×16 | 182.9409 | 57.9443 | | ×64 | 731.7637 | 57.9443 | Even with a 64-fold increase in input norm, the output norm is nearly constant. The residual path retains the enlarged input, so its magnitude changes the relative influence of the decoder update.   ### Concern 1. $\mathrm{TAN}_0$ and Initialization only performed worse than Baseline on certain metrics The MMAU-Pro difference between Baseline and Initialization only is small: 0.35 percentage points (34.01% versus 33.66%). $\mathrm{TAN}_0$ scored below Baseline on MMAU and MMAU-Pro in the manuscript, but the coefficient control performed better than Baseline on all five main-text benchmarks. Our focus is the emergence and role of norm inflation, so we deliberately avoided extensive coefficient tuning for performance optimization. Following Reviewer uyfY's suggestion, we trained $\mathrm{TAN}_0$ with a coefficient of 0.1/12, matching the coefficient $\mathrm{TAN}_e$ applies to its input norm penalty. The comparison is also included in the revised manuscript (Section 5.3, Table 1). | Condition | LS-clean WER ↓ | LibriMix WER ↓ | SpokenSQuAD F1 ↑ | MMAU ↑ | MMAU-Pro ↑ | |---|---:|---:|---:|---:|---:| | Baseline | 3.26% | 9.66% | 78.47% | 59.10% | 34.01% | | Initialization only | 2.76% | 8.11% | 86.53% | 59.80% | 33.66% | | $\mathrm{TAN}_0$ ($\lambda=0.1$) | 3.07% | 8.66% | 85.86% | 54.00% | 33.90% | | $\mathrm{TAN}_e$ | 2.81% | 7.70% | 87.32% | 69.00% | 35.86% | | $\mathrm{TAN}_0$ ($\lambda=0.1/12$) | 2.86% | 8.32% | 86.94% | 67.50% | 36.42% | Reducing $\mathrm{TAN}_0$'s coefficient to 0.1/12 resolves its two reported drops. Its task performance and inter-modal cosine similarity also become closer to $\mathrm{TAN}_e$'s (Section 5.3, Table 1 and Figure 6).   ### Concern 2. Replacing original representations with input tokens impacts accuracy Input replacement tests how far into the decoder the learned inputs can substitute for intermediate representations. Decoder computation continues to transform representations even while large norms help preserve the inputs. Some accuracy loss when replacing intermediate states with the original input tokens is therefore expected. For example, Input accuracy at depth 3 of Qwen3.5-4B vision is 91.0%. Original is 100% because the cohort contains 300 originally correct items. At the same norm, Norm-matched achieves 90.7% and Random 63.3%. The key finding is that the original inputs can still substitute for the non-text hidden states at depth 3, retaining much of Original's accuracy and outperforming random directions (Sections 4.2–4.3, Figure 3(a)).   ### Concern 3. The narrative in the introduction that norm inflation is the cause of modality gap The manuscript does not claim that norm inflation causes the modality gap. The introduction presents prior reports of norm discrepancy in MLLMs as motivation. We define norm inflation as the increase in non-text token norms during alignment training (Section 1). We show that norms increase during optimization beyond the initial scales set by the encoder and projector, and investigate the role of these learned large norms. The earlier cross-modal alignment you describe is our interpretation of the representation changes observed after mitigating norm inflation with $\mathrm{TAN}_e$. We interpret the earlier departure from input directions together with higher audio–text cosine similarity as earlier formation of joint representations, and discuss these changes alongside task performance (Section 5.3, Table 2 and Figure 6).

I appreciate kind response from the authors regarding the question of RMSNorm and extra experiments on TAN_0. However, according to the response regarding concern 3, the early cross-modal alignment is the true cause behind the change of performance, it's still unclear if norm inflation has real dependency with this phenomenon, or simply a surface-level correlation. Based on this, I would tend to keep my original review score.

AuthorsOpenReview ↗

Response 2 to Reviewer j8u9

Thank you for your follow-up. Input norm causally affects the decoder’s representation trajectory and can thereby influence when joint audio–text representations form. As we explain in Sections 1 and 5.3, smaller residual norms give decoder updates more influence over token direction. Li et al. (2026) analyzed the same dependence for visual tokens in pre-norm MLLMs.

In the rescaling control, we change only the norm of the projected audio inputs while holding their directions and the model weights fixed. The changes in internal audio–text cosine similarity therefore arise through decoder updates (Appendix E.5). Experiment 2 also shows that changing the norm of replacement inputs at fixed directions affects prediction accuracy (Section 4.3).

ConditionInput (0)Early (8)Mid (16)Late (24)Mean ± SDLS-clean WER (%) ↓SpokenSQuAD F1 (%) ↑
Baseline0.0290.3400.5490.7670.510 ± 0.2143.2678.47
Baseline rescaling0.0290.7630.7480.7550.724 ± 0.136346.1521.75
Initialization only0.1070.6400.6750.7570.652 ± 0.1302.7686.53
Initialization rescaling0.1070.7140.7320.7720.704 ± 0.1223.6483.44
TANe\mathrm{TAN}_e (reference)0.0900.8020.7990.7950.752 ± 0.1272.8187.32

The cosine mean and SD are across depths 0–32. Baseline rescaling raises mean similarity while worsening transcription and spoken question answering. We introduced TAN in Experiment 3 to study the effects of mitigating norm inflation. We report the task gains under TANe\mathrm{TAN}_e alongside the representation changes as outcomes of that intervention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.