- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the role of text and non-text token norms in multimodal large language models (MLLMs). In particular, the authors observe a phenomenon they call non-text token norm inflation, where the norms of non-text tokens increase during modality alignment and become larger than those of text tokens. The authors analyze this phenomenon across different models and investigate its potential role in preserving non-text information through the decoder layers. They then introduce Text-Anchored Norm Regularization (TAN), which aims to mitigate this effect by aligning the norm statistics of non-text tokens with those of text tokens. They report improvements on several downstream audio tasks.
Critical strengths
The paper addresses an interesting research question. Understanding how the norms of text and non-text tokens evolve during multimodal alignment, how their distributions differ, and how they interact across decoder layers is certainly worth investigating. Moreover, the analysis goes beyond simply observing norm differences, including experiments on token replacement and the decomposition of representations into common and individual components. The proposed regularization is relatively simple, and the reported improvements on some downstream tasks are encouraging.
Critical weaknesses
My main concern is the overall execution and presentation of the study. I found the paper quite difficult to follow. Several concepts are repeated throughout the manuscript, sometimes without adding much to the discussion. In general, I struggled to identify a clear flow connecting the introduction, methodology, experiments, and conclusions.
The methodology section is particularly difficult to follow. While some mathematical definitions are provided, I think the overall formalization is insufficient. For instance, the expression (H^+ = H + F(\operatorname{Norm}(H))) is very general, since (F) can represent either attention or an FFN and the same is not linked to anything else later on in the text so even useless. In general, I would have appreciated a more precise description of the architecture, clearly defining the encoder, projector, decoder, and the representations being analyzed. Although these components are familiar in the MLLM literature, their specific roles and interactions should be made explicit in this work.
Another issue is the organization of the methodology. Several implementation and evaluation details are introduced in the Methods section, while other important details are only provided in the appendices. I think a clearer separation between the proposed methodology and experimental setup would considerably improve readability.
My main experimental concern relates to the training procedure. As stated in Section 3.1 and illustrated in Figure 1(a), the analysis of norm inflation during alignment is conducted by training only the projector, while keeping the encoder and decoder frozen. This is a reasonable setup for an initial investigation, but it only represents one stage of a typical MLLM training pipeline.
In many MLLMs, a subsequent alignment or instruction-tuning stage involves jointly training the projector and the LLM, and sometimes also the modality-specific encoder. What happens to the norms of text and non-text tokens during this stage? Does the same inflation behavior emerge, and does the proposed regularization remain beneficial?
To be precise, the authors do include a subsequent decoder LoRA fine-tuning stage in Section 3.3. However, I think the evolution of token norms during this stage deserves more attention, particularly considering the main research question of the paper. Minor problem: If also Lora finetuning is perfomed then Figure 1 is misleading. The only trainable module in that picture is the projector one. Readability and correctness of Figure 1 should be improved.
Finally, I find the interpretation of the results not entirely convincing. The authors associate reduced norm inflation with earlier cross-modal representation formation, mainly based on cosine similarity and token-replacement experiments. While these results are interesting, I am not sure they are sufficient to support this interpretation. Moreover, TAN is evaluated mainly on one audio alignment configuration, making it difficult to understand how general the proposed method actually is.
All in all, I think the paper starts from an interesting observation and investigates a relevant problem in MLLMs. However, I find the current execution rather weak for this prestigeous venue, both in terms of presentation, limited novelty and weak experimental validation. The difficulty in following the methodology, together with the limited investigation of different alignment stages, makes it hard for me to fully assess the significance and generality of the results.
Questions for authors
-
Why is the emergence of norm inflation investigated primarily under projector-only optimization? How do text and non-text token norms evolve during decoder LoRA fine-tuning or joint projector–decoder training?
-
What evidence establishes that increased inter-modal cosine similarity under TAN reflects improved semantic integration rather than a geometric effect of the regularizer?
Plus See Weaknesses
4 comments
Response to Reviewer GiYq (Part 2)
Concern 1. The overall execution and presentation of the study
We have organized the manuscript around three experiments: the emergence of norm inflation, its role in input preservation, and the effects of mitigating it (Sections 3–5). Shared definitions appear early in the paper (Section 1), and each experiment connects its question with the results and interpretation. Methodology and experimental setup are now separate (Sections 3.1–3.2, 4.1–4.2, and 5.1–5.2). Each Experimental Setup includes the settings needed to interpret that experiment, with detailed reproducibility information in the appendix (Appendices B–C).
Concern 2. The overall formalization is insufficient
We deliberately use a general formulation to emphasize that our analysis rests on a property shared by pre-norm decoders.
We have also specified that the encoder extracts features from non-text inputs and the projector maps those features into the decoder's input space. A token denotes the representation vector at a decoder sequence position. For non-text tokens, depth 0 denotes the projector output, and each subsequent depth denotes the corresponding decoder block's output (Section 1).
Concern 3. Figure 1 is misleading if LoRA fine-tuning is also performed
Subsequent decoder LoRA fine-tuning serves downstream task evaluation. Because it is not central to our methodology, Figure 1(a) focuses on the projector alignment used to investigate norm inflation. Fine-tuning used a low maximum learning rate of and a small training set of 15,000 examples, with the same procedure applied to all conditions, including Baseline. Experiment 3 specifies the settings (Section 5.2; Appendix B.1).
Concern 4. TAN is evaluated mainly on one audio alignment configuration
TAN is a simple regularizer that matches non-text token norm statistics to text references. Although evaluated here only in Qwen3.5-4B audio alignment, it can also be applied to other modalities. We chose audio because vision models are more widely available than audio-visual models, making audio adaptation of an existing VLM a useful small-scale setting under limited resources.
Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts. However, two main concerns remain.
1. Paper structure and organization.
I acknowledge that the readability of the manuscript has slightly improved. However, in my honest point of view, I still find the overall structure not so much appropriate for a scientific paper. I apologize if my original comment was not sufficiently clear. What I meant was a clear separation between methodology and experimental evaluation, rather than organizing the paper into three macro-sections titled "Experiment 1", "Experiment 2", and "Experiment 3", each containing its own methodology and experimental setup.
In my view, a more appropriate organization would include a unified methodology section presenting the framework, definitions, and proposed approach, followed by a separate experimental section presenting the evaluations and analyses. The current organization remains fragmented and does not adequately address my original concern.
2. The central research question.
The paper is titled "Why Does Non-Text Token Norm Inflation Emerge?" However, I am still not convinced that the work actually answers this question.
The experiments show that norm inflation occurs during alignment, that the common component contributes substantially to its early development, and that larger token norms can help preserve input representations. These are interesting observations, but demonstrating what norm inflation does is not the same as explaining why it emerges.
I want to say again that I find the overall phenomenon interesting and worthy of further investigation. However, at its current stage, the work mainly provides empirical observations without a sufficiently developed theoretical foundation. The proposed explanation then is plausible, but the experimental evidence does not establish the claimed causal mechanism.
I sincerely appreciate the authors' efforts in investigating this topic and in rebuttal phase and I hope they find my feedback constructive.
Response 2 to Reviewer GiYq
Thank you for clarifying your concern that the methodology is still presented in a fragmented way. We designed the study to examine how norm inflation emerges, what role it plays, and what changes when it is mitigated. We used a different method for each question. Had we been able to establish the causal mechanism directly, we could have presented a unified methodology and organized the experimental evaluation around that mechanism. We understand your concern about the paper’s organization to be related to how directly the study explains why norm inflation emerges.
We agree that demonstrating that large norms help preserve input representations does not by itself establish why norm inflation emerges. Although our study does not directly establish a causal mechanism for the emergence of norm inflation, the following steps show how we developed our explanation and tested its predictions.
-
In Experiment 1, we observed norm inflation when we used next-token prediction (NTP) to train projectors for feature alignment in multimodal LLMs.
-
The NTP objective does not explicitly favor large non-text token norms.
-
We hypothesized that large norms emerge because they help achieve a function “A” that benefits NTP optimization. During projector training, the norms of the projected inputs can change and thereby affect “A.” The benefit of “A” can therefore favor large norms during alignment. The title focuses on the emergence of norm inflation, the phenomenon we observe directly during training.
-
To identify “A,” we considered the following hypothesis: “Making the encoder’s already structured features compatible with the decoder can help the model rapidly reduce NTP loss during early alignment by learning associations between inputs and responses in the training data. Preserving the projected encoder features across decoder layers may help the model use the learned associations for prediction.”
-
In Experiment 2, we investigated input preservation as the proposed function “A.” We examined how closely non-text tokens retained their projected input directions and used input replacement with norm and direction controls to test the role of large norms in preserving the projected inputs.
-
With reduced input preservation, we expected decoder updates to have greater relative influence on the residual in early layers than under Baseline, so that non-text tokens would depart from their input directions at shallower depths and accuracy after input replacement would be lower.
-
In Experiment 3, we tested the predictions in step 6. Because input preservation cannot be manipulated in isolation in our training setup, we used TAN to mitigate norm inflation during alignment. We observed both expected changes under relative to Baseline, providing indirect empirical support for our explanation of why norm inflation emerges.
The current manuscript does not explicitly state the optimization hypothesis in step 4. We will state the hypothesis at the start of Experiment 2 to explain why preserving the decoder-compatible inputs discussed in Experiment 1 may benefit early optimization and how the potential benefit motivates testing the role of large norms in input preservation.
We appreciate your constructive feedback, which prompted us to clarify how the experiments address the central research question.
Response to Reviewer GiYq (Part 1)
Thank you for your detailed feedback on the paper's presentation, training procedure, and interpretation of its results. We provide a revised manuscript and plan to supplement its appendix with additional experimental results. Our responses to your questions and concerns follow.
Question 1. Projector-only optimization and subsequent training
We froze the encoder and decoder and trained only the projector to show that norm inflation already emerges during initial alignment. This is a standard setting for initial feature alignment in MLLMs. For vision in Experiment 1, we reinitialized and trained the projector between a jointly pretrained encoder and LLM. Norms grew rapidly, then decreased and stabilized above text norms. Audio also showed an initial increase followed by a stable norm trajectory (Sections 3.2–3.3, Figure 2).
To examine the effect of subsequent decoder training, we compared layer-wise mean norms before and after fine-tuning on the same 256 samples. With the projector frozen during LoRA fine-tuning, relative changes in both text and non-text norms were below 1% for Baseline, Initialization only, TAN0 (λ=0.1), and TANe. We also jointly trained the projector and the decoder's LoRA parameters for 625 steps from the TANe alignment checkpoint at a learning rate of 10−5. No norm regularizer was applied during this stage. The largest relative changes in layer-wise mean norm were +2.8% for non-text tokens and +0.9% for text tokens. In our setting, subsequent training changed the norms established during initial alignment only slightly.
Question 2. Semantic integration rather than a geometric effect of the regularizer
The manuscript discusses earlier formation of joint representations and does not use the term “semantic integration.” Smaller non-text norms can increase the relative influence of decoder updates on the residual and thereby change inter-modal cosine similarity (Section 5.3, Figure 6). We also reproduced an increase in cosine similarity through pure rescaling (Appendix E.5, Figure 20).
TANe learns projector representations under reduced norms, so the differences after training cannot be equated with post-hoc rescaling alone. To examine the representations and how the decoder uses them, we compared audio inputs with their transcript text embeddings and analyzed internal computation in a SpokenSQuAD case, as described in our response to Reviewer uyfY, Question 4. For the same recording, Initialization only and TANe differed in how representations of particular audio segments affected the answer. Selective interventions within each model confirmed these effects. The procedure and results are included in the revised manuscript (Section 5.3; Appendices F.1–F.2).